- Email: [email protected]
Impact of Residents on Surgical Outcomes in High-Complexity Procedures
Impact of Residents on Surgical Outcomes in High-Complexity Procedures Victor A Ferraris, MD, PhD, FACS, Jennifer W Harris, Sibu P Saha, MD, MBA, Eric D Endean, MD, FACS
MD, PhD,
Jeremiah T Martin,
MBBCh, FRCSI,
There are different views on the effects of resident involvement on surgical outcomes. We hypothesized that resident participation in surgical care does not appreciably alter outcomes. STUDY DESIGN: We analyzed an American College of Surgeons NSQIP subset of inpatients having procedures with high complexity, including 4 surgical specialties (general surgery, cardiothoracic surgery, neurosurgery, and vascular surgery) with the highest mean work relative value units. We evaluated surgical outcomes in patients having procedures performed by the attending surgeon alone, or by the attending surgeon with assistance from at least one surgical resident (PGY1 to PGY6). Outcomes measures included operative mortality, composite morbidity, and failure to rescue (FTR). Propensity-score matching minimized the effects of nonrandom assignment of residents to procedures. RESULTS: In 266,411 patients, unmatched comparisons showed significantly higher operative mortality and composite morbidity rates, but decreased FTR, in operations performed with resident involvement. After propensity-score matching, there were small but significant residentrelated increases in composite morbidity, but significant improvement in FTR. Senior-level resident involvement translated into improved outcomes, especially in cardiothoracic surgery procedures where >63.6% of procedures had PGY6 resident involvement. Resident involvement attenuated the significant worsening of operative mortality and FTR associated with multiple serious complications in individual patients. Measures of resource use increased modestly with resident involvement. CONCLUSIONS: We found substantial improvement in FTR with resident involvement, both in unmatched and propensity-matched comparisons. Senior-level resident participation seemed to attenuate, and even improve, surgical outcomes, despite slightly increased resource use. These results provide some reassurance about teaching paradigms. (J Am Coll Surg 2016;222:545e555. 2016 by the American College of Surgeons. Published by Elsevier Inc. All rights reserved.)
BACKGROUND:
The form and substance of resident engagement in surgical care changed dramatically during the last decade, especially with the advent of restricted duty hours. Reports surfaced that suggest benefit,1 risk,2-4 and no effect5-7 from resident involvement with surgical patients. Neither teaching faculty nor the residents themselves seem happy with all of the changes that occurred. There is a suggestion that residents think they are unprepared for independent practice and, in most cases, teaching faculty agree.8-11 Because of divergent physician perceptions and conflicting literature reports, we wondered how resident involvement impacted surgical care, especially in highcomplexity patients. If differences exist between procedures done with and without resident involvement, we
Disclosure Information: Nothing to disclose. Disclosures outside the scope of the current work: Dr Ferraris has performed CME events for CMEology/Baxter Healthcare and is on the advisory board of the Acelity division of KCI. Dr Saha has performed a research trial for CVRx. Disclaimer: American College of Surgeons NSQIP and the hospitals participating in the ACS NSQIP are the source of the data used herein; they have not verified and are not responsible for the statistical validity of the data analysis or the conclusions derived by the authors. Presented at the Southern Surgical Association 127th Annual Meeting, Hot Springs, VA, December 2015. Received December 18, 2015; Accepted December 21, 2015. From the Department of Surgery, University of Kentucky (Ferraris, Harris, Martin, Saha, Endean) and Department of Surgery, Lexington Veteran’s Affairs Medical Center (Ferraris), Lexington, KY. Correspondence address: Victor A Ferraris, MD, PhD, FACS, Department of Surgery, University of Kentucky, A301 Kentucky Clinic, 740 South Limestone, Lexington, KY 40536-0284. email: [email protected]
ª 2016 by the American College of Surgeons. Published by Elsevier Inc. All rights reserved.
545
http://dx.doi.org/10.1016/j.jamcollsurg.2015.12.056 ISSN 1072-7515/16
Ferraris et al
546
Surgical Outcomes and Resident Involvement
Abbreviations and Acronyms
ACS ¼ American College of Surgeons FTR ¼ failure to rescue wRVUs ¼ work-related relative value units
hypothesize that high-complexity procedures would be most likely to expose these differences. We used the American College of Surgeons (ACS) NSQIP database to identify patients with high complexity based on work-related relative value units (wRVUs). We selected the 4 surgical specialties with high wRVUs to obtain a high-complexity patient dataset. From this dataset, we assessed differences in operative mortality, morbidity, and failure to rescue (FTR) in procedures done with and without surgical resident involvement. We were particularly interested in ability of residents to manage patients with complications, believing that duty hour limits and new teaching paradigms might have the biggest impact on the care of the sickest patients, especially those requiring constant bedside care, often after normal duty hours. Importantly, postoperative mortality that follows development of perioperative complications is termed failure to rescue, and several authors suggest that FTR rates reflect hospital and provider quality.12-15 If resident involvement does alter surgical outcomes, we suspected that FTR might be a sensitive indicator of any resident-related benefit or deficit, if it exists.
METHODS Study population We used the ACS NSQIP database to identify patients having major operations with inpatient postoperative stays. The ACS NSQIP database contains patient deidentified information available to participants who sign and comply with the ACS NSQIP Data Use Agreement. The Data Use Agreement uses the data protections of the Health Insurance Portability and Accountability Act of 1996. We analyzed the ACS NSQIP participant use file containing surgical cases submitted by >300 acute care hospitals throughout the United States between 2008 and 2012. This database excludes trauma and pediatric patients. We excluded database patients with CPT codes listed as “procedure not otherwise specified” because of uncertainty about the type of procedures performed and because of the lack of associated wRVUs with the unspecified procedures. The study group included patients from the 4 surgical specialties in the database (ie, general surgery, cardiothoracic surgery, neurosurgery, and vascular surgery) with the high mean
J Am Coll Surg
total wRVUs. Additional exclusions from the analysis included patients with missing values in resident involvement database fields. Study design We evaluated surgical outcomes in patients having procedures performed by the attending surgeon alone, or by the attending surgeon with assistance from at least one surgical resident (PGY1 to PGY>7). Outcomes measures included operative mortality, composite morbidity, and FTR. Propensity-score matching minimized the effects of nonrandom assignment of residents to procedures. Measurement of effect sizes of outcomes differences estimated the clinical importance of significant group differences. We used total wRVUs, as recorded in ACS NSQIP, as an index of operation complexity.16 Outcomes measures We analyzed outcomes recorded in ACS NSQIP, including mortality within 30 days of operation or within the same hospitalization, individual morbidities (1 or more of 7 serious adverse events defined by the ACS NSQIP), composite morbidity consisting of any combination of the 7 serious individual morbidities, and FTR defined as death after development of any of the following 7 serious complications:
1. Wound complications: deep organ space surgical site infection, deep surgical wound infection, and wound dehiscence; 2. Pulmonary complications: pneumonia, unplanned intubation, pulmonary embolism with deep vein thrombosis, or mechanical ventilation exceeding 48 hours; 3. Renal complications: acute kidney injury or new renal failure requiring dialysis; 4. Central nervous system complications: new postoperative transient ischemic attack, stroke, or coma; 5. Cardiac complications: postoperative myocardial infarction or cardiac arrest; 6. Sepsis: postoperative systemic inflammatory response syndrome, septic shock, or blood-borne sepsis; and 7. Unplanned return to the operating room within 30 days of the initial procedure. Statistical comparisons Comparisons between operations done with and without resident involvement included assessment of differences in operative mortality, composite morbidity, FTR, resource use, and type of surgical specialty. Univariate statistics (chi-square and Student’s t-test) assessed differences between outcomes with and without resident involvement.
Vol. 222, No. 4, April 2016
Ferraris et al
A probability derived from univariate statistical tests of 0.05 implied a significant difference for all of the analyses. Because of the likelihood of detecting significant differences among the comparison groups, given the large sample size in the ACS NSQIP, we measure standardized effect sizes of most significant differences identified by statistical comparisons. We used previously published methods to measure the effect sizes of significantly different comparisons.17,18 The standardized effect size is independent of sample size and does not depend on the measurement units of the sample. Calculation of the standardized effect size is the ratio of SDs of 2 samples. Effect sizes of 0.2 are considered small, 0.5 are considered medium, and >0.8 are considered large. An effect size ratio of 0.2 measured between 2 significantly different comparisons suggests a possibly clinically unimportant, although statistically different, comparison. All analyses used IBM SPSS, version 23 (IBM Corp) for calculation of univariate statistics, regression models, and propensity matching. Multivariate determinates of serious postoperative complications Logistic regression models evaluated multivariate determinants of the development of any serious postoperative complication, of operative mortality, and of FTR. Independent variables used in the model included >50 ACS NSQIP variables that reflected patient demographic characteristics, preoperative risk indicators, wRVUs as an index of procedure complexity, and the level of resident involvement. A backward stepwise selection of prognostic variables identified significant predictors of the development of postoperative outcomes. The level of resident involvement was forced into the model to allow calculation of multivariate odds ratios of dependence of outcomes on resident involvement, and adjusted for other variables in the model. A probability value of 0.05 determined variable entry into the model and probability of 0.10 determined variable removal from final models. Propensity matching in patients having operations with and without resident involvement We used propensity-score matching to minimize the effects of confounding due to nonrandom assignment of residents to procedures when comparing outcomes between patients having operations with or without resident involvement.19,20 For this analysis, we derived propensity scores for each patient using a robust logistic regression model that included all preoperative risks in the NSQIP database entered into the regression equation. Exclusions
Surgical Outcomes and Resident Involvement
547
from the analysis included patients with incomplete preoperative risk variables (6.7% of patients). For the purposes of this propensity-score-matching analysis, we included resident involvement as the dependent variable to account for differences in this variable across the spectrum of risk. The IBM SPSS software matched patients with resident involvement to an equal number of patients without resident involvement using an algorithm that required matching on the propensity score (logistic regression derived) to the nearest 0.0001. Subsequent comparisons between propensity-matched groups used McNemar tests for comparison of discrete variables and paired t-tests for comparison of continuous variables.
RESULTS Study population and resident involvement After exclusions from the ACS NSQIP database, there were 266,411 patients with complete descriptions of resident involvement. Figure 1 displays the frequency of resident involvement for each of the 4 surgical specialties and for the total population. There was significantly less resident involvement in cardiothoracic procedures compared with other specialties. For example, residents participated in 50.1% of cardiothoracic procedures compared with 60.8% of neurosurgery procedures (p < 0.05 by chisquare testing with corrections for multiple comparisons). The study population is heavily weighted toward general surgical patients, with 77.1% of patient procedures performed by self-identified general surgeons. There were significant differences in the PGY level of residents who participated in the various specialty procedures. Cardiothoracic procedures were less likely to have junior-level residents involved in the operations and were more likely to have advanced-level residents, especially PGY6 residents, involved in the operations compared with other surgical specialties (Fig. 2). For example, residents in the PGY6 level participated in 63.6% of residentrelated cardiothoracic procedures, significantly more than 16.9% of general surgical procedures, 42.4% of neurosurgical procedures, and 45.6% of vascular surgical procedures (p < 0.001 by chi-square corrected for multiple comparisons). Multivariate risk predictors of resident-related operative outcomes We measured multivariate predictors of development of any of 3 serious postoperative complications (operative mortality, composite serious morbidity, and FTR) using logistic regression modeling. To assess the impact of resident contribution to outcomes, the level of resident involvement was forced into each of 3 separate regression
Ferraris et al
J Am Coll Surg
Surgical Outcomes and Resident Involvement
Percent resident involvement
80
300,000
70 250,000
60 200,000
50 40
150,000
30 100,000
Number of paƟents
548
20 50,000
10
0 Percent with resident invovlement Number of paƟents
CT 50.1
GS 67.3
NS 60.1
VS 65.4
Total 66
11,906
205,511
11,855
37,169
266,441
0
Figure 1. Resident involvement in high-complexity procedures. CT, cardiothoracic surgery; GS, general surgery; NS, neurosurgery; VS, vascular surgery.
models. The logistic regression coefficients allowed measurement of odds ratios for prediction of the impact of PGY level on each of the 3 serious postoperative outcomes. Figure 3 shows the resident-related multivariate odds ratios for each of these 3 serious postoperative outcomes. Logistic modeling summarized in Figure 3
suggests that resident involvement predicts significantly increased composite morbidity, especially with seniorlevel resident participation. This figure also shows that the level of resident involvement does not provide a significant contribution to operative mortality or to FTR in the logistic models.
Figure 2. Level of resident involvement in specialty procedures.
Vol. 222, No. 4, April 2016
Ferraris et al
Surgical Outcomes and Resident Involvement
549
Figure 3. Multivariate odds ratio for resident-related outcomes derived from logistic regression models of predictors of operative mortality, composite morbidity, and failure to rescue.
Unmatched and propensity-matched comparison of outcomes with and without resident involvement Table 1 summarizes the results of outcomes with and without resident involvement. Unmatched and propensityscore-matched comparisons measured resident-related differences in surgical outcomes for each of the 4 surgical specialties and for the entire study group. In unmatched and propensity-score-matched comparisons, resident involvement was associated with improved FTR across all specialties and in the total group. After propensity-score matching, significant differences in operative mortality between resident-related and attending-only procedures disappeared for the total study group and for general surgical, neurosurgical, and vascular surgical groups. Cardiothoracic surgical patients with resident involvement had an associated improved operative mortality in both unmatched and propensity-score-matched evaluations compared with patients with attending surgeon involvement only. Because there are differences in patient complexity within the study groups, we used propensity-score matching to derive a set of patients with equivalent perioperative complexity, differing in whether or not residents were involved in the index procedure. After propensity-score matching, there were 83,773 pairs of patients whose procedures were done with either resident involvement or with only attending surgeons. In propensity-score-matched comparisons, there was significantly decreased operative mortality, composite
morbidity, and FTR only for resident involvement in cardiothoracic procedures (Table 1). There was a small but significant increase in resident-related composite morbidity for the total study group and for neurosurgical and general surgical patients. Although these increases in morbidity with resident involvement are significant, the differences are small and do not meet the criteria for meaningful effect sizes. Effect sizes of >0.2 generally indicate clinically important differences,17,18,21 but the calculated effect size for these differences in composite morbidity range between 0.12 and 0.20, suggesting a modest, possibly clinically unimportant, risk of composite morbidity associated with resident involvement in propensity-score-matched comparisons.17,18,21 Effect of sequential complications on failure to rescue The ability to rescue patients with complications and to prevent additional complications varies with the type and timing of complications, with the procedure performed, and with the patients’ underlying risk. We evaluated the impact of resident involvement on development of one or more postoperative complications. There was a noticeable progression of complications with a few days intervening between each subsequent complication (Fig. 4). In our study group, there was usually a sentinel complication (frequently postoperative bleeding) that often foreshadowed other complications separated by a
550
Table 1.
Ferraris et al
J Am Coll Surg
Surgical Outcomes and Resident Involvement
Unmatched and Propensity Matched Measures of Resident Involvement on Surgical Outcomes
Measures
All procedures (mean wRVUs ¼ 46.7 20.7) Patients, n Operative mortality, % Composite morbidity, % Failure to rescue, % Cardiothoracic surgery (mean wRVUs ¼ 52.2 25.3) Patients, n Operative mortality, % Composite morbidity, % Failure to rescue, % General surgery (mean wRVUs ¼ 46.2 19.1) Patients, n Operative mortality, % Composite morbidity, % Failure to rescue, % Neurosurgery (mean wRVUs ¼ 50.1 23.1) Patients, n Operative mortality, % Composite morbidity, % Failure to rescue, % Vascular surgery (mean wRVUs ¼ 46.6 19.1) Patients, n Operative mortality, % Composite morbidity, % Failure to rescue, %
Percent outcomes Unmatched Propensity matched No resident Any resident No resident Any resident
90,484 3.7 19.4 14.5
175,957 3.9* 24.4* 12.7y
83,773 3.6 18.6 14.7
83,773 3.4 19.5* 13.5y
5,937 4.5 39.6 9.4
5,969 3.1y 39.4 7.3y
2,800 5.0 41.8 9.6
2,185 3.1y 36.0y 8.3y
67,048 3.3 16.7 15.1
138,463 3.6* 24.6* 12.8y
64,653 3.2 16.5 14.9
66,436 3.1 18.0* 13.4y
4,651 3.4 15.5 16.1
7,204 3.6 23.5* 12.6y
4,068 3.4 15.5 16.2
2,514 2.9 18.0* 11.5y
12,848 5.4 25.5 15.8
24,321 5.9* 32.7* 13.9y
12,252 5.3 25.1 15.6
12,638 5.3 24.8 15.5y
*Values suggest significantly worse outcomes associated with resident involvement. y Values suggest significantly better outcomes associated with resident involvement (all p < 0.001). wRVU, work-related relative value unit.
few days. This complication sequence resulted in increasing operative mortality as the number of complications increased. Of the 10,168 deaths in the study group, 78.6% of patients died after multiple serious postoperative complications developed in a sequential manner. The odds of perioperative death increased dramatically as the number of complications increased. Figure 5 shows the multivariate odds ratios for FTR derived from mortality logistic regression models both for resident-related procedures and for attending-only procedures. There are several compelling features of this graphic representation of the effect of sequential complications. If a patient has only one serious postoperative complication, there is no significant difference in the odds ratios for development of FTR with or without resident involvement. As the number of complications increases beyond a single complication, there is a progressive and significant separation between the odds ratios for FTR of resident-related procedures vs attending only procedures (Fig. 5).
Multiple complications translate into a significant decrease in the odds of FTR (ie, improved survival) in procedures with resident involvement compared with attending-only procedures. This suggests a favorable resident-related effect on FTR. Similar results that are shown in Table 1 support this observation. Resource use and resident involvement We measured 3 indices of resource use derived from the ACS NSQIP databasedoperating room time, hospital length of stay from operation to discharge or death, and total hospital length of stay. Tables 2 and 3 summarize the impact of resident involvement on resource use for the 4 surgical specialties and for the entire study group, both before and after propensity matching. With very few exceptions, resident involvement in procedures translated into significantly increased resource use using these measures.
Ferraris et al
Vol. 222, No. 4, April 2016
Surgical Outcomes and Resident Involvement
25
Days from operaƟon to complicaƟon
30000 20 25000
15 20000
15000 10
10000 5 5000
0
Days from operaƟon to onset of complicaƟon Number of paƟents with complicaƟon
1st complicaƟon
2nd complicaƟon
3rd complicaƟon
4th complicaƟon
5th or greater complicaƟon
5.6
9.1
10.3
11.9
14.9
32088
12807
6281
3629
3220
0
Figure 4. Timing and sequencing of multiple complications on failure to rescue.
Figure 5. Odds of failure to rescue as a function of sequential complications with and without resident involvement.
Number of paƟents with complicaƟon
35000
551
Ferraris et al
552
Table 2.
J Am Coll Surg
Surgical Outcomes and Resident Involvement
Unmatched Indices of Resource Use With and Without Resident Involvement in the Study Groups
Resource use
Operating room time, min Attend only Any resident Days from operation to discharge Attend only Any resident Total hospital length of stay, d Attending only Any resident
Cardiothoracic surgery
General surgery
Neurosurgery
Vascular surgery
All patients (n ¼ 266,441)
245 102 269 122*
152 97 199 119*
184 109 239 130*
188 101 236 119*
165 102 208 122*
8.4 7.9 8.7 8.5*
6.4 7.9 8.0 9.6*
5.2 6.9 6.3 8.0*
6.6 8.0 7.8 9.8*
6.5 7.9 7.9 9.6*
10.6 11.5 10.6 10.6
7.5 11.8 9.3 12.6*
6.3 8.7 7.5 11.3*
7.8 9.3 9.3 11.7*
7.7 11.3 9.3 12.4*
Date are presented as mean SD. *Comparisons show significantly increased resource use in resident-related procedures (all p < 0.05 with corrections for multiple comparisons).
DISCUSSION Resident involvement in high-complexity operations Our study encompassed a robust global assessment of outcomes of surgical procedures performed with and without resident involvement. The study aimed to evaluate highcomplexity procedures from the ACS NSQIP database to examine resident-related differences if they exist. After careful propensity-score matching for comorbidities and other complexity measures in a large patient cohort, we found small increases in composite morbidity in procedures performed with resident involvement. These differences, although statistically significant in the large dataset, might be clinically unimportant when subjected to effect size measurements. Interestingly, we found a statistically significant improved FTR without significant differences in operative mortality with resident involvement. In addition, our results summarized in Figure 4 suggest that an increased number of complications is associated with an increased risk of operative mortality. This association was partially attenuated when residents were involved in surgical care. These findings would seem to support and reaffirm the achievements of current teaching paradigms. Table 3.
A unique feature of the ACS NSQIP database is the collection of data from multiple surgical specialties with varying teaching behavior and with different resident profiles. Predictably, we found differences in resident-related patient outcomes among the 4 surgical specialties included in our study. The pattern of resident involvement in cardiothoracic surgical procedures is distinctly different than the other specialties included in our study. Figure 2 suggests that cardiothoracic procedures have very different resident profiles compared with other specialties in our study. More than 60% of the resident-involved cardiothoracic procedures had PGY6 residents participating in their procedures. No other specialty had such a high ratio of advanced-level residents involved in resident-related procedures. We suspect that this difference in seniority of resident involvement accounts for some of the specialty-related outcome variances seen in our analysis. Similar observations appear in the literature for other specialties.2,22-26 However, there are counter opinions suggesting that the experience of residents does not impact surgical outcomes,6,27-31 some of it from analysis of the ACS NSQIP database.6,29 The majority of these conflicting studies do not assess operative complexity.
Indices of Resource Use With and Without Resident Involvement in the Propensity-Matched Study Groups
Resource use
Operating room time, min Attend only Any resident Days from operation to discharge Attend only Any resident Total hospital length of stay, d Attending only Any resident
Cardiothoracic surgery
General surgery
Neurosurgery
Vascular surgery
All patients (n ¼ 167,546)
251 107 270 123*
152 96 187 108*
185 111 232 125*
188 100 232 114*
162 100 197 112*
8.5 10.0 8.2 10.4
6.3 8.7 7.4 9.9*
5.0 8.3 5.7 9.3*
6.4 8.6 7.7 10.8*
6.3 8.7 7.4 10.0*
10.6 10.0 10.0 12.1
7.30 12.3 8.6 12.1*
6.0 9.8 6.9 13.3*
7.6 9.9 9.1 12.4*
7.4 11.9 8.6 12.2*
*Comparisons show significantly increased resource use in resident-related procedures (all p < 0.05 with corrections for multiple comparisons).
Vol. 222, No. 4, April 2016
Ferraris et al
These varied results suggest a need for better understanding of the impact of the level of resident training on operative outcomes, especially in high-risk, high-complexity procedures. Our results suggest that better outcomes are associated with senior-level residents, but this observation warrants additional study. Implications of improved resident-related outcomes and the postgraduate-year level of resident involvement Using propensity-matched comparisons, we found a strong association between resident involvement in cardiothoracic operations and improved operative mortality, composite morbidity, and FTR rates. There was a 40% increased mortality in cardiothoracic procedures performed by attending surgeons alone compared with procedures done with resident involvement. Similarly, there was a 30% increased FTR rate and a 20% increase in composite morbidity associated with cardiothoracic procedures done with only attending surgeons present. The greater senior-level resident involvement in cardiothoracic procedures might have an impact on these improved outcomes. It is likely, and seems obvious, that the care environment differs between hospitals where surgical residents, especially senior-level residents, are present compared with those hospitals in which residents are not actively engaged in perioperative care. We suspect that resident involvement in perioperative care is a marker of this difference, not necessarily the cause of this difference. Multiple reports suggest that high-complexity interventions, like veno-venous extracorporeal circulation or left ventricular assist devices, which are more commonly present in teaching hospitals, can improve FTR rates.32,33 One study found that a more advanced ICU can be an important factor that contributes to better FTR rates.34 As a counter to these observations, Pucher35 and colleagues reviewed the factors that contribute to improving salvage from postoperative complications. Their findings suggest that process measures, such as low nurse to patient ratios, are more important than the effects of hospital size, resources, or subspecialist care availability. Although our observations about improved FTR outcomes in resident-related procedures are intriguing, the exact causes of these improved outcomes require more study and our observations must be considered hypothesis-generating, not a cause-andeffect relationship. Implications of resident-related differences in resource use We found improved FTR rates in resident-related operations (Table 1). We also noted significant increases in
Surgical Outcomes and Resident Involvement
553
indices of resource use associated with resident involvement. Many studies suggest that increased health care costs track with resident involvement.36-38 An extrapolation from our findings shown in Table 1 suggests that 1,005 more patients (1.2% of 83,773) might have survived in association with resident-involved vs attendingalone procedures. Table 3 suggests that this increased survival occurs at a mean cost of 1.1 excess postoperative days, 35 extra minutes time in the operating room, and of extra days of total hospital length of stay. The implication is that the added resource use might pay off in increased survival if residents are involved in perioperative care. Our results suggest that the increased costs associated with resident training can achieve a substantial return on investment by improving FTR rates and ultimately improving overall survival. This benefit might parallel involvement of senior-level residents. We suggest that more careful downstream assessment of resource use is justified, especially studying the impact of senior-level residents on patient resource use. Study strengths and weaknesses The ACS NSQIP database suffers from the shortcomings of most large databases. In the current database, missing values in certain variable fields limit the validity of conclusions. Results obtained must be considered hypothesis generating rather than firm inferences, and observations made with the database need additional testing in controlled trials. Nonetheless, the strength of the ACS NSQIP lies in the breadth and depth of participation. This database has proven accuracy, validity, and consistency from data extraction done by nurses who undergo rigorous training. The number of patients included in the database exceeds 2 million. There is no other comparable prospective clinical repository for patient information that encompasses multiple surgical specialties and includes the diversity of surgical procedures besides the ACS NSQIP. The database strength is its ability to uncover trends and to serve as a foundation for quality improvement. Our current study makes use of these imperatives, but we recognize that our results do not have the same impact as higher-level evidence, like randomized controlled trials or meta-analyses. It is important to state this caveat when describing results from analysis of the ACS NSQIP. How well the ACS NSQIP database reflects all surgical specialties is uncertain. For example, it is possible that the cross-section of cardiothoracic procedures used in our analysis is not a true sample of thoracic surgeons’ practice. There are >300 hospital practices represented in the ACS NSQIP database. We believe that the selected thoracic surgery practices included in the database are a
554
Ferraris et al
Surgical Outcomes and Resident Involvement
representative sample of all cardiothoracic procedures performed nationwide, but the possibility of selection bias is a concern, especially because other large cardiothoracicspecific databases exist. In addition, a significant fraction of hospitals in the database do not have thoracic surgeons, so there is referral pattern bias that limits random referral to thoracic surgeons for cardiothoracic operations. How these patterns of bias influence our results is uncertain, but other surgical specialties in the ACS NSQIP database have similar chances of biased results, with the possible exception of general surgery. We chose to study patients based on operative complexity as measured by wRVUs. This approach was used previously with some success for comparative purposes.16 Of the 4 surgical specialties, there are differences in the wRVU levels and these differences might skew some of our findings. The differences between wRVUs of each of the surgical specialties are small and the effect sizes of these differences are modest and might be clinically unimportant (see Table 1). Nonetheless, differences in the wRVUs of the 4 surgical specialties can introduce bias in favor of high or low wRVU specialties. We deliberately picked surgical specialties for our analysis that had similar wRVUs levels. Despite these concerns, we believe that the choice of wRVUs as an indicator of operative complexity is the best available option of the variables included in the ACS NSQIP database.
CONCLUSIONS We analyzed an ACS NSQIP database subset that reflects surgical outcomes with and without surgical resident involvement. Unmatched and propensity-score-matched comparisons showed a slight increase in resident-related composite morbidity and resource use, but a significant decrease in FTR rates. Our data suggest that resident involvement is associated with slightly increased composite morbidity, but the presence of resident participation portends improved FTR, such that overall mortality is not effected. Propensity-score matching showed that cardiothoracic procedures with resident involvement had significantly decreased operative mortality, composite morbidity, and FTR compared with procedures performed by attending surgeons alone. This pattern was different than residentrelated outcomes for the other surgical specialties in the study group. Residents involved in cardiothoracic operations were significantly more likely to be senior-level residents (PGY6) compared with other specialties. This preponderance of senior-level residents might explain some of the improved outcomes seen with resident involvement in cardiothoracic operations. These results
J Am Coll Surg
suggest that resident involvement in high-complexity procedures has significant benefit if senior-level residents are involved in these operations. Resident-related procedures are associated with increased resource use and the tradeoff for this increase translates to improved FTR rates and no effect on operative mortality. The consequences of this study offer some reassurance that resident involvement in surgical procedures provides safe and effective outcomes, but also suggests areas of additional study. Our analysis raises important questions about the level of resident training required to optimize outcomes in complicated patients and argues in favor of more experienced residents as an important ingredient in the mix that improves outcomes in these complex patients. Author Contributions Study conception and design: Ferraris, Harris, Endean Acquisition of data: Ferraris, Harris, Martin Analysis and interpretation of data: Ferraris, Harris, Martin, Saha, Endean Drafting of manuscript: Ferraris, Harris, Endean Critical revision: Ferraris, Harris, Saha, Martin, Endean REFERENCES 1. Seib CD, Greenblatt DY, Campbell MJ, et al. Adrenalectomy outcomes are superior with the participation of residents and fellows. J Am Coll Surg 2014;219:53e60. 2. Castleberry AW, Clary BM, Migaly J, et al. Resident education in the era of patient safety: a nationwide analysis of outcomes and complications in resident-assisted oncologic surgery. Ann Surg Oncol 2013;20:3715e3724. 3. Krell RW, Birkmeyer NJ, Reames BN, et al. Effects of resident involvement on complication rates after laparoscopic gastric bypass. J Am Coll Surg 2014;218:253e260. 4. Babu R, Thomas S, Hazzard MA, et al. Worse outcomes for patients undergoing brain tumor and cerebrovascular procedures following the ACGME resident duty-hour restrictions. J Neurosurg 2014;121:262e276. 5. Matulewicz RS, Pilecki M, Rambachan A, et al. Impact of resident involvement on urological surgery outcomes: an analysis of 40,000 patients from the ACS NSQIP database. J Urol 2014;192:885e890. 6. Venkat R, Valdivia PL, Guerrero MA. Resident participation and postoperative outcomes in adrenal surgery. J Surg Res 2014;190:559e564. 7. Norby K, Siddiq F, Adil MM, Haines SJ. The effect of duty hour regulations on outcomes of neurological surgery in training hospitals in the United States: duty hour regulations and patient outcomes. J Neurosurg 2014;121:247e261. 8. Tapia NM, Milewicz A, Whitney SE, et al. Identifying and eliminating deficiencies in the general surgery resident core competency curriculum. JAMA Surg 2014;149:514e518. 9. Harris JD, Staheli G, LeClere L, et al. What effects have resident work-hour changes had on education, quality of life, and safety? A systematic review. Clin Orthop Relat Res 2015;473: 1600e1608.
Ferraris et al
Vol. 222, No. 4, April 2016
10. Fitzgibbons SC, Chen J, Jagsi R, Weinstein D. Long-term follow-up on the educational impact of ACGME duty hour limits: a pre-post survey study. Ann Surg 2012;256: 1108e1112. 11. Yeo H, Viola K, Berg D, et al. Attitudes, training experiences, and professional expectations of US general surgery residents: a national survey. JAMA 2009;302:1301e1308. 12. Sheetz KH, Waits SA, Krell RW, et al. Improving mortality following emergent surgery in older patients requires focus on complication rescue. Ann Surg 2013;258:614e617; discussion 617e618. 13. Reddy HG, Shih T, Englesbe MJ, et al. Analyzing “failure to rescue”: is this an opportunity for outcome improvement in cardiac surgery? Ann Thorac Surg 2013;95:1976e1981; discussion 1981. 14. Ahmed EO, Butler R, Novick RJ. Failure-to-rescue rate as a measure of quality of care in a cardiac surgery recovery unit: a five-year study. Ann Thorac Surg 2014;97:147e152. 15. Ghaferi AA, Birkmeyer JD, Dimick JB. Complications, failure to rescue, and mortality with major inpatient surgery in medicare patients. Ann Surg 2009;250:1029e1034. 16. Davenport DL, Henderson WG, Khuri SF, Mentzer RM Jr. Preoperative risk factors and surgical complexity are more predictive of costs than postoperative complications: a case study using the National Surgical Quality Improvement Program (NSQIP) database. Ann Surg 2005;242:463e468; discussion 468e471. 17. Livingston EH, Elliot A, Hynan L, Cao J. Effect size estimation: a necessary component of statistical analysis. Arch Surg 2009;144:706e712. 18. McGlothlin AE, Lewis RJ. Minimal clinically important difference: defining what really matters to patients. JAMA 2014;312:1342e1343. 19. Blackstone EH. Comparing apples and oranges. J Thorac Cardiovasc Surg 2002;123:8e15. 20. Austin PC. Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006: a systematic review and suggestions for improvement. J Thorac Cardiovasc Surg 2007;134:1128e1135. 21. Copay AG, Subach BR, Glassman SD, et al. Understanding the minimum clinically important difference: a review of concepts and methods. Spine J 2007;7:541e546. 22. Heffernan MJ, Seehausen DA, Andras LM, Skaggs DL. Comparison of outcomes after posterior spinal fusion for adolescent idiopathic and neuromuscular scoliosis: does the surgical first assistant’s level of training matter? Spine 2014;39:648e655. 23. Beland MD, Anderson TJ, Atalay MK, et al. Resident experience increases diagnostic rate of thyroid fine-needle aspiration biopsies. Acad Radiol 2014;21:1490e1494. 24. Schoenfeld AJ, Serrano JA, Waterman BR, et al. The impact of resident involvement on post-operative morbidity and mortality following orthopaedic procedures: a study of 43,343 cases. Arch Orthop Trauma Surg 2013;133:1483e1491. 25. Hope WW, Hooks WB 3rd, Kilbourne SN, et al. Assessing resident performance and training of colonoscopy in a general surgery training program. Surg Endosc 2013;27: 1706e1710. 26. Advani V, Ahad S, Gonczy C, et al. Does resident involvement effect surgical times and complication rates during laparoscopic appendectomy for uncomplicated appendicitis? An analysis of 16,849 cases from the ACS-NSQIP. Am J Surg 2012;203:347e351; discussion 351e352.
Discussion
555
27. Ross SW, Oommen B, Kim M, et al. A little slower, but just as good: postgraduate year resident versus attending outcomes in laparoscopic ventral hernia repair. Surg Endosc 2014;28: 3092e3100. 28. Singh P, Turner EJ, Cornish J, et al. Safety assessment of resident grade and supervision level during emergency appendectomy: analysis of a multicenter, prospective study. Surgery 2014;156:28e38. 29. Monteiro R, Mino JS, Siperstein AE. Trends and disparities in education between specialties in thyroid and parathyroid surgery: an analysis of 55,402 NSQIP patients. Surgery 2013; 154:720e728; discussion 728e729. 30. Herrick BW, Yap RL. It is safe to teach residents laser prostate surgery in the private practice setting. Urology 2013;81: 629e632. 31. Naiditch JA, Lautz TB, Raval MV, et al. Effect of resident postgraduate year on outcomes after laparoscopic appendectomy for appendicitis in children. J Laparoendosc Adv Surg Tech A 2012;22:715e719. 32. Guirand DM, Okoye OT, Schmidt BS, et al. Venovenous extracorporeal life support improves survival in adult trauma patients with acute hypoxemic respiratory failure: a multicenter retrospective cohort study. J Trauma Acute Care Surg 2014;76:1275e1281. 33. Al-Adhami A, Macfie A, Mathieson C, et al. Ventricular assist devices as rescue therapy in cardiogenic shock after subarachnoid hemorrhage. Ann Thorac Surg 2014;97:1440e1443. 34. Henneman D, van Leersum NJ, Ten Berge M, et al. Failureto-rescue after colorectal cancer surgery and the association with three structural hospital factors. Ann Surg Oncol 2013; 20:3370e3376. 35. Pucher PH, Aggarwal R, Singh P, Darzi A. Enhancing surgical performance outcomes through process-driven care: a systematic review. World J Surg 2014;38:1362e1373. 36. Sasor SE, Flores RL, Wooden WA, Tholpady S. The cost of intraoperative plastic surgery education. J Surg Educ 2013; 70:655e659. 37. Pentiak PA, Schuch-Miller D, Streetman RT, et al. Barriers to adoption of the surgical resident skills curriculum of the American College of Surgeons/Association of Program Directors in Surgery. Surgery 2013;154:23e28. 38. Hwang CS, Wichterman KA, Alfrey EJ. The cost of resident education. J Surg Res 2010;163:18e23.
Discussion DR CARL HAISCH (Greenville, NC): I would like to compliment the authors on the huge amount of work and statistical analysis they have done. The authors have used the NSQIP database to analyze patient outcomes in high acuity procedures in general surgery, vascular, cardiothoracic, and neurosurgery. They have also reviewed the deaths and complications of these patients, including pulmonary failure, wound complications, central nervous system, cardiac complications, sepsis, or unplanned returns to the operating room (OR). They hypothesized that a failure to rescue patients with major complications would be a measure of resident benefit or deficit caused by the resident care. I have some methodologic
MD, PhD,
Jeremiah T Martin,
MBBCh, FRCSI,
There are different views on the effects of resident involvement on surgical outcomes. We hypothesized that resident participation in surgical care does not appreciably alter outcomes. STUDY DESIGN: We analyzed an American College of Surgeons NSQIP subset of inpatients having procedures with high complexity, including 4 surgical specialties (general surgery, cardiothoracic surgery, neurosurgery, and vascular surgery) with the highest mean work relative value units. We evaluated surgical outcomes in patients having procedures performed by the attending surgeon alone, or by the attending surgeon with assistance from at least one surgical resident (PGY1 to PGY6). Outcomes measures included operative mortality, composite morbidity, and failure to rescue (FTR). Propensity-score matching minimized the effects of nonrandom assignment of residents to procedures. RESULTS: In 266,411 patients, unmatched comparisons showed significantly higher operative mortality and composite morbidity rates, but decreased FTR, in operations performed with resident involvement. After propensity-score matching, there were small but significant residentrelated increases in composite morbidity, but significant improvement in FTR. Senior-level resident involvement translated into improved outcomes, especially in cardiothoracic surgery procedures where >63.6% of procedures had PGY6 resident involvement. Resident involvement attenuated the significant worsening of operative mortality and FTR associated with multiple serious complications in individual patients. Measures of resource use increased modestly with resident involvement. CONCLUSIONS: We found substantial improvement in FTR with resident involvement, both in unmatched and propensity-matched comparisons. Senior-level resident participation seemed to attenuate, and even improve, surgical outcomes, despite slightly increased resource use. These results provide some reassurance about teaching paradigms. (J Am Coll Surg 2016;222:545e555. 2016 by the American College of Surgeons. Published by Elsevier Inc. All rights reserved.)
BACKGROUND:
The form and substance of resident engagement in surgical care changed dramatically during the last decade, especially with the advent of restricted duty hours. Reports surfaced that suggest benefit,1 risk,2-4 and no effect5-7 from resident involvement with surgical patients. Neither teaching faculty nor the residents themselves seem happy with all of the changes that occurred. There is a suggestion that residents think they are unprepared for independent practice and, in most cases, teaching faculty agree.8-11 Because of divergent physician perceptions and conflicting literature reports, we wondered how resident involvement impacted surgical care, especially in highcomplexity patients. If differences exist between procedures done with and without resident involvement, we
Disclosure Information: Nothing to disclose. Disclosures outside the scope of the current work: Dr Ferraris has performed CME events for CMEology/Baxter Healthcare and is on the advisory board of the Acelity division of KCI. Dr Saha has performed a research trial for CVRx. Disclaimer: American College of Surgeons NSQIP and the hospitals participating in the ACS NSQIP are the source of the data used herein; they have not verified and are not responsible for the statistical validity of the data analysis or the conclusions derived by the authors. Presented at the Southern Surgical Association 127th Annual Meeting, Hot Springs, VA, December 2015. Received December 18, 2015; Accepted December 21, 2015. From the Department of Surgery, University of Kentucky (Ferraris, Harris, Martin, Saha, Endean) and Department of Surgery, Lexington Veteran’s Affairs Medical Center (Ferraris), Lexington, KY. Correspondence address: Victor A Ferraris, MD, PhD, FACS, Department of Surgery, University of Kentucky, A301 Kentucky Clinic, 740 South Limestone, Lexington, KY 40536-0284. email: [email protected]
ª 2016 by the American College of Surgeons. Published by Elsevier Inc. All rights reserved.
545
http://dx.doi.org/10.1016/j.jamcollsurg.2015.12.056 ISSN 1072-7515/16
Ferraris et al
546
Surgical Outcomes and Resident Involvement
Abbreviations and Acronyms
ACS ¼ American College of Surgeons FTR ¼ failure to rescue wRVUs ¼ work-related relative value units
hypothesize that high-complexity procedures would be most likely to expose these differences. We used the American College of Surgeons (ACS) NSQIP database to identify patients with high complexity based on work-related relative value units (wRVUs). We selected the 4 surgical specialties with high wRVUs to obtain a high-complexity patient dataset. From this dataset, we assessed differences in operative mortality, morbidity, and failure to rescue (FTR) in procedures done with and without surgical resident involvement. We were particularly interested in ability of residents to manage patients with complications, believing that duty hour limits and new teaching paradigms might have the biggest impact on the care of the sickest patients, especially those requiring constant bedside care, often after normal duty hours. Importantly, postoperative mortality that follows development of perioperative complications is termed failure to rescue, and several authors suggest that FTR rates reflect hospital and provider quality.12-15 If resident involvement does alter surgical outcomes, we suspected that FTR might be a sensitive indicator of any resident-related benefit or deficit, if it exists.
METHODS Study population We used the ACS NSQIP database to identify patients having major operations with inpatient postoperative stays. The ACS NSQIP database contains patient deidentified information available to participants who sign and comply with the ACS NSQIP Data Use Agreement. The Data Use Agreement uses the data protections of the Health Insurance Portability and Accountability Act of 1996. We analyzed the ACS NSQIP participant use file containing surgical cases submitted by >300 acute care hospitals throughout the United States between 2008 and 2012. This database excludes trauma and pediatric patients. We excluded database patients with CPT codes listed as “procedure not otherwise specified” because of uncertainty about the type of procedures performed and because of the lack of associated wRVUs with the unspecified procedures. The study group included patients from the 4 surgical specialties in the database (ie, general surgery, cardiothoracic surgery, neurosurgery, and vascular surgery) with the high mean
J Am Coll Surg
total wRVUs. Additional exclusions from the analysis included patients with missing values in resident involvement database fields. Study design We evaluated surgical outcomes in patients having procedures performed by the attending surgeon alone, or by the attending surgeon with assistance from at least one surgical resident (PGY1 to PGY>7). Outcomes measures included operative mortality, composite morbidity, and FTR. Propensity-score matching minimized the effects of nonrandom assignment of residents to procedures. Measurement of effect sizes of outcomes differences estimated the clinical importance of significant group differences. We used total wRVUs, as recorded in ACS NSQIP, as an index of operation complexity.16 Outcomes measures We analyzed outcomes recorded in ACS NSQIP, including mortality within 30 days of operation or within the same hospitalization, individual morbidities (1 or more of 7 serious adverse events defined by the ACS NSQIP), composite morbidity consisting of any combination of the 7 serious individual morbidities, and FTR defined as death after development of any of the following 7 serious complications:
1. Wound complications: deep organ space surgical site infection, deep surgical wound infection, and wound dehiscence; 2. Pulmonary complications: pneumonia, unplanned intubation, pulmonary embolism with deep vein thrombosis, or mechanical ventilation exceeding 48 hours; 3. Renal complications: acute kidney injury or new renal failure requiring dialysis; 4. Central nervous system complications: new postoperative transient ischemic attack, stroke, or coma; 5. Cardiac complications: postoperative myocardial infarction or cardiac arrest; 6. Sepsis: postoperative systemic inflammatory response syndrome, septic shock, or blood-borne sepsis; and 7. Unplanned return to the operating room within 30 days of the initial procedure. Statistical comparisons Comparisons between operations done with and without resident involvement included assessment of differences in operative mortality, composite morbidity, FTR, resource use, and type of surgical specialty. Univariate statistics (chi-square and Student’s t-test) assessed differences between outcomes with and without resident involvement.
Vol. 222, No. 4, April 2016
Ferraris et al
A probability derived from univariate statistical tests of 0.05 implied a significant difference for all of the analyses. Because of the likelihood of detecting significant differences among the comparison groups, given the large sample size in the ACS NSQIP, we measure standardized effect sizes of most significant differences identified by statistical comparisons. We used previously published methods to measure the effect sizes of significantly different comparisons.17,18 The standardized effect size is independent of sample size and does not depend on the measurement units of the sample. Calculation of the standardized effect size is the ratio of SDs of 2 samples. Effect sizes of 0.2 are considered small, 0.5 are considered medium, and >0.8 are considered large. An effect size ratio of 0.2 measured between 2 significantly different comparisons suggests a possibly clinically unimportant, although statistically different, comparison. All analyses used IBM SPSS, version 23 (IBM Corp) for calculation of univariate statistics, regression models, and propensity matching. Multivariate determinates of serious postoperative complications Logistic regression models evaluated multivariate determinants of the development of any serious postoperative complication, of operative mortality, and of FTR. Independent variables used in the model included >50 ACS NSQIP variables that reflected patient demographic characteristics, preoperative risk indicators, wRVUs as an index of procedure complexity, and the level of resident involvement. A backward stepwise selection of prognostic variables identified significant predictors of the development of postoperative outcomes. The level of resident involvement was forced into the model to allow calculation of multivariate odds ratios of dependence of outcomes on resident involvement, and adjusted for other variables in the model. A probability value of 0.05 determined variable entry into the model and probability of 0.10 determined variable removal from final models. Propensity matching in patients having operations with and without resident involvement We used propensity-score matching to minimize the effects of confounding due to nonrandom assignment of residents to procedures when comparing outcomes between patients having operations with or without resident involvement.19,20 For this analysis, we derived propensity scores for each patient using a robust logistic regression model that included all preoperative risks in the NSQIP database entered into the regression equation. Exclusions
Surgical Outcomes and Resident Involvement
547
from the analysis included patients with incomplete preoperative risk variables (6.7% of patients). For the purposes of this propensity-score-matching analysis, we included resident involvement as the dependent variable to account for differences in this variable across the spectrum of risk. The IBM SPSS software matched patients with resident involvement to an equal number of patients without resident involvement using an algorithm that required matching on the propensity score (logistic regression derived) to the nearest 0.0001. Subsequent comparisons between propensity-matched groups used McNemar tests for comparison of discrete variables and paired t-tests for comparison of continuous variables.
RESULTS Study population and resident involvement After exclusions from the ACS NSQIP database, there were 266,411 patients with complete descriptions of resident involvement. Figure 1 displays the frequency of resident involvement for each of the 4 surgical specialties and for the total population. There was significantly less resident involvement in cardiothoracic procedures compared with other specialties. For example, residents participated in 50.1% of cardiothoracic procedures compared with 60.8% of neurosurgery procedures (p < 0.05 by chisquare testing with corrections for multiple comparisons). The study population is heavily weighted toward general surgical patients, with 77.1% of patient procedures performed by self-identified general surgeons. There were significant differences in the PGY level of residents who participated in the various specialty procedures. Cardiothoracic procedures were less likely to have junior-level residents involved in the operations and were more likely to have advanced-level residents, especially PGY6 residents, involved in the operations compared with other surgical specialties (Fig. 2). For example, residents in the PGY6 level participated in 63.6% of residentrelated cardiothoracic procedures, significantly more than 16.9% of general surgical procedures, 42.4% of neurosurgical procedures, and 45.6% of vascular surgical procedures (p < 0.001 by chi-square corrected for multiple comparisons). Multivariate risk predictors of resident-related operative outcomes We measured multivariate predictors of development of any of 3 serious postoperative complications (operative mortality, composite serious morbidity, and FTR) using logistic regression modeling. To assess the impact of resident contribution to outcomes, the level of resident involvement was forced into each of 3 separate regression
Ferraris et al
J Am Coll Surg
Surgical Outcomes and Resident Involvement
Percent resident involvement
80
300,000
70 250,000
60 200,000
50 40
150,000
30 100,000
Number of paƟents
548
20 50,000
10
0 Percent with resident invovlement Number of paƟents
CT 50.1
GS 67.3
NS 60.1
VS 65.4
Total 66
11,906
205,511
11,855
37,169
266,441
0
Figure 1. Resident involvement in high-complexity procedures. CT, cardiothoracic surgery; GS, general surgery; NS, neurosurgery; VS, vascular surgery.
models. The logistic regression coefficients allowed measurement of odds ratios for prediction of the impact of PGY level on each of the 3 serious postoperative outcomes. Figure 3 shows the resident-related multivariate odds ratios for each of these 3 serious postoperative outcomes. Logistic modeling summarized in Figure 3
suggests that resident involvement predicts significantly increased composite morbidity, especially with seniorlevel resident participation. This figure also shows that the level of resident involvement does not provide a significant contribution to operative mortality or to FTR in the logistic models.
Figure 2. Level of resident involvement in specialty procedures.
Vol. 222, No. 4, April 2016
Ferraris et al
Surgical Outcomes and Resident Involvement
549
Figure 3. Multivariate odds ratio for resident-related outcomes derived from logistic regression models of predictors of operative mortality, composite morbidity, and failure to rescue.
Unmatched and propensity-matched comparison of outcomes with and without resident involvement Table 1 summarizes the results of outcomes with and without resident involvement. Unmatched and propensityscore-matched comparisons measured resident-related differences in surgical outcomes for each of the 4 surgical specialties and for the entire study group. In unmatched and propensity-score-matched comparisons, resident involvement was associated with improved FTR across all specialties and in the total group. After propensity-score matching, significant differences in operative mortality between resident-related and attending-only procedures disappeared for the total study group and for general surgical, neurosurgical, and vascular surgical groups. Cardiothoracic surgical patients with resident involvement had an associated improved operative mortality in both unmatched and propensity-score-matched evaluations compared with patients with attending surgeon involvement only. Because there are differences in patient complexity within the study groups, we used propensity-score matching to derive a set of patients with equivalent perioperative complexity, differing in whether or not residents were involved in the index procedure. After propensity-score matching, there were 83,773 pairs of patients whose procedures were done with either resident involvement or with only attending surgeons. In propensity-score-matched comparisons, there was significantly decreased operative mortality, composite
morbidity, and FTR only for resident involvement in cardiothoracic procedures (Table 1). There was a small but significant increase in resident-related composite morbidity for the total study group and for neurosurgical and general surgical patients. Although these increases in morbidity with resident involvement are significant, the differences are small and do not meet the criteria for meaningful effect sizes. Effect sizes of >0.2 generally indicate clinically important differences,17,18,21 but the calculated effect size for these differences in composite morbidity range between 0.12 and 0.20, suggesting a modest, possibly clinically unimportant, risk of composite morbidity associated with resident involvement in propensity-score-matched comparisons.17,18,21 Effect of sequential complications on failure to rescue The ability to rescue patients with complications and to prevent additional complications varies with the type and timing of complications, with the procedure performed, and with the patients’ underlying risk. We evaluated the impact of resident involvement on development of one or more postoperative complications. There was a noticeable progression of complications with a few days intervening between each subsequent complication (Fig. 4). In our study group, there was usually a sentinel complication (frequently postoperative bleeding) that often foreshadowed other complications separated by a
550
Table 1.
Ferraris et al
J Am Coll Surg
Surgical Outcomes and Resident Involvement
Unmatched and Propensity Matched Measures of Resident Involvement on Surgical Outcomes
Measures
All procedures (mean wRVUs ¼ 46.7 20.7) Patients, n Operative mortality, % Composite morbidity, % Failure to rescue, % Cardiothoracic surgery (mean wRVUs ¼ 52.2 25.3) Patients, n Operative mortality, % Composite morbidity, % Failure to rescue, % General surgery (mean wRVUs ¼ 46.2 19.1) Patients, n Operative mortality, % Composite morbidity, % Failure to rescue, % Neurosurgery (mean wRVUs ¼ 50.1 23.1) Patients, n Operative mortality, % Composite morbidity, % Failure to rescue, % Vascular surgery (mean wRVUs ¼ 46.6 19.1) Patients, n Operative mortality, % Composite morbidity, % Failure to rescue, %
Percent outcomes Unmatched Propensity matched No resident Any resident No resident Any resident
90,484 3.7 19.4 14.5
175,957 3.9* 24.4* 12.7y
83,773 3.6 18.6 14.7
83,773 3.4 19.5* 13.5y
5,937 4.5 39.6 9.4
5,969 3.1y 39.4 7.3y
2,800 5.0 41.8 9.6
2,185 3.1y 36.0y 8.3y
67,048 3.3 16.7 15.1
138,463 3.6* 24.6* 12.8y
64,653 3.2 16.5 14.9
66,436 3.1 18.0* 13.4y
4,651 3.4 15.5 16.1
7,204 3.6 23.5* 12.6y
4,068 3.4 15.5 16.2
2,514 2.9 18.0* 11.5y
12,848 5.4 25.5 15.8
24,321 5.9* 32.7* 13.9y
12,252 5.3 25.1 15.6
12,638 5.3 24.8 15.5y
*Values suggest significantly worse outcomes associated with resident involvement. y Values suggest significantly better outcomes associated with resident involvement (all p < 0.001). wRVU, work-related relative value unit.
few days. This complication sequence resulted in increasing operative mortality as the number of complications increased. Of the 10,168 deaths in the study group, 78.6% of patients died after multiple serious postoperative complications developed in a sequential manner. The odds of perioperative death increased dramatically as the number of complications increased. Figure 5 shows the multivariate odds ratios for FTR derived from mortality logistic regression models both for resident-related procedures and for attending-only procedures. There are several compelling features of this graphic representation of the effect of sequential complications. If a patient has only one serious postoperative complication, there is no significant difference in the odds ratios for development of FTR with or without resident involvement. As the number of complications increases beyond a single complication, there is a progressive and significant separation between the odds ratios for FTR of resident-related procedures vs attending only procedures (Fig. 5).
Multiple complications translate into a significant decrease in the odds of FTR (ie, improved survival) in procedures with resident involvement compared with attending-only procedures. This suggests a favorable resident-related effect on FTR. Similar results that are shown in Table 1 support this observation. Resource use and resident involvement We measured 3 indices of resource use derived from the ACS NSQIP databasedoperating room time, hospital length of stay from operation to discharge or death, and total hospital length of stay. Tables 2 and 3 summarize the impact of resident involvement on resource use for the 4 surgical specialties and for the entire study group, both before and after propensity matching. With very few exceptions, resident involvement in procedures translated into significantly increased resource use using these measures.
Ferraris et al
Vol. 222, No. 4, April 2016
Surgical Outcomes and Resident Involvement
25
Days from operaƟon to complicaƟon
30000 20 25000
15 20000
15000 10
10000 5 5000
0
Days from operaƟon to onset of complicaƟon Number of paƟents with complicaƟon
1st complicaƟon
2nd complicaƟon
3rd complicaƟon
4th complicaƟon
5th or greater complicaƟon
5.6
9.1
10.3
11.9
14.9
32088
12807
6281
3629
3220
0
Figure 4. Timing and sequencing of multiple complications on failure to rescue.
Figure 5. Odds of failure to rescue as a function of sequential complications with and without resident involvement.
Number of paƟents with complicaƟon
35000
551
Ferraris et al
552
Table 2.
J Am Coll Surg
Surgical Outcomes and Resident Involvement
Unmatched Indices of Resource Use With and Without Resident Involvement in the Study Groups
Resource use
Operating room time, min Attend only Any resident Days from operation to discharge Attend only Any resident Total hospital length of stay, d Attending only Any resident
Cardiothoracic surgery
General surgery
Neurosurgery
Vascular surgery
All patients (n ¼ 266,441)
245 102 269 122*
152 97 199 119*
184 109 239 130*
188 101 236 119*
165 102 208 122*
8.4 7.9 8.7 8.5*
6.4 7.9 8.0 9.6*
5.2 6.9 6.3 8.0*
6.6 8.0 7.8 9.8*
6.5 7.9 7.9 9.6*
10.6 11.5 10.6 10.6
7.5 11.8 9.3 12.6*
6.3 8.7 7.5 11.3*
7.8 9.3 9.3 11.7*
7.7 11.3 9.3 12.4*
Date are presented as mean SD. *Comparisons show significantly increased resource use in resident-related procedures (all p < 0.05 with corrections for multiple comparisons).
DISCUSSION Resident involvement in high-complexity operations Our study encompassed a robust global assessment of outcomes of surgical procedures performed with and without resident involvement. The study aimed to evaluate highcomplexity procedures from the ACS NSQIP database to examine resident-related differences if they exist. After careful propensity-score matching for comorbidities and other complexity measures in a large patient cohort, we found small increases in composite morbidity in procedures performed with resident involvement. These differences, although statistically significant in the large dataset, might be clinically unimportant when subjected to effect size measurements. Interestingly, we found a statistically significant improved FTR without significant differences in operative mortality with resident involvement. In addition, our results summarized in Figure 4 suggest that an increased number of complications is associated with an increased risk of operative mortality. This association was partially attenuated when residents were involved in surgical care. These findings would seem to support and reaffirm the achievements of current teaching paradigms. Table 3.
A unique feature of the ACS NSQIP database is the collection of data from multiple surgical specialties with varying teaching behavior and with different resident profiles. Predictably, we found differences in resident-related patient outcomes among the 4 surgical specialties included in our study. The pattern of resident involvement in cardiothoracic surgical procedures is distinctly different than the other specialties included in our study. Figure 2 suggests that cardiothoracic procedures have very different resident profiles compared with other specialties in our study. More than 60% of the resident-involved cardiothoracic procedures had PGY6 residents participating in their procedures. No other specialty had such a high ratio of advanced-level residents involved in resident-related procedures. We suspect that this difference in seniority of resident involvement accounts for some of the specialty-related outcome variances seen in our analysis. Similar observations appear in the literature for other specialties.2,22-26 However, there are counter opinions suggesting that the experience of residents does not impact surgical outcomes,6,27-31 some of it from analysis of the ACS NSQIP database.6,29 The majority of these conflicting studies do not assess operative complexity.
Indices of Resource Use With and Without Resident Involvement in the Propensity-Matched Study Groups
Resource use
Operating room time, min Attend only Any resident Days from operation to discharge Attend only Any resident Total hospital length of stay, d Attending only Any resident
Cardiothoracic surgery
General surgery
Neurosurgery
Vascular surgery
All patients (n ¼ 167,546)
251 107 270 123*
152 96 187 108*
185 111 232 125*
188 100 232 114*
162 100 197 112*
8.5 10.0 8.2 10.4
6.3 8.7 7.4 9.9*
5.0 8.3 5.7 9.3*
6.4 8.6 7.7 10.8*
6.3 8.7 7.4 10.0*
10.6 10.0 10.0 12.1
7.30 12.3 8.6 12.1*
6.0 9.8 6.9 13.3*
7.6 9.9 9.1 12.4*
7.4 11.9 8.6 12.2*
*Comparisons show significantly increased resource use in resident-related procedures (all p < 0.05 with corrections for multiple comparisons).
Vol. 222, No. 4, April 2016
Ferraris et al
These varied results suggest a need for better understanding of the impact of the level of resident training on operative outcomes, especially in high-risk, high-complexity procedures. Our results suggest that better outcomes are associated with senior-level residents, but this observation warrants additional study. Implications of improved resident-related outcomes and the postgraduate-year level of resident involvement Using propensity-matched comparisons, we found a strong association between resident involvement in cardiothoracic operations and improved operative mortality, composite morbidity, and FTR rates. There was a 40% increased mortality in cardiothoracic procedures performed by attending surgeons alone compared with procedures done with resident involvement. Similarly, there was a 30% increased FTR rate and a 20% increase in composite morbidity associated with cardiothoracic procedures done with only attending surgeons present. The greater senior-level resident involvement in cardiothoracic procedures might have an impact on these improved outcomes. It is likely, and seems obvious, that the care environment differs between hospitals where surgical residents, especially senior-level residents, are present compared with those hospitals in which residents are not actively engaged in perioperative care. We suspect that resident involvement in perioperative care is a marker of this difference, not necessarily the cause of this difference. Multiple reports suggest that high-complexity interventions, like veno-venous extracorporeal circulation or left ventricular assist devices, which are more commonly present in teaching hospitals, can improve FTR rates.32,33 One study found that a more advanced ICU can be an important factor that contributes to better FTR rates.34 As a counter to these observations, Pucher35 and colleagues reviewed the factors that contribute to improving salvage from postoperative complications. Their findings suggest that process measures, such as low nurse to patient ratios, are more important than the effects of hospital size, resources, or subspecialist care availability. Although our observations about improved FTR outcomes in resident-related procedures are intriguing, the exact causes of these improved outcomes require more study and our observations must be considered hypothesis-generating, not a cause-andeffect relationship. Implications of resident-related differences in resource use We found improved FTR rates in resident-related operations (Table 1). We also noted significant increases in
Surgical Outcomes and Resident Involvement
553
indices of resource use associated with resident involvement. Many studies suggest that increased health care costs track with resident involvement.36-38 An extrapolation from our findings shown in Table 1 suggests that 1,005 more patients (1.2% of 83,773) might have survived in association with resident-involved vs attendingalone procedures. Table 3 suggests that this increased survival occurs at a mean cost of 1.1 excess postoperative days, 35 extra minutes time in the operating room, and of extra days of total hospital length of stay. The implication is that the added resource use might pay off in increased survival if residents are involved in perioperative care. Our results suggest that the increased costs associated with resident training can achieve a substantial return on investment by improving FTR rates and ultimately improving overall survival. This benefit might parallel involvement of senior-level residents. We suggest that more careful downstream assessment of resource use is justified, especially studying the impact of senior-level residents on patient resource use. Study strengths and weaknesses The ACS NSQIP database suffers from the shortcomings of most large databases. In the current database, missing values in certain variable fields limit the validity of conclusions. Results obtained must be considered hypothesis generating rather than firm inferences, and observations made with the database need additional testing in controlled trials. Nonetheless, the strength of the ACS NSQIP lies in the breadth and depth of participation. This database has proven accuracy, validity, and consistency from data extraction done by nurses who undergo rigorous training. The number of patients included in the database exceeds 2 million. There is no other comparable prospective clinical repository for patient information that encompasses multiple surgical specialties and includes the diversity of surgical procedures besides the ACS NSQIP. The database strength is its ability to uncover trends and to serve as a foundation for quality improvement. Our current study makes use of these imperatives, but we recognize that our results do not have the same impact as higher-level evidence, like randomized controlled trials or meta-analyses. It is important to state this caveat when describing results from analysis of the ACS NSQIP. How well the ACS NSQIP database reflects all surgical specialties is uncertain. For example, it is possible that the cross-section of cardiothoracic procedures used in our analysis is not a true sample of thoracic surgeons’ practice. There are >300 hospital practices represented in the ACS NSQIP database. We believe that the selected thoracic surgery practices included in the database are a
554
Ferraris et al
Surgical Outcomes and Resident Involvement
representative sample of all cardiothoracic procedures performed nationwide, but the possibility of selection bias is a concern, especially because other large cardiothoracicspecific databases exist. In addition, a significant fraction of hospitals in the database do not have thoracic surgeons, so there is referral pattern bias that limits random referral to thoracic surgeons for cardiothoracic operations. How these patterns of bias influence our results is uncertain, but other surgical specialties in the ACS NSQIP database have similar chances of biased results, with the possible exception of general surgery. We chose to study patients based on operative complexity as measured by wRVUs. This approach was used previously with some success for comparative purposes.16 Of the 4 surgical specialties, there are differences in the wRVU levels and these differences might skew some of our findings. The differences between wRVUs of each of the surgical specialties are small and the effect sizes of these differences are modest and might be clinically unimportant (see Table 1). Nonetheless, differences in the wRVUs of the 4 surgical specialties can introduce bias in favor of high or low wRVU specialties. We deliberately picked surgical specialties for our analysis that had similar wRVUs levels. Despite these concerns, we believe that the choice of wRVUs as an indicator of operative complexity is the best available option of the variables included in the ACS NSQIP database.
CONCLUSIONS We analyzed an ACS NSQIP database subset that reflects surgical outcomes with and without surgical resident involvement. Unmatched and propensity-score-matched comparisons showed a slight increase in resident-related composite morbidity and resource use, but a significant decrease in FTR rates. Our data suggest that resident involvement is associated with slightly increased composite morbidity, but the presence of resident participation portends improved FTR, such that overall mortality is not effected. Propensity-score matching showed that cardiothoracic procedures with resident involvement had significantly decreased operative mortality, composite morbidity, and FTR compared with procedures performed by attending surgeons alone. This pattern was different than residentrelated outcomes for the other surgical specialties in the study group. Residents involved in cardiothoracic operations were significantly more likely to be senior-level residents (PGY6) compared with other specialties. This preponderance of senior-level residents might explain some of the improved outcomes seen with resident involvement in cardiothoracic operations. These results
J Am Coll Surg
suggest that resident involvement in high-complexity procedures has significant benefit if senior-level residents are involved in these operations. Resident-related procedures are associated with increased resource use and the tradeoff for this increase translates to improved FTR rates and no effect on operative mortality. The consequences of this study offer some reassurance that resident involvement in surgical procedures provides safe and effective outcomes, but also suggests areas of additional study. Our analysis raises important questions about the level of resident training required to optimize outcomes in complicated patients and argues in favor of more experienced residents as an important ingredient in the mix that improves outcomes in these complex patients. Author Contributions Study conception and design: Ferraris, Harris, Endean Acquisition of data: Ferraris, Harris, Martin Analysis and interpretation of data: Ferraris, Harris, Martin, Saha, Endean Drafting of manuscript: Ferraris, Harris, Endean Critical revision: Ferraris, Harris, Saha, Martin, Endean REFERENCES 1. Seib CD, Greenblatt DY, Campbell MJ, et al. Adrenalectomy outcomes are superior with the participation of residents and fellows. J Am Coll Surg 2014;219:53e60. 2. Castleberry AW, Clary BM, Migaly J, et al. Resident education in the era of patient safety: a nationwide analysis of outcomes and complications in resident-assisted oncologic surgery. Ann Surg Oncol 2013;20:3715e3724. 3. Krell RW, Birkmeyer NJ, Reames BN, et al. Effects of resident involvement on complication rates after laparoscopic gastric bypass. J Am Coll Surg 2014;218:253e260. 4. Babu R, Thomas S, Hazzard MA, et al. Worse outcomes for patients undergoing brain tumor and cerebrovascular procedures following the ACGME resident duty-hour restrictions. J Neurosurg 2014;121:262e276. 5. Matulewicz RS, Pilecki M, Rambachan A, et al. Impact of resident involvement on urological surgery outcomes: an analysis of 40,000 patients from the ACS NSQIP database. J Urol 2014;192:885e890. 6. Venkat R, Valdivia PL, Guerrero MA. Resident participation and postoperative outcomes in adrenal surgery. J Surg Res 2014;190:559e564. 7. Norby K, Siddiq F, Adil MM, Haines SJ. The effect of duty hour regulations on outcomes of neurological surgery in training hospitals in the United States: duty hour regulations and patient outcomes. J Neurosurg 2014;121:247e261. 8. Tapia NM, Milewicz A, Whitney SE, et al. Identifying and eliminating deficiencies in the general surgery resident core competency curriculum. JAMA Surg 2014;149:514e518. 9. Harris JD, Staheli G, LeClere L, et al. What effects have resident work-hour changes had on education, quality of life, and safety? A systematic review. Clin Orthop Relat Res 2015;473: 1600e1608.
Ferraris et al
Vol. 222, No. 4, April 2016
10. Fitzgibbons SC, Chen J, Jagsi R, Weinstein D. Long-term follow-up on the educational impact of ACGME duty hour limits: a pre-post survey study. Ann Surg 2012;256: 1108e1112. 11. Yeo H, Viola K, Berg D, et al. Attitudes, training experiences, and professional expectations of US general surgery residents: a national survey. JAMA 2009;302:1301e1308. 12. Sheetz KH, Waits SA, Krell RW, et al. Improving mortality following emergent surgery in older patients requires focus on complication rescue. Ann Surg 2013;258:614e617; discussion 617e618. 13. Reddy HG, Shih T, Englesbe MJ, et al. Analyzing “failure to rescue”: is this an opportunity for outcome improvement in cardiac surgery? Ann Thorac Surg 2013;95:1976e1981; discussion 1981. 14. Ahmed EO, Butler R, Novick RJ. Failure-to-rescue rate as a measure of quality of care in a cardiac surgery recovery unit: a five-year study. Ann Thorac Surg 2014;97:147e152. 15. Ghaferi AA, Birkmeyer JD, Dimick JB. Complications, failure to rescue, and mortality with major inpatient surgery in medicare patients. Ann Surg 2009;250:1029e1034. 16. Davenport DL, Henderson WG, Khuri SF, Mentzer RM Jr. Preoperative risk factors and surgical complexity are more predictive of costs than postoperative complications: a case study using the National Surgical Quality Improvement Program (NSQIP) database. Ann Surg 2005;242:463e468; discussion 468e471. 17. Livingston EH, Elliot A, Hynan L, Cao J. Effect size estimation: a necessary component of statistical analysis. Arch Surg 2009;144:706e712. 18. McGlothlin AE, Lewis RJ. Minimal clinically important difference: defining what really matters to patients. JAMA 2014;312:1342e1343. 19. Blackstone EH. Comparing apples and oranges. J Thorac Cardiovasc Surg 2002;123:8e15. 20. Austin PC. Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006: a systematic review and suggestions for improvement. J Thorac Cardiovasc Surg 2007;134:1128e1135. 21. Copay AG, Subach BR, Glassman SD, et al. Understanding the minimum clinically important difference: a review of concepts and methods. Spine J 2007;7:541e546. 22. Heffernan MJ, Seehausen DA, Andras LM, Skaggs DL. Comparison of outcomes after posterior spinal fusion for adolescent idiopathic and neuromuscular scoliosis: does the surgical first assistant’s level of training matter? Spine 2014;39:648e655. 23. Beland MD, Anderson TJ, Atalay MK, et al. Resident experience increases diagnostic rate of thyroid fine-needle aspiration biopsies. Acad Radiol 2014;21:1490e1494. 24. Schoenfeld AJ, Serrano JA, Waterman BR, et al. The impact of resident involvement on post-operative morbidity and mortality following orthopaedic procedures: a study of 43,343 cases. Arch Orthop Trauma Surg 2013;133:1483e1491. 25. Hope WW, Hooks WB 3rd, Kilbourne SN, et al. Assessing resident performance and training of colonoscopy in a general surgery training program. Surg Endosc 2013;27: 1706e1710. 26. Advani V, Ahad S, Gonczy C, et al. Does resident involvement effect surgical times and complication rates during laparoscopic appendectomy for uncomplicated appendicitis? An analysis of 16,849 cases from the ACS-NSQIP. Am J Surg 2012;203:347e351; discussion 351e352.
Discussion
555
27. Ross SW, Oommen B, Kim M, et al. A little slower, but just as good: postgraduate year resident versus attending outcomes in laparoscopic ventral hernia repair. Surg Endosc 2014;28: 3092e3100. 28. Singh P, Turner EJ, Cornish J, et al. Safety assessment of resident grade and supervision level during emergency appendectomy: analysis of a multicenter, prospective study. Surgery 2014;156:28e38. 29. Monteiro R, Mino JS, Siperstein AE. Trends and disparities in education between specialties in thyroid and parathyroid surgery: an analysis of 55,402 NSQIP patients. Surgery 2013; 154:720e728; discussion 728e729. 30. Herrick BW, Yap RL. It is safe to teach residents laser prostate surgery in the private practice setting. Urology 2013;81: 629e632. 31. Naiditch JA, Lautz TB, Raval MV, et al. Effect of resident postgraduate year on outcomes after laparoscopic appendectomy for appendicitis in children. J Laparoendosc Adv Surg Tech A 2012;22:715e719. 32. Guirand DM, Okoye OT, Schmidt BS, et al. Venovenous extracorporeal life support improves survival in adult trauma patients with acute hypoxemic respiratory failure: a multicenter retrospective cohort study. J Trauma Acute Care Surg 2014;76:1275e1281. 33. Al-Adhami A, Macfie A, Mathieson C, et al. Ventricular assist devices as rescue therapy in cardiogenic shock after subarachnoid hemorrhage. Ann Thorac Surg 2014;97:1440e1443. 34. Henneman D, van Leersum NJ, Ten Berge M, et al. Failureto-rescue after colorectal cancer surgery and the association with three structural hospital factors. Ann Surg Oncol 2013; 20:3370e3376. 35. Pucher PH, Aggarwal R, Singh P, Darzi A. Enhancing surgical performance outcomes through process-driven care: a systematic review. World J Surg 2014;38:1362e1373. 36. Sasor SE, Flores RL, Wooden WA, Tholpady S. The cost of intraoperative plastic surgery education. J Surg Educ 2013; 70:655e659. 37. Pentiak PA, Schuch-Miller D, Streetman RT, et al. Barriers to adoption of the surgical resident skills curriculum of the American College of Surgeons/Association of Program Directors in Surgery. Surgery 2013;154:23e28. 38. Hwang CS, Wichterman KA, Alfrey EJ. The cost of resident education. J Surg Res 2010;163:18e23.
Discussion DR CARL HAISCH (Greenville, NC): I would like to compliment the authors on the huge amount of work and statistical analysis they have done. The authors have used the NSQIP database to analyze patient outcomes in high acuity procedures in general surgery, vascular, cardiothoracic, and neurosurgery. They have also reviewed the deaths and complications of these patients, including pulmonary failure, wound complications, central nervous system, cardiac complications, sepsis, or unplanned returns to the operating room (OR). They hypothesized that a failure to rescue patients with major complications would be a measure of resident benefit or deficit caused by the resident care. I have some methodologic