Objective measure of learning curves for trainees in cardiac surgery via cumulative sum failure analysis

Journal Pre-proof Objective Measure of Learning Curves for Trainees in Cardiac Surgery via Cumulative Sum Failure (Cusum) Analysis Elizabeth D. Krebs, MD, MSc, William Z. Chancellor, MD, Robert B. Hawkins, MD, MSc, Jared P. Beller, MD, J. Hunter Mehaffey, MD, MSc, Nicholas R. Teman, MD, Gorav Ailawadi, MD, Leora T. Yarboro, MD PII:

S0022-5223(19)32194-4

DOI:

https://doi.org/10.1016/j.jtcvs.2019.09.147

Reference:

YMTC 15151

To appear in:

The Journal of Thoracic and Cardiovascular Surgery

Received Date: 3 May 2019 Revised Date:

7 September 2019

Accepted Date: 25 September 2019

Please cite this article as: Krebs ED, Chancellor WZ, Hawkins RB, Beller JP, Mehaffey JH, Teman NR, Ailawadi G, Yarboro LT, Objective Measure of Learning Curves for Trainees in Cardiac Surgery via Cumulative Sum Failure (Cusum) Analysis, The Journal of Thoracic and Cardiovascular Surgery (2019), doi: https://doi.org/10.1016/j.jtcvs.2019.09.147. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. Copyright © 2019 Published by Elsevier Inc. on behalf of The American Association for Thoracic Surgery

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

Objective Measure of Learning Curves for Trainees in Cardiac Surgery via Cumulative Sum Failure (Cusum) Analysis

Authors: Elizabeth D. Krebs, MD, MSc1, William Z. Chancellor, MD1, Robert B. Hawkins, MD, MSc1, Jared P. Beller, MD1, J. Hunter Mehaffey, MD, MSc1, Nicholas R. Teman, MD1, Gorav Ailawadi, MD1, Leora T. Yarboro, MD1 Author Affiliations: 1 Department of Surgery, University of Virginia, Charlottesville, VA Disclosures: Gorav Ailawadi is a consultant for Abbott, Edwards, Medtronic, Admedus, and Gore. All other authors have nothing to disclose. Funding: The National Institutes of Health supported research in this publication under award numbers T32 HL007849 (JPB, WZC) and UM1 HL088925 (EDK) Presentation: Presented at American Association for Thoracic Surgery 99th Annual Meeting, Toronto, ON, May 6, 2019 Corresponding Author: Leora Yarboro, MD Division of Thoracic and Cardiovascular Surgery P.O. Box 800679 Charlottesville, VA, 22908-0709 Telephone: (434) 243-6828 Fax: (434) 244-7588 Manuscript Word Count: 2219/3500 Abstract Word Count: 237/250

1

40

Central Picture:

41 42 43 44 45 46 47 48 49 50 51 52 53

Central Picture Legend: Risk-adjusted learning curves of cardiac surgery residents. (58/90)

Perspective Statement: Though the safety of resident surgeons in cardiac surgery is well established, objective, outcome-based assessment measures of resident progress are lacking. Cumulative sum failure (Cusum) analysis has been applied to senior surgeons adopting new techniques, but has not previously described resident learning curves. (316/405) Central Message: Trainee-performed cases have excellent results, with better than expected outcomes. Cusum analysis characterizes learning curves and is a promising tool for resident evaluation throughout training. (196/200)

2

54

Abstract (237/250)

55

Objectives: Objective measures of cardiac surgery trainee progress are limited despite a push for

56

competency-based assessments. We hypothesized that Cumulative sum failure (Cusum) technique could

57

provide a risk-adjusted, quantitative measure of resident learning curves and competence.

58

Methods: Records of all coronary artery bypass grafting and/or valve operations performed by cardiac-

59

track residents from 2007-2017 at a single institution were stratified by operative resident. Multivariable

60

regression evaluated the association between resident, case number and postoperative outcomes. To

61

evaluate performance over time, risk-adjusted Cusum analysis was performed, taking into account

62

institutional expected values and comparing residents to study-defined ‘early alert’ and ‘concern’

63

boundaries.

64

Results: A total of 3,937 STS-PROM cases were evaluated from 19 residents. Observed-to-expected

65

ratios for mortality and combined morbidity-mortality were 0.66 and 0.72, and each individual resident

66

exhibited better than predicted outcomes (all observed:expected ratios less than one.)When evaluating

67

Cusum learning curves, residents exhibited an initial slight increase in complications, followed by

68

improvement and better than expected performance. The ‘early alert’ boundary was crossed by 36.8%

69

residents at any point in training, with 94.7% of residents under this boundary at the end of training. The

70

higher ‘concern’ boundary was crossed by 2 residents (10.5%), although all residents ended their training

71

below this boundary.

72

Conclusion: Outcomes for trainee-performed cardiac surgery procedures were excellent, with no

73

association between individual trainees and adverse events. Cusum analysis based on postoperative

74

outcomes is a potential tool for objective evaluation of resident proficiency.

75 76

3

77

Introduction

78

Providing excellent surgical training for residents and fellows without compromising patient

79

outcomes is the paramount goal of academic cardiothoracic surgical programs. The safety of training

80

thoracic surgical residents is well established. Academic programs consistently demonstrate satisfactory

81

short and long-term outcomes for procedures such as coronary artery bypass grafting (CABG), mitral

82

valve surgery, and aortic valve replacement. 1-5 However, objectively evaluating the safety and

83

progression of an individual resident’s skills remains a challenge across surgical disciplines.6-8 Assessing

84

operative skills in real-world environments is particularly difficult, due to patient variability, safety,

85

subjectivity of evaluator, and variability in assessment metrics. 6, 9-11 Despite these barriers, surgical

86

education is shifting to a competency-based training paradigm, further necessitating the development of

87

objective metrics for trainee success.6

88

Cumulative sum failure (Cusum) analysis, initially used for quality control in the industrial

89

sector, has previously been used to evaluate senior surgeons performing new cardiac surgical

90

procedures.12-15 This technique provides for real-time performance monitoring and is best for detecting

91

small but persistent process deviations.16 Risk-adjusted Cusum analysis evaluates observed outcomes

92

compared to expected outcomes over time, using predefined boundaries to specify when a larger or

93

smaller amount of adverse events are occurring than expected. Though the method has evaluated trainees

94

performing simulated procedures in a variety of surgical disciplines, it has not to our knowledge been

95

used to evaluate the impact of resident experience on outcomes following cardiac surgical procedures.

96

We aimed to evaluate the impact of individual cardiothoracic surgical residents’ experience levels

97

on postoperative outcomes. Given the Cusum technique’s strength in providing real-time performance

98

modeling, we hypothesized that this analysis would identify targetable areas for improvement in surgical

99

resident education. Furthermore, we aimed to characterize the learning curve of residents performing

100

standard procedures, to evaluate if postoperative outcomes provide a reasonable means for evaluating

101

resident improvement.

102 4

103

Methods

104

Study Population

105

Patients undergoing cardiac operations with an STS predicted risk score, including CABG, aortic

106

valve replacement (AVR), mitral valve replacement (MVR), mitral valve repair (MVr), or combination of

107

CABG and valve surgery (STS-PROM cases) between 2007-2017 were selected from an institutional

108

Society of Thoracic Surgeons (STS) database. These cases were chosen to represent standard, commonly

109

performed operations, and also as these are the operations for which standardized preoperative scores

110

exist. The primary resident recorded for each case was abstracted and deidentified. To include only

111

cardiac-track residents, cases performed by residents with fewer than 140 STS-PROM cases over the

112

course of their fellowship were excluded (this number represents the minimum number of CABG and

113

valve operations required, and also a natural break in resident case volume.) At our institution, fellows act

114

as primary surgeon for all portions of standard procedures, from incision to closure. Primary outcome was

115

the composite of operative mortality (composite deaths during hospitalization and within 30 days from

116

surgery) and major morbidity (prolonged mechanical ventilation, renal failure, permanent stroke,

117

reoperation, or deep sternal wound infection), as defined by standard STS definitions.17 The study was

118

approved by the institutional review board at the University of Virginia (IRB-HSR #19762).

119 120

Univariate and Regression Analysis

121

Univariate analysis evaluated overall demographics and outcomes for cases in the study,

122

reporting characteristics as median [interquartile range], mean (standard deviation) or n (%). Analysis of

123

residents evaluated the observed:expected ratios of composite morbidity and mortality for each resident.

124

Regression analyses evaluated the impact of both case number and individual resident on composite

125

morbidity and mortality, controlling for preoperative risk by incorporating the STS predicted risk of

126

morbidity or mortality into the model. Statistical analysis was performed using SAS Version 9.4 (SAS

127

Institute, Cary, NC) using p<0.05 as the significance threshold.

128 5

129

CUSUM Analysis

130

Risk-adjusted Cusum analysis was performed for the outcome of composite mortality or

131

morbidity, according to methods described by Rogers et al. and Grunkemeier et al.16, 18 For each resident,

132

the sum of observed minus expected failure (composite morbidity and mortality) rates were plotted along

133

the y axis, with number of STS-PROM cases performed along the x axis. Because institutional outcomes

134

were better than expected by STS predictive risk scores alone, institutional expected values were derived

135

by constructing a logistic regression model, accounting for STS predicted morbidity or mortality

136

(PROMM). (Supplemental table) Boundary lines were constructed according to a modified version of

137

equations described by Grunkemeier et al, and are intended solely to aid in interpretation and not for

138

formal hypothesis testing.18 (see Appendix) These lines are drawn to approximate a 95% confidence

139

interval (‘early alert’ boundary) and 98% confidence interval (‘concern’ boundary.) Analysis was

140

conducted using Graphpad Prism (GraphPad Software Inc., San Diego, CA) and SAS Version 9.4 (SAS

141

Institute, Cary, NC).

142 143 144

Results A total of 3,937 STS-PROM cases performed by 19 cardiac-track residents were included in the

145

study. The most common case included was an isolated CABG (50.88%) followed by aortic valve

146

replacement (22.07%). Patients had a mean PROM of 3.34%, and a mean PROMM of 20.14%, with an

147

actual mortality rate of 2.39% and composite morbidity-mortality rate of 13.21%, giving observed-to-

148

expected ratios of 0.66 and 0.72, respectively. Residents included in the study performed a mean of 207 ±

149

26 STS-PROM cases. Further case demographics are shown in Table 1.

150

Every resident exhibited an observed:expected ratio of composite morbidity and mortality less

151

than one (Table 2), demonstrating that each resident had a less than expected overall rate of morbidity

152

and mortality. On multivariable regression analysis, only one individual resident was associated with an

153

increase in composite morbidity and mortality (Table 3). Case experience was not statistically associated

6

154

with these risk-adjusted adverse outcomes, however it did exhibit a trend towards being protective for

155

composite morbidity and mortality. (Table 3).

156

Risk-adjusted Cusum plots for all residents studied is shown in Figure 1. The ‘concern’ boundary

157

line (representing 98% confidence interval) was crossed by 2 residents (10.5%) at any point in training,

158

with 100% of residents below this boundary at the end of training. The ‘early alert’ boundary line

159

(approximating 95% confidence interval) was crossed by 7 (36.8%) residents at any point in training,

160

however by the end of training all but one (94.7%) resident were under the ‘early alert’ boundary. A

161

figure of all CUSUM curves, with those of residents crossing the alert boundary colored red, is shown in

162

Figure 2.

163

The mean curve for all residents is shown in Figure 3. This curve features an initial upward

164

slope, representing more observed than expected events initially, peaking at around 70 cases, followed by

165

a gradual downward slope that levels out around 0 (with observed morbidity matching expected at the

166

institution) around 140 total STS-PROM cases.

167 168 169

Discussion This retrospective study utilized regression and Cusum techniques to evaluate outcome-related

170

proficiency for 19 residents performing almost 4,000 routine cardiac surgery procedures. Overall

171

outcomes were superior, with far fewer adverse events than predicted based on STS risk scores. There

172

was a trend towards improved outcomes throughout training and only one individual resident associated

173

with increased morbidity or mortality compared to the others. Upon Cusum analysis, residents

174

demonstrated a slight increase in adverse events at the beginning of training, peaking at 70 cases,

175

followed by improvement that ultimately resulted in better-than-expected outcomes by 140 cases, and

176

well before the end of training. Although 7 residents crossed an ‘alarm’ limit at some point during

177

training, all were within predefined proficiency bounds at the end of training, demonstrating the method’s

178

value as an objective early-warning assessment for trainees.

7

179

Importantly, overall outcomes were excellent, with each resident demonstrating observed

180

morbidity and mortality rates much lower than predicted based on STS predictive risk scores.

181

Additionally, there was only one resident with a statistically greater rate of adverse events compared to

182

the others. This is consistent with existing evidence demonstrating safety of trainees in cardiac surgery.1-5

183

However, existing studies have also noted learning curves for trainees, with shorter operative and

184

cardiopulmonary bypass time later in training.5 Similarly, the present investigation noted an incremental

185

improvement in outcomes later in training upon regression analysis, with Cusum analysis demonstrating

186

improvement later in training. Thresholds in our study were noted at approximately 70 cases, when the

187

learning curve peaked, with a decrease until 140 PROM cases, after which outcomes matched the

188

institution’s historical average. Coincidentally, the Accreditation Council for Graduate Medical Education

189

(ACGME) minimum requirement for this cases- 80 myocardial revascularization and 60 acquired valvular

190

heart disease cases- is also 140. Though this may be solely coincidental, it is also possible that the

191

CUSUM analysis mirrors the current surgeon-determined gestalt for the minimum cases needed to

192

achieve full competency.

193

Cusum analysis has previously evaluated practicing cardiac surgeons adopting new techniques.

194

The method has been used to evaluate a variety of procedures, including minimally invasive and off-

195

pump CABG, thoracic aortic procedures with hypothermic circulatory arrest, and minimally invasive

196

aortic valve replacement.12, 14, 15, 19 Though there is much variation, the resident learning curves in this

197

intervention closely parallel the learning curves of experienced surgeons, with initial upslope of more

198

complications, followed by eventual downslope and continued improvements. Similar to the present

199

study, the majority of these evaluations have been retrospective, however have noted the technique’s

200

promise as a prospective quality improvement tool. One barrier is lack of user-friendly software for

201

prospective analysis. For example, with accessible, web-based software, a surgeon could log the expected

202

and actual outcomes for a series of cases and receive a real-time update of their progress.

203

Though the safety of resident surgeons is well established, upon Cusum analysis there were a

204

number of residents identified to have crossed an “early alert” boundary at any point in training. This

8

205

suggests that CUSUM techniques provide a potential opportunity for early identification of trainees in

206

need of increased guidance. By visually identifying outcome-related trends, mentors may be able to detect

207

areas for improvement or provide targeted education earlier in training. Cardiac surgery centers have

208

previously reported success using Cusum methods for departmental quality control, which could be

209

similarly translated to trainees.13 For instance, when noted to cross the boundary lines early in training, a

210

trainee could receive increased guidance and supervision during cases, increased one-on-one skills

211

training in a simulation environment, or targeted evaluation of postoperative management to identify

212

deficiencies in clinical judgement. Cusum methods have been previously used to assess skills-related

213

progress in general surgery trainees, evaluating simulation-based laparoscopic, open, and endoscopic

214

proficiency.20 These simulation-based studies suggest that Cusum analysis is more sensitive to assess

215

competency than existing metrics, which may translate to Cusum assessment of postoperative outcomes.20

216

Though the concern boundaries in this study were chosen to represent 95% and 98% confidence intervals,

217

individual programs could alter their confidence boundaries to more adequately identify residents in need

218

of intervention. By using less lenient boundaries, more trainees may be identified, earlier in their training.

219

As objective measures of trainee performance are lacking, Cusum analysis of outcomes for resident cases

220

represents a promising tool for gauging resident progress during training.6

221

This study does have noted limitations. First, the Cusum methodology is designed primarily for

222

process control rather than rigorous statistical testing, and as with all studies utilizing Cusum

223

methodology, the results should not be overinterpreted. Secondly, the analysis evaluates only cases with

224

an STS risk score (CABG and/or left sided valve procedures), and does not consider other cases that the

225

resident has performed throughout their training. Though this was necessary for risk-adjustment and for

226

adding uniformity, the impact of these other cases is not known. The case mix was also variable, with

227

CABG and AVR cases overrepresented in the sample. Furthermore, there is likely some variability in

228

resident role from case to case, though our institution does intend to have residents act as senior surgeon

229

for the entirety of the case. Additionally, though not necessarily a limitation, it should be noted that for

230

Cusum risk-adjustment we derived institution-specific risk scores, taking into account STS risk scores,

9

231

rather than using the STS risk scores alone. Though using STS risk scores would have shown all learning

232

curves as continual better-than-expected downslopes, we felt deriving institution-specific expected values

233

provided a better representation of true learning curves. Finally, this investigation is retrospective and

234

observational in nature.

235 236 237

Conclusion Based on analysis of nearly 4,000 cases performed by 19 surgical residents, outcomes for routine

238

cardiac surgery procedures were better than predicted, with no resident demonstrating a greater than

239

expected rate of morbidity and mortality. However, trainees did demonstrate a learning curve, with

240

improved outcomes throughout training. Cusum analysis based on patient-level postoperative outcomes is

241

a promising tool for objective evaluation of residents throughout training, and may provide early

242

identification of trainees in need of increased guidance to achieve proficiency. As surgical education

243

shifts towards a competency-based education paradigm, the development of objective evaluation metrics

244

will become increasingly important for ensuring success for the next generation of cardiac surgeons.

245

10

246

Appendix:

247

For prediction limits in figures 2 and 3, we used a modified version of the equations described by

248

Grunkemeier et al.18 Their equation describes the standard error (SE) of the risk-adjusted Cusum at time t

249

as the square root of the cumulative sum of E(1-E) for all patients operated on up to time t, where E is the

250

expected value at that time point. The 95% and 98% prediction limits are determined by multiplying the

251

SE by 1.96 and 2.58, respectively. To estimate the curve for all residents, E was assumed to be the mean

252

expected value of all operations (meanE), such that the SE at any point equaled the square root of

253

t*meanE*(1-meanE).

254

11

255

References

256

1.

Bakaeen FG, Sethi G, Wagner TH, Kelly R, Lee K, Upadhyay A, et al. Coronary artery bypass

257

graft patency: residents versus attending surgeons. The Annals of thoracic surgery. 2012;94:482-

258

488; discussion 488.

259

2.

Caputo M, Chamberlain MH, Ozalp F, Underwood MJ, Ciulli F, Angelini GD. Off-pump

260

coronary operations can be safely taught to cardiothoracic trainees. The Annals of thoracic

261

surgery. 2001;71:1215-1219.

262

3.

263 264

graft surgery. Eur J Cardiothorac Surg. 2004;25:591-596. 4.

265 266

Oo AY, Grayson AD, Rashid A. Effect of training on outcomes following coronary artery bypass

Saxena A, Virk SA, Bowman SRA, Jeremy R, Bannon PG. Heart Valve Surgery Performed by Trainee Surgeons: Meta-Analysis of Clinical Outcomes. Heart Lung Circ. 2018;27:420-426.

5.

Yount KW, Yarboro LT, Narahari AK, Ghanta RK, Tribble CG, Kron IL, et al. Outcomes of

267

Trainees Performing Coronary Artery Bypass Grafting: Does Resident Experience Matter? The

268

Annals of thoracic surgery. 2017;103:975-981.

269

6.

270 271

Vaporciyan AA, Yang SC, Baker CJ, Fann JI, Verrier ED. Cardiothoracic surgery residency training: past, present, and future. J Thorac Cardiovasc Surg. 2013;146:759-767.

7.

Dijksterhuis MG, Voorhuis M, Teunissen PW, Schuwirth LW, ten Cate OT, Braat DD, et al.

272

Assessment of competence and progressive independence in postgraduate clinical training. Med

273

Educ. 2009;43:1156-1165.

274

8.

275 276

2018;267:820-822. 9.

277 278 279

George BC, Dunnington GL, DaRosa DA. Trainee Autonomy and Patient Safety. Ann Surg.

van Hove PD, Tuijthof GJ, Verdaasdonk EG, Stassen LP, Dankelman J. Objective assessment of technical surgical skills. The British journal of surgery. 2010;97:972-987.

10.

Reznick RK, MacRae H. Teaching surgical skills--changes in the wind. The New England journal of medicine. 2006;355:2664-2669.

12

280

11.

DaRosa DA, Zwischenberger JB, Meyerson SL, George BC, Teitelbaum EN, Soper NJ, et al. A

281

theory-based model for teaching and assessing residents in the operating room. J Surg Educ.

282

2013;70:24-30.

283

12.

Murzi M, Cerillo AG, Gilmanov D, Concistre G, Farneti P, Glauber M, et al. Exploring the

284

learning curve for minimally invasive sutureless aortic valve replacement. J Thorac Cardiovasc

285

Surg. 2016;152:1537-1546 e1531.

286

13.

Murzi M, Cerillo AG, Bevilacqua S, Gasbarri T, Kallushi E, Farneti P, et al. Enhancing

287

departmental quality control in minimally invasive mitral valve surgery: a single-institution

288

experience. Eur J Cardiothorac Surg. 2012;42:500-506.

289

14.

Novick RJ, Fox SA, Stitt LW, Kiaii BB, Swinamer SA, Rayman R, et al. Assessing the learning

290

curve in off-pump coronary artery surgery via CUSUM failure analysis. The Annals of thoracic

291

surgery. 2002;73:S358-362.

292

15.

Mazine A, Stevens LM, Ghoneim A, Chung J, Ouzounian M, Dagenais F, et al. Developing skills

293

for thoracic aortic surgery with hypothermic circulatory arrest. J Thorac Cardiovasc Surg.

294

2019;157:1360-1368 e1368.

295

16.

Rogers CA, Reeves BC, Caputo M, Ganesh JS, Bonser RS, Angelini GD. Control chart methods

296

for monitoring cardiac surgical performance and their interpretation. J Thorac Cardiovasc Surg.

297

2004;128:811-819.

298

17.

Shahian DM, O'brien SM, Filardo G, Ferraris VA, Haan CK, Rich JB, et al. The Society of

299

Thoracic Surgeons 2008 cardiac surgery risk models: part 1—coronary artery bypass grafting

300

surgery. The Annals of thoracic surgery. 2009;88:S2-S22.

301

18.

302 303

Grunkemeier GL, Wu YX, Furnary AP. Cumulative sum techniques for assessing surgical results. The Annals of thoracic surgery. 2003;76:663-667.

19.

Holzhey DM, Jacobs S, Walther T, Mochalski M, Mohr FW, Falk V. Cumulative sum failure

304

analysis for eight surgeons performing minimally invasive direct coronary artery bypass. J

305

Thorac Cardiovasc Surg. 2007;134:663-669.

13

306 307

20.

Hu Y, Jolissaint JS, Ramirez A, Gordon R, Yang Z, Sawyer RG. Cumulative sum: a proficiency metric for basic endoscopic training. The Journal of surgical research. 2014;192:62-67.

308 309

14

310

Table 1: Demographics and Outcomes of Cases with an STS Predicted Risk Score Performed By

311

Residents

Demographics: Patient Age Sex (Female) Peripheral Arterial Disease Heart Failure Hypertension Diabetes Stroke Predicted Risk of Mortality Predicted Morbidity or Mortality Procedure Type: AV Replacement AVR + CABG Isolated CABG MVr MVr + CABG MVR + CABG MVR Outcomes: Cardiopulmonary Bypass Time Postoperative LOS Renal Failure Prolonged Ventilation Deep Sternal Wound Infection Reoperation Major Morbidity Operative Mortality Mortality or Morbidity

Total Cases (n=3937) Median [IQR], mean (standard deviation) or n (%) 68 [59 - 76] 1208 (30.68) 651 (16.54) 1695 (43.05) 3209 (81.51) 1587 (40.31) 67 (1.70) 3.34% (4.61%) 20.14% (14.50%) 869 (22.07) 472 (11.99) 2003 (50.88) 267 (6.78) 102 (2.59) 42 (1.07) 182 (4.62) 95 [78 - 119] 6 [5-8] 157 (3.99) 329 (8.36) 9 (0.23) 110 (2.80) 498 (12.65) 94 (2.39) 520 (13.21)

312 313

AV, aortic valve; AVR, aortic valve replacement; CABG, coronary artery bypass grafting; MVr, mitral

314

valve repair; MVR, mitral valve replacement; LOS, length of stay

315

15

316

Table 2: Observed and Expected Rates of Composite Morbidity and Mortality for Cases Performed

317

by Cardiac-Track Thoracic Surgery Residents

Resident B C D E F G I K L M N O P Q S T W Y Z

Composite MorbidityMortality (%) 12.98% 13.06% 14.16% 14.29% 14.22% 13.11% 12.21% 12.33% 10.97% 10.49% 10.97% 12.95% 18.41% 14.00% 13.93% 14.06% 12.75% 12.95% 13.57%

Expected Rate of MorbidityMortality (%)a 17.62% 21.77% 21.59% 20.94% 20.84% 21.44% 20.08% 21.11% 21.30% 18.22% 19.06% 18.99% 19.57% 19.37% 19.94% 20.68% 21.09% 18.77% 19.06%

O:E Ratio 0.74 0.60 0.66 0.68 0.68 0.61 0.61 0.58 0.51 0.58 0.58 0.68 0.94 0.72 0.70 0.68 0.60 0.69 0.71

318 319

a

320

O:E Ratio; observed:expected ratio

Expected rate based on Society of Thoracic Surgeons predictive risk scores

321

16

322

Table 3: Logistic Regression Model for Risk-Adjusted Impact of Residents and Case Experience on

323

Composite Morbidity and Mortality Factor Case Number Resident B Resdent C Resident D Resident E Resident F Resident G Resident I Resident K Resident L Resident M Resident N Resident P Resident Q Resident S Resident T Resident W Resident Y Resident Z PROMM Score (logit)

Odds Ratio 0.998 1.09 0.824 0.912 0.964 0.982 0.797 0.882 0.803 0.694 0.785 0.758 1.449 1.032 1.009 0.949 0.811 1.015 1.042 2.649

95% CI Odds Ratio 0.997 - 1.000 0.603 - 1.969 0.460 - 1.475 0.515 - 1.615 0.542 - 1.713 0.552 - 1.746 0.435 - 1.459 0.502 - 1.551 0.446 - 1.448 0.382 - 1.261 0.404 - 1.525 0.386 - 1.486 0.800 - 2.624 0.570 - 1.871 0.561 - 1.814 0.523 - 1.724 0.444 - 1.481 0.569 - 1.808 0.578 - 1.877 2.376 - 2.953

p-value <0.001 0.436 0.579 0.950 0.834 0.764 0.502 0.812 0.506 0.185 0.527 0.450 0.035 0.604 0.673 0.899 0.552 0.645 0.567 <.0001

324 325 326

CI, confidence interval; PROMM, Society of Thoracic Surgeons predicted risk of morbidity or mortality; logit = log (1/(1 – PROMM))

17

327 328 329

Figure 1: Individual Learning Curves of All Residents

330 331 332 333 334

Figure 1 Legend: Learning curves of residents performing cardiac surgery procedures, riskadjusted by institutional predicted outcomes. The shaded gray boundary represents the 95% confidence interval ‘early alert’ boundary, while the dashed boundary represents the ‘concern’ 98% confidence interval boundary. (see appendix)

18

335 336

Figure 2:

337 338

Figure 2 Legend: Learning curves of selected residents performing cardiac surgery procedures,

339

risk-adjusted by institutional predicted outcomes. The shaded gray boundary represents the 95%

340

confidence interval ‘early alert’ boundary, while the dashed boundary represents the ‘concern’

341

98% confidence interval boundary. Residents crossing an ‘early alert’ boundary are colored

342

orange, while those crossing a ‘concern’ boundary are colored red.

343

19

345 346 347 348

Figure 3:

Morbidity or Mortality (Observed - Expected)

344

2

0 50 -2

100

150

200

Number of Performed STS-PROM Cases

Figure 3 Legend: Mean learning curve of cardiac-track residents performing routine cardiac surgery procedures.

20