The Society of Thoracic Surgeons Congenital Heart Surgery Database Mortality Risk Model: Part 2—Clinical Application

The Society of Thoracic Surgeons Congenital Heart Surgery Database Mortality Risk Model: Part 2—Clinical Application

Johns Hopkins All Children’s Heart Institute, Saint Petersburg, Tampa, and Orlando, Florida (JPJ, JAQ, CM, MLJ); Division of Cardiac Surgery, Department of Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland (JPJ, JAQ, CM, MLJ); Florida Hospital for Children, Orlando, Florida (JPJ, JAQ, CM, MLJ); Duke Clinical Research Institute, Duke University School of Medicine, Durham, North Carolina (SMO, SK); Department of Pediatrics and Communicable Diseases, C.S. Mott Children’s Hospital, University of Michigan, Ann Arbor, Michigan (SKP); Division of Cardiothoracic Surgery, Department of Surgery, Children’s Hospital of Philadelphia, Philadelphia, Pennsylvania (JWG); Department of Cardiac Surgery, Boston Children’s Hospital, Harvard Medical School, Boston, Massachusetts (JEM); Division of Pediatric Cardiac Surgery, Benioff Children’s Hospital, University of California, San Francisco, San Francisco, California (TK); Section of Congenital Cardiovascular Surgery, University of Illinois College of Medicine at Peoria, Children’s Hospital of Illinois, Peoria, Illinois (KFW); Institute for Health Care Research and Improvement, Baylor Health Care System, Dallas, Texas (GF); The Society of Thoracic Surgeons, Chicago, Illinois (JMH); Nemours/Alfred I. duPont Hospital, Wilmington, Delaware (CP); McGill University, Montreal Children’s Hospital, Montreal, Ontario, Canada (CIT); Royal Brompton Hospital, London, United Kingdom (FL-G); Northwestern University Feinberg School of Medicine, Ann & Robert H. Lurie Children’s Hospital of Chicago, Chicago, Illinois (CLB); Kosair Children’s Hospital, University of Louisville, Louisville, Kentucky (EHA); Texas Children’s Hospital, Baylor College of Medicine, Houston, Texas (CDF); Children’s Hospital of Wisconsin, Milwaukee, Wisconsin (JST); Children’s National Medical Center, Washington D.C. (RAJ); University of Florida, College of Medicine–Jacksonville, Jacksonville, Florida (FHE); University of Colorado Denver, School of Medicine, Aurora, Colorado (FLG); University of Michigan, Ann Arbor, Michigan (RLP); and Massachusetts General Hospital Department of Surgery and Center for Quality and Safety, and Harvard Medical School, Boston, Massachusetts (DMS)

Background. The empirically derived 2014 Society of Thoracic Surgeons Congenital Heart Surgery Database Mortality Risk Model incorporates adjustment for procedure type and patient-specific factors. The purpose of this report is to describe this model and its application in the assessment of variation in outcomes across centers. Methods. All index cardiac operations in The Society of Thoracic Surgeons Congenital Heart Surgery Database (January 1, 2010, to December 31, 2013) were eligible for inclusion. Isolated patent ductus arteriosus closures in patients weighing less than or equal to 2.5 kg were excluded, as were centers with more than 10% missing data and patients with missing data for key variables. The model includes the following covariates: primary procedure, age, any prior cardiovascular operation, any noncardiac abnormality, any chromosomal abnormality or syndrome, important preoperative factors (mechanical circulatory support, shock persisting at time of operation,

Accepted for publication July 2, 2015. Presented at the Sixty-first Annual Meeting of the Southern Thoracic Surgical Association, Tucson, AZ, Nov 5-8, 2014. Address correspondence to Dr Jacobs, Cardiac Surgery, Johns Hopkins All Children’s Heart Institute, 601 Fifth St South, Ste 607, St. Petersburg, FL 33701; e-mail: [email protected].

Ó 2015 by The Society of Thoracic Surgeons Published by Elsevier

mechanical ventilation, renal failure requiring dialysis or renal dysfunction (or both), and neurological deficit), any other preoperative factor, prematurity (neonates and infants), and weight (neonates and infants). Variation across centers was assessed. Centers for which the 95% confidence interval for the observed-to-expected mortality ratio does not include unity are identified as lower-performing or higher-performing programs with respect to operative mortality. Results. Included were 52,224 operations from 86 centers. Overall discharge mortality was 3.7% (1,931 of 52,224). Discharge mortality by age category was neonates, 10.1% (1,129 of 11,144); infants, 3.0% (564 of 18,554), children, 0.9% (167 of 18,407), and adults, 1.7% (71 of 4,119). For all patients, 12 of 86 centers (14%) were lowerperforming programs, 67 (78%) were not outliers, and 7 (8%) were higher-performing programs. Conclusions. The 2014 Society of Thoracic Surgeons Congenital Heart Surgery Database Mortality Risk Model facilitates description of outcomes (mortality) adjusted for procedural and for patient-level factors. Identification of low-performing and high-performing programs may be useful in facilitating quality improvement efforts. (Ann Thorac Surg 2015;100:1063–70) Ó 2015 by The Society of Thoracic Surgeons 0003-4975/$36.00 http://dx.doi.org/10.1016/j.athoracsur.2015.07.011

CONGENITAL HEART

Jeffrey P. Jacobs, MD, Sean M. O’Brien, PhD, Sara K. Pasquali, MD, J. William Gaynor, MD, John E. Mayer, Jr, MD, Tara Karamlou, MD, Karl F. Welke, MD, Giovanni Filardo, PhD, Jane M. Han, MSW, Sunghee Kim, PhD, James A. Quintessenza, MD, Christian Pizarro, MD, Christo I. Tchervenkov, MD, Francois Lacour-Gayet, MD, Constantine Mavroudis, MD, Carl L. Backer, MD, Erle H. Austin, III, MD, Charles D. Fraser, MD, James S. Tweddell, MD, Richard A. Jonas, MD, Fred H. Edwards, MD, Frederick L. Grover, MD, Richard L. Prager, MD, David M. Shahian, MD, and Marshall L. Jacobs, MD

1064

JACOBS ET AL APPLICATION OF STS-CHSD MORTALITY RISK MODEL

CONGENITAL HEART

T

he assessment of variation in outcomes across pediatric and congenital cardiac surgical programs requires adjustment for differences in case mix across hospitals. Outcomes analysis without risk adjustment is misleading and can lead to risk aversion, with efforts made to avoid caring for the sickest of patients who might benefit the most from a cardiac operation [1–3]. The most common forms of risk adjustment for analysis and reporting of outcomes from pediatric and congenital cardiac surgery in use today are based mainly on the estimated risk of mortality of the primary procedural component of the operative encounter, as defined by The Society of Thoracic Surgeons (STS)—European Association for Cardio-Thoracic Surgery (EACTS) Congenital Heart Surgery (STAT) Mortality Categories [4, 5]. Because of the increased availability of robust clinical data, adding a variety of specific patient characteristics to pediatric and congenital cardiac surgical risk models is now possible [6]. The empirically derived 2014 STS Congenital Heart Surgery Database (STS-CHSD) Mortality Risk Model incorporates procedural factors and patient factors. This report describes this risk model and its application in the assessment of variation in pediatric and congenital cardiac surgical outcomes across centers.

Patients and Methods Data Source The STS-CHSD was used for this study. STS-CHSD is a randomly audited, comprehensive database of patients who have undergone congenital and pediatric cardiac surgical operations at centers in the United States and Canada. STS-CHSD is a voluntary registry that contains preoperative, operative, and outcomes data for all patients undergoing congenital and pediatric cardiovascular operations at participating centers. STS-CHSD uses the following age groupings: neonates (0 to 30 days), infants (31 days to 1 year), children (>1 year to <18 years), and adults (
18 years). The Report of the 2010 STS Congenital Heart Surgery Practice and Manpower Survey, undertaken by the STS Workforce on Congenital Heart Surgery, estimated that 125 hospitals in the United States and 8 hospitals in Canada perform pediatric and congenital heart operations [7]. In 2014, the STS-CHSD included 113 North American congenital heart surgical programs representing 119 of these 125 hospitals (95.2% penetrance by hospital) in the United States and 3 of the 8 centers in Canada. Coding for this database is accomplished by clinicians and ancillary support staff using the International Pediatric and Congenital Cardiac Code [8, 9] and is entered into the contemporary version of the STS-CHSD data collection form (version 3.0 was used in this report) [10]. The definitions of all terms and codes from the STS-CHSD used in this report have been standardized and published [10]. Evaluation of data quality in the STS-CHSD includes intrinsic verification of data (eg, identification and correction of values that are missing or out of range and inconsistencies across fields), along with a formal process

Ann Thorac Surg 2015;100:1063–70

of random site audits at approximately 10% of all participating centers each year conducted by a panel of independent quality personnel and pediatric cardiac surgeons [11]. Audit of the STS-CHSD has documented the following rates of completeness and accuracy for the specified fields of data [12]:  Primary diagnosis (completeness, 100%; accuracy, 96.2%)  Primary procedure (completeness, 100%; accuracy, 98.7%)  Mortality status at hospital discharge (completeness, 100%; accuracy, 98.8%) The Duke Clinical Research Institute serves as the data warehouse and analytic center for all STS National Databases. Approval for this study was obtained from the Duke University Medical Center Institutional Review Board and the Quality Measurement Task Force of the STS Workforce on National Databases.

Study Population All index cardiac operations in the STS-CHSD during January 1, 2010, to December 31, 2013, inclusive, were eligible for inclusion (index operations are defined as the first cardiac operation of a hospitalization). Patients weighing less than or equal to 2.5 kg undergoing isolated closure of the patent arterial duct were excluded. Centers with more than 10% missing data and patients with missing data for discharge mortality or other key variables were excluded. The first 3.5 years of data were used for estimation, and the last 0.5 years were used as an independent validation set. The determination of outlier status of hospitals was based on all 4 years of data. Table 1 documents the inclusionary and exclusionary criteria applied to obtain the final study cohort, which included 52,224 index cardiac operations from 86 centers from January 1, 2010, to December 31, 2013, inclusive. Data collected included procedure type, patient age, weight, prematurity, chromosomal abnormalities, syndromes, noncardiac congenital anatomic abnormalities, preoperative factors, prior cardiac operation, primary procedure, and operative mortality.

The 2014 STS-CHSD Mortality Risk Model The STS-CHSD Mortality Risk Model adjusts for the variables listed in Table 2. These variables include procedural stratification based on the primary procedure of each operation and also a variety of categories of patient characteristics, including prematurity, chromosomal abnormalities, syndromes, noncardiac congenital anatomic abnormalities, and preoperative factors. The factors coded in the STS-CHSD under the heading “Preoperative Factors” are patient preoperative “status” factors, such as preoperative mechanical circulatory support and preoperative renal dysfunction, among many others. These preoperative status factors can be distinguished from other patient characteristics, including patient-related genetic and structural factors that are included in the

Ann Thorac Surg 2015;100:1063–70

JACOBS ET AL APPLICATION OF STS-CHSD MORTALITY RISK MODEL

1065

Table 1. Inclusionary and Exclusionary Criteria Applied to Obtain the Study Cohort Records (No.)

Criteria

PDA ¼ patent ductus arteriosus; Surgery Mortality Categories.

(29,238 excluded) (11,480 excluded) (3,400 excluded) (1,302 excluded) (937 excluded) (304 excluded) 52,224

113

86

STAT ¼ The Society of Thoracic Surgeons—European Association for Cardio-Thoracic Surgery Congenital Heart

model, such as chromosomal abnormalities, syndromes, and noncardiac congenital anatomic abnormalities. The six specific “Preoperative Factors” incorporated into this model were selected from a previous report of 25,476 index cardiac operations performed in 23,019 patients at 72 centers from January 1, 2010, to December 31, 2012 [6]. Preoperative factors selected for inclusion in the model included all factors whose association with discharge mortality was highly significant (p < 0.0001) for the three pediatric age categories (neonates, infants, and children): mechanical circulatory support, renal failure requiring dialysis or renal dysfunction (or both), shock, and mechanical ventilation, and factors whose association with discharge mortality was significant (p < 0.05) for all four age categories: preoperative mechanical ventilation and preoperative neurological deficit. Table 2. The Society of Thoracic Surgeons Congenital Heart Surgery Database Mortality Risk Model: List of Included Variables for Which the Model Adjusts Variable Age group Primary procedurea Weight (neonates and infants) Prior cardiothoracic operation Any noncardiac congenital anatomic abnormality Any chromosomal abnormality or syndrome Prematurity (neonates and infants) Preoperative factors  Mechanical circulatory supportb  Shock, persistent at time of operation  Mechanical ventilation to treat cardiorespiratory failure  Renal failure requiring dialysis or renal dysfunction, or both  Neurological deficit  Any other factor a

98,885

The model adjusts for each combination of primary procedure and age group. Coefficients obtained by shrinkage estimation with The Society of Thoracic Surgery—European Association for Cardio-Thoracic Surgery Congenital Heart Surgery Mortality Category as an auxiliary variable; see b text for details. Includes intraaortic balloon pump, ventricular assist device, extracorporeal membrane oxygenation, mechanical cardiopulmonary support system.

The detailed statistical methodology involved with the development and validation of this new 2014 STS-CHSD Mortality Risk Model is described in a companion report [13]. As described in that report [13], a statistical technique known as empirical Bayes shrinkage estimation was adopted to account for small sample sizes when estimating the baseline risk associated with individual procedures. Briefly, the empirical Bayes estimator uses data from the entire ensemble of procedures in the database when estimating the risk for any single procedure. Heuristically, the empirical Bayes estimate is a weighted average of a procedure’s actual observed risk and a model-based estimate of risk derived from other similar procedures. The model weights the data of an individual procedure more heavily when the procedurespecific sample size is large enough to be reliable and weights the model-based prediction more heavily when the procedure-specific sample size is too small to be reliable. To further increase precision, the STAT Mortality Category of each procedure was incorporated in the shrinkage estimator. As a result, the estimated risk of each procedure is a weighted average of its actual observed risk and information borrowed from other procedures in the same STAT Mortality Category.

Variation Across Centers Risk-adjusted mortality results for each center are expressed as the observed-to-expected (O/E) operative mortality ratio. The O/E ratio is defined as the number of observed deaths (numerator “O”) divided by the number of expected deaths (denominator “E”). The number of observed deaths is the center’s observed operative mortality. The number of expected deaths is determined by the 2014 STS-CHSD Mortality Risk Model [13] and reflects the center’s case mix; that is, the mix of age, weight, procedure types, and other model-specific variables, including prior cardiothoracic operations, noncardiac congenital anatomic abnormalities, chromosomal abnormalities, syndromes, and preoperative risk factors. An O/ E ratio exceeding 1.0 implies that the hospital had more deaths than would have been expected given the case mix. Conversely, an O/E ratio of less than 1.0 implies that the number of deaths was fewer than expected. Because

CONGENITAL HEART

Starting population: cardiac operations performed 2010 to 2013, inclusive, with patient data collected under STS Version 3 or later at North America programs Exclude data from 27 hospitals with <90% data completeness Exclude nonindex operations of a hospitalization Exclude PDA closures in 2.5 kg or organ procurement Exclude operations with an undefined STAT score Exclude operations with missing mortality Exclude operations with missing data for key model covariates Final population

Participants (No.)

CONGENITAL HEART

1066

JACOBS ET AL APPLICATION OF STS-CHSD MORTALITY RISK MODEL

small differences in the O/E ratio are usually not statistically significant and may reflect random statistical variation, O/E ratios are presented with 95% confidence intervals (CIs). Centers were classified as having mortality that was lower than expected if their 95% CI for the O/E mortality ratio fell entirely below 1, higher than expected if their 95% CI for the O/E mortality ratio was entirely above 1, and same as expected if their 95% CI for the O/E mortality ratio overlapped 1. Data are presented documenting the number of centers that would fall into each of the three performance categories using 80%, 90%, 95%, and 99% CIs for the O/E mortality ratio.

Ann Thorac Surg 2015;100:1063–70

Fig 1. Distribution of hospital-specific observed-to-expected (O/E) ratios for operative mortality with 95% confidence intervals (gray lines).

Institutional Review Board Approval The Duke University Health System Institutional Review Board approved this study and provided a waiver of informed consent. Although the STS data used in the analysis contain patient identifiers, they were originally collected for nonresearch purposes, and the risk to patients was deemed to be minimal [14].

Results The final analysis included 52,224 index cardiac operations with an overall operative mortality of 3.7% (n ¼ 1,931). Table 3 reports operative mortality overall and stratified by age category. Assessment of model fit and discrimination in the development sample and the validation sample reveals overall C statistics of 0.875 and 0.858, respectively. Variation across centers was assessed for all patients and within the age categories. Table 4 reports the distribution of institutions into performance categories with 4-year samples of data, for all patients and within age categories, by 95% CI for the O/E mortality ratio. Table 5 reports the number of centers identified as programs with mortality that was higher than expected, the same as expected, and lower than expected, using 4-year samples with 80%, 90%, 95%, and 99% CIs. Figure 1 shows the distribution of hospital-specific O/E ratios for operative mortality with 95% CIs.

Comment The earliest forms of risk adjustment used by STS-CHSD were based on complexity stratification, a method of analysis in which the data are divided into relatively homogeneous groups (called strata). The data are analyzed within each stratum. STS-CHSD currently uses three methods of procedural complexity stratification [6]: (1) Table 3. Overall Operative Mortality and Operative Mortality Stratified by Age Category Variable Operations, No. Deaths, No. Death, %

Overall Neonates Infants Children Adults 52,224 1,931 3.7

11,144 1,129 10.1

18,554 564 3.0

18,407 167 0.9

4,119 71 1.7

The STAT Mortality Categories, (2) Aristotle Basic Complexity (ABC) levels, and (3) Risk Adjustment for Congenital Heart Surgery-1 (RACHS-1) categories. The ABC score and ABC levels were incorporated in STSCHSD in 2002. RACHS-1 categories were incorporated in STS-CHSD in 2006. RACHS-1 and the ABC scores were developed at a time when limited multiinstitutional clinical data were available and were, therefore, largely based on subjective probability (expert opinion). With the increasing availability of multiinstitutional clinical data, the STAT Mortality Score and STAT Mortality Categories were introduced in STS-CHSD in 2010. These three methods provide three different ways of grouping types of pediatric and congenital cardiac operations according to their estimated risk or complexity. The STAT Mortality Categories are empirically derived from data in the STS and EACTS CHSDs and use five categories; the STAT Mortality Categories serve as the main complexity adjustment tool for the STS-CHSD. The ABC method uses four categories. The RACHS-1 method uses six categories but functionally has five categories when applied to STS-CHSD. STS and EACTS have transitioned from the primary use of ABC and RACHS-1 to the primary use of the STAT Mortality Categories. For the past several years, risk adjustment in STS-CHSD has been based on estimated risk of death of the primary procedure of the operation, age, weight, and prematurity. Our previous report demonstrated the significant association of certain patient-specific preoperative risk factors with discharge mortality after operations for pediatric and congenital cardiac disease [6]. That previous report suggested that the inclusion of additional patient-specific preoperative factors in risk models for pediatric and congenital cardiac operations could lead to increased precision in predicting the risk of operative mortality and comparison of observed versus expected outcomes. One of the limitations of current systems of risk stratification used for pediatric and congenital cardiac operations is the limited adjustment made for patient-specific factors. With the increased availability of verified clinical data, incorporating patient-specific factors into the risk models in the STS-CHSD is now possible. The empirically derived 2014 STS-CHSD Mortality Risk Model incorporates adjustment for procedure type and

Ann Thorac Surg 2015;100:1063–70

JACOBS ET AL APPLICATION OF STS-CHSD MORTALITY RISK MODEL

1067

Table 4. The Society of Thoracic Surgeons Congenital Heart Surgery Database Mortality Risk Model: Performance Categories Using 95% Confidence Intervals Programs With Mortality That Is Total Programs No. (%)

Age Category Neonates Neonates and infants Neonates, infants, and children Neonates, infants, children, and adults

85 86 86 86

Higher Than Expected No. (%)

(100) (100) (100) (100)

8 11 12 12

Future Directions The new 2014 STS-CHSD Mortality Risk Model includes a number of patient-specific characteristics, including prematurity, chromosomal abnormalities, syndromes, noncardiac congenital anatomic abnormalities, and preoperative factors. This present report describes this model and uses it to assess variation in outcomes across centers. This 2014 STS-CHSD Mortality Risk Model will improve the ability of the STS-CHSD to be used as a tool to improve the quality of surgical care delivered to patients with pediatric and congenital cardiac disease [15–17]. It is anticipated that with the accumulation of larger numbers of patients and further understanding of the importance of various variables to each primary procedure or patient diagnosis, the C statistics will improve. The development of this risk model and the

(9) (13) (14) (14)

72 68 67 67

(85) (79) (78) (78)

Lower Than Expected No. (%) 5 7 7 7

(6) (8) (8) (8)

incorporation of these patient-specific preoperative data elements is a large step forward, but as in all new ideas, this model will require refinement. In our prior STS-CHSD report [6], the only preoperative factors that were associated with operative mortality among adults undergoing operations for congenital heart disease were preoperative mechanical ventilation and preoperative neurological deficit. A specific tool for surgical risk adjustment for adults undergoing operations for congenital heart disease is currently under development. Many adults with congenital heart disease have unique preoperative factors, including ventricular dysfunction and pulmonary hypertension. Many of the preoperative factors in adults undergoing operations for congenital heart disease tend to be quite different from the preoperative factors present in neonates, infants, and children. Eventually, age-specific risk models will complement the overall risk models in the STS-CHSD. In the January 2014 upgrade of the STS-CHSD, several “procedure-specific factors” were added to the data collection form. These new procedure-specific factors pertain to the previously published benchmark operations [16] and should eventually facilitate the development of procedure-specific risk models for these benchmark operations. In reality, meaningful evaluation and comparison of outcomes require consideration of mortality and morbidity, but the latter is much harder to quantify. The observed versus expected operative mortality ratio is only one aspect of overall performance and should not be equated with overall performance of a surgical program. The STAT Mortality Categories are an empirically based tool that statistically estimates the risk of mortality associated with operations for congenital heart disease [4, 5]. The addition of patient characteristics, including

Table 5. Identification of Outliers Using 80%, 90%, 95%, and 99% Confidence Intervals for the Model Using Neonates, Infants, Children, and Adults Programs With Mortality That Is Confidence Interval 80% 90% 95% 99%

Total Programs No. (%) 86 86 86 86

(100) (100) (100) (100)

Higher Than Expected No. (%) 19 13 12 6

(22) (15) (14) (7)

Same As Expected No. (%) 52 63 67 78

(60) (73) (78) (91)

Lower Than Expected No. (%) 15 10 7 2

(17) (12) (8) (2)

CONGENITAL HEART

patient-specific factors. The 2014 STS-CHSD Mortality Risk Model is designed to facilitate comparison of observed versus expected operative mortality across centers but is not designed to function as a procedurebased risk calculator. Several procedure-specific factors have been added to the STS-CHSD to enhance future risk adjustment. A short-term limitation of the 2014 STS-CHSD Mortality Risk Model is the requirement that operations be coded with patient data collected under STS version 3 or later, whereas some operations performed in 2010 to 2013 (in patients who had prior cardiac operations) are coded in earlier versions of the database. Software updates will facilitate coding of the necessary fields for these operations under STS version 3 or later. Furthermore, coefficients of the final model will be reestimated twice yearly, using a 4year analytic window, to coincide with the production of each STS-CHSD participant feedback report.

Same As Expected No. (%)

CONGENITAL HEART

1068

JACOBS ET AL APPLICATION OF STS-CHSD MORTALITY RISK MODEL

prematurity, chromosomal abnormalities, syndromes, noncardiac congenital anatomic abnormalities, and preoperative factors, can enhance risk adjustment using the STAT Mortality Categories. To complement the evaluation of quality of care in pediatric and congenital cardiac surgery using the analysis of risk-adjusted mortality, STS has also developed a tool to analyze risk-adjusted morbidity: the STAT Morbidity Categories [18], which are based on major postoperative complications and postoperative length of stay. Major postoperative complications and postoperative length of stay were both used because models that assume a perfect one-to-one relationship between postoperative complications and postoperative length of stay are not likely to fit the data well. The STAT Morbidity Categories are an empirically based tool that statistically estimates the risk of morbidity associated with operations for congenital heart disease [18]. The new 2014 STS-CHSD Mortality Risk Model described in this report adjusts for individual procedures, which is an even more granular adjustment than the STAT Mortality Category. Future initiatives to assess quality and improve outcomes using STS-CHSD will adjust for both mortality and morbidity based not only on the operation performed but also on patient-specific factors. In the future, when models have been developed that encompass other outcomes in addition to mortality, pediatric and congenital cardiac surgical performance may be able to be assessed using a multi-domain composite score that incorporates both mortality and morbidity and adjusts for the operation performed and patient-specific factors. It is our expectation that this information will also be publicly reported.

Conclusions Current STS-CHSD risk adjustment is based on estimated risk of mortality of the primary procedure of the operation, age, and weight. The 2014 STS-CHSD Mortality Risk Model includes additional patient-specific preoperative factors that lead to increased precision in predicting risk of operative mortality and comparison of observed versus expected outcomes. The 2014 STS-CHSD Mortality Risk Model can be used to describe center-level performance. Identification of low-performing and high-performing programs may facilitate quality improvement.

References 1. Jacobs JP, Cerfolio RJ, Sade RM. The ethics of transparency: publication of cardiothoracic surgical outcomes in the lay press. Ann Thorac Surg 2009;87:679–86. 2. Shahian DM, Edwards FH, Jacobs JP, et al. Public reporting of cardiac surgery performance: part 1–history, rationale, consequences. Ann Thorac Surg 2011;92(3 Suppl):S2–11.

Ann Thorac Surg 2015;100:1063–70

3. Shahian DM, Edwards FH, Jacobs JP, et al. Public reporting of cardiac surgery performance: part 2–implementation. Ann Thorac Surg 2011;92(3 Suppl):S12–23. 4. O’Brien SM, Clarke DR, Jacobs JP, et al. An empirically based tool for analyzing mortality associated with congenital heart surgery. J Thorac Cardiovasc Surg 2009;138: 1139–53. 5. Jacobs JP, Jacobs ML, Maruszewski B, et al. Initial application in the EACTS and STS Congenital Heart Surgery Databases of an empirically derived methodology of complexity adjustment to evaluate surgical case mix and results. Eur J Cardiothorac Surg 2012;42:775–80. 6. Jacobs JP, O’Brien SM, Pasquali SK, et al. The importance of patient-specific preoperative factors: an analysis of The Society of Thoracic Surgeons Congenital Heart Surgery Database. Ann Thorac Surg 2014;98:1653–9. 7. Jacobs ML, Daniel M, Mavroudis C, et al. Report of the 2010 Society of Thoracic Surgeons Congenital Heart Surgery Practice and Manpower Survey. Ann Thorac Surg 2011;92: 762–9. 8. International Pediatric and Congenital Cardiac Code. Available: at http://www.ipccc.net. Accessed December 30, 2013. 9. Franklin RC, Jacobs JP, Krogmann ON, et al. Nomenclature for congenital and paediatric cardiac disease: historical perspectives and The International Pediatric and Congenital Cardiac Code. Cardiol Young 2008;18(Suppl 2):70–80. 10. STS Congenital Heart Surgery Database Data Specifications. Version 3.0. Available at: http://www.sts.org/sites/default/ files/documents/pdf/CongenitalDataSpecificationsV3_0_20090904. pdf. Accessed July 4, 2014. 11. Clarke DR, Breen LS, Jacobs ML, et al. Verification of data in congenital cardiac surgery. Cardiol Young 2008;18(Suppl 2): 177–87. 12. Jacobs JP, Jacobs ML, Mavroudis C, Tchervenkov CI, Pasquali SK. Executive summary: The Society of Thoracic Surgeons Congenital Heart Surgery Database—twentieth harvest—(January 1, 2010—December 21, 2013). Durham, NC: The Society of Thoracic Surgeons (STS) and Duke Clinical Research Institute (DCRI), Duke University Medical Center; Spring 2014 Harvest. 13. O’Brien SM, Jacobs JP, Pasquali SK, et al. The Society of Thoracic Surgeons Congenital Heart Surgery Database mortality risk model: part 1—statistical methodology. Ann Thorac Surg 2015;100:1054–62. 14. Dokholyan RS, Muhlbaier LH, Falletta J, et al. Regulatory and ethical considerations for linking clinical and administrative databases. Am Heart J 2009;157:971–82. 15. Jacobs JP, Jacobs ML, Austin EH, et al. Quality measures for congenital and pediatric cardiac surgery. World J Pediatr Congenit Heart Surg 2012;3:32–47. 16. Jacobs JP, O’Brien SM, Pasquali SK, et al. Variation in outcomes for benchmark operations: an analysis of The Society of Thoracic Surgeons Congenital Heart Surgery Database. Ann Thorac Surg 2011;92:2184–92. 17. Jacobs JP, O’Brien SM, Pasquali SK, et al. Variation in outcomes for risk-stratified pediatric cardiac surgical operations: an analysis of the STS Congenital Heart Surgery Database. Ann Thorac Surg 2012;94:564–72. 18. Jacobs ML, O’Brien SM, Jacobs JP, et al. An empirically based tool for analyzing morbidity associated with operations for congenital heart disease. J Thorac Cardiovasc Surg 2013;145: 1046–57.e1.

DISCUSSION DR GROVER: Jeff, this was a great presentation, a real step forward for The Society of Thoracic Surgeons (STS) Congenital Heart Surgery Database. You and your colleagues are to be congratulated. It is a beautiful description of the historic development of

STS Congenital Heart Surgery Database, and all the hard work that you and your colleagues have put into it. A C statistic of 0.86 is remarkable considering that we are usually willing to accept 0.75 as acceptable for preoperative risk-adjustment databases.

Ann Thorac Surg 2015;100:1063–70

DR JACOBS: Thank you, Dr Grover. Those words of congratulations mean a lot coming from you as the past Chair of the STS Database and as the past President of both the Southern Thoracic Surgical Association and The Society of Thoracic Surgeons. Indeed, all of the names of our colleagues that you have mentioned had fundamental roles in the development of the STS Congenital Heart Surgery Database. Dr Grover, you have asked two important questions that I am happy to answer. First, our timeline for adding morbidity metrics into a composite performance measure is soon. In fact, we are about to start our efforts to develop a composite performance measure for congenital and pediatric cardiac surgery that incorporates the two domains of mortality and morbidity. Later this year, we will begin this project, which is funded by the National Heart, Lung, and Blood Institute (NHLBI) of the National Institutes of Health (NIH) via an NHLBI R01-funded grant titled, “Understanding Quality and Costs in Congenital Heart Surgery.” Sara Pasquali, at the University of Michigan, is the Principal Investigator, I am the STS Principal Investigator, and many of the names that you previously mentioned are very involved in this grant. The NIH will be funding our efforts to enhance our current STS Congenital Heart Surgery Database Mortality Risk Model that was presented today by developing a two-domain composite model that incorporates both mortality and morbidity. This grant is important, because it is the first NIH R01-funded grant to support research involving the STS Congenital Heart Surgery Database. Your second question is about the assessment of longitudinal long-term outcomes. I believe that long-term outcomes represent the Holy Grail of research about outcomes and quality improvement. Furthermore, data about longitudinal outcomes will provide answers to the questions that the parents of our patients really want to know. When parents comes to see us in the office, they want to know not only whether or not their babies are going to go home alive and be alive 30 days after surgery, but also whether or not their babies will be healthy

1069

enough to go to high school and college and one day have children of their own. Multiple strategies are being considered and implemented to try to transform the STS Database into a platform for longitudinal follow-up. These strategies include adding longitudinal variables to the STS Database itself and linking the STS Database with other registries and sources of information about longitudinal outcomes. DR MAYER, JR: I did just want to emphasize one other point. I think the concept here is that what we have been looking at are general risk factors that presumably apply equally across all patient populations, and the new version of the STS Congenital Heart Surgery Database now actually also includes procedurespecific risk factors, which I think is pretty critical. For example, nobody really cares what your coronary anatomy is if you are having your ventricular septal defect closed or your atrial septal defect closed, but you probably care more when it is tetralogy of Fallot and you care a lot when it is an arterial switch operation. So that procedure-level risk factor analysis is something that we are preparing for, and I think it will be important in raising this C statistic even higher. DR JACOBS: I agree completely. Procedure-specific factors have been added to the STS Congenital Heart Surgery Database for a variety of benchmark operations. Future risk models will likely incorporate these procedure specific factors. DR KARAMLOU: Jeff, a wonderful presentation, as usual. I just wanted to have you say one or two thoughts about public reporting and the power of the STS Database. I think Marshall Jacobs said it best when he said, “Public reporting evokes both emotional and logical responses.” So could you just please say a sentence or two about what our plans for this initiative are and how central the STS will be to facilitate accurate and fair public reporting? DR JACOBS: Absolutely, Tara. I think Marshall is absolutely correct with that statement. STS will begin public reporting of pediatric and congenital cardiac outcomes in January 2015, using a platform for pediatric and congenital cardiac surgical public reporting of outcomes that is based on the STS Congenital Heart Surgery Database Mortality Risk Model that we have presented today. DR BACKER: Jeff, congratulations. I wanted to make two comments and then I have a question. First I congratulate you on having over 50,000 congenital patients for your analysis. Second, as a co-author, I saw behind the scene the multiple Email chains of the innumerable interactions among some very capable and insightful physicians and statisticians. This thorough, careful, in-depth analysis is now summarized beautifully by your slides. The question I have is regarding the confidence intervals that are going to be selected for the one-star, two-star, and three-star programs. What makes you feel certain that use of 95% confidence intervals is the best, most informative metric to identify centers with outcomes that are importantly different from those achieved at other centers and across the aggregate of the 86 STS Congenital Database participants in this study? For example, one might consider using different confidence intervals to identify the one-star and three-star programs. For the three-star programs one might consider using the 80% confidence interval. Looking at your graph there is clearly not much difference between program No. 4

CONGENITAL HEART

I think it is important to pause just for a second to talk about some of the giants in this area that you have worked with. I think many of you know, but maybe not all of you, know that the congenital heart surgeons have been meeting since we began this database in 1989. It is a very complex database because of all the nuances of all the various congenital anomalies and the procedures, and, as mentioned, probably 150 to 200. Our former Southern Thoracic Surgical Association president, Gus Mavroudis, was the first to lead the STS Congenital Heart Surgery Database by developing the definitions of the anatomic pathology and physiology, and many other colleagues made important contributions, including Francois Lacour-Gayet, Martin Elliot, Marshall Jacobs, Bohdan Maruszewski, and multiple members from the United States, Canada, Europe, and Asia. This has been truly, therefore, a worldwide effort. You are to be congratulated on this great effort going forward. This work has resulted in not one but two manuscripts being submitted to The Annals, one on the clinical aspects and one on the statistics, which is very impressive in itself. I have basically a couple of questions to ask you. As you mentioned, you are going to be initiating morbidity and composite data reports very similar to what we do with the adult database. My question is on your timeline for that. And the second question is, are there plans for tracking long-term outcomes? For example, you are collecting chromosomal abnormalities. Are you going to carry that into genetics, too? And what are the long-term effects of all of these efforts? Again, congratulations to you and everybody in this room that has participated in this effort.

JACOBS ET AL APPLICATION OF STS-CHSD MORTALITY RISK MODEL

CONGENITAL HEART

1070

JACOBS ET AL APPLICATION OF STS-CHSD MORTALITY RISK MODEL

and program No. 9 or program No. 9 and program No. 15. They all have a very similar low observed-to-expected ratio for operative mortality. These are all superb programs that the public should know about. Changing the confidence interval from 95% to 80% increases the number of three-star program from 7 to 15. In contrast, for the low performing programs, the one-star programs, there seems to be a natural cutoff at the 95% range. In fact, going from 95% to 90% only adds one more program (12 to 13). This group has a distinctly high observed-to-expected mortality ratio that is more than twice what is expected. That cut point seems very relevant from a clinical standpoint. The low performers by this analysis are clearly outliers. I think that the most important part of this analysis is to identify the low performing programs. They either need to be improved or they may have to be phased out. The public deserves to be protected and that slide you showed with the difference in mortality is very dramatic, especially when you consider that it is risk adjusted data. What do you think about that strategy for the one-star, two-star, and three-star programs: 80% confidence interval for three-star programs (resulting in 15

Ann Thorac Surg 2015;100:1063–70

programs), 95% confidence interval for one-star programs (resulting in 12 programs)? DR JACOBS: I think the selection of confidence intervals is a critical topic and is certainly something that we could discuss for quite some time. I personally believe that the approach that you are describing makes reasonably good sense and that we could have different confidence intervals for high-performing outliers vs low-performing outliers and that those different confidence intervals at each end of the spectrum could be selected to maximize clinical reliability and clinical face validity. I do not know if this approach would be uniformly acceptable amongst all stakeholders, but it sounds like a good idea to me. Dr Grover is a co-author and invited discussant on this article. He was asked to serve as the invited discussant based on his involvement as Chair of the STS National Database Workforce for many years, his knowledge and historical perspective of the STS Congenital Heart Surgery Database, and his support of its development.