- Email: [email protected]
What the Surgeon Needs to Know About Databases
CURRENT READING
What the Surgeon Needs to Know About Databases Michele Salati, MD,* and Alessandro Brunelli, MD† Data collection is one of the most important instruments of any quality improvement initiatives. We have selected, summarized, and discussed 5 recent contributions mostly based on large international databases, which we considered most relevant to our specialty. They focused on different aspects: the selection and rigorous definition of the variables contained in the data set, the evaluation of a treatment or a pathway of care by the analysis of the observed outcomes, the identification of risk factors able to affect the surgical course, the measurement of the quality provided by a care giver, and the assessment of the quality of the data collected and the planning of quality improvement activities.
Alessandro Brunelli, MD
Semin Thoracic Surg 27:250–255 I 2015 Elsevier Inc. All rights reserved.
Central Message
Keywords: database, quality of care, quality of data, thoracic surgery, lung resection, VATS lobectomy
Collection of clinical data is essential to improve standards of care and evaluate quality of care in a reliable and accountable way.
INTRODUCTION The clinical practice is increasingly based on the scientific evidence. Original studies, review papers, meta-analysis, and guidelines represent a solid foundation that supports everyday our clinical strategies. In all the aforementioned studies, the analytic process is preceded by the data collection phase. The creation and maintenance of a database is a complex activity able to influence the process of knowledge and decision-making. To obtain a reliable database, especially if we are building or examining a surgical registry, we need to focus our attention on – the selection and rigorous definition of the variables contained in the data set, – the finality of the database: – evaluation of a treatment or a pathway of care by the analysis of the observed outcomes, – identification of risk factors able to affect the surgical course, – measurement of the quality provided by a care giver, and – the assessment of the quality of the data collected and the planning of quality improvement activities. *
Division of Thoracic Surgery, Ospedali Riuniti Ancona, Italy. Department of Thoracic Surgery, St. James’s University Hospital, Leeds, UK.
†
Address reprint requests to Alessandro Brunelli, Dept. Thoracic Surgery, St. James’s University Hospital, Beckett St, Bexley Wing, LS7 9FT, Leeds, UK. E-mail: [email protected]
250
The aim of this work is to explain these points using recently published articles based on the analysis of large international databases.
SELECTION AND DEFINITION OF THE VARIABLES The Society of Thoracic Surgeons and the European Society of Thoracic Surgeons General Thoracic Surgery databases: Joint standardization of variable definition and terminology. Fernandez FG, Falcoz PE, Kozower BD, et al. Ann Thorac Surg 99:368-376, 20151 Representatives of the Society of Thoracic Surgeons (STS) and of the European Society of Thoracic Surgeons (ESTS) Database Committees contributed to the development of a joint document for the variables standardization in thoracic surgery registries. They defined, harmonizing different terminologies, 62 variables usually collected in monospecialistic thoracic surgery databases. The first step for the creation of a medical registry is represented by the identification of the aim of the data collection. In fact, taking into account the real word that the database should describe and the objectives and information that it would provide, it is possible to select those variables for collection. The next step is the definition of the variables. This phase is extremely important, being able to heavily influence the quality of data across the years, especially in multi-institutional registries.2 In this work, the authors compared the 2 major international general thoracic surgery databases, the STS and the ESTS
1043-0679/$-see front matter ª 2015 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1053/j.semtcvs.2015.06.004
THORACIC DATABASES registries, which collect medical and thoracic surgery data from North American and European patients. Even if these databases share the same objective of quality of care monitoring and improvement, the authors identified some discrepancies in the definitions of 50 variables common to both registries. Each organization reviewed and proposed a specific definition for each of these variables. Then, after several meetings, a common set of variable definitions was agreed by both societies. A similar process was adopted for a second group of 12 variables, initially not in common but desirable in both databases. In conclusion, the article presented a list of 62 parameters strictly defined by a joint task force of the STS and ESTS, which could be used as foundation for the creation of mono- or multi-institutional base of data in thoracic surgery. SURGICAL TREATMENT EVALUATION Thoracoscopic lobectomy is associated with acceptable morbidity and mortality in patients with predicted postoperative forced expiratory volume in 1 second or diffusing capacity for carbon monoxide less than 40% of normal. Burt BM, Kosinski AS, Shrager JB, et al. J Thorac Cardiovasc Surg 148:19-29, 20143 The authors tried to verify if the innovative thoracoscopic surgical (VATS) lobectomy approach was superior to the standard open lobectomy, considered the cornerstone for the treatment of primary lung cancer, in patients with reduced pulmonary function. The analysis was performed on data derived from the Society of Thoracic Surgeons General Thoracic Database. This article represents an example of how the data collected in international registries could be used to derive information able to lead the clinical practice in the future. In fact, the most recent guidelines proposed by the American College of Chest Physicians, regarding the preoperative physiological evaluation of lung resection candidates, defined the group of patients with a predicted postoperative (ppo) forced expiratory volume in 1 second (FEV1) or diffusing capacity for carbon monoxide (DLCO) or both less than 40% of predicted as at high surgical risk for postoperative complications and death.4 At the same time, growing evidence showed the advantages of the VATS lobectomy technique in terms of reduced morbidity, mortality, postoperative chest surgical discomfort, and immunologic impairment.5 Nevertheless, a study aimed at testing the VATS lobectomy results in the specific subgroup of highrisk lung resection candidates was not yet performed.
To verify their hypothesis, the authors queried the Society of Thoracic Surgeons General Thoracic Database. This large registry gave them the possibility to compare 6802 patients submitted to open lobectomy vs 6574 submitted to VATS lobectomy. Firstly, a multivariate analysis was performed demonstrating the following: although decreasing ppoFEV1 and ppoDLCO values were independent predictors of cardiopulmonary morbidity in both the open lobectomy and VATS lobectomy groups, the ppoFEV1 was not related to the postoperative mortality in VATS lobectomy patients. Then the authors performed a propensity score analysis to match 1:1 open and VATS patients, taking into account several preoperative characteristics (21 factors, excluding pulmonary function) as well as the pathologic stage of disease. They obtained 4215 well-matched pairs of patients for verifying the rates of cardiopulmonary morbidity and mortality at incremental ranges of ppoFEV1% and ppoDLCO%. They found that in patients with ppoFEV1 o 40%, those submitted to VATS lobectomy had a lower rates of complications and mortality compared with those submitted to open surgery (13% vs 22% and 0.7% vs 4.8%, respectively). Similarly, in patients with ppoDLCO o 40% those operated on through VATS had a much lower incidence of complications and mortality compared with those operated through thoracotomy (10% vs 15% and 2% vs 5%, respectively). Consequently, the authors concluded that VATS lobectomy could be considered a reasonably safe surgical procedure to treat patients with lung cancer traditionally considered at high risk for compromised pulmonary function. RISK FACTORS IDENTIFICATION The European Thoracic Surgery Database project: Modeling the risk of inhospital death following lung resection. Berrisford R, Brunelli A, Rocco G, et al. Eur J Cardiothorac Surg 28:306-311, 20056 This article published by Berrisford and colleagues in 2005 on behalf of the Audit and guidelines committee of the European Society of Thoracic Surgeons and the European Association of Cardiothoracic Surgeons is an example of risk model analysis performed on data derived from a large international registry, such as the European Society of Thoracic Surgeons Database, which, at that time, collected data from 112 thoracic surgery units from different European Countries (only 27 of these units, sending to the database data of sufficient completeness, were enrolled in the study). Complex analytic models are usually needed to identify risk factors influencing the results of a
Seminars in Thoracic and Cardiovascular Surgery Volume 27, Number 2
251
THORACIC DATABASES therapeutic strategy. This is particularly evident in studies of risk modeling for complications and deaths that are rare events after surgical treatment. The reliability of the analysis is dependent on the amount and quality of data collected. Consequently, these types of studies are based on large databases, where the information should be clearly encoded and stable, gathering data from multiple institutions for several years. The authors developed a model to identify factors able to predict the inhospital mortality in patients submitted to lung resection. A total of 3426 patients (wedge or segmentectomies 26%, lobectomies 59%, pneumonectomies 14%, and volume reduction 1%) were used in the analysis, with an overall mortality of 1.9% (66 patients). The initial population was divided into 2 different groups: a first group (60% of the patients) was used for performing a logistic regression analysis aimed at identifying the baseline characteristics associated with the inhospital mortality. Then the mortality model, incorporating as risk factors age, dyspnea score, American Society of Anaesthesiologists Score, and type of resection, was built and tested on the remaining 40% of patients. Taking into account the model tendency at underestimating mortality for patients at medium risk and overestimating for those at high risk and considering that the model was based mostly on subjective factors, the authors decided to refine the analysis, developing a second model focused only on patients with lung cancer (85% of the initial cohort). This analysis yielded the European Society Objective Score (ESOS), which is a mortality risk model based just on age and ppoFEV1 and derived from a population of 1694 patients. The ESOS was able to predict inhospital death with high concordance between predicted and observed mortality for all the risk classes of the 1128 patients testing group. The authors recommended the use of the ESOS mostly for monitoring the performance of a unit and improving the results on long-term survival. In fact, special caution should be taken applying the information derived from a risk model for the clinical management of individual patients, especially in terms of early postsurgical mortality. QUALITY OF CARE EVALUATION The European Thoracic Database project: Composite performance score to measure quality of care after major lung resection. Brunelli A, Berrisford RG, Rocco G, et al. Eur J Cardiothorac Surg 35:769-774, 20097 This article represents a template for developing a unique performance score (the Composite
252
Performance Score [CPS]), that, incorporating multiple indicators of outcome (morbidity and mortality), intraoperative and perioperative processes, is able to measure the overall health care quality. This project, directly implemented by the European Society of Thoracic Surgeons, was performed using the platform of the European Society of Thoracic Surgeons database. This study shows the need for a careful planning of the variable selection and inclusion within a data set to obtain valuable data to analyze. In fact, considering the aim of this project, which was to develop a model for measuring the quality of care at a European level, it was necessary to collect information about several domains of the thoracic surgery clinical practice for a long period of time and from multiple contributors. A total of 1656 procedures (major lung resections for primary lung cancer), uploaded from 10 European thoracic surgery units, were selected and the correspondent data validated for completeness and consistency before the analysis. The authors identified and clearly defined 4 different quality indicators (2 process indicators and 2 outcome indicators). The process indicators were represented by the proportion of patients with predicted postoperative carbon monoxide lung diffusion capacity estimated before the surgical procedure and the proportion of patients with a systematic lymph nodes dissection performed during the operation. The outcome indicators were represented by the risk-adjusted mortality and the risk-adjusted cardiopulmonary morbidity. The predictors of morbidity and mortality used for building the risk models were identified by a 3-step process, consisting of a univariate analysis, a multivariate regression analysis, and a bootstrap procedure for the final models validation. The resulting predictors associated with cardiopulmonary morbidity were age, predicted postoperative forced expiratory volume in 1 second, and extended resection (ie, lung resection associated with resection of chest wall, diaphragm, pericardium, or other mediastinal structures); the ones associated with mortality were age and extended resection. For each unit the 4 indicators were calculated and after a procedure of rescaling, necessary to harmonize parameters with a high range of variability, summed to obtain the CPS. The authors verified that using the CPS as quality of care indicator all the evaluated units changed their position in comparison with the rank obtained by using a single indicator such as the mortality. This project could be used as a methodological template (in terms of data collection and the data mining strategies) for the development of
Seminars in Thoracic and Cardiovascular Surgery Volume 27, Number 2
THORACIC DATABASES multidomain scores for the quality of care assessment. A composite score such as the CPS, including process and outcome measures of quality, is a reliable indicator of performance, which is able to express the level of quality as a single numeric parameter. The CPS could lead processes of quality of care monitoring and quality improvement strategies in thoracic surgery either in a single institution or at an international level. DATA QUALITY Task-independent metrics to assess the data quality of medical registries using the European Society of Thoracic Surgeons (ESTS) database. Salati M, Brunelli A, Dahan M, et al. Eur J Cardiothorac Surg 40(1):91-98, 20118 The process of knowledge is a pathway that starts from the data collection. The interpretation of the available data allows obtaining information for planning the decision-making strategies.9,10 Taking into account this chain, to ensure a high quality of data is essential, because, despite the best analytic process, it directly influences the reliability of the resulting evidence. In the present work the authors described the methodological approach to measure the quality of data in medical registries. This innovative analysis was performed using the ESTS Database Platform, with the aim of reducing the gap with other areas (business, banking, and government) in the field of data quality management. In the present study the authors defined for the first time in the thoracic surgery specialty 3 specific indicators able to assess the quality of data from different perspectives in medical registries. In particular they tested this methodology using the data of those units contributing to the ESTS database with at least 100 pulmonary resections during the period July 2007 to October 2009. They explained the following concepts. – Completeness: it measures the extent to which data are not missing. – Correctness or accuracy: it measures the extent to which data are correct and reliable. – Consistency: it measures the extent to which data are correspondent and coherent with each other within the same record. Each metric was calculated on the basis of a standardized formula, that was applied to verify the completeness, correctness, and consistency of a sample of 15 variables selected from the ESTS database. This implied that the level of quality was limited to the aforementioned group of variables and not generalizable to the entire database.
The obtained cumulative level of completeness for the tested variables was 0.85, the cumulative level of correctness was 0.99, and the cumulative level of consistency was 0.98 (possible range of variability: from 0-1). Assuming from the evidences about data quality that the threshold for a good quality could be above 0.8, the overall quality level for each metric was satisfying. The authors concluded that, independently from the data quality level verified using the ESTS database, it was possible to provide a first template to evaluate the quality of medical registry. This methodological process could be useful to measure and monitor the quality of a base of data as well as to lead data quality improvement strategies. COMMENTARY Data collection is certainly the most important tool of any quality improvement initiative. The gold standard for data collection is a specialtyspecific, procedure-specific, prospectively maintained, and periodically audited database. Administrative databases are often used to this purpose. They are appealing as they are readily available and relatively inexpensive. Nevertheless they have several shortcomings: specific variables are often unavailable, which limits risk adjustment; differentiation between comorbidities and complications may be often difficult; important variables that are not billable diagnoses are excluded; and poor flexibility to classify certain comorbidities. Owing to these limitations, claims data should be avoided whenever possible. There are 2 essential steps that should be followed to construct a database: 1. the data source should be clearly defined and 2. a list of variables (and their definitions) that constitute the database should be created. These steps allow the database to be used even by subjects who did not participate in its construction and it can be audited by external data managers to assess quality of data. In this regard the General Thoracic Surgery database task forces of the Society of Thoracic Surgeons and of the European Society of Thoracic Surgeons create a joint standardization working group with the aim of creating a commonly agreed definition of variables and nomenclature of variables present in the 2 databases. More than 50 variables including risk factors, staging, and outcomes were assigned jointly agreed definitions. This allows future analyses to take place based on the data present in the 2 databases as the variables definitions are now standardized. Moreover, these definitions can be adopted
Seminars in Thoracic and Cardiovascular Surgery Volume 27, Number 2
253
THORACIC DATABASES by individual institutions during the construction or update of their institutional databases.1 Along with standardization of variable definition, the importance of quality of the data collected in a database cannot be emphasized enough. Although the research on data quality is extremely advanced in the business and financial sectors, in the medical field very few studies have focused their attention on data quality and without a systematic evaluation of the subject. The main phases composing the data quality methodology are
conceptual framework and methodology described by the Society of Thoracic Surgeons Adult Cardiac Surgery Quality measurement Task force.11,12 The CPS contains both outcome and process indicators and covers the 3 temporal domains of the thoracic surgical activity (preoperative, intraoperative, and postoperative). ESTS initially focused on lung resection for cancer as the most representative operation in our specialty. All the information used to construct the CPS must be present as data elements in the ESTS database. The followings indicators were included in the CPS.
– state reconstruction phase, which collects information about organizational processes, data collection and management procedures, quality issues, and cost of a given database; – assessment phase, which measures quality of data through relevant quality dimensions, and – improvement phase, which defines steps, strategies, and techniques for achieving new data quality targets.9,10
1. Preoperative care a. % of pts having DLCO measured, b. % of pts with CT enlarged or PETþ med nodes undergoing preoperative invasive mediastinal staging. 2. Operative care – % of pts operated on for primary neoplastic disease submitted to systematic lymph node dissection. 3. Postoperative care (outcome indicator) a. inhospital mortality, b. inhospital cardiopulmonary morbidity (pneumonia, atelectasis requiring bronchoscopy, adult respiratory distress syndrome, mechanical ventilation longer than 24 hours, pulmonary edema, pulmonary embolism, myocardial ischemia, cardiac failure, arrhythmia, stroke, and acute renal insufficiency).
Considering the assessment phase of medical registries, the literature provides a wide spectrum of well-defined and tested data quality metrics. The most widely used ones are the following: – completeness—the extent to which data are not missing and is of sufficient breadth and depth to describe the corresponding set of real world objects, – correctness or accuracy—the extent to which data are correct and reliable, and – consistency—the extent to which data are correspondent and coherent with each other (crossrecord consistency). In this regard, the European Society of Thoracic Surgeons (ESTS) Database Committee has published a preliminary study analyzing the data quality of key variables in the ESTS database with the purpose of providing a methodological template in our specialty.8 One of the most important uses of clinical data is for quality improvement initiatives. Quality of care is an elusive concept that cannot be directly measured. Like other complex constructs such as intelligence or musical ability, we often rely on composite metrics to obtain a more reliable and comprehensive assessment. In this regard, ESTS has recently proposed a Composite Performance Score based on the
254
All process indicators have been selected because they represent evidence-based recommendations from the Scientific Organizations.4,13-15 The risk-adjusted mortality and morbidity models were developed using a sample of approximately 4000 patients registered in the ESTS database from 2007-2010 and were published elsewhere.16 These models are now in the process to be revised and updated based on an analysis on a population of more than 39,000 patients registered in the ESTS database until 2014. The methodology of the Composite Performance Score calculation has been described elsewhere and it is out of the scope of this commentary.7 The CPS is currently used to assess eligibility of European Units for the Institutional Accreditation program set by the ESTS. More than 10 units have been awarded with this quality certification, acknowledging their high standards of care in line with recommended structural and procedural criteria.
Seminars in Thoracic and Cardiovascular Surgery Volume 27, Number 2
THORACIC DATABASES CONCLUSIONS Data collection is essential for any quality initiative. Clinical, prospectively maintained, and audited databases should be ideally used to this purpose to ensure high data quality.
The use of databases in the clinical practice appears valuable to improve standards of care, assess implementation of new treatments or new technologies, and evaluate performance in a reliable and accountable way.
1. Fernandez FG, Falcoz PE, Kozower BD, et al: 6. Berrisford R, Brunelli A, Rocco G, et al: The The Society of Thoracic Surgeons and the European Thoracic Surgery Database project: Modelling the risk of in-hospital death followEuropean Society of Thoracic Surgeons general ing lung resection. Eur J Cardiothorac Surg 28 thoracic surgery databases: Joint standardiza(2):306-311, 2005 tion of variable definitions and terminology. Ann Thorac Surg 99(1):368-376, 2015 7. Brunelli A, Berrisford RG, Rocco G, et al: The 2. Laudon KC. Data quality and due process in European Thoracic Database project: Compolarge inter-organizational record systems. Comsite performance score to measure quality of care after major lung resection. Eur J Cardiomunications of the ACM;29:4-11, 1986. thorac Surg 35(5):769-774, 2009 3. Burt BM, Kosinski AS, Shrager JB, et al: Thoracoscopiclobectomy is associated with acceptable 8. Salati M, Brunelli A, Dahan M, et al: Taskmorbidity and mortality in patients with predicted independent metrics to assess the data quality of medical registries using the European Society postoperative forced expiratory volume in 1secof Thoracic Surgeons (ESTS) Database. Eur J ond or diffusing capacity for carbon monoxide Cardiothorac Surg 40(1):91-98, 2011 less than 40% of normal. J Thorac Cardiovasc 9. Batini C, Cappiello C, Francalanci C, et al: Surg 148(1):19-28, 2014; [discussion 28-29] 4. Brunelli A, Kim AW, Berger KI, et al: PhysioMethodologies for data quality assessment and improvement. ACM Comput Surveys 41: logic evaluation of the patient with lung cancer 1-52, 2009 being considered for resectional surgery: Diagnosis and management of lung cancer, 3rd ed: 10. Cappiello C, Francalanci C, Pernici B: A ruleAmerican College of Chest Physicians evidencebased methodology to support information quality assessment and improvement. Informabased clinical practice guidelines. Chest 143: tion Quality of Studies in Communication e166S-e1690, 2013 (suppl 5) Sciences’ 2:137-154, 2004 5. Onaitis MW, Petersen RP, Balderson SS, et al: Thoracoscopic lobectomy is a safe and versatile 11. O’Brien SM, Shahian DM, DeLong ER, et al: Quality measurement in adult cardiac surgery: procedure: Experience with 500 consecutive Part 2—Statistical considerations in composite patients. Ann Surg 244:420-425, 2006
Seminars in Thoracic and Cardiovascular Surgery Volume 27, Number 2
12.
13.
14.
15.
16.
measure scoring and provider rating. Ann Thorac Surg 83:S13-S26, 2007 (suppl 4) Shahian DM, Edwards FH, Ferraris VA, et al: Quality measurement in adult cardiac surgery: Part 1—Conceptual framework and measure selection. Ann Thorac Surg 83:S3-S12, 2007 (suppl 4) Brunelli A, Charloux A, Bolliger CT, et al: European Respiratory Society and European Society of Thoracic Surgeons joint task force on fitness for radical therapy. ERS/ESTS clinical guidelines on fitness for radical therapy in lung cancer patients (surgery and chemo-radiotherapy). Eur Respir J 34(1):17-41, 2009 De Leyn P, Dooms C, Kuzdzal J, et al: Revised ESTS guidelines for preoperative mediastinal lymph node staging for non-small-cell lung cancer. Eur J Cardiothorac Surg 45(5): 787-798, 2014 Lardinois D, De Leyn P, Van Schil P, et al: ESTS guidelines for intraoperative lymph node staging in non-small cell lung cancer. Eur J Cardiothorac Surg 30(5):787-792, 2006 Brunelli A, Rocco G, Van Raemdonck D, et al: Lessons learned from the European thoracic surgery database: The Composite Performance Score. Eur J Surg Oncol 36:S93-S99, 2010 (suppl 1)
255
What the Surgeon Needs to Know About Databases Michele Salati, MD,* and Alessandro Brunelli, MD† Data collection is one of the most important instruments of any quality improvement initiatives. We have selected, summarized, and discussed 5 recent contributions mostly based on large international databases, which we considered most relevant to our specialty. They focused on different aspects: the selection and rigorous definition of the variables contained in the data set, the evaluation of a treatment or a pathway of care by the analysis of the observed outcomes, the identification of risk factors able to affect the surgical course, the measurement of the quality provided by a care giver, and the assessment of the quality of the data collected and the planning of quality improvement activities.
Alessandro Brunelli, MD
Semin Thoracic Surg 27:250–255 I 2015 Elsevier Inc. All rights reserved.
Central Message
Keywords: database, quality of care, quality of data, thoracic surgery, lung resection, VATS lobectomy
Collection of clinical data is essential to improve standards of care and evaluate quality of care in a reliable and accountable way.
INTRODUCTION The clinical practice is increasingly based on the scientific evidence. Original studies, review papers, meta-analysis, and guidelines represent a solid foundation that supports everyday our clinical strategies. In all the aforementioned studies, the analytic process is preceded by the data collection phase. The creation and maintenance of a database is a complex activity able to influence the process of knowledge and decision-making. To obtain a reliable database, especially if we are building or examining a surgical registry, we need to focus our attention on – the selection and rigorous definition of the variables contained in the data set, – the finality of the database: – evaluation of a treatment or a pathway of care by the analysis of the observed outcomes, – identification of risk factors able to affect the surgical course, – measurement of the quality provided by a care giver, and – the assessment of the quality of the data collected and the planning of quality improvement activities. *
Division of Thoracic Surgery, Ospedali Riuniti Ancona, Italy. Department of Thoracic Surgery, St. James’s University Hospital, Leeds, UK.
†
Address reprint requests to Alessandro Brunelli, Dept. Thoracic Surgery, St. James’s University Hospital, Beckett St, Bexley Wing, LS7 9FT, Leeds, UK. E-mail: [email protected]
250
The aim of this work is to explain these points using recently published articles based on the analysis of large international databases.
SELECTION AND DEFINITION OF THE VARIABLES The Society of Thoracic Surgeons and the European Society of Thoracic Surgeons General Thoracic Surgery databases: Joint standardization of variable definition and terminology. Fernandez FG, Falcoz PE, Kozower BD, et al. Ann Thorac Surg 99:368-376, 20151 Representatives of the Society of Thoracic Surgeons (STS) and of the European Society of Thoracic Surgeons (ESTS) Database Committees contributed to the development of a joint document for the variables standardization in thoracic surgery registries. They defined, harmonizing different terminologies, 62 variables usually collected in monospecialistic thoracic surgery databases. The first step for the creation of a medical registry is represented by the identification of the aim of the data collection. In fact, taking into account the real word that the database should describe and the objectives and information that it would provide, it is possible to select those variables for collection. The next step is the definition of the variables. This phase is extremely important, being able to heavily influence the quality of data across the years, especially in multi-institutional registries.2 In this work, the authors compared the 2 major international general thoracic surgery databases, the STS and the ESTS
1043-0679/$-see front matter ª 2015 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1053/j.semtcvs.2015.06.004
THORACIC DATABASES registries, which collect medical and thoracic surgery data from North American and European patients. Even if these databases share the same objective of quality of care monitoring and improvement, the authors identified some discrepancies in the definitions of 50 variables common to both registries. Each organization reviewed and proposed a specific definition for each of these variables. Then, after several meetings, a common set of variable definitions was agreed by both societies. A similar process was adopted for a second group of 12 variables, initially not in common but desirable in both databases. In conclusion, the article presented a list of 62 parameters strictly defined by a joint task force of the STS and ESTS, which could be used as foundation for the creation of mono- or multi-institutional base of data in thoracic surgery. SURGICAL TREATMENT EVALUATION Thoracoscopic lobectomy is associated with acceptable morbidity and mortality in patients with predicted postoperative forced expiratory volume in 1 second or diffusing capacity for carbon monoxide less than 40% of normal. Burt BM, Kosinski AS, Shrager JB, et al. J Thorac Cardiovasc Surg 148:19-29, 20143 The authors tried to verify if the innovative thoracoscopic surgical (VATS) lobectomy approach was superior to the standard open lobectomy, considered the cornerstone for the treatment of primary lung cancer, in patients with reduced pulmonary function. The analysis was performed on data derived from the Society of Thoracic Surgeons General Thoracic Database. This article represents an example of how the data collected in international registries could be used to derive information able to lead the clinical practice in the future. In fact, the most recent guidelines proposed by the American College of Chest Physicians, regarding the preoperative physiological evaluation of lung resection candidates, defined the group of patients with a predicted postoperative (ppo) forced expiratory volume in 1 second (FEV1) or diffusing capacity for carbon monoxide (DLCO) or both less than 40% of predicted as at high surgical risk for postoperative complications and death.4 At the same time, growing evidence showed the advantages of the VATS lobectomy technique in terms of reduced morbidity, mortality, postoperative chest surgical discomfort, and immunologic impairment.5 Nevertheless, a study aimed at testing the VATS lobectomy results in the specific subgroup of highrisk lung resection candidates was not yet performed.
To verify their hypothesis, the authors queried the Society of Thoracic Surgeons General Thoracic Database. This large registry gave them the possibility to compare 6802 patients submitted to open lobectomy vs 6574 submitted to VATS lobectomy. Firstly, a multivariate analysis was performed demonstrating the following: although decreasing ppoFEV1 and ppoDLCO values were independent predictors of cardiopulmonary morbidity in both the open lobectomy and VATS lobectomy groups, the ppoFEV1 was not related to the postoperative mortality in VATS lobectomy patients. Then the authors performed a propensity score analysis to match 1:1 open and VATS patients, taking into account several preoperative characteristics (21 factors, excluding pulmonary function) as well as the pathologic stage of disease. They obtained 4215 well-matched pairs of patients for verifying the rates of cardiopulmonary morbidity and mortality at incremental ranges of ppoFEV1% and ppoDLCO%. They found that in patients with ppoFEV1 o 40%, those submitted to VATS lobectomy had a lower rates of complications and mortality compared with those submitted to open surgery (13% vs 22% and 0.7% vs 4.8%, respectively). Similarly, in patients with ppoDLCO o 40% those operated on through VATS had a much lower incidence of complications and mortality compared with those operated through thoracotomy (10% vs 15% and 2% vs 5%, respectively). Consequently, the authors concluded that VATS lobectomy could be considered a reasonably safe surgical procedure to treat patients with lung cancer traditionally considered at high risk for compromised pulmonary function. RISK FACTORS IDENTIFICATION The European Thoracic Surgery Database project: Modeling the risk of inhospital death following lung resection. Berrisford R, Brunelli A, Rocco G, et al. Eur J Cardiothorac Surg 28:306-311, 20056 This article published by Berrisford and colleagues in 2005 on behalf of the Audit and guidelines committee of the European Society of Thoracic Surgeons and the European Association of Cardiothoracic Surgeons is an example of risk model analysis performed on data derived from a large international registry, such as the European Society of Thoracic Surgeons Database, which, at that time, collected data from 112 thoracic surgery units from different European Countries (only 27 of these units, sending to the database data of sufficient completeness, were enrolled in the study). Complex analytic models are usually needed to identify risk factors influencing the results of a
Seminars in Thoracic and Cardiovascular Surgery Volume 27, Number 2
251
THORACIC DATABASES therapeutic strategy. This is particularly evident in studies of risk modeling for complications and deaths that are rare events after surgical treatment. The reliability of the analysis is dependent on the amount and quality of data collected. Consequently, these types of studies are based on large databases, where the information should be clearly encoded and stable, gathering data from multiple institutions for several years. The authors developed a model to identify factors able to predict the inhospital mortality in patients submitted to lung resection. A total of 3426 patients (wedge or segmentectomies 26%, lobectomies 59%, pneumonectomies 14%, and volume reduction 1%) were used in the analysis, with an overall mortality of 1.9% (66 patients). The initial population was divided into 2 different groups: a first group (60% of the patients) was used for performing a logistic regression analysis aimed at identifying the baseline characteristics associated with the inhospital mortality. Then the mortality model, incorporating as risk factors age, dyspnea score, American Society of Anaesthesiologists Score, and type of resection, was built and tested on the remaining 40% of patients. Taking into account the model tendency at underestimating mortality for patients at medium risk and overestimating for those at high risk and considering that the model was based mostly on subjective factors, the authors decided to refine the analysis, developing a second model focused only on patients with lung cancer (85% of the initial cohort). This analysis yielded the European Society Objective Score (ESOS), which is a mortality risk model based just on age and ppoFEV1 and derived from a population of 1694 patients. The ESOS was able to predict inhospital death with high concordance between predicted and observed mortality for all the risk classes of the 1128 patients testing group. The authors recommended the use of the ESOS mostly for monitoring the performance of a unit and improving the results on long-term survival. In fact, special caution should be taken applying the information derived from a risk model for the clinical management of individual patients, especially in terms of early postsurgical mortality. QUALITY OF CARE EVALUATION The European Thoracic Database project: Composite performance score to measure quality of care after major lung resection. Brunelli A, Berrisford RG, Rocco G, et al. Eur J Cardiothorac Surg 35:769-774, 20097 This article represents a template for developing a unique performance score (the Composite
252
Performance Score [CPS]), that, incorporating multiple indicators of outcome (morbidity and mortality), intraoperative and perioperative processes, is able to measure the overall health care quality. This project, directly implemented by the European Society of Thoracic Surgeons, was performed using the platform of the European Society of Thoracic Surgeons database. This study shows the need for a careful planning of the variable selection and inclusion within a data set to obtain valuable data to analyze. In fact, considering the aim of this project, which was to develop a model for measuring the quality of care at a European level, it was necessary to collect information about several domains of the thoracic surgery clinical practice for a long period of time and from multiple contributors. A total of 1656 procedures (major lung resections for primary lung cancer), uploaded from 10 European thoracic surgery units, were selected and the correspondent data validated for completeness and consistency before the analysis. The authors identified and clearly defined 4 different quality indicators (2 process indicators and 2 outcome indicators). The process indicators were represented by the proportion of patients with predicted postoperative carbon monoxide lung diffusion capacity estimated before the surgical procedure and the proportion of patients with a systematic lymph nodes dissection performed during the operation. The outcome indicators were represented by the risk-adjusted mortality and the risk-adjusted cardiopulmonary morbidity. The predictors of morbidity and mortality used for building the risk models were identified by a 3-step process, consisting of a univariate analysis, a multivariate regression analysis, and a bootstrap procedure for the final models validation. The resulting predictors associated with cardiopulmonary morbidity were age, predicted postoperative forced expiratory volume in 1 second, and extended resection (ie, lung resection associated with resection of chest wall, diaphragm, pericardium, or other mediastinal structures); the ones associated with mortality were age and extended resection. For each unit the 4 indicators were calculated and after a procedure of rescaling, necessary to harmonize parameters with a high range of variability, summed to obtain the CPS. The authors verified that using the CPS as quality of care indicator all the evaluated units changed their position in comparison with the rank obtained by using a single indicator such as the mortality. This project could be used as a methodological template (in terms of data collection and the data mining strategies) for the development of
Seminars in Thoracic and Cardiovascular Surgery Volume 27, Number 2
THORACIC DATABASES multidomain scores for the quality of care assessment. A composite score such as the CPS, including process and outcome measures of quality, is a reliable indicator of performance, which is able to express the level of quality as a single numeric parameter. The CPS could lead processes of quality of care monitoring and quality improvement strategies in thoracic surgery either in a single institution or at an international level. DATA QUALITY Task-independent metrics to assess the data quality of medical registries using the European Society of Thoracic Surgeons (ESTS) database. Salati M, Brunelli A, Dahan M, et al. Eur J Cardiothorac Surg 40(1):91-98, 20118 The process of knowledge is a pathway that starts from the data collection. The interpretation of the available data allows obtaining information for planning the decision-making strategies.9,10 Taking into account this chain, to ensure a high quality of data is essential, because, despite the best analytic process, it directly influences the reliability of the resulting evidence. In the present work the authors described the methodological approach to measure the quality of data in medical registries. This innovative analysis was performed using the ESTS Database Platform, with the aim of reducing the gap with other areas (business, banking, and government) in the field of data quality management. In the present study the authors defined for the first time in the thoracic surgery specialty 3 specific indicators able to assess the quality of data from different perspectives in medical registries. In particular they tested this methodology using the data of those units contributing to the ESTS database with at least 100 pulmonary resections during the period July 2007 to October 2009. They explained the following concepts. – Completeness: it measures the extent to which data are not missing. – Correctness or accuracy: it measures the extent to which data are correct and reliable. – Consistency: it measures the extent to which data are correspondent and coherent with each other within the same record. Each metric was calculated on the basis of a standardized formula, that was applied to verify the completeness, correctness, and consistency of a sample of 15 variables selected from the ESTS database. This implied that the level of quality was limited to the aforementioned group of variables and not generalizable to the entire database.
The obtained cumulative level of completeness for the tested variables was 0.85, the cumulative level of correctness was 0.99, and the cumulative level of consistency was 0.98 (possible range of variability: from 0-1). Assuming from the evidences about data quality that the threshold for a good quality could be above 0.8, the overall quality level for each metric was satisfying. The authors concluded that, independently from the data quality level verified using the ESTS database, it was possible to provide a first template to evaluate the quality of medical registry. This methodological process could be useful to measure and monitor the quality of a base of data as well as to lead data quality improvement strategies. COMMENTARY Data collection is certainly the most important tool of any quality improvement initiative. The gold standard for data collection is a specialtyspecific, procedure-specific, prospectively maintained, and periodically audited database. Administrative databases are often used to this purpose. They are appealing as they are readily available and relatively inexpensive. Nevertheless they have several shortcomings: specific variables are often unavailable, which limits risk adjustment; differentiation between comorbidities and complications may be often difficult; important variables that are not billable diagnoses are excluded; and poor flexibility to classify certain comorbidities. Owing to these limitations, claims data should be avoided whenever possible. There are 2 essential steps that should be followed to construct a database: 1. the data source should be clearly defined and 2. a list of variables (and their definitions) that constitute the database should be created. These steps allow the database to be used even by subjects who did not participate in its construction and it can be audited by external data managers to assess quality of data. In this regard the General Thoracic Surgery database task forces of the Society of Thoracic Surgeons and of the European Society of Thoracic Surgeons create a joint standardization working group with the aim of creating a commonly agreed definition of variables and nomenclature of variables present in the 2 databases. More than 50 variables including risk factors, staging, and outcomes were assigned jointly agreed definitions. This allows future analyses to take place based on the data present in the 2 databases as the variables definitions are now standardized. Moreover, these definitions can be adopted
Seminars in Thoracic and Cardiovascular Surgery Volume 27, Number 2
253
THORACIC DATABASES by individual institutions during the construction or update of their institutional databases.1 Along with standardization of variable definition, the importance of quality of the data collected in a database cannot be emphasized enough. Although the research on data quality is extremely advanced in the business and financial sectors, in the medical field very few studies have focused their attention on data quality and without a systematic evaluation of the subject. The main phases composing the data quality methodology are
conceptual framework and methodology described by the Society of Thoracic Surgeons Adult Cardiac Surgery Quality measurement Task force.11,12 The CPS contains both outcome and process indicators and covers the 3 temporal domains of the thoracic surgical activity (preoperative, intraoperative, and postoperative). ESTS initially focused on lung resection for cancer as the most representative operation in our specialty. All the information used to construct the CPS must be present as data elements in the ESTS database. The followings indicators were included in the CPS.
– state reconstruction phase, which collects information about organizational processes, data collection and management procedures, quality issues, and cost of a given database; – assessment phase, which measures quality of data through relevant quality dimensions, and – improvement phase, which defines steps, strategies, and techniques for achieving new data quality targets.9,10
1. Preoperative care a. % of pts having DLCO measured, b. % of pts with CT enlarged or PETþ med nodes undergoing preoperative invasive mediastinal staging. 2. Operative care – % of pts operated on for primary neoplastic disease submitted to systematic lymph node dissection. 3. Postoperative care (outcome indicator) a. inhospital mortality, b. inhospital cardiopulmonary morbidity (pneumonia, atelectasis requiring bronchoscopy, adult respiratory distress syndrome, mechanical ventilation longer than 24 hours, pulmonary edema, pulmonary embolism, myocardial ischemia, cardiac failure, arrhythmia, stroke, and acute renal insufficiency).
Considering the assessment phase of medical registries, the literature provides a wide spectrum of well-defined and tested data quality metrics. The most widely used ones are the following: – completeness—the extent to which data are not missing and is of sufficient breadth and depth to describe the corresponding set of real world objects, – correctness or accuracy—the extent to which data are correct and reliable, and – consistency—the extent to which data are correspondent and coherent with each other (crossrecord consistency). In this regard, the European Society of Thoracic Surgeons (ESTS) Database Committee has published a preliminary study analyzing the data quality of key variables in the ESTS database with the purpose of providing a methodological template in our specialty.8 One of the most important uses of clinical data is for quality improvement initiatives. Quality of care is an elusive concept that cannot be directly measured. Like other complex constructs such as intelligence or musical ability, we often rely on composite metrics to obtain a more reliable and comprehensive assessment. In this regard, ESTS has recently proposed a Composite Performance Score based on the
254
All process indicators have been selected because they represent evidence-based recommendations from the Scientific Organizations.4,13-15 The risk-adjusted mortality and morbidity models were developed using a sample of approximately 4000 patients registered in the ESTS database from 2007-2010 and were published elsewhere.16 These models are now in the process to be revised and updated based on an analysis on a population of more than 39,000 patients registered in the ESTS database until 2014. The methodology of the Composite Performance Score calculation has been described elsewhere and it is out of the scope of this commentary.7 The CPS is currently used to assess eligibility of European Units for the Institutional Accreditation program set by the ESTS. More than 10 units have been awarded with this quality certification, acknowledging their high standards of care in line with recommended structural and procedural criteria.
Seminars in Thoracic and Cardiovascular Surgery Volume 27, Number 2
THORACIC DATABASES CONCLUSIONS Data collection is essential for any quality initiative. Clinical, prospectively maintained, and audited databases should be ideally used to this purpose to ensure high data quality.
The use of databases in the clinical practice appears valuable to improve standards of care, assess implementation of new treatments or new technologies, and evaluate performance in a reliable and accountable way.
1. Fernandez FG, Falcoz PE, Kozower BD, et al: 6. Berrisford R, Brunelli A, Rocco G, et al: The The Society of Thoracic Surgeons and the European Thoracic Surgery Database project: Modelling the risk of in-hospital death followEuropean Society of Thoracic Surgeons general ing lung resection. Eur J Cardiothorac Surg 28 thoracic surgery databases: Joint standardiza(2):306-311, 2005 tion of variable definitions and terminology. Ann Thorac Surg 99(1):368-376, 2015 7. Brunelli A, Berrisford RG, Rocco G, et al: The 2. Laudon KC. Data quality and due process in European Thoracic Database project: Compolarge inter-organizational record systems. Comsite performance score to measure quality of care after major lung resection. Eur J Cardiomunications of the ACM;29:4-11, 1986. thorac Surg 35(5):769-774, 2009 3. Burt BM, Kosinski AS, Shrager JB, et al: Thoracoscopiclobectomy is associated with acceptable 8. Salati M, Brunelli A, Dahan M, et al: Taskmorbidity and mortality in patients with predicted independent metrics to assess the data quality of medical registries using the European Society postoperative forced expiratory volume in 1secof Thoracic Surgeons (ESTS) Database. Eur J ond or diffusing capacity for carbon monoxide Cardiothorac Surg 40(1):91-98, 2011 less than 40% of normal. J Thorac Cardiovasc 9. Batini C, Cappiello C, Francalanci C, et al: Surg 148(1):19-28, 2014; [discussion 28-29] 4. Brunelli A, Kim AW, Berger KI, et al: PhysioMethodologies for data quality assessment and improvement. ACM Comput Surveys 41: logic evaluation of the patient with lung cancer 1-52, 2009 being considered for resectional surgery: Diagnosis and management of lung cancer, 3rd ed: 10. Cappiello C, Francalanci C, Pernici B: A ruleAmerican College of Chest Physicians evidencebased methodology to support information quality assessment and improvement. Informabased clinical practice guidelines. Chest 143: tion Quality of Studies in Communication e166S-e1690, 2013 (suppl 5) Sciences’ 2:137-154, 2004 5. Onaitis MW, Petersen RP, Balderson SS, et al: Thoracoscopic lobectomy is a safe and versatile 11. O’Brien SM, Shahian DM, DeLong ER, et al: Quality measurement in adult cardiac surgery: procedure: Experience with 500 consecutive Part 2—Statistical considerations in composite patients. Ann Surg 244:420-425, 2006
Seminars in Thoracic and Cardiovascular Surgery Volume 27, Number 2
12.
13.
14.
15.
16.
measure scoring and provider rating. Ann Thorac Surg 83:S13-S26, 2007 (suppl 4) Shahian DM, Edwards FH, Ferraris VA, et al: Quality measurement in adult cardiac surgery: Part 1—Conceptual framework and measure selection. Ann Thorac Surg 83:S3-S12, 2007 (suppl 4) Brunelli A, Charloux A, Bolliger CT, et al: European Respiratory Society and European Society of Thoracic Surgeons joint task force on fitness for radical therapy. ERS/ESTS clinical guidelines on fitness for radical therapy in lung cancer patients (surgery and chemo-radiotherapy). Eur Respir J 34(1):17-41, 2009 De Leyn P, Dooms C, Kuzdzal J, et al: Revised ESTS guidelines for preoperative mediastinal lymph node staging for non-small-cell lung cancer. Eur J Cardiothorac Surg 45(5): 787-798, 2014 Lardinois D, De Leyn P, Van Schil P, et al: ESTS guidelines for intraoperative lymph node staging in non-small cell lung cancer. Eur J Cardiothorac Surg 30(5):787-792, 2006 Brunelli A, Rocco G, Van Raemdonck D, et al: Lessons learned from the European thoracic surgery database: The Composite Performance Score. Eur J Surg Oncol 36:S93-S99, 2010 (suppl 1)
255