Use of Control Charts for Identifying Worsening Postoperative Mortality or Serious Morbidity Performance After Colectomy

Use of Control Charts for Identifying Worsening Postoperative Mortality or Serious Morbidity Performance After Colectomy Elise H. Lawson, MD, MSHS, Jeffrey Lin, and Clifford Y. Ko, MD, MS, MSHS Control charts are increasingly being used by hospitals as a tool for monitoring and improving health care quality. Our objective was to determine whether control charts of mortality or serious morbidity (M&M) rates after colectomy predict changes in outlier status for risk-adjusted M&M rates using data from a surgical registry, the American College of Surgeons National Surgical Quality Improvement Program. Control charts of monthly M&M rates for 95 hospitals were analyzed for indicators of a performance change in 2009 (vs 2008) using standard rules. Hospitals were also classified as having better, worse, or no change in outlier status for risk-adjusted M&M rates in 2009 (vs 2008). Agreement between these methods was fair (weighted ␬ ⴝ 0.379). There were no hospitals labeled as improving by one method and worsening by the other. Control charts predicted nonworsening performance well (specificity 0.866), but failed to identify 38.5% of hospitals with worsened outlier status. Although we did not demonstrate perfect agreement, our results suggest that these methods are measuring similar constructs of quality and are likely complementary uses of the same clinical data source. Semin Colon Rectal Surg 23:146-152 © 2012 Elsevier Inc. All rights reserved.

N

umerous reports have demonstrated high rates of postoperative complications, with colectomy procedures in particular having significant associated risk of morbidity and mortality. As a result, policy makers and payers have developed pay-for-performance and public reporting initiatives aimed at rewarding high-quality hospitals and/or penalizing hospitals demonstrated to have substandard performance. Additionally, hospitals have responded to reports of poor quality by participating in national clinical data registries and by forming local collaboratives focused on shared learning. However, despite this increased interest and commitment, implementing and maintaining interventions that successfully improve quality can be challenging. Six sigma is a business management strategy widely used in industries to improve quality and efficiency and to reduce Department of Surgery, David Geffen School of Medicine, University of California Los Angeles, Los Angeles, CA. E.H.L.’s time was supported by the VA Health Services Research and Development program (RWJ 65-020) and the American College of Surgeons through the Robert Wood Johnson Foundation Clinical Scholars Program. Address reprint requests to: Elise H. Lawson, MD, MSHS, Department of Surgery, David Geffen School of Medicine, University of California Los Angeles, 10833 Le Conte Ave 72–215 CHS, Los Angeles, CA 90095. E-mail: [email protected]

146

1043-1489/$-see front matter © 2012 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1053/j.scrs.2012.07.003

costs. Groups such as the Commonwealth Health Corporation, the Joint Commission Center for Transforming Healthcare, and the Institute for Healthcare Improvement have championed the use of this framework for quality improvement projects in health care, and it has been adopted by many hospitals. A key component of six sigma is the use of control charts for measuring and monitoring manufacturing processes or clinical outcomes. When used within the framework of a broader quality improvement methodology, such as six sigma, control charts can be a powerful tool for improving both postoperative outcomes and patient satisfaction, as well as for sustaining change.1-4 In this article, we compare the use of control charts for detecting changes in hospital-level postoperative mortality or serious morbidity (M&M) rates with another validated method for benchmarking performance: outlier status for risk-adjusted M&M as determined by the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP). We chose to focus on colectomy procedures because they are commonly performed and have relatively high associated rates of postoperative morbidity and mortality.5 Later in the text, we outline the statistical methods underlying each of these techniques and discuss the relative strengths and weaknesses of each as a measurement tool. Additionally, we assessed agreement between the 2 methods

Use of control charts for identifying worsening postoperative M&M performance after colectomy

147

Table 1 Current Procedural Terminology Codes for Colectomy Procedures Included in Study Current Procedural Terminology Code 44140 44141 44143 44144 44145 44146 44147 44150 44151 44160 44204 44205 44206 44207 44208 44210

Code Description Colectomy, partial; with anastomosis Colectomy, partial; with skin level cecostomy or colostomy Colectomy, partial; with end colostomy and distal closure of segment (Hartmann-type procedure) Colectomy, partial; with resection, with colostomy or ileostomy and creation of mucous fistula Colectomy, partial; with coloproctostomy (low pelvic anastomosis) Colectomy, partial; with coloproctostomy (low pelvic anastomosis), with colostomy Colectomy, partial; abdominal and transanal approach Colectomy, total, abdominal, without proctectomy; with ileostomy or ileoproctostomy Colectomy, total, abdominal, without proctectomy; with continent ileostomy Colectomy, partial, with removal of terminal ileum with ileocolostomy Laparoscopy, surgical; colectomy, partial, with anastomosis Laparoscopy, surgical; colectomy, partial with removal of terminal ileum with ileocolostomy Laparoscopy, surgical; colectomy, partial, with anastomosis, with end colostomy and closure of distal segment (Hartmann type procedure) Laparoscopy, surgical; colectomy, partial, with anastomosis, with coloproctostomy (low pelvic anastomosis) Laparoscopy, surgical; colectomy, partial, with anastomosis, with coloproctostomy (low pelvic anastomosis) with colostomy Laparoscopy, surgical; colectomy, total, abdominal, without proctectomy, with ileostomy or ileoproctostomy

on the classification of a hospital as having worsening performance for postoperative M&M after colectomy.

Methods Data Source and Study Sample The data source for this study was the ACS-NSQIP, which is an institution-based multispecialty surgical registry of patient risk factors and postoperative outcomes. Hospital participation in ACS-NSQIP is voluntary. However, all data are collected by abstractors who undergo continued training and testing by ACS-NSQIP to ensure uniform use of strict variable definitions. In addition, hospitals are audited to ensure standardized data collection, with audit results demonstrating substantial or almost perfect agreement on the coding of most variables.6 The sampling strategy consists of collecting data for the first 40 cases performed within consecutive 8-day cycles. Data are collected across several surgical specialties, including general, vascular, and specific subspecialties. Sources for data are clinical medical records and the patients. Data collected include demographics, preoperative risk factors and laboratory values, operative information, and postoperative outcomes within 30 days of the index operation. Participating hospitals receive semiannual reports with riskadjusted outcomes from ACS-NSQIP that allow them to benchmark their performance with national averages.7,8 The study sample was derived from hospitals that participated in ACS-NSQIP in both 2008 and 2009. Colectomies were identified using the Current Procedural Terminology codes listed in Table 1. Only hospitals performing at least 2 colectomy procedures per month for 2008 and 2009 were included in the analysis. Patient characteristics were compared by year (2008 vs 2009).

Outcome of Interest The outcome of interest was any occurrence of 30-day postoperative M&M for colectomy procedures. Serious morbidity includes any occurrence of myocardial infarction, cardiac arrest requiring cardiopulmonary resuscitation, deep venous thrombosis requiring therapy, pulmonary embolism, pneumonia without preoperative pneumonia, unplanned reintubation without prior ventilator dependence, sepsis, septic shock, deep or organ-space wound infection, wound dehiscence, progressive renal insufficiency or acute renal failure without preoperative renal failure or dialysis, urinary tract infection, or unplanned reoperation. This definition is consistent with that used in a colectomy surgery quality measure developed by the ACS-NSQIP and endorsed by the National Quality Forum.

Control Charts There are many types of control charts that can be used in the health care setting, depending on the type of process or outcome being analyzed. For this study, we created p-charts, which are used to evaluate variation in rates of categorical variables during several periods (in this case, variation in the monthly rate of M&M after colectomy). These p-charts allow the subgroup size to vary between periods (ie, the number of patients undergoing a colectomy procedure each month). A p-chart is composed of raw data graphed during several periods, a centerline, and control limits, as demonstrated in Fig. 1. The steps for creating the p-charts in this study are as follows: (1) Determine the number of patients who developed the outcome of interest each month; (2) Determine the total number of patients who were at risk of developing the outcome each month (n) (ie, the number of patients who

E.H. Lawson, J. Lin, and C.Y. Ko

148

Figure 1 Example of p-chart using raw rates of mortality or serious morbidity, by month. In some cases, the lower limits will be calculated to be less than zero. When this occurs (as in the above p-chart), the lower control limit defaults to the x-axis, as a negative rate of M&M would not make sense. Standard rules for analyzing p-charts for indicators of a change in performance (also known as indicators of special cause variation) are listed below the p-chart.9 UCL, upper control limit (⫹3 sigma); LCL, lower control limit (⫺3 sigma).

underwent a colectomy procedure each month); (3) Calculate and plot the percentage of patients who developed the outcome each month (solid line, Fig. 1); (4) Calculate the average rate of the outcome during all the periods (p៮ ) and plot as a horizontal line across the chart (centerline); (5) Calculate 1, 2, and 3 sigma of the data at each time point and plot as sigma limits and control limits. These limits are similar to confidence intervals in that they provide information about the expected variation in values for a data point in a given period. They are calculated using sigmas, which are estimates of the standard deviation of the data based on the sample size during a given period.3,4,9 All p-charts for this study were created using the Excel QI Macro, which automatically performs the necessary calculations and creates the chart using raw data entered in a spreadsheet. Non–risk-adjusted monthly rates of M&M after colectomy were calculated for each hospital for 2008 and 2009 and used to create p-charts using 15, 18, 21, and 24 months of data (ie, 4 p-charts for each hospital). The p-charts were analyzed using standard rules (Fig. 1), and hospitals were classified into 3 groups, using 2008 as the baseline: (1) Indicator of worsening performance and higher raw rate in 2009, (2)

Indicator of improving performance and lower raw rate in 2009, and (3) No indicator of change in performance or unclear if performance is worsening or improving.

ACS-NSQIP Outlier Status Hospital outlier status for risk-adjusted postoperative M&M in 2008 and 2009 was determined using standard ACS-NSQIP methodology. Briefly, hierarchical multivariate logistic models were developed for 2008 and 2009, with hospital treated as a random effect. This approach accounts for clustering of patients within hospitals by allowing each hospital to have a different random intercept and incorporates the empirical Bayes method, which optimally combines information from a given hospital with information from the entire sample of hospitals to arrive at a best prediction for the given hospital’s performance. This adjustment tends to shrink predicted hospital performance toward the grand mean hospital value, with the greatest shrinkage effect occurring when the hospital sample size is small and when the hospital’s estimate is extreme compared with that of other hospitals. The entire ACS-NSQIP dataset was used for each year to develop the models and to determine outlier status for the

Use of control charts for identifying worsening postoperative M&M performance after colectomy hospitals included in the study. Covariates included procedure, indication for operation, functional status before surgery, American Society of Anesthesiology class, wound class, and emergency procedure, as defined and collected within the ACS-NSQIP. These covariates are consistent with those included in the colectomy surgery quality measure developed by the ACS-NSQIP and endorsed by the National Quality Forum. Individual hospital performance for 2008 and 2009 was estimated using the hospital intercept odds ratio derived from the applicable hierarchical multivariate logistic regression model. This odds ratio estimates the odds of the outcome at the specified hospital versus the odds of the outcome at a theoretical “average” hospital, adjusted for other covariates included in the model. A hospital was considered a high outlier (worse than expected performance) if its odds ratio was ⬎1 (P ⬍ 0.05) and a low outlier (better than expected performance) if its odds ratio was ⬍1 (P ⬍ 0.05). Hospitals with odds ratios not significantly different from 1 (P ⱖ 0.05) were labeled “as expected” for their given patient population. Owing to the small number of hospitals that are classified as statistically significant outliers using this method, ACSNSQIP additionally identifies hospitals as worse or better performers if they are in the top or bottom decile of odds ratios, respectively. Outlier status in 2008 and 2009 was compared, and hospitals were classified into 3 groups, using 2008 as the baseline: (1) Worse— hospitals that became high outliers or stopped being low outliers in 2009, (2) Improved— hospitals that stopped being high outliers or became low outliers in 2009, and (3) No change in outlier status in 2009 (Table 2).

Statistical Analysis Classification of hospital performance for postoperative M&M after colectomy by each method was cross-tabulated. Groups were compared with the ␹2 and weighted ␬ statistics. The “improved” and “no change” groups were then combined to test for correlation between the binary categories “worse”

Table 2 Classification of Hospitals by Change in Outlier Status for Risk-Adjusted Mortality or Serious Morbidity (M&M) After Colectomy in 2009 vs 2008 2009

2008 High outlier As expected Low outlier

High Outlier

As Expected

Low Outlier

No change Worse Worse

Improved No change Worse

Improved Improved No change

High outlier: risk-adjusted M&M rate is significantly higher than expected (odds ratio >1, P < 0.05) or in top decile for odds ratios. As expected: risk-adjusted M&M rate is as expected (odds ratio not significantly different from 1, P > 0.05). Low outlier: risk-adjusted M&M rate is significantly lower than expected (odds ratio <1, P < 0.05) or in bottom decile for odds ratios.

149

versus “not worse.” Sensitivity, specificity, positive predictive value, and negative predictive value of control charts for predicting worse versus not-worse ACS-NSQIP outlier status were calculated. All analyses were performed using SAS version 9.2 software (SAS Institute, Inc, Cary, NC).

Results There were 211 and 236 hospitals in the ACS-NSQIP database that performed colectomy procedures in 2008 and 2009, respectively. After applying the exclusion criteria, the study population included 95 hospitals, with 12,348 patients in 2008 and 13,163 patients in 2009. The patient populations in 2008 and 2009 were similar, with no clinically meaningful differences between the groups, although some differences were statistically significant because of the large sample size (Table 3). The overall rate of M&M was 22.0% in 2008 and 22.4% in 2009 (P ⫽ 0.487). Using 2008 as the baseline, 19 hospitals had a control chart indicator of worsening performance and a higher rate for postoperative M&M for colectomy procedures in 2009, 17 had an indicator of improving performance and a lower rate, and 59 had no indication of a change in performance noted. ACS-NSQIP outlier status for postoperative M&M worsened for 13 hospitals, improved for 17 hospitals, and stayed the same for 65 hospitals in 2009, compared with 2008. Of the hospitals that worsened, 6 became high outliers in 2009 and 7 were no longer low outliers. Of the hospitals that improved, 8 became low outliers in 2009 and 9 were no longer high outliers. There were no hospitals that went from low to high outlier status, or vice versa, between 2008 and 2009. There was fair agreement beyond chance between the control chart and outlier status methods on whether a hospital had a change in performance between 2008 and 2009 (weighted ␬ ⫽ 0.379) (Table 4). There were no hospitals labeled as improving by one method and worsening by the other. Correlation between the control chart and outlier status methods for determining worsening versus not-worsening performance in 2009 compared with that of 2008 was medium (correlation coefficient 0.41, P ⬍ 0.001). Control charts identified worsening performance for 61.5% of hospitals that had a worsened outlier status (sensitivity) and identified nonworsening performance for 86.6% of hospitals that had a nonworsened outlier status (specificity). In other words, control charts failed to identify 38.5% of the hospitals with worsened outlier status and 13.4% of hospitals with outlier status that did not worsen. The positive predictive value of a control chart indicator of worsening performance was 42.1% and the negative predictive value was 93.4%. In other words, 57.9% of hospitals with a control chart indicator of worsening performance did not have a change in outlier status between 2008 and 2009, and 6.6% of hospitals without a control chart indicator of worsening performance did have a worse outlier status in 2009.

E.H. Lawson, J. Lin, and C.Y. Ko

150 Table 3 Characteristics of Study Population, By Year Patient Characteristic Age, years (SD) Male, % Body mass index (kg/m2), % Underweight (<18.5) Normal (18.5-24.9) Overweight (25-29.9) Class I obesity (30-34.9) Class II obesity (35-39.9) Class III obesity (>40) Unknown Functional status, % Independent Partially dependent Totally dependent Smoker, % Diabetes, % None Oral medication Insulin dependent ASA class, % I II III IV V Unknown Wound class, % II—clean/contaminated III—contaminated IV—dirty/infected Emergency case, % Number of comorbidities, n (SD) Mortality or serious morbidity, %

2008 (12,348 Patients)

2009 (13,163 Patients)

62.7 (15.7) 47.8

62.8 (15.4) 46.2

3.2 30 33 18 7.9 5.4 2.6

3.2 30.6 33.4 19.7 7.7 5.3 0

89 7.4 3.7 19

88.4 7.6 3.9 19

85.7 9.8 4.6

84.6 10.4 5.1

3.4 45.5 41 9.3 0.8 0.04

2.8 44.9 41.3 10 0.9 0.1

71.9 13 15.1 17.5 1.6 (1.5) 22

72.4 12.2 15.4 17.2 2.0 (1.9) 22.4

P Value 0.773 0.011 <0.001

0.311

0.998 0.032

0.029

0.19

0.58 <0.001 0.487

Study sample derived from 95 hospitals that participated in ACS-NSQIP in 2008 and 2009. Numbers do not add to 100 for some variables because of rounding error. SD, standard deviation; ASA, American Society of Anesthesiologists.

Discussion This study demonstrates that control charts using raw outcome data and outlier status for risk-adjusted outcomes are complementary quality measurement tools. Because of the unique strengths and weaknesses of each method, hospitals interested in quality assessment and improvement would likely benefit from using both methods in conjunction. We found that the control charts and outlier status methods never contradicted each other for the 95 hospitals studied (ie, no instance where a hospital is labeled as improving by one method and worsening by the other), and control charts were able to detect worsening performance for a majority of hospitals with outlier status that worsened. However, agreement beyond chance on whether a hospital had a change in performance between 2008 and 2009 was fair. A key aspect of both methods used herein is that they used the same data source, ACS-NSQIP, which is a national surgical clinical registry of patient risk factors and 30-day postoperative outcomes. Use of robust data that are collected in a

systematic and rigorous manner is a key component of meaningful quality measurement, regardless of the methodology used for assessment. ACS-NSQIP ensures the reliability of data collected by participating hospitals through auditing and by requiring data abstractors to be trained and tested on strict variable definitions, inclusion/exclusion criteria, and data collection methods. In addition, ACS-NSQIP requires a rigorous 30-day follow-up for postoperative outcomes, as many complications are known to occur after discharge from the primary hospitalization.10 One drawback to data registries such as ACS-NSQIP is that clinical data can be burdensome for surgeons and hospitals to collect. Administrative data, which are collected for the purpose of submitting claims for payment, can be used for quality measurement using the methods described herein without an added data collection burden. However, the validity and reliability of using this data source for surgical quality measurement has been questioned. Thus, we believe the added burden of clinical data collection is outweighed by the superior ability of clinical

Use of control charts for identifying worsening postoperative M&M performance after colectomy

151

Table 4 Comparison of 2 Methods for Monitoring for a Change in Postoperative Mortality or Serious Morbidity (M&M) Performance for Colectomy Procedures in 95 Hospitals Number of Hospitals With a Change in ACSNSQIP Outlier Status for Risk-Adjusted M&M Between 2008 and 2009 Number of hospitals with a control chart indicator of change in monthly M&M rate in 2009, using 2008 as a baseline Indicator of worsening performance No indicator of change in performance Indicator of improving performance

Worse

No Change

Improved

8 5 0

11 45 9

0 9 8

␹2 ⴝ 27.9, P < 0.0001; Weighted ␬ ⴝ 0.379 (95% CI, 0.211-0.546). ACS-NSQIP, American College of Surgeons National Surgical Quality Improvement Program.

data to properly and consistently identify high- and lowquality hospitals.11-13 Despite use of the same data source, one would not expect perfect agreement between control charts and outlier status determinations because they are different methodologies for measuring quality. The major strength of ACS-NSQIP outlier status determinations is that it allows hospitals to benchmark their performance with that of their peers in a manner that accounts for differences in a hospital’s procedure-mix and case-mix of patients. However, outlier status for risk-adjusted postoperative outcomes is reported back to hospitals only twice a year, and there is a 6-month lag from the time when the last case included in the report was performed and issuance of the report. For example, the semiannual report issued in July 2009 used all cases entered for the year 2008. In contrast, control charts allow quality measurement to be more timely and locally controlled. Hospitals have ready access to their own data as soon as it is collected and can use it to construct control charts using the ACS-NSQIP Web site or readily available software, such as Minitab or the Excel QI macro. Although this method does not allow for external comparison of performance with other hospitals, it can serve as a powerful internal management tool for monitoring surgical processes of care or postoperative outcomes. Control charts can also be used to better understand why a hospital had improving or worsening performance, and for determining whether a process or outcome is in control (ie, do monthly rates vary widely or cluster closely around the overall average line). Such information can help direct root cause analyses to identify the etiology of improving or worsening performance, and can also be informative for the development of interventions to improve care. In this study, we constructed p-charts, which are used to evaluate variation in the rates of categorical variables during several periods. The p-charts could similarly be used for monitoring a wide range of surgical processes and postoperative outcomes. Other types of control charts that may be useful in the health care setting include the u-chart and X-MR (X ⫽ observation, MR ⫽ moving range) chart. The u-chart is similar to the p-chart and is used when there can be ⬎1 type of error per patient. For example, a patient could have ⬎1 medication error during the course of a hospital stay or could

have ⬎1 postoperative complication. An X-MR chart is used when there is a single observation per period, and outcomes are measured on an interval scale (for example, operative time or cost).4,9 Some have advocated for the use of riskadjusted control charts for assessing health care quality, as the aforementioned methods use raw data that do not account for patient disease severity, which may vary during different periods.14-16 Quality improvement methodologies such as Lean six sigma use the previously described control charts within the framework of a broader system of tools for identifying problems, measuring their impact, discovering their root causes, developing specific and targeted interventions, testing the effect of interventions, and ensuring the sustainability of improvements to quality, safety, and efficiency. These tools include change management strategies for engaging and managing stakeholders, which is an essential component of any quality improvement initiative. The results of our study should be interpreted in light of certain limitations. First, this study only included ACSNSQIP hospitals. Although ACS-NSQIP is a nationwide program, most hospitals are still large academic medical centers, although this is changing over time. Thus, our results may not be generalizable to all smaller institutions. Second, we considered ACS-NSQIP outlier status as the gold standard. Although the statistical methods used to identify outliers are transparent and logical, it is certainly possible that some hospitals were incorrectly labeled as high or low outliers, or that hospitals with worse or better performance were incorrectly labeled as having “as expected” performance. Third, this is a retrospective observational study on a prospectively gathered clinical database. Although ACS-NSQIP uses audits and strict variable definitions and methods of data collection, we have no way of ensuring the accuracy of all the data. In addition, we are unable to ascertain whether hospitals included in the study sample were actively engaging in quality improvement initiatives focused on colectomy or if they had reasons for external variation in performance, which could affect the way control charts are constructed and interpreted. In conclusion, we found that control charts of monthly raw rates of M&M after colectomy and ACS-NSQIP outlier status for risk-adjusted M&M produced results that were significantly, but not perfectly, correlated. Although we did not

152 demonstrate perfect agreement, our results support the assertion that these methods are measuring similar constructs of quality and are likely complementary uses of the same clinical data source.

References 1. Thor J, Lundberg J, Ask J, et al: Application of statistical process control in healthcare improvement: Systematic review. Qual Saf Health Care 16:387-399, 2007 2. Lee K, McGreevey C: Using control charts to assess performance measurement data. Jt Comm J Qual Improv 28:90-101, 2002 3. Duclos A, Voirin N: The p-control chart: A tool for care improvement. Int J Qual Health Care 22:402-407, 2010 4. Arthur J: Statistical process control for health care. Qual Dig 28:3438, 2008 5. Schilling PL, Dimick JB, Birkmeyer JD: Prioritizing quality improvement in general surgery. J Am Coll Surg 207:698-704, 2008 6. Shiloach M, Frencher SK Jr, Steeger JE, et al: Toward robust information: Data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program. J Am Coll Surg 210:6-16, 2010 7. Khuri SF: The NSQIP: A new frontier in surgery. Surgery 138:837843, 2005

E.H. Lawson, J. Lin, and C.Y. Ko 8. Ingraham AM, Richards KE, Hall BL, et al: Quality improvement in surgery: The American College of Surgeons National Surgical Quality Improvement Program approach. Adv Surg 44:251-267, 2010 9. Lighter D, Fair D: Statitistical process control approaches: Basic theory and use of control charts, in Quality Management in Health Care: Principles and Methods. Sudbury, MA, Jones and Bartlett (ed 2), 2004 10. Bilimoria KY, Cohen ME, Ingraham AM, et al: Effect of postdischarge morbidity and mortality on comparisons of hospital surgical quality. Ann Surg 252:183-190, 2010 11. Steinberg SM, Popa MR, Michalek JA, et al: Comparison of risk adjustment methodologies in surgical quality improvement. Surgery 144: 662-667, 2008 12. Jollis JG, Ancukiewicz M, DeLong ER, et al: Discordance of databases designed for claims payment versus clinical information systems. Implications for outcomes research. Ann Intern Med 119:844-850, 1993 13. Hannan EL, Kilburn H Jr, Lindsey ML, et al: Clinical versus administrative data bases for CABG surgery. Does it matter? Med Care 30:892907, 1992 14. Gombay E, Hussein AA, Steiner SH: Monitoring binary outcomes using risk-adjusted charts: A comparative study. Stat Med 30:28152826, 2011 15. Rogers CA, Ganesh JS, Banner NR, et al: Cumulative risk adjusted monitoring of 30-day mortality after cardiothoracic transplantation: UK experience. Eur J Cardiothorac Surg 27:1022-1029, 2005 16. Alemi F, Rom W, Eisenstein E: Risk-adjusted control charts for health care assessment. Ann Oper Res 67:45-60, 1996