- Email: [email protected]
The Need for “Compassionate Provider Profiling”
Journal of the American College of Cardiology © 2011 by the American College of Cardiology Foundation Published by Elsevier Inc.
EDITORIAL COMMENT
The Need for “Compassionate Provider Profiling” Refining Risk Assessment for Percutaneous Coronary Intervention* Eric D. Peterson, MD, MPH Durham, North Carolina
In today’s “transparent” society, the public is increasingly demanding more information on the quality and outcomes of medical care (1,2). Most would agree that patients need more data regarding their health care providers’ results, yet the devil is always in the details. For outcome comparisons to be fair and valid, they first need to be adjusted for underlying patient risks (3). For percutaneous coronary intervention (PCI), outcome metrics (e.g., acute mortality) must account for multiple potential confounding factors including patient demographics, disease severity, comorbid illness, and, most importantly, procedural acuity. See page 904
The science of outcomes risk assessment needs to continually evolve to keep up with an ever-changing medical field and care environment (4). Historically, hospital comparisons were based on claims (or billing) data. Although claims data were ubiquitous, they often proved inaccurate and lacked many critical factors needed for proper risk adjustment (5). During the past 2 decades, multicenter clinical registries evolved and spread, helping to accelerate our understanding of PCI procedural risk. In 1998, a group of 8 pioneer PCI databases were used to identify key variables associated with PCI risk (6). Since then, the majority of U.S. PCI hospitals have consolidated data submission into a single National Cardiovascular Data Registry (NCDR). The NCDR expanded the number of data elements collected, standardized their definitions, and improved their completeness and accuracy (7). The resulting
*Editorials published in the Journal of the American College of Cardiology reflect the views of the authors and do not necessarily represent the views of JACC or the American College of Cardiology. From the Duke Clinical Research Institute, Duke University Medical Center, Durham, North Carolina. Dr. Peterson has received research grants from the American College of Cardiology, Bristol-Myers Squibb Company, Eli Lilly & Company, Johnson & Johnson, Merck & Co., Sanofi-Aventis, Society of Thoracic Surgeons, and the American Heart Association; and is a consultant to Bristol-Myers Squibb, Merck & Co., Tethysbio, and AstraZeneca.
Vol. 57, No. 8, 2011 ISSN 0735-1097/$36.00 doi:10.1016/j.jacc.2010.10.022
larger, higher quality database has facilitated the development of more precise PCI risk models. The current NCDR risk model adjusts for ⬎21 independent factors and accurately identifies those who will survive a PCI hospitalization and those who will die in 93% of cases (c-index 0.93) (8). In this issue of the Journal, Resnic et al. (9) assess whether PCI risk algorithms could be further improved by refining the variables used to summarize procedural acuity. Specifically, a previous case review had demonstrated that most PCI deaths occurred among patients entering the catheterization lab in extremis (10). Although the NCDR data capture many indicators of disease acuity (e.g., ST-segment elevation myocardial infarction, heart failure, shock), some of these life-threatening factors were missed. To address this gap, Resnic et al. (9) studied the incremental impact of 3 novel risk data elements on predicting PCI mortality: 1) cardiopulmonary resuscitation; 2) patients in a coma; or 3) those with a ventricular assist device or extracorporeal bypass device. These 3 factors were combined into a single composite term entitled compassionate use (CU) PCI. Resnic et al. (9) found that while PCI was rarely performed for CU indications (0.03% of all PCI cases), these CU cases did indeed face very high mortality risks, with in-hospital mortality rates approaching 70%. They also found that the CU indicator added independent prognostic information beyond traditional clinical risk factors within Massachusetts’ state PCI database. Measuring how these data affect patient risk stratification, hospital performance assessment, and resource requirements deserves further discussion. From a statistical standpoint, the gains in risk prediction provided by the inclusion of CU in the PCI risk model were modest. Overall model discrimination (c-index) changed little (0.03), and ⬍10% of patients were reclassified correctly to a higher (or lower) risk group when CU was added (9). Beyond patient-level risk prediction, one also needs to consider how inclusion of the CU variable affects hospital-level risk adjustment. With regards to performance assessment, Resnic et al. (9) found individual hospital risk-adjusted mortality rates were fairly static before and after adding the CU indicator. In fact, in its first year of implementation, consideration of CU had no effect on the classification of centers rated as superior or inferior “outliers.” These findings were not surprising because centers (on average) treated ⱕ3 CU cases per year (9). Thus, while the risk models were incrementally improved with inclusion of CU, these changes had limited measurable impact on aggregated provider-level conclusions. Additionally, one should consider the costs associated with CU data implementation. Human resource costs include the need to educate sites regarding the definition of CU, the personal time required for CU data collection, and the subsequent central adjudication effort needed to validate each reported CU case. For example, Resnic et al. (9) found that in Massachusetts, despite state-wide training, up to one-third of investigator-reported CU procedures were later reversed after external audit. This stringent audit system
Peterson Compassionate Provider Profiling
JACC Vol. 57, No. 8, 2011 February 22, 2011:912–3
used in the state further contributed to delays in release of the outcomes information to the public. As a result, the Massachusetts PCI mortality model reports were released ⬎2 years after the last procedures were performed (11). Despite these potential statistical and operational challenges, the work of Resnic et al. (9) points out an extremely important consequence of provider profiling programs; namely, how provider perceptions of the PCI risk adjustment process itself can affect their subsequent clinical actions. Faced with fears of appearing on the front page of their local newspapers, interventionalists in Massachusetts may have thought twice before attempting PCI on very high-risk patients (12). In fact, Resnic et al. (9) found empirical support for this hypothesis. In the first 3 years after Massachusetts’ decision to publicly report PCI mortality, the use of coronary revascularization procedures in shock patients decreased by 50%. However, after the Massachusetts Board began adjusting for CU factors, this trend reversed, and interventionalists reverted to their preprofiling procedure levels. Although such trends will need ongoing evaluation, these initial findings are provocative indicators of the influence of profiling methodology on provider case selection. Can the Massachusetts CU experience be generalized to the NCDR PCI database? Perhaps. Currently, the NCDR remains a voluntary registry whose goal is to provide clinicians with confidential feedback on their comparative care and outcomes. In an era without the pressures of public reporting, hospitals have limited incentive to up-code or “game” their reported PCI risk data. Looking toward the future, however, the world may be radically different. Beginning last fall, the Society of Thoracic Surgeons has agreed to publicly release hospitals’ bypass surgery outcomes and care quality information to Consumer Reports (13). This decision will put pressure on the NCDR and other clinical registries to also release their data to the public. Although such increased transparency is welcomed by policy leaders who have advocated for these types of changes (1,2), it will also likely (at least temporarily) increase the anxiety of practicing clinicians. Thus, if and when professional society registries are also used as public report cards, then the requirements for data accuracy and verification will be just as high as those currently faced by Massachusetts. Therefore, the study by Resnic et al. (9) has helped our profession prepare in multiple ways for this near-term future. First, Resnic et al. (9) have identified a rare but important set of high-risk factors not previously collected in national registries. Second, the investigators have shown that these data variables were potentially feasible to collect, providing that an external validation is in place to ensure accuracy. Third, and perhaps most important, they demonstrated that the extra data collection and validation may be considered worthwhile, given the positive impact of this process on provider behaviors. Many have and will debate the relative merits and unintended consequences of releasing outcomes data to the public (14). Nevertheless, if such provider profiling is to be implemented, I strongly believe that this profiling must be used compassionately, in a manner that demonstrates concern for
913
accuracy and fairness to the physicians affected by its critique. As a practicing coronary care unit physician, I rely on the skills and judgment of my interventionalist partners. When I call on my colleagues to consider taking an extremely sick patient to the laboratory, I do not want them worrying about how a case may adversely affect their outcomes score card; rather, I want them to compassionately consider what can be done to save the patient’s life. Reprint requests and correspondence: Dr. Eric D. Peterson, Duke Clinical Research Institute, 2400 Pratt Street, Durham, North Carolina 27705. E-mail: [email protected]. REFERENCES
1. Swensen SJ, Meyer GS, Nelson EC, et al. Cottage industry to postindustrial care—the revolution in health care delivery. N Engl J Med 2010;362:e12. 2. Chassin MR, Loeb JM, Schmaltz SP, Wachter RM. Accountability measures— using measurement to promote quality improvement. N Engl J Med 2010;363:683– 8. 3. Krumholz HM, Brindis RG, Brush JE, et al. Standards for statistical models used for public reporting of health outcomes: an American Heart Association Scientific Statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group: cosponsored by the Council on Epidemiology and Prevention and the Stroke Council. Endorsed by the American College of Cardiology Foundation. Circulation 2006;113:456 – 62. 4. Topol E, Califf R. Scorecard cardiovascular medicine. Its impact and future directions. Ann Intern Med 1994;120:65–70. 5. Jollis JG, Ancukiewicz M, DeLong ER, Pryor DB, Muhlbaier LH, Mark DB. Discordance of databases designed for claims payment versus clinical information systems. Implications for outcomes research. Ann Intern Med 1993;119:844 –50. 6. Block PC, Peterson ED, Krone R, et al. Identification of variables needed to risk adjust outcomes of coronary interventions: evidence-based guidelines for efficient data collection. J Am Coll Cardiol 1998;32:275–82. 7. Brindis RG, Fitzgerald S, Anderson HV, Shaw RE, Weintraub WS, Williams JF. The American College of Cardiology–National Cardiovascular Data Registry (ACC-NCDR): building a national clinical data repository. J Am Coll Cardiol 2001;37:2240 –5. 8. Peterson ED, Dai D, DeLong ER, et al. Contemporary mortality risk prediction for percutaneous coronary intervention: results from 588,398 procedures in the National Cardiovascular Data Registry. J Am Coll Cardiol 2010;55:1923–32. 9. Resnic FS, Normand SLT, Piemonte TC, et al. Improvement in mortality risk prediction after percutaneous coronary intervention through the addition of a “compassionate use” variable to the National Cardiovascular Data Registry CathPCI dataset: a study from the Massachusetts Angioplasty Registry. J Am Coll Cardiol 2011;57:904–11. 10. Resnic FS, Welt FG. The public health hazards of risk avoidance associated with public reporting of risk-adjusted outcomes in coronary intervention. J Am Coll Cardiol 2009;53:825–30. 11. Massachusetts Data Analysis Center. Adult percutaneous coronary intervention in the Commonwealth of Massachusetts: fiscal year 2008 report (October 1, 2007–September 30, 2008): hospital riskstandardized in-hospital mortality rates. Available at: http:// www.massdac.org/sites/default/files/reports/PCI%20FY2008.pdf. Accessed October 4, 2010. 12. Narins CR, Dozier AM, Ling FS, Zareba W. The influence of public reporting of outcome data on medical decision making by physicians. Arch Intern Med 2005;165:83–7. 13. Ferris TG, Torchiana DF. Public release of clinical outcomes data— online CABG report cards. N Engl J Med 2010;363:1593–5. 14. Califf RM, Peterson ED. Public reporting of quality measures: what are we trying to accomplish? J Am Coll Cardiol 2009;53:831–3. Key Words: American College of Cardiology National Cardiovascular Data Registry CathPCI Registry y hierarchical risk prediction models y percutaneous coronary intervention y predictive models.
EDITORIAL COMMENT
The Need for “Compassionate Provider Profiling” Refining Risk Assessment for Percutaneous Coronary Intervention* Eric D. Peterson, MD, MPH Durham, North Carolina
In today’s “transparent” society, the public is increasingly demanding more information on the quality and outcomes of medical care (1,2). Most would agree that patients need more data regarding their health care providers’ results, yet the devil is always in the details. For outcome comparisons to be fair and valid, they first need to be adjusted for underlying patient risks (3). For percutaneous coronary intervention (PCI), outcome metrics (e.g., acute mortality) must account for multiple potential confounding factors including patient demographics, disease severity, comorbid illness, and, most importantly, procedural acuity. See page 904
The science of outcomes risk assessment needs to continually evolve to keep up with an ever-changing medical field and care environment (4). Historically, hospital comparisons were based on claims (or billing) data. Although claims data were ubiquitous, they often proved inaccurate and lacked many critical factors needed for proper risk adjustment (5). During the past 2 decades, multicenter clinical registries evolved and spread, helping to accelerate our understanding of PCI procedural risk. In 1998, a group of 8 pioneer PCI databases were used to identify key variables associated with PCI risk (6). Since then, the majority of U.S. PCI hospitals have consolidated data submission into a single National Cardiovascular Data Registry (NCDR). The NCDR expanded the number of data elements collected, standardized their definitions, and improved their completeness and accuracy (7). The resulting
*Editorials published in the Journal of the American College of Cardiology reflect the views of the authors and do not necessarily represent the views of JACC or the American College of Cardiology. From the Duke Clinical Research Institute, Duke University Medical Center, Durham, North Carolina. Dr. Peterson has received research grants from the American College of Cardiology, Bristol-Myers Squibb Company, Eli Lilly & Company, Johnson & Johnson, Merck & Co., Sanofi-Aventis, Society of Thoracic Surgeons, and the American Heart Association; and is a consultant to Bristol-Myers Squibb, Merck & Co., Tethysbio, and AstraZeneca.
Vol. 57, No. 8, 2011 ISSN 0735-1097/$36.00 doi:10.1016/j.jacc.2010.10.022
larger, higher quality database has facilitated the development of more precise PCI risk models. The current NCDR risk model adjusts for ⬎21 independent factors and accurately identifies those who will survive a PCI hospitalization and those who will die in 93% of cases (c-index 0.93) (8). In this issue of the Journal, Resnic et al. (9) assess whether PCI risk algorithms could be further improved by refining the variables used to summarize procedural acuity. Specifically, a previous case review had demonstrated that most PCI deaths occurred among patients entering the catheterization lab in extremis (10). Although the NCDR data capture many indicators of disease acuity (e.g., ST-segment elevation myocardial infarction, heart failure, shock), some of these life-threatening factors were missed. To address this gap, Resnic et al. (9) studied the incremental impact of 3 novel risk data elements on predicting PCI mortality: 1) cardiopulmonary resuscitation; 2) patients in a coma; or 3) those with a ventricular assist device or extracorporeal bypass device. These 3 factors were combined into a single composite term entitled compassionate use (CU) PCI. Resnic et al. (9) found that while PCI was rarely performed for CU indications (0.03% of all PCI cases), these CU cases did indeed face very high mortality risks, with in-hospital mortality rates approaching 70%. They also found that the CU indicator added independent prognostic information beyond traditional clinical risk factors within Massachusetts’ state PCI database. Measuring how these data affect patient risk stratification, hospital performance assessment, and resource requirements deserves further discussion. From a statistical standpoint, the gains in risk prediction provided by the inclusion of CU in the PCI risk model were modest. Overall model discrimination (c-index) changed little (0.03), and ⬍10% of patients were reclassified correctly to a higher (or lower) risk group when CU was added (9). Beyond patient-level risk prediction, one also needs to consider how inclusion of the CU variable affects hospital-level risk adjustment. With regards to performance assessment, Resnic et al. (9) found individual hospital risk-adjusted mortality rates were fairly static before and after adding the CU indicator. In fact, in its first year of implementation, consideration of CU had no effect on the classification of centers rated as superior or inferior “outliers.” These findings were not surprising because centers (on average) treated ⱕ3 CU cases per year (9). Thus, while the risk models were incrementally improved with inclusion of CU, these changes had limited measurable impact on aggregated provider-level conclusions. Additionally, one should consider the costs associated with CU data implementation. Human resource costs include the need to educate sites regarding the definition of CU, the personal time required for CU data collection, and the subsequent central adjudication effort needed to validate each reported CU case. For example, Resnic et al. (9) found that in Massachusetts, despite state-wide training, up to one-third of investigator-reported CU procedures were later reversed after external audit. This stringent audit system
Peterson Compassionate Provider Profiling
JACC Vol. 57, No. 8, 2011 February 22, 2011:912–3
used in the state further contributed to delays in release of the outcomes information to the public. As a result, the Massachusetts PCI mortality model reports were released ⬎2 years after the last procedures were performed (11). Despite these potential statistical and operational challenges, the work of Resnic et al. (9) points out an extremely important consequence of provider profiling programs; namely, how provider perceptions of the PCI risk adjustment process itself can affect their subsequent clinical actions. Faced with fears of appearing on the front page of their local newspapers, interventionalists in Massachusetts may have thought twice before attempting PCI on very high-risk patients (12). In fact, Resnic et al. (9) found empirical support for this hypothesis. In the first 3 years after Massachusetts’ decision to publicly report PCI mortality, the use of coronary revascularization procedures in shock patients decreased by 50%. However, after the Massachusetts Board began adjusting for CU factors, this trend reversed, and interventionalists reverted to their preprofiling procedure levels. Although such trends will need ongoing evaluation, these initial findings are provocative indicators of the influence of profiling methodology on provider case selection. Can the Massachusetts CU experience be generalized to the NCDR PCI database? Perhaps. Currently, the NCDR remains a voluntary registry whose goal is to provide clinicians with confidential feedback on their comparative care and outcomes. In an era without the pressures of public reporting, hospitals have limited incentive to up-code or “game” their reported PCI risk data. Looking toward the future, however, the world may be radically different. Beginning last fall, the Society of Thoracic Surgeons has agreed to publicly release hospitals’ bypass surgery outcomes and care quality information to Consumer Reports (13). This decision will put pressure on the NCDR and other clinical registries to also release their data to the public. Although such increased transparency is welcomed by policy leaders who have advocated for these types of changes (1,2), it will also likely (at least temporarily) increase the anxiety of practicing clinicians. Thus, if and when professional society registries are also used as public report cards, then the requirements for data accuracy and verification will be just as high as those currently faced by Massachusetts. Therefore, the study by Resnic et al. (9) has helped our profession prepare in multiple ways for this near-term future. First, Resnic et al. (9) have identified a rare but important set of high-risk factors not previously collected in national registries. Second, the investigators have shown that these data variables were potentially feasible to collect, providing that an external validation is in place to ensure accuracy. Third, and perhaps most important, they demonstrated that the extra data collection and validation may be considered worthwhile, given the positive impact of this process on provider behaviors. Many have and will debate the relative merits and unintended consequences of releasing outcomes data to the public (14). Nevertheless, if such provider profiling is to be implemented, I strongly believe that this profiling must be used compassionately, in a manner that demonstrates concern for
913
accuracy and fairness to the physicians affected by its critique. As a practicing coronary care unit physician, I rely on the skills and judgment of my interventionalist partners. When I call on my colleagues to consider taking an extremely sick patient to the laboratory, I do not want them worrying about how a case may adversely affect their outcomes score card; rather, I want them to compassionately consider what can be done to save the patient’s life. Reprint requests and correspondence: Dr. Eric D. Peterson, Duke Clinical Research Institute, 2400 Pratt Street, Durham, North Carolina 27705. E-mail: [email protected]. REFERENCES
1. Swensen SJ, Meyer GS, Nelson EC, et al. Cottage industry to postindustrial care—the revolution in health care delivery. N Engl J Med 2010;362:e12. 2. Chassin MR, Loeb JM, Schmaltz SP, Wachter RM. Accountability measures— using measurement to promote quality improvement. N Engl J Med 2010;363:683– 8. 3. Krumholz HM, Brindis RG, Brush JE, et al. Standards for statistical models used for public reporting of health outcomes: an American Heart Association Scientific Statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group: cosponsored by the Council on Epidemiology and Prevention and the Stroke Council. Endorsed by the American College of Cardiology Foundation. Circulation 2006;113:456 – 62. 4. Topol E, Califf R. Scorecard cardiovascular medicine. Its impact and future directions. Ann Intern Med 1994;120:65–70. 5. Jollis JG, Ancukiewicz M, DeLong ER, Pryor DB, Muhlbaier LH, Mark DB. Discordance of databases designed for claims payment versus clinical information systems. Implications for outcomes research. Ann Intern Med 1993;119:844 –50. 6. Block PC, Peterson ED, Krone R, et al. Identification of variables needed to risk adjust outcomes of coronary interventions: evidence-based guidelines for efficient data collection. J Am Coll Cardiol 1998;32:275–82. 7. Brindis RG, Fitzgerald S, Anderson HV, Shaw RE, Weintraub WS, Williams JF. The American College of Cardiology–National Cardiovascular Data Registry (ACC-NCDR): building a national clinical data repository. J Am Coll Cardiol 2001;37:2240 –5. 8. Peterson ED, Dai D, DeLong ER, et al. Contemporary mortality risk prediction for percutaneous coronary intervention: results from 588,398 procedures in the National Cardiovascular Data Registry. J Am Coll Cardiol 2010;55:1923–32. 9. Resnic FS, Normand SLT, Piemonte TC, et al. Improvement in mortality risk prediction after percutaneous coronary intervention through the addition of a “compassionate use” variable to the National Cardiovascular Data Registry CathPCI dataset: a study from the Massachusetts Angioplasty Registry. J Am Coll Cardiol 2011;57:904–11. 10. Resnic FS, Welt FG. The public health hazards of risk avoidance associated with public reporting of risk-adjusted outcomes in coronary intervention. J Am Coll Cardiol 2009;53:825–30. 11. Massachusetts Data Analysis Center. Adult percutaneous coronary intervention in the Commonwealth of Massachusetts: fiscal year 2008 report (October 1, 2007–September 30, 2008): hospital riskstandardized in-hospital mortality rates. Available at: http:// www.massdac.org/sites/default/files/reports/PCI%20FY2008.pdf. Accessed October 4, 2010. 12. Narins CR, Dozier AM, Ling FS, Zareba W. The influence of public reporting of outcome data on medical decision making by physicians. Arch Intern Med 2005;165:83–7. 13. Ferris TG, Torchiana DF. Public release of clinical outcomes data— online CABG report cards. N Engl J Med 2010;363:1593–5. 14. Califf RM, Peterson ED. Public reporting of quality measures: what are we trying to accomplish? J Am Coll Cardiol 2009;53:831–3. Key Words: American College of Cardiology National Cardiovascular Data Registry CathPCI Registry y hierarchical risk prediction models y percutaneous coronary intervention y predictive models.