Performance Measurement Through Audit, Feedback, and Profiling as Tools for Improving Clinical Care

Performance Measurement Through Audit, Feedback, and Profiling as Tools for Improving Clinical Care* Kevin B. Weiss, MD; and Robin Wagner, RN, MHSA

Clinical audits and practice profiling have become popular tools in the attempt to change physician behavior to improve quality of care. Unfortunately, the growing need for information on quality of care has often outpaced the development of standard, valid, and reliable approaches to using these tools. The studies of performance measurement published in the literature to date demonstrate varying impact on ability to improve clinical care; few are randomized controlled trials. While performance measurement has become a common practice, the science surrounding this field is still in its early stages of development; while it seems promising, it should be viewed as largely experimental. (CHEST 2000; 118:53S–58S) Key words: Health Plan Employer Data Information Set; quality of care Abbreviation: HEDIS ⫽ Health Plan Employer Data Information Set

the seminal work of Wennberg and GittelS ince sohn in the 1970s, numerous studies have sought 1

to understand the reasons for large geographic variations in the use of clinical services, as well as variations in patient outcomes.2 For much of the 1980s and early 1990s, these studies have served to indicate that there might be important differences in the quality of medical care as a result of overuse, underuse, and/or misuse of medical interventions.3, 4 In response, the health-care system has begun to explore ways to use this information to influence changes in provider behavior to improve care. Use of clinical data to improve outcomes is not a new concept. For decades, hospitals have routinely held surgical morbidity and mortality reviews as a means by which to learn from experience. More recently, the focus has shifted to clinical data review across multiple settings, and among differing types of provider groups. This report presents an overview of the use of clinical audits/practice feedback, practice profiling/benchmarking, and regulatory oversight as tools for changing physician behavior to improve outcomes. Before exploring these different tools, it may be useful to review a couple of the basic concepts of performance measurement. *From the Center for Health Services Research, Rush Primary Care Institute, Rush-Presbyterian-St. Luke’s Medical Center, Chicago, IL. Correspondence to: Kevin B. Weiss, MD, Center for Health Services Research, Rush Primary Care Institute, Rush-Presbyterian-St. Luke’s Medical Center, 1653 West Congress Parkway, Chicago, IL 60612

Individual vs Population Performance Measurement Individual case review provides different data from population-based reviews, and both types of review have their strengths and weaknesses. Individual case review is principally used to explore concerns that are associated with rare, but sentinel, events. An example would be the review of an incident in which a patient received mismatched blood products. This type of review is best suited for events that are infrequent but important enough to warrant the use of resources so as to minimize the chances of a similar error in the future. Yet, one may or may not be able to generalize from the knowledge gained from such a review, and the infrequent nature of such events makes it infeasible to examine multiple similar events. There is also little opportunity to use epidemiologic and statistical tools to assist in judging the degree of certainty of the findings from individual case reporting. The alternative to individual case review is the population-based approach. Aggregated experience from multiple cases can provide insights to patterns of clinical behavior for more common conditions that affect many more patients. An example would be a measure of the proportion of patients within a particular health plan who received the flu vaccine. With population-based assessment, it is possible to use standard epidemiologic and statistical techniques to help assess the degree of certainty of the conclusions drawn from the observed clinical experiences. CHEST / 118 / 2 / AUGUST, 2000 SUPPLEMENT

53S

Quality as Measured by Structure, Process, and Outcomes Donabedian5 offered the concept that quality could be measured based on structure, process, and outcomes. Structure encompasses physical factors, such as buildings, as well as professional and institutional factors, such as the regulatory and financing environments in which health care is delivered. Process refers to the actions that health-care providers take in delivering medical care, such as performing examinations, ordering tests, and prescribing medications. Outcomes are the end result of the process interventions: the effects on the patient’s health and well being. While early attempts at measuring the quality of health care focused on the structure, much of the current focus relates to exploring clinical processes and outcomes. Although patient outcomes are the ultimate judge of the quality of health care, there are several advantages to using process measures instead of outcome measures for purposes of performance evaluation. Most notably, it is much easier for physicians or other health-care providers to accept responsibility for their actions in providing care than to accept responsibility for their patients’ outcomes, because there are many factors affecting patient outcomes that are not directly under the control of providers. For example, while a provider might make a concerted effort to ensure that a patient has been offered the flu vaccine, the patient may choose not to take the vaccine and may subsequently develop influenza. In this situation, performance evaluation will produce very different results depending on whether it is the process (providing access to the vaccine) or the outcome (influenza) that is measured. Process measures are also useful in evaluating the quality of care for common chronic conditions for which the ultimate outcomes may take years to determine, such as hypertension and stroke, or glycemic control and complications from diabetes. For these reasons, it is attractive to focus on using process measures rather than outcomes measures for performance measurement. However, it would seem that the best measure of health-care performance rests with patient outcomes, including physiologic status, health-related quality of life, and satisfaction with the health-care system. Formative vs Evaluative Information A third central concept for performance measurement relates to how the data are used. Formative data are gathered for immediate use, to guide clinical decisions affecting ongoing patient care.6 As such, this type of information is different from the kind 54S

used for evaluation. Although evaluative data may be collected at any time in the process of care, they are generally examined retrospectively in an attempt to evaluate good vs bad quality health care, overuse vs underuse of services, or perhaps to compare one type of service to another. Currently, there are several major types of performance measurement in use. These include clinical audits/practice feedback, practice profiling/benchmarking, and regulatory oversight of performance indicator systems.

Clinical Audits and Practice Feedback For decades, health-care systems have used clinical audits as a tool for quality assessment. Audits of this type usually seek to characterize care through the systematic review of a series of patient experiences. Most often, the information is obtained by examining charts or medical records for documentation of specific clinical practices/procedures. Since the 1970s, the British have used audits to examine issues of quality surrounding clinical management of minor acute problems or preventive health practices,7,8 chronic disease management (eg, diabetes,9 and asthma10), and the use of specialty consultations.11 While clinical audits are widely used to assess performance, there is conflicting evidence regarding whether or not they are effective in changing provider behavior. For example, a study at one hospital demonstrated significant improvements in preventive health processes that were audited vs other health-care processes that were not monitored.12 Two small studies, examining the quality of Papanicolaou smears, demonstrated that performance of both residents and faculty physicians substantively improved after they received feedback from clinical audits.13,14 By contrast, a study by Reilly and Patten15 demonstrated little change in targeted prescribing patterns for various clinical conditions as a result of audit and feedback. The Ambulatory Care Medical Audit Demonstration Project16 is the largest formal study of the use of audit information in the United States. The project was designed as a randomized controlled clinical trial of the use of quality-improvement techniques to improve clinical performance in areas of primary care. Although audit information was only one element in a multidimensional intervention, this study demonstrated that it is possible to improve the quality of care with feedback of audit information. Unfortunately, there has been no formal synthesis of studies on the use of audits to affect clinical performance. Many of the studies conducted to date were not well controlled and did not include a Translating Guidelines Into Practice

strategy for randomizing the physicians who were given feedback. Rather, most were preevaluation/ postevaluation designs, based on interventions conducted at a single site or with a small number of practices. Therefore, while clinical audit with feedback is an attractive approach to changing physician behavior, its efficacy is unclear. Practice Profiling/Benchmarking Another approach to performance measurement compares the performance of a single provider against that of a panel of similar providers. This type of measurement is often referred to as practice profiling or benchmarking.17 In practice profiling the performance of a single physician or a group is expressed as a rate, a measure of resource use during a defined period for the population served. A profile is created by comparing this rate to that of a community norm based on the practices of other physicians or on other standards such as guidelines.18 Figure 1 provides an example of the differences in the case mix-adjusted number of relative value units per hospital admission by physician specialty among physicians providing care to Medicare patients in Oregon vs Florida.19 As with audits, there is also little known about how profiling affects clinical performance. A meta-analysis of randomized trials of profiling revealed only 12 eligible trials; many of the studies under evaluation

had notable design flaws.20 The analysis found that while profiling had a statistically significant positive effect on utilization, the effect was of minimal clinical importance. Nevertheless, practice profiling is widely used and has attracted much attention and controversy.21–23 In a 1994 American Medical Association survey of practicing physicians, over half of the respondents reported that they were subject to clinical or economic practice profiling.24 Much of the controversy rests on the quality, validity, and reliability of the profiling data. For performance assessments to provide useful information, they must meet certain methodologic criteria. Among these is the need for well-defined, similar patient populations. It is important that practice data are adjusted for case mix severity and for other nonmedical care factors that are known to affect clinical performance, and that sufficient numbers of events are measured to ensure that differences are not due to chance alone.25,26 This last issue is particularly problematic. A study by Hofer et al27 examined the usefulness of physician profiling for patients with diabetes, one of the most prevalent conditions in clinical practice. The authors conducted a study of approximately 3,600 patients with type II diabetes, under the care of 232 different physicians. Yet, as Figure 2 illustrates, they were unable to reliably detect any true differences in care among the physicians. They observed that “ . . . a physician would need to have more than 100 patients

Figure 1. Case mix-adjusted number of relative value units (RVUs) per hospital admission in Florida and Oregon according to medical specialty. CHEST / 118 / 2 / AUGUST, 2000 SUPPLEMENT

55S

Figure 2. Comparison of visit rate profiles for 232 physicians in three health plans, showing the relative visit rate by physician (with 1.0 being an average profile) after adjustment for patient demographics and detailed case mix measures. The error bars represent a 1.4 SE confidence interval, so that overlapping confidence intervals suggest that the overall difference between two physician visit rates is not statistically significant (p ⬎ 0.05). In this graph, although the overall physician effect on visits is statistically significant, it is not possible to say that the physicians at the extremes are significantly different in their visit rates from any of the other physicians.

with diabetes in their panel for the profiles to have a reliability of 0.80 or better (while more than 90% of all primary care physicians in the health maintenance organization had fewer than 50 patients with diabetes).”27 Therefore, the studies of physician profiling as a tool for changing practice behavior present a very mixed picture. The randomized, controlled trial literature suggests that profiling can produce a modest, but statistically significant effect on improving physician behavior.28 However, more recent studies on the validity and reliability of this measurement technique have opened up new questions about its usefulness. Regulatory Oversight and Performance Indicator Systems With such increased interest in attempting to improve the quality of care through feedback of clinical data, it is perhaps no surprise that there have been efforts to create complex systems to evaluate clinical performance. The apparent premise behind such performance measurement systems is to use them as administrative tools, either voluntary or regulatory, to broadly measure quality-improvement activities. 56S

In the United States, one of the prototypes of these systems is the Health Plan Employer Data Information Set (HEDIS). The HEDIS was developed in the early 1990s by the National Committee on Quality Assurance, a not-for-profit organization committed to evaluating and reporting the quality of care delivered by managed care plans. Using standardized methodology, HEDIS data are gathered from several sources within each health plan, including administrative claims and encounter information, medical records, and survey information. The National Committee on Quality Assurance, which uses the information from HEDIS as part of its accreditation program, makes the results publicly available through a national database of HEDIS information and accreditation results.29 Employers and consumers alike can use this information about quality of care to make choices among health plans. Figure 3 is an example of HEDIS data comparing mammography rates for Medicare managed-care organizations in Orange County, CA.29 In addition to HEDIS, the United States has developed or proposed several other performance indicator systems.30 –33 Unlike HEDIS, the role and utility of these other systems have not yet fully evolved. Many of these newer systems do not clearly specify the primary audience for their information. Is Translating Guidelines Into Practice

term credibility. There are still many unanswered questions, such as the appropriate population size to study and the types of data adjustments (eg, case mix, severity, sociodemographic) that need to be applied in order to be able to make accurate comparisons. Also, the literature has yet to determine which clinical conditions or administrative issues benefit the most from these types of data collection and feedback, and which methods work best to produce positive changes in the delivery of care. Performance measurement appears to be most useful when it is used as a formative tool as part of a more complex set of quality-improvement activities.34 However, this field has yet to determine which types of quality-improvement efforts will lead to better care. Until such questions are adequately addressed, performance measurement should be considered to be still in the “experimental stage” in the challenge toward reducing unintended variations in the quality of health care.

Figure 3. Medicare compare graph showing mammography rates for Medicare managed-care organizations in Orange County, CA.

the primary audience the provider, the health plan, the employer, or the consumer? Is the primary focus on process measures or outcomes measures? What are the costs and burdens of collecting such complex and comprehensive data? Perhaps most importantly, what effect will such data have on changing the quality of health care in the future? What Is the Future for Using Performance Measurement To Evaluate Quality Improvement Activities? Although not always approached with rigorous research methods, health-care performance measurement is now pervasive, and it seems likely that these activities will continue, if not increase, in the future. On the positive side, these efforts have helped to focus attention on the overall importance of evaluating the quality of health care; for the public, they have removed some of the mystery surrounding the delivery of care. As performance measurement continues to evolve nationally, clearer standards will emerge to define the types of measures that are most appropriate for this field, and valid and reliable methods will emerge for the collection, analysis, and reporting of data. Alternatively, there are many limitations to this evolving practice of performance measurement that, if not adequately addressed, will undermine its long-

References 1 Wennberg J, Gittelsohn A. Small area variations in health care delivery. Science 1973; 182:1102–1108 2 Detsky AS. Regional variation in medical care. N Engl J Med 1995; 333:589 –590 3 Wennberg DE. Variation in the delivery of health care: the stakes are high. Ann Intern Med 1998; 128:866 – 868 4 Brook RH. Appropriate and necessary. Health Manage Q 1991; 13:8 –11 5 Donabedian A. Basic approaches to assessment: structure, process and outcome. In: The definition of quality and approaches to its assessment: explorations in quality assessment and monitoring (vol 1). Ann Arbor, MI: Health Administration Press, 1980; 77–128 6 Goodman RM, Wandersman A. FORECAST: a formative approach to evaluating community coalitions and communitybased initiatives. J Commun Psychol 1994; special issue:6 –25 7 Ryan MP, Buchan IC, Buckley EG. Medical audit: a preliminary report from general practice. J R Coll Gen Pract 1979; 29:719 –722 8 LeBaron CW, Chaney M, Baughman AL, et al. Impact of measurement and feedback on vaccination coverage in public clinics, 1988 –1994. JAMA 1997; 277:631– 635 9 Doney BJ. An audit of the care of diabetics in a group practice. J R Coll Gen Pract 1976; 26:734 –742 10 Hewer SM, Hambleton G, McKenzie S, et al. Asthma audit: a multicentre pilot study. Arch Dis Child 1994; 71:167–169 11 Verby JE, Holden P, Davis RH. Peer review of consultations in primary care: the use of audiovisual recordings. BMJ 1979; 1:1686 –1688 12 Holmboe E, Scranton R, Sumption K, et al. Effect of medical record audit and feedback on residents’ compliance with preventive health care guidelines. Acad Med 1998; 73:901– 903 13 Curtis P, Skinner B, Varenholt J, et al. Papanicolaou smear quality assurance: providing feedback to physicians. J Fam Pract 1993; 36:309 –312 14 Norton PF, Shaw PA, Murray MA. Quality improvement in family practice: program for Pap smears. Can Fam Physician 1997; 43:503–508 CHEST / 118 / 2 / AUGUST, 2000 SUPPLEMENT

57S

15 Reilly PM, Patten MP. An audit of prescribing by peer review. J R Coll Gen Pract 1978; 28:525–530 16 Palmer RH, Hargraves JL. The Ambulatory Care Medical Audit Demonstration Project: research design. Med Care 1996; 34:S12–S28 17 Lasker RD, Shapiro DW, Tucker AM. Realizing the potential of practice pattern profiling. Inquiry 1992; 29:287–297 18 Lee PR, Shapiro DW, Lasker RD, et al. Managed care: provider profiling. J Insur Med 1992; 24:179 –181 19 Welch HG, Miller ME, Welch WP. Physician profiling: an analysis of inpatient practice patterns in Florida and Oregon. N Engl J Med 1994; 330:607– 612 20 Balas EA, Boren SA, Brown GD, et al. Effect of physician profiling on utilization: meta-analysis of randomized clinical trials. J Gen Intern Med 1996; 11:584 –590 21 Krakauer H, Bailery RC, Skellan KJ, et al. Evaluation of the HCFA model for the analysis of mortality following hospitalization. Health Serv Res 1992; 27:317–335 22 Green J, Wintfeld N. How accurate are hospital discharge data for evaluation effectiveness of care? Med Care 1993; 31:719–731 23 Kassirer JP. The use and abuse of practice profiles. N Engl J Med 1994; 330:634 – 636 24 Emmons DW, Wozniak GD. Profiles and feedback: who measures physician performance? In: Gonzalez ML, ed. Socioeconomic characteristics of medical practice. Chicago, IL: American Medical Association, 1994; 11–16 25 Physician Payment Review Commission Conference on Profiling Washington, DC: Physician Payment Review Commis-

58S

sion, May 1992; Publication No. 92–2 26 Orav EJ, Wright EA, Palmer RH, et al. Issues of variability and bias affecting multisite measurement of quality of care. Med Care 1996; 34:SS87–SS101 27 Hofer TP, Hayward RA, Greenfield S, et al. The unreliability of individual physician “report cards” for assessing the costs and quality of care of a chronic disease. JAMA 1999; 281: 2098 –2105 28 Kim CS, Kristopaitis RJ, Stone E, et al. Physician education and report cards: do they make the grade? Results from a randomized controlled trial. Am J Med 1999; 107:556 –560 29 HEDIS 2000 narrative: what’s in it and why it matters. Washington, DC: National Committee for Quality Assurance, 1999; Publication No.10234 –100-00 30 Palmer RH, Peterson LE. Development and testing of performance measures for quality improvement in clinical preventive services. Am J Prev Med 1995; 11:402– 406 31 Lawtheres AB, Palmer RH, Edwards FE, et al. Developing and evaluating measures of ambulatory care quality: a preliminary report of the DEMPAY Project. J Qual Improv 1993; 19:552–565 32 Lansky D. Measuring what matters to the public. Health Aff 1998; 17:40 – 41 33 Braun BI, Koss RG, Loeb JM. Integrating performance measure data into the Joint Commission accreditation process. Eval Health Prof 1999; 22:283–297 34 Thompson RS. Systems approach and the delivery of health services. JAMA 1997; 277:668 – 671

Translating Guidelines Into Practice