- Email: [email protected]
Development and testing of performance measures for pharmacy services
special feature
Development and testing of performance measures for pharmacy services Donna Pillittere-Dugan, David P. Nau, Kimberly McDonough, and Zakiya Pierre
Received January 22, 2009, and in revised form February 14, 2009. Accepted for publication February 14, 2009.
Abstract Objective: To report on the status of the pilot work of PQA, a pharmacy quality alliance, to develop and test performance metrics of pharmacy services for use in quality improvement, benchmarking, and pay-for-performance benchmarks. Design: Observational cohort study. Setting: Three health plans (commercial, Medicare and Medicaid) located in the northeastern United States and one nationwide prescription drug plan. Patients: Pharmacies of health plans with membership ranging from approximately 3,330 to nearly 1.7 million members. Intervention: Pharmaceutical claims data for prescriptions dispensed at community pharmacies were analyzed. Main outcome measures: Not applicable. Results: The four plans had pharmacy networks ranging from 653 to 53,153 pharmacies. When using a minimum sample of 30 members per measure, less than 10% of the pharmacies within the plans’ networks were evaluable for all measures except the measure of high-risk drugs in the elderly. The measure for high-risk drugs in the elderly had 6,210 evaluable pharmacies in a network of 53,153. The measures for high-risk drugs in the elderly and medication adherence appear to have the greatest potential for use as performance measures in that they show room for improvement and variation among pharmacies. Conclusion: The ideal performance measure is relevant, scientifically sound, and feasible. Several of the measures that underwent testing possessed some, if not all, of the properties of an ideal performance measure. Strategies for aggregating data across health and drug plans may be useful for overcoming sample size challenges. Keywords: Quality control, quality improvement, practice standards, pay for performance. J Am Pharm Assoc. 2009;49:212–219. doi: 10.1331/JAPhA.2009.09012
Donna Pillittere-Dugan, MS, was Director, Performance Measurement, National Committee for Quality Assurance, Washington, DC, at the time this study was conducted; she is currently a private consultant in Washington, DC. David P. Nau, PhD, BPharm, CPHQ, was Director, Practice Improvement, PQA, Alexandria, VA, at the time this study was conducted; he is currently Research Manager, Competitive Health Analytics, Humana Pharmacy Solutions, Louisville, KY. Kimberly McDonough, PharmD, is President, Advanced Pharmacy Concepts, Inc., Kingston, RI. Zakiya Pierre, is Analyst, Performance Measurement, National Committee for Quality Assurance, Washington, DC. Correspondence: David Nau, PhD, BPharm, CPHQ, Humana Pharmacy Solutions, 500 W Main St., Louisville, KY 40202. Fax: 859-2460141. E-mail: [email protected] Disclosure: Other than their listed employers, the authors declare no conflicts of interest or financial interests in any product or service mentioned in this article, including grants, employment, gifts, stock holdings, or honoraria. Acknowledgments: The views expressed in this article are those of the authors and do not reflect the official policy or position of the U.S. Government or the Department of Defense. Development and testing of these measures was led by the National Committee for Quality Assurance, partnering with Advanced Pharmacy Concepts. The following experts served as part of the technical expert panel: Co-chair: Julie Kuhle, BPharm; Co-chair: Brad Tice, PharmD; Emily Cox, PhD; Ajit Dhavle, PharmD, MBA; Debra Dullinger, PharmD; Cathy Graeff, BPharm; David Medvedeff, PharmD, MBA; Jay Nadas, PharmD; David Nau, PhD, BPharm, CPHQ; Matt Palmgren, PharmD; Darren Triller, PharmD; and Alan Zillich, PharmD. Funding: By PQA, a pharmacy quality alliance. Previous presentation: AcademyHealth Annual Research Meeting, Washington DC, July 2008. See related articles on pages 143 and 153.
212 • JAPhA • 4 9 : 2 • M a r /A p r 2009
www.j aph a. or g
Journal of the American Pharmacists Association
Performance measures
J
ohnson and Bootman1 estimated that as many as 28% of all hospital admissions are the result of drug-related morbidity and mortality. Their landmark study related emergency department visits, long-term care admissions, and physician visits to medication misuse. They estimated that appropriate pharmaceutical care could prevent almost 5 million hospital admissions and more than 100,000 deaths each year. A more recent study examined the economic impact of pharmaceutical care and estimated that appropriate medication use could reduce the cost of drug-related mortality by $45 billion annually.2 Pharmacies and pharmacists have a unique opportunity to identify medication misuse and to intervene to improve the quality of a patient’s health. Despite
At a Glance
Synopsis: How can the quality of performance in ambulatory community pharmacy be measured? What are the criteria for determining whether measures are useful in assessing quality? These authors present the results of pilot work undertaken by PQA, a pharmacy quality alliance, to develop performance metrics that can be aggregated across health and drug plans for use in quality improvement and in pay-for-performance benchmarking in ambulatory care and community pharmacy. Prescription claims data for three regional health plans and one national prescription drug plan (membership ranged from about 3,330 to 1.7 million people; number of pharmacies ranged from 653 to 53,135) were evaluated on 22 measures that were developed through a conceptual process and then field tested. These quality measures were in the areas of adherence and persistence; safety; and diabetes, cardiovascular, and respiratory care. Two performance measures, high-risk drugs in the elderly and medication adherence, show the greatest potential for use as quality metrics because they show sufficient variation among pharmacies. Analysis: PQA was formed in 2006 by a group of stakeholders committed to the improvement of health care quality and patient safety in pharmacy practice. PQA’s goal is to develop a strategy for measuring pharmacy and pharmacist performance by using data that have been collected in the least burdensome way and reporting meaningful information to help consumers, pharmacists, employers, health plans, and others in making informed choices, improving outcomes, and developing new payment models. The initial work described in this article reveals the complexities involved in developing performance measures. This analysis indicates that measures related to medication adherence may be feasible and scientifically sound. The next phase of this work will examine and test the assumption that pharmacy personnel can influence scores on these measures.
Journal of the American Pharmacists Association
special feature
the potential impact of pharmacy services, little information is available to judge the value and quality of services provided by a pharmacy or pharmacist. In April 2006, a group of stakeholders committed to improving health care quality and patient safety in pharmacy practice formed PQA, a pharmacy quality alliance. PQA is a voluntary, membership-based collaborative comprising organizations from the pharmacy, patient, employer, and health insurance plan communities, as well as state and federal government. PQA’s mission is to improve health care quality and patient safety through a collaborative process in which key stakeholders agree on a strategy for measuring performance at the pharmacy and pharmacist levels, collecting data in the least burdensome way, and reporting meaningful information to patients, pharmacists, employers, health insurance plans, and other health care decision makers to help them make informed choices, improve outcomes, and stimulate the development of new payment models. More information on PQA is available at www.pqaalliance.org. Given the 2006 implementation of the Medicare Part D drug benefit, an initial goal of PQA was to identify potential measures of pharmacy quality that would be relevant to patients enrolled in Medicare drug plans and that could be put into place using existing data. An environmental scan of quality measures revealed no measures that were widely used to evaluate the quality of ambulatory/community pharmacies. Thus, the PQA convened a Quality Metrics Workgroup, as well as multiple subgroups, to develop quality measure concepts that could be further specified and tested. In November 2006, PQA identified a starter set of 37 quality measure concepts in the areas of adherence and persistence, efficiency, safety, and diabetes, cardiovascular, and respiratory care. PQA then sought competitive bids from organizations with expertise in measure development to evaluate the feasibility of creating measures for each concept area using only prescription drug claims data, create technical specifications for each measure, and test the measures using drug claims data. The measures’ specifications included definitions, numerator, denominator, inclusion/exclusion criteria, and drug lists. The National Committee for Quality Assurance (NCQA) was awarded the contract. NCQA is a not-for-profit organization dedicated to improving health care quality through measurement, transparency, and accountability. NCQA develops and maintains the measures within the Healthcare Effectiveness Data and Information Set (HEDIS). HEDIS is a tool used by more than 90% of America’s managed care organizations to measure performance on important dimensions of care and service. NCQA, in collaboration with Advanced Pharmacy Concepts (APC) as a subcontractor, was tasked with assembling a technical expert panel (TEP) to evaluate the feasibility of creating measures in each concept area, developing detailed technical measure specification algorithms, and conducting initial measure testing. As subcontractor, APC performed data aggregation and analysis and offered technical experwww. japh a. or g
M a r /A p r 20 0 9 • 4 9 : 2 •
JAPhA • 213
special feature
Performance measures
tise based on their experience with pharmacy databases and pharmaceutical claims.
Objectives This report describes the results of pilot testing a subset of the starter set of measures that were created by PQA and further defined by NCQA. Performance measures were tested using only pharmacy claim data because these quality measures are intended for use by pharmacies and drug plans that do not typically have ready access to patient-specific medical claims data.
as a marker for diagnosis. Using this methodology, members who suffer from the clinical condition of interest (e.g., asthma, diabetes) might be excluded, resulting in a false-negative misclassification error. Likewise, a patient receiving a target drug might not suffer from the clinical condition of interest (i.e., a false-positive). Fortunately, the majority of the measures focused on the patient’s treatment persistence or safety of treatment. In these cases, confirmation of the diagnosis is not germane to questions about the validity of the measures. Pharmacy performance rate calculation
NCQA and APC worked with the TEP to screen the initial set of measure concepts identified by PQA. Fifteen measures from the starter set were referred back to PQA because they were not feasible or presented conceptual challenges that required refinement. A total of 22 measures were deemed to be feasible to create using only prescription drug claims data and were specified more fully and further evaluated through field testing (Table 1). More information on the technical specifications for the measures is available from NCQA (www.ncqa. org).
A sample size of 30 or more is required within NCQA performance-reporting programs for providers. NCQA has recommended not reporting performance scores for a pharmacy that has fewer than 30 eligible members.3 Therefore, performance rates for each measure were calculated only for pharmacies that had a minimum sample size of 30 patients. In addition, performance rates were not reported for measures with 10 or fewer pharmacies meeting the minimum sample size criterion. This additional requirement was added because an accurate comparison of pharmacy performance may be less reliable in a sample size that contains 10 or fewer pharmacies.
Field testing
Attribution method
Methods
The field test was structured to provide varying perspectives of pharmacy services and of the measures. A crosssection of pharmacies serving members from commercial, Medicaid, and Medicare health plans was included. Pharmaceutical claims data for prescriptions dispensed at community pharmacies were analyzed from three plans in the northeastern United States and one national prescription drug plan (PDP) (Table 2). Plans participated in the field test by providing pharmacy claims data to APC under the terms of a formal data-sharing agreement. The field test research protocol was reviewed and approved by the Chesapeake Research Review Institutional Review Board (IRB). Data were analyzed and reported at the pharmacy and plan levels for each health or PDP. Eligible population
Eligible patients were identified from the drug claims data for each performance measure. Patients were included in the test of the measures if they received a drug from within the targeted class and were continuously eligible during the measurement period. Plan enrollment information was not available in the pharmacy claims database. Therefore, an algorithm was developed to serve as a proxy for continuous enrollment. Members were defined as continuously enrolled if they filled any two prescriptions within 150 days between the first and last prescription dispensed during a 12-month period. The algorithm ensured that the members included in the measures had adequate claims information for calculating the measure. Because the algorithm used only prescriptions as the basis for inclusion in the eligible population, the presence of medications dispensed in key therapeutic classes was used 214 • JAPhA • 4 9 : 2 • M a r /A p r 2009
www.j aph a. or g
For interpharmacy comparisons, accurately attributing patients to the correct pharmacy is important. In doing this, a balance must be sought between providing a sufficient number of measurement opportunities with patients and holding pharmacies accountable for services provided to patients. The more rigorous the methodology (i.e., requiring more fills or a higher percentage of fills before a patient is attributed to a pharmacy), the more restrictive the attribution. The downside of this more rigorous strategy is that it leads to a lower number of attributable patients/events. In most cases, an ongoing relationship between a pharmacy and a patient must be established before the pharmacy can be considered accountable for the ongoing management of services that patients receive. The exception to this guideline is a “never–never” situation. An example of a never–never situation is two contraindicated medications being dispensed at the same time, which should never occur. Specific rules were established to define how the pharmacy–patient link should be applied to determine which pharmacy will be considered accountable for pharmacy services rendered to a patient. ■■ For measures in which a performance event (i.e., one prescription dispensed) qualifies for the denominator or numerator of the measure, the pharmacy that dispenses the prescription is assumed to be accountable. ■■ When patients receive prescriptions from only one pharmacy during the measurement year, patients are attributed to that pharmacy. ■■ When patients use more than one pharmacy during the measurement year for medications within an identified drug class, patients are attributed to the pharmacy that dispensed the majority of the prescriptions in that drug class or drug classes. Journal of the American Pharmacists Association
Performance measures
Table 1. Recommended measures of pharmacy performance Measure PDC
Gap in therapy
Diabetes: ACEI/ ARB
Antidiabetic medication dosing
Suboptimal asthma control
Absence of asthma controller Use of high-risk medications in the elderly
Definition The percentage of patients who were dispensed a medication within the targeted drug class who met the PDC threshold of 80%. This category contained seven measures within targeted drug classes: ACEI/ARBs, beta-blockers, calcium-channel blockers, lipid modifiers, biguanides, sulfonylureas, and thiazolidinediones. The percentage of prevalent users of a medication within the targeted drug class who had a significant gap (>30 days) in medication therapy. This category contained seven measures with targeted drug classes: ACEI/ARBs, beta-blockers, calcium-channel blockers, lipid modifiers, biguanides, sulfonylureas, and thiazolidinediones. The percentage of patients who were dispensed a medication for diabetes and a medication for hypertension who are not receiving an ACEI or ARB. The percentage of patients who were dispensed a dose higher than the FDA-indicated maximum dose for the following three therapeutic categories of oral antihyperglycemic agents: biguanides, sulfonylureas, and thiazolidinediones. The percentage of patients with persistent asthma who were dispensed more than five canisters of a short-acting beta2 agonist inhaler over any 3-month period. The percentage of patients with persistent asthma and suboptimal control who received controller therapy. This category contains two measures: the percentage of patients 65 years or older who received at least one high-risk medication and the percentage of patients 65 years or older who received at least two different high-risk medications.
Abbreviations used: ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin II receptor blocker; FDA, Food and Drug Administration; PDC, proportion of days covered.
These rules were applied in addition to the continuous enrollment criterion. Analytic strategy
One concern for implementing pharmacy performance measures is that patients may use more than one pharmacy. If so, attribution of performance cannot be made to a single pharmacy. To assess the extent of this problem, we evaluated the percentage of patients that visited more than one phar-
Journal of the American Pharmacists Association
special feature
Table 2. Drug plan characteristics
Plan A
Plan B
Plan C Plan D
Description Commercial & Medicare health plan Medicare prescription drug plan Medicaid health plan Medicare Advantage plan
No. eligible membersa 850,461
Scope Regional
Population Commercial and Medicare
National
Medicare
867,016
One state Regional
Medicaid
35,369
Medicare
1,185
a Members who were continuously enrolled and who filled a prescription at a community pharmacy.
macy for medications within the targeted therapeutic class for each measure and the proportion of patients that could be attributed to a single pharmacy. Another concern is that the sample size requirement of 30 eligible members per measure per pharmacy would substantially limit the number of pharmacies that could be evaluated. Therefore, we determined the number of pharmacies within each plan’s network that met the sample size criterion. For performance scores to be useful, they should meet two additional criteria: (1) scores should vary substantially across pharmacies so that the measure can distinguish between “good” performers and “poor” performers, and (2) there should be room for improvement in performance for most pharmacies. We examined the distribution of scores by calculating the median score for all evaluable pharmacies, as well as the scores at the 10th and 90th percentiles. The median score gives an estimate of the room for improvement in performance, while the 10th and 90th percentiles identify the spread of scores across the evaluable pharmacies. Higher scores indicate better performance for some measures, and the 90th percentile identifies the top performers. For other measures, lower scores indicate better performance; therefore, the scores were inverted so that the 90th percentile would represent the top performers.
Results Attribution
For most measures, 85% to 90% of patients obtained the medication from the targeted therapeutic class from only one pharmacy. Measures for asthma and diabetes were the only exceptions. Only 72% to 78% of patients used a single pharmacy for asthma medications, and only 68% to 79% of patients received both an antidiabetic medication and an antihypertensive medication at the same pharmacy for the Diabetes: ACEI/ARB (angiotensin-converting enzyme inhibitor/ angiotensin II receptor blocker) measure. Fewer than 1% of www. japh a. or g
M a r /A p r 20 0 9 • 4 9 : 2 •
JAPhA • 215
special feature
Performance measures
Table 3. Pharmacies with sample size of at least 30 eligible members per measure No. network pharmacies Performance measuresa High-risk drugs in the elderly PDC beta-blocker PDC ACEI/ARB PDC calcium-channel blocker PDC dyslipidemia PDC diabetes Gap beta-blocker Gap ACEI/ARB Gap calcium-channel blocker Gap dyslipidemia Gap diabetes Diabetes: ACEI/ARB Diabetes medication dosing Suboptimal asthma control Absence of asthma controller
Health plan A 34,177
Drug plan B 53,153
1,088 821 860 536 883 402 806 833 479 861 372 112 402 99 1
62,10 2,001 2,001 1,504 2,357 990 1,842 1,980 1,379 2,160 927 686 990 NA NA
Health plan C 653 0 7 10 3 10 0 3 9 2 7 0 0 0 2 0
Health plan D 658 0 0 0 0 0 0 0 0 0 0 0 0 0 NA NA
Abbreviations used: ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin II receptor blocker; Gap, gap in therapy; PDC, proportion of days covered. Numbers within cells show the number of pharmacies with at least 30 members for that measure.
a
patients obtained medications from the targeted therapeutic class from more than two pharmacies. The attribution logic allowed nearly all patients to be assigned to a single pharmacy.
with at least 30 patients meeting the inclusion criteria, and plan B (Medicare PDP) did not have any evaluable pharmacies.
Sample size
Discussion
A minimum sample size of 30 eligible members per measure per pharmacy leads to a considerable limitation on the number of pharmacies that can be evaluated (Table 3). The larger plans (i.e., plan A and plan B) had several hundred to several thousand pharmacies with at least 30 eligible members per measure. However, this represented fewer than 10% of the plan’s pharmacies within their networks. Too few pharmacies were evaluable for plan C (Medicaid), and none was evaluable for plan D (Medicare Advantage). The asthma measures were not evaluated in plans B and D because the technical specifications for those measures exclude patients older than 50 years because older adults will frequently use short-acting inhalers for chronic obstructive pulmonary disease. For plans A and C, the number of pharmacies meeting the sample size criterion was too small to test the absence of asthma controller measure. Distribution of scores
The distribution of scores was examined for the eligible pharmacies. There is room for improvement in pharmacy performance and substantial variation among pharmacies for nearly all measures (Table 4). The notable exception was the medication dosing for biguanides and sulfonylureas. The majority of pharmacies had no patients with excessive doses of these drugs. Additionally, the asthma controller measure could not be evaluated because plan A had only one pharmacy 216 • JAPhA • 4 9 : 2 • M a r /A p r 2009
www.j aph a. or g
The PQA process for developing and evaluating measures is very complex and evolving, as illustrated by this report of PQA’s first pilot of the starter set of concept measures. The process experienced by PQA parallels that of other organizations involved in measure development and endorsement activities.4,5 The process consisted of obtaining expert guidance in selecting measure concepts that are meaningful, based on sound clinical evidence, and that represent gaps in clinical quality care. The selected process also facilitates development of precise measure specifications and algorithms to enable apples-to-apples comparisons. Moreover, use of relevant data sets informs decisions regarding whether the measures would be useful and feasible for future implementation. Further validation of specific proxy algorithms, such as enrollment criteria and diagnosis information, are planned for future PQA demonstration projects. The ideal performance measure is relevant, scientifically sound, feasible, and usable for quality improvement.5 The PQA membership and the numerous workgroups that formulated the measure concepts determined the relevance of the pharmacy performance measures. Future PQA demonstration projects will test the measures’ usability in partnership with community pharmacies. This report supports the notion that performance measures related to medication adherence may be feasible and scientifically sound because they showed variation and room for improvement for pharmacy services. Journal of the American Pharmacists Association
Performance measures
However, more work is necessary to fully evaluate the scientific soundness of these measures. The greatest limitation identified in our initial work is that only a small proportion of pharmacies within a single plan’s network may be reliably evaluated. When using a minimum sample of 30 members per pharmacy, fewer than 10% of the pharmacies, even within a large plan’s network, may be evaluable. For small health plans, measures of quality at the pharmacy level may be more difficult. Plan D represented a Medicare Advantage health maintenance organization with fewer than 5,000 members. No pharmacies in plan D’s network met the minimum of 30 patients per pharmacy. Plan C (Medicaid) also had very few pharmacies that were evaluable, and thus the expense and effort of implementing a performance evaluation system may not be warranted. The problem of too few units to evaluate is not unique to pharmacy. Scholle and colleagues evaluated physician performance measures across nine health plans and found that very few primary care physicians met a sample size threshold of 30 quality events.3 However, the majority of quality events were accounted for within the small number of physicians that did meet the threshold. Thus, a single health plan may be able to evaluate the physicians that provide care to a large portion of the plan’s membership. The same pattern may be true of pharmacy measures for large, regional drug plans for which the membership is concentrated in a smaller number of pharmacies. One plausible strategy for overcoming the problem of a small number of eligible units is to aggregate provider’s data across health plans. In California, the Integrated Healthcare Association aggregates physician performance data across multiple health plans.6 Small health plans (<500,000 members) could only evaluate 16% of their physician groups. But after creating an aggregated dataset together with other health plans, more than 70% of physician groups were evaluable. Demonstration projects supported by the federal government and by PQA are examining ways to aggregate data to enhance performance reporting.7,8 An aggregated model for performance assessment may be a viable option to increase the number of pharmacies that can be reliably evaluated since most community pharmacies serve patients enrolled in multiple drug plans. Another strategy to increase the number of evaluable pharmacies is to combine similar performance measures into a composite measure.9 For example, a composite measure of persistence could include the compilation of results from individual measures of persistence within targeted therapeutic classes. Thus, if a pharmacy had 10 patients each within the measures of persistence for beta-blockers, calcium-channel blockers, ACEI/ARBs, and lipid-lowering drugs, a composite measure could include 40 patients across the four classes. One drawback to this method is that the composite measure may be less useful for quality improvement because the pharmacy could not identify whether problems with persistence are limited to a single therapeutic class or across all four classes. Journal of the American Pharmacists Association
special feature
An entirely different approach could be used to overcome the same problem. Pharmacies could calculate their performance on each measure and report the results to an independent entity. An external auditor could validate the report’s accuracy at the pharmacy’s expense. This would be analogous to the approach used by The Joint Commission when evaluating hospital quality and by NCQA when evaluating health plan quality. The cost of gathering data, calculating and reporting their performance, and auditing the results would be a considerable burden for pharmacies that lack the resources of a national chain. Nonetheless, a pharmacy that seeks to continuously improve the quality of its services will need to invest in performance measurement. Although quantitative benchmarks of pharmacy quality are not widely available, the results for some of the performance measures (Table 4) can be compared with reports from other sectors. The rates of medication adherence, as measured by PDC and gaps in therapy, seem relatively consistent with the extant literature across the various drug classes.10–12 For example, measures of high-risk medication use in the elderly have been included in HEDIS for Medicare health plans. Estimates for plan A and plan B pharmacies were inbetween the 2007 HEDIS rates for this measure.13 This study is not the first to report quality improvement efforts in community pharmacies. Other researchers and leaders have described ways to build quality improvement systems for pharmacies.14–18 The endeavors of PQA differ in that they sought to identify and test a core set of performance measures that could be widely implemented and that could generate comparative information on community pharmacies. The Secretary of Health and Human Services has called for greater transparency in health care.19 This first initiative by the PQA and its stakeholders, including the American Pharmacists Association, moves us one step toward transparent goals for measuring the quality of pharmacy services.
Limitations This pilot study of pharmacy quality measures has several limitations. The three participating health plans were located in the same geographic region (northeastern United States), and national coverage was represented by only one PDP. Thus, conclusions cannot be drawn about the quality of pharmacies nationwide. The quality measures were based solely on prescription drug claims data from health/drug plans; therefore, the measures did not include medications for which no drug claim was submitted for payment (e.g., cash purchases, over-the-counter medications, injectable medications billed through Medicare Part B). In addition, the diagnosis of a patient could not be confirmed for the measures intended to focus on persons with asthma or diabetes, although the inclusion criteria were constructed to minimize the likelihood of misidentifying the clinical condition. Important limitations were associated with the size of the pharmacy samples. Within an individual health plan, a small number of pharmacies provide the majority of prescriptions while a very large number of pharmacies dispense a smaller www. japh a. or g
M a r /A p r 20 0 9 • 4 9 : 2 •
JAPhA • 217
special feature
PERFORMANCE MEASURES
Table 4. Distribution of pharmacy performance measuresa 10th percentileb % High-risk medications in the elderly One or more Two or more PDC beta-blocker PDC ACEI/ARB PDC calcium-channel blocker PDC dyslipidemia PDC diabetes Biguanides Sulfonylureas Thiazolidinediones Gap beta-blocker Gap ACEI/ARB Gap calcium-channel blocker Gap dyslipidemia Gap diabetes Biguanides Sulfonylureas Thiazolidinediones Diabetes: ACEI/ARB Diabetes medication dosing Biguanides Sulfonylureas Thiazolidinediones Suboptimal asthma control Absence of asthma controllerd
Plan A Median %
90th percentilec %
10th percentileb %
Plan B Median %
90th percentilec %
21.4 13.2
15.3 8.6
8.8 4.4
45.7 33.8
32.3 22.0
16.2 9.6
73.5 75.6 77.7 66.7
80.5 82.7 85.8 75.0
86.5 88.9 92.3 81.7
69.2 72.3 75.0 68.1
79.5 81.8 84.0 78.4
89.1 90.4 92.1 88.4
56.0 60.7 57.1 31.4 28.4 26.6 36.4
69.2 75.7 75.0 23.8 21.0 17.9 28.4
81.0 87.1 92.3 17.1 14.9 10.8 22.2
58.3 63.3 60.0 36.7 34.7 31.4 39.5
73.7 78.6 76.9 26.3 25.0 22.0 29.4
87.5 90.9 91.7 16.6 15.6 13.2 18.8
41.2 40.0 42.9 39.3
30.4 26.7 25.0 26.5
20.0 14.3 10.0 15.6
42.1 40.0 43.5 40.4
29.2 25.8 27.3 26.5
15.8 13.3 13.0 14.0
4.9 4.5 20.0 8.8 —
0 0 7.7 3.0 —
0 0 0 0 —
5.6 7.1 18.8 NA NA
0 0 6.7 NA NA
0 0 0 NA NA
Abbreviations used: ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin II receptor blocker; Gap, gap in therapy; NA, not applicable (asthma measures were not tested in Plan B because it was a Medicare prescription drug plan); PDC, proportion of days covered. a Cells show the percent of patients at a pharmacy who met the numerator threshold for each measure. Only pharmacies with 30 eligible members per measure were included in this analysis. b The 10th percentile column shows the demarcation for the bottom 10% of pharmacies. c The 90th percentile column shows the demarcation for the top 10% of pharmacies. d Too few pharmacies met the sample size criteria for a reliable estimate to be calculated.
quantity. Because measurement of quality was limited only to those pharmacies with a sufficient number of prescription events, quality measures apply to only a minority of participating pharmacies in a health plan’s panel. A final limitation is the assumption that the pharmacy personnel can influence the scores on these measures and that the measures therefore reflect the quality of a pharmacy. This important assumption will be tested in Phase II of the PQA demonstration projects, but the results of these projects will not be available until 2010 at the earliest.
mentation in a more “real-world” setting, along with the validity of certain proxy algorithms, will be tested. Given the challenge of obtaining sufficient denominator sizes, strategies for aggregating pharmacy performance data across health and drug plans will be crucial for generating performance reports that provide reliable information about the majority of community pharmacies. 1.
Johnson J, Bootman JL. Drug-related morbidity and mortality. Arch Intern Med. 1995; 155:1949–55.
conclusion
2.
Based on the results of this analysis, some measures for evaluating pharmacy performance appear to show variation between and room for improvement within pharmacies. The viability of these measures will be further evaluated through their use in PQA demonstration projects, where their imple-
Johnson J, Bootman JL. Drug-related morbidity and mortality and the economic impact of pharmaceutical care. Am J Health Syst Pharm. 1997;54:554–8.
3.
Scholle SH, Roski J, Dunn DL, et al. Availability of data for measuring physician quality performance. Am J Manag Care. 2009;15:67–72.
218 • JAPhA • 4 9 : 2 • M a r /a p r 2009
www.j aph a. or g
references
Journal of the American Pharmacists Association
PERFORMANCE MEASURES
special feature
4.
National Committee for Quality Assurance (NCQA). Measure Development Process. Accessed at www.ncqa.org/tabid/414/ Default.aspx, January 14, 2009.
12. Rozenfeld Y, Hunt JL, Plauschinat C, Wong KS. Oral antidiabetic medication adherence and glycemic control in managed care. Am J Manag Care. 2008;14,71–5.
5.
National Quality Forum. Measure evaluation criteria. Accessed at www.qualityforum.org/about/leadership/measure_evaluation.asp, January 14, 2009.
13. National Committee for Quality Assurance (NCQA). State of health care quality 2007. Accessed at www.ncqa.org, January 14, 2009.
6.
Integrated Healthcare Association. Advancing quality through collaboration: the California Pay-for-Performance Program. Accessed at www.iha.org/wp020606.pdf, January 14, 2009.
14. Hepler CD, Segal R. Preventing medication errors and improving drug therapy. Boca Raton, FL: CRC Press; 2003.
7.
Centers for Medicare and Medicaid Services. Better quality information. Accessed at www.cms.hhs.gov/BQI/, January 14, 2009.
8.
9.
Pharmacy Quality Alliance (PQA). PQA announces official launch of demonstration projects. Accessed at www.pqaalliance.org/files/PQALaunchofDemoProjectsJuly2008.pdf, January 14, 2009. Scholle SH, Roski J, Adams JL, et al. Benchmarking physician performance: reliability of individual and composite measures. Am J Manag Care. 2008;14:829–38.
10. Chernew ME, Shah MR, Weigh A, et al. Impact of decreasing copayments on medication adherence in a disease-management environment. Health Aff. 2008;27:103–12. 11. Nau DP, Steinke DT, Williams LK, et al. Adherence analysis using a visual analog scale versus claims-based estimation. Ann Pharmacother. 2007;41:1792–7.
15. Angaran DM. Selecting, developing, and evaluating indicators. Am J Hosp Pharm. 1991;48:1931–7. 16. Isetts BJ, Brown LM, Schondelmeyer SW, Lenarz LA. Quality assessment of a collaborative approach for decreasing drugrelated morbidity and achieving therapeutic goals. Arch Intern Med. 2003;163:1813–20. 17. Farris KB, Kirking DM. Assessing the quality of pharmaceutical care. II. Application of concepts of quality assessment from medical care. Ann Pharmacother. 1993;27:215–23. 18. Fry RN, Avey SG. Framework for pharmacy services quality improvement—a bridge to cross the quality chasm. Part I. The opportunity and the tool. J Manag Care Pharm. 2004;10:60–78.
19. Department of Health and Human Services. Building a valuebased health care system. Accessed at www.hhs.gov/valuedriven/, January 11, 2009.
Certified Pharmacy Technician trained
tested trusted
Continuous Testing Begins April 1, 2009 The PTCB Certification Program is the only certification endorsed by the American Pharmacists Association (APhA), the American Society of Health-System Pharmacists (ASHP), the National Association of Boards J oof u r nPharmacy a l o f t h e A m(NABP) e r i c a n P hand a r m a cother i s t s A s professional sociation pharmacy In 2008, PTCB tested 50,000 www.organizations. japh a. or g M a r /a p r 20 0 9 • 4 9 : 2over • JAPhA • 219 pharmacy technicians and has certified over 330,000 CPhTs since 1995.
Development and testing of performance measures for pharmacy services Donna Pillittere-Dugan, David P. Nau, Kimberly McDonough, and Zakiya Pierre
Received January 22, 2009, and in revised form February 14, 2009. Accepted for publication February 14, 2009.
Abstract Objective: To report on the status of the pilot work of PQA, a pharmacy quality alliance, to develop and test performance metrics of pharmacy services for use in quality improvement, benchmarking, and pay-for-performance benchmarks. Design: Observational cohort study. Setting: Three health plans (commercial, Medicare and Medicaid) located in the northeastern United States and one nationwide prescription drug plan. Patients: Pharmacies of health plans with membership ranging from approximately 3,330 to nearly 1.7 million members. Intervention: Pharmaceutical claims data for prescriptions dispensed at community pharmacies were analyzed. Main outcome measures: Not applicable. Results: The four plans had pharmacy networks ranging from 653 to 53,153 pharmacies. When using a minimum sample of 30 members per measure, less than 10% of the pharmacies within the plans’ networks were evaluable for all measures except the measure of high-risk drugs in the elderly. The measure for high-risk drugs in the elderly had 6,210 evaluable pharmacies in a network of 53,153. The measures for high-risk drugs in the elderly and medication adherence appear to have the greatest potential for use as performance measures in that they show room for improvement and variation among pharmacies. Conclusion: The ideal performance measure is relevant, scientifically sound, and feasible. Several of the measures that underwent testing possessed some, if not all, of the properties of an ideal performance measure. Strategies for aggregating data across health and drug plans may be useful for overcoming sample size challenges. Keywords: Quality control, quality improvement, practice standards, pay for performance. J Am Pharm Assoc. 2009;49:212–219. doi: 10.1331/JAPhA.2009.09012
Donna Pillittere-Dugan, MS, was Director, Performance Measurement, National Committee for Quality Assurance, Washington, DC, at the time this study was conducted; she is currently a private consultant in Washington, DC. David P. Nau, PhD, BPharm, CPHQ, was Director, Practice Improvement, PQA, Alexandria, VA, at the time this study was conducted; he is currently Research Manager, Competitive Health Analytics, Humana Pharmacy Solutions, Louisville, KY. Kimberly McDonough, PharmD, is President, Advanced Pharmacy Concepts, Inc., Kingston, RI. Zakiya Pierre, is Analyst, Performance Measurement, National Committee for Quality Assurance, Washington, DC. Correspondence: David Nau, PhD, BPharm, CPHQ, Humana Pharmacy Solutions, 500 W Main St., Louisville, KY 40202. Fax: 859-2460141. E-mail: [email protected] Disclosure: Other than their listed employers, the authors declare no conflicts of interest or financial interests in any product or service mentioned in this article, including grants, employment, gifts, stock holdings, or honoraria. Acknowledgments: The views expressed in this article are those of the authors and do not reflect the official policy or position of the U.S. Government or the Department of Defense. Development and testing of these measures was led by the National Committee for Quality Assurance, partnering with Advanced Pharmacy Concepts. The following experts served as part of the technical expert panel: Co-chair: Julie Kuhle, BPharm; Co-chair: Brad Tice, PharmD; Emily Cox, PhD; Ajit Dhavle, PharmD, MBA; Debra Dullinger, PharmD; Cathy Graeff, BPharm; David Medvedeff, PharmD, MBA; Jay Nadas, PharmD; David Nau, PhD, BPharm, CPHQ; Matt Palmgren, PharmD; Darren Triller, PharmD; and Alan Zillich, PharmD. Funding: By PQA, a pharmacy quality alliance. Previous presentation: AcademyHealth Annual Research Meeting, Washington DC, July 2008. See related articles on pages 143 and 153.
212 • JAPhA • 4 9 : 2 • M a r /A p r 2009
www.j aph a. or g
Journal of the American Pharmacists Association
Performance measures
J
ohnson and Bootman1 estimated that as many as 28% of all hospital admissions are the result of drug-related morbidity and mortality. Their landmark study related emergency department visits, long-term care admissions, and physician visits to medication misuse. They estimated that appropriate pharmaceutical care could prevent almost 5 million hospital admissions and more than 100,000 deaths each year. A more recent study examined the economic impact of pharmaceutical care and estimated that appropriate medication use could reduce the cost of drug-related mortality by $45 billion annually.2 Pharmacies and pharmacists have a unique opportunity to identify medication misuse and to intervene to improve the quality of a patient’s health. Despite
At a Glance
Synopsis: How can the quality of performance in ambulatory community pharmacy be measured? What are the criteria for determining whether measures are useful in assessing quality? These authors present the results of pilot work undertaken by PQA, a pharmacy quality alliance, to develop performance metrics that can be aggregated across health and drug plans for use in quality improvement and in pay-for-performance benchmarking in ambulatory care and community pharmacy. Prescription claims data for three regional health plans and one national prescription drug plan (membership ranged from about 3,330 to 1.7 million people; number of pharmacies ranged from 653 to 53,135) were evaluated on 22 measures that were developed through a conceptual process and then field tested. These quality measures were in the areas of adherence and persistence; safety; and diabetes, cardiovascular, and respiratory care. Two performance measures, high-risk drugs in the elderly and medication adherence, show the greatest potential for use as quality metrics because they show sufficient variation among pharmacies. Analysis: PQA was formed in 2006 by a group of stakeholders committed to the improvement of health care quality and patient safety in pharmacy practice. PQA’s goal is to develop a strategy for measuring pharmacy and pharmacist performance by using data that have been collected in the least burdensome way and reporting meaningful information to help consumers, pharmacists, employers, health plans, and others in making informed choices, improving outcomes, and developing new payment models. The initial work described in this article reveals the complexities involved in developing performance measures. This analysis indicates that measures related to medication adherence may be feasible and scientifically sound. The next phase of this work will examine and test the assumption that pharmacy personnel can influence scores on these measures.
Journal of the American Pharmacists Association
special feature
the potential impact of pharmacy services, little information is available to judge the value and quality of services provided by a pharmacy or pharmacist. In April 2006, a group of stakeholders committed to improving health care quality and patient safety in pharmacy practice formed PQA, a pharmacy quality alliance. PQA is a voluntary, membership-based collaborative comprising organizations from the pharmacy, patient, employer, and health insurance plan communities, as well as state and federal government. PQA’s mission is to improve health care quality and patient safety through a collaborative process in which key stakeholders agree on a strategy for measuring performance at the pharmacy and pharmacist levels, collecting data in the least burdensome way, and reporting meaningful information to patients, pharmacists, employers, health insurance plans, and other health care decision makers to help them make informed choices, improve outcomes, and stimulate the development of new payment models. More information on PQA is available at www.pqaalliance.org. Given the 2006 implementation of the Medicare Part D drug benefit, an initial goal of PQA was to identify potential measures of pharmacy quality that would be relevant to patients enrolled in Medicare drug plans and that could be put into place using existing data. An environmental scan of quality measures revealed no measures that were widely used to evaluate the quality of ambulatory/community pharmacies. Thus, the PQA convened a Quality Metrics Workgroup, as well as multiple subgroups, to develop quality measure concepts that could be further specified and tested. In November 2006, PQA identified a starter set of 37 quality measure concepts in the areas of adherence and persistence, efficiency, safety, and diabetes, cardiovascular, and respiratory care. PQA then sought competitive bids from organizations with expertise in measure development to evaluate the feasibility of creating measures for each concept area using only prescription drug claims data, create technical specifications for each measure, and test the measures using drug claims data. The measures’ specifications included definitions, numerator, denominator, inclusion/exclusion criteria, and drug lists. The National Committee for Quality Assurance (NCQA) was awarded the contract. NCQA is a not-for-profit organization dedicated to improving health care quality through measurement, transparency, and accountability. NCQA develops and maintains the measures within the Healthcare Effectiveness Data and Information Set (HEDIS). HEDIS is a tool used by more than 90% of America’s managed care organizations to measure performance on important dimensions of care and service. NCQA, in collaboration with Advanced Pharmacy Concepts (APC) as a subcontractor, was tasked with assembling a technical expert panel (TEP) to evaluate the feasibility of creating measures in each concept area, developing detailed technical measure specification algorithms, and conducting initial measure testing. As subcontractor, APC performed data aggregation and analysis and offered technical experwww. japh a. or g
M a r /A p r 20 0 9 • 4 9 : 2 •
JAPhA • 213
special feature
Performance measures
tise based on their experience with pharmacy databases and pharmaceutical claims.
Objectives This report describes the results of pilot testing a subset of the starter set of measures that were created by PQA and further defined by NCQA. Performance measures were tested using only pharmacy claim data because these quality measures are intended for use by pharmacies and drug plans that do not typically have ready access to patient-specific medical claims data.
as a marker for diagnosis. Using this methodology, members who suffer from the clinical condition of interest (e.g., asthma, diabetes) might be excluded, resulting in a false-negative misclassification error. Likewise, a patient receiving a target drug might not suffer from the clinical condition of interest (i.e., a false-positive). Fortunately, the majority of the measures focused on the patient’s treatment persistence or safety of treatment. In these cases, confirmation of the diagnosis is not germane to questions about the validity of the measures. Pharmacy performance rate calculation
NCQA and APC worked with the TEP to screen the initial set of measure concepts identified by PQA. Fifteen measures from the starter set were referred back to PQA because they were not feasible or presented conceptual challenges that required refinement. A total of 22 measures were deemed to be feasible to create using only prescription drug claims data and were specified more fully and further evaluated through field testing (Table 1). More information on the technical specifications for the measures is available from NCQA (www.ncqa. org).
A sample size of 30 or more is required within NCQA performance-reporting programs for providers. NCQA has recommended not reporting performance scores for a pharmacy that has fewer than 30 eligible members.3 Therefore, performance rates for each measure were calculated only for pharmacies that had a minimum sample size of 30 patients. In addition, performance rates were not reported for measures with 10 or fewer pharmacies meeting the minimum sample size criterion. This additional requirement was added because an accurate comparison of pharmacy performance may be less reliable in a sample size that contains 10 or fewer pharmacies.
Field testing
Attribution method
Methods
The field test was structured to provide varying perspectives of pharmacy services and of the measures. A crosssection of pharmacies serving members from commercial, Medicaid, and Medicare health plans was included. Pharmaceutical claims data for prescriptions dispensed at community pharmacies were analyzed from three plans in the northeastern United States and one national prescription drug plan (PDP) (Table 2). Plans participated in the field test by providing pharmacy claims data to APC under the terms of a formal data-sharing agreement. The field test research protocol was reviewed and approved by the Chesapeake Research Review Institutional Review Board (IRB). Data were analyzed and reported at the pharmacy and plan levels for each health or PDP. Eligible population
Eligible patients were identified from the drug claims data for each performance measure. Patients were included in the test of the measures if they received a drug from within the targeted class and were continuously eligible during the measurement period. Plan enrollment information was not available in the pharmacy claims database. Therefore, an algorithm was developed to serve as a proxy for continuous enrollment. Members were defined as continuously enrolled if they filled any two prescriptions within 150 days between the first and last prescription dispensed during a 12-month period. The algorithm ensured that the members included in the measures had adequate claims information for calculating the measure. Because the algorithm used only prescriptions as the basis for inclusion in the eligible population, the presence of medications dispensed in key therapeutic classes was used 214 • JAPhA • 4 9 : 2 • M a r /A p r 2009
www.j aph a. or g
For interpharmacy comparisons, accurately attributing patients to the correct pharmacy is important. In doing this, a balance must be sought between providing a sufficient number of measurement opportunities with patients and holding pharmacies accountable for services provided to patients. The more rigorous the methodology (i.e., requiring more fills or a higher percentage of fills before a patient is attributed to a pharmacy), the more restrictive the attribution. The downside of this more rigorous strategy is that it leads to a lower number of attributable patients/events. In most cases, an ongoing relationship between a pharmacy and a patient must be established before the pharmacy can be considered accountable for the ongoing management of services that patients receive. The exception to this guideline is a “never–never” situation. An example of a never–never situation is two contraindicated medications being dispensed at the same time, which should never occur. Specific rules were established to define how the pharmacy–patient link should be applied to determine which pharmacy will be considered accountable for pharmacy services rendered to a patient. ■■ For measures in which a performance event (i.e., one prescription dispensed) qualifies for the denominator or numerator of the measure, the pharmacy that dispenses the prescription is assumed to be accountable. ■■ When patients receive prescriptions from only one pharmacy during the measurement year, patients are attributed to that pharmacy. ■■ When patients use more than one pharmacy during the measurement year for medications within an identified drug class, patients are attributed to the pharmacy that dispensed the majority of the prescriptions in that drug class or drug classes. Journal of the American Pharmacists Association
Performance measures
Table 1. Recommended measures of pharmacy performance Measure PDC
Gap in therapy
Diabetes: ACEI/ ARB
Antidiabetic medication dosing
Suboptimal asthma control
Absence of asthma controller Use of high-risk medications in the elderly
Definition The percentage of patients who were dispensed a medication within the targeted drug class who met the PDC threshold of 80%. This category contained seven measures within targeted drug classes: ACEI/ARBs, beta-blockers, calcium-channel blockers, lipid modifiers, biguanides, sulfonylureas, and thiazolidinediones. The percentage of prevalent users of a medication within the targeted drug class who had a significant gap (>30 days) in medication therapy. This category contained seven measures with targeted drug classes: ACEI/ARBs, beta-blockers, calcium-channel blockers, lipid modifiers, biguanides, sulfonylureas, and thiazolidinediones. The percentage of patients who were dispensed a medication for diabetes and a medication for hypertension who are not receiving an ACEI or ARB. The percentage of patients who were dispensed a dose higher than the FDA-indicated maximum dose for the following three therapeutic categories of oral antihyperglycemic agents: biguanides, sulfonylureas, and thiazolidinediones. The percentage of patients with persistent asthma who were dispensed more than five canisters of a short-acting beta2 agonist inhaler over any 3-month period. The percentage of patients with persistent asthma and suboptimal control who received controller therapy. This category contains two measures: the percentage of patients 65 years or older who received at least one high-risk medication and the percentage of patients 65 years or older who received at least two different high-risk medications.
Abbreviations used: ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin II receptor blocker; FDA, Food and Drug Administration; PDC, proportion of days covered.
These rules were applied in addition to the continuous enrollment criterion. Analytic strategy
One concern for implementing pharmacy performance measures is that patients may use more than one pharmacy. If so, attribution of performance cannot be made to a single pharmacy. To assess the extent of this problem, we evaluated the percentage of patients that visited more than one phar-
Journal of the American Pharmacists Association
special feature
Table 2. Drug plan characteristics
Plan A
Plan B
Plan C Plan D
Description Commercial & Medicare health plan Medicare prescription drug plan Medicaid health plan Medicare Advantage plan
No. eligible membersa 850,461
Scope Regional
Population Commercial and Medicare
National
Medicare
867,016
One state Regional
Medicaid
35,369
Medicare
1,185
a Members who were continuously enrolled and who filled a prescription at a community pharmacy.
macy for medications within the targeted therapeutic class for each measure and the proportion of patients that could be attributed to a single pharmacy. Another concern is that the sample size requirement of 30 eligible members per measure per pharmacy would substantially limit the number of pharmacies that could be evaluated. Therefore, we determined the number of pharmacies within each plan’s network that met the sample size criterion. For performance scores to be useful, they should meet two additional criteria: (1) scores should vary substantially across pharmacies so that the measure can distinguish between “good” performers and “poor” performers, and (2) there should be room for improvement in performance for most pharmacies. We examined the distribution of scores by calculating the median score for all evaluable pharmacies, as well as the scores at the 10th and 90th percentiles. The median score gives an estimate of the room for improvement in performance, while the 10th and 90th percentiles identify the spread of scores across the evaluable pharmacies. Higher scores indicate better performance for some measures, and the 90th percentile identifies the top performers. For other measures, lower scores indicate better performance; therefore, the scores were inverted so that the 90th percentile would represent the top performers.
Results Attribution
For most measures, 85% to 90% of patients obtained the medication from the targeted therapeutic class from only one pharmacy. Measures for asthma and diabetes were the only exceptions. Only 72% to 78% of patients used a single pharmacy for asthma medications, and only 68% to 79% of patients received both an antidiabetic medication and an antihypertensive medication at the same pharmacy for the Diabetes: ACEI/ARB (angiotensin-converting enzyme inhibitor/ angiotensin II receptor blocker) measure. Fewer than 1% of www. japh a. or g
M a r /A p r 20 0 9 • 4 9 : 2 •
JAPhA • 215
special feature
Performance measures
Table 3. Pharmacies with sample size of at least 30 eligible members per measure No. network pharmacies Performance measuresa High-risk drugs in the elderly PDC beta-blocker PDC ACEI/ARB PDC calcium-channel blocker PDC dyslipidemia PDC diabetes Gap beta-blocker Gap ACEI/ARB Gap calcium-channel blocker Gap dyslipidemia Gap diabetes Diabetes: ACEI/ARB Diabetes medication dosing Suboptimal asthma control Absence of asthma controller
Health plan A 34,177
Drug plan B 53,153
1,088 821 860 536 883 402 806 833 479 861 372 112 402 99 1
62,10 2,001 2,001 1,504 2,357 990 1,842 1,980 1,379 2,160 927 686 990 NA NA
Health plan C 653 0 7 10 3 10 0 3 9 2 7 0 0 0 2 0
Health plan D 658 0 0 0 0 0 0 0 0 0 0 0 0 0 NA NA
Abbreviations used: ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin II receptor blocker; Gap, gap in therapy; PDC, proportion of days covered. Numbers within cells show the number of pharmacies with at least 30 members for that measure.
a
patients obtained medications from the targeted therapeutic class from more than two pharmacies. The attribution logic allowed nearly all patients to be assigned to a single pharmacy.
with at least 30 patients meeting the inclusion criteria, and plan B (Medicare PDP) did not have any evaluable pharmacies.
Sample size
Discussion
A minimum sample size of 30 eligible members per measure per pharmacy leads to a considerable limitation on the number of pharmacies that can be evaluated (Table 3). The larger plans (i.e., plan A and plan B) had several hundred to several thousand pharmacies with at least 30 eligible members per measure. However, this represented fewer than 10% of the plan’s pharmacies within their networks. Too few pharmacies were evaluable for plan C (Medicaid), and none was evaluable for plan D (Medicare Advantage). The asthma measures were not evaluated in plans B and D because the technical specifications for those measures exclude patients older than 50 years because older adults will frequently use short-acting inhalers for chronic obstructive pulmonary disease. For plans A and C, the number of pharmacies meeting the sample size criterion was too small to test the absence of asthma controller measure. Distribution of scores
The distribution of scores was examined for the eligible pharmacies. There is room for improvement in pharmacy performance and substantial variation among pharmacies for nearly all measures (Table 4). The notable exception was the medication dosing for biguanides and sulfonylureas. The majority of pharmacies had no patients with excessive doses of these drugs. Additionally, the asthma controller measure could not be evaluated because plan A had only one pharmacy 216 • JAPhA • 4 9 : 2 • M a r /A p r 2009
www.j aph a. or g
The PQA process for developing and evaluating measures is very complex and evolving, as illustrated by this report of PQA’s first pilot of the starter set of concept measures. The process experienced by PQA parallels that of other organizations involved in measure development and endorsement activities.4,5 The process consisted of obtaining expert guidance in selecting measure concepts that are meaningful, based on sound clinical evidence, and that represent gaps in clinical quality care. The selected process also facilitates development of precise measure specifications and algorithms to enable apples-to-apples comparisons. Moreover, use of relevant data sets informs decisions regarding whether the measures would be useful and feasible for future implementation. Further validation of specific proxy algorithms, such as enrollment criteria and diagnosis information, are planned for future PQA demonstration projects. The ideal performance measure is relevant, scientifically sound, feasible, and usable for quality improvement.5 The PQA membership and the numerous workgroups that formulated the measure concepts determined the relevance of the pharmacy performance measures. Future PQA demonstration projects will test the measures’ usability in partnership with community pharmacies. This report supports the notion that performance measures related to medication adherence may be feasible and scientifically sound because they showed variation and room for improvement for pharmacy services. Journal of the American Pharmacists Association
Performance measures
However, more work is necessary to fully evaluate the scientific soundness of these measures. The greatest limitation identified in our initial work is that only a small proportion of pharmacies within a single plan’s network may be reliably evaluated. When using a minimum sample of 30 members per pharmacy, fewer than 10% of the pharmacies, even within a large plan’s network, may be evaluable. For small health plans, measures of quality at the pharmacy level may be more difficult. Plan D represented a Medicare Advantage health maintenance organization with fewer than 5,000 members. No pharmacies in plan D’s network met the minimum of 30 patients per pharmacy. Plan C (Medicaid) also had very few pharmacies that were evaluable, and thus the expense and effort of implementing a performance evaluation system may not be warranted. The problem of too few units to evaluate is not unique to pharmacy. Scholle and colleagues evaluated physician performance measures across nine health plans and found that very few primary care physicians met a sample size threshold of 30 quality events.3 However, the majority of quality events were accounted for within the small number of physicians that did meet the threshold. Thus, a single health plan may be able to evaluate the physicians that provide care to a large portion of the plan’s membership. The same pattern may be true of pharmacy measures for large, regional drug plans for which the membership is concentrated in a smaller number of pharmacies. One plausible strategy for overcoming the problem of a small number of eligible units is to aggregate provider’s data across health plans. In California, the Integrated Healthcare Association aggregates physician performance data across multiple health plans.6 Small health plans (<500,000 members) could only evaluate 16% of their physician groups. But after creating an aggregated dataset together with other health plans, more than 70% of physician groups were evaluable. Demonstration projects supported by the federal government and by PQA are examining ways to aggregate data to enhance performance reporting.7,8 An aggregated model for performance assessment may be a viable option to increase the number of pharmacies that can be reliably evaluated since most community pharmacies serve patients enrolled in multiple drug plans. Another strategy to increase the number of evaluable pharmacies is to combine similar performance measures into a composite measure.9 For example, a composite measure of persistence could include the compilation of results from individual measures of persistence within targeted therapeutic classes. Thus, if a pharmacy had 10 patients each within the measures of persistence for beta-blockers, calcium-channel blockers, ACEI/ARBs, and lipid-lowering drugs, a composite measure could include 40 patients across the four classes. One drawback to this method is that the composite measure may be less useful for quality improvement because the pharmacy could not identify whether problems with persistence are limited to a single therapeutic class or across all four classes. Journal of the American Pharmacists Association
special feature
An entirely different approach could be used to overcome the same problem. Pharmacies could calculate their performance on each measure and report the results to an independent entity. An external auditor could validate the report’s accuracy at the pharmacy’s expense. This would be analogous to the approach used by The Joint Commission when evaluating hospital quality and by NCQA when evaluating health plan quality. The cost of gathering data, calculating and reporting their performance, and auditing the results would be a considerable burden for pharmacies that lack the resources of a national chain. Nonetheless, a pharmacy that seeks to continuously improve the quality of its services will need to invest in performance measurement. Although quantitative benchmarks of pharmacy quality are not widely available, the results for some of the performance measures (Table 4) can be compared with reports from other sectors. The rates of medication adherence, as measured by PDC and gaps in therapy, seem relatively consistent with the extant literature across the various drug classes.10–12 For example, measures of high-risk medication use in the elderly have been included in HEDIS for Medicare health plans. Estimates for plan A and plan B pharmacies were inbetween the 2007 HEDIS rates for this measure.13 This study is not the first to report quality improvement efforts in community pharmacies. Other researchers and leaders have described ways to build quality improvement systems for pharmacies.14–18 The endeavors of PQA differ in that they sought to identify and test a core set of performance measures that could be widely implemented and that could generate comparative information on community pharmacies. The Secretary of Health and Human Services has called for greater transparency in health care.19 This first initiative by the PQA and its stakeholders, including the American Pharmacists Association, moves us one step toward transparent goals for measuring the quality of pharmacy services.
Limitations This pilot study of pharmacy quality measures has several limitations. The three participating health plans were located in the same geographic region (northeastern United States), and national coverage was represented by only one PDP. Thus, conclusions cannot be drawn about the quality of pharmacies nationwide. The quality measures were based solely on prescription drug claims data from health/drug plans; therefore, the measures did not include medications for which no drug claim was submitted for payment (e.g., cash purchases, over-the-counter medications, injectable medications billed through Medicare Part B). In addition, the diagnosis of a patient could not be confirmed for the measures intended to focus on persons with asthma or diabetes, although the inclusion criteria were constructed to minimize the likelihood of misidentifying the clinical condition. Important limitations were associated with the size of the pharmacy samples. Within an individual health plan, a small number of pharmacies provide the majority of prescriptions while a very large number of pharmacies dispense a smaller www. japh a. or g
M a r /A p r 20 0 9 • 4 9 : 2 •
JAPhA • 217
special feature
PERFORMANCE MEASURES
Table 4. Distribution of pharmacy performance measuresa 10th percentileb % High-risk medications in the elderly One or more Two or more PDC beta-blocker PDC ACEI/ARB PDC calcium-channel blocker PDC dyslipidemia PDC diabetes Biguanides Sulfonylureas Thiazolidinediones Gap beta-blocker Gap ACEI/ARB Gap calcium-channel blocker Gap dyslipidemia Gap diabetes Biguanides Sulfonylureas Thiazolidinediones Diabetes: ACEI/ARB Diabetes medication dosing Biguanides Sulfonylureas Thiazolidinediones Suboptimal asthma control Absence of asthma controllerd
Plan A Median %
90th percentilec %
10th percentileb %
Plan B Median %
90th percentilec %
21.4 13.2
15.3 8.6
8.8 4.4
45.7 33.8
32.3 22.0
16.2 9.6
73.5 75.6 77.7 66.7
80.5 82.7 85.8 75.0
86.5 88.9 92.3 81.7
69.2 72.3 75.0 68.1
79.5 81.8 84.0 78.4
89.1 90.4 92.1 88.4
56.0 60.7 57.1 31.4 28.4 26.6 36.4
69.2 75.7 75.0 23.8 21.0 17.9 28.4
81.0 87.1 92.3 17.1 14.9 10.8 22.2
58.3 63.3 60.0 36.7 34.7 31.4 39.5
73.7 78.6 76.9 26.3 25.0 22.0 29.4
87.5 90.9 91.7 16.6 15.6 13.2 18.8
41.2 40.0 42.9 39.3
30.4 26.7 25.0 26.5
20.0 14.3 10.0 15.6
42.1 40.0 43.5 40.4
29.2 25.8 27.3 26.5
15.8 13.3 13.0 14.0
4.9 4.5 20.0 8.8 —
0 0 7.7 3.0 —
0 0 0 0 —
5.6 7.1 18.8 NA NA
0 0 6.7 NA NA
0 0 0 NA NA
Abbreviations used: ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin II receptor blocker; Gap, gap in therapy; NA, not applicable (asthma measures were not tested in Plan B because it was a Medicare prescription drug plan); PDC, proportion of days covered. a Cells show the percent of patients at a pharmacy who met the numerator threshold for each measure. Only pharmacies with 30 eligible members per measure were included in this analysis. b The 10th percentile column shows the demarcation for the bottom 10% of pharmacies. c The 90th percentile column shows the demarcation for the top 10% of pharmacies. d Too few pharmacies met the sample size criteria for a reliable estimate to be calculated.
quantity. Because measurement of quality was limited only to those pharmacies with a sufficient number of prescription events, quality measures apply to only a minority of participating pharmacies in a health plan’s panel. A final limitation is the assumption that the pharmacy personnel can influence the scores on these measures and that the measures therefore reflect the quality of a pharmacy. This important assumption will be tested in Phase II of the PQA demonstration projects, but the results of these projects will not be available until 2010 at the earliest.
mentation in a more “real-world” setting, along with the validity of certain proxy algorithms, will be tested. Given the challenge of obtaining sufficient denominator sizes, strategies for aggregating pharmacy performance data across health and drug plans will be crucial for generating performance reports that provide reliable information about the majority of community pharmacies. 1.
Johnson J, Bootman JL. Drug-related morbidity and mortality. Arch Intern Med. 1995; 155:1949–55.
conclusion
2.
Based on the results of this analysis, some measures for evaluating pharmacy performance appear to show variation between and room for improvement within pharmacies. The viability of these measures will be further evaluated through their use in PQA demonstration projects, where their imple-
Johnson J, Bootman JL. Drug-related morbidity and mortality and the economic impact of pharmaceutical care. Am J Health Syst Pharm. 1997;54:554–8.
3.
Scholle SH, Roski J, Dunn DL, et al. Availability of data for measuring physician quality performance. Am J Manag Care. 2009;15:67–72.
218 • JAPhA • 4 9 : 2 • M a r /a p r 2009
www.j aph a. or g
references
Journal of the American Pharmacists Association
PERFORMANCE MEASURES
special feature
4.
National Committee for Quality Assurance (NCQA). Measure Development Process. Accessed at www.ncqa.org/tabid/414/ Default.aspx, January 14, 2009.
12. Rozenfeld Y, Hunt JL, Plauschinat C, Wong KS. Oral antidiabetic medication adherence and glycemic control in managed care. Am J Manag Care. 2008;14,71–5.
5.
National Quality Forum. Measure evaluation criteria. Accessed at www.qualityforum.org/about/leadership/measure_evaluation.asp, January 14, 2009.
13. National Committee for Quality Assurance (NCQA). State of health care quality 2007. Accessed at www.ncqa.org, January 14, 2009.
6.
Integrated Healthcare Association. Advancing quality through collaboration: the California Pay-for-Performance Program. Accessed at www.iha.org/wp020606.pdf, January 14, 2009.
14. Hepler CD, Segal R. Preventing medication errors and improving drug therapy. Boca Raton, FL: CRC Press; 2003.
7.
Centers for Medicare and Medicaid Services. Better quality information. Accessed at www.cms.hhs.gov/BQI/, January 14, 2009.
8.
9.
Pharmacy Quality Alliance (PQA). PQA announces official launch of demonstration projects. Accessed at www.pqaalliance.org/files/PQALaunchofDemoProjectsJuly2008.pdf, January 14, 2009. Scholle SH, Roski J, Adams JL, et al. Benchmarking physician performance: reliability of individual and composite measures. Am J Manag Care. 2008;14:829–38.
10. Chernew ME, Shah MR, Weigh A, et al. Impact of decreasing copayments on medication adherence in a disease-management environment. Health Aff. 2008;27:103–12. 11. Nau DP, Steinke DT, Williams LK, et al. Adherence analysis using a visual analog scale versus claims-based estimation. Ann Pharmacother. 2007;41:1792–7.
15. Angaran DM. Selecting, developing, and evaluating indicators. Am J Hosp Pharm. 1991;48:1931–7. 16. Isetts BJ, Brown LM, Schondelmeyer SW, Lenarz LA. Quality assessment of a collaborative approach for decreasing drugrelated morbidity and achieving therapeutic goals. Arch Intern Med. 2003;163:1813–20. 17. Farris KB, Kirking DM. Assessing the quality of pharmaceutical care. II. Application of concepts of quality assessment from medical care. Ann Pharmacother. 1993;27:215–23. 18. Fry RN, Avey SG. Framework for pharmacy services quality improvement—a bridge to cross the quality chasm. Part I. The opportunity and the tool. J Manag Care Pharm. 2004;10:60–78.
19. Department of Health and Human Services. Building a valuebased health care system. Accessed at www.hhs.gov/valuedriven/, January 11, 2009.
Certified Pharmacy Technician trained
tested trusted
Continuous Testing Begins April 1, 2009 The PTCB Certification Program is the only certification endorsed by the American Pharmacists Association (APhA), the American Society of Health-System Pharmacists (ASHP), the National Association of Boards J oof u r nPharmacy a l o f t h e A m(NABP) e r i c a n P hand a r m a cother i s t s A s professional sociation pharmacy In 2008, PTCB tested 50,000 www.organizations. japh a. or g M a r /a p r 20 0 9 • 4 9 : 2over • JAPhA • 219 pharmacy technicians and has certified over 330,000 CPhTs since 1995.