An International Environmental Scan of Quality Indicators for Cardiovascular Care

Canadian Journal of Cardiology 28 (2012) 110 –118

Review

An International Environmental Scan of Quality Indicators for Cardiovascular Care Lusine Abrahamyan, MD, PhD, MPH,a Nicole Boom, MSc,a Linda R. Donovan, BScN, MBA,a and Jack V. Tu, MD, PhD;a,b,c,d for the Canadian Cardiovascular Society Quality Indicators Steering Committee* a b

c d

Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada

Division of Cardiology, Schulich Heart Centre, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada Department of Medicine, University of Toronto, Toronto, Ontario, Canada

Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada

ABSTRACT

RÉSUMÉ

Quality indicators (QIs) are increasingly being used to measure and improve the quality of cardiac care. We conducted an international environmental scan to identify and critically appraise published QI development initiatives addressing cardiovascular disease (CVD). A review of the peer-reviewed and grey English-language literature was conducted to identify published CVD QI development initiatives. The quality of identified studies was assessed using a modified version of the Appraisal of Guidelines for Research and Evaluation (AGREE) II QI tool—an instrument originally developed for the assessment of the quality of clinical practice guidelines. An initial literature search identified 2314 potentially relevant abstracts of peer-reviewed articles. After a review of the abstracts, 120 full text articles were retrieved and reviewed. Of these, 20 articles and 1 peer-reviewed monograph were selected for critical appraisal (n ⫽ 21). Most of the initiatives were conducted in North America (76%) and were published after 2005 (62%). The majority (5 of 6) of the AGREE II QI domain scores were skewed toward higher values, including the median score for the ‘overall quality’ rating (83.3%). Of the CVD categories addressed within the 21 initiatives, heart failure was the most common (n ⫽ 10 QI indicator sets), followed by acute coronary syndromes (n ⫽ 8). Considerable

Les indicateurs de la qualité (IQ) sont de plus en plus utilisés pour mesurer et améliorer la qualité des soins cardiaques. Nous avons mené une analyse environnementale internationale pour définir et évaluer de manière critique les publications sur les initiatives de développement des IQ portant sur la maladie cardiovasculaire (MCV). Une revue de la littérature grise anglophone examinée par des pairs a été menée pour définir les publications sur les initiatives de développement des IQ portant sur les MCV. La qualité de ces études a été évaluée en utilisant une version modifiée de l’outil d’IQ Appraisal of Guidelines for Research and Evaluation (AGREE) II – un instrument conçu au départ pour l’évaluation de la qualité des lignes directrices de la pratique clinique. Une recherche initiale de littérature a déterminé 2 314 résumés potentiellement pertinents d’articles examinés par des pairs. Après une revue des résumés, 120 articles en texte intégral ont été extraits et revus. Parmi ceux-ci, 20 articles et 1 monographie examinée par des pairs ont été sélectionnés pour une évaluation critique (n ⫽ 21). La plupart des initiatives ont été menées en Amérique du Nord (76 %) et ont été publiées après 2005 (62 %). La majorité (5 de 6) des scores du domaine des IQ de AGREE II ont été faussées par des valeurs plus élevées, incluant le score médian pour la

Despite the availability of multiple evidence-based guidelines for the prevention and treatment of cardiovascular disease (CVD), a large practice gap exists between optimal and actual patterns of cardiac care.1–3 Increasingly, quality indicators

(QIs) or performance measures are being used as tools to assess adherence to practice guidelines in routine clinical care and serve as the foundation of many quality improvement initiatives.4 QIs are typically developed to address clinical practices that have been identified as needing improvement, where significant variation in performance exists, and where evidence indicates that specific interventions can improve the quality of care.5,6 Clinical practice guidelines often serve as the basis for the development of QIs; however, they are distinct from one another. Guidelines are systematically developed statements designed to help clinicians make decisions about appropriate

Received for publication July 11, 2011. Accepted September 19, 2011. *See Appendix I for a list of Quality Indicators Steering Committee members. Corresponding author: Dr Jack V. Tu, Institute for Clinical Evaluative Sciences, 2075 Bayview Avenue – G250, Toronto, Ontario M4N 3M5, Canada. Tel.: ⫹1-416-480-4700; fax: ⫹1-416-480-6048. E-mail: [email protected] See page 116 for disclosure information.

0828-282X/$ – see front matter © 2012 Canadian Cardiovascular Society. Published by Elsevier Inc. All rights reserved. doi:10.1016/j.cjca.2011.09.019

Abrahamyan et al. Quality of Cardiovascular QIs

111

variation was observed in the methods utilized and the degree of scientific rigour applied in the published international CVD QI development initiatives. Adoption of standardized methods could help improve the quality of QI development initiatives.

notation de la « qualité globale » (83,3 %). Des catégories de MCV dont il est question dans les 21 initiatives, l’insuffisance cardiaque a été la plus fréquente (ensembles d’indicateurs IQ, n ⫽ 10), suivie par les syndromes coronariens aigus (n ⫽ 8). Une variation importante a été observée dans les méthodes utilisées et le degré de rigueur scientifique appliquée aux publications des initiatives internationales de développement des IQ portant sur la MCV. L’adoption de méthodes standardisées aiderait à améliorer la qualité des initiatives de développement des IQ.

health care interventions for specific circumstances.4 They are comprehensive (aiming to cover all aspects of care for a clinical condition); prescriptive (intending to influence provider behaviour prospectively at the level of the individual patient); and flexible (leaving room for physician judgement and interpretation).7–9 In comparison, QIs are more precise as they apply to specific clinical circumstances for which the evidence is so strong that failure to perform the actions inherent in the QIs reduces the likelihood of optimal patient outcomes.4,7,9 QIs should reflect explicit actions performed for clearly specified and easily identified patients.10 Moreover, QIs must be applied correctly with an understanding of their limitations and the potential adverse consequences of inaccurate measurement and reporting.11 Data generated from these measures can be used for multiple purposes including assessing past performance, comparing performance across organizations, identifying suboptimal practices, and targeting areas for future improvement.6,12 The results may be distributed confidentially to those best situated to implement change such as providers and health care administrators.13 Alternatively, publicly released QI reports may also drive improvement and enhance public accountability of providers and institutions.14 Although there is increasing interest in many countries around the world in assessing the quality of cardiac care using QIs, there are no published systematic reviews or critical appraisal instruments for assessing the QI literature. In 2010, the Canadian Cardiovascular Society (CCS) received funding from the Public Health Agency of Canada (PHAC) to lead development of a set of pan-Canadian QIs for measuring the quality of cardiac care in Canada. It was recognized at the outset that the CCS initiative should build upon previously published Canadian and international QI development initiatives and lessons learned. Accordingly, a team of researchers with experience in developing QIs was commissioned to conduct an environmental scan of the English-language scientific literature on previously published CVD QI development initiatives. The environmental scan, initially focused on Canadian initiatives, was subsequently expanded to include major international initiatives. The ultimate goal of this initiative is to assist the CCS to identify a set of best practices for developing QIs that could be used to develop future pan-Canadian cardiac QIs. This report presents a summary of the key methods and results of the environmental scan of published cardiac QI initiatives.

initiatives; and (3) preparation of a summary of the published QI’s by disease category.

Methods The environmental scan involved 3 main steps: (1) a search of the peer-reviewed and grey literature for CVD QI development initiatives; (2) a critical appraisal of the identified QI

Search strategy—peer-reviewed literature A systematic literature search was conducted using Ovid MEDLINE and EMBASE databases, from 1996 to 2010, restricted to English articles of human studies. To identify studies related to CVD, we used database-appropriate terms for the following clinical categories: coronary artery disease, acute coronary syndrome, acute myocardial infarction, coronary artery bypass grafting, percutaneous coronary intervention, heart failure, heart valve disease, heart transplantation, peripheral arterial disease, atrial fibrillation, and echocardiography. To identify studies related to QI development the following search terms were used: ‘performance indicator,’ ‘quality indicator,’ ‘performance measure,’ ‘quality measure,’ ‘report card,’ ‘benchmark,’ and ‘quality assurance.’ The results from these 2 search steps were then combined. The following terms were deemed out of scope and excluded from the search process: ‘access to care,’ ‘primary care,’ and ‘patient safety,’ as well as those related to stroke. One of the authors (L.A.) applied the defined search strategy, reviewed all selected abstracts for relevance, and retrieved the full articles of all relevant abstracts. Additionally, references of selected articles were screened for other potentially eligible studies. Eligibility of the full text articles for inclusion in the critical appraisal was assessed by the study team. In some instances, multiple articles were found, for example where a set of QIs had been updated. In these instances, only the most current article and indicator set were abstracted. The following information was abstracted for all articles included in the scan: author, journal, year of publication, name of organization and/or initiative, country of origin, and the final list(s) of indicators. Search strategy— grey literature A search of the grey English literature using the ‘Google’ internet search engine was conducted, using the same terms as for the peer-reviewed literature search but looking for other relevant reports and working papers from various organizations or research groups, books, and unpublished material. In addition, we searched publicly available quality indicator/measure repositories, such as the National Quality Measures Clearinghouse (NQMC) (http://www.qualitymeasures.ahrq.gov), the National Quality Forum (NQF) (http://www.qualityforum. org) from the United States (US), and web sites of major organizations with a known interest in evaluating the quality of cardiac care. This step also involved a review of websites of organizations identified from peer-reviewed publications. We

112

abstracted the same variables for both the grey and peer-reviewed literature publications. Exclusion criteria To ensure a focus on the most pertinent publications we applied 8 exclusion criteria to identified QI initiatives (see Supplemental Appendix S1). Critical appraisal of the quality of QI development initiatives Due to the considerable heterogeneity observed in the retrieved articles, the study team identified a need for a consistent appraisal method. After some discussion and a search of the peer-reviewed literature, we identified the Appraisal of Guidelines for Research and Evaluation (AGREE) II tool—a validated instrument that provides a framework for evaluating the quality of clinical practice guidelines.15–17 The use of this tool was felt to be reasonable given the lack of a specific appraisal tool for QIs and the fact that the cardiac QI development initiatives under review were mostly derived from clinical practice guidelines. The original AGREE instrument, published in 2003, was developed by a group of international guideline developers and researchers—the AGREE Collaboration.18 Since then it has been translated into 13 languages and applied in a variety of clinical settings. In 2010, the AGREE Next Steps Consortium published the revised AGREE II instrument,15 along with a user manual available at http://www.agreetrust.org. The AGREE II instrument is a valid and reliable instrument, consisting of 6 domains, 23 items, and an overall guideline assessment score. Each domain consists of a set of relevant items, where each item is rated on a 7-point scale ranging from ‘strongly disagree’ to ‘strongly agree.’ The AGREE II instrument was modified, by the study team, creating AGREE II QI, for use in evaluating the quality of the QI initiatives. The resulting AGREE II QI tool consists of 6 domains (similar to AGREE II QI) with 16 items and an overall quality score, with a similar 7-point rating scale and domain score calculation (see Supplemental Appendix S2). The first domain ‘scope and purpose’ has 2 items and evaluates the description and clarity of a quality initiative’s scope, overall objectives, and target population. The ‘stakeholder involvement’ domain with 2 items evaluates the level of stakeholder involvement in the QI development process and alignment with the study’s overall purpose. The ‘rigour of development’ domain has the largest number of items (n ⫽ 7) and addresses key steps in the QI development process. The first item assesses whether a systematic literature review was conducted, focusing on specific details of the search process. The second item evaluates whether explicit criteria were used for selecting QIs (eg, validity, importance) and, if yes, whether these criteria were clearly described. The next 2 items evaluate the methods used for formulating the QIs and if a predefined quantitative process was used for indicator selection. The final 3 items of this domain assess whether supporting evidence accompanied the selected indicators; whether they were reviewed by experts or end-users prior to publication, and if a process to facilitate updating of the indicators was specified. While many of the items in AGREE II QI are closely aligned with those of the original AGREE II instrument, item 6 (“Use of explicit criteria for the selection of the initial list of indica-

Canadian Journal of Cardiology Volume 28 2012

tors”) and item 8 (“Use of a predefined quantitative process for the selection of the final list of indicators”) are specific to the QI development process. The ‘clarity of presentation’ domain assesses how well the QIs are defined, in the final set, in terms of specifying an indicator’s numerator, denominator, and related inclusion and exclusion criteria. The domain ‘applicability’ evaluates whether the QIs are supported with tools and if potential implementation barriers are discussed. Both factors are important for the uptake of indicators by the target users. The domain ‘editorial independence’ consists of 2 items related to the funding source as well as reporting and management of conflicts of interest among the QI development team. The final component of the AGREE II QI instrument, ‘rate the overall quality of this initiative,’ represents a global rating of the overall quality of the QI initiative and is calculated in the same way as the other domain scores. The AGREE II QI instrument was pilot-tested on a set of 5 QI development publications (the first 5 identified by the literature search) by 3 appraisers (L.A., N.B., L.R.D.), and then updated. Next, 2 reviewers (L.A. and N.B.) independently appraised all selected initiatives using AGREE II QI. The resulting item scores were entered into an Excel spreadsheet (Excel 2007; Microsoft, Redmond, WA), and domain scores were calculated according to the AGREE II methods, as described in the user’s manual. After the critical appraisal, QIs were abstracted from the articles and presented according to Donabedian’s ‘structureprocess-outcome’ framework.19 Process QIs, although not always mutually exclusive, were further categorized as ‘therapeutics,’ ‘other interventions,’ ‘documentation of care,’ ‘investigations/therapeutic monitoring,’ and ‘education and follow-up care.’ Indicators that measured the same aspects of care were combined.

Results Search results—peer-reviewed literature The systematic literature review identified 2314 abstracts from MEDLINE and EMBASE. After the review of abstracts, 2194 were excluded and 120 were deemed potentially relevant (Fig. 1). The full texts of these abstracts were obtained for the review. Of these, 20 articles20 –39 and 1 peer-reviewed monograph7 (identified from the screening of references) met the criteria for evaluation and were critically appraised using the AGREE II QI tool. The majority of excluded abstracts focused on the application of QIs or the impact of quality improvement activities on care processes and patient outcomes as opposed to QI development. One initiative by Thomas et al., published in 2007, described the development of QIs for cardiac rehabilitation.30 A subsequent publication in 2010 updated the specifications for 2 indicators in the set.40 Only the first publication30 was included in our critical appraisal. For the Assessing Care for Vulnerable Elders (ACOVE) initiative, we included the publication by Wenger at al.,39 describing the QI development methodology in the critical appraisal, and content from related articles about the specific QIs41– 44 within the relevant tables of indicators by CVD disease category.

Abrahamyan et al. Quality of Cardiovascular QIs

113

Abstracts iden
fied by search strategy n = 2314 Excluded aer review of abstracts n = 2194

(OECD). A majority (62%, n ⫽ 13) of the 21 QI initiatives were published between 2006 and 2010, while the remaining 8 were published between 2000 and 2005. Search results— grey literature Several initiatives and indicator sets were identified from the grey literature search; however, we observed a high degree of heterogeneity in QI development and reporting methods. For example, some initiatives described only the final list of indicators; others described how they had modified their QI development methods over time. A lack of information made it difficult to apply the AGREE II QI tool consistently across the publications retrieved from the grey literature. Many of these lacked comprehensive information about the methodologies used, and considerable effort was expended trying to locate required information for various components of the AGREE II QI instrument. Due to time and resource constraints, we did not contact the authors for additional information and did not include these initiatives in the critical appraisal.

Full-text ar
cles obtained for more detailed review n = 120 Excluded aer review of full texts n = 100 Ar
cles included in the cri
cal appraisal by AGREE II QI n = 20 ar
cles and 1 monograph Figure 1. Flow diagram of selection of published quality indicator (QI) development initiatives. AGREE, Appraisal of Guidelines for Research and Evaluation.

Quality of the QI development initiatives in the peer-reviewed literature

The majority of the initiatives included in the final set (16/ 21) were conducted in North America—10 in the US and 6 in Canada (Table 1). Of the remaining 5 initiatives, 2 were conducted in Australia, 1 in The Netherlands, 1 jointly represented 9 European Union countries and 1 represented the Organisation for Economic Cooperation and Development

All QI development initiatives identified in the peer-reviewed literature were critically appraised using the AGREE II QI tool. Summary statistics of domain scores and the overall quality assessment scores are presented in Table 2. Scores for 5 out of the tool’s 6 domains were skewed toward higher values. For example, the median score for the domain ‘scope and purpose’ was 100%. The lowest median score, 41.7%, was ob-

Table 1. Description of QI development initiatives in the peer-reviewed literature (n ⫽ 21) First author 20

Bonow Brand21 Burge22 Campbell23 Estes24 Guru25 Hickey26 Idanpaan-Heikkila27 Kerr7 Ko28 Krumholz29 Lee31 Lindsay32 Masoudi33 Olin34 Redberg35 Saliba36 Vermeulen37 Thomas30 Tu38 Wenger39

Year

Organization/initiative

Country

2005 2009 2007 2008 2008 2005 2004 2006 2000 2008 2008 2003 2002 2000 2010 2009 2005 2008 2007 2008 2007

ACC/AHA — CCORT EPA Cardio Project ACC/AHA/PCPI CCORT BCC OECD HCQI Project RAND CCORT ACC/AHA CCORT/CCS HRRC HCFA NHF ACCF/AHA/ACR/SCAI/SIR/SVM/SVN/SVS ACCF/AHA RAND — AACVPR/ACC/AHA CCORT/CCS RAND/ACOVE

United States Australia Canada European Union* United States Canada Australia OECD United States Canada United States Canada Canada United States United States United States United States The Netherlands United States Canada United States

AACVPR, American Association of Cardiovascular and Pulmonary Rehabilitation; ACC, American College of Cardiology; ACCF, American College of Cardiology Foundation; ACOVE, Assessing the Care of Vulnerable Elders Project; ACR, American College of Radiology; AHA, American Heart Association; BCC, Brisbane Cardiac Consortium; CCORT, Canadian Cardiovascular Outcomes Research Team; CCS, Canadian Cardiovascular Society; EPA, European Practice Assessment; HCFA NHF, Health Care Financing Administration National Heart Failure Quality Improvement Project; HCQI, Health Care Quality Indicators Project; HRRC, Hospital Report Research Collaborative; OECD, Organisation for Economic Co-operation and Development; PCPI, Physician Consortium for Performance Improvement; QI, quality indicator; RAND, Research and Development; SCAI, Society for Cardiac Angiography and Interventions; SIR, Society for Interventional Radiology; SVM, Society for Vascular Medicine; SVN, Society for Vascular Nursing; SVS, Society for Vascular Surgery. * Nine European countries participated in the panel: Austria, Belgium, Finland, France, Germany, The Netherlands, Slovenia, United Kingdom, and Switzerland.

114

Canadian Journal of Cardiology Volume 28 2012

Table 2. AGREE II QI domain summary statistics (n ⫽ 21 initiatives) Domain Scope and purpose Stakeholder involvement Rigour of development Clarity of presentation Applicability Editorial independence Overall quality

Interquartile range (%)

Median (%)

Domain correlation* with the ‘overall quality’

87.5-100 83.3-95.8 58.3-83.3 58.3-100 33.3-83.3 20.8-100 66.7-83.3

100 87.5 71.4 83.3 41.7 66.7 83.3

0.671 0.517 0.852 0.866 0.349 0.639 —

AGREE, Appraisal of Guidelines for Research and Evaluation; QI, quality indicator. * Numbers represent Spearman correlation coefficients. Except the coefficient for ‘applicability’, all coefficients have P values ⬍ 0.05.

served for ‘applicability’ domain. The largest interquartile range was observed for ‘editorial independence’ (21%-100%). The ‘overall quality’ scores ranged from 25% to 100% across the 21 studies, with a median score of 83.3% (Fig. 2). Eleven of the 21 studies (52%) had ‘overall quality’ scores above 80%. In addition, significant correlations were observed between ‘overall quality’ scores and 5 of the 6 domain scores (Table 2) with the highest correlation coefficients for ‘clarity of presentation’ (Spearman’s ␳ ⫽ 0.866) and the lowest for ‘applicability’ (␳ ⫽ 0.349). QIs by CVD category Table 3 presents the 21 QI development initiatives sorted according to 10 CVD categories. Of these, heart failure (HF) had the greatest number of QI sets (n ⫽ 10) followed by acute coronary syndrome (ACS) (n ⫽ 8). In Supplemental Appendix S3, Supplemental Tables S1-S11, provide high level summaries of individual QIs for each CVD category. The number of unique indicators within a single QI development initiative ranged from 2 to 52. Readers are encouraged to refer to the original articles for complete indicator specifications.

Discussion The need for the development and application of a systematic, rigourous process for QI development and reporting is well recognized by the scientific community.4,6,11,45 The aim of this review was to identify and critically appraise published international CVD QI development initiatives to assist the CCS in its efforts to establish best practices for pan-Canadian cardiac QIs and to build upon pre-existing indicator development work. This study represents the first application of AGREE II QI, a modification of the AGREE II instrument originally developed to evaluate the quality of clinical practice guidelines, to critically appraise the quality of QI development initiatives. AGREE II QI domains reflect all 4 stages of indicator development described by Adair et al. including: “(a) conceptualization, (b) selection and/or development of measures, (c) data collection and processing, and (d) reporting and using results.”11 All 21 QI development initiatives identified were critically appraised using this tool. We found that most of these studies were conducted in North America, published within the past 5 years and described indicators developed for application in local, regional, national, or international evaluations

Figure 2. Ranking of the ‘overall quality’ assessment scores by Appraisal of Guidelines for Research and Evaluation (AGREE) II quality indicator (QI) instrument.

Abrahamyan et al. Quality of Cardiovascular QIs

115

Table 3. Quality indicators by CVD category* CVD categories Prev

IHD

ACS

✓ ✓

✓

✓

✓ ✓ ✓ ✓ ✓

✓

3

PCI

CABG

AF

PAD

✓ ✓ ✓

20

Bonow Brand21 Burge22 Campbell23 Estes24 Guru25 Hickey26 Idanpaan-Heikkila27 Kerr7 Ko28 Krumholz29 Lee31 Lindsay32 Masoudi33 Olin34 Redberg35 Saliba36 Vermeulen37 Thomas30 Tu38 Wenger39 Total number of QI sets

HF

✓

✓

✓ 4

✓ ✓ 8

✓ ✓ ✓

✓ ✓

✓

✓

HTN

HLP

✓

✓

✓

✓

CR

✓

✓

✓ ✓ ✓

✓ 10

✓ ✓

✓

3

✓ ✓

2

✓ 4

1

✓ 4

2

1

ACS, acute coronary syndrome (includes indicators for ST-elevation myocardial infarction, non–ST-elevation myocardial infarction, and unstable angina); AF, atrial fibrillation and atrial flatter; CABG, coronary artery bypass grafting; CR, cardiac rehabilitation; CVD, cardiovascular disease; HF, heart failure; HLP, hyperlipidemia; HTN, hypertension; IHD, ischemic heart disease (includes indicators for stable angina); PAD, peripheral arterial disease; PCI, percutaneous coronary intervention and percutaneous transluminal coronary angioplasty; Prev, primary prevention and management of CVD. * To summarize indicators under distinct CVD categories, some of the original indicator sets were regrouped.

of quality of care. Significant heterogeneity was observed among the 21 studies, in terms of various aspects of QI development including project scope, development methods, scientific rigour, application of the indicators, and overall reporting strategies. AGREE II QI domain scores The ‘scope and purpose’ domain had the highest median score; however, the initiatives varied in terms of overall scope and purpose. For example, the objective of the studies from the American College of Cardiology (ACC)/American Heart Association (AHA) were to develop performance measures, from a provider’s perspective, for HF,20 atrial fibrillation,24 and acute myocardial infarction,29 in order to evaluate the quality of care and to identify opportunities for improvement. The OECD Health Care Quality Indicators Project27 and the European Practice Assessment (EPA) Cardio Project23 developed cardiac QIs to enable international comparisons of the performance of health care systems. Other studies focused on developing QIs for local quality improvement activities.21,26 In the ‘rigour of development’ domain, while some studies did not provide a clear description of the search process, many others described a literature search for relevant peer-reviewed studies and clinical guidelines complemented with an evaluation of the strength of evidence. The majority of the initiatives had explicit indicator selection criteria; however, the criteria and their definitions varied greatly across the different initiatives. Summaries of indicator selection criteria have been published previously elsewhere.11,39,46 The most common method used to formulate QIs was the Research and Development (RAND)/University of California at Los Angeles appropriateness method (n ⫽ 9 initiatives), also known as mod-

ified or 2-stage Delphi panel method.7,22,25,27,28,31,32,35,36,39,40 A unique QI formulation method was applied across all 6 initiatives led by the ACC/AHA Task Force on Performance Measures.20,24,29,30,34,35 Although the majority of the initiatives reviewed received high scores for the ‘clarity of presentation’ domain, information was presented in various formats; some initiatives reported only the numerator and denominator, while others only provided a list of indicators. The scores for the ‘applicability’ domain ranged from 17% to 92% depending on the initiatives’ efforts to support application of the QIs. For example, all performance measure sets developed by the ACC/AHA Task force on Performance Measures included a sample data collection tool and noted potential implementation challenges for each selected measure.20,24,29,30,34,35 The scores for the ‘editorial independence’ domain ranged from 0 to 100% demonstrating inconsistent reporting. For example, some studies reported the source of project funding but did not specify whether the funder played a role in the development or influenced the content of the QIs. Many studies did not report competing interests. All QI development initiatives led by the ACC/AHA Task Force on Performance Measures reported funding sources, required members to recuse themselves from voting on measures involving conflicts of interest, and published conflict of interest information of all participants.24,29,30,34,35,40 A summary of the QI sets identified in the review, categorized by CVD category, was prepared to assess similarities and differences as well as change over time. The CVD category with the highest number of QI sets was HF followed by ACS which was not surprising given that selection of an area for QI development may be driven by disease burden, variation in care, and

116

the potential for quality improvement,6 all of which are significant for HF and ACS. In contrast, our review identified only 1 set of QIs for cardiac rehabilitation and 1 set for peripheral arterial disease. This review has several limitations. First, it was restricted to studies published in English, potentially omitting relevant studies in other languages. Second, although we conducted a grey literature search, it was not exhaustive and we were unable to apply the AGREE II QI tool to critically appraise the initiatives located in the grey literature due to incomplete/missing information. Further, it is possible that relevant initiatives were missed from the peer-reviewed literature search. Initiatives lacking information about the indicator development process or those providing only limited information also were excluded. It is possible that some of the information we were looking for was not included in the reviewed articles due to journal word limits. We did not contact any author/organization to elicit additional information due to time and budget constraints limiting our findings to what was available in the published articles. As this was the first application of the AGREE II QI instrument, additional studies are needed to further test and validate the instrument. There may be some concerns about a potential conflict of interest among the study team, as 1 of the authors of this report (J.V.T.) was involved in the development of the Canadian Cardiovascular Outcomes Research Team (CCORT) indicators in Canada that were included in the 21 initiatives reviewed. However, the assessment of the quality of the initiatives included in this report was conducted independently by 2 other team members (L.A. and N.B.), neither of whom had any direct involvement in the CCORT quality initiatives under review. In conclusion, our review found that a number of organizations have developed QIs for CVD. While many QI initiatives used rigourous and evidence-based methods, we observed significant heterogeneity in terms of methodological approaches and level of scientific rigour. The results of this review can assist the CCS and other organizations pursuing QI development and quality improvement activities by informing them about currently available indicators and options for QI development approaches. Considering the lack of uniform standards for QI development and reporting, the AGREE II QI instrument may potentially serve as a methodological framework to guide both the QI development and related reporting. A more rigourous approach to development and reporting of QIs will improve transparency, strengthen the quality of the indicators developed, and ultimately help to improve the quality of patient care.

Acknowledgements The views expressed herein do not necessarily represent the views of the PHAC or the CCS.

Funding Sources Production of these materials has been made possible through a financial contribution from the PHAC and by a Canadian Institutes of Health Research Team Grant in Cardiovascular Outcomes Research to the CCORT.

Canadian Journal of Cardiology Volume 28 2012

Disclosures Dr Tu has received operating grants to develop cardiac QIs and to conduct QI-related research from the following agencies: Canadian Institutes of Health Research, Heart and Stroke Foundation of Canada, Heart and Stroke Foundation of Ontario, Public Health Agency of Canada, and Ministry of Health and Long-Term Care. He is also the Principal Investigator of the CCORT initiative, which is included in this environmental scan. Dr Abrahamyan is supported by a CCORT postdoctoral fellowship award. None of the other authors have any conflicts of interest to disclose. References 1. Fonarow GC, Yancy CW, Heywood JT. Adherence to heart failure quality-of-care indicators in US hospitals: analysis of the ADHERE Registry. Arch Intern Med 2005;165:1469-77. 2. Fox KA, Goodman SG, Klein W, et al. Management of acute coronary syndromes. Variations in practice and outcome; findings from the Global Registry of Acute Coronary Events (GRACE). Eur Heart J 2002;23:1177-89. 3. Eagle KA, Goodman SG, Avezum A, et al. Practice variation and missed opportunities for reperfusion in ST-segment-elevation myocardial infarction: findings from the Global Registry of Acute Coronary Events (GRACE). Lancet 2002;359:373-7. 4. Campbell SM, Braspenning J, Hutchinson A, Marshall M. Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care 2002;11:358-64. 5. Evans SM, Lowinger JS, Sprivulis PC, Copnell B, Cameron PA. Prioritizing quality indicator development across the healthcare system: identifying what to measure. Intern Med J 2009;39:648-54. 6. Mainz J. Developing evidence-based clinical indicators: a state of the art methods primer. Int J Qual Health Care 2003;15(suppl 1):i5-11. 7. Kerr EA, Asch SM, Hamilton EG, McGlynn EA, eds. Quality of Care for Cardiopulmonary Conditions: A Review of the Literature and Quality Indicators. RAND monograph report: MR-1282-AHRQ, 2000. 8. Fleming BB, Greenfield S, Engelgau MM, et al. The Diabetes Quality Improvement Project: moving science into health policy to gain an edge on the diabetes epidemic. Diabetes Care 2001;24:1815-20. 9. Spertus JA, Eagle KA, Krumholz HM, Mitchell KR, Normand SL. American College of Cardiology and American Heart Association methodology for the selection and creation of performance measures for quantifying the quality of cardiovascular care. J Am Coll Cardiol 2005;45:1147-56. 10. Measuring and improving quality of care: a report from the American Heart Association/American College of Cardiology First Scientific Forum on Assessment of Healthcare Quality in Cardiovascular Disease and Stroke. Circulation 2000;101:1483-93. 11. Adair CE, Simpson E, Casebeer AL, et al. Performance measurement in healthcare: part II - state of the science findings by stage of the performance measurement process. Healthcare Policy 2006;2:56-78. 12. Copnell B, Hagger V, Wilson SG, et al. Measuring the quality of hospital care: an inventory of indicators. Intern Med J 2009;39:352-60. 13. Brien SE, Dixon E, Ghali WA. Measuring and reporting on quality in health care: a framework and road map for improving care. J Surg Oncol 2009;99:462-6. 14. Rich JB. Quality indicators, performance measures, and accountability: the right thing, at the right time, for the right reason. J Thorac Cardiovasc Surg 2006;131:4-8.

Abrahamyan et al. Quality of Cardiovascular QIs 15. Brouwers MC, Kho ME, Browman GP, et al. AGREE II: Advancing guideline development, reporting and evaluation in health care. CMAJ 2010;182:E839-42. 16. Brouwers MC, Kho ME, Browman GP, et al. Development of the AGREE II, part 1: performance, usefulness and areas for improvement. CMAJ 2010;182:1045-52. 17. Brouwers MC, Kho ME, Browman GP, et al. Development of the AGREE II, part 2: assessment of validity of items and tools to support application. CMAJ 2010;182:E472-8. 18. AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003;12:18-23. 19. Donabedian A. Evaluating the quality of medical care. 1966. Milbank Q 2005;83:691-729. 20. Bonow RO, Bennett S, Casey DE Jr, et al. ACC/AHA clinical performance measures for adults with chronic heart failure: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures (Writing Committee to Develop Heart Failure Clinical Performance Measures) endorsed by the Heart Failure Society of America. J Am Coll Cardiol 2005;46:1144-78. 21. Brand C, Lam SK, Roberts C, et al. Measuring performance to drive improvement: development of a clinical indicator set for general medicine. Intern Med J 2009;39:361-9. 22. Burge FI, Bower K, Putnam W, Cox JL. Quality indicators for cardiovascular primary care. Can J Cardiol 2007;23:383-8. 23. Campbell SM, Ludt S, Van Lieshout J, et al. Quality indicators for the prevention and management of cardiovascular disease in primary care in nine European countries. Eur J Cardiovasc Prev Rehabil 2008;15:509-15. 24. Estes NA 3rd, Halperin JL, Calkins H, et al. ACC/AHA/Physician Consortium 2008 Clinical Performance Measures for Adults with Nonvalvular Atrial Fibrillation or Atrial Flutter: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures and the Physician Consortium for Performance Improvement (Writing Committee to Develop Clinical Performance Measures for Atrial Fibrillation) developed in Collaboration with the Heart Rhythm Society. J Am Coll Cardiol 2008;51:865-84. 25. Guru V, Anderson GM, Fremes SE, et al. The identification and development of Canadian coronary artery bypass graft surgery quality indicators. J Thorac Cardiovasc Surg 2005;130:1257. 26. Hickey A, Scott I, Denaro C, et al. Using clinical indicators in a quality improvement programme targeting cardiac care. Int J Qual Health Care 2004;16(suppl 1):i11-25. 27. Idanpaan-Heikkila UM, Lambie L, Mattke S, et al. Selecting indicators for the quality of cardiac care at the health system level in Organization for Economic Co-operation and Development countries. Int J Qual Health Care 2006;18(suppl 1):39-44. 28. Ko DT, Wijeysundera HC, Zhu X, Richards J, Tu JV. Canadian quality indicators for percutaneous coronary interventions. Can J Cardiol 2008; 24:899-903. 29. Krumholz HM, Anderson JL, Bachelder BL, et al. ACC/AHA 2008 performance measures for adults with ST-elevation and non-ST-elevation myocardial infarction: a report of the American College of Cardiology/ American Heart Association Task Force on Performance Measures (Writing Committee to Develop Performance Measures for ST-Elevation and Non-ST-Elevation Myocardial Infarction) developed in collaboration with the American Academy of Family Physicians and American College of Emergency Physicians endorsed by the American Association of Cardiovascular and Pulmonary Rehabilitation, Society for Cardiovascular

117 Angiography and Interventions, and Society of Hospital Medicine. J Am Coll Cardiol 2008;52:2046-99. 30. Thomas RJ, King M, Lui K, et al. AACVPR/ACC/AHA 2007 performance measures on cardiac rehabilitation for referral to and delivery of cardiac rehabilitation/secondary prevention services endorsed by the American College of Chest Physicians, American College of Sports Medicine, American Physical Therapy Association, Canadian Association of Cardiac Rehabilitation, European Association for Cardiovascular Prevention and Rehabilitation, Inter-American Heart Foundation, National Association of Clinical Nurse Specialists, Preventive Cardiovascular Nurses Association, and the Society of Thoracic Surgeons. J Am Coll Cardiol 2007;50:1400-33. 31. Lee DS, Tran C, Flintoft V, et al. CCORT/CCS quality indicators for congestive heart failure care. Can J Cardiol 2003;19:357-64. 32. Lindsay P, Schull M, Bronskill S, Anderson G. The development of indicators to measure the quality of clinical care in emergency departments following a modified-delphi approach. Acad Emerg Med 2002;9:1131-9. 33. Masoudi FA, Ordin DL, Delaney RJ, Krumholz HM, Havranek EP. The National Heart Failure Project: a health care financing administration initiative to improve the care of Medicare beneficiaries with heart failure. Congest Heart Fail 2000;6:337-9. 34. Olin JW, Allie DE, Belkin M, et al. ACCF/AHA/ACR/SCAI/SIR/SVM/ SVN/SVS 2010 performance measures for adults with peripheral artery disease: a report of the American College of Cardiology Foundation/ American Heart Association Task Force on Performance Measures, the American College of Radiology, the Society for Cardiac Angiography and Interventions, the Society for Interventional Radiology, the Society for Vascular Medicine, the Society for Vascular Nursing, and the Society for Vascular Surgery (Writing Committee to Develop Clinical Performance Measures for Peripheral Artery Disease). J Am Coll Cardiol 2010;56: 2147-81. 35. Redberg RF, Benjamin EJ, Bittner V, et al. ACCF/AHA 2009 performance measures for primary prevention of cardiovascular disease in adults: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Performance Measures (Writing Committee to Develop Performance Measures for Primary Prevention of Cardiovascular Disease) developed in collaboration with the American Academy of Family Physicians; American Association of Cardiovascular and Pulmonary Rehabilitation; and Preventive Cardiovascular Nurses Association: endorsed by the American College of Preventive Medicine, American College of Sports Medicine, and Society for Women’s Health Research. J Am Coll Cardiol 2009;54:1364-405. 36. Saliba D, Solomon D, Rubenstein L, et al. Quality indicators for the management of medical conditions in nursing home residents. J Am Med Dir Assoc 2005;6:S36-48. 37. Vermeulen RP, Jessurun GA, Peels HO, Jaarsma T, Zijlstra F. Clinical performance indicators for percutaneous coronary intervention. Crit Pathw Cardiol 2008;7:126-32. 38. Tu JV, Khalid L, Donovan LR, Ko DT. Indicators of quality of care for patients with acute myocardial infarction. CMAJ 2008;179:909-15. 39. Wenger NS, Roth CP, Shekelle P. Introduction to the assessing care of vulnerable elders-3 quality indicator measurement set. J Am Geriatr Soc 2007;55(suppl 2):S247-52. 40. Thomas RJ, King M, Lui K, et al. AACVPR/ACCF/AHA 2010 Update: Performance Measures on Cardiac Rehabilitation for Referral to Cardiac Rehabilitation/Secondary Prevention Services endorsed by the American College of Chest Physicians, the American College of Sports Medicine, the American Physical Therapy Association, the Canadian Association of Cardiac Rehabilitation, the Clinical Exercise Physiology Association, the European Association for Cardiovascular Prevention and Rehabilita-

118 tion, the Inter-American Heart Foundation, the National Association of Clinical Nurse Specialists, the Preventive Cardiovascular Nurses Association, and the Society of Thoracic Surgeons. J Am Coll Cardiol 2010;56:1159-67. 41. Cheng EM, Fung CH. Quality indicators for the care of stroke and atrial fibrillation in vulnerable elders. J Am Geriatr Soc 2007;55(suppl 2): S431-7. 42. Heidenreich PA, Fonarow GC. Quality indicators for the care of heart failure in vulnerable elders. J Am Geriatr Soc 2007;55(suppl 2):S340-6. 43. Min LC, Mehrotra R, Fung CH. Quality indicators for the care of hypertension in vulnerable elders. J Am Geriatr Soc 2007;55(suppl 2):S359-65. 44. Watson K, Fung CH, Budoff M. Quality indicators for the care of ischemic heart disease in vulnerable elders. J Am Geriatr Soc 2007;55(suppl 2):S366-72.

Canadian Journal of Cardiology Volume 28 2012 45. Spertus JA, Bonow RO, Chan P, et al. ACCF/AHA new insights into the methodology of performance measurement. J Am Coll Cardiol 2010;56: 1767-82. 46. National Centre for Health Outcomes Development. Evaluating the quality of clinical and health indicators. Annex 12. Extract from Compendium of Clinical and Health Indicators User Guide. Available at: https://indicators.ic.nhs.uk/download/Additional%20Reading/Methods% 20annexes/Compendium%20User%20Guide%202005%20Annex%2012. doc. Accessed October 5, 2010

Supplementary Material To access the supplementary material accompanying this article, visit the online version of the Canadian Journal of Cardiology at www.onlinecjc.ca, and at doi:10.1016/j.cjca.2011.09.019.

Appendix I. Quality Indicators Steering Committee Members 2010-2011 David E. Johnstone, MD, FRCPC, (Quality Indicators Steering Committee Chair), Mazankowski Alberta Heart Institute Jafna L. Cox, MD, FRCPC, Dalhousie University, Cardiovascular Health, Nova Scotia Karin Humphries, British Columbia Cardiac Registry Kori J. Kingsbury, CEO, Cardiac Care Network of Ontario Merril Knudtson, MD, Libin Cardiovascular Institute of Alberta Paul MacDonald, MD, Canadian Cardiovascular Society Guidelines Committee Hugh MacLeod, CEO, Canadian Patient Safety Institute Anne McFarlane, Canadian Institute for Health Information François Philippon, MD, FRCPC, President, Réseau québécois de cardiologie tertiaire Jack V. Tu, MD, PhD, FRCPC, Canadian Cardiovascular Outcomes Research Team Blair J. O’Neill, MD, FRCPC, President, Canadian Cardiovascular Society, ex-officio Charles Kerr, MD, FRCPC, Past President, Canadian Cardiovascular Society, ex-officio