- Email: [email protected]
Audit and feedback interventions to improve endoscopist performance: Principles and effectiveness
Accepted Manuscript Audit and feedback interventions to improve endoscopist performance: principles and effectiveness Jill Tinmouth, MD, PhD, FRCPC, Lead Scientist, Jigisha Patel, MSc, Research Coordinator, Robert J. Hilsden, MD, PhD, FRCPC, Associate Professor, Director of Research, Noah Ivers, MD, PhD, Assistant Professor, Diego Llovet, PhD, Behavioural Scientist PII:
S1521-6918(16)30012-9
DOI:
10.1016/j.bpg.2016.04.002
Reference:
YBEGA 1423
To appear in:
Best Practice & Research Clinical Gastroenterology
Received Date: 16 March 2016 Revised Date:
31 March 2016
Accepted Date: 7 April 2016
Please cite this article as: Tinmouth J, Patel J, Hilsden RJ, Ivers N, Llovet D, Audit and feedback interventions to improve endoscopist performance: principles and effectiveness, Best Practice & Research Clinical Gastroenterology (2016), doi: 10.1016/j.bpg.2016.04.002. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Audit and feedback interventions to improve endoscopist performance: principles and effectiveness
RI PT
Jill Tinmouth (corresponding author), MD PhD FRCPC, Lead Scientist, ColonCancerCheck Program, Cancer Care Ontario; Scientist and Staff Gastroenterologist, Sunnybrook Health Sciences Centre, Address: 2075 Bayview Avenue, Room HG40, Toronto, ON M4N 3M5 Tel: (416) 480-5910 Fax: (416) 480-4845 E-mail: [email protected]
M AN U
SC
Jigisha Patel MSc, Research Coordinator Sunnybrook Health Sciences Centre, Address: 2075 Bayview Avenue, Room HG40, Toronto, ON M4N 3M5 Tel: (416) 480-6100 x7941 Fax: (416) 480-4845 E-mail: [email protected]
TE D
Robert J. Hilsden, MD PhD FRCPC, Associate Professor, Departments of Medicine/Community Health Sciences, Cummings School of Medicine, University of Calgary; Director of Research, Forzani & MacPhail Colon Cancer Screening Center, Alberta Health Services Address: 3280 Hospital Drive NW, Room 6E17, TRW Building, Calgary, AB, CANADA T2N 4Z6 Tel: (403) 592-5089 Fax: (403) 592-5090 E-mail: [email protected]
AC C
EP
Noah Ivers, MD PhD, Assistant Professor, Department of Family and Community Medicine, University of Toronto; Scientist and Family Physician, Women's College Hospital Address: 77 Grenville Street, Toronto, ON M5S 1B3 Tel: (416) 323-6400 Fax: (416)323-6255 E-mail: [email protected] Diego Llovet, PhD, Behavioural Scientist, Cancer Screening, Prevention & Cancer Control, Cancer Care Ontario Address: 505 University Ave, 18-37, Toronto, Ontario, M5G 1X3 Tel: (416) 971-9800 x 2540 E-mail: [email protected]
1
ACCEPTED MANUSCRIPT
Abstract: There is considerable variation in quality of colonoscopy, attributable in part to endoscopist performance. Audit and feedback (A&F) provides health professionals
RI PT
with a summary of their performance over a period of time and is a common strategy used to improve provider performance. In this review, we discuss current understanding of the mechanism of A&F and describe specific features of effective A&F. To date, trials
SC
of A&F to improve colonoscopy performance report heterogeneous results, in part because colonoscopy is a complex procedural skill but also because the quality
M AN U
improvement interventions were sub-optimally implemented or inadequately evaluated. Nonetheless, evidence from a wide range of literature suggests that A&F has the potential to improve endoscopist performance. We discuss future directions for research in this area and provide guidance for providers or health system planners
jurisdiction.
TE D
wishing to implement A&F to address quality of colonoscopy in their practice and/or
AC C
EP
Key words: Colonoscopy, audit & feedback, quality improvement
2
ACCEPTED MANUSCRIPT
Words: 6785 Overview
RI PT
High quality colonoscopy is integral to the diagnosis and prevention of colorectal cancer (CRC) as well as to the management of gastrointestinal disease. Unfortunately,
colonoscopy quality is highly variable [1, 2] and poor quality colonoscopy adversely
SC
affects patients. For example, poor quality is associated with post-colonoscopy CRC [3-5]
M AN U
and therefore, increased patient morbidity and mortality [3].
Variation in the quality of colonoscopy is likely multifactorial, including patient, physician, equipment and system factors; however, variation in the quality of the endoscopist’s performance is an important contributor. As audit and feedback (A&F) is
TE D
one of the few performance improvement tools that has been shown to effect a change in provider behaviour [6], it is commonly incorporated into quality management programs aiming to improve quality of colonoscopy [7, 8]. However, colonoscopy is a
EP
complex procedural skill and A&F is an equally complex intervention. Therefore it is important to have a thorough understanding of the principles and features of effective
AC C
A&F prior to implementation in colonoscopy settings. In addition, the use of A&F to improve colonoscopy presents unique challenges, largely because colonoscopy is comprised of multiple interacting motor and cognitive components, which may make A&F less effective.
3
ACCEPTED MANUSCRIPT
In this review, we describe what is known about A&F in general, including current understanding about how A&F works, its effectiveness in changing provider practice and
RI PT
features of effective A&F. We also review the use of A&F to improve colonoscopy performance in particular and identify opportunities to optimize future interventions. The audience for this review is broad and includes health system planners and policy
M AN U
activities in their practices or hospitals.
SC
makers as well as physicians who are interested in engaging in or who are leading A&F
1. What is A&F?
A&F is a common intervention that is used to improve health professional practice in a wide variety of clinical contexts. It is broadly defined as ‘any summary of clinical
TE D
performance of healthcare over a specified period of time’ [9] and is one of the few quality improvement interventions that has been shown to improve physician performance, leading to better patient care [6]. By providing objective data, A&F can
EP
highlight discrepancies between current practice and target performance, and accordingly, can prompt action for practice improvement when clinical practice is
AC C
recognized as suboptimal [10].
2. How does A&F work? To best ensure that an A&F intervention succeeds, it is important to consider the mechanism of action of a successful A&F intervention and how a specific intervention might work in a given context.
4
ACCEPTED MANUSCRIPT
In behavioural science, theoretical frameworks are used to explain the mechanism of
RI PT
action of interventions such as A&F [11]. Unfortunately, most A&F interventions are designed without reference to these frameworks and without building on previous
research [12], which may account for some of the observed disparity in findings from
SC
A&F studies. An important observation is that at least 25% of A&F interventions in
health care settings are ineffective or minimally effective and a small number of studies
M AN U
appear to have a negative effect on performance [9]. Thus, leaders in the field need to carefully and thoughtfully develop A&F interventions in order to be successful in improving the quality of colonoscopy. An understanding of these theoretical frameworks is essential to this process [11]. One of these frameworks is discussed
TE D
below.
Kluger and DeNisi’s Feedback Intervention Theory, developed for industrial/business
EP
environments [13], provides a useful starting point for health system planners and physicians considering A&F to improve colonoscopy performance. According to
AC C
Feedback Intervention Theory, effective feedback shifts a physician’s attention towards a practice-based task where there is room for improvement. Three key factors determine whether A&F will accomplish this shift: 1) characteristics of the feedback itself (e.g., content, format of feedback); 2) the nature of the task that is to be improved (e.g., bowel preparation, reaching the cecum); and 3) situational (e.g., monetary incentives, organizational pressures, public disclosure) and personality variables (e.g.,
5
ACCEPTED MANUSCRIPT
self-efficacy, goal-commitment). See Figure 1 for an illustration of this theory for the
RI PT
task of cecal intubation.
In their seminal paper on Feedback Intervention Theory, Kluger and DeNisi performed a meta-analysis of studies from industrial/organizational psychology to test their
SC
theoretical framework [13]. Largely, their results supported the validity of the
framework. They found that the following characteristics augmented the feedback’s
M AN U
effect on performance: 1) including information that helps the recipient understand what needs to change in order to improve performance (called “correct solution information”); 2) including information regarding trends in performance (if prior data is available); 3) providing frequent feedback; and 4) providing computerized feedback.
TE D
Conversely, feedback characteristics that attenuated its effect included: 1) a focus on praise or discouragement (which directs attention towards the self and away from the task) and 2) providing verbal feedback alone. The effect of the feedback was also
EP
influenced by the nature of the task and personality and situational variables (see Table 1). A more recent meta-analysis using the Feedback Intervention Theory framework on
AC C
studies in the health care literature yielded similar findings [14].
3. How effective is A&F? What can be done to make A&F more effective?
The study of A&F has continued and since 1995, the literature on its effectiveness in health care has burgeoned [15]. The most recent Cochrane review comprises 140
6
ACCEPTED MANUSCRIPT
randomized trials of A&F in a health care setting and found that overall, A&F leads to a modest improvement in provider performance with a median 4.3% absolute increase in
RI PT
compliance with desired practice (IQR: 0.5 – 16%). However, results from individual trials continue to be heterogeneous with examples of both “positive” and “negative”
trials in the literature [9]. The reasons for these varied results are likely multifactorial,
SC
ranging from differing (and often poor) trial designs, the nature of the targeted
behaviour, the potential for variation in the A&F interventions across studies, and the
M AN U
context in which it was studied. In a pre-specified exploratory analysis, the Cochrane review found that A&F is more effective at improving quality of care when baseline performance is low, when feedback is provided more than once, when it is delivered by a trusted source (e.g. a colleague or supervisor), when it is delivered in both verbal and
TE D
written formats, when the goal is to decrease current behaviour, and when it includes explicit targets and action plans [9].
EP
Other systematic reviews of A&F in health care settings inform us about additional factors that make A&F more effective. One review examined feedback using medical
AC C
registries and found that feedback targeting process measures to be generally more effective than that targeting outcome measures [16]. A second study identified characteristics of Veterans Affairs Medical Centers with high and low compliance with clinical practice guidelines [17]. In both studies, facilitators of effective feedback included data characteristics (trust in the data [16]and individualized data [16, 17]), motivation of recipients [16], timeliness [17], non-punitiveness [17], and organizational
7
ACCEPTED MANUSCRIPT
factors [16] (e.g., quality improvement infrastructure). By identifying features of effective A&F, these reviews contribute to the growing understanding of how A&F
RI PT
works and highlight its potential mechanisms of action [18].
Although in many ways still in its infancy, the science of A&F is advancing and
SC
knowledge is accumulating. It is clear that A&F can be effective in improving physician performance. Thoughtful implementation includes careful design and delivery of the
M AN U
audit (maximizing the validity of the data, e.g., case-mix adjustment) and feedback (attention to the way data is presented, including choice of comparisons, and attention to how it is delivered, e.g., a trusted source), careful consideration of what aspect of physician performance or physician behaviour is targeted for change and inclusion of
TE D
targets, goals and an action plan [10]. The desired action (i.e., do more, do less or for interpretation/information) should be clear to the recipient. Implementation of a new A&F intervention should include a plan for evaluation and refinement to ensure
EP
alignment with goals. There are useful, evidence-based resources that can provide guidance to leaders in colonoscopy who wish to design and implement A&F
AC C
interventions [19, 20].
4. Can A&F be used for colonoscopy improvement?
The use of A&F for colonoscopy presents unique challenges. Colonoscopy is a complex task; arguably it is comprised of multiple “sub-tasks”, which can be broadly categorized
8
ACCEPTED MANUSCRIPT
as either cognitive or motor [21]. Motor or technical tasks include positioning the patient, handling and advancing the colonoscope, loop reduction, withdrawal and
RI PT
inspection techniques as well as polypectomy [21, 22]. Some have separated the cognitive aspects of colonoscopy into content area knowledge, consisting of pre-
procedure, insertion, withdrawal and diagnosis (including lesion recognition), treatment
SC
(including lesion removal), post-procedure and complications, and non-technical skills, which include analytical and personal behaviour [22]. Other non-technical skills are
M AN U
considered “social” and include communication and team-work [22].
Application of the Feedback Intervention Theory framework to the case of colonoscopy is informative. First of all, it is important to consider the overall “task”: colonoscopy is
TE D
physical and is clearly complex, two features which have been shown to attenuate the impact of feedback [13]. Figure 1 applies Feedback Intervention Theory to an example from colonoscopy and illustrates reasons that A&F might be less effective in this context
EP
using the example of cecal intubation: A&F can be expected to modify behaviour (e.g. attending a skills improvement course or practicing skills) but cannot reasonably be
AC C
expected to modify the motor skills required to insert the colonoscope. The challenges of A&F when applied to procedures such as colonoscopy are reflected in the fact that most recent meta-analyses of A&F exclude trials of physician procedure performance entirely [9, 14, 23].
9
ACCEPTED MANUSCRIPT
The cognitive tasks involved in colonoscopy, in particular the content-area knowledge, may be amenable to this intervention as feedback appears to augment memory tasks
RI PT
[13]. Furthermore, a “performance gap” for adenoma detection may motivate the recipient to examine the colon more carefully upon withdrawal. Hence, in theory, A&F may be useful to improve specific aspects of colonoscopy such as bowel preparation,
SC
adherence to surveillance guidelines and adenoma detection rates (ADR). However, as noted above, A&F alone should not be expected to improve colonoscopy technical
M AN U
tasks/skills such as those required for safe and effective cecal intubation or polypectomy. Rather, A&F’s expected mechanism of action if a technical “performance gap” were identified would be to motivate the recipient to take additional steps to improve their technical skills. Additional resources such as hands-on colonoscopy
TE D
courses [22, 24], detailed educational videos [25], or multimedia presentations [26] might be useful. Feedback facilitated by a trusted supervisor or peer (see section 6 below) may be particularly useful in this context as it may help the recipient to
EP
understand and interpret their data and to develop an action plan for improvement [27], which should include identifying the appropriate resources to effect a change in
AC C
performance. It is important to point out that because this is a multistep process, A&F may be successful (i.e., it motivates the recipient to take steps to improve their motor skills) but performance may not be improved if the subsequent educational interventions are ineffective. Therefore, intermediate processes should be measured [28] in order to optimize the effectiveness of A&F in improving colonoscopy technical skills.
10
ACCEPTED MANUSCRIPT
An understanding of the sub-tasks of colonoscopy can be used to inform the design of
RI PT
A&F. Specific elements of A&F can be tailored to address individual sub-tasks; when doing so, it is important to anticipate how the recipient might respond to the A&F intervention. Failure to do so may have unintended consequences. For example,
SC
providing feedback on withdrawal time (time of colon inspection, from the intubation of the cecum to removal of colonoscope) in order to improve ADR without providing
M AN U
information on how to improve inspection of the colon (i.e., correct solution information) is at best, unlikely to be effective (see next section) or at worst, lead to “gaming” where endoscopists linger in the rectum in order to achieve acceptable
TE D
withdrawal times [29].
5. How effective is A&F for colonoscopy improvement? There are relatively few studies of A&F interventions for colonoscopy. Earlier studies,
EP
which tended to be quasi-experimental (i.e., tracked one group using a “pre/post” design) [30], often focused on the measurement and reporting of a single performance
AC C
measure--withdrawal time--and overall, results were disappointing. A systematic review of these studies [31] found that the measurement of withdrawal time alone (either in real time or as an aggregate measure over a number of colonoscopies) was generally ineffective. The one study that did show a benefit also incorporated an educational intervention providing tips on good colonoscopy inspection technique (i.e., adequate
11
ACCEPTED MANUSCRIPT
insufflation, repetitive examination of colonic segments, torqueing maneuvers to
RI PT
improve visualization between folds) [32].
Unfortunately, the field has not advanced significantly since the systematic review in
2011. Subsequently published studies testing the effectiveness of A&F specifically (i.e.,
SC
comparing A&F to usual care) continue to use quasi-experimental designs [33-35],
generally testing A&F in a small number of endoscopists at a single centre over a short
M AN U
period of time [33, 35]. While these studies have shown a significant benefit of A&F, the lack of a control group makes the findings questionable as they could also be due to secular trends.
TE D
To date, there are two published randomized controlled trials that include the use of A&F [36, 37] but did not compare it to a control group that did not receive A&F. In these trials, A&F was provided to both arms of the study and the intervention groups
EP
received supplementary interventions. In one study [36], physicians in the intervention group who received 2 hours of training on methods and techniques to improve ADR and
AC C
on polyp classification methods experienced a significant improvement in their ADR (11% vs -1% change from baseline in the controls). The second study [37] comprised leaders at colonoscopy screening centre in Poland whose ADR at baseline was lower than 25% (n=38). The intervention group received a 3-4 day hands on course on teaching high quality colonoscopy with the hypothesis that this intervention would diffuse into their home screening centre leading to improvements in care. The study
12
ACCEPTED MANUSCRIPT
showed a significant improvement in the primary outcome (change in the leaders’ ADR from baseline) in the intervention group (8.2%) compared to controls (1.1%). These
RI PT
results suggest that A&F is effective for colonoscopy improvement when paired with a supplementary educational intervention that provides correct solution information but do not address its effectiveness when implemented alone. Results from endoscopy
SC
studies published to date are summarized in Table 2.
M AN U
6. Improving the effectiveness of A&F in colonoscopy: future directions It has been noted that research on A&F is “stagnating” and that there is a need to incorporate best practices in A&F and address gaps in the literature in future work [10]. Leading professional societies and others have published comprehensive lists of pre-,
TE D
intra- and post-procedural colonoscopy quality indicators [38-41]. While there is evidence to support some of these indicators, many are based on expert opinion only. Indicators used in A&F for colonoscopy improvement should be carefully selected to
EP
ensure that they are meaningful. Unfortunately, currently, ease of measurement may drive selection. For example, polyp detection rate (PDR) is often used instead of ADR as
AC C
the former can often be captured using claims data whereas the latter generally requires manual reconciliation of the endoscopy record with pathology results [38]. Although there is data to support the association between lower PDR and lower rates of missed CRCs [5], there is arguably greater potential for gaming with PDR than ADR [38].
13
ACCEPTED MANUSCRIPT
As another example, cecal intubation rate is often reported but it may be more meaningful to have a measure of how often the cecum is reached while keeping the
RI PT
patient safe and comfortable (i.e., without excess sedation and with good comfort scores) [42]. It is important to ensure that as much as possible, the data and design of
the A&F produce the intended response in the recipient thereby achieving the goals of
SC
providing the feedback. An understanding of the factors that contribute to variation in indicators such as ADR [43] is also important as these factors can be incorporated into
M AN U
the design (i.e., correct solution information) of A&F interventions.
The medical education and broader behavioral science literature offer strategies that may enhance the impact of A&F interventions, including the use of social norms,
TE D
pledges/commitment devices, defaults (opt-out vs. opt-in approaches to uptake of recommended behaviour) [44] and facilitated feedback [27]. Social norms are the rules or standards that influence the behavior of members of a group [45]. There are two
EP
types of social norms. An individual is influenced by descriptive social norms when s/he tries to align his or her behavior with the behavior of the rest of the group in order to
AC C
avoid being labeled an outlier. An individual is influenced by prescriptive (or injunctive) social norms when s/he tries to align his or her behavior with the behavior prescribed by a recognizable authority [46]. A&F interventions can incorporate descriptive social norms by using peer comparator information that will help the individual physician understand where his or her performance stands vis-à-vis that of his or her peers. Prescriptive social norms can be incorporated by referencing performance targets set by
14
ACCEPTED MANUSCRIPT
credible professional bodies that will help the physician know whether s/he is in
RI PT
compliance with accepted recommendations.
The use of pledges or commitment devices has long been studied in behavioural science and has been successful in continuing medical education [47, 48]. It highlights that once
SC
a commitment is made, an individual will experience personal and interpersonal
pressures to honour that commitment [49]. In the context of A&F, the intervention
M AN U
could include use of a pledge whereby the physician commits to attend a course or seek other educational opportunities to help him or her improve his or her colonoscopy performance. The pledge may make it more likely that the physician will attend the
TE D
course.
Behavioral theory suggests that inertia plays an important role in influencing how individuals behave –when confronted with options, individuals will often go with the
EP
default option because doing so is the easy thing to do (i.e. it requires no effort on their part). This insight has led behavioral scientists to recommend making the preferred
AC C
option the default option when designing programs [50]. Purposeful use of default options can help increase the chances of success of an A&F intervention. For example, physicians below a certain threshold-score on their A&F can be prescheduled to attend a colonoscopy skills improvement course and given the option to opt-out if they choose to (instead of asking them to make the effort to schedule the course themselves). The
15
ACCEPTED MANUSCRIPT
advantage of default options is that they influence decisions without restricting choice
RI PT
[50].
Facilitated feedback involves engaging recipients of A&F in a reflective conversation
about their performance and the feedback itself in order to gain a deeper understanding
SC
of both. One model of facilitated feedback developed for physicians specifically employs a 4 step approach: building rapport and relationship, exploring reactions and
M AN U
perceptions of the report, exploring understanding of the content of the report and coaching for performance change [27]. Facilitated feedback can help mitigate a response known as ‘cognitive dissonance’, which refers to the struggle of maintaining two conflicting beliefs simultaneously. For example, if A&F suggests substandard
TE D
performance, the recipient must wrestle with 2 beliefs: that s/he is underperforming and the self-perception that s/he is a capable and competent practitioner [51]. Because it is easier to question the validity of the data than to question oneself, the recipient
EP
may resolve this conflict by discounting the data. Facilitation of feedback, particularly from a trusted source, may help the recipient to set practice improvement goals rather
AC C
than reject the data altogether [27, 52]. A&F presents the recipient with data that identifies a performance gap, facilitated feedback helps to close the circle leading to improvement in patient care by validating the performance gap (Figure 2).
7. Summary and conclusion
16
ACCEPTED MANUSCRIPT
A&F has the potential to improve endoscopist performance thereby contributing to the improvement of quality of colonoscopy. Careful implementation of A&F, based on a
RI PT
sound understanding of its mechanism, is necessary to ensure that the intervention is thoughtfully designed (see Box 1) to maximize the likelihood of achieving its goals.
More research in A&F to improve the quality of colonoscopy is needed (Box 2) to inform
AC C
EP
TE D
Conflict of interest: None
M AN U
performance improvement intervention.
SC
implementation efforts and to maximize the benefit from this promising endoscopist
17
ACCEPTED MANUSCRIPT
References 1. Rabeneck L, Rumble RB, Axler J, Smith A, Armstrong D, Vinden C et al. Cancer Care
RI PT
Ontario Colonoscopy Standards: standards and evidentiary base. Can J Gastroenterol 2007;21 Suppl D:5D-24D.
2. Tinmouth J, Baxter NN, Li Q, Paszat L, Patel J, Rabeneck L. Measuring the quality of
SC
colonoscopy delivered to an entire population in Ontario, Canada. Gastrointest Endosc 2014;79(5):AB118.
M AN U
3. Corley DA, Jensen CD, Marks AR, Zhao WK, Lee JK, Doubeni CA et al. Adenoma detection rate and risk of colorectal cancer and death. N Engl J Med 2014;370(14):1298306.
4. Kaminski MF, Regula J, Kraszewska E, Polkowski M, Wojciechowska U, Didkowska J et
TE D
al. Quality indicators for colonoscopy and the risk of interval cancer. N Engl J Med 2010;362(19):1795-803.
5. Baxter NN, Sutradhar R, Forbes SS, Paszat LF, Saskin R, Rabeneck L. Analysis of
EP
administrative data finds endoscopist quality measures associated with postcolonoscopy colorectal cancer. Gastroenterology 2011;140(1):65-72.
AC C
6. Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess 2004;8(6):1-84. 7. Joint Advisory Group on GI Endoscopy. http://www.thejag.org.uk. Accessed March 10, 2016.
18
ACCEPTED MANUSCRIPT
8. Quality Management Partnership: Colonoscopy. https://www.qmpontario.ca/colonoscopy/. Accessed March 10, 2016.
RI PT
9. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD et al. Audit and feedback: effects on professional practice and healthcare outcomes Cochrane Database Syst Rev 2012;6(CD000259).
SC
10. Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ et al. No more ‘business as usual’with audit and feedback interventions: towards an agenda for a reinvigorated
M AN U
intervention. Implement Sci 2014;9(1):14.
11. Shute VJ. Focus on formative feedback. Rev Educ Res 2008;78(1):153-89. 12. Colquhoun HL, Brehaut JC, Sales A, Ivers N, Grimshaw J, Michie S et al. A systematic review of the use of theory in randomized controlled trials of audit and feedback.
TE D
Implement Sci 2013;8(1):66.
13. Kluger AN, DeNisi A. The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory.
EP
Psychol Bull 1996;119(2):254-84.
14. Hysong SJ. Meta-analysis: audit & feedback features impact effectiveness on care
AC C
quality. Med Care 2009;47(3):356. 15. Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O’Brien MA, French SD et al. Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care. J Gen Intern Med 2014;29:1534-41.
19
ACCEPTED MANUSCRIPT
16. van der Veer SN, de Keizer NF, Ravelli AC, Tenkink S, Jager KJ. Improving quality of care. A systematic review on how medical registries provide information feedback to
RI PT
health care providers. Int J Med Inform 2010;79(5):305-23. 17. Hysong SJ, Best RG, Pugh JA. Audit and feedback and clinical practice guideline adherence: Making feedback actionable. Implement Sci 2006;1(9):5-3.
SC
18. Foy R, Eccles M, Jamtvedt G, Young J, Grimshaw J, Baker R. What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review.
M AN U
BMC Health Serv Res 2005;5(1):50.
19. Brehaut JC, Colquhoun HL, Eva KW, Carroll K, Sales A, Michie S et al. Practice Feedback Interventions: 15 Suggestions for Optimizing Effectiveness. Ann Intern Med 2016;164:435-41.
TE D
20. French SD, Green SE, O’Connor DA, McKenzie JE, Francis JJ, Michie S et al. Developing theory-informed behaviour change interventions to implement evidence into practice: a systematic approach using the Theoretical Domains Framework.
EP
Implementat Sci 2012;7(1):38.
21. Sedlack RE. Training to competency in colonoscopy: assessing and defining
AC C
competency standards. Gastrointest Endosc 2011;74(2):355-66. 22. Robertson DJ, Kaminski MF, Bretthauer M. Effectiveness, training and quality assurance of colonoscopy screening for colorectal cancer. Gut 2015;64(6):982-90. 23. Gardner B, Whittington C, McAteer J, Eccles M, Michie S. Using theory to synthesise evidence from behaviour change interventions: The example of audit and feedback. Soc Sci Med 2010(70):1618-25.
20
ACCEPTED MANUSCRIPT
24. Thomas-Gibson S, Williams CB. Colonoscopy training—new approaches, old problems. Gastrointest Endosc Clin N Am 2005;15(4):813-27.
RI PT
25. American Society fornGastrointestinal Endoscopy (ASGE). Endoscopic Learning Library: Endoscopy’s most comprehensive CME video collection. 2015.
http://www.asge.org/uploadedFiles/Public_E-Blast_PDFs/ASGE_ELL_brochure_2015.pdf
SC
26. Marx R, le Clercq C, Bogie R, Winkens B, Kruimel J, Conchillo JM et al. Training on detection and resection of nonpolypoid colorectal neoplasms reduces the
M AN U
postcolonoscopy colorectal cancer rate. Gastroenterology 2015;148(4):S-96. 27. Sargeant J, Lockyer J, Mann K, Holmboe E, Silver I, Armson H et al. Facilitated reflective performance feedback: developing an evidence-and theory-based model that builds relationship, explores reactions and content, and coaches for performance
TE D
change (R2C2). Acad Med 2015;90(12):1698-706.
28. Grimshaw JM, Eccles MP, Walker AE, Thomas RE. Changing physicians' behavior: what works and thoughts on getting more things to work. J Contin Educ Health Prof
EP
2002;22(4):237-43.
29. Inadomi JM. In search of quality colonoscopy. Gastroenterology 2008;135(6):1845-7.
AC C
30. Harris AD, McGregor JC, Perencevich EN, Furuno JP, Zhu J, Peterson DE et al. The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc 2006;13(1):16-23. 31. Corley DA, Jensen CD, Marks AR. Can we improve adenoma detection rates? A systematic review of intervention studies. Gastrointest Endosc 2011;74(3):656-65.
21
ACCEPTED MANUSCRIPT
32. Barclay RL, Vicari JJ, Greenlaw RL. Effect of a time-dependent colonoscopic withdrawal protocol on adenoma detection during screening colonoscopy. Clin
RI PT
Gastroenterol Hepatol 2008;6(10):1091-8. 33. Keswani RN, Yadlapati R, Gleason KM, Ciolino JD, Manka M, O'Leary KJ et al.
Physician report cards and implementing standards of practice are both significantly
SC
associated with improved screening colonoscopy quality. Am J Gastroenterol 2015;110(8):1134-9.
M AN U
34. Fraser AG, Gamble GD, Rose TR, Dunn JP. Colonoscopy audit over 10 years--what can be learnt? The New Zealand Medical Journal (Online) 2013;126(1382). 35. Kahi CJ, Ballard D, Shah AS, Mears R, Johnson CS. Impact of a quarterly report card on colonoscopy quality measures. Gastrointest Endosc 2013;77 (6 ):925-31.
TE D
36. Coe SG, Crook JE, Diehl NN, Wallace MB. An endoscopic quality improvement program improves detection of colorectal adenomas. Am J Gastroenterol 2013;08(2):219-26.
EP
37. Kaminski MF, Anderson J, Valori R, Kraszewska E, Rupinski M, Pachlewski J et al. Leadership training to improve adenoma detection rate in screening colonoscopy: a
AC C
randomised trial. Gut 2016;65:616-24. 38. Rex D, Schoenfeld P, Cohen J, Pike I, Adler D, Fennerty M et al. Quality indicators for colonoscopy. Gastrointest Endosc 2015;81(1):31-53. 39. Tinmouth J, Kennedy E, Baron D, Burke M, Feinberg S, Gould M. Guideline for colonoscopy quality assurance in Ontario. Cancer Care Ontario 2013.
22
ACCEPTED MANUSCRIPT
40. European Commission. European guidelines for quality assurance in colorectal cancer screening and diagnosis. First ed. Luxembourg: International Agency for Research
RI PT
on Cancer 2010. 41. Chilton A, Rutter M. Quality assurance guidelines for colonoscopy. Sheffield: NHS Cancer Screening Programmes 2011.
SC
42. Valori RM, Gavin D, Damery S, Swarbrick E, Donnelly M, Anderson J et al. Sa1400 A
composite measure of colonic intubation (Circ) is better able to distinguish performance
M AN U
of colonoscopy and is associated with higher polyp detection rates. Gastrointest Endosc 2014;79(5):AB197-AB8.
43. Atkins L, Hunkeler EM, Jensen CD, Michie S, Lee JK, Doubeni CA et al. Factors influencing variation in physician adenoma detection rates: a theory-based approach for
TE D
performance improvement. Gastrointest Endosc 2016;83:617-26. 44. Dolan P, Hallsworth M, Halpern D, King D, Metcalfe R, Vlaev I. Influencing behaviour: The mindspace way. J Econ Psychol 2012;33(1):264-77.
EP
45. Cialdini RB, Trost MR. Social influence: Social norms, conformity and compliance. In: D. T. Gilbert STF, & G. Lindzey, editor. The handbook of social psychology. Boston:
AC C
McGraw-Hill; 1998.
46. Cialdini RB, Reno RR, Kallgren CA. A focus theory of normative conduct: recycling the concept of norms to reduce littering in public places. J Pers Soc Psychol 1990;58(6):1015.
23
ACCEPTED MANUSCRIPT
47. Domino FJ, Chopra S, Seligman M, Sullivan K, Quirk ME. The impact on medical practice of commitments to change following CME lectures: A randomized controlled
RI PT
trial. Med Teach 2011;33(9):e495-e500. 48. Wakefield J, Herbert CP, Maclure M, Dormuth C, Wright JM, Legare J et al.
Health Prof 2003;23(2):81-92.
SC
Commitment to change statements can predict actual change in practice. J Contin Educ
49. Cialdini RB. Influence: Science and practice Fourth Editiion. New York:
M AN U
HarperCollinsCollegePublishers; 2001.
50. Halpern SD, Ubel PA, Asch DA. Harnessing the power of default options to improve health care. N Engl J Med 2007;357(13):1340-4.
51. Eva KW, Regehr G. Effective feedback for maintenance of competence: from data
TE D
delivery to trusting dialogues. Can Med Assoc J 2013;185(6):463-4. 52. Sargeant JM, Mann KV, Van der Vleuten CP, Metsemakers JF. Reflection: a link between receiving and using assessment feedback. Adv Health Sci Educ 2009;14(3):399-
EP
410.
53. Lin OS, Kozarek RA, Arai A, Gluck M, Jiranek GC, Kowdley KV et al. The effect of
AC C
periodic monitoring and feedback on screening colonoscopy withdrawal times, polyp detection rates, and patient satisfaction scores. Gastrointest Endosc 2010;71(7):1253-9. 54. Harewood GC, Murray F, Winder S, Patchett S. Evaluation of formal feedback on endoscopic competence among trainees: the EFFECT trial. Ir J Med Sci 2008;177(3):2536.
24
ACCEPTED MANUSCRIPT
55. Sawhney MS, Cury MS, Neeman N, Ngo LH, Lewis JM, Chuttani R et al. Effect of institution-wide policy of colonoscopy withdrawal time ≥ 7 minutes on polyp detection.
RI PT
Gastroenterology 2008;135(6):1892-8. 56. Harewood GC, Petersen BT, Ott BJ. Prospective assessment of the impact of
feedback on colonoscopy performance. Aliment Pharmacol Ther 2006;24(2):313-8.
SC
57. Shaukat A, Oancea C, Bond JH, Church TR, Allen JI. Variation in detection of
adenomas and polyps by colonoscopy and change over time with a performance
M AN U
improvement program. Clin Gastroenterol Hepatol 2009;7(12 ):1335-40. 58. Imperiali G, Minoli G, Meucci GM, Spinzi G, Strocchi E, Terruzzi V et al. Effectiveness of a continuous quality improvement program on colonoscopy practice. Endoscopy
AC C
EP
TE D
2007;39(04 ):314-8.
25
ACCEPTED MANUSCRIPT
Box 1: Practice Points 1.
Physicians are poor at self-assessment and A&F can help physicians identify
2.
RI PT
areas of practice in need of improvement. A&F colonoscopy initiatives must be implemented thoughtfully. A framework that can guide these types of initiatives [20] is applied to colonoscopy below:
SC
a. What is the colonoscopy performance-gap you are trying to change? What behaviour needs to change to close this gap?
behaviour change?
M AN U
b. What are the levers (barriers and enablers) that would lead to this
c. What components could be included in the A&F initiative that could help overcome the barriers and enhance the enablers to change? A&F colonoscopy initiatives should anticipate and pre-emptively address
TE D
3.
common recipient reactions: rejecting the data or attributing their poor performance to their patient case-mix (i.e., their patients are different from
4.
EP
others).
The desired effect of each quality indictor in an A&F initiative should be clear to
AC C
the recipient (i.e., do more, do less, or indicator provided for information/interpretation).
5.
A&F colonoscopy initiatives should strive to incorporate meaningful quality indicators, rather than simply reporting data that is easy to measure.
26
ACCEPTED MANUSCRIPT
6.
A&F alone may be insufficient to improve certain aspects of colonoscopy performance, such as technical skills. Using A&F to motivate recipients to
RI PT
engage with other resources such as colonoscopy skills improvement courses, educational videos or multimedia presentations may be more effective. 7.
Ongoing evaluation of A&F colonoscopy initiatives is important to assess
SC
effectiveness and alignment with goals. Leaders in colonoscopy should initiate A&F in their local practice contexts but expect that initial attempts will be
M AN U
imperfect. Leaders should incorporate evaluation and iterative revisions of the A&F initiative into the implementation plan. 8.
There are useful, evidence-based resources that can provide guidance to leaders in colonoscopy who wish to design and implement A&F interventions
TE D
[19, 20]
Box 2: Research Agenda
There is a need to move beyond the study of A&F alone to improve physician
EP
1.
performance [10]. The social sciences provide insight into
AC C
ancillary/complementary interventions to study. Future study should consider formal evaluation of A&F interventions that incorporate facilitated feedback, social norms, pledges, and use of defaults [44].
2.
Research is needed to develop meaningful quality indicators to measure colonoscopy performance and to assess whether the use of accepted, existing indicators in A&F is leading to meaningful change in colonoscopy performance.
27
ACCEPTED MANUSCRIPT
3.
Colonoscopy-specific A&F elements, which are specifically designed to target colonoscopy sub-tasks such as technical tasks (e.g., colonoscope insertion) and
RI PT
cognitive tasks (e.g., content area knowledge, communication), should be
AC C
EP
TE D
M AN U
SC
developed and evaluated.
28
ACCEPTED MANUSCRIPT
Figure legends Figure 1. Simple schematic of Feedback Intervention Theory, adapted from Kluger and
RI PT
DeNisi [13], using cecal intubation as an example. According to the theory, feedback is successful if it shifts the recipient’s attention to the performance gap; 3 key factors
contribute to this success (see boxes with: --- ): characteristics of the feedback, task
SC
characteristics and situational/personality variables. In this example, the behaviour
change resulting from audit and feedback is that the recipient takes steps to improve
M AN U
(take a course, practice). Performance improvement depends on both successful behaviour change and the effectiveness of the course. [ ], component of theory.
Figure2.Critical elements necessary for audit and feedback to lead to improvement in
TE D
patient care. A motivated recipient (1) is an essential first step. Feedback highlights a performance-gap (2). To prevent the recipient from disregarding the data, the performance-gap needs to be validated (3), for instance through facilitated feedback or
EP
by emphasizing the accuracy of the audit. Finally, tangible changes to patient care (4)
AC C
can be implemented to address the performance-gap.
29
ACCEPTED MANUSCRIPT
Tables
RI PT
Table 1. Characteristics of feedback, nature of the task and situational and personality variables that influenced the effectiveness of feedback according to Kluger and DeNisi [13]
SC
M AN U
TE D
Situational variables Personality variables
EP
Tasks
AC C
Feedback Characteristics
Effect on feedback Augmented Attenuated • information about what • praise needs to change in order • discouragement to improve performance • verbal feedback only (“correct solution information”) • trend in performance • frequent feedback • computerized feedback • memory tasks • physical tasks • complex tasks • “following the rules” tasks • goal-setting intervention • self-esteem threatened
30
ACCEPTED MANUSCRIPT
Fraser et al, 2013 [34]
6 endoscopists, ≥15 procedures in each study period / 928 screening colonoscopies in average risk patients aged 50+ QuasiEndoscopists (# not experimental provided), >100 (1G, Single colonoscopies over period), two study period/ 67570 private diagnostic, family endoscopy units, history screening, & New Zealand surveillance colonoscopies
ADR
Pre: Nov 2012-Mar 2013 Intervention 1 (Int1): Apr 2013-March 2014 Intervention 2 (Int2): Apr 2014-Oc2014 CIR and ADR, adjusted for physician, patient age, and sex / Pre: July – Dec 2008 Int: Apr 2009-Mar 2011
TE D
Quasiexperimental (1G, Pre/Int) / Single centre, tertiary care, USA.
EP
Kahi et al, 2013 [35]
20 endoscopists, >15 colonoscopies per month / 12894 screening colonoscopies
Primary outcome / Study period
Intervention(s)%
Findings
Int1: Individualized annual A&F (ADR, WT) derived from EMR with peer comparison (10th, median, 90th percentile)
ADR after Intervention #1: 28% (Pre) 31% (Int1)*
Individualized quarterly A&F (BPrep, pre procedure assessment, CIR, WT, ADR) derived from EMR with anonymized peer comparison Annual A&F meetings, other details not provided
CIR: 95.6% (Pre) 98.1% (Int)* ADR: 44.7% (Pre) 53.9% (Int)*
SC
Studies testing A&F vs usual care Keswani et Quasi$ al, 2015 [33] experimental (1G, Pre/Int) / Single centre, tertiary care, USA.
Study cohort / Procedures
M AN U
Design / Setting
AC C
Author, year
RI PT
Table 2. Selected studies of audit and feedback (A&F) for colonoscopy performance improvement organized by studies evaluating audit and feedback vs usual care and studies of A&F plus supplemental interventions compared to A&F alone or to usual care.
CIR, insertion time, WT, PDR Study period: 19992010
CIR: 96.3% (1999) 99.0% (2010)* Insertion time: 7.5 min (2001) 8.9 min (2010)* PDR: 29% (1999)
31
ACCEPTED MANUSCRIPT
4 gastroenterology fellowship trainees / 581 colonoscopies
Quasiexperimental (1G, Single period) / Single centre,
AC C
Sawhney et al, 2008 [55]
42 endoscopists, 23,910 colonoscopies
Individualized written A&F (WT, PDR, patient satisfaction) every 3-6 months anonymized peer comparison
SC
WT, PDR, ADR, and patient satisfaction scores
Pre: July 2005-June 2006
M AN U
Randomized controlled trial / Single centre, tertiary care, Ireland
10 gastroenterologists / 1391 asymptomatic adult screening colonoscopies
Int: July 2006-Apr 2007 CIR and PDR Pre: July-Aug 2006 Int: Oct 2006-Jan2007
EP
Harewood et al, 2008 [54]
Quasiexperimental (1G, Pre/Int) / Single centre, tertiary care, USA
TE D
Lin et al, 2010 [53]
RI PT
49% (2010)*
% WT ≥7 minutes, PDR Pre: Prior to Feb 2006 Study period: Feb 2006June 2007
Individualized written and graphed monthly A&F (CIR, PDR) reviewed at in person meeting with supervisor derived from EMR with anonymized peer comparison
Monthly individualized A&F (WT) derived from EMR
WT: 5.6 min (2001) 6.6 min (2010)* WT: 6.57 min (Pre) 8.07 min (Int)* Polyp detection: 33.1% (Pre) 38.1% (Int) ADR: 19.6% (Pre) 22.7% (Int) CIR: 72.9% (Pre) 83.4% (Int) in intervention vs. 78% (Pre) 71.9% (Post) in control* PDR: 12.9% (Pre) 18% (Int) in intervention vs. 16.9% (Pre) 19.6% (Post) in control % WT ≥7-minute#: 65% (1st month) almost 99% (last month)* PDR#: 48% (1st month)
32
ACCEPTED MANUSCRIPT
almost 55% (last month)
58 endoscopists / median of 310.5 outpatient colonoscopies/year per endoscopist from during study period (Pre & Int)
CIR, insertion time and WT
Individualized emailed quarterly A&F (CIR, insertion time, WT) derived from EMR with anonymized peer comparison
SC
Quasiexperimental (1G, Pre/Int), Single centre, USA
Pre: Jan 2004-Dec 2004 Int: Jan 2005-Dec 2005
M AN U
Harewood et al, 2006 [56]
RI PT
tertiary care, USA
AC C
EP
TE D
Studies of A&F + supplemental interventions compared to A&F alone or to usual care Kaminski et Randomized 38 endoscopists ADR Individualized mailed al, 2016 [37] controlled trial / (screening centre & emailed A&F (ADR & 40 colonoscopy leaders), ≥30 Pre: Jan 2011-Dec 2011 CIR) derived from EMR screening screening Early post-intervention with anonymized peer centres, Poland colonoscopies and (Early PI): June 2012comparison, issued at ADR <25% in 2011 / Dec 2012 end of Pre and Early 24582 screening Late post-intervention post-intervention colonoscopies (Late PI): Jan 2013-Dec phases + Train2013 Colonoscopy-Leaders (TCLs) programme (pretraining assessment, 2 days of hands-on training, post-training evaluation and feedback) vs
CIR: 96.3% >97.2% Insertion time: 10.6min 9.5min* WT: 9.1min 8.9min
ADR (early): 17.4% (Pre) 25.6% (Early PI) in intervention vs 18.5% (Pre) 19.6% (Early PI) in control* ADR (late): 17.4% (Pre) 23.9% (Late PI) in intervention vs 18.5% (Pre) 20.8% (Late PI) in control *
33
ACCEPTED MANUSCRIPT
ADR
M AN U
20 endoscopists, >15 colonoscopies per month / 12894 screening colonoscopies
Pre: Nov 2012-Mar 2013 Intervention 1 (Int1): Apr 2013-March 2014 Intervention 2 (Int2): Apr 2014-Oc2014
Randomized controlled trial / Single centre, tertiary care, USA
15 endoscopists, ≥10 procedures per month/ 2400 screening, surveillance, & diagnostic colonoscopies
AC C
Coe et al, 2013 [36]
EP
TE D
Keswani et Quasi$ al, 2015 [33] experimental (1G, Pre/Int) / Single centre, tertiary care, USA.
SC
RI PT
individualized mailed & emailed A&F (ADR & CIR) derived from EMR with anonymized peer comparison, issued at end of Pre and Early post-intervention phases alone Int1: Individualized ADR after Intervention annual A&F (ADR, WT) #2: 31% (Int1) 39% derived from EMR (Int2)* with peer comparison (10th, median, 90th percentile)
ADR, adjusted for patient age, sex, BPrep and indication Pre: Feb 2010-Jan 2011 Intervention: Feb 2011Apr 2011 Post: May 2011-Dec 2011
Int2: Failure to meet standards of practice (>5min WT and ADR>20%) training, endoscopy restrictions Individualized A&F (ADR & WT) derived from prospectively collected data, issued at end of Pre phase + 2 X 1 hour in person sessions: videos, still images, literature + post-test for
ADR: 36% (Pre) 47% (Post) in intervention vs. 36% (Pre) 35% (Post) in control*
34
ACCEPTED MANUSCRIPT
RI PT
ADR, adjusted for patient age, sex and BPrep
SC
43 gastroenterologists, >100 colonoscopies per year of study, >5000 colonoscopies over career/ 47253 1st time screening colonoscopies
M AN U
Quasiexperimental (1G, Pre/Int), community based, 5 ambulatory endoscopy centers, USA
Pre: Jan 2004 to Dec 2004 Int: Jan 2005-Dec2006
Imperiali et al, 2007 [58]
Quasiexperimental (1G, Pre/Int) Single centre, secondary care,
AC C
EP
TE D
Shaukat et al, 2009 [57]
8 gastroenterologists; 10,705 colonoscopies
competency vs. A&F (ADR & WT) derived from prospectively collected data, issued at end of Pre phase alone Individualized mailed biannual A&F (CIR, ADR) with peer comparison. Further interventions over study period: 1) blinded group ADR review, 2) unblinded individual ADR review at partnership meetings, 3) education re literature of ADR and WT standards, 4) poor performers met w/ practice leaders 5) 1% financial penalty for failure to meet standards Individualized biannual A&F (CIR, PDR) via chart audit. Blinded group A&F review at departmental
CIR and PDR Pre: Jan 2001-Dec 2001 Int: Jan 2002-Dec 2004
ADR: No significant change (data not provided)
CIR: 84.6% (Pre) 93.1% (Int)* Polyp detection: 33.8% (Pre) 33% (Int)
35
ACCEPTED MANUSCRIPT
meetings and action plan developed: (1) sedation mandated, (2) less experienced/ low performers more examinations, (3) less skilled examiners periodically supervised by veteran examiners, (4) incomplete another endoscopist attempts; and (5) physicians with low PDR met with unit head % A&F presentation and delivery details included where reported in original manuscript $ This study compared A&F to usual care (Int1) and A&F + supplemental intervention to A&F alone (Int2), therefore the study is listed twice, with relevant results reported in the appropriate sections of this table * p < 0.05, #monthly % regressed and significance determined based on slope of line 1G = one group ADR = adenoma detection rate BPrep = bowel preparation score CIR = cecal intubation rate PDR = polyp detection rate Pre / Int / Post= measurements prior to (Pre), during (Int) or after intervention (Post) WT = withdrawal time
AC C
EP
TE D
M AN U
SC
RI PT
Italy
36
AC C
EP
TE D
M AN U
SC
RI PT
ACCEPTED MANUSCRIPT
AC C
EP
TE D
M AN U
SC
RI PT
ACCEPTED MANUSCRIPT
S1521-6918(16)30012-9
DOI:
10.1016/j.bpg.2016.04.002
Reference:
YBEGA 1423
To appear in:
Best Practice & Research Clinical Gastroenterology
Received Date: 16 March 2016 Revised Date:
31 March 2016
Accepted Date: 7 April 2016
Please cite this article as: Tinmouth J, Patel J, Hilsden RJ, Ivers N, Llovet D, Audit and feedback interventions to improve endoscopist performance: principles and effectiveness, Best Practice & Research Clinical Gastroenterology (2016), doi: 10.1016/j.bpg.2016.04.002. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Audit and feedback interventions to improve endoscopist performance: principles and effectiveness
RI PT
Jill Tinmouth (corresponding author), MD PhD FRCPC, Lead Scientist, ColonCancerCheck Program, Cancer Care Ontario; Scientist and Staff Gastroenterologist, Sunnybrook Health Sciences Centre, Address: 2075 Bayview Avenue, Room HG40, Toronto, ON M4N 3M5 Tel: (416) 480-5910 Fax: (416) 480-4845 E-mail: [email protected]
M AN U
SC
Jigisha Patel MSc, Research Coordinator Sunnybrook Health Sciences Centre, Address: 2075 Bayview Avenue, Room HG40, Toronto, ON M4N 3M5 Tel: (416) 480-6100 x7941 Fax: (416) 480-4845 E-mail: [email protected]
TE D
Robert J. Hilsden, MD PhD FRCPC, Associate Professor, Departments of Medicine/Community Health Sciences, Cummings School of Medicine, University of Calgary; Director of Research, Forzani & MacPhail Colon Cancer Screening Center, Alberta Health Services Address: 3280 Hospital Drive NW, Room 6E17, TRW Building, Calgary, AB, CANADA T2N 4Z6 Tel: (403) 592-5089 Fax: (403) 592-5090 E-mail: [email protected]
AC C
EP
Noah Ivers, MD PhD, Assistant Professor, Department of Family and Community Medicine, University of Toronto; Scientist and Family Physician, Women's College Hospital Address: 77 Grenville Street, Toronto, ON M5S 1B3 Tel: (416) 323-6400 Fax: (416)323-6255 E-mail: [email protected] Diego Llovet, PhD, Behavioural Scientist, Cancer Screening, Prevention & Cancer Control, Cancer Care Ontario Address: 505 University Ave, 18-37, Toronto, Ontario, M5G 1X3 Tel: (416) 971-9800 x 2540 E-mail: [email protected]
1
ACCEPTED MANUSCRIPT
Abstract: There is considerable variation in quality of colonoscopy, attributable in part to endoscopist performance. Audit and feedback (A&F) provides health professionals
RI PT
with a summary of their performance over a period of time and is a common strategy used to improve provider performance. In this review, we discuss current understanding of the mechanism of A&F and describe specific features of effective A&F. To date, trials
SC
of A&F to improve colonoscopy performance report heterogeneous results, in part because colonoscopy is a complex procedural skill but also because the quality
M AN U
improvement interventions were sub-optimally implemented or inadequately evaluated. Nonetheless, evidence from a wide range of literature suggests that A&F has the potential to improve endoscopist performance. We discuss future directions for research in this area and provide guidance for providers or health system planners
jurisdiction.
TE D
wishing to implement A&F to address quality of colonoscopy in their practice and/or
AC C
EP
Key words: Colonoscopy, audit & feedback, quality improvement
2
ACCEPTED MANUSCRIPT
Words: 6785 Overview
RI PT
High quality colonoscopy is integral to the diagnosis and prevention of colorectal cancer (CRC) as well as to the management of gastrointestinal disease. Unfortunately,
colonoscopy quality is highly variable [1, 2] and poor quality colonoscopy adversely
SC
affects patients. For example, poor quality is associated with post-colonoscopy CRC [3-5]
M AN U
and therefore, increased patient morbidity and mortality [3].
Variation in the quality of colonoscopy is likely multifactorial, including patient, physician, equipment and system factors; however, variation in the quality of the endoscopist’s performance is an important contributor. As audit and feedback (A&F) is
TE D
one of the few performance improvement tools that has been shown to effect a change in provider behaviour [6], it is commonly incorporated into quality management programs aiming to improve quality of colonoscopy [7, 8]. However, colonoscopy is a
EP
complex procedural skill and A&F is an equally complex intervention. Therefore it is important to have a thorough understanding of the principles and features of effective
AC C
A&F prior to implementation in colonoscopy settings. In addition, the use of A&F to improve colonoscopy presents unique challenges, largely because colonoscopy is comprised of multiple interacting motor and cognitive components, which may make A&F less effective.
3
ACCEPTED MANUSCRIPT
In this review, we describe what is known about A&F in general, including current understanding about how A&F works, its effectiveness in changing provider practice and
RI PT
features of effective A&F. We also review the use of A&F to improve colonoscopy performance in particular and identify opportunities to optimize future interventions. The audience for this review is broad and includes health system planners and policy
M AN U
activities in their practices or hospitals.
SC
makers as well as physicians who are interested in engaging in or who are leading A&F
1. What is A&F?
A&F is a common intervention that is used to improve health professional practice in a wide variety of clinical contexts. It is broadly defined as ‘any summary of clinical
TE D
performance of healthcare over a specified period of time’ [9] and is one of the few quality improvement interventions that has been shown to improve physician performance, leading to better patient care [6]. By providing objective data, A&F can
EP
highlight discrepancies between current practice and target performance, and accordingly, can prompt action for practice improvement when clinical practice is
AC C
recognized as suboptimal [10].
2. How does A&F work? To best ensure that an A&F intervention succeeds, it is important to consider the mechanism of action of a successful A&F intervention and how a specific intervention might work in a given context.
4
ACCEPTED MANUSCRIPT
In behavioural science, theoretical frameworks are used to explain the mechanism of
RI PT
action of interventions such as A&F [11]. Unfortunately, most A&F interventions are designed without reference to these frameworks and without building on previous
research [12], which may account for some of the observed disparity in findings from
SC
A&F studies. An important observation is that at least 25% of A&F interventions in
health care settings are ineffective or minimally effective and a small number of studies
M AN U
appear to have a negative effect on performance [9]. Thus, leaders in the field need to carefully and thoughtfully develop A&F interventions in order to be successful in improving the quality of colonoscopy. An understanding of these theoretical frameworks is essential to this process [11]. One of these frameworks is discussed
TE D
below.
Kluger and DeNisi’s Feedback Intervention Theory, developed for industrial/business
EP
environments [13], provides a useful starting point for health system planners and physicians considering A&F to improve colonoscopy performance. According to
AC C
Feedback Intervention Theory, effective feedback shifts a physician’s attention towards a practice-based task where there is room for improvement. Three key factors determine whether A&F will accomplish this shift: 1) characteristics of the feedback itself (e.g., content, format of feedback); 2) the nature of the task that is to be improved (e.g., bowel preparation, reaching the cecum); and 3) situational (e.g., monetary incentives, organizational pressures, public disclosure) and personality variables (e.g.,
5
ACCEPTED MANUSCRIPT
self-efficacy, goal-commitment). See Figure 1 for an illustration of this theory for the
RI PT
task of cecal intubation.
In their seminal paper on Feedback Intervention Theory, Kluger and DeNisi performed a meta-analysis of studies from industrial/organizational psychology to test their
SC
theoretical framework [13]. Largely, their results supported the validity of the
framework. They found that the following characteristics augmented the feedback’s
M AN U
effect on performance: 1) including information that helps the recipient understand what needs to change in order to improve performance (called “correct solution information”); 2) including information regarding trends in performance (if prior data is available); 3) providing frequent feedback; and 4) providing computerized feedback.
TE D
Conversely, feedback characteristics that attenuated its effect included: 1) a focus on praise or discouragement (which directs attention towards the self and away from the task) and 2) providing verbal feedback alone. The effect of the feedback was also
EP
influenced by the nature of the task and personality and situational variables (see Table 1). A more recent meta-analysis using the Feedback Intervention Theory framework on
AC C
studies in the health care literature yielded similar findings [14].
3. How effective is A&F? What can be done to make A&F more effective?
The study of A&F has continued and since 1995, the literature on its effectiveness in health care has burgeoned [15]. The most recent Cochrane review comprises 140
6
ACCEPTED MANUSCRIPT
randomized trials of A&F in a health care setting and found that overall, A&F leads to a modest improvement in provider performance with a median 4.3% absolute increase in
RI PT
compliance with desired practice (IQR: 0.5 – 16%). However, results from individual trials continue to be heterogeneous with examples of both “positive” and “negative”
trials in the literature [9]. The reasons for these varied results are likely multifactorial,
SC
ranging from differing (and often poor) trial designs, the nature of the targeted
behaviour, the potential for variation in the A&F interventions across studies, and the
M AN U
context in which it was studied. In a pre-specified exploratory analysis, the Cochrane review found that A&F is more effective at improving quality of care when baseline performance is low, when feedback is provided more than once, when it is delivered by a trusted source (e.g. a colleague or supervisor), when it is delivered in both verbal and
TE D
written formats, when the goal is to decrease current behaviour, and when it includes explicit targets and action plans [9].
EP
Other systematic reviews of A&F in health care settings inform us about additional factors that make A&F more effective. One review examined feedback using medical
AC C
registries and found that feedback targeting process measures to be generally more effective than that targeting outcome measures [16]. A second study identified characteristics of Veterans Affairs Medical Centers with high and low compliance with clinical practice guidelines [17]. In both studies, facilitators of effective feedback included data characteristics (trust in the data [16]and individualized data [16, 17]), motivation of recipients [16], timeliness [17], non-punitiveness [17], and organizational
7
ACCEPTED MANUSCRIPT
factors [16] (e.g., quality improvement infrastructure). By identifying features of effective A&F, these reviews contribute to the growing understanding of how A&F
RI PT
works and highlight its potential mechanisms of action [18].
Although in many ways still in its infancy, the science of A&F is advancing and
SC
knowledge is accumulating. It is clear that A&F can be effective in improving physician performance. Thoughtful implementation includes careful design and delivery of the
M AN U
audit (maximizing the validity of the data, e.g., case-mix adjustment) and feedback (attention to the way data is presented, including choice of comparisons, and attention to how it is delivered, e.g., a trusted source), careful consideration of what aspect of physician performance or physician behaviour is targeted for change and inclusion of
TE D
targets, goals and an action plan [10]. The desired action (i.e., do more, do less or for interpretation/information) should be clear to the recipient. Implementation of a new A&F intervention should include a plan for evaluation and refinement to ensure
EP
alignment with goals. There are useful, evidence-based resources that can provide guidance to leaders in colonoscopy who wish to design and implement A&F
AC C
interventions [19, 20].
4. Can A&F be used for colonoscopy improvement?
The use of A&F for colonoscopy presents unique challenges. Colonoscopy is a complex task; arguably it is comprised of multiple “sub-tasks”, which can be broadly categorized
8
ACCEPTED MANUSCRIPT
as either cognitive or motor [21]. Motor or technical tasks include positioning the patient, handling and advancing the colonoscope, loop reduction, withdrawal and
RI PT
inspection techniques as well as polypectomy [21, 22]. Some have separated the cognitive aspects of colonoscopy into content area knowledge, consisting of pre-
procedure, insertion, withdrawal and diagnosis (including lesion recognition), treatment
SC
(including lesion removal), post-procedure and complications, and non-technical skills, which include analytical and personal behaviour [22]. Other non-technical skills are
M AN U
considered “social” and include communication and team-work [22].
Application of the Feedback Intervention Theory framework to the case of colonoscopy is informative. First of all, it is important to consider the overall “task”: colonoscopy is
TE D
physical and is clearly complex, two features which have been shown to attenuate the impact of feedback [13]. Figure 1 applies Feedback Intervention Theory to an example from colonoscopy and illustrates reasons that A&F might be less effective in this context
EP
using the example of cecal intubation: A&F can be expected to modify behaviour (e.g. attending a skills improvement course or practicing skills) but cannot reasonably be
AC C
expected to modify the motor skills required to insert the colonoscope. The challenges of A&F when applied to procedures such as colonoscopy are reflected in the fact that most recent meta-analyses of A&F exclude trials of physician procedure performance entirely [9, 14, 23].
9
ACCEPTED MANUSCRIPT
The cognitive tasks involved in colonoscopy, in particular the content-area knowledge, may be amenable to this intervention as feedback appears to augment memory tasks
RI PT
[13]. Furthermore, a “performance gap” for adenoma detection may motivate the recipient to examine the colon more carefully upon withdrawal. Hence, in theory, A&F may be useful to improve specific aspects of colonoscopy such as bowel preparation,
SC
adherence to surveillance guidelines and adenoma detection rates (ADR). However, as noted above, A&F alone should not be expected to improve colonoscopy technical
M AN U
tasks/skills such as those required for safe and effective cecal intubation or polypectomy. Rather, A&F’s expected mechanism of action if a technical “performance gap” were identified would be to motivate the recipient to take additional steps to improve their technical skills. Additional resources such as hands-on colonoscopy
TE D
courses [22, 24], detailed educational videos [25], or multimedia presentations [26] might be useful. Feedback facilitated by a trusted supervisor or peer (see section 6 below) may be particularly useful in this context as it may help the recipient to
EP
understand and interpret their data and to develop an action plan for improvement [27], which should include identifying the appropriate resources to effect a change in
AC C
performance. It is important to point out that because this is a multistep process, A&F may be successful (i.e., it motivates the recipient to take steps to improve their motor skills) but performance may not be improved if the subsequent educational interventions are ineffective. Therefore, intermediate processes should be measured [28] in order to optimize the effectiveness of A&F in improving colonoscopy technical skills.
10
ACCEPTED MANUSCRIPT
An understanding of the sub-tasks of colonoscopy can be used to inform the design of
RI PT
A&F. Specific elements of A&F can be tailored to address individual sub-tasks; when doing so, it is important to anticipate how the recipient might respond to the A&F intervention. Failure to do so may have unintended consequences. For example,
SC
providing feedback on withdrawal time (time of colon inspection, from the intubation of the cecum to removal of colonoscope) in order to improve ADR without providing
M AN U
information on how to improve inspection of the colon (i.e., correct solution information) is at best, unlikely to be effective (see next section) or at worst, lead to “gaming” where endoscopists linger in the rectum in order to achieve acceptable
TE D
withdrawal times [29].
5. How effective is A&F for colonoscopy improvement? There are relatively few studies of A&F interventions for colonoscopy. Earlier studies,
EP
which tended to be quasi-experimental (i.e., tracked one group using a “pre/post” design) [30], often focused on the measurement and reporting of a single performance
AC C
measure--withdrawal time--and overall, results were disappointing. A systematic review of these studies [31] found that the measurement of withdrawal time alone (either in real time or as an aggregate measure over a number of colonoscopies) was generally ineffective. The one study that did show a benefit also incorporated an educational intervention providing tips on good colonoscopy inspection technique (i.e., adequate
11
ACCEPTED MANUSCRIPT
insufflation, repetitive examination of colonic segments, torqueing maneuvers to
RI PT
improve visualization between folds) [32].
Unfortunately, the field has not advanced significantly since the systematic review in
2011. Subsequently published studies testing the effectiveness of A&F specifically (i.e.,
SC
comparing A&F to usual care) continue to use quasi-experimental designs [33-35],
generally testing A&F in a small number of endoscopists at a single centre over a short
M AN U
period of time [33, 35]. While these studies have shown a significant benefit of A&F, the lack of a control group makes the findings questionable as they could also be due to secular trends.
TE D
To date, there are two published randomized controlled trials that include the use of A&F [36, 37] but did not compare it to a control group that did not receive A&F. In these trials, A&F was provided to both arms of the study and the intervention groups
EP
received supplementary interventions. In one study [36], physicians in the intervention group who received 2 hours of training on methods and techniques to improve ADR and
AC C
on polyp classification methods experienced a significant improvement in their ADR (11% vs -1% change from baseline in the controls). The second study [37] comprised leaders at colonoscopy screening centre in Poland whose ADR at baseline was lower than 25% (n=38). The intervention group received a 3-4 day hands on course on teaching high quality colonoscopy with the hypothesis that this intervention would diffuse into their home screening centre leading to improvements in care. The study
12
ACCEPTED MANUSCRIPT
showed a significant improvement in the primary outcome (change in the leaders’ ADR from baseline) in the intervention group (8.2%) compared to controls (1.1%). These
RI PT
results suggest that A&F is effective for colonoscopy improvement when paired with a supplementary educational intervention that provides correct solution information but do not address its effectiveness when implemented alone. Results from endoscopy
SC
studies published to date are summarized in Table 2.
M AN U
6. Improving the effectiveness of A&F in colonoscopy: future directions It has been noted that research on A&F is “stagnating” and that there is a need to incorporate best practices in A&F and address gaps in the literature in future work [10]. Leading professional societies and others have published comprehensive lists of pre-,
TE D
intra- and post-procedural colonoscopy quality indicators [38-41]. While there is evidence to support some of these indicators, many are based on expert opinion only. Indicators used in A&F for colonoscopy improvement should be carefully selected to
EP
ensure that they are meaningful. Unfortunately, currently, ease of measurement may drive selection. For example, polyp detection rate (PDR) is often used instead of ADR as
AC C
the former can often be captured using claims data whereas the latter generally requires manual reconciliation of the endoscopy record with pathology results [38]. Although there is data to support the association between lower PDR and lower rates of missed CRCs [5], there is arguably greater potential for gaming with PDR than ADR [38].
13
ACCEPTED MANUSCRIPT
As another example, cecal intubation rate is often reported but it may be more meaningful to have a measure of how often the cecum is reached while keeping the
RI PT
patient safe and comfortable (i.e., without excess sedation and with good comfort scores) [42]. It is important to ensure that as much as possible, the data and design of
the A&F produce the intended response in the recipient thereby achieving the goals of
SC
providing the feedback. An understanding of the factors that contribute to variation in indicators such as ADR [43] is also important as these factors can be incorporated into
M AN U
the design (i.e., correct solution information) of A&F interventions.
The medical education and broader behavioral science literature offer strategies that may enhance the impact of A&F interventions, including the use of social norms,
TE D
pledges/commitment devices, defaults (opt-out vs. opt-in approaches to uptake of recommended behaviour) [44] and facilitated feedback [27]. Social norms are the rules or standards that influence the behavior of members of a group [45]. There are two
EP
types of social norms. An individual is influenced by descriptive social norms when s/he tries to align his or her behavior with the behavior of the rest of the group in order to
AC C
avoid being labeled an outlier. An individual is influenced by prescriptive (or injunctive) social norms when s/he tries to align his or her behavior with the behavior prescribed by a recognizable authority [46]. A&F interventions can incorporate descriptive social norms by using peer comparator information that will help the individual physician understand where his or her performance stands vis-à-vis that of his or her peers. Prescriptive social norms can be incorporated by referencing performance targets set by
14
ACCEPTED MANUSCRIPT
credible professional bodies that will help the physician know whether s/he is in
RI PT
compliance with accepted recommendations.
The use of pledges or commitment devices has long been studied in behavioural science and has been successful in continuing medical education [47, 48]. It highlights that once
SC
a commitment is made, an individual will experience personal and interpersonal
pressures to honour that commitment [49]. In the context of A&F, the intervention
M AN U
could include use of a pledge whereby the physician commits to attend a course or seek other educational opportunities to help him or her improve his or her colonoscopy performance. The pledge may make it more likely that the physician will attend the
TE D
course.
Behavioral theory suggests that inertia plays an important role in influencing how individuals behave –when confronted with options, individuals will often go with the
EP
default option because doing so is the easy thing to do (i.e. it requires no effort on their part). This insight has led behavioral scientists to recommend making the preferred
AC C
option the default option when designing programs [50]. Purposeful use of default options can help increase the chances of success of an A&F intervention. For example, physicians below a certain threshold-score on their A&F can be prescheduled to attend a colonoscopy skills improvement course and given the option to opt-out if they choose to (instead of asking them to make the effort to schedule the course themselves). The
15
ACCEPTED MANUSCRIPT
advantage of default options is that they influence decisions without restricting choice
RI PT
[50].
Facilitated feedback involves engaging recipients of A&F in a reflective conversation
about their performance and the feedback itself in order to gain a deeper understanding
SC
of both. One model of facilitated feedback developed for physicians specifically employs a 4 step approach: building rapport and relationship, exploring reactions and
M AN U
perceptions of the report, exploring understanding of the content of the report and coaching for performance change [27]. Facilitated feedback can help mitigate a response known as ‘cognitive dissonance’, which refers to the struggle of maintaining two conflicting beliefs simultaneously. For example, if A&F suggests substandard
TE D
performance, the recipient must wrestle with 2 beliefs: that s/he is underperforming and the self-perception that s/he is a capable and competent practitioner [51]. Because it is easier to question the validity of the data than to question oneself, the recipient
EP
may resolve this conflict by discounting the data. Facilitation of feedback, particularly from a trusted source, may help the recipient to set practice improvement goals rather
AC C
than reject the data altogether [27, 52]. A&F presents the recipient with data that identifies a performance gap, facilitated feedback helps to close the circle leading to improvement in patient care by validating the performance gap (Figure 2).
7. Summary and conclusion
16
ACCEPTED MANUSCRIPT
A&F has the potential to improve endoscopist performance thereby contributing to the improvement of quality of colonoscopy. Careful implementation of A&F, based on a
RI PT
sound understanding of its mechanism, is necessary to ensure that the intervention is thoughtfully designed (see Box 1) to maximize the likelihood of achieving its goals.
More research in A&F to improve the quality of colonoscopy is needed (Box 2) to inform
AC C
EP
TE D
Conflict of interest: None
M AN U
performance improvement intervention.
SC
implementation efforts and to maximize the benefit from this promising endoscopist
17
ACCEPTED MANUSCRIPT
References 1. Rabeneck L, Rumble RB, Axler J, Smith A, Armstrong D, Vinden C et al. Cancer Care
RI PT
Ontario Colonoscopy Standards: standards and evidentiary base. Can J Gastroenterol 2007;21 Suppl D:5D-24D.
2. Tinmouth J, Baxter NN, Li Q, Paszat L, Patel J, Rabeneck L. Measuring the quality of
SC
colonoscopy delivered to an entire population in Ontario, Canada. Gastrointest Endosc 2014;79(5):AB118.
M AN U
3. Corley DA, Jensen CD, Marks AR, Zhao WK, Lee JK, Doubeni CA et al. Adenoma detection rate and risk of colorectal cancer and death. N Engl J Med 2014;370(14):1298306.
4. Kaminski MF, Regula J, Kraszewska E, Polkowski M, Wojciechowska U, Didkowska J et
TE D
al. Quality indicators for colonoscopy and the risk of interval cancer. N Engl J Med 2010;362(19):1795-803.
5. Baxter NN, Sutradhar R, Forbes SS, Paszat LF, Saskin R, Rabeneck L. Analysis of
EP
administrative data finds endoscopist quality measures associated with postcolonoscopy colorectal cancer. Gastroenterology 2011;140(1):65-72.
AC C
6. Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess 2004;8(6):1-84. 7. Joint Advisory Group on GI Endoscopy. http://www.thejag.org.uk. Accessed March 10, 2016.
18
ACCEPTED MANUSCRIPT
8. Quality Management Partnership: Colonoscopy. https://www.qmpontario.ca/colonoscopy/. Accessed March 10, 2016.
RI PT
9. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD et al. Audit and feedback: effects on professional practice and healthcare outcomes Cochrane Database Syst Rev 2012;6(CD000259).
SC
10. Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ et al. No more ‘business as usual’with audit and feedback interventions: towards an agenda for a reinvigorated
M AN U
intervention. Implement Sci 2014;9(1):14.
11. Shute VJ. Focus on formative feedback. Rev Educ Res 2008;78(1):153-89. 12. Colquhoun HL, Brehaut JC, Sales A, Ivers N, Grimshaw J, Michie S et al. A systematic review of the use of theory in randomized controlled trials of audit and feedback.
TE D
Implement Sci 2013;8(1):66.
13. Kluger AN, DeNisi A. The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory.
EP
Psychol Bull 1996;119(2):254-84.
14. Hysong SJ. Meta-analysis: audit & feedback features impact effectiveness on care
AC C
quality. Med Care 2009;47(3):356. 15. Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O’Brien MA, French SD et al. Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care. J Gen Intern Med 2014;29:1534-41.
19
ACCEPTED MANUSCRIPT
16. van der Veer SN, de Keizer NF, Ravelli AC, Tenkink S, Jager KJ. Improving quality of care. A systematic review on how medical registries provide information feedback to
RI PT
health care providers. Int J Med Inform 2010;79(5):305-23. 17. Hysong SJ, Best RG, Pugh JA. Audit and feedback and clinical practice guideline adherence: Making feedback actionable. Implement Sci 2006;1(9):5-3.
SC
18. Foy R, Eccles M, Jamtvedt G, Young J, Grimshaw J, Baker R. What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review.
M AN U
BMC Health Serv Res 2005;5(1):50.
19. Brehaut JC, Colquhoun HL, Eva KW, Carroll K, Sales A, Michie S et al. Practice Feedback Interventions: 15 Suggestions for Optimizing Effectiveness. Ann Intern Med 2016;164:435-41.
TE D
20. French SD, Green SE, O’Connor DA, McKenzie JE, Francis JJ, Michie S et al. Developing theory-informed behaviour change interventions to implement evidence into practice: a systematic approach using the Theoretical Domains Framework.
EP
Implementat Sci 2012;7(1):38.
21. Sedlack RE. Training to competency in colonoscopy: assessing and defining
AC C
competency standards. Gastrointest Endosc 2011;74(2):355-66. 22. Robertson DJ, Kaminski MF, Bretthauer M. Effectiveness, training and quality assurance of colonoscopy screening for colorectal cancer. Gut 2015;64(6):982-90. 23. Gardner B, Whittington C, McAteer J, Eccles M, Michie S. Using theory to synthesise evidence from behaviour change interventions: The example of audit and feedback. Soc Sci Med 2010(70):1618-25.
20
ACCEPTED MANUSCRIPT
24. Thomas-Gibson S, Williams CB. Colonoscopy training—new approaches, old problems. Gastrointest Endosc Clin N Am 2005;15(4):813-27.
RI PT
25. American Society fornGastrointestinal Endoscopy (ASGE). Endoscopic Learning Library: Endoscopy’s most comprehensive CME video collection. 2015.
http://www.asge.org/uploadedFiles/Public_E-Blast_PDFs/ASGE_ELL_brochure_2015.pdf
SC
26. Marx R, le Clercq C, Bogie R, Winkens B, Kruimel J, Conchillo JM et al. Training on detection and resection of nonpolypoid colorectal neoplasms reduces the
M AN U
postcolonoscopy colorectal cancer rate. Gastroenterology 2015;148(4):S-96. 27. Sargeant J, Lockyer J, Mann K, Holmboe E, Silver I, Armson H et al. Facilitated reflective performance feedback: developing an evidence-and theory-based model that builds relationship, explores reactions and content, and coaches for performance
TE D
change (R2C2). Acad Med 2015;90(12):1698-706.
28. Grimshaw JM, Eccles MP, Walker AE, Thomas RE. Changing physicians' behavior: what works and thoughts on getting more things to work. J Contin Educ Health Prof
EP
2002;22(4):237-43.
29. Inadomi JM. In search of quality colonoscopy. Gastroenterology 2008;135(6):1845-7.
AC C
30. Harris AD, McGregor JC, Perencevich EN, Furuno JP, Zhu J, Peterson DE et al. The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc 2006;13(1):16-23. 31. Corley DA, Jensen CD, Marks AR. Can we improve adenoma detection rates? A systematic review of intervention studies. Gastrointest Endosc 2011;74(3):656-65.
21
ACCEPTED MANUSCRIPT
32. Barclay RL, Vicari JJ, Greenlaw RL. Effect of a time-dependent colonoscopic withdrawal protocol on adenoma detection during screening colonoscopy. Clin
RI PT
Gastroenterol Hepatol 2008;6(10):1091-8. 33. Keswani RN, Yadlapati R, Gleason KM, Ciolino JD, Manka M, O'Leary KJ et al.
Physician report cards and implementing standards of practice are both significantly
SC
associated with improved screening colonoscopy quality. Am J Gastroenterol 2015;110(8):1134-9.
M AN U
34. Fraser AG, Gamble GD, Rose TR, Dunn JP. Colonoscopy audit over 10 years--what can be learnt? The New Zealand Medical Journal (Online) 2013;126(1382). 35. Kahi CJ, Ballard D, Shah AS, Mears R, Johnson CS. Impact of a quarterly report card on colonoscopy quality measures. Gastrointest Endosc 2013;77 (6 ):925-31.
TE D
36. Coe SG, Crook JE, Diehl NN, Wallace MB. An endoscopic quality improvement program improves detection of colorectal adenomas. Am J Gastroenterol 2013;08(2):219-26.
EP
37. Kaminski MF, Anderson J, Valori R, Kraszewska E, Rupinski M, Pachlewski J et al. Leadership training to improve adenoma detection rate in screening colonoscopy: a
AC C
randomised trial. Gut 2016;65:616-24. 38. Rex D, Schoenfeld P, Cohen J, Pike I, Adler D, Fennerty M et al. Quality indicators for colonoscopy. Gastrointest Endosc 2015;81(1):31-53. 39. Tinmouth J, Kennedy E, Baron D, Burke M, Feinberg S, Gould M. Guideline for colonoscopy quality assurance in Ontario. Cancer Care Ontario 2013.
22
ACCEPTED MANUSCRIPT
40. European Commission. European guidelines for quality assurance in colorectal cancer screening and diagnosis. First ed. Luxembourg: International Agency for Research
RI PT
on Cancer 2010. 41. Chilton A, Rutter M. Quality assurance guidelines for colonoscopy. Sheffield: NHS Cancer Screening Programmes 2011.
SC
42. Valori RM, Gavin D, Damery S, Swarbrick E, Donnelly M, Anderson J et al. Sa1400 A
composite measure of colonic intubation (Circ) is better able to distinguish performance
M AN U
of colonoscopy and is associated with higher polyp detection rates. Gastrointest Endosc 2014;79(5):AB197-AB8.
43. Atkins L, Hunkeler EM, Jensen CD, Michie S, Lee JK, Doubeni CA et al. Factors influencing variation in physician adenoma detection rates: a theory-based approach for
TE D
performance improvement. Gastrointest Endosc 2016;83:617-26. 44. Dolan P, Hallsworth M, Halpern D, King D, Metcalfe R, Vlaev I. Influencing behaviour: The mindspace way. J Econ Psychol 2012;33(1):264-77.
EP
45. Cialdini RB, Trost MR. Social influence: Social norms, conformity and compliance. In: D. T. Gilbert STF, & G. Lindzey, editor. The handbook of social psychology. Boston:
AC C
McGraw-Hill; 1998.
46. Cialdini RB, Reno RR, Kallgren CA. A focus theory of normative conduct: recycling the concept of norms to reduce littering in public places. J Pers Soc Psychol 1990;58(6):1015.
23
ACCEPTED MANUSCRIPT
47. Domino FJ, Chopra S, Seligman M, Sullivan K, Quirk ME. The impact on medical practice of commitments to change following CME lectures: A randomized controlled
RI PT
trial. Med Teach 2011;33(9):e495-e500. 48. Wakefield J, Herbert CP, Maclure M, Dormuth C, Wright JM, Legare J et al.
Health Prof 2003;23(2):81-92.
SC
Commitment to change statements can predict actual change in practice. J Contin Educ
49. Cialdini RB. Influence: Science and practice Fourth Editiion. New York:
M AN U
HarperCollinsCollegePublishers; 2001.
50. Halpern SD, Ubel PA, Asch DA. Harnessing the power of default options to improve health care. N Engl J Med 2007;357(13):1340-4.
51. Eva KW, Regehr G. Effective feedback for maintenance of competence: from data
TE D
delivery to trusting dialogues. Can Med Assoc J 2013;185(6):463-4. 52. Sargeant JM, Mann KV, Van der Vleuten CP, Metsemakers JF. Reflection: a link between receiving and using assessment feedback. Adv Health Sci Educ 2009;14(3):399-
EP
410.
53. Lin OS, Kozarek RA, Arai A, Gluck M, Jiranek GC, Kowdley KV et al. The effect of
AC C
periodic monitoring and feedback on screening colonoscopy withdrawal times, polyp detection rates, and patient satisfaction scores. Gastrointest Endosc 2010;71(7):1253-9. 54. Harewood GC, Murray F, Winder S, Patchett S. Evaluation of formal feedback on endoscopic competence among trainees: the EFFECT trial. Ir J Med Sci 2008;177(3):2536.
24
ACCEPTED MANUSCRIPT
55. Sawhney MS, Cury MS, Neeman N, Ngo LH, Lewis JM, Chuttani R et al. Effect of institution-wide policy of colonoscopy withdrawal time ≥ 7 minutes on polyp detection.
RI PT
Gastroenterology 2008;135(6):1892-8. 56. Harewood GC, Petersen BT, Ott BJ. Prospective assessment of the impact of
feedback on colonoscopy performance. Aliment Pharmacol Ther 2006;24(2):313-8.
SC
57. Shaukat A, Oancea C, Bond JH, Church TR, Allen JI. Variation in detection of
adenomas and polyps by colonoscopy and change over time with a performance
M AN U
improvement program. Clin Gastroenterol Hepatol 2009;7(12 ):1335-40. 58. Imperiali G, Minoli G, Meucci GM, Spinzi G, Strocchi E, Terruzzi V et al. Effectiveness of a continuous quality improvement program on colonoscopy practice. Endoscopy
AC C
EP
TE D
2007;39(04 ):314-8.
25
ACCEPTED MANUSCRIPT
Box 1: Practice Points 1.
Physicians are poor at self-assessment and A&F can help physicians identify
2.
RI PT
areas of practice in need of improvement. A&F colonoscopy initiatives must be implemented thoughtfully. A framework that can guide these types of initiatives [20] is applied to colonoscopy below:
SC
a. What is the colonoscopy performance-gap you are trying to change? What behaviour needs to change to close this gap?
behaviour change?
M AN U
b. What are the levers (barriers and enablers) that would lead to this
c. What components could be included in the A&F initiative that could help overcome the barriers and enhance the enablers to change? A&F colonoscopy initiatives should anticipate and pre-emptively address
TE D
3.
common recipient reactions: rejecting the data or attributing their poor performance to their patient case-mix (i.e., their patients are different from
4.
EP
others).
The desired effect of each quality indictor in an A&F initiative should be clear to
AC C
the recipient (i.e., do more, do less, or indicator provided for information/interpretation).
5.
A&F colonoscopy initiatives should strive to incorporate meaningful quality indicators, rather than simply reporting data that is easy to measure.
26
ACCEPTED MANUSCRIPT
6.
A&F alone may be insufficient to improve certain aspects of colonoscopy performance, such as technical skills. Using A&F to motivate recipients to
RI PT
engage with other resources such as colonoscopy skills improvement courses, educational videos or multimedia presentations may be more effective. 7.
Ongoing evaluation of A&F colonoscopy initiatives is important to assess
SC
effectiveness and alignment with goals. Leaders in colonoscopy should initiate A&F in their local practice contexts but expect that initial attempts will be
M AN U
imperfect. Leaders should incorporate evaluation and iterative revisions of the A&F initiative into the implementation plan. 8.
There are useful, evidence-based resources that can provide guidance to leaders in colonoscopy who wish to design and implement A&F interventions
TE D
[19, 20]
Box 2: Research Agenda
There is a need to move beyond the study of A&F alone to improve physician
EP
1.
performance [10]. The social sciences provide insight into
AC C
ancillary/complementary interventions to study. Future study should consider formal evaluation of A&F interventions that incorporate facilitated feedback, social norms, pledges, and use of defaults [44].
2.
Research is needed to develop meaningful quality indicators to measure colonoscopy performance and to assess whether the use of accepted, existing indicators in A&F is leading to meaningful change in colonoscopy performance.
27
ACCEPTED MANUSCRIPT
3.
Colonoscopy-specific A&F elements, which are specifically designed to target colonoscopy sub-tasks such as technical tasks (e.g., colonoscope insertion) and
RI PT
cognitive tasks (e.g., content area knowledge, communication), should be
AC C
EP
TE D
M AN U
SC
developed and evaluated.
28
ACCEPTED MANUSCRIPT
Figure legends Figure 1. Simple schematic of Feedback Intervention Theory, adapted from Kluger and
RI PT
DeNisi [13], using cecal intubation as an example. According to the theory, feedback is successful if it shifts the recipient’s attention to the performance gap; 3 key factors
contribute to this success (see boxes with: --- ): characteristics of the feedback, task
SC
characteristics and situational/personality variables. In this example, the behaviour
change resulting from audit and feedback is that the recipient takes steps to improve
M AN U
(take a course, practice). Performance improvement depends on both successful behaviour change and the effectiveness of the course. [ ], component of theory.
Figure2.Critical elements necessary for audit and feedback to lead to improvement in
TE D
patient care. A motivated recipient (1) is an essential first step. Feedback highlights a performance-gap (2). To prevent the recipient from disregarding the data, the performance-gap needs to be validated (3), for instance through facilitated feedback or
EP
by emphasizing the accuracy of the audit. Finally, tangible changes to patient care (4)
AC C
can be implemented to address the performance-gap.
29
ACCEPTED MANUSCRIPT
Tables
RI PT
Table 1. Characteristics of feedback, nature of the task and situational and personality variables that influenced the effectiveness of feedback according to Kluger and DeNisi [13]
SC
M AN U
TE D
Situational variables Personality variables
EP
Tasks
AC C
Feedback Characteristics
Effect on feedback Augmented Attenuated • information about what • praise needs to change in order • discouragement to improve performance • verbal feedback only (“correct solution information”) • trend in performance • frequent feedback • computerized feedback • memory tasks • physical tasks • complex tasks • “following the rules” tasks • goal-setting intervention • self-esteem threatened
30
ACCEPTED MANUSCRIPT
Fraser et al, 2013 [34]
6 endoscopists, ≥15 procedures in each study period / 928 screening colonoscopies in average risk patients aged 50+ QuasiEndoscopists (# not experimental provided), >100 (1G, Single colonoscopies over period), two study period/ 67570 private diagnostic, family endoscopy units, history screening, & New Zealand surveillance colonoscopies
ADR
Pre: Nov 2012-Mar 2013 Intervention 1 (Int1): Apr 2013-March 2014 Intervention 2 (Int2): Apr 2014-Oc2014 CIR and ADR, adjusted for physician, patient age, and sex / Pre: July – Dec 2008 Int: Apr 2009-Mar 2011
TE D
Quasiexperimental (1G, Pre/Int) / Single centre, tertiary care, USA.
EP
Kahi et al, 2013 [35]
20 endoscopists, >15 colonoscopies per month / 12894 screening colonoscopies
Primary outcome / Study period
Intervention(s)%
Findings
Int1: Individualized annual A&F (ADR, WT) derived from EMR with peer comparison (10th, median, 90th percentile)
ADR after Intervention #1: 28% (Pre) 31% (Int1)*
Individualized quarterly A&F (BPrep, pre procedure assessment, CIR, WT, ADR) derived from EMR with anonymized peer comparison Annual A&F meetings, other details not provided
CIR: 95.6% (Pre) 98.1% (Int)* ADR: 44.7% (Pre) 53.9% (Int)*
SC
Studies testing A&F vs usual care Keswani et Quasi$ al, 2015 [33] experimental (1G, Pre/Int) / Single centre, tertiary care, USA.
Study cohort / Procedures
M AN U
Design / Setting
AC C
Author, year
RI PT
Table 2. Selected studies of audit and feedback (A&F) for colonoscopy performance improvement organized by studies evaluating audit and feedback vs usual care and studies of A&F plus supplemental interventions compared to A&F alone or to usual care.
CIR, insertion time, WT, PDR Study period: 19992010
CIR: 96.3% (1999) 99.0% (2010)* Insertion time: 7.5 min (2001) 8.9 min (2010)* PDR: 29% (1999)
31
ACCEPTED MANUSCRIPT
4 gastroenterology fellowship trainees / 581 colonoscopies
Quasiexperimental (1G, Single period) / Single centre,
AC C
Sawhney et al, 2008 [55]
42 endoscopists, 23,910 colonoscopies
Individualized written A&F (WT, PDR, patient satisfaction) every 3-6 months anonymized peer comparison
SC
WT, PDR, ADR, and patient satisfaction scores
Pre: July 2005-June 2006
M AN U
Randomized controlled trial / Single centre, tertiary care, Ireland
10 gastroenterologists / 1391 asymptomatic adult screening colonoscopies
Int: July 2006-Apr 2007 CIR and PDR Pre: July-Aug 2006 Int: Oct 2006-Jan2007
EP
Harewood et al, 2008 [54]
Quasiexperimental (1G, Pre/Int) / Single centre, tertiary care, USA
TE D
Lin et al, 2010 [53]
RI PT
49% (2010)*
% WT ≥7 minutes, PDR Pre: Prior to Feb 2006 Study period: Feb 2006June 2007
Individualized written and graphed monthly A&F (CIR, PDR) reviewed at in person meeting with supervisor derived from EMR with anonymized peer comparison
Monthly individualized A&F (WT) derived from EMR
WT: 5.6 min (2001) 6.6 min (2010)* WT: 6.57 min (Pre) 8.07 min (Int)* Polyp detection: 33.1% (Pre) 38.1% (Int) ADR: 19.6% (Pre) 22.7% (Int) CIR: 72.9% (Pre) 83.4% (Int) in intervention vs. 78% (Pre) 71.9% (Post) in control* PDR: 12.9% (Pre) 18% (Int) in intervention vs. 16.9% (Pre) 19.6% (Post) in control % WT ≥7-minute#: 65% (1st month) almost 99% (last month)* PDR#: 48% (1st month)
32
ACCEPTED MANUSCRIPT
almost 55% (last month)
58 endoscopists / median of 310.5 outpatient colonoscopies/year per endoscopist from during study period (Pre & Int)
CIR, insertion time and WT
Individualized emailed quarterly A&F (CIR, insertion time, WT) derived from EMR with anonymized peer comparison
SC
Quasiexperimental (1G, Pre/Int), Single centre, USA
Pre: Jan 2004-Dec 2004 Int: Jan 2005-Dec 2005
M AN U
Harewood et al, 2006 [56]
RI PT
tertiary care, USA
AC C
EP
TE D
Studies of A&F + supplemental interventions compared to A&F alone or to usual care Kaminski et Randomized 38 endoscopists ADR Individualized mailed al, 2016 [37] controlled trial / (screening centre & emailed A&F (ADR & 40 colonoscopy leaders), ≥30 Pre: Jan 2011-Dec 2011 CIR) derived from EMR screening screening Early post-intervention with anonymized peer centres, Poland colonoscopies and (Early PI): June 2012comparison, issued at ADR <25% in 2011 / Dec 2012 end of Pre and Early 24582 screening Late post-intervention post-intervention colonoscopies (Late PI): Jan 2013-Dec phases + Train2013 Colonoscopy-Leaders (TCLs) programme (pretraining assessment, 2 days of hands-on training, post-training evaluation and feedback) vs
CIR: 96.3% >97.2% Insertion time: 10.6min 9.5min* WT: 9.1min 8.9min
ADR (early): 17.4% (Pre) 25.6% (Early PI) in intervention vs 18.5% (Pre) 19.6% (Early PI) in control* ADR (late): 17.4% (Pre) 23.9% (Late PI) in intervention vs 18.5% (Pre) 20.8% (Late PI) in control *
33
ACCEPTED MANUSCRIPT
ADR
M AN U
20 endoscopists, >15 colonoscopies per month / 12894 screening colonoscopies
Pre: Nov 2012-Mar 2013 Intervention 1 (Int1): Apr 2013-March 2014 Intervention 2 (Int2): Apr 2014-Oc2014
Randomized controlled trial / Single centre, tertiary care, USA
15 endoscopists, ≥10 procedures per month/ 2400 screening, surveillance, & diagnostic colonoscopies
AC C
Coe et al, 2013 [36]
EP
TE D
Keswani et Quasi$ al, 2015 [33] experimental (1G, Pre/Int) / Single centre, tertiary care, USA.
SC
RI PT
individualized mailed & emailed A&F (ADR & CIR) derived from EMR with anonymized peer comparison, issued at end of Pre and Early post-intervention phases alone Int1: Individualized ADR after Intervention annual A&F (ADR, WT) #2: 31% (Int1) 39% derived from EMR (Int2)* with peer comparison (10th, median, 90th percentile)
ADR, adjusted for patient age, sex, BPrep and indication Pre: Feb 2010-Jan 2011 Intervention: Feb 2011Apr 2011 Post: May 2011-Dec 2011
Int2: Failure to meet standards of practice (>5min WT and ADR>20%) training, endoscopy restrictions Individualized A&F (ADR & WT) derived from prospectively collected data, issued at end of Pre phase + 2 X 1 hour in person sessions: videos, still images, literature + post-test for
ADR: 36% (Pre) 47% (Post) in intervention vs. 36% (Pre) 35% (Post) in control*
34
ACCEPTED MANUSCRIPT
RI PT
ADR, adjusted for patient age, sex and BPrep
SC
43 gastroenterologists, >100 colonoscopies per year of study, >5000 colonoscopies over career/ 47253 1st time screening colonoscopies
M AN U
Quasiexperimental (1G, Pre/Int), community based, 5 ambulatory endoscopy centers, USA
Pre: Jan 2004 to Dec 2004 Int: Jan 2005-Dec2006
Imperiali et al, 2007 [58]
Quasiexperimental (1G, Pre/Int) Single centre, secondary care,
AC C
EP
TE D
Shaukat et al, 2009 [57]
8 gastroenterologists; 10,705 colonoscopies
competency vs. A&F (ADR & WT) derived from prospectively collected data, issued at end of Pre phase alone Individualized mailed biannual A&F (CIR, ADR) with peer comparison. Further interventions over study period: 1) blinded group ADR review, 2) unblinded individual ADR review at partnership meetings, 3) education re literature of ADR and WT standards, 4) poor performers met w/ practice leaders 5) 1% financial penalty for failure to meet standards Individualized biannual A&F (CIR, PDR) via chart audit. Blinded group A&F review at departmental
CIR and PDR Pre: Jan 2001-Dec 2001 Int: Jan 2002-Dec 2004
ADR: No significant change (data not provided)
CIR: 84.6% (Pre) 93.1% (Int)* Polyp detection: 33.8% (Pre) 33% (Int)
35
ACCEPTED MANUSCRIPT
meetings and action plan developed: (1) sedation mandated, (2) less experienced/ low performers more examinations, (3) less skilled examiners periodically supervised by veteran examiners, (4) incomplete another endoscopist attempts; and (5) physicians with low PDR met with unit head % A&F presentation and delivery details included where reported in original manuscript $ This study compared A&F to usual care (Int1) and A&F + supplemental intervention to A&F alone (Int2), therefore the study is listed twice, with relevant results reported in the appropriate sections of this table * p < 0.05, #monthly % regressed and significance determined based on slope of line 1G = one group ADR = adenoma detection rate BPrep = bowel preparation score CIR = cecal intubation rate PDR = polyp detection rate Pre / Int / Post= measurements prior to (Pre), during (Int) or after intervention (Post) WT = withdrawal time
AC C
EP
TE D
M AN U
SC
RI PT
Italy
36
AC C
EP
TE D
M AN U
SC
RI PT
ACCEPTED MANUSCRIPT
AC C
EP
TE D
M AN U
SC
RI PT
ACCEPTED MANUSCRIPT