- Email: [email protected]
Identifying effective computerized strategies to prevent drug–drug interactions in hospital: A user-centered approach
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 595–600
journal homepage: www.ijmijournal.com
Identifying effective computerized strategies to prevent drug–drug interactions in hospital: A user-centered approach Olivia Missiakos a,b , Melissa T. Baysari c,b,∗ , Richard O. Day b,a a
UNSW Medicine, UNSW, Sydney, Australia Department of Clinical Pharmacology & Toxicology, St Vincent’s Hospital, Sydney, Australia c Centre for Health Systems & Safety Research, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia b
a r t i c l e
i n f o
a b s t r a c t
Article history:
Background: Drug–drug interactions (DDIs) are an important and preventable cause of med-
Received in revised form
ication errors in hospitals. Recent developments in technology have seen new strategies
21 July 2014
emerge for preventing DDIs but these computerized strategies are rarely evaluated and are
Accepted 1 April 2015
typically implemented with little input from the individuals using them.
Keywords:
strategies available to assist in the identification and prevention of DDIs in hospitals.
Aim: To determine the opinions of both experts and users (prescribers) on computerized Drug–drug interaction
Method: Eight drug safety experts and 18 prescribers took part in semi-structured inter-
Alerts
views. Participants were asked about their confidence in identifying DDIs and their views
Alert fatigue
on potential computerized strategies to prevent DDIs.
Interviews
Results: No prescribers reported complete confidence in identifying dangerous DDIs, with junior prescribers appearing less confident than senior prescribers. Most prescribers believed that computerized alerts would be the most effective strategy for preventing DDIs, while experts were more critical of alerts. Conclusion: The lack of confidence displayed by prescribers in their ability to identify DDIs suggests that an appropriate strategy would be one that does not rely on individuals seeking out the information themselves. While a large number of problems related to DDI alert implementation have been reported in the literature (e.g. alert overload), prescribers appeared to be receptive to the idea of being alerted. By ensuring users are aware of the limitations of the system and involving them in DDI strategy design we expect greater use and satisfaction with the adopted strategy. © 2015 Elsevier Ireland Ltd. All rights reserved.
1.
Introduction
Drug–drug interactions (DDIs) are a preventable cause of medication errors in community and hospital settings and account
for 2.4–4.4% of hospital admissions [1,2]. They occur when two or more drugs are taken in combination that leads to a change in the activity of either or both drugs [3,4]. DDIs can result in adverse effects; commonly these include low blood pressure, bleeding or kidney damage [5]. Additionally, DDIs can lead to
∗ Corresponding author at: Centre for Health Systems & Safety Research, Level 6, 75 Talavera Rd, Macquarie University, NSW 2109, Australia. Tel.: +61 29850 2416; fax: +61 280886234. E-mail address: [email protected] (M.T. Baysari).
http://dx.doi.org/10.1016/j.ijmedinf.2015.04.001 1386-5056/© 2015 Elsevier Ireland Ltd. All rights reserved.
596
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 595–600
therapeutic failure, where one or both of the drugs are unable to achieve their desired clinical effect [5]. Research has shown that both prescribers and pharmacists are often unable to recognize potential DDIs [6,7]. Recent developments in technology have seen new strategies emerge to assist in DDI identification and prevention. In particular, alerts integrated into electronic prescribing systems (ePS) have frequently been adopted by hospitals in an attempt to minimize DDI occurrence [8,9]. To date, there is limited evidence demonstrating reductions in DDI errors or adverse drug events following DDI alert introduction, with evaluations typically comprising a review of the number of alerts generated and acted on by prescribers [10,11]. Two studies examined the impact of a single customized DDI alert on the concurrent ordering of two medications, but they report inconsistent findings [12,13] and in one case, introduction of a near hard-stop DDI alert resulted in unintended consequences (e.g. delays in appropriate treatment) [13]. Computerized DDI checking programs are also commonly discussed in the literature as a strategy to target DDI errors [14,15]. Prescribers enter medication names into the program, which then checks medication combinations for potential DDIs. The main difference between this strategy and an alert system is that software programs are typically voluntarily used and so are non-interruptive. Evaluation of DDI checking software usually includes an assessment of its ability to identify DDIs, most often in the form of an analysis of sensitivity and specificity, [16,17] but in one study it was demonstrated that compulsory use of a DDI checking program resulted in a 50% reduction in the incidence of DDI errors [18]. Given the complexity of the emerging field of health informatics, the focus is now shifting toward consulting users to develop more effective and efficient systems [19]. Users’ views are important because users have a unique ability to pick up problems and suggest ideas for improvement that system developers sometimes overlook [19]. Research has also shown that user involvement in system design can lead to greater system usage and satisfaction [20]. The aim of this study was to determine the opinions of both experts and users on computerized strategies available to target DDIs. Experts’ ideas about potential DDI strategies in Phase 1 were used in Phase 2 to ascertain what users perceived to be the best strategy to implement in hospital for preventing unwanted DDIs. This study is unique in its approach to DDIs; most studies only assess user views postimplementation of a specific system [6,21]. We hoped seeking input from users before implementation would allow us to identify user needs and perceived system requirements, and to determine perceived barriers and facilitators to successful uptake of a DDI computerized strategy.
2.
Methods
2.1.
Setting
all wards of the hospital used an ePS (MedChart® version 4.2.0) except the emergency department. MedChart® (www.isofthealth.com) is a commercial electronic medication management system that links prescribing, pharmacy and drug administration. The system interfaces with a locally developed computerized provider order entry system and results reporting system. When MedChart® was implemented (pilot 2005, complete implementation 2010), a decision was made not to incorporate DDI alerts into the system because it was felt that having a large number of alerts would lead to prescribers being over-alerted and so to alerts being ignored.
2.2.
Recruitment
2.2.1.
Phase 1
Purposive sampling was used to recruit participants for Phase 1. Members of clinical pharmacology or pharmacy with expertise in the area of medication safety were invited to participate in the study via telephone, email or face-to-face. Of the 11 participants contacted, eight agreed to take part in the study. Three were clinical pharmacologists and five were pharmacists.
2.2.2.
2.3.
Data collection
Semi-structured interviews were carried out for both phases. During Phase 1, participants were asked to identify and discuss potential strategies to prevent DDIs. The responses from Phase 1 were used to shape the focus of the subsequent phase. In Phase 2, prescribers were asked about their confidence in identifying DDIs and about their views of the different strategies identified in Phase 1. To minimize interviewer bias and encourage maximal discussion of the issues raised, questions were open-ended and were piloted with a clinical pharmacology expert and prescriber prior to commencement of data collection (see on-line Appendix). Interviews were approximately 20 min in duration in Phase 1, and 10 min in duration in Phase 2. Ethics committee approval was obtained by the human research ethics committee of the participating hospital and the University of NSW.
2.4.
This study was conducted at a 326-bed teaching hospital in metropolitan Sydney. At the time of the study,
Phase 2
Convenience sampling was used to recruit participants for Phase 2. Prescribers working in a variety of specialities, and of differing levels of seniority were contacted via telephone, email, or face-to-face. Of the 35 participants contacted, 18 agreed to take part in the study (eight JMOs (junior medical officers–interns, residents and registrars) and 10 staff specialists). Recruitment in both phases continued until saturation of themes had been achieved (see Section 2.4 below).
Data analysis
Interviews were audio recorded and transcribed. Two investigators separately reviewed each of the transcripts to identify key themes. The investigators discussed the themes to ensure
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 595–600
consistency in coding and to determine when theme saturation had occurred. Any discrepancies that emerged were resolved by reaching a consensus. Analysis of the transcripts in both phases was performed with no pre-conceived ideas about emergent themes. The software program, NVivo version 9.0.204.0 (http://www.qsrinternational.com) was used by the primary investigator to code themes and compare themes between senior and junior prescribers.
3.
Results
3.1.
Phase 1
3.1.1. What computerized strategies could be useful in reducing DDIs? When participants were asked for their opinion on what would be an effective strategy to reduce DDIs in hospital, many experts mentioned an alert system. Despite this, all felt strongly about a number of problems that could arise following DDI alert implementation. The main concern was that having too many alerts would be disruptive and lead to ‘alert fatigue’. One participant emphasized the importance of ensuring that DDI alerts are “the right ones that really count”. Other participants noted the complicated nature of selecting specific DDI alerts to put in place and discussed the legal liability problems associated with this. Participants also discussed the potential long-term effects of implementing DDI alerts. Experts felt that having an alert system in place might lead to prescribers beginning to over-rely on the system to detect DDIs, resulting in fewer prescribers remembering DDIs on their own: “If you have an alert system people rely on the alert system. . .. . .And they forget that if it doesn’t go “bing” in front of them, then it doesn’t mean that there’s no interaction, it just means that that particular database doesn’t think there’s an interaction” (Pharmacist). Participants in Phase 1 also suggested a second, more novel strategy to target DDIs: that of a reference “look-up” tool integrated into the ePS. This strategy differs from DDI checking software because it uses the patient’s current medication list available in the ePS. With one mouse-click, a prescriber can perform a DDI check of all a patient’s medications. The main benefit of this strategy was said to be the absence of unnecessary alerts annoying prescribers. With this kind of a system in place, prescribers “. . .would only do [the DDI check] when they were interested enough to actually read the results.” (Clinical Pharmacologist).
3.2.
Phase 2
3.2.1. What are prescribers’ thoughts on DDIs and the current prescribing process? All prescribers described giving some level of thought to DDIs while prescribing, although the extent of this varied considerably. Less than half of those interviewed reported thinking about DDIs every time they wrote a prescription, while the remainder said it was often dependent on the specific drug
597
being prescribed, and the condition of the patient. No prescriber reported having complete confidence in their ability to recognize dangerous DDIs. Most JMOs reported not being confident in recognizing dangerous DDIs. For example, one participant said: “I don’t think anyone can know absolutely every drug–drug interaction, particularly with the fact that more drugs are coming out every day”
3.2.2. What are prescribers’ ideas on an effective computerized strategy for preventing DDIs? All participants agreed that there needed to be some kind of strategy in place to prevent DDIs. When asked for their opinions on the two main ideas from Phase 1–an alert system or a reference ‘look-up’ tool – the majority of prescribers preferred an alert system. The main advantage of an alert system was reported to be the fact that alerts occurred independently of a conscious decision by a prescriber to check an interaction. One prescriber said about the look-up tool: “You have to be looking for interactions, so if you’re not thinking about it then you probably won’t look it up, where at least if there’s an alert that prompts you. . . it’s already done the work.” Participants also felt that prescribers’ ability to recognize DDIs was compromised when they were busy or timeconstrained, so a look-up tool would not be used. This was particularly a problem for JMOs: “When you’re in a hurry, in an emergency things may not occur to you which the alert system would remind you of”. Some participants recognized that with the addition of DDI alerts to the current alerts in their hospital ePS prescribers may be exposed to too many alerts, to the point where they may begin to ignore and overlook them. When asked about the features they would like to see in an ideal alert system, most prescribers supported a system where DDIs were differentiated according to severity. In this way, prescribers could be alerted only about the more serious and most clinically significant interactions. Some participants thought that overriding DDI alerts should be allowed while other participants felt that some medication combinations should require the approval of a staff specialist, or pharmacist, before an order could be completed.
4.
Discussion
This study used interviews to explore the perspectives of both drug-safety experts and prescribers on effective DDI strategies in a hospital setting. We discovered that prescribers did not always consider potential DDIs when prescribing, were not confident about knowing all DDIs and so supported introducing DDI alerts into the current alert set at their site. This result was unexpected given that our previous research at the site revealed that alerts currently operational in the ePS were not seen as helpful in guiding prescribing
598
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 595–600
decisions and were largely ignored by prescribers on wardrounds [22]. Previous research has also shown that prescribers override most DDI alerts presented [11,23] with a number of systemic factors consistently hampering the effectiveness of alert systems and of DDI alerts specifically. A recent review described ‘alert fatigue’ as the main problem associated with incorporating DDIs alerts into an ePS [9]. Participants in our study were aware of the potential for alert fatigue following DDI alert introduction but still preferred alerts to the ‘look-up’ tool integrated into the ePS. This was primarily because the ‘look-up’ tool is reliant on prescribers actively searching for information. Prescribers believed that the tool would not be used given the busy nature of hospitals and the lack of awareness and knowledge of DDIs among prescribers. A major barrier to the success of similar software programs in healthcare is prescribers not using them [24] and one of the most significant independent predictors of improved clinical outcomes with clinical decision support (CDS) is the automatic provision of decision support, instead of the requirement for users to seek out the advice on their own [25]. Our findings are consistent with those of a recent survey study that showed that prescribers working in an intensive care unit preferred to receive pop-up alerts than be presented with the same information on request [26]. If a look-up tool is implemented at our study hospital, there is a real and significant risk that prescribers will not consult the source as often as they need to, particularly because they are not aware of many potentially harmful medication combinations [6,7,27]. In general, participants in Phase 1 were more critical of DDI alerts as a strategy, compared to the users interviewed in Phase 2. This was most likely because as experts, Phase 1 individuals were more aware of the intricacies and difficulties associated with DDI alert implementation. For example, participants in Phase 1 spoke of legal liability issues that might arise following the development of a site-specific DDI alert set but this was not mentioned by users in Phase 2. Although prescribers expressed a preference for only being alerted about the more serious and clinically important interactions as a way of reducing the number of alerts experienced, experts in Phase 1 recognized the complexity and difficulty of identifying what constitutes a serious or important DDI. Multiple studies have been conducted in an attempt to generate a small set of contraindicated DDIs for incorporation into alert sets, [28–30] but no consensus has yet been reached on which are the most critical and important DDIs to include. It has also been shown that particular DDI alerts may be more relevant in some contexts than others (e.g. intensive care units vs. general wards) [31], highlighting the complexity of designing a set of clinically appropriate DDI alerts for ePS. Experts in Phase 1 also expressed a concern that DDI alerts might lead to an over-reliance on the system by prescribers. A consequence of this ‘automation bias’ is that users can miss important clinical errors because the system has not warned them about them [32,33]. This problem can be minimized by ensuring that users receive regular updates and training about the limitations of the alert system.
4.1.
Study limitations
This study was conducted at a single site so may have limited generalizability. Our recruitment mechanism for Phase 2 – convenience sampling – may have resulted in a biased sample, although we ensured prescribers from a range of specialties and levels of experience were interviewed. Similarly, we expect prescribers’ views of the potential strategies to be influenced by their previous experiences with DDI prevention strategies and we did not explore these experiences in-depth during interviews.
5.
Conclusion
Prescribers, particularly junior medical officers, were not confident in their ability to identify potential DDIs. This suggests that a strategy, which is not reliant on individuals seeking out information for themselves, would be most appropriate for targeting DDIs. While a number of problems with the implementation of DDI alerts have been discussed in the literature, prescribers felt that alerts would be the most effective strategy to introduce for preventing DDIs. When incorporating DDI alerts into an alert set, care must be taken to ensure that prescribers are not being over-alerted, or becoming too reliant on alerts to identify all potential errors. These harms can be partly mitigated by ensuring that users are aware of the limitations of the system and by continuing to consult users during the implementation process. By involving users in DDI strategy design we expect greater adherence and satisfaction with the strategy implemented.
Author contributions O.M. contributed to data collection and interpretation, M.B. and R.D. contributed to conception and design of study and interpretation of findings, all authors contributed to preparation of the manuscript and approved the final version for submission.
Conflict of interest The authors declare that they have no conflict of interest.
Funding This research was supported by National Health and Medical Research Council Program grant #569612. This funding source had no involvement in the design, data collection, data analysis, interpretation of data, writing of report or decision to submit the paper for publication.
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 595–600
Summary points What was already known on the topic? • Drug–drug interactions (DDIs) are an important and preventable cause of medication errors in hospitals. • Prescribers are often unable to recognize potential DDIs when prescribing. • Several computerized strategies have been introduced in hospitals to prevent DDIs but these are rarely evaluated and are typically implemented with little input from users.
[7]
[8]
[9]
[10]
What this study added to our knowledge? • Prescribers were not confident in their ability to identify dangerous DDIs and were supportive of introducing a computerized strategy to prevent DDIs in hospitals. • Prescribers were aware of the potential for ‘alert fatigue’ following DDI alert introduction but still preferred alerts to a ‘look-up’ tool. • Computerized strategies, which are not reliant on individuals seeking information out are likely to be most effective in preventing DDIs. • ‘Experts’ and prescribers expressed different views about potential strategies for targeting DDIs, highlighting the value of seeking input from users prior to electronic system implementation.
[11]
[12]
[13]
[14]
[15]
Appendix A. Supplementary data Supplementary material related to this article can be found, in the online version, at http://dx.doi.org/10.1016/ j.ijmedinf.2015.04.001.
[16]
[17]
references [18] [1] D. Bates, D. Cullen, N. Laird, Incidence of adverse drug events and potential adverse drug events: implications for prevention, JAMA 274 (1995) 29–34. [2] L. Stanton, G. Peterson, R. Rumble, Drug-related admissions to an Australian hospital, J. Clin. Pharm. Ther. 19 (1994) 341–347. [3] S.E. Wolverton, Chapter 236. Drug interactions, in: L.A. Goldsmith, S.I. Katz, B.A. Gilchrest, A.S. Paller, D.J. Leffell, D. N.A. (Eds.), Ftizpatrick’s Dermatology in General Medicine, 2012, p. 8e, http://www.accessmedicine.com.wwwproxy0. library.unsw.edu.au/content.aspx?aID=56102812 [4] D.M. Roden, Chapter 5: principles of clinical pharmacology, in: D.L. Long, A.S. Facui, D.L. Kasper, S.L. Hauser, J.L. Jamesone, J. Loscalzo (Eds.), Harrison’s Principles of Internal Medicine, 2012, p. 18e. [5] J. Zwart-van Rijkom, E. Uijtendaal, M. ten Berg, W. van Solinge, A. Egberts, Frequency and nature of drug–drug interactions in a Dutch univeristy hospital, Br. J. Clin. Pharmacol. 68 (2) (2009) 187–193. [6] P. Glassman, B. Simon, P. Belperio, A. Lanto, Improving recognition of drug interactions: benefits and barriers to
[19]
[20]
[21]
[22]
[23]
[24]
599
using automated drug alerts, Med. Care 40 (12) (2002) 1161–1171. Y. Ko, D. Malone, G. Skrepnek, Prescribers’ knowledge of and sources of information for potential drug–drug interactions; a postal survey of US prescribers, Drug Saf. 31 (2008) 525–536. R. Tamblyn, A. Huang, L. Taylor, Y. Kawasumi, G. Bartlett, R. Grad, A randomised trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care, J. Am. Med. Inform. Assoc. 15 (2008) 430–438. P. Smithburger, M. Buckley, S. Bejian, K. Burenheide, S. Kane-Gill, A critical evaluation of clinical decision support for the detection of drug–drug interactions, Expert Opin. Drug Saf. 10 (6) (2011) 871–882. P.L. Smithburger, M.S. Buckley, S. Bejian, K. Burenheide, S.L. Kane-Gill, A critical evaluation of clinical decision support for the detection of drug–drug interactions, Expert Opin. Drug Saf. 10 (6) (2011) 871–882. J.D. Duke, X. Li, P. Dexter, Adherence to drug–drug interaction alerts in high-risk patients: a trial of context-enhanced alerting, J. Am. Med. Inform. Assoc. 20 (3) (2013) 494–498. B.L. Strom, R. Schinnar, W. Bilker, S. Hennessy, C.E. Leanard, E.A. Pifer, Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAID-warfarin co-prescribing as a test case, J. Am. Med. Inform. Assoc. 17 (2010) 411–415. B.L. Strom, R. Schinnar, F. Aberra, et al., Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial, Arch. Intern. Med. 170 (17) (2010) 1578–1583. B. Mannheimer, J. Ulfvarson, S. Eklof, M. Bergqvist, C. von Bahr, A clinical evaluation of the Janus Web Application, a software screening tool for drug–drug interactions, Eur. J. Clin. Pharmacol. 64 (2008) 1209–1214. J. Abarca, L. Colon, V. Wang, D. Malone, J. Murphy, E. Armstrong, Evaluation of the performance of drug–drug interaction screening software in community and hospital pharmacies, J. Manag. Care Pharm. 12 (5) (2006) 383–389. M. Sweidan, J. Reeve, J. Brien, P. Jayasuriya, J. Martin, G. Vernon, Quality of drug interaction alerts in prescribing and dispensing software, MJA 190 (2009) 251–254. A.M.M. Reis, S.H.D. Cassiani, Evaluation of three brands of drug interaction software for use in intensive care units, Pharm. World Sci. 32 (6) (2010) 822–828. A. Oliven, I. Michalake, D. Zalman, E. Dorman, D. Yeshurun, M. Odeh, Prevention of prescription errors by computerized, on-line suveillance of drug order entry, Int. J. Med. Inf. 74 (2005) 377–386. J.M.C. Bastien, Usability testing: a review of some methodological and technical aspects of the method, Int. J. Med. Inf. 79 (4) (2010) E18–E23. J.J. Baroudi, M.H. Olson, B. Ives, An empirical study of the impact of user involvement on system usage and information satisfaction, CACM 29 (3) (1986) 232–238. D. Magnus, D. Rodgers, J. Avery, GP’s views on computerized drug interaction alerts: questionnaire survey, J. Clin. Pharm. Ther. 27 (2002) 377–382. M. Baysari, J. Westbrook, K. Richardson, R. Day, The influence of computerized decision support on prescribing during ward-rounds: are the decision-makers targeted, J. Am. Med. Inform. Assoc. 18 (2011) 754–759. K.C. Nanji, S.P. Slight, D.L. Seger, et al., Overrides of medication-related clinical decision support alerts in outpatients, J. Am. Med. Inform. Assoc. (2013). N. Sandson, Economic grand rounds: drug–drug interactions: the silent epidemic, Psychiatr. Serv. 56 (1) (2005) 22–24.
600
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 595–600
[25] K. Kawamoto, C.A. Houlihan, E.A. Balas, D.F. Lobach, Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success, Br. Med. J. 330 (7494) (2005) 765E–768E. [26] A.M. Scheepers-Hoeks, R.J. Grouls, C. Neef, E.W. Ackerman, E.H. Korsten, Physicians’ responses to clinical decision support on an intensive care unit – comparison of four different alerting methods, Artif. Intell. Med. 59 (1) (2013) 33–38. [27] R. Weideman, I. Bernstein, W. McKinney, Pharmacist recognition of potential drug interactions, Am. J. Health Syst. Pharm. 56 (15) (1999) 1524–1529. [28] D. Malone, J. Abarca, P. Hansten, A. Grizzle, E. Armstrong, R. Van Bergen, Identification of serious drug–drug interactions: results of the partnership to prevent drug–drug interactions, J. Am. Pharm. Assoc. 44 (2) (2004).
[29] D. Classen, P. Shobha, D. Bates, Critical drug–drug interactions for use in electronic health records systems with computerised physician order entry: review of leading approaches, J. Patient Saf. 7 (2) (2011) 61–65. [30] S. Phansalkar, A.A. Desai, D. Bell, et al., High-priority drug–drug interactions for use in electronic health records, J. Am. Med. Inform. Assoc. 19 (5) (2012) 735–743. [31] M. Askari, S. Eslami, M. Louws, et al., Relevance of drug–drug interaction in the ICU – perceptions of intensivists and pharmacists, Stud. Health Technol. Inform. 180 (2012) 716–720. [32] E. Coiera, J.I. Westbrook, J.C. Wyatt, The safety and quality of decision support systems, Methods Inf. Med. 45 (2006) 20–25. [33] L.G. Bergman, U.G.H. Fors, Decision support in psychiatry – a comparison between the diagnostic outcomes using a computerized decision support system versus manual diagnosis, BMC Med. Inform. Decis. Mak. (2008) 8.
journal homepage: www.ijmijournal.com
Identifying effective computerized strategies to prevent drug–drug interactions in hospital: A user-centered approach Olivia Missiakos a,b , Melissa T. Baysari c,b,∗ , Richard O. Day b,a a
UNSW Medicine, UNSW, Sydney, Australia Department of Clinical Pharmacology & Toxicology, St Vincent’s Hospital, Sydney, Australia c Centre for Health Systems & Safety Research, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia b
a r t i c l e
i n f o
a b s t r a c t
Article history:
Background: Drug–drug interactions (DDIs) are an important and preventable cause of med-
Received in revised form
ication errors in hospitals. Recent developments in technology have seen new strategies
21 July 2014
emerge for preventing DDIs but these computerized strategies are rarely evaluated and are
Accepted 1 April 2015
typically implemented with little input from the individuals using them.
Keywords:
strategies available to assist in the identification and prevention of DDIs in hospitals.
Aim: To determine the opinions of both experts and users (prescribers) on computerized Drug–drug interaction
Method: Eight drug safety experts and 18 prescribers took part in semi-structured inter-
Alerts
views. Participants were asked about their confidence in identifying DDIs and their views
Alert fatigue
on potential computerized strategies to prevent DDIs.
Interviews
Results: No prescribers reported complete confidence in identifying dangerous DDIs, with junior prescribers appearing less confident than senior prescribers. Most prescribers believed that computerized alerts would be the most effective strategy for preventing DDIs, while experts were more critical of alerts. Conclusion: The lack of confidence displayed by prescribers in their ability to identify DDIs suggests that an appropriate strategy would be one that does not rely on individuals seeking out the information themselves. While a large number of problems related to DDI alert implementation have been reported in the literature (e.g. alert overload), prescribers appeared to be receptive to the idea of being alerted. By ensuring users are aware of the limitations of the system and involving them in DDI strategy design we expect greater use and satisfaction with the adopted strategy. © 2015 Elsevier Ireland Ltd. All rights reserved.
1.
Introduction
Drug–drug interactions (DDIs) are a preventable cause of medication errors in community and hospital settings and account
for 2.4–4.4% of hospital admissions [1,2]. They occur when two or more drugs are taken in combination that leads to a change in the activity of either or both drugs [3,4]. DDIs can result in adverse effects; commonly these include low blood pressure, bleeding or kidney damage [5]. Additionally, DDIs can lead to
∗ Corresponding author at: Centre for Health Systems & Safety Research, Level 6, 75 Talavera Rd, Macquarie University, NSW 2109, Australia. Tel.: +61 29850 2416; fax: +61 280886234. E-mail address: [email protected] (M.T. Baysari).
http://dx.doi.org/10.1016/j.ijmedinf.2015.04.001 1386-5056/© 2015 Elsevier Ireland Ltd. All rights reserved.
596
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 595–600
therapeutic failure, where one or both of the drugs are unable to achieve their desired clinical effect [5]. Research has shown that both prescribers and pharmacists are often unable to recognize potential DDIs [6,7]. Recent developments in technology have seen new strategies emerge to assist in DDI identification and prevention. In particular, alerts integrated into electronic prescribing systems (ePS) have frequently been adopted by hospitals in an attempt to minimize DDI occurrence [8,9]. To date, there is limited evidence demonstrating reductions in DDI errors or adverse drug events following DDI alert introduction, with evaluations typically comprising a review of the number of alerts generated and acted on by prescribers [10,11]. Two studies examined the impact of a single customized DDI alert on the concurrent ordering of two medications, but they report inconsistent findings [12,13] and in one case, introduction of a near hard-stop DDI alert resulted in unintended consequences (e.g. delays in appropriate treatment) [13]. Computerized DDI checking programs are also commonly discussed in the literature as a strategy to target DDI errors [14,15]. Prescribers enter medication names into the program, which then checks medication combinations for potential DDIs. The main difference between this strategy and an alert system is that software programs are typically voluntarily used and so are non-interruptive. Evaluation of DDI checking software usually includes an assessment of its ability to identify DDIs, most often in the form of an analysis of sensitivity and specificity, [16,17] but in one study it was demonstrated that compulsory use of a DDI checking program resulted in a 50% reduction in the incidence of DDI errors [18]. Given the complexity of the emerging field of health informatics, the focus is now shifting toward consulting users to develop more effective and efficient systems [19]. Users’ views are important because users have a unique ability to pick up problems and suggest ideas for improvement that system developers sometimes overlook [19]. Research has also shown that user involvement in system design can lead to greater system usage and satisfaction [20]. The aim of this study was to determine the opinions of both experts and users on computerized strategies available to target DDIs. Experts’ ideas about potential DDI strategies in Phase 1 were used in Phase 2 to ascertain what users perceived to be the best strategy to implement in hospital for preventing unwanted DDIs. This study is unique in its approach to DDIs; most studies only assess user views postimplementation of a specific system [6,21]. We hoped seeking input from users before implementation would allow us to identify user needs and perceived system requirements, and to determine perceived barriers and facilitators to successful uptake of a DDI computerized strategy.
2.
Methods
2.1.
Setting
all wards of the hospital used an ePS (MedChart® version 4.2.0) except the emergency department. MedChart® (www.isofthealth.com) is a commercial electronic medication management system that links prescribing, pharmacy and drug administration. The system interfaces with a locally developed computerized provider order entry system and results reporting system. When MedChart® was implemented (pilot 2005, complete implementation 2010), a decision was made not to incorporate DDI alerts into the system because it was felt that having a large number of alerts would lead to prescribers being over-alerted and so to alerts being ignored.
2.2.
Recruitment
2.2.1.
Phase 1
Purposive sampling was used to recruit participants for Phase 1. Members of clinical pharmacology or pharmacy with expertise in the area of medication safety were invited to participate in the study via telephone, email or face-to-face. Of the 11 participants contacted, eight agreed to take part in the study. Three were clinical pharmacologists and five were pharmacists.
2.2.2.
2.3.
Data collection
Semi-structured interviews were carried out for both phases. During Phase 1, participants were asked to identify and discuss potential strategies to prevent DDIs. The responses from Phase 1 were used to shape the focus of the subsequent phase. In Phase 2, prescribers were asked about their confidence in identifying DDIs and about their views of the different strategies identified in Phase 1. To minimize interviewer bias and encourage maximal discussion of the issues raised, questions were open-ended and were piloted with a clinical pharmacology expert and prescriber prior to commencement of data collection (see on-line Appendix). Interviews were approximately 20 min in duration in Phase 1, and 10 min in duration in Phase 2. Ethics committee approval was obtained by the human research ethics committee of the participating hospital and the University of NSW.
2.4.
This study was conducted at a 326-bed teaching hospital in metropolitan Sydney. At the time of the study,
Phase 2
Convenience sampling was used to recruit participants for Phase 2. Prescribers working in a variety of specialities, and of differing levels of seniority were contacted via telephone, email, or face-to-face. Of the 35 participants contacted, 18 agreed to take part in the study (eight JMOs (junior medical officers–interns, residents and registrars) and 10 staff specialists). Recruitment in both phases continued until saturation of themes had been achieved (see Section 2.4 below).
Data analysis
Interviews were audio recorded and transcribed. Two investigators separately reviewed each of the transcripts to identify key themes. The investigators discussed the themes to ensure
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 595–600
consistency in coding and to determine when theme saturation had occurred. Any discrepancies that emerged were resolved by reaching a consensus. Analysis of the transcripts in both phases was performed with no pre-conceived ideas about emergent themes. The software program, NVivo version 9.0.204.0 (http://www.qsrinternational.com) was used by the primary investigator to code themes and compare themes between senior and junior prescribers.
3.
Results
3.1.
Phase 1
3.1.1. What computerized strategies could be useful in reducing DDIs? When participants were asked for their opinion on what would be an effective strategy to reduce DDIs in hospital, many experts mentioned an alert system. Despite this, all felt strongly about a number of problems that could arise following DDI alert implementation. The main concern was that having too many alerts would be disruptive and lead to ‘alert fatigue’. One participant emphasized the importance of ensuring that DDI alerts are “the right ones that really count”. Other participants noted the complicated nature of selecting specific DDI alerts to put in place and discussed the legal liability problems associated with this. Participants also discussed the potential long-term effects of implementing DDI alerts. Experts felt that having an alert system in place might lead to prescribers beginning to over-rely on the system to detect DDIs, resulting in fewer prescribers remembering DDIs on their own: “If you have an alert system people rely on the alert system. . .. . .And they forget that if it doesn’t go “bing” in front of them, then it doesn’t mean that there’s no interaction, it just means that that particular database doesn’t think there’s an interaction” (Pharmacist). Participants in Phase 1 also suggested a second, more novel strategy to target DDIs: that of a reference “look-up” tool integrated into the ePS. This strategy differs from DDI checking software because it uses the patient’s current medication list available in the ePS. With one mouse-click, a prescriber can perform a DDI check of all a patient’s medications. The main benefit of this strategy was said to be the absence of unnecessary alerts annoying prescribers. With this kind of a system in place, prescribers “. . .would only do [the DDI check] when they were interested enough to actually read the results.” (Clinical Pharmacologist).
3.2.
Phase 2
3.2.1. What are prescribers’ thoughts on DDIs and the current prescribing process? All prescribers described giving some level of thought to DDIs while prescribing, although the extent of this varied considerably. Less than half of those interviewed reported thinking about DDIs every time they wrote a prescription, while the remainder said it was often dependent on the specific drug
597
being prescribed, and the condition of the patient. No prescriber reported having complete confidence in their ability to recognize dangerous DDIs. Most JMOs reported not being confident in recognizing dangerous DDIs. For example, one participant said: “I don’t think anyone can know absolutely every drug–drug interaction, particularly with the fact that more drugs are coming out every day”
3.2.2. What are prescribers’ ideas on an effective computerized strategy for preventing DDIs? All participants agreed that there needed to be some kind of strategy in place to prevent DDIs. When asked for their opinions on the two main ideas from Phase 1–an alert system or a reference ‘look-up’ tool – the majority of prescribers preferred an alert system. The main advantage of an alert system was reported to be the fact that alerts occurred independently of a conscious decision by a prescriber to check an interaction. One prescriber said about the look-up tool: “You have to be looking for interactions, so if you’re not thinking about it then you probably won’t look it up, where at least if there’s an alert that prompts you. . . it’s already done the work.” Participants also felt that prescribers’ ability to recognize DDIs was compromised when they were busy or timeconstrained, so a look-up tool would not be used. This was particularly a problem for JMOs: “When you’re in a hurry, in an emergency things may not occur to you which the alert system would remind you of”. Some participants recognized that with the addition of DDI alerts to the current alerts in their hospital ePS prescribers may be exposed to too many alerts, to the point where they may begin to ignore and overlook them. When asked about the features they would like to see in an ideal alert system, most prescribers supported a system where DDIs were differentiated according to severity. In this way, prescribers could be alerted only about the more serious and most clinically significant interactions. Some participants thought that overriding DDI alerts should be allowed while other participants felt that some medication combinations should require the approval of a staff specialist, or pharmacist, before an order could be completed.
4.
Discussion
This study used interviews to explore the perspectives of both drug-safety experts and prescribers on effective DDI strategies in a hospital setting. We discovered that prescribers did not always consider potential DDIs when prescribing, were not confident about knowing all DDIs and so supported introducing DDI alerts into the current alert set at their site. This result was unexpected given that our previous research at the site revealed that alerts currently operational in the ePS were not seen as helpful in guiding prescribing
598
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 595–600
decisions and were largely ignored by prescribers on wardrounds [22]. Previous research has also shown that prescribers override most DDI alerts presented [11,23] with a number of systemic factors consistently hampering the effectiveness of alert systems and of DDI alerts specifically. A recent review described ‘alert fatigue’ as the main problem associated with incorporating DDIs alerts into an ePS [9]. Participants in our study were aware of the potential for alert fatigue following DDI alert introduction but still preferred alerts to the ‘look-up’ tool integrated into the ePS. This was primarily because the ‘look-up’ tool is reliant on prescribers actively searching for information. Prescribers believed that the tool would not be used given the busy nature of hospitals and the lack of awareness and knowledge of DDIs among prescribers. A major barrier to the success of similar software programs in healthcare is prescribers not using them [24] and one of the most significant independent predictors of improved clinical outcomes with clinical decision support (CDS) is the automatic provision of decision support, instead of the requirement for users to seek out the advice on their own [25]. Our findings are consistent with those of a recent survey study that showed that prescribers working in an intensive care unit preferred to receive pop-up alerts than be presented with the same information on request [26]. If a look-up tool is implemented at our study hospital, there is a real and significant risk that prescribers will not consult the source as often as they need to, particularly because they are not aware of many potentially harmful medication combinations [6,7,27]. In general, participants in Phase 1 were more critical of DDI alerts as a strategy, compared to the users interviewed in Phase 2. This was most likely because as experts, Phase 1 individuals were more aware of the intricacies and difficulties associated with DDI alert implementation. For example, participants in Phase 1 spoke of legal liability issues that might arise following the development of a site-specific DDI alert set but this was not mentioned by users in Phase 2. Although prescribers expressed a preference for only being alerted about the more serious and clinically important interactions as a way of reducing the number of alerts experienced, experts in Phase 1 recognized the complexity and difficulty of identifying what constitutes a serious or important DDI. Multiple studies have been conducted in an attempt to generate a small set of contraindicated DDIs for incorporation into alert sets, [28–30] but no consensus has yet been reached on which are the most critical and important DDIs to include. It has also been shown that particular DDI alerts may be more relevant in some contexts than others (e.g. intensive care units vs. general wards) [31], highlighting the complexity of designing a set of clinically appropriate DDI alerts for ePS. Experts in Phase 1 also expressed a concern that DDI alerts might lead to an over-reliance on the system by prescribers. A consequence of this ‘automation bias’ is that users can miss important clinical errors because the system has not warned them about them [32,33]. This problem can be minimized by ensuring that users receive regular updates and training about the limitations of the alert system.
4.1.
Study limitations
This study was conducted at a single site so may have limited generalizability. Our recruitment mechanism for Phase 2 – convenience sampling – may have resulted in a biased sample, although we ensured prescribers from a range of specialties and levels of experience were interviewed. Similarly, we expect prescribers’ views of the potential strategies to be influenced by their previous experiences with DDI prevention strategies and we did not explore these experiences in-depth during interviews.
5.
Conclusion
Prescribers, particularly junior medical officers, were not confident in their ability to identify potential DDIs. This suggests that a strategy, which is not reliant on individuals seeking out information for themselves, would be most appropriate for targeting DDIs. While a number of problems with the implementation of DDI alerts have been discussed in the literature, prescribers felt that alerts would be the most effective strategy to introduce for preventing DDIs. When incorporating DDI alerts into an alert set, care must be taken to ensure that prescribers are not being over-alerted, or becoming too reliant on alerts to identify all potential errors. These harms can be partly mitigated by ensuring that users are aware of the limitations of the system and by continuing to consult users during the implementation process. By involving users in DDI strategy design we expect greater adherence and satisfaction with the strategy implemented.
Author contributions O.M. contributed to data collection and interpretation, M.B. and R.D. contributed to conception and design of study and interpretation of findings, all authors contributed to preparation of the manuscript and approved the final version for submission.
Conflict of interest The authors declare that they have no conflict of interest.
Funding This research was supported by National Health and Medical Research Council Program grant #569612. This funding source had no involvement in the design, data collection, data analysis, interpretation of data, writing of report or decision to submit the paper for publication.
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 595–600
Summary points What was already known on the topic? • Drug–drug interactions (DDIs) are an important and preventable cause of medication errors in hospitals. • Prescribers are often unable to recognize potential DDIs when prescribing. • Several computerized strategies have been introduced in hospitals to prevent DDIs but these are rarely evaluated and are typically implemented with little input from users.
[7]
[8]
[9]
[10]
What this study added to our knowledge? • Prescribers were not confident in their ability to identify dangerous DDIs and were supportive of introducing a computerized strategy to prevent DDIs in hospitals. • Prescribers were aware of the potential for ‘alert fatigue’ following DDI alert introduction but still preferred alerts to a ‘look-up’ tool. • Computerized strategies, which are not reliant on individuals seeking information out are likely to be most effective in preventing DDIs. • ‘Experts’ and prescribers expressed different views about potential strategies for targeting DDIs, highlighting the value of seeking input from users prior to electronic system implementation.
[11]
[12]
[13]
[14]
[15]
Appendix A. Supplementary data Supplementary material related to this article can be found, in the online version, at http://dx.doi.org/10.1016/ j.ijmedinf.2015.04.001.
[16]
[17]
references [18] [1] D. Bates, D. Cullen, N. Laird, Incidence of adverse drug events and potential adverse drug events: implications for prevention, JAMA 274 (1995) 29–34. [2] L. Stanton, G. Peterson, R. Rumble, Drug-related admissions to an Australian hospital, J. Clin. Pharm. Ther. 19 (1994) 341–347. [3] S.E. Wolverton, Chapter 236. Drug interactions, in: L.A. Goldsmith, S.I. Katz, B.A. Gilchrest, A.S. Paller, D.J. Leffell, D. N.A. (Eds.), Ftizpatrick’s Dermatology in General Medicine, 2012, p. 8e, http://www.accessmedicine.com.wwwproxy0. library.unsw.edu.au/content.aspx?aID=56102812 [4] D.M. Roden, Chapter 5: principles of clinical pharmacology, in: D.L. Long, A.S. Facui, D.L. Kasper, S.L. Hauser, J.L. Jamesone, J. Loscalzo (Eds.), Harrison’s Principles of Internal Medicine, 2012, p. 18e. [5] J. Zwart-van Rijkom, E. Uijtendaal, M. ten Berg, W. van Solinge, A. Egberts, Frequency and nature of drug–drug interactions in a Dutch univeristy hospital, Br. J. Clin. Pharmacol. 68 (2) (2009) 187–193. [6] P. Glassman, B. Simon, P. Belperio, A. Lanto, Improving recognition of drug interactions: benefits and barriers to
[19]
[20]
[21]
[22]
[23]
[24]
599
using automated drug alerts, Med. Care 40 (12) (2002) 1161–1171. Y. Ko, D. Malone, G. Skrepnek, Prescribers’ knowledge of and sources of information for potential drug–drug interactions; a postal survey of US prescribers, Drug Saf. 31 (2008) 525–536. R. Tamblyn, A. Huang, L. Taylor, Y. Kawasumi, G. Bartlett, R. Grad, A randomised trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care, J. Am. Med. Inform. Assoc. 15 (2008) 430–438. P. Smithburger, M. Buckley, S. Bejian, K. Burenheide, S. Kane-Gill, A critical evaluation of clinical decision support for the detection of drug–drug interactions, Expert Opin. Drug Saf. 10 (6) (2011) 871–882. P.L. Smithburger, M.S. Buckley, S. Bejian, K. Burenheide, S.L. Kane-Gill, A critical evaluation of clinical decision support for the detection of drug–drug interactions, Expert Opin. Drug Saf. 10 (6) (2011) 871–882. J.D. Duke, X. Li, P. Dexter, Adherence to drug–drug interaction alerts in high-risk patients: a trial of context-enhanced alerting, J. Am. Med. Inform. Assoc. 20 (3) (2013) 494–498. B.L. Strom, R. Schinnar, W. Bilker, S. Hennessy, C.E. Leanard, E.A. Pifer, Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAID-warfarin co-prescribing as a test case, J. Am. Med. Inform. Assoc. 17 (2010) 411–415. B.L. Strom, R. Schinnar, F. Aberra, et al., Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial, Arch. Intern. Med. 170 (17) (2010) 1578–1583. B. Mannheimer, J. Ulfvarson, S. Eklof, M. Bergqvist, C. von Bahr, A clinical evaluation of the Janus Web Application, a software screening tool for drug–drug interactions, Eur. J. Clin. Pharmacol. 64 (2008) 1209–1214. J. Abarca, L. Colon, V. Wang, D. Malone, J. Murphy, E. Armstrong, Evaluation of the performance of drug–drug interaction screening software in community and hospital pharmacies, J. Manag. Care Pharm. 12 (5) (2006) 383–389. M. Sweidan, J. Reeve, J. Brien, P. Jayasuriya, J. Martin, G. Vernon, Quality of drug interaction alerts in prescribing and dispensing software, MJA 190 (2009) 251–254. A.M.M. Reis, S.H.D. Cassiani, Evaluation of three brands of drug interaction software for use in intensive care units, Pharm. World Sci. 32 (6) (2010) 822–828. A. Oliven, I. Michalake, D. Zalman, E. Dorman, D. Yeshurun, M. Odeh, Prevention of prescription errors by computerized, on-line suveillance of drug order entry, Int. J. Med. Inf. 74 (2005) 377–386. J.M.C. Bastien, Usability testing: a review of some methodological and technical aspects of the method, Int. J. Med. Inf. 79 (4) (2010) E18–E23. J.J. Baroudi, M.H. Olson, B. Ives, An empirical study of the impact of user involvement on system usage and information satisfaction, CACM 29 (3) (1986) 232–238. D. Magnus, D. Rodgers, J. Avery, GP’s views on computerized drug interaction alerts: questionnaire survey, J. Clin. Pharm. Ther. 27 (2002) 377–382. M. Baysari, J. Westbrook, K. Richardson, R. Day, The influence of computerized decision support on prescribing during ward-rounds: are the decision-makers targeted, J. Am. Med. Inform. Assoc. 18 (2011) 754–759. K.C. Nanji, S.P. Slight, D.L. Seger, et al., Overrides of medication-related clinical decision support alerts in outpatients, J. Am. Med. Inform. Assoc. (2013). N. Sandson, Economic grand rounds: drug–drug interactions: the silent epidemic, Psychiatr. Serv. 56 (1) (2005) 22–24.
600
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 595–600
[25] K. Kawamoto, C.A. Houlihan, E.A. Balas, D.F. Lobach, Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success, Br. Med. J. 330 (7494) (2005) 765E–768E. [26] A.M. Scheepers-Hoeks, R.J. Grouls, C. Neef, E.W. Ackerman, E.H. Korsten, Physicians’ responses to clinical decision support on an intensive care unit – comparison of four different alerting methods, Artif. Intell. Med. 59 (1) (2013) 33–38. [27] R. Weideman, I. Bernstein, W. McKinney, Pharmacist recognition of potential drug interactions, Am. J. Health Syst. Pharm. 56 (15) (1999) 1524–1529. [28] D. Malone, J. Abarca, P. Hansten, A. Grizzle, E. Armstrong, R. Van Bergen, Identification of serious drug–drug interactions: results of the partnership to prevent drug–drug interactions, J. Am. Pharm. Assoc. 44 (2) (2004).
[29] D. Classen, P. Shobha, D. Bates, Critical drug–drug interactions for use in electronic health records systems with computerised physician order entry: review of leading approaches, J. Patient Saf. 7 (2) (2011) 61–65. [30] S. Phansalkar, A.A. Desai, D. Bell, et al., High-priority drug–drug interactions for use in electronic health records, J. Am. Med. Inform. Assoc. 19 (5) (2012) 735–743. [31] M. Askari, S. Eslami, M. Louws, et al., Relevance of drug–drug interaction in the ICU – perceptions of intensivists and pharmacists, Stud. Health Technol. Inform. 180 (2012) 716–720. [32] E. Coiera, J.I. Westbrook, J.C. Wyatt, The safety and quality of decision support systems, Methods Inf. Med. 45 (2006) 20–25. [33] L.G. Bergman, U.G.H. Fors, Decision support in psychiatry – a comparison between the diagnostic outcomes using a computerized decision support system versus manual diagnosis, BMC Med. Inform. Decis. Mak. (2008) 8.