- Email: [email protected]
Preparing student pharmacists to identify opioid misuse, prevent overdose and prescribe naloxone
Currents in Pharmacy Teaching and Learning 11 (2019) 522–527
Contents lists available at ScienceDirect
Currents in Pharmacy Teaching and Learning journal homepage: www.elsevier.com/locate/cptl
Experiences in Teaching and Learning
Preparing student pharmacists to identify opioid misuse, prevent overdose and prescribe naloxone
T
Elizabeth Skoy , Heidi Eukel, Jeanne Frenzel, Amy Werremeyer ⁎
North Dakota State University School of Pharmacy, PO Box 6050, Dept 2660, Fargo, ND 58018-6050, United States
ARTICLE INFO
ABSTRACT
Keywords: Naloxone Opioid Simulation Laboratory activity
Background and Purpose: All 50 states have implemented policies to improve access to naloxone through community pharmacies. Many states require naloxone-based training for pharmacists before participating in these activities. The purpose of this study was to determine the effects of an opioid misuse and overdose training program on students' knowledge, self-efficacy, and value of pharmacists' role in preventing overdose. Educational Activity and Setting: The training program was implemented with third-year professional pharmacy students. Students participated in a 50-min lecture followed by a two-hour hands-on laboratory activity. Students took a pre-survey immediately before the lecture and a post-survey immediately following the laboratory activity. Findings: Seventy students completed the pre- and post-surveys. Students showed a significant (p < 0.05) gain in self-efficacy and value after completing the training program, and a significant gain in knowledge from six of the eight related survey items. There was a significant correlation between students' self-efficacy and perceived value of the pharmacists' role in preventing opioid overdose. Summary: A training program implemented for third-year pharmacy students increases students' knowledge, self-efficacy, and perceived value. Specifically, the strong correlation seen between self-efficacy and perceived value after completing the training program is important to note as we prepare future pharmacists for their role in fighting the opioid epidemic.
Background and Purpose In 2017, the United States (US) Department of Health and Human Services declared the opioid epidemic a public health emergency and published five strategies for combatting opioid misuse. One of the strategies was to increase availability and use of overdose reversing drugs, such as naloxone.1 Pharmacists have a unique opportunity to dispense naloxone and educate patients on the use of this life-saving medication due to their expertise and their position as the most accessible healthcare provider.2–4 All 50 states have implemented a community pharmacy naloxone access policy. These policies range from the ability for pharmacists to prescribe and dispense naloxone, dispense naloxone without a prescription, and the ability for pharmacists to directly dispense naloxone under a state issued standing order.5 Currently, 19 states, including North Dakota, mandate that pharmacists complete a naloxone-based training before participating in activities allowed through the naloxone access policies. The most common training requirement among these states is a one- to two-hour continuing education (CE) course.6 In addition, some states require pharmacists
Corresponding author. E-mail addresses: [email protected] (E. Skoy), [email protected] (H. Eukel), [email protected] (J. Frenzel), [email protected] (A. Werremeyer). ⁎
https://doi.org/10.1016/j.cptl.2019.02.013
1877-1297/ © 2019 Elsevier Inc. All rights reserved.
Currents in Pharmacy Teaching and Learning 11 (2019) 522–527
E. Skoy, et al.
to complete annual or biennial CE to maintain their ability to participate in naloxone access programs.6 Research has shown that naloxone training increases student pharmacists' and pharmacists' willingness and confidence in their ability to dispense and counsel on naloxone.6–8 A survey of pharmacists in Kentucky found that an increase in reported pharmacist confidence in identifying individuals at risk for overdose was associated with a significant increase in willingness to dispense naloxone. The same study also found that an increase in reported pharmacist confidence in the ability to educate patients about overdose was associated with a significant increase in willingness to dispense naloxone.9 An additional study reviewing naloxone training programs called for the inclusion of content addressing how to communicate the need for naloxone with patients.6 The American Association of Colleges of Pharmacy (AACP) has called for all pharmacy graduates to be able to provide and administer naloxone as well as properly educate individuals who may experience or encounter an opioid overdose.10 In alignment with this call, the 2013 Center for the Advancement of Pharmacy Education (CAPE) Educational Outcomes outline the expectation that graduates must be able to educate patients and assess their understanding.11 Since, there have been published works addressing the need for training and educating future pharmacists on the expansion of opioid overdose interventions, including naloxone prescribing.7,8,12–14 The description of these educational activities have included the incorporation of a related Objective Structured Clinical Examination (OSCE), community naloxone counseling, lectures, hands-on workshops, and laboratory activities. The purpose of this study was to determine the effects of an opioid misuse and accidental overdose training program on students' knowledge, selfefficacy, and perceived value of pharmacists' role in preventing overdose. Educational Activity and Setting The training program described in this manuscript took place in Pharmacy Practice Laboratory IV, which takes place in the spring semester of the third-year professional pharmacy program, prior to advanced pharmacy practice experiences (APPEs). Pharmacy Practice Laboratory IV is part of a four-semester course series designed to teach and assess the practical application of skills needed for students to be able to practice at the top of their license. The course is comprised of a weekly 50-min lecture and a two-hour laboratory session that are conducted in small groups of 16 students. Topics in this course focus on advancing roles of the pharmacist, such as point-of care testing, medication therapy management (MTM), and disease state management. To address the AACP call to action and expose students to the advancing role of the pharmacist in the fight against the opioid epidemic, a training program was created and added to the curriculum to develop a student's ability to assess for opioid misuse and accidental overdose and to prescribe and dispense naloxone. The training program consisted of a 50-min lecture and a two-hour hands-on activity. Pre-activity Preparation During the fall semester of the third professional year, 12 weeks prior to the laboratory activity, students received two and onehalf hours of didactic instruction on opioid misuse and accidental overdose in a neuropsychiatric pharmacotherapy course. As part of this course, students also completed a one-hour online CE training module focused on identifying persons at risk of respiratory depression from opioids, signs of opioid overdose, proper administration of naloxone, and a pharmacists' role in dispensing naloxone.15 Verification of program completion included handing in the certificate of completion attesting they had passed the final CE quiz. It is important to note that this CE is a suggested module used to fulfill the state requirement of one-hour topic-related education to allow pharmacists to prescribe naloxone. Although this fulfilled the requirement, faculty determined additional education and a hands-on laboratory activity would be beneficial to further advance the curriculum. The opioid misuse and overdose training program took place the semester after the didactic instruction on this content. A 50-min didactic lecture was used to reinforce topics previously addressed including North Dakota state-specific protocol for pharmacists' prescribing naloxone, risk factors for opioid misuse and accidental overdose, how to respond to an opioid overdose situation, and information regarding the four available naloxone dosage forms. During the lecture, students were provided with an opioid misuse and overdose toolkit (Fig. 1) that was developed by faculty based on their practice experience and expertise. The toolkit included practice-based tools for students to utilize during the laboratory activity and included useful mobile apps such as Opi-Rescue®, that could be used to instruct and aid patients in how to respond to an opioid overdose, and the Centers for Disease Control and Prevention (CDC) Opioid Guidelines app, which has a built-in feature to calculate daily morphine milligram equivalents (MMEs).16,17 The toolkit also included a red flags checklist supported by the literature to identify individuals who may be misusing controlled substances. These red flags included but are not limited to individuals requesting to pay cash, receiving medications from multiple providers, and requesting early medication refills.18,19 A faculty-developed naloxone consultation checklist, supported by the North Dakota Board of ■ ■ ■ ■ ■ ■
State protocol for naloxone prescribing and dispensing15 Opioid Risk Tool16 Pharmacy Red Flag Checklist17 Naloxone Consultation Checklist15 OpiRescue App11 CDC Opioid MME Calculator12
CDC = Center for Disease Control; MME = morphine milligram equivalents. Fig. 1. Opioid misuse and overdose toolkit. CDC = Center for Disease Control; MME = morphine milligram equivalents. 523
Currents in Pharmacy Teaching and Learning 11 (2019) 522–527
E. Skoy, et al.
Pharmacy, and the Opioid Risk Tool (ORT) validated for predicting individuals at risk for opioid medication misuse, were also included in the toolkit and discussed in the didactic lecture.20–22 At the conclusion of the lecture, students were given the assignment of reviewing the naloxone administration videos found on the Prescribe to Prevent® website prior to attending the hands-on activity.23 Activity Description After the 50-min didactic lecture, students attended a two-hour opioid misuse and overdose simulation. Each simulation group had a maximum number of 16 students where they worked in pairs to complete the activity. Faculty developed three case-specific simulation scenarios. All students had case number 1, and one student from each pair was assigned either case number 2 or 3. For each case, students were given a new prescription for a simulated patient, a paper-based medication profile for the patient including directions for use and dates of previously dispensed medications, and the patient's ORT results. Students had access to the previously provided toolkit while working through each case. Guided questions were provided to students, which were designed to walk them through the process of evaluating patients for opioid misuse and/or overdose. The guided questions were: 1. 2. 3. 4. 5.
Identify red flags for potential controlled substance misuse. Use the CDC Opioid Guidelines app to evaluate the prescribed morphine equivalency dose. Does this patient have any risk factors for potential overdose? If so, identify them. Use the ORT to determine the patient's risk for opioid misuse score and evaluate. Based on the information above, is this patient a candidate for naloxone?
Faculty worked with the Board of Pharmacy to input the simulated patients' information in the prescription drug monitoring program (PDMP) to create a realistic learning environment. Prior to completing each case, faculty demonstrated how to access the simulated patient's information in the state's PDMP. The PDMP is a valuable resource for pharmacists and use of the PDMP during this activity was designed to prepare students for advanced experiences and practice. Case 1 involved a patient presenting with an opioid prescription that met multiple objective red flag criteria for opioid misuse, including a concerning PDMP report. All students completed case 1 individually, after which they were directed to engage in the think-pair-share model to build knowledge and confidence with the use of the toolkit. By following the guided questions, students identified potential red flags of prescription misuse based on the patient's medication profile and the PDMP. Students also evaluated the patient's medication profile to determine if they were at risk for respiratory depression while taking a newly prescribed opioid medication. Students evaluated the simulated patient's ORT score, and the daily MME dose. Following independent evaluation and the sharing of their findings with their partner, faculty led the collective group through determining a course of action. Students were asked if they would prescribe and dispense naloxone for the patient. Debriefing following Case 1 included mock conversations with a patient and provider regarding concern for opioid misuse and ethical and legal concerns associated with this complex topic. A faculty member played the roles of the patient and provider during the mock conversations where possible situations were discussed and evidence-based communication skills such as motivational interviewing and screening, brief interventions, and referral to treatment (SBIRT) were modeled.24 Following Case 1, students independently completed a second case. Cases 2 and 3 were two unique scenarios with student pairs each having a different simulated patient. Independently, students completed the same process used with Case 1. Again, students shared their individual findings with their partner to discuss and outline potential next steps based on the findings. The students also used role-play to simulate a conversation with the patient and the provider based on the findings. For Case 2, the patient was a candidate for naloxone due to the risk for accidental overdose from respiratory depression. For Case 3, the patient was a candidate for naloxone due to a high risk for opioid misuse. Based on the assigned patient scenario students were expected to either prescribe or dispense Narcan® nasal spray or the Evzio® auto injector. Students prescribed the naloxone for the simulated patient in accordance with the state protocol, and then dispensed the naloxone using pharmacy dispensing software. Students recommended and counseled the patient on their newly prescribed naloxone medication based on communication techniques discussed during Case 1. Faculty were available for observation of the consultation and provided formative feedback when needed. Each student was then required to counsel their partner on the generic naloxone intranasal with the atomizer or the injectable naloxone. By the end of the activity, students had either counseled or heard a consultation for all four available naloxone dosage forms. Survey Design and Survey Methods Faculty who had experience and expertise in the topic area created a 22-item survey. The survey was administered to students at the beginning of the course lecture and immediately upon completion of the laboratory activity. Each student created a unique code to allow faculty to blindly match the survey results. Survey questions evaluated knowledge, self-efficacy, or perceived value. Of the 22 questions, eight assessed knowledge (Table 1), 11 assessed students' self-efficacy of assessing for opioid misuse and accidental overdose and for prescribing and dispensing of naloxone (Table 2), and three assessed students' perceived value of pharmacists and naloxone's role in fighting the opioid epidemic (Table 3). The eight knowledge questions were taken directly from the course content and online training assessment of the neuropsychiatric pharmacy course the previous semester. Knowledge questions were assessed in a multiple-choice format, and self-efficacy and perceived value questions were assessed using a five-point Likert scale (1 = strongly 524
Currents in Pharmacy Teaching and Learning 11 (2019) 522–527
E. Skoy, et al.
Table 1 Items evaluating student knowledge of naloxone. Knowledge questions
Number of students answering pre-survey correctly (n = 70)
Number of students answering post-survey correctly (n = 70)
t-Test significance
59 (84%)
69 (99%)
0.001
61 (87%)
70 (100%)
0.002
50 (71%)
62 (89%)
0.006
64 (91%)
69 (99%)
0.058
32 (46%)
40 (57%)
0.159
18 (26%)
62 (89%)
< 0.001
59 (84%)
70 (100%)
0.001
45 (64%)
67 (96%)
< 0.001
What can you tell a patient or family member about the risks associated with using naloxone? What should a rescuer do if a person experiencing an overdose doesn't respond to a dose of naloxone? What can you tell a patient and their caregiver about withdrawal symptoms if naloxone is given? If a person experiencing an overdose needs to be left alone, what is the proper position they should be left in? Labeling for this FDA-approved naloxone preparation includes instructions for layperson use. Narcan nasal spray and Evzio auto-injector must be stored at controlled room temperature at all times in order to guarantee their effectiveness. In the state of North Dakota pharmacists may dispense naloxone therapy to patients and/or their caregivers Patient education for naloxone therapy may be waived based on patient request, just like any other prescription medication.
FDA = Food and Drug Administration. Table 2 Student self-efficacy.a Self-efficacy questions “I feel confident in my ability to…” Counsel patients how to respond to an emergency opioid overdose situation. Instruct patients on the signs of opioid overdose. Counsel patients on appropriate options for the safe use and disposal of opioids. Counsel patients on how to administer Evzio auto-injector. Counsel patients on how to administer Narcan nasal spray. Counsel patients on how to administer naloxone with a vial and syringe. Counsel patients on how to administer naloxone via the intranasal route with an atomizer. Determine if a patient is at risk for opioid misuse. Determine patients whose PDMP profile should be examined. Identify patients who should be prescribed naloxone. Initiate a conversation with a patient and recommend the possession of naloxone.
Pre-survey
Post-survey
t-Test significance
2.89 2.24 1.90 2.91 2.94 2.76 3.19 2.27 2.27 2.54 2.79
1.56 1.59 1.56 1.49 1.53 1.54 1.61 1.69 1.60 1.61 1.77
< 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001
PDMP = prescription drug monitoring program. a 1 = strongly agree; 2 = agree; 3 = neutral; 4 = disagree; 5 = strongly disagree. Table 3 Student perceived value of pharmacists and naloxone.a Perceived value questions I will recommend naloxone to patients in my future practice. Pharmacists are a key player in the fight against the opioid epidemic. Naloxone is an important tool of patients at risk of opioid overdose. a
Pre-survey
Post-survey
t-Test significance
2.19 4.13 2.37
1.70 1.36 1.34
< 0.001 < 0.001 < 0.001
1 = strongly agree; 2 = agree; 3 = neutral; 4 = disagree; 5 = strongly disagree.
agree, 2 = agree, 3 = neutral, 4 = disagree, and 5 = strongly disagree). Survey items were analyzed with a paired sample t-test using SPSS.25 Cronbach's Alpha test was deployed to determine reliability of questions labeled as self-efficacy and perceived value. Correlation statistics were calculated using composite scores from all questions within each of the three question categories using Pearson correlation coefficient. The North Dakota State University Institutional Review Board approved the research for this project. Findings A total of 78 students were enrolled in the course and of those students, 70 completed the pre- and post-survey for a response rate of 89.7%. For six out of the eight knowledge questions, students demonstrated statistically significant gains in knowledge (p < 0.05) following the laboratory activity (Table 1). Students also showed a significant gain (p < 0.05) in self-efficacy and perceived value that was reflected in responses from all corresponding survey items. The Cronbach alpha coefficients for the self-efficacy and perceived value questions were 0.858 and 0.938, respectively, showing 525
Currents in Pharmacy Teaching and Learning 11 (2019) 522–527
E. Skoy, et al.
high reliability of the identified question categories. The Pearson correlation showed a significant (p < 0.05) correlation (0.587) between self-efficacy and perceived value of the pharmacists role. There was not a significant correlation with knowledge and perceived value (p = 0.59) or knowledge and self-efficacy (p = 0.73). Discussion Students showed a significant gain in knowledge, self-efficacy, and perceived value of the pharmacist's role in opioid misuse and the provision of naloxone. This is important because students had already completed the state requirement for training for pharmacists to prescribe naloxone. This implies that a training program consisting of a hands-on laboratory activity is of added value to regular didactic content. Faculty deemed the training program necessary and successful without the need for substantial changes; however, they have considered emphasizing characteristics of the various naloxone formulations due to the lower performance on the corresponding question from the knowledge portion of the survey. Although survey questions evaluating self-efficacy and perceived value were faculty developed, the questions were shown to be reliable which demonstrated that the questions were consistently able to differentiate among individuals. The correlation between perceived self-efficacy and perceived value is also important. If a student has confidence in their ability to complete a task they are more likely to see the value in it. The fact that knowledge does not correlate with self-efficacy or value is also of note. This shows that although students may have the necessary knowledge to address the opioid epidemic and prescribe naloxone as obtained through online modules or didactic training, knowledge alone may not sufficiently impact self-efficacy and perceived value. This significant gain in self-efficacy and perceived value is especially important since the literature has shown that pharmacists who are confident and find value in their role are more likely to dispense naloxone for patients.9 Other curricular activities, including didactic, laboratory, and OSCE-based simulations related to naloxone have been described in the literature.7,8,12–14 Results from other educational activities have reported students' gain in confidence to counsel about opioid overdose and administer or counsel on naloxone.7,8,13 Likewise, two studies have reported an increase in perceived value of pharmacist-provided services addressing the opioid crisis.7,13 This manuscript contributes to the literature by describing the design and assessment of a an opioid misuse and overdose training program, including a practice-ready toolkit, which focused on opioid misuse, accidental overdose, and naloxone prescribing and dispensing. This study is the first to evaluate student perceived self-efficacy of using tools and objective measures to identify patients at risk of accidental overdose and/or opioid misuse and to prescribe naloxone to appropriate patients. At the time of writing this manuscript, it is also the first study to evaluate the correlation between knowledge, self-efficacy, and perceived value of pharmacists' role in preventing overdose after the completion of a opioid misuse and overdose training program. In addition, this is the first study to compare knowledge, self-efficacy, and perceived-value of pharmacists' role in preventing overdose with the addition of a training program after the completion of an online training program, which is the most common state training requirement for pharmacists to participate in naloxone prescribing activities.6 The researchers did not survey students prior to the neuropsychiatry course content to obtain student's baseline levels in knowledge, self-efficacy, and perceived value. In addition, the researchers are unable to determine which component of the training played a more important role in student change (the didactic lecture that reinforced previously learned topics or the hands-on activity). This study was conducted within one institution, and it remains unknown if the same results would hold true for all student pharmacists. Lastly, although there was a significant gain in knowledge, self-efficacy, and perceived value of the pharmacist's role in opioid misuse and the provision of naloxone, our study did not assess how this would be transferred into their future pharmacy practice. Future research is needed in this area. Summary The important role of the pharmacist in fighting the opioid-epidemic through identifying opioid misuse, preventing overdose, and prescribing naloxone cannot be ignored. From this study we have identified that a training program can increase pharmacy students' knowledge, self-efficacy, and perceived value in their role in preventing accidental opioid overdose. In addition, we have shown that although students may gain knowledge through readings and didactic lectures, hands on practical application may make a significant impact on students' self-efficacy and perceived value. Conflict of Interest None. Disclosures None. References 1. U.S. Department of Health and Human Services. What is the U.S. Opioid Epidemic? https://www.hhs.gov/opioids/about-the-epidemic/index.html; 2018 Updated 2018. Accessed 13 April 2019.
526
Currents in Pharmacy Teaching and Learning 11 (2019) 522–527
E. Skoy, et al.
2. Bailey AM, Wermeling DP. Naloxone for opioid overdose prevention: pharmacists' role in community-based practice settings. Ann Pharmacother. 2014;48(5):601–606. 3. American Pharmacists Association. Resource Guide for Immunization Servies. https://www.pharmacist.com/resource-guide-immunization-services; 2019 Updated 2019. Accessed 13 April 2019. 4. American Society of Health-System Pharmacists. ASHP statement on the pharmacist's role in substance abuse prevention, education, and assistance. Am J Health Syst Pharm. 2014;71(3):243–246. 5. National Alliance of State Pharmacy Associations. Naloxone Access in Community Pharmacies. https://naspa.us/resource/naloxone-access-communitypharmacies; 2018 Published 5 April 2018. Accessed 13 April 2019. 6. Roberts AW, Carpenter DM, Smith A, Look KA. Reviewing state-mandated training requirements for naloxone-dispensing pharmacists. Res Soc Adm Pharm. 2018. https://doi.org/10.1016/j.sapharm.2018.04.002. 7. Schartel A, Lardieri A, Mattingly A, Feemster AA. Implementation and assessment of a naloxone-training program for first-year student pharmacists. Curr Pharm Teach Learn. 2018;10(6):717–722. 8. Hines J, Deja E, Black EP. Student pharmacist perceptions of participation in hands-on naloxone counseling. Curr Pharm Teach Learn. 2018;10(6):712–716. 9. Freeman PR, Goodin A, Troske S, Strahl A, Fallin A, Green TC. Pharmacists' role in opioid overdose: Kentucky pharmacists' willingness to participate in naloxone dispensing. J Am Pharm Assoc (2003). 2017;57(2):S28–S33. 10. House of Delegates Sessions. July 24–27, Anaheim, California. Am J Pharm Educ. 2016;80(7) https://doi.org/10.5688/ajpe807S8. 11. Medina MS, Plaza CM, Stowe CD, et al. Center for the advancement of pharmacy education 2013 educational outcomes. Am J Pharm Educ. 2013;77(8) https://doi. org/10.5688/ajpe778162. 12. Maguire MA, Pavlakos RN, Mehta BH, Schmuhl KK, Beatty SJ. A naloxone and harm reduction educational program across four years of a doctor of pharmacy program. Curr Pharm Teach Learn. 2018;10(1):72–77. 13. Donohoe KL, Raghavan A, Tran TT, Alotaibi FM, Powers KE, Morgan L. Preparing pharmacy students to manage the opioid crisis. Am J Pharm Educ. 2018 https:// www.ajpe.org/doi/abs/10.5688/ajpe6988 Accessed 13 April 2019. 14. Jacobson AN, Bratberg JP, Monk M, Ferrentino J. Retention of student pharmacists' knowledge and skills regarding overdose management with naloxone. Subst Abus. 2018;39(2):193–198. 15. CE Online. Pharmacist's Letter. Naloxone for the Prevention of Opioid-Related Deaths. http://pharmacistsletter.therapeuticresearch.com/ce/cecourse.aspx?pc= 16-242; 2019 Updated 2019. Accessed 13 April 2019. 16. OpiSafe, Inc. OpiRescue. https://opirescue.com; 2019 [Updated 2019. Accessed 6 February 2019]. 17. Centers for Disease Control and Prevention. Guideline Resources: CDC Opioid Guideline Mobile App. https://www.cdc.gov/drugoverdose/prescribing/app.html; 2017 Updated August 30, 2017. Accessed 13 April 2019. 18. Substance Abuse and Mental Health Services Administration. SAMHSA Opioid Overdose Prevention TOOLKIT: Information for Prescribers. https://store.samhsa. gov/system/files/information-for-prescribers.pdf; 2018 Updated 2018. Accessed 13 April 2019. 19. U.S. Department of Justice. Drug Enforcement Administration, Diversion Control Division. Pharmacist's Manual: Appendix D: Pharmacist's Guide to Prescription Fraud. https://www.deadiversion.usdoj.gov/pubs/manuals/pharm2/appendix/appdx_d.htm; 2010 Published 2010. Accessed 13 April 2019. 20. North Dakota Board of Pharmacy. Naloxone Prescribing for Pharmacists. https://www.nodakpharmacy.com/naloxone.asp; 2019 Updated 2019. Accessed 13 April 2019. 21. Webster LR, Webster RM. Predicting aberrant behaviors in opioid-treated patients: preliminary validation of the opioid risk tool. Pain Med. 2005;6(6):432–442. 22. Crawford C. ‘Red Flag’ Document Offers Guidance on Prescribing Controlled Substances: AAFP Joins Physician, Pharmacist Associations, Retail Chains to Create Resource. https://www.aafp.org/news/practice-professional-issues/20150313redflagdoc.html; 2015 Published March 13, 2015. Accessed 13 April 2019. 23. Prescribe to Prevent. Videos. http://prescribetoprevent.org/patient-education/videos/; 2015 Updated 2015. Accessed 13 April 2019. 24. Institute of Medicine. Broadening the Base of Treatment for Alcohol Problems. Washington, DC: The National Academies Press; 1990. 25. Armonk, NY: IBM Corp. SPSS for Windows [Computer Program]. Version 25.0. 2017; 2017.
527
Contents lists available at ScienceDirect
Currents in Pharmacy Teaching and Learning journal homepage: www.elsevier.com/locate/cptl
Experiences in Teaching and Learning
Preparing student pharmacists to identify opioid misuse, prevent overdose and prescribe naloxone
T
Elizabeth Skoy , Heidi Eukel, Jeanne Frenzel, Amy Werremeyer ⁎
North Dakota State University School of Pharmacy, PO Box 6050, Dept 2660, Fargo, ND 58018-6050, United States
ARTICLE INFO
ABSTRACT
Keywords: Naloxone Opioid Simulation Laboratory activity
Background and Purpose: All 50 states have implemented policies to improve access to naloxone through community pharmacies. Many states require naloxone-based training for pharmacists before participating in these activities. The purpose of this study was to determine the effects of an opioid misuse and overdose training program on students' knowledge, self-efficacy, and value of pharmacists' role in preventing overdose. Educational Activity and Setting: The training program was implemented with third-year professional pharmacy students. Students participated in a 50-min lecture followed by a two-hour hands-on laboratory activity. Students took a pre-survey immediately before the lecture and a post-survey immediately following the laboratory activity. Findings: Seventy students completed the pre- and post-surveys. Students showed a significant (p < 0.05) gain in self-efficacy and value after completing the training program, and a significant gain in knowledge from six of the eight related survey items. There was a significant correlation between students' self-efficacy and perceived value of the pharmacists' role in preventing opioid overdose. Summary: A training program implemented for third-year pharmacy students increases students' knowledge, self-efficacy, and perceived value. Specifically, the strong correlation seen between self-efficacy and perceived value after completing the training program is important to note as we prepare future pharmacists for their role in fighting the opioid epidemic.
Background and Purpose In 2017, the United States (US) Department of Health and Human Services declared the opioid epidemic a public health emergency and published five strategies for combatting opioid misuse. One of the strategies was to increase availability and use of overdose reversing drugs, such as naloxone.1 Pharmacists have a unique opportunity to dispense naloxone and educate patients on the use of this life-saving medication due to their expertise and their position as the most accessible healthcare provider.2–4 All 50 states have implemented a community pharmacy naloxone access policy. These policies range from the ability for pharmacists to prescribe and dispense naloxone, dispense naloxone without a prescription, and the ability for pharmacists to directly dispense naloxone under a state issued standing order.5 Currently, 19 states, including North Dakota, mandate that pharmacists complete a naloxone-based training before participating in activities allowed through the naloxone access policies. The most common training requirement among these states is a one- to two-hour continuing education (CE) course.6 In addition, some states require pharmacists
Corresponding author. E-mail addresses: [email protected] (E. Skoy), [email protected] (H. Eukel), [email protected] (J. Frenzel), [email protected] (A. Werremeyer). ⁎
https://doi.org/10.1016/j.cptl.2019.02.013
1877-1297/ © 2019 Elsevier Inc. All rights reserved.
Currents in Pharmacy Teaching and Learning 11 (2019) 522–527
E. Skoy, et al.
to complete annual or biennial CE to maintain their ability to participate in naloxone access programs.6 Research has shown that naloxone training increases student pharmacists' and pharmacists' willingness and confidence in their ability to dispense and counsel on naloxone.6–8 A survey of pharmacists in Kentucky found that an increase in reported pharmacist confidence in identifying individuals at risk for overdose was associated with a significant increase in willingness to dispense naloxone. The same study also found that an increase in reported pharmacist confidence in the ability to educate patients about overdose was associated with a significant increase in willingness to dispense naloxone.9 An additional study reviewing naloxone training programs called for the inclusion of content addressing how to communicate the need for naloxone with patients.6 The American Association of Colleges of Pharmacy (AACP) has called for all pharmacy graduates to be able to provide and administer naloxone as well as properly educate individuals who may experience or encounter an opioid overdose.10 In alignment with this call, the 2013 Center for the Advancement of Pharmacy Education (CAPE) Educational Outcomes outline the expectation that graduates must be able to educate patients and assess their understanding.11 Since, there have been published works addressing the need for training and educating future pharmacists on the expansion of opioid overdose interventions, including naloxone prescribing.7,8,12–14 The description of these educational activities have included the incorporation of a related Objective Structured Clinical Examination (OSCE), community naloxone counseling, lectures, hands-on workshops, and laboratory activities. The purpose of this study was to determine the effects of an opioid misuse and accidental overdose training program on students' knowledge, selfefficacy, and perceived value of pharmacists' role in preventing overdose. Educational Activity and Setting The training program described in this manuscript took place in Pharmacy Practice Laboratory IV, which takes place in the spring semester of the third-year professional pharmacy program, prior to advanced pharmacy practice experiences (APPEs). Pharmacy Practice Laboratory IV is part of a four-semester course series designed to teach and assess the practical application of skills needed for students to be able to practice at the top of their license. The course is comprised of a weekly 50-min lecture and a two-hour laboratory session that are conducted in small groups of 16 students. Topics in this course focus on advancing roles of the pharmacist, such as point-of care testing, medication therapy management (MTM), and disease state management. To address the AACP call to action and expose students to the advancing role of the pharmacist in the fight against the opioid epidemic, a training program was created and added to the curriculum to develop a student's ability to assess for opioid misuse and accidental overdose and to prescribe and dispense naloxone. The training program consisted of a 50-min lecture and a two-hour hands-on activity. Pre-activity Preparation During the fall semester of the third professional year, 12 weeks prior to the laboratory activity, students received two and onehalf hours of didactic instruction on opioid misuse and accidental overdose in a neuropsychiatric pharmacotherapy course. As part of this course, students also completed a one-hour online CE training module focused on identifying persons at risk of respiratory depression from opioids, signs of opioid overdose, proper administration of naloxone, and a pharmacists' role in dispensing naloxone.15 Verification of program completion included handing in the certificate of completion attesting they had passed the final CE quiz. It is important to note that this CE is a suggested module used to fulfill the state requirement of one-hour topic-related education to allow pharmacists to prescribe naloxone. Although this fulfilled the requirement, faculty determined additional education and a hands-on laboratory activity would be beneficial to further advance the curriculum. The opioid misuse and overdose training program took place the semester after the didactic instruction on this content. A 50-min didactic lecture was used to reinforce topics previously addressed including North Dakota state-specific protocol for pharmacists' prescribing naloxone, risk factors for opioid misuse and accidental overdose, how to respond to an opioid overdose situation, and information regarding the four available naloxone dosage forms. During the lecture, students were provided with an opioid misuse and overdose toolkit (Fig. 1) that was developed by faculty based on their practice experience and expertise. The toolkit included practice-based tools for students to utilize during the laboratory activity and included useful mobile apps such as Opi-Rescue®, that could be used to instruct and aid patients in how to respond to an opioid overdose, and the Centers for Disease Control and Prevention (CDC) Opioid Guidelines app, which has a built-in feature to calculate daily morphine milligram equivalents (MMEs).16,17 The toolkit also included a red flags checklist supported by the literature to identify individuals who may be misusing controlled substances. These red flags included but are not limited to individuals requesting to pay cash, receiving medications from multiple providers, and requesting early medication refills.18,19 A faculty-developed naloxone consultation checklist, supported by the North Dakota Board of ■ ■ ■ ■ ■ ■
State protocol for naloxone prescribing and dispensing15 Opioid Risk Tool16 Pharmacy Red Flag Checklist17 Naloxone Consultation Checklist15 OpiRescue App11 CDC Opioid MME Calculator12
CDC = Center for Disease Control; MME = morphine milligram equivalents. Fig. 1. Opioid misuse and overdose toolkit. CDC = Center for Disease Control; MME = morphine milligram equivalents. 523
Currents in Pharmacy Teaching and Learning 11 (2019) 522–527
E. Skoy, et al.
Pharmacy, and the Opioid Risk Tool (ORT) validated for predicting individuals at risk for opioid medication misuse, were also included in the toolkit and discussed in the didactic lecture.20–22 At the conclusion of the lecture, students were given the assignment of reviewing the naloxone administration videos found on the Prescribe to Prevent® website prior to attending the hands-on activity.23 Activity Description After the 50-min didactic lecture, students attended a two-hour opioid misuse and overdose simulation. Each simulation group had a maximum number of 16 students where they worked in pairs to complete the activity. Faculty developed three case-specific simulation scenarios. All students had case number 1, and one student from each pair was assigned either case number 2 or 3. For each case, students were given a new prescription for a simulated patient, a paper-based medication profile for the patient including directions for use and dates of previously dispensed medications, and the patient's ORT results. Students had access to the previously provided toolkit while working through each case. Guided questions were provided to students, which were designed to walk them through the process of evaluating patients for opioid misuse and/or overdose. The guided questions were: 1. 2. 3. 4. 5.
Identify red flags for potential controlled substance misuse. Use the CDC Opioid Guidelines app to evaluate the prescribed morphine equivalency dose. Does this patient have any risk factors for potential overdose? If so, identify them. Use the ORT to determine the patient's risk for opioid misuse score and evaluate. Based on the information above, is this patient a candidate for naloxone?
Faculty worked with the Board of Pharmacy to input the simulated patients' information in the prescription drug monitoring program (PDMP) to create a realistic learning environment. Prior to completing each case, faculty demonstrated how to access the simulated patient's information in the state's PDMP. The PDMP is a valuable resource for pharmacists and use of the PDMP during this activity was designed to prepare students for advanced experiences and practice. Case 1 involved a patient presenting with an opioid prescription that met multiple objective red flag criteria for opioid misuse, including a concerning PDMP report. All students completed case 1 individually, after which they were directed to engage in the think-pair-share model to build knowledge and confidence with the use of the toolkit. By following the guided questions, students identified potential red flags of prescription misuse based on the patient's medication profile and the PDMP. Students also evaluated the patient's medication profile to determine if they were at risk for respiratory depression while taking a newly prescribed opioid medication. Students evaluated the simulated patient's ORT score, and the daily MME dose. Following independent evaluation and the sharing of their findings with their partner, faculty led the collective group through determining a course of action. Students were asked if they would prescribe and dispense naloxone for the patient. Debriefing following Case 1 included mock conversations with a patient and provider regarding concern for opioid misuse and ethical and legal concerns associated with this complex topic. A faculty member played the roles of the patient and provider during the mock conversations where possible situations were discussed and evidence-based communication skills such as motivational interviewing and screening, brief interventions, and referral to treatment (SBIRT) were modeled.24 Following Case 1, students independently completed a second case. Cases 2 and 3 were two unique scenarios with student pairs each having a different simulated patient. Independently, students completed the same process used with Case 1. Again, students shared their individual findings with their partner to discuss and outline potential next steps based on the findings. The students also used role-play to simulate a conversation with the patient and the provider based on the findings. For Case 2, the patient was a candidate for naloxone due to the risk for accidental overdose from respiratory depression. For Case 3, the patient was a candidate for naloxone due to a high risk for opioid misuse. Based on the assigned patient scenario students were expected to either prescribe or dispense Narcan® nasal spray or the Evzio® auto injector. Students prescribed the naloxone for the simulated patient in accordance with the state protocol, and then dispensed the naloxone using pharmacy dispensing software. Students recommended and counseled the patient on their newly prescribed naloxone medication based on communication techniques discussed during Case 1. Faculty were available for observation of the consultation and provided formative feedback when needed. Each student was then required to counsel their partner on the generic naloxone intranasal with the atomizer or the injectable naloxone. By the end of the activity, students had either counseled or heard a consultation for all four available naloxone dosage forms. Survey Design and Survey Methods Faculty who had experience and expertise in the topic area created a 22-item survey. The survey was administered to students at the beginning of the course lecture and immediately upon completion of the laboratory activity. Each student created a unique code to allow faculty to blindly match the survey results. Survey questions evaluated knowledge, self-efficacy, or perceived value. Of the 22 questions, eight assessed knowledge (Table 1), 11 assessed students' self-efficacy of assessing for opioid misuse and accidental overdose and for prescribing and dispensing of naloxone (Table 2), and three assessed students' perceived value of pharmacists and naloxone's role in fighting the opioid epidemic (Table 3). The eight knowledge questions were taken directly from the course content and online training assessment of the neuropsychiatric pharmacy course the previous semester. Knowledge questions were assessed in a multiple-choice format, and self-efficacy and perceived value questions were assessed using a five-point Likert scale (1 = strongly 524
Currents in Pharmacy Teaching and Learning 11 (2019) 522–527
E. Skoy, et al.
Table 1 Items evaluating student knowledge of naloxone. Knowledge questions
Number of students answering pre-survey correctly (n = 70)
Number of students answering post-survey correctly (n = 70)
t-Test significance
59 (84%)
69 (99%)
0.001
61 (87%)
70 (100%)
0.002
50 (71%)
62 (89%)
0.006
64 (91%)
69 (99%)
0.058
32 (46%)
40 (57%)
0.159
18 (26%)
62 (89%)
< 0.001
59 (84%)
70 (100%)
0.001
45 (64%)
67 (96%)
< 0.001
What can you tell a patient or family member about the risks associated with using naloxone? What should a rescuer do if a person experiencing an overdose doesn't respond to a dose of naloxone? What can you tell a patient and their caregiver about withdrawal symptoms if naloxone is given? If a person experiencing an overdose needs to be left alone, what is the proper position they should be left in? Labeling for this FDA-approved naloxone preparation includes instructions for layperson use. Narcan nasal spray and Evzio auto-injector must be stored at controlled room temperature at all times in order to guarantee their effectiveness. In the state of North Dakota pharmacists may dispense naloxone therapy to patients and/or their caregivers Patient education for naloxone therapy may be waived based on patient request, just like any other prescription medication.
FDA = Food and Drug Administration. Table 2 Student self-efficacy.a Self-efficacy questions “I feel confident in my ability to…” Counsel patients how to respond to an emergency opioid overdose situation. Instruct patients on the signs of opioid overdose. Counsel patients on appropriate options for the safe use and disposal of opioids. Counsel patients on how to administer Evzio auto-injector. Counsel patients on how to administer Narcan nasal spray. Counsel patients on how to administer naloxone with a vial and syringe. Counsel patients on how to administer naloxone via the intranasal route with an atomizer. Determine if a patient is at risk for opioid misuse. Determine patients whose PDMP profile should be examined. Identify patients who should be prescribed naloxone. Initiate a conversation with a patient and recommend the possession of naloxone.
Pre-survey
Post-survey
t-Test significance
2.89 2.24 1.90 2.91 2.94 2.76 3.19 2.27 2.27 2.54 2.79
1.56 1.59 1.56 1.49 1.53 1.54 1.61 1.69 1.60 1.61 1.77
< 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001
PDMP = prescription drug monitoring program. a 1 = strongly agree; 2 = agree; 3 = neutral; 4 = disagree; 5 = strongly disagree. Table 3 Student perceived value of pharmacists and naloxone.a Perceived value questions I will recommend naloxone to patients in my future practice. Pharmacists are a key player in the fight against the opioid epidemic. Naloxone is an important tool of patients at risk of opioid overdose. a
Pre-survey
Post-survey
t-Test significance
2.19 4.13 2.37
1.70 1.36 1.34
< 0.001 < 0.001 < 0.001
1 = strongly agree; 2 = agree; 3 = neutral; 4 = disagree; 5 = strongly disagree.
agree, 2 = agree, 3 = neutral, 4 = disagree, and 5 = strongly disagree). Survey items were analyzed with a paired sample t-test using SPSS.25 Cronbach's Alpha test was deployed to determine reliability of questions labeled as self-efficacy and perceived value. Correlation statistics were calculated using composite scores from all questions within each of the three question categories using Pearson correlation coefficient. The North Dakota State University Institutional Review Board approved the research for this project. Findings A total of 78 students were enrolled in the course and of those students, 70 completed the pre- and post-survey for a response rate of 89.7%. For six out of the eight knowledge questions, students demonstrated statistically significant gains in knowledge (p < 0.05) following the laboratory activity (Table 1). Students also showed a significant gain (p < 0.05) in self-efficacy and perceived value that was reflected in responses from all corresponding survey items. The Cronbach alpha coefficients for the self-efficacy and perceived value questions were 0.858 and 0.938, respectively, showing 525
Currents in Pharmacy Teaching and Learning 11 (2019) 522–527
E. Skoy, et al.
high reliability of the identified question categories. The Pearson correlation showed a significant (p < 0.05) correlation (0.587) between self-efficacy and perceived value of the pharmacists role. There was not a significant correlation with knowledge and perceived value (p = 0.59) or knowledge and self-efficacy (p = 0.73). Discussion Students showed a significant gain in knowledge, self-efficacy, and perceived value of the pharmacist's role in opioid misuse and the provision of naloxone. This is important because students had already completed the state requirement for training for pharmacists to prescribe naloxone. This implies that a training program consisting of a hands-on laboratory activity is of added value to regular didactic content. Faculty deemed the training program necessary and successful without the need for substantial changes; however, they have considered emphasizing characteristics of the various naloxone formulations due to the lower performance on the corresponding question from the knowledge portion of the survey. Although survey questions evaluating self-efficacy and perceived value were faculty developed, the questions were shown to be reliable which demonstrated that the questions were consistently able to differentiate among individuals. The correlation between perceived self-efficacy and perceived value is also important. If a student has confidence in their ability to complete a task they are more likely to see the value in it. The fact that knowledge does not correlate with self-efficacy or value is also of note. This shows that although students may have the necessary knowledge to address the opioid epidemic and prescribe naloxone as obtained through online modules or didactic training, knowledge alone may not sufficiently impact self-efficacy and perceived value. This significant gain in self-efficacy and perceived value is especially important since the literature has shown that pharmacists who are confident and find value in their role are more likely to dispense naloxone for patients.9 Other curricular activities, including didactic, laboratory, and OSCE-based simulations related to naloxone have been described in the literature.7,8,12–14 Results from other educational activities have reported students' gain in confidence to counsel about opioid overdose and administer or counsel on naloxone.7,8,13 Likewise, two studies have reported an increase in perceived value of pharmacist-provided services addressing the opioid crisis.7,13 This manuscript contributes to the literature by describing the design and assessment of a an opioid misuse and overdose training program, including a practice-ready toolkit, which focused on opioid misuse, accidental overdose, and naloxone prescribing and dispensing. This study is the first to evaluate student perceived self-efficacy of using tools and objective measures to identify patients at risk of accidental overdose and/or opioid misuse and to prescribe naloxone to appropriate patients. At the time of writing this manuscript, it is also the first study to evaluate the correlation between knowledge, self-efficacy, and perceived value of pharmacists' role in preventing overdose after the completion of a opioid misuse and overdose training program. In addition, this is the first study to compare knowledge, self-efficacy, and perceived-value of pharmacists' role in preventing overdose with the addition of a training program after the completion of an online training program, which is the most common state training requirement for pharmacists to participate in naloxone prescribing activities.6 The researchers did not survey students prior to the neuropsychiatry course content to obtain student's baseline levels in knowledge, self-efficacy, and perceived value. In addition, the researchers are unable to determine which component of the training played a more important role in student change (the didactic lecture that reinforced previously learned topics or the hands-on activity). This study was conducted within one institution, and it remains unknown if the same results would hold true for all student pharmacists. Lastly, although there was a significant gain in knowledge, self-efficacy, and perceived value of the pharmacist's role in opioid misuse and the provision of naloxone, our study did not assess how this would be transferred into their future pharmacy practice. Future research is needed in this area. Summary The important role of the pharmacist in fighting the opioid-epidemic through identifying opioid misuse, preventing overdose, and prescribing naloxone cannot be ignored. From this study we have identified that a training program can increase pharmacy students' knowledge, self-efficacy, and perceived value in their role in preventing accidental opioid overdose. In addition, we have shown that although students may gain knowledge through readings and didactic lectures, hands on practical application may make a significant impact on students' self-efficacy and perceived value. Conflict of Interest None. Disclosures None. References 1. U.S. Department of Health and Human Services. What is the U.S. Opioid Epidemic? https://www.hhs.gov/opioids/about-the-epidemic/index.html; 2018 Updated 2018. Accessed 13 April 2019.
526
Currents in Pharmacy Teaching and Learning 11 (2019) 522–527
E. Skoy, et al.
2. Bailey AM, Wermeling DP. Naloxone for opioid overdose prevention: pharmacists' role in community-based practice settings. Ann Pharmacother. 2014;48(5):601–606. 3. American Pharmacists Association. Resource Guide for Immunization Servies. https://www.pharmacist.com/resource-guide-immunization-services; 2019 Updated 2019. Accessed 13 April 2019. 4. American Society of Health-System Pharmacists. ASHP statement on the pharmacist's role in substance abuse prevention, education, and assistance. Am J Health Syst Pharm. 2014;71(3):243–246. 5. National Alliance of State Pharmacy Associations. Naloxone Access in Community Pharmacies. https://naspa.us/resource/naloxone-access-communitypharmacies; 2018 Published 5 April 2018. Accessed 13 April 2019. 6. Roberts AW, Carpenter DM, Smith A, Look KA. Reviewing state-mandated training requirements for naloxone-dispensing pharmacists. Res Soc Adm Pharm. 2018. https://doi.org/10.1016/j.sapharm.2018.04.002. 7. Schartel A, Lardieri A, Mattingly A, Feemster AA. Implementation and assessment of a naloxone-training program for first-year student pharmacists. Curr Pharm Teach Learn. 2018;10(6):717–722. 8. Hines J, Deja E, Black EP. Student pharmacist perceptions of participation in hands-on naloxone counseling. Curr Pharm Teach Learn. 2018;10(6):712–716. 9. Freeman PR, Goodin A, Troske S, Strahl A, Fallin A, Green TC. Pharmacists' role in opioid overdose: Kentucky pharmacists' willingness to participate in naloxone dispensing. J Am Pharm Assoc (2003). 2017;57(2):S28–S33. 10. House of Delegates Sessions. July 24–27, Anaheim, California. Am J Pharm Educ. 2016;80(7) https://doi.org/10.5688/ajpe807S8. 11. Medina MS, Plaza CM, Stowe CD, et al. Center for the advancement of pharmacy education 2013 educational outcomes. Am J Pharm Educ. 2013;77(8) https://doi. org/10.5688/ajpe778162. 12. Maguire MA, Pavlakos RN, Mehta BH, Schmuhl KK, Beatty SJ. A naloxone and harm reduction educational program across four years of a doctor of pharmacy program. Curr Pharm Teach Learn. 2018;10(1):72–77. 13. Donohoe KL, Raghavan A, Tran TT, Alotaibi FM, Powers KE, Morgan L. Preparing pharmacy students to manage the opioid crisis. Am J Pharm Educ. 2018 https:// www.ajpe.org/doi/abs/10.5688/ajpe6988 Accessed 13 April 2019. 14. Jacobson AN, Bratberg JP, Monk M, Ferrentino J. Retention of student pharmacists' knowledge and skills regarding overdose management with naloxone. Subst Abus. 2018;39(2):193–198. 15. CE Online. Pharmacist's Letter. Naloxone for the Prevention of Opioid-Related Deaths. http://pharmacistsletter.therapeuticresearch.com/ce/cecourse.aspx?pc= 16-242; 2019 Updated 2019. Accessed 13 April 2019. 16. OpiSafe, Inc. OpiRescue. https://opirescue.com; 2019 [Updated 2019. Accessed 6 February 2019]. 17. Centers for Disease Control and Prevention. Guideline Resources: CDC Opioid Guideline Mobile App. https://www.cdc.gov/drugoverdose/prescribing/app.html; 2017 Updated August 30, 2017. Accessed 13 April 2019. 18. Substance Abuse and Mental Health Services Administration. SAMHSA Opioid Overdose Prevention TOOLKIT: Information for Prescribers. https://store.samhsa. gov/system/files/information-for-prescribers.pdf; 2018 Updated 2018. Accessed 13 April 2019. 19. U.S. Department of Justice. Drug Enforcement Administration, Diversion Control Division. Pharmacist's Manual: Appendix D: Pharmacist's Guide to Prescription Fraud. https://www.deadiversion.usdoj.gov/pubs/manuals/pharm2/appendix/appdx_d.htm; 2010 Published 2010. Accessed 13 April 2019. 20. North Dakota Board of Pharmacy. Naloxone Prescribing for Pharmacists. https://www.nodakpharmacy.com/naloxone.asp; 2019 Updated 2019. Accessed 13 April 2019. 21. Webster LR, Webster RM. Predicting aberrant behaviors in opioid-treated patients: preliminary validation of the opioid risk tool. Pain Med. 2005;6(6):432–442. 22. Crawford C. ‘Red Flag’ Document Offers Guidance on Prescribing Controlled Substances: AAFP Joins Physician, Pharmacist Associations, Retail Chains to Create Resource. https://www.aafp.org/news/practice-professional-issues/20150313redflagdoc.html; 2015 Published March 13, 2015. Accessed 13 April 2019. 23. Prescribe to Prevent. Videos. http://prescribetoprevent.org/patient-education/videos/; 2015 Updated 2015. Accessed 13 April 2019. 24. Institute of Medicine. Broadening the Base of Treatment for Alcohol Problems. Washington, DC: The National Academies Press; 1990. 25. Armonk, NY: IBM Corp. SPSS for Windows [Computer Program]. Version 25.0. 2017; 2017.
527