- Email: [email protected]
Effectiveness of a pharmacy teaching certificate program offered to practicing pharmacists
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Contents lists available at ScienceDirect
Currents in Pharmacy Teaching and Learning journal homepage: www.elsevier.com/locate/cptl
Research Note
Effectiveness of a pharmacy teaching certificate program offered to practicing pharmacists ⁎
Jaclyn D. Cole, Melissa J. Ruble , Gwendolyn Wantuch, Kamila A. Dell, Erini Serag-Bolos, Carol Fox, Janelle Perkins Pharmacotherapeutics and Clinical Research Department, University of South Florida College of Pharmacy, United States
A R T IC LE I N F O
ABS TRA CT
Keywords: Teaching certificate program Preceptor development Academia Pharmacists Knowledge
Introduction: Regardless of the practice setting, pharmacists' roles often expand beyond patientcentered services and into health-related educational roles for patients, healthcare providers, and student pharmacists. Teaching roles of a clinical pharmacist can include providing invited presentations (such as continuing education lectures), teaching colleagues about new clinical initiatives, precepting students during advanced pharmacy practice experiences (APPEs) and introductory pharmacy practice experiences (IPPEs), and precepting residents on rotations. This program was designed to evaluate the effect of a university-affiliated pharmacy teaching certificate program on participant knowledge and perceived confidence in specific areas of academic pharmacy and preceptor development. Methods: A 10-month program consisting of four live modules with associated continuing education (CE) credit was offered to pharmacy residents and practicing pharmacists in the surrounding area. The program format utilized didactic and active learning strategies with “at home” assignments to reinforce material presented. Content knowledge was measured with written summative assessments provided at the start and end of each module. Perceived confidence was assessed through online surveys provided at the start and end of the program. All results were evaluated using paired t-tests. Results: Data were obtained from 14 participants in the first cohort and 18 participants in the second cohort. For each of the four modules, there was a significant difference in pre- and postknowledge assessments. Confidence data was also significantly increased regarding topics in each module. Conclusion: Knowledge assessments and perception surveys suggest that this university-affiliated teaching certificate program improved both knowledge and perceived confidence for program participants.
Introduction Pharmacy school curricula emphasize appropriate strategies for patient counseling and education, but an emphasis on teaching approaches and mentoring students from an academic perspective is lacking. Recent survey results from United States colleges and schools of pharmacy suggest that academic-focused advanced pharmacy practice experiences (APPEs) are the most common training
⁎
Corresponding author. E-mail addresses: [email protected] (J.D. Cole), [email protected] (M.J. Ruble), [email protected] (G. Wantuch), [email protected] (K.A. Dell), [email protected] (E. Serag-Bolos), [email protected] (C. Fox), [email protected] (J. Perkins). https://doi.org/10.1016/j.cptl.2019.07.002
1877-1297/ © 2019 Elsevier Inc. All rights reserved.
Please cite this article as: Jaclyn D. Cole, et al., Currents in Pharmacy Teaching and Learning, https://doi.org/10.1016/j.cptl.2019.07.002
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
J.D. Cole, et al.
method offered to cover these aspects, but less than half of the programs offer didactic coursework or study opportunities for those interested in an academic career.1 A shortage of experienced faculty members at newer colleges and schools of pharmacy, as well as a shortage of trained clinical pharmacists who are obliged to take on these expanded teaching roles, further emphasize the need for more structured programs to mentor potential academicians and preceptors.2 Postgraduate year one (PGY1) and postgraduate year two (PGY2) pharmacy residency programs may offer teaching certificate program opportunities in conjunction with colleges of pharmacy. Although this is not specifically required by the American Society of Health-System Pharmacists (ASHP) for accreditation, there are standards related to residency program competency in “teaching, education, and dissemination of knowledge” that remain unclear and lack standardization.3 The American College of Clinical Pharmacy (ACCP) Task Force on Residencies has developed two guidelines that clarify roles and expectations of teaching certificate programs within residencies.4 The American Association of Colleges of Pharmacy (AACP) and ASHP also teamed up to guide the development of teaching certificate programs within postgraduate education.5 Both guides emphasize a greater focus on a teaching and learning curricula, encourage colleges and schools of pharmacy to provide these programs, and provide suggestions for curriculum content and assessment. A literature review of Medline, Ovid, and Embase using “teaching certificate programs and pharmacy” as search terms was conducted to identify common components of existing successful programs and to determine the type of programmatic outcomes evaluated. Guidelines mentioned above were also reviewed for specific content and best practices in developing such programs. Common components that previous teaching certificate programs have utilized include educational presentations, lecture objectives, small group discussion facilitation, and student progress evaluation.6 The type of outcomes evaluated through previous studies included participant satisfaction, perceptions of improvement, and perceived value of program content.7,8 Longitudinal studies have been conducted and suggest that implementation of teaching certificate programs increase participant confidence and interest in academia and precepting.9,10 Wahl and colleagues9 surveyed graduates of a teaching certificate program over twelve years and found that delivery of such programs may encourage participants to pursue academia as a career. These results also reinforced previous studies that have shown increased confidence in teaching and precepting students. In addition, McNatty and colleagues10 surveyed residency graduates from ASHP-accredited programs from 2003 through 2006 and found that those who had received education in teaching methods during their postgraduate education were more likely to aspire to academic positions. Finally, literature has also noted the importance of professional mentorship for participants completing such a program, with emphasis placed on pharmacy faculty to provide this guidance for potential future pharmacy educators.11 Although several articles discuss teaching certificate programs, none were found that included pharmacists outside of postgraduate training programs or assessed participant knowledge and confidence. There was also limited information regarding the use of mentors as an integral part of such programs. Since precepting and clinical education roles contain components of teaching, evaluation, and assessment, the authors of this study decided to create and offer such a program to not only pharmacy residents but practicing pharmacists as well. This program was different from many others as it included one-on-one mentoring for all participants which allowed for a more personalized training plan. The objective of this study was to evaluate the impact of a university-affiliated teaching certificate program on both participant knowledge and perceived confidence in specific areas of academic pharmacy and preceptor development. Methods A cohort study of participants enrolled in a teaching certificate program developed for practicing pharmacists in the surrounding area of the host university was undertaken. The intent of the longitudinal program was to offer preceptor development with a focus on academic aspects of teaching pedagogy, assessment, and evaluation. This 10-month program consisted of four live modules that offered continuing education (CE) credit and a final graduation module in which participants summarized key concepts they learned and how they were going to take these back to their practice. Enrollment numbers were dictated by the number of available faculty mentors who were willing to take up to two participants per cohort. Modules utilized didactic and active learning strategies with “at home” assignments to reinforce the material presented. Initially 15 hours of CE credit were offered, but in the second iteration of the program the topics were streamlined to offer 12 hours of CE credit due to participant feedback. The format of the program and associated assignments are depicted in Table 1, with each module running four hours in length. The topics and content included in this program were specifically chosen by the founding program coordinators as areas that would meet the needs of individuals interested in precepting and/or academia. For example, “Module 2 Publications” may initially seem more directly aligned with expectations in academia, but it is also an area with which preceptors can support lifelong learning through student/resident involvement in research projects and scholarly activities. Content was presented by college of pharmacy faculty who founded and coordinated the program. Each faculty member had been teaching at the university for at least three years and had participated in a teaching certificate program at a separate institution prior to creation of this program. Additional pharmacy faculty volunteered to serve as mentors for certificate participants in a 1:1 match to provide individualized, effective and specific feedback on all assignments and mentorship throughout the program. All mentors attended a mandatory training session in which the coordinators discussed the overall goals of the program, explained mentor/ mentee roles and expectations, and trained mentors on standardized program feedback forms/rubrics for required assignments. The program was marketed through emails to college preceptors, local residency directors, alumni listservs, posts on electronic platforms (e.g., college Facebook page, LinkedIn), and by word of mouth. Applications were available from June to July each year, and the program faculty met to review all applications in early August. Each applicant packet was evaluated for completeness, individual ability to complete the program successfully, and interest in precepting and/or academia. All applicants were accepted for 2
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
J.D. Cole, et al.
Table 1 Teaching certificate program format. Module 1 Introduction to Teaching
2a Publications
3 Advanced Teaching Principles
4 Precepting
Concepts Philosophy • Teaching Alignment and Mapping • Course and Course Objective • Syllabus Creation and Templates • Rubrics Expectations • Publication Tips for Success • Publication Review Board (IRB) • Institutional and Grant Writing
Objectives and Format • Lecture Delivery Technique • Lecture Learning Techniques • Active Techniques • Assessment to Write Exam Questions and • How Utilize Item Analysis Differences • Generational and Mentoring: • Precepting Expectations, Planning, and Execution
Conversations and • Difficult Effective Feedback and Feedback • Graduation • Presentation of Certificates
5
a
First iteration assignments ❏ Teaching philosophy ❏ Syllabus ❏ Rubric
Complete ONE of the following: ❏ ❏ ❏ ❏ ❏
Create protocol Complete IRB application Complete grant application Create electronic poster Develop manuscript for submission
❏ Live didactic lecture with active learning OR recorded lecture with live active learning ❏ Reflection paper on lecture ❏ Write five multiple choice questions ❏ Evaluate exam question item analysis ❏ Facilitate at least one simulation activity ❏ Reflection paper on simulation activity ❏ Create/modify APPE rotation schedule ❏ Reflection paper on how to engage in difficult conversations None
Second iteration assignments ❏ Teaching philosophy ❏ Syllabus
Complete ONE of the following: ❏ Create protocol and identify type of IRB approval needed ❏ Complete grant application ❏ Create electronic abstract and poster and identify where it should be presented ❏ Complete manuscript for submission ❏ Live didactic lecture with active learning OR recorded lecture with live active learning ❏ Rubric ❏ Write 5 multiple choice questions ❏ Evaluate exam question item analysis ❏ Facilitate at least one simulation activity ❏ Create/modify APPE rotation schedule ❏ Reflection paper on how to engage in difficult conversations None
Moved to Module 2 in the second iteration based on feedback from first participant cohort. APPE: Advanced Pharmacy Practice Experiences.
the first offering of the program, and only one applicant was denied for the second cycle due to the inability to attend required modules. During the selection process, the program faculty reviewed areas of expertise and interest for each applicant and used this information to match them with a volunteer faculty mentor with a similar profile. All applicants were notified of acceptance status and assigned mentor by mid-August to prepare for attendance at Module 1 in early fall. Each cohort was encouraged to email their individual mentor prior to Module 1 in order to begin introductions and review expectations and goals for the program. The impact of content knowledge was measured with written summative assessments provided at the start and end of each module. Module assessments contained three to seven free response or multiple-choice questions for each CE topic covered. Therefore, the pre- and post-module assessments contained 9 to 13 questions to reflect the three topics covered during each module. Questions were created by the faculty who delivered the content and were peer-reviewed by program coordinators prior to implementation. Questions were not validated. All pre- and post-assessments were proctored by program faculty, allowing for up to 15 min for completion. Continuing education credit was not reliant on post-test scoring as the intent was to assess the impact of the program. All data were analyzed using Microsoft Excel®.12 Confidence data was assessed through an online Qualtrics®13 survey sent to participants at the start and the end of the 10-month program. Participants were asked to rate their confidence regarding skills and abilities to perform module objectives via a sliding scale Likert score from 1 to 5, with 1 being “not confident at all” and 5 being “very confident”. The 17-item confidence survey can be found in Appendix 1. All knowledge and confidence data were evaluated utilizing paired t-tests. Means of the pre- and post-assessment scores for each content module were compared utilizing all participant data that was complete. Content modules were delineated as: 1 = Introduction to Teaching, 2 = Publications, 3 = Advanced Teaching Principles and 4 = Precepting. Complete data was defined as a complete pre- and post- assessment for the module being evaluated. An ANOVA was utilized to determine if mean differences between the modules were statistically significantly different. Bonferroni (all-pairwise) multiple comparison tests were utilized to determine what modules demonstrated the most change. This study received exempt status from the host institution's Institutional Review Board (IRB). Results The inaugural teaching certificate program cohort consisted of 14 participants, and the second cohort consisted of 18 participants. Of note, each year there was 1 participant who began the program but withdrew due to a career change and/or personal reasons. Data from those individuals was obtained and included up until the time of their withdrawal. Baseline demographic characteristics for the cohort participants are listed in Table 2. Twenty nine of the 32 participants completed the pre-confidence survey, whereas all 30 participants who graduated the program completed the post-confidence survey. Knowledge assessments for participants who were 3
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
J.D. Cole, et al.
Table 2 Baseline demographic characteristics for program participants. Demographics
First cohort (n = 14) No. (%)
Second cohort (n = 18) No. (%)
10 (71.4)
13 (72.2)
2 (14.3) 8 (57.4) 0 1 (7.1) 1 (7.1) 0 0 2 (14.3)
3 (16.7) 7 (38.9) 4 (22.2) 0 0 1 (5.6) 2 (11.1) 1 (5.6)
0 2 (14.3) 4 (28.6) 6 (57.1) 0 2 (14.3) 0 0
2 (11.1) 1 (5.6) 9 (50) 1 (5.6) 2 (11.1) 1 (5.6) 1 (5.6) 1 (5.6)
3 (21.4) 6 (42.9) 5 (35.7)
8 (44.4) 4 (22.2) 6 (33.3)
3 (21.4) 6 (42.9) 5 (35.7)
2 (11.1) 9 (50) 7 (38.9)
Gender Female Area of practice Community/outpatient Hospital PBM pharmacy Consulting Infusion pharmacy Mail Order pharmacy Residency Fellowship Employment position PGY-1 resident Fellow Staff/clinical Manager/supervisor Clinical coordinator Pharmacy director Chief clinical officer Vice President of clinical services Years since graduation 0–2 years 3–5 years > 5 years Reason for enrollment Academic interest Preceptor interest Both
absent or late to any of the modules were not included as a pre- and post- assessment could not be compared. For each of the four modules, there was a significant difference in pre- and post-knowledge assessments. Knowledge data for each module can be found in Table 3. There was a statistically significant difference between the mean differences (p = 0.005), with the significant change between module 1 and module 2, and module 2 and module 4. Confidence data was significantly increased regarding topics in each module. The average pre- and post-confidence scores for each module can be found in Table 4. Although participants scored an average of 82% on the post-module assessment for Publications (module 2), their confidence of this material was lower than the other modules (3.7 vs. > 4). The mean differences for the confidence data were also significantly different (p = 0.044). This difference was noted to be between the mean differences for module 1 and 4.
Discussion This teaching certificate program incorporated the recommended core content for teaching and learning curricula and common components previously established in programs nationally.6–8,11 However, the format of this program was unique as it included preceptor development, CE credits for participation, one-on-one mentorship, and was not limited to participants enrolled in Table 3 Comparison of pre- and post-assessment of knowledge by module. Knowledge
N
Mean % (SD)
95% CI
p-Value
Introduction to teaching pre-assessment Introduction to teaching post-assessment Mean difference Publications pre-assessment Publications post-assessment Mean difference Advanced teaching principles pre-assessment Advanced teaching principles post-assessment Mean difference Precepting pre-assessment Precepting pre-assessment Mean difference
13 13
64.1 (13.3) 72.2 (15.2) 8.1 (13.1) 60.5 (13.1) 80.7 (12.7) 20.2 (15.1) 42 (13.3) 59.9 (14) 17.9 (6.3) 80 (12.7) 90.9 (7.6) 10.9 (12.7)
56–72.1 63–81.4 0.2–16.1 55.4–65.6 75.8–85.7 14.3–26.1 36.6–47.4 54.3–65.6 15.3–20.5 75–85.1 87.9–93.9 5.9–15.9
0.046
28 28 27 27 27 27
SD: standard deviation; CI: confidence interval. 4
< 0.001
< 0.001
< 0.001
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
J.D. Cole, et al.
Table 4 Confidence differences in module objectives before and after the teaching certificate program. Confidence
N
Mean (SD)
95% CI
p-Value
Introduction to teaching pre-assessment Introduction to teaching post-assessment Mean difference Publications pre-assessment Publications post-assessment Mean difference Advanced teaching principles pre-assessment Advanced teaching principles post-assessment Mean difference Precepting pre-assessment Precepting post-assessment Mean difference
29 30
2.6 4.6 2.1 2.1 3.7 1.6 2.7 4.4 1.7 3.2 4.5 1.4
2.3–2.8 4.5–4.8 1.7–2.4 1.8–2.4 3.4–4 1.2–2 2.5–2.9 4.2–4.6 1.4–1.9 2.8–3.5 1.4–4.7 1–1.7
< 0.001
29 30 29 30 29 30
(0.7) (0.4) (0.8) (0.8) (0.9) (1) (0.7) (0.5) (0.8) (0.9) (0.4) (0.9)
< 0.001
< 0.001
< 0.001
SD: standard deviation; CI: confidence interval.
postgraduate training. Data collected from this study offer insight to knowledge obtained from a teaching certificate program in addition to perceived confidence, which has not previously been reported in the literature.6,7 The results obtained suggest that this university-affiliated teaching certificate program improved both knowledge and perceived confidence for participants. Knowledge data obtained from pre- and post-module assessments showed statistically significant improvement in participant scores for all modules. The Publications (module 2) and Advanced Teaching Principles (module 3) assessment data revealed the largest improvement from pre- to post-module results. This difference may be attributed to lower baseline knowledge in those areas, as the participant pre-scores appear lower than the other modules. Post-scores for all modules were similar except for the Precepting module (module 4) in which participants scored the highest. This may be due to the fact that the content was directly relatable to current practice. There was also a significant improvement in participant confidence reported for all four modules. The current study compared participant perceptions from baseline to the end of the 10-month program, when they had attended all modules and completed all assignments. The largest average difference in participant confidence was reported for Introduction to Teaching (module 1), followed closely by Advanced Teaching Principles (module 3). These results are not surprising as many participants did not have extensive education in or baseline exposure to teaching and learning in a pharmacy didactic curriculum. It was noted during the program that the participants may not have been exposed to or felt confident in executing these activities without proper training. Post-module outcomes for knowledge assessments and confidence for Publications (module 2) show an unanticipated disparity as participants scored high for their assessment but ranked their confidence lower than other modules. The nature of the assignments and/or the inability to complete the publication assignments in the required time frame may have contributed to the lower confidence. The higher pre-module scores for both knowledge and confidence on Precepting content (module 4) were not surprising as many participants were already serving in this role at their practice sites. The confidence is likely higher to begin with as compared to the other modules because of their experience, but it is important to note that a significant increase in confidence was still observed based on material presented in the module. Both Introduction to Teaching (module 1) and Advanced Teaching Principles (module 3) had post-assessment knowledge scores that fell below 80%. This may be because the content learned in these modules remains abstract until the participants are able to apply the skills in the classroom or through precepting. By nature, this material requires further learning through distributed practice and application. One way this may be addressed would be to evaluate cumulative final assessments in addition to the pre- and postassessments already utilized. This may provide insight regarding the impact assignments have on overall knowledge and longitudinal retention of information. After the first offering of this program, the coordinators reflected on logistical challenges and participant feedback to determine what changes would provide enhancement for the future. Challenges identified included time to implement and complete research requirements, overlapping content between topics presented, and time provided for assignment completion between modules. The coordinators took this feedback and made changes to the order of module content, assignment deadlines, and streamlined content. After the second program cohort feedback, the coordinators identified additional areas for continued program improvement. One main concern has been the time required to attend live modules on Saturdays throughout the year. As a result, the subsequent cohort will complete a live-online hybrid program that contains two live modules, one to two online modules provided through a free learning management system (CANVAS), and one live graduation module. Another concern has been a lack of a central program platform as content information was disseminated by program faculty through Google Docs while individual assignments were submitted to faculty mentors. To address this concern, CANVAS will be utilized for the program's overall execution and assignment submission due to its ability to organize the information in one central location while providing a user-friendly platform for both participants and faculty mentors. Future opportunities for improvement may also be found in alignment with teaching certificate program recommendations provided by ACCP and a joint recommendation by the AACP and ASHP.4,5 Although these recommendations are focused on teaching opportunities provided to pharmacy residents, it does seem feasible that they could be extrapolated to additional practicing pharmacists and therefore would be relevant for enhancing this program. Upon review, program faculty feel that the current program 5
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
J.D. Cole, et al.
already includes the main requirements proposed for the ACCP Basic Teaching Experiences and a large portion of those for the Expanded Teaching and Learning Curriculum, including the creation of a teaching philosophy, involvement in pedagogical discussions, and participation in teaching experiences.4 However, there are some higher level teaching experiences, such as course development and academic committee work, that are not met with this program. There may, therefore, be an opportunity to create a second-level teaching certificate program that more closely focuses on this level of academic application for those individuals interested in pursuing this course of training. When reviewing the 12 recommendations for best practices for teaching and learning curricula provided by AACP/ASHP, the program faculty feel that these items are also already being addressed with the current program. These areas include alignment with a college of pharmacy, clear marketing of program content and requirements, identification of core program content delivered in varied teaching styles, education tailored to participant's level of knowledge of subject matter, opportunities for application in various educational modalities, evaluation by mentors at regular intervals, development of a teaching portfolio, and faculty use of assessment for continued program enhancement. Since there is no current standardized tool for external validation of the program, the program is currently unable to meet this final recommendation. Conclusion Implementation and execution of a university-affiliated pharmacy teaching certificate program for practicing pharmacists improved both participant knowledge and perceived confidence. Although the lowest knowledge assessment scores were observed in the Introduction to Teaching (module 1) and Advanced Teaching Principles (module 3), these same modules also had the largest average difference in participant confidence. This association may be due to the fact that the abstract content delivered requires continuous application, but participants still had improved confidence as a result of limited baseline exposure to teaching and learning in a pharmacy didactic curriculum. Future research is needed to determine the longitudinal benefits of a pharmacy teaching certificate program in regard to knowledge retention and practical application of content. Declaration of Competing Interest An internal grant was provided by the Dean of the college for the first two offerings of this program. Appendix 1. Confidence survey On a Scale of 1 to 5, with 1 being Not Confident at All and 5 being Very Confident, please answer how confident you feel in the following areas: Developing a teaching philosophy Developing a course syllabus Developing appropriate course objectives Developing a well-designed grading rubric Developing lecture objectives Developing active learning activities Developing lecture materials Designing effective assessments to meet objectives Preparing appropriate exam/CE questions Analyzing exam questions Recognizing generational differences Precepting students effectively Providing feedback in difficult situations Successfully completing an IRB application Successfully completing a grant application Developing a poster for presentation at a professional meeting Writing a manuscript for submission to a professional journal References 1. Haines SL, Dy-Borman EA, Clifford KM, et al. Methods used by colleges and schools of pharmacy to prepare student pharmacists for careers in academia. Am J Pharm Educ. 2017;81(1):6. https://doi.org/10.5688/ajpe8116. 2. Assemi M, Hudmon KS, Sowinski KM, Corelli RL. Educational background and academic rank of faculty members within US schools pharmacy. Am J Pharm Educ. 2016;80(4) https://doi.org/10.5688/ajpe80461. 3. ASHP Accreditation Standard for Postgraduate Year One (PGY1) Pharmacy Residency Programs. American Society of Health-System Pharmacists (ASHP). https:// www.ashp.org/-/media/assets/professional-development/residencies/docs/pgy1-accreditation-standard-2016.ashx?la=en&hash= 82D0575273AD83E720B114D62B7926FD35792AFD. Updated 2016. Accessed 6 August 2019. 4. Engle JP, Franks AM, Ashijan E, et al. A self-assessment guide for resident teaching experiences. Pharmacotherapy. 2016 Jun;36(6):e58–e79. https://doi.org/10. 1002/phar.1768. 5. Wright EA, Brown B, Gettig J, et al. Teaching and learning curriculum programs: recommendations for postgraduate pharmacy experiences in education. Am J Health Syst Pharm. 2014;71(15):1292–1302.
6
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
J.D. Cole, et al.
6. Falter RA, Arrendale JR. Benefits of a teaching certificate program for pharmacy residents. Am J Health Syst Pharm. 2009;66(21):1905–1906. 7. Strang AF, Baia P. An investigation of teaching and learning programs in pharmacy education. Am J Pharm Educ. 2016;80(4):59. https://doi.org/10.5688/ ajpe80459. 8. Gonzalvo JD, Ramsey DC, Sheehan AH, Sprunger TL. Redesign of a statewide teaching certificate program for pharmacy residents. Am J Pharm Educ. 2013;77(4) https://doi.org/10.5688/ajpe44579. 9. Wahl KR, Margolis A, Lintner K, Hartkopf K, Martin B. Impact and application of material learned in a pharmacy residency teaching certificate program. Am J Pharm Educ. 2014;78(6) https://doi.org/10.5688/ajpe786123. 10. McNatty D, Cox CD, Seifert CF. Assessment of teaching experiences completed during accredited pharmacy residency programs. Am J Pharm Educ. 2007;71(5) https://doi.org/10.5688/aj710588. 11. Sheehan AH, Gonzalvo JD, Ramsey DC, Sprunger TL. Teaching certificate program participants' perceptions of mentor-mentee relationships. Am J Pharm Educ. 2016;80(3) https://doi.org/10.5688/ajpe80342. 12. Microsoft Excel (for windows) [computer program]. Version 13.0. Redmond, WA: Microsoft; 2013. 13. Qualtrics™ Software. Version 2015–2016 and 2016–2017. Provo, UT, USA 2019; 2019.
7
Contents lists available at ScienceDirect
Currents in Pharmacy Teaching and Learning journal homepage: www.elsevier.com/locate/cptl
Research Note
Effectiveness of a pharmacy teaching certificate program offered to practicing pharmacists ⁎
Jaclyn D. Cole, Melissa J. Ruble , Gwendolyn Wantuch, Kamila A. Dell, Erini Serag-Bolos, Carol Fox, Janelle Perkins Pharmacotherapeutics and Clinical Research Department, University of South Florida College of Pharmacy, United States
A R T IC LE I N F O
ABS TRA CT
Keywords: Teaching certificate program Preceptor development Academia Pharmacists Knowledge
Introduction: Regardless of the practice setting, pharmacists' roles often expand beyond patientcentered services and into health-related educational roles for patients, healthcare providers, and student pharmacists. Teaching roles of a clinical pharmacist can include providing invited presentations (such as continuing education lectures), teaching colleagues about new clinical initiatives, precepting students during advanced pharmacy practice experiences (APPEs) and introductory pharmacy practice experiences (IPPEs), and precepting residents on rotations. This program was designed to evaluate the effect of a university-affiliated pharmacy teaching certificate program on participant knowledge and perceived confidence in specific areas of academic pharmacy and preceptor development. Methods: A 10-month program consisting of four live modules with associated continuing education (CE) credit was offered to pharmacy residents and practicing pharmacists in the surrounding area. The program format utilized didactic and active learning strategies with “at home” assignments to reinforce material presented. Content knowledge was measured with written summative assessments provided at the start and end of each module. Perceived confidence was assessed through online surveys provided at the start and end of the program. All results were evaluated using paired t-tests. Results: Data were obtained from 14 participants in the first cohort and 18 participants in the second cohort. For each of the four modules, there was a significant difference in pre- and postknowledge assessments. Confidence data was also significantly increased regarding topics in each module. Conclusion: Knowledge assessments and perception surveys suggest that this university-affiliated teaching certificate program improved both knowledge and perceived confidence for program participants.
Introduction Pharmacy school curricula emphasize appropriate strategies for patient counseling and education, but an emphasis on teaching approaches and mentoring students from an academic perspective is lacking. Recent survey results from United States colleges and schools of pharmacy suggest that academic-focused advanced pharmacy practice experiences (APPEs) are the most common training
⁎
Corresponding author. E-mail addresses: [email protected] (J.D. Cole), [email protected] (M.J. Ruble), [email protected] (G. Wantuch), [email protected] (K.A. Dell), [email protected] (E. Serag-Bolos), [email protected] (C. Fox), [email protected] (J. Perkins). https://doi.org/10.1016/j.cptl.2019.07.002
1877-1297/ © 2019 Elsevier Inc. All rights reserved.
Please cite this article as: Jaclyn D. Cole, et al., Currents in Pharmacy Teaching and Learning, https://doi.org/10.1016/j.cptl.2019.07.002
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
J.D. Cole, et al.
method offered to cover these aspects, but less than half of the programs offer didactic coursework or study opportunities for those interested in an academic career.1 A shortage of experienced faculty members at newer colleges and schools of pharmacy, as well as a shortage of trained clinical pharmacists who are obliged to take on these expanded teaching roles, further emphasize the need for more structured programs to mentor potential academicians and preceptors.2 Postgraduate year one (PGY1) and postgraduate year two (PGY2) pharmacy residency programs may offer teaching certificate program opportunities in conjunction with colleges of pharmacy. Although this is not specifically required by the American Society of Health-System Pharmacists (ASHP) for accreditation, there are standards related to residency program competency in “teaching, education, and dissemination of knowledge” that remain unclear and lack standardization.3 The American College of Clinical Pharmacy (ACCP) Task Force on Residencies has developed two guidelines that clarify roles and expectations of teaching certificate programs within residencies.4 The American Association of Colleges of Pharmacy (AACP) and ASHP also teamed up to guide the development of teaching certificate programs within postgraduate education.5 Both guides emphasize a greater focus on a teaching and learning curricula, encourage colleges and schools of pharmacy to provide these programs, and provide suggestions for curriculum content and assessment. A literature review of Medline, Ovid, and Embase using “teaching certificate programs and pharmacy” as search terms was conducted to identify common components of existing successful programs and to determine the type of programmatic outcomes evaluated. Guidelines mentioned above were also reviewed for specific content and best practices in developing such programs. Common components that previous teaching certificate programs have utilized include educational presentations, lecture objectives, small group discussion facilitation, and student progress evaluation.6 The type of outcomes evaluated through previous studies included participant satisfaction, perceptions of improvement, and perceived value of program content.7,8 Longitudinal studies have been conducted and suggest that implementation of teaching certificate programs increase participant confidence and interest in academia and precepting.9,10 Wahl and colleagues9 surveyed graduates of a teaching certificate program over twelve years and found that delivery of such programs may encourage participants to pursue academia as a career. These results also reinforced previous studies that have shown increased confidence in teaching and precepting students. In addition, McNatty and colleagues10 surveyed residency graduates from ASHP-accredited programs from 2003 through 2006 and found that those who had received education in teaching methods during their postgraduate education were more likely to aspire to academic positions. Finally, literature has also noted the importance of professional mentorship for participants completing such a program, with emphasis placed on pharmacy faculty to provide this guidance for potential future pharmacy educators.11 Although several articles discuss teaching certificate programs, none were found that included pharmacists outside of postgraduate training programs or assessed participant knowledge and confidence. There was also limited information regarding the use of mentors as an integral part of such programs. Since precepting and clinical education roles contain components of teaching, evaluation, and assessment, the authors of this study decided to create and offer such a program to not only pharmacy residents but practicing pharmacists as well. This program was different from many others as it included one-on-one mentoring for all participants which allowed for a more personalized training plan. The objective of this study was to evaluate the impact of a university-affiliated teaching certificate program on both participant knowledge and perceived confidence in specific areas of academic pharmacy and preceptor development. Methods A cohort study of participants enrolled in a teaching certificate program developed for practicing pharmacists in the surrounding area of the host university was undertaken. The intent of the longitudinal program was to offer preceptor development with a focus on academic aspects of teaching pedagogy, assessment, and evaluation. This 10-month program consisted of four live modules that offered continuing education (CE) credit and a final graduation module in which participants summarized key concepts they learned and how they were going to take these back to their practice. Enrollment numbers were dictated by the number of available faculty mentors who were willing to take up to two participants per cohort. Modules utilized didactic and active learning strategies with “at home” assignments to reinforce the material presented. Initially 15 hours of CE credit were offered, but in the second iteration of the program the topics were streamlined to offer 12 hours of CE credit due to participant feedback. The format of the program and associated assignments are depicted in Table 1, with each module running four hours in length. The topics and content included in this program were specifically chosen by the founding program coordinators as areas that would meet the needs of individuals interested in precepting and/or academia. For example, “Module 2 Publications” may initially seem more directly aligned with expectations in academia, but it is also an area with which preceptors can support lifelong learning through student/resident involvement in research projects and scholarly activities. Content was presented by college of pharmacy faculty who founded and coordinated the program. Each faculty member had been teaching at the university for at least three years and had participated in a teaching certificate program at a separate institution prior to creation of this program. Additional pharmacy faculty volunteered to serve as mentors for certificate participants in a 1:1 match to provide individualized, effective and specific feedback on all assignments and mentorship throughout the program. All mentors attended a mandatory training session in which the coordinators discussed the overall goals of the program, explained mentor/ mentee roles and expectations, and trained mentors on standardized program feedback forms/rubrics for required assignments. The program was marketed through emails to college preceptors, local residency directors, alumni listservs, posts on electronic platforms (e.g., college Facebook page, LinkedIn), and by word of mouth. Applications were available from June to July each year, and the program faculty met to review all applications in early August. Each applicant packet was evaluated for completeness, individual ability to complete the program successfully, and interest in precepting and/or academia. All applicants were accepted for 2
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
J.D. Cole, et al.
Table 1 Teaching certificate program format. Module 1 Introduction to Teaching
2a Publications
3 Advanced Teaching Principles
4 Precepting
Concepts Philosophy • Teaching Alignment and Mapping • Course and Course Objective • Syllabus Creation and Templates • Rubrics Expectations • Publication Tips for Success • Publication Review Board (IRB) • Institutional and Grant Writing
Objectives and Format • Lecture Delivery Technique • Lecture Learning Techniques • Active Techniques • Assessment to Write Exam Questions and • How Utilize Item Analysis Differences • Generational and Mentoring: • Precepting Expectations, Planning, and Execution
Conversations and • Difficult Effective Feedback and Feedback • Graduation • Presentation of Certificates
5
a
First iteration assignments ❏ Teaching philosophy ❏ Syllabus ❏ Rubric
Complete ONE of the following: ❏ ❏ ❏ ❏ ❏
Create protocol Complete IRB application Complete grant application Create electronic poster Develop manuscript for submission
❏ Live didactic lecture with active learning OR recorded lecture with live active learning ❏ Reflection paper on lecture ❏ Write five multiple choice questions ❏ Evaluate exam question item analysis ❏ Facilitate at least one simulation activity ❏ Reflection paper on simulation activity ❏ Create/modify APPE rotation schedule ❏ Reflection paper on how to engage in difficult conversations None
Second iteration assignments ❏ Teaching philosophy ❏ Syllabus
Complete ONE of the following: ❏ Create protocol and identify type of IRB approval needed ❏ Complete grant application ❏ Create electronic abstract and poster and identify where it should be presented ❏ Complete manuscript for submission ❏ Live didactic lecture with active learning OR recorded lecture with live active learning ❏ Rubric ❏ Write 5 multiple choice questions ❏ Evaluate exam question item analysis ❏ Facilitate at least one simulation activity ❏ Create/modify APPE rotation schedule ❏ Reflection paper on how to engage in difficult conversations None
Moved to Module 2 in the second iteration based on feedback from first participant cohort. APPE: Advanced Pharmacy Practice Experiences.
the first offering of the program, and only one applicant was denied for the second cycle due to the inability to attend required modules. During the selection process, the program faculty reviewed areas of expertise and interest for each applicant and used this information to match them with a volunteer faculty mentor with a similar profile. All applicants were notified of acceptance status and assigned mentor by mid-August to prepare for attendance at Module 1 in early fall. Each cohort was encouraged to email their individual mentor prior to Module 1 in order to begin introductions and review expectations and goals for the program. The impact of content knowledge was measured with written summative assessments provided at the start and end of each module. Module assessments contained three to seven free response or multiple-choice questions for each CE topic covered. Therefore, the pre- and post-module assessments contained 9 to 13 questions to reflect the three topics covered during each module. Questions were created by the faculty who delivered the content and were peer-reviewed by program coordinators prior to implementation. Questions were not validated. All pre- and post-assessments were proctored by program faculty, allowing for up to 15 min for completion. Continuing education credit was not reliant on post-test scoring as the intent was to assess the impact of the program. All data were analyzed using Microsoft Excel®.12 Confidence data was assessed through an online Qualtrics®13 survey sent to participants at the start and the end of the 10-month program. Participants were asked to rate their confidence regarding skills and abilities to perform module objectives via a sliding scale Likert score from 1 to 5, with 1 being “not confident at all” and 5 being “very confident”. The 17-item confidence survey can be found in Appendix 1. All knowledge and confidence data were evaluated utilizing paired t-tests. Means of the pre- and post-assessment scores for each content module were compared utilizing all participant data that was complete. Content modules were delineated as: 1 = Introduction to Teaching, 2 = Publications, 3 = Advanced Teaching Principles and 4 = Precepting. Complete data was defined as a complete pre- and post- assessment for the module being evaluated. An ANOVA was utilized to determine if mean differences between the modules were statistically significantly different. Bonferroni (all-pairwise) multiple comparison tests were utilized to determine what modules demonstrated the most change. This study received exempt status from the host institution's Institutional Review Board (IRB). Results The inaugural teaching certificate program cohort consisted of 14 participants, and the second cohort consisted of 18 participants. Of note, each year there was 1 participant who began the program but withdrew due to a career change and/or personal reasons. Data from those individuals was obtained and included up until the time of their withdrawal. Baseline demographic characteristics for the cohort participants are listed in Table 2. Twenty nine of the 32 participants completed the pre-confidence survey, whereas all 30 participants who graduated the program completed the post-confidence survey. Knowledge assessments for participants who were 3
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
J.D. Cole, et al.
Table 2 Baseline demographic characteristics for program participants. Demographics
First cohort (n = 14) No. (%)
Second cohort (n = 18) No. (%)
10 (71.4)
13 (72.2)
2 (14.3) 8 (57.4) 0 1 (7.1) 1 (7.1) 0 0 2 (14.3)
3 (16.7) 7 (38.9) 4 (22.2) 0 0 1 (5.6) 2 (11.1) 1 (5.6)
0 2 (14.3) 4 (28.6) 6 (57.1) 0 2 (14.3) 0 0
2 (11.1) 1 (5.6) 9 (50) 1 (5.6) 2 (11.1) 1 (5.6) 1 (5.6) 1 (5.6)
3 (21.4) 6 (42.9) 5 (35.7)
8 (44.4) 4 (22.2) 6 (33.3)
3 (21.4) 6 (42.9) 5 (35.7)
2 (11.1) 9 (50) 7 (38.9)
Gender Female Area of practice Community/outpatient Hospital PBM pharmacy Consulting Infusion pharmacy Mail Order pharmacy Residency Fellowship Employment position PGY-1 resident Fellow Staff/clinical Manager/supervisor Clinical coordinator Pharmacy director Chief clinical officer Vice President of clinical services Years since graduation 0–2 years 3–5 years > 5 years Reason for enrollment Academic interest Preceptor interest Both
absent or late to any of the modules were not included as a pre- and post- assessment could not be compared. For each of the four modules, there was a significant difference in pre- and post-knowledge assessments. Knowledge data for each module can be found in Table 3. There was a statistically significant difference between the mean differences (p = 0.005), with the significant change between module 1 and module 2, and module 2 and module 4. Confidence data was significantly increased regarding topics in each module. The average pre- and post-confidence scores for each module can be found in Table 4. Although participants scored an average of 82% on the post-module assessment for Publications (module 2), their confidence of this material was lower than the other modules (3.7 vs. > 4). The mean differences for the confidence data were also significantly different (p = 0.044). This difference was noted to be between the mean differences for module 1 and 4.
Discussion This teaching certificate program incorporated the recommended core content for teaching and learning curricula and common components previously established in programs nationally.6–8,11 However, the format of this program was unique as it included preceptor development, CE credits for participation, one-on-one mentorship, and was not limited to participants enrolled in Table 3 Comparison of pre- and post-assessment of knowledge by module. Knowledge
N
Mean % (SD)
95% CI
p-Value
Introduction to teaching pre-assessment Introduction to teaching post-assessment Mean difference Publications pre-assessment Publications post-assessment Mean difference Advanced teaching principles pre-assessment Advanced teaching principles post-assessment Mean difference Precepting pre-assessment Precepting pre-assessment Mean difference
13 13
64.1 (13.3) 72.2 (15.2) 8.1 (13.1) 60.5 (13.1) 80.7 (12.7) 20.2 (15.1) 42 (13.3) 59.9 (14) 17.9 (6.3) 80 (12.7) 90.9 (7.6) 10.9 (12.7)
56–72.1 63–81.4 0.2–16.1 55.4–65.6 75.8–85.7 14.3–26.1 36.6–47.4 54.3–65.6 15.3–20.5 75–85.1 87.9–93.9 5.9–15.9
0.046
28 28 27 27 27 27
SD: standard deviation; CI: confidence interval. 4
< 0.001
< 0.001
< 0.001
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
J.D. Cole, et al.
Table 4 Confidence differences in module objectives before and after the teaching certificate program. Confidence
N
Mean (SD)
95% CI
p-Value
Introduction to teaching pre-assessment Introduction to teaching post-assessment Mean difference Publications pre-assessment Publications post-assessment Mean difference Advanced teaching principles pre-assessment Advanced teaching principles post-assessment Mean difference Precepting pre-assessment Precepting post-assessment Mean difference
29 30
2.6 4.6 2.1 2.1 3.7 1.6 2.7 4.4 1.7 3.2 4.5 1.4
2.3–2.8 4.5–4.8 1.7–2.4 1.8–2.4 3.4–4 1.2–2 2.5–2.9 4.2–4.6 1.4–1.9 2.8–3.5 1.4–4.7 1–1.7
< 0.001
29 30 29 30 29 30
(0.7) (0.4) (0.8) (0.8) (0.9) (1) (0.7) (0.5) (0.8) (0.9) (0.4) (0.9)
< 0.001
< 0.001
< 0.001
SD: standard deviation; CI: confidence interval.
postgraduate training. Data collected from this study offer insight to knowledge obtained from a teaching certificate program in addition to perceived confidence, which has not previously been reported in the literature.6,7 The results obtained suggest that this university-affiliated teaching certificate program improved both knowledge and perceived confidence for participants. Knowledge data obtained from pre- and post-module assessments showed statistically significant improvement in participant scores for all modules. The Publications (module 2) and Advanced Teaching Principles (module 3) assessment data revealed the largest improvement from pre- to post-module results. This difference may be attributed to lower baseline knowledge in those areas, as the participant pre-scores appear lower than the other modules. Post-scores for all modules were similar except for the Precepting module (module 4) in which participants scored the highest. This may be due to the fact that the content was directly relatable to current practice. There was also a significant improvement in participant confidence reported for all four modules. The current study compared participant perceptions from baseline to the end of the 10-month program, when they had attended all modules and completed all assignments. The largest average difference in participant confidence was reported for Introduction to Teaching (module 1), followed closely by Advanced Teaching Principles (module 3). These results are not surprising as many participants did not have extensive education in or baseline exposure to teaching and learning in a pharmacy didactic curriculum. It was noted during the program that the participants may not have been exposed to or felt confident in executing these activities without proper training. Post-module outcomes for knowledge assessments and confidence for Publications (module 2) show an unanticipated disparity as participants scored high for their assessment but ranked their confidence lower than other modules. The nature of the assignments and/or the inability to complete the publication assignments in the required time frame may have contributed to the lower confidence. The higher pre-module scores for both knowledge and confidence on Precepting content (module 4) were not surprising as many participants were already serving in this role at their practice sites. The confidence is likely higher to begin with as compared to the other modules because of their experience, but it is important to note that a significant increase in confidence was still observed based on material presented in the module. Both Introduction to Teaching (module 1) and Advanced Teaching Principles (module 3) had post-assessment knowledge scores that fell below 80%. This may be because the content learned in these modules remains abstract until the participants are able to apply the skills in the classroom or through precepting. By nature, this material requires further learning through distributed practice and application. One way this may be addressed would be to evaluate cumulative final assessments in addition to the pre- and postassessments already utilized. This may provide insight regarding the impact assignments have on overall knowledge and longitudinal retention of information. After the first offering of this program, the coordinators reflected on logistical challenges and participant feedback to determine what changes would provide enhancement for the future. Challenges identified included time to implement and complete research requirements, overlapping content between topics presented, and time provided for assignment completion between modules. The coordinators took this feedback and made changes to the order of module content, assignment deadlines, and streamlined content. After the second program cohort feedback, the coordinators identified additional areas for continued program improvement. One main concern has been the time required to attend live modules on Saturdays throughout the year. As a result, the subsequent cohort will complete a live-online hybrid program that contains two live modules, one to two online modules provided through a free learning management system (CANVAS), and one live graduation module. Another concern has been a lack of a central program platform as content information was disseminated by program faculty through Google Docs while individual assignments were submitted to faculty mentors. To address this concern, CANVAS will be utilized for the program's overall execution and assignment submission due to its ability to organize the information in one central location while providing a user-friendly platform for both participants and faculty mentors. Future opportunities for improvement may also be found in alignment with teaching certificate program recommendations provided by ACCP and a joint recommendation by the AACP and ASHP.4,5 Although these recommendations are focused on teaching opportunities provided to pharmacy residents, it does seem feasible that they could be extrapolated to additional practicing pharmacists and therefore would be relevant for enhancing this program. Upon review, program faculty feel that the current program 5
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
J.D. Cole, et al.
already includes the main requirements proposed for the ACCP Basic Teaching Experiences and a large portion of those for the Expanded Teaching and Learning Curriculum, including the creation of a teaching philosophy, involvement in pedagogical discussions, and participation in teaching experiences.4 However, there are some higher level teaching experiences, such as course development and academic committee work, that are not met with this program. There may, therefore, be an opportunity to create a second-level teaching certificate program that more closely focuses on this level of academic application for those individuals interested in pursuing this course of training. When reviewing the 12 recommendations for best practices for teaching and learning curricula provided by AACP/ASHP, the program faculty feel that these items are also already being addressed with the current program. These areas include alignment with a college of pharmacy, clear marketing of program content and requirements, identification of core program content delivered in varied teaching styles, education tailored to participant's level of knowledge of subject matter, opportunities for application in various educational modalities, evaluation by mentors at regular intervals, development of a teaching portfolio, and faculty use of assessment for continued program enhancement. Since there is no current standardized tool for external validation of the program, the program is currently unable to meet this final recommendation. Conclusion Implementation and execution of a university-affiliated pharmacy teaching certificate program for practicing pharmacists improved both participant knowledge and perceived confidence. Although the lowest knowledge assessment scores were observed in the Introduction to Teaching (module 1) and Advanced Teaching Principles (module 3), these same modules also had the largest average difference in participant confidence. This association may be due to the fact that the abstract content delivered requires continuous application, but participants still had improved confidence as a result of limited baseline exposure to teaching and learning in a pharmacy didactic curriculum. Future research is needed to determine the longitudinal benefits of a pharmacy teaching certificate program in regard to knowledge retention and practical application of content. Declaration of Competing Interest An internal grant was provided by the Dean of the college for the first two offerings of this program. Appendix 1. Confidence survey On a Scale of 1 to 5, with 1 being Not Confident at All and 5 being Very Confident, please answer how confident you feel in the following areas: Developing a teaching philosophy Developing a course syllabus Developing appropriate course objectives Developing a well-designed grading rubric Developing lecture objectives Developing active learning activities Developing lecture materials Designing effective assessments to meet objectives Preparing appropriate exam/CE questions Analyzing exam questions Recognizing generational differences Precepting students effectively Providing feedback in difficult situations Successfully completing an IRB application Successfully completing a grant application Developing a poster for presentation at a professional meeting Writing a manuscript for submission to a professional journal References 1. Haines SL, Dy-Borman EA, Clifford KM, et al. Methods used by colleges and schools of pharmacy to prepare student pharmacists for careers in academia. Am J Pharm Educ. 2017;81(1):6. https://doi.org/10.5688/ajpe8116. 2. Assemi M, Hudmon KS, Sowinski KM, Corelli RL. Educational background and academic rank of faculty members within US schools pharmacy. Am J Pharm Educ. 2016;80(4) https://doi.org/10.5688/ajpe80461. 3. ASHP Accreditation Standard for Postgraduate Year One (PGY1) Pharmacy Residency Programs. American Society of Health-System Pharmacists (ASHP). https:// www.ashp.org/-/media/assets/professional-development/residencies/docs/pgy1-accreditation-standard-2016.ashx?la=en&hash= 82D0575273AD83E720B114D62B7926FD35792AFD. Updated 2016. Accessed 6 August 2019. 4. Engle JP, Franks AM, Ashijan E, et al. A self-assessment guide for resident teaching experiences. Pharmacotherapy. 2016 Jun;36(6):e58–e79. https://doi.org/10. 1002/phar.1768. 5. Wright EA, Brown B, Gettig J, et al. Teaching and learning curriculum programs: recommendations for postgraduate pharmacy experiences in education. Am J Health Syst Pharm. 2014;71(15):1292–1302.
6
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
J.D. Cole, et al.
6. Falter RA, Arrendale JR. Benefits of a teaching certificate program for pharmacy residents. Am J Health Syst Pharm. 2009;66(21):1905–1906. 7. Strang AF, Baia P. An investigation of teaching and learning programs in pharmacy education. Am J Pharm Educ. 2016;80(4):59. https://doi.org/10.5688/ ajpe80459. 8. Gonzalvo JD, Ramsey DC, Sheehan AH, Sprunger TL. Redesign of a statewide teaching certificate program for pharmacy residents. Am J Pharm Educ. 2013;77(4) https://doi.org/10.5688/ajpe44579. 9. Wahl KR, Margolis A, Lintner K, Hartkopf K, Martin B. Impact and application of material learned in a pharmacy residency teaching certificate program. Am J Pharm Educ. 2014;78(6) https://doi.org/10.5688/ajpe786123. 10. McNatty D, Cox CD, Seifert CF. Assessment of teaching experiences completed during accredited pharmacy residency programs. Am J Pharm Educ. 2007;71(5) https://doi.org/10.5688/aj710588. 11. Sheehan AH, Gonzalvo JD, Ramsey DC, Sprunger TL. Teaching certificate program participants' perceptions of mentor-mentee relationships. Am J Pharm Educ. 2016;80(3) https://doi.org/10.5688/ajpe80342. 12. Microsoft Excel (for windows) [computer program]. Version 13.0. Redmond, WA: Microsoft; 2013. 13. Qualtrics™ Software. Version 2015–2016 and 2016–2017. Provo, UT, USA 2019; 2019.
7