- Email: [email protected]
Impact of a teaching objective structured clinical examination (TOSCE) on student confidence in a pharmacy skills laboratory
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Contents lists available at ScienceDirect
Currents in Pharmacy Teaching and Learning journal homepage: www.elsevier.com/locate/cptl
Research Note
Impact of a teaching objective structured clinical examination (TOSCE) on student confidence in a pharmacy skills laboratory Bin Denga, Norman E Fenn IIIb, Kimberly S. Plakec, a b c
⁎
California Northstate University College of Pharmacy, 9700 West Taron Drive, Elk Grove, CA 95757, United States The University of Texas at Tyler, Department of Clinical Sciences, 3900 University Blvd, Tyler, TX 75799, United States Purdue University College of Pharmacy, Department of Pharmacy Practice, 575 Stadium Mall Drive, West Lafayette, IN 47907, United States
A R T IC LE I N F O
ABS TRA CT
Keywords: Teaching objective structured clinical examination Pharmacy skills laboratory Student confidence Interpersonal communication skills Clinical skills delivery Medication-related problem resolution
Objective: To describe the relationship between the use of a teaching objective structured clinical examination (TOSCE) and student confidence in a pharmacy skills laboratory. Methods: To develop students’ abilities in providing recommendations and patient education on selected drug products, third-year pharmacy students participated in a TOSCE in a men's and women's health laboratory. A 19-item, 5-point Likert scale survey was administered longitudinally to assess student confidence (1 = not all confident to 5 = extremely confident). An additional seven items (1 = strongly disagree to 5 = strongly agree) were added to the original 19-item survey to measure changes in confidence and attitude before and after a problem-based assessment. Results: Seventy-two students completed all five surveys administered throughout the semester. Students’ confidence improved in multiple areas within the clinical skills and interpersonal communication skills categories. Students also indicated their confidence in using primary literature, electronic drug references, and package inserts to answer medication-related questions improved after the TOSCE implementation. Students agreed that the feedback they received from the TOSCE was beneficial to their performance on a problem-based assessment. Conclusions: A TOSCE is a formative assessment that has the potential to improve student confidence in core competency areas immediately after an activity. However, its benefit may be short-term when a time gap exists. Further study is needed to assess long-term sustainability in student confidence over multiple laboratory activities throughout a semester.
Introduction An objective structured clinical examination (OSCE) is an assessment method used to evaluate learners’ abilities to perform various clinical and technical tasks.1 In an OSCE, learners rotate through a series of timed interactive and non-interactive stations.2 In an interactive station, a learner performs a task with a standardized patient or an individual who receives specific training on portraying a patient with a specific medical condition. The learner's interaction is observed and assessed by a facilitator who is either a faculty member, pharmacy resident, or a pharmacy fellow using a standardized assessment called an analytical checklist.2 At a noninteractive station, the learner provides a written response to a task or problem without direct observation from a faculty member.2 Confidence is essential in ensuring students have the ability and willingness to effectively perform clinical tasks in experiential
⁎
Corresponding author. E-mail addresses: [email protected] (B. Deng), [email protected] (N.E. Fenn), [email protected] (K.S. Plake).
https://doi.org/10.1016/j.cptl.2018.11.009
1877-1297/ © 2018 Elsevier Inc. All rights reserved.
Please cite this article as: Deng, B., Currents in Pharmacy Teaching and Learning, https://doi.org/10.1016/j.cptl.2018.11.009
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
B. Deng et al.
practice settings. Social cognitive theory defines the role of confidence as belief in one's own ability, and this can be developed by creating self-directed lifelong learners as required by the Accreditation Council for Pharmacy Education Accreditation Standards.3–5 Students who have self-confidence perform better and have a positive attitude when solving complex problems.5 In a study conducted by Bandura,5 school children were given difficult mathematical questions and separated into three ability levels (high, medium, and low) as related to perceived self-confidence or self-doubt. At each level, children with perceived self-confidence spent more time solving wrong questions and had higher problem resolution rate as compared to children with self-doubt.5 The same theory can also applies to pharmacy education, where self-confidence can be developed through skill utilization in both didactic and experiential education. In the didactic curriculum, students acquire clinical skills through classroom discussion and laboratory application. In experiential education, students continue to build their self-confidence by applying clinical skills learned in the classroom to different pharmacy practice settings. In order to successfully develop students’ clinical confidence and prepare them for experiential education, it is necessary for them to build confidence in applying these skills throughout the didactic curriculum. The pharmaceutical care laboratory provides students with opportunities to practice clinical skills in a small group setting and to receive frequent feedback from faculty instructors.3 OSCEs are often implemented in the pharmaceutical care laboratory curriculum to assess students’ confidence and competence to perform specific tasks in simulated clinical settings. In one study, second-, third-, and fourth-year pharmacy students completed an OSCE as part of their course requirement.6 Their performance was compared to six licensed pharmacists who completed the same OSCE as the students. Although licensed pharmacists scored higher in communication and clinical skills, students improved significantly in these two areas over time. Eighty-six percent of students expressed confidence that they would be able to perform these types of clinical activities as licensed pharmacists.6 In a similar study, students’ confidence and knowledge scores improved significantly after completion of an active-learning course that focused on student-centered activities such as student-led presentations and peer evaluations.7 These findings suggested that active-learning strategies and summative assessments, such as OSCEs, can evaluate student confidence and mastery of subject matters over time.6 OSCEs have been used to evaluate learners’ competencies in multiple health professions entry examinations, including the United States Medical Licensing Examination, the Canadian Pharmacist Qualification Examination, and the Medical Council of Canada Qualifying Examination.2,8,9 In comparison to written multiple-choice examination, an OSCE allows assessment of multiple dimensions, including interpersonal skills, problem-solving abilities, and clinical decision-making abilities.10 However, students have also reported higher levels of stress and anxiety in completing OSCEs in comparison to traditional multiple-choice examinations.11 In one study, 37% of students reported they are more anxious about OSCEs than traditional multiple choice examinations.11 Some potential approaches to address stress and anxiety are problem-based assessment (PBA) and teaching OSCE (TOSCE), which both are formative assessments. PBA is a nationally recognized learning methodology where students are assigned to a small group with a faculty facilitator.12 In one study, groups of seven or eight students were placed with a faculty facilitator for a one-hour formative learning experience in a physical assessment course.12 At the conclusion of this learning experience, students rated their confidence in performing physical assessment skills as a 3 or 4 on a 5-point Likert scale.12 Unlike PBA, TOSCE is an individual formative assessment where an evaluator provides performance feedback to a learner at the conclusion of his or her performance.13 This feedback can then be used by the learner to prepare for future summative OSCE.13 In the literature, several approaches to using TOSCEs have been reported.13–15 In one study, students were given 10 min to perform a physical assessment of a standardized patient under the observation of a faculty facilitator and multiple student peers. Immediately following the assessment, faculty and student peers provided performance feedback to the student completing the TOSCE.14 In another approach, a student was given 10 min to counsel a standardized patient on an ophthalmic prescription. After the counseling component, a clinical pharmacist spent four minutes providing individualized verbal feedback on the student's clinical knowledge and communication technique.15 In both studies, students and faculty alike were positive in its potential to improve student performance in summative assessments.14–15 A study conducted in the United Kingdom demonstrated improvement in students’ grades with summative OSCE after a formative OSCE (i.e. TOSCE) was implemented.13 A subsequent formative OSCE in 2013 utilized the same content in a different format, demonstrating improvement in overall station scores and communication grades regardless of confounding variables (e.g., curriculum format, pharmacy coursework, Grade Point Average).15 Students viewed the TOSCE as a positive experience to prepare them for summative assessments as well as a valuable opportunity to discuss and receive constructive feedback on their strengths and weaknesses from faculty members and student peers.13–15 Despite students’ positive reviews of their experiences with TOSCEs, there is no literature describing a TOSCE and its impact on student confidence in their clinical decision-making skills and interpersonal communication skills over the course of multiple laboratory sessions. Therefore, the purpose of this study was to examine the relationship between TOSCE implementation and student confidence in three competency areas (clinical skills delivery, interpersonal communication, and problem resolution) and to explore its influence on student performance on summative problem-based assessment (PBA). Methods This study was conducted at the Purdue University College of Pharmacy and was deemed exempt by the institutional review board. All third-year professional students (n = 150) participated in this study as part of a one-credit hour Professional Program Laboratory (PPL).16 In PPL, students apply didactic clinical knowledge in simulated clinical situations, such as formulating pharmacotherapy plans, counseling patients on medications and devices, and communicating recommendations with providers.17 As part of the college's assessment strategy, the summative PBA is completed each semester to evaluate student competency and progress. Students must pass each PBA with a score of at least 80% to advance through the PPL curriculum. Each topic in the PPL curriculum 2
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
B. Deng et al.
lasts one week (five days), and there are 30 students in each laboratory group. Students were required to complete five surveys throughout the semester to receive full participation credit for this course and were informed that their survey responses would have no impact on their course grades. Men's and women's health (MWH) laboratory description The goals of the MWH laboratory were to recommend and counsel a patient on appropriate men's and women's nonprescription and prescription products, use drug references to answer specific questions from patients and providers, and communicate medication recommendations with patients and providers. Students spent 35 min at each of the four stations: men's product counseling, women's product counseling, drug-information (DI) scavenger hunt, and clinical oral case. In the men's and women's product counseling sessions, students were divided into pairs and assigned to a facilitator. One student played the role of the patient, and the other student played the role of the pharmacist. These roles were reversed for the other session. The student pharmacist was given 5 min to counsel the patient on a men's or women's product. In the DI scavenger hunt, students were asked to identify solutions to a series of clinical questions using a variety of drug references, such as primary literature, electronic drug references, evidence-based clinical guidelines, and package inserts. The facilitator provided answers to clinical questions and reviewed the appropriate use of drug references after the laboratory session. TOSCE description and implementation A TOSCE was implemented as a part of the MWH laboratory as two substations of the clinical oral case station. One substation focused on male-related disease states, such as erectile dysfunction and benign prostatic hypertrophy, while the other substation focused on female-related disease states, such as menopause and contraception. One facilitator (clinical pharmacy faculty, pharmacy residents, or pharmacy fellows) and one standardized patient (either a hired actor or a fourth year pharmacy student) was assigned to each TOSCE station. Detailed instruction was distributed to all facilitators (clinical pharmacy faculty, pharmacy residents, pharmacy fellows, and standardized patients) one week prior to the TOSCE implementation. Instructions were provided on the structure and length of the activity (5 min student case presentation with 3 min of verbal feedback) and on the method for providing feedback on the analytical checklist for each patient case. Students also received a brief description and procedure for all four stations in the MWH laboratory one week prior to the first laboratory session. Just prior to the TOSCE, each student was given a case scenario and allowed 5 min to formulate patient-specific recommendations using electronic (e.g., journal articles, Micromedex®, clinical guidelines) and printed (e.g., drug package insert) drug references before interacting with the standardized patient. The student was allowed to bring his or her personal computer to access these online drug references, and the course instructors provided the student with sample drug products and package inserts. The student was then given 5 min to interview a standardized patient to elicit a symptom analysis and counsel the patient on a recommended product. The facilitator used a case-specific analytical checklist (23 items: patient interview = 7 items, pharmacotherapy and patient counseling = 7 items, and communication = 9 items) to evaluate students’ performance on clinical decision-making skills, problemsolving skills, and interpersonal skills. The facilitator then provided 3 min of verbal feedback on these competency areas after the counseling session. PBA description The PBA took place three weeks after the MWH laboratory. This PBA activity simulated a routine student-preceptor interaction on an advanced pharmacy practice experience (APPE). Each student was assigned a patient case with two common chronic disease states (anticoagulation, hypertension, diabetes, hyperlipidemia, or heart failure). The student had 30 min to review the assigned patient case and formulate evidence-based recommendations based on the patient's disease presentation using a teaching electronic medical record and various electronic and printed drug references. After case preparation, each student had 7 min to present therapy recommendations and counseling points to a faculty facilitator, who played the role of an APPE preceptor. The faculty facilitators used a case-specific analytical checklist (21 items: pharmacotherapy and counseling = 18 items, and communication = 3 items) to evaluate students’ performance and provide verbal feedback on communication and verbal case presentation delivery. To minimize academic dishonesty, faculty facilitators were instructed not to provide verbal feedback on the quality of the care plan (pharmacotherapy and counseling points) or inform the student of their performance on the PBA. Survey structure and administration A 19-item survey was constructed based on learning objectives from PPL and contained four sections: student demographics, clinical skills delivery, interpersonal communication skills, and drug-related problem resolution. The survey was developed and administered to students using Qualtrics online.18 Students responded to each item using a 5-point Likert scale (1 = not at all confident to 5 = extremely confident). Student demographic information was collected in the first survey administration. To minimize social desirability, the study team did not access survey responses until the conclusion of the semester after assignment of grades was completed. The course coordinator, who was not part of the study team, monitored students’ progress in survey completion and assigned participation credit as appropriate. A timeline for survey administration is provided in Appendix 1. In summary, the first survey (Survey 1) was administered to students prior to the communication laboratory to establish students’ baseline confidence prior 3
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
B. Deng et al.
Table 1 Survey completion rate for the TOSCE study (n = 150). Survey number
Survey name
Number of student-completed surveys
Survey completion rate (%)
1 2 3 4 5
Pre-communication Pre-MWH lab Post-MWH lab Pre-PBA Post-PBA
136 139 95 128 127
90.7 92.7 63.3 85.3 84.7
TOSCE: teaching objective structured clinical examination; MWH: men's and women's health; PBA: problem-based assessment.
to intervention. The same survey, without demographic questions, was administered two weeks after the communication laboratory immediately prior to the MWH laboratory (Survey 2) as well as just after the MWH laboratory (Survey 3) to measure student confidence changes before and after TOSCE implementation. An additional seven items (1 = strongly disagree to 5 = strongly agree) were added to the original 19-item survey to measure students’ confidence and attitudes specifically towards the PBA, and this version was administered before (Survey 4) and after (Survey 5) the PBA. Survey 4 took place three weeks after Survey 3. All analyses were performed using SPSS.19 An a priori level of statistical significance was established at α = 0.05. Descriptive statistics were performed for all items. Repeated measures ANOVA tests were used to identify statistical significance for each survey item throughout the study period. If a statistical difference was identified, pair-wise comparisons with Bonferroni correction were then performed. Finally, Pearson correlation analysis between student confidence and their performance was conducted. Results All third-year pharmacy students (n = 150) participated in the TOSCE as part of their course requirements. There were 134 students (89%) who completed at least one of the study's surveys. Of these, 72 students (54%) completed all five surveys administered throughout the study period, and their responses were included in the analysis. Table 1 summarizes the response rates for each survey. The mean age was 24 years (range 21–45), with the majority being Caucasian (71%) and female (63%). A total of 100 students (75%) were currently employed in a pharmacy setting, such as community, hospital, or independent pharmacy. Approximately 80% of the respondents reported previous experience in community pharmacy practice. There was no statistical difference in age, gender, race, employment status, or employment settings. Clinical skills delivery Student confidence improved in ability to evaluate appropriate drug therapy for chronic MWH disorders from before to after the MWH laboratory (p = 0.048, Table 2). Students also reported non-statistically significant improvements in confidence for the following: ability to gather relevant patient data; identify, resolve, and prioritize drug-related problems; and evaluate appropriate treatment options in managing acute men's and women's health disorders. Students reported that their confidence improved in prioritizing drug-related issues (p = 0.046) and managing chronic men's and women's health-related disorders (p = 0.004) between the pre-communication and pre-MWH laboratories. Students’ confidence also improved in their ability to evaluate appropriate treatment options in managing acute (p = 0.008) and chronic (p = 0.001) MWH disorders and in developing patient-specific monitoring plans (p = 0.001) between the pre-communication and post-MWH laboratories. Interpersonal communication Students reported non-statistically significant improvement in confidence to counsel a patient on lifestyle and medications and utilize appropriate techniques to assess a patient's understanding of the counseling session from before to after the MWH laboratory (Table 3). Students also showed improved confidence in their ability to counsel a patient on different MWH devices between the precommunication and post-MWH laboratories (p = 0.036). Drug-related problem resolution Student responses indicated that confidence improved in all areas measured in this competency, although these improvements were not statistically significant (Table 4). The change in student confidence in using package inserts to answer medication questions was statistically significant (p = 0.023) between the pre-communication and post-MWH laboratories. In addition, there was a statistically significant difference in student confidence in using electronic drug references (p = 0.046) between the pre-MWH laboratory and after the PBA. Correlation between student confidence and predicted performance on PBA Students felt confident in their preparation for the PBA (p = 0.029) and believed they performed well on the assessment (p < 0.001) despite not being aware of their grade (Table 5). Students also agreed that the feedback they received from the TOSCE in 4
3.07 2.79 2.83 2.99 2.93 2.86 2.50 2.75 3.25
Obtain focused and relevant patient history using the SCHOLAR methodd Identify drug-related problems (e.g., inappropriate therapy, drug-drug interactions)d Resolve drug-related problems (e.g., inappropriate therapy, drug-drug interactions)d Prioritize drug-related issuesd,f,g Identify OTC drug therapyd Evaluate prescription drug therapy options for a chronic disease state (e.g., BPH)d,e,f,g Evaluate prescription drug therapy options for an acute disease state (e.g., UTI)d,e,f Develop a patient monitoring pland,e Recommend lifestyle modifications
5
3.17 2.92 2.90 3.03 3.00 2.90 2.69 2.88 3.27
(0.44) (0.47) (0.55) (0.62) (0.61) (0.60) (0.63) (0.64) (0.60)
Pre-MWH lab 3.21 2.93 2.99 3.15 3.07 3.10 2.90 3.01 3.30
(0.47) (0.51) (0.53) (0.59) (0.54) (0.60) (0.65) (0.64) (0.58)
Post- MWH lab 3.10 2.90 2.94 3.07 3.01 3.07 2.76 2.98 3.29
(0.45) (0.56) (0.48) (0.56) (0.61) (0.45) (0.64) (0.56) (0.53)
Pre- PBA
3.28 3.03 3.06 3.24 3.20 3.13 2.89 3.07 3.26
(0.54) (0.56) (0.56) (0.57) (0.73) (0.60) (0.68) (0.70) (0.67)
Post-PBA
1.000 1.000 1.000 0.716 1.000 0.048 0.081 0.404 0.062
p-value (pre- and post-MWH Lab)
SCHOLAR: symptoms, characteristics, history, onset, location, aggravating factors, remitting factors; MWH: men's and women's health; PBA: problem-based assessment; OTC: over-the-counter; BPH: benign prostatic hypertrophy; UTI: urinary tract infection. a Scale: 1 = not at all confident; 2 = minimally confident; 3 = moderately confident; 4 = very confident; 5 = extremely confident. b Repeated ANOVA was performed to assess difference in student confidence throughout the same period. c Pair-wise comparison with Bonferronic Correction was performed to assess difference between two time points in the study period. d p < 0.05 for statistically difference between pre-communication and post-PBA. e p < 0.05 for statistically difference between pre-communication and post-MWH lab. f p < 0.05 for statistically difference between pre-communication and pre-PBA. g p < 0.05 for statistically difference between pre-MWH lab and post-PBA.
(0.48) (0.53) (0.57) (0.65) (0.68) (0.54) (0.68) (0.62) (0.61)
Pre-communication
Item
Table 2 Mean (standard deviation) for student confidence in clinical skills delivery (n = 72).a,b,c
B. Deng et al.
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
d
6
3.11 2.96 2.99 2.74 3.22 3.15 3.14
(0.64) (0.66) (0.52) (0.60) (0.61) (0.66) (0.53)
Pre-communication 3.15 3.00 3.07 2.82 3.29 3.18 3.14
(0.67) (0.65) (0.56) (0.63) (0.64) (0.66) (0.68)
Pre-MWH lab 3.19 3.13 3.11 2.93 3.31 3.28 3.14
(0.59) (0.64) (0.58) (0.63) (0.63) (0.54) (0.59)
Post-MWH lab 3.18 3.01 3.06 2.93 3.32 3.35 3.24
(0.56) (0.62) (0.49) (0.63) (0.62) (0.57) (0.60)
Pre-PBA
MWH: men's and women's health; PBA: problem-based assessment; OTC: over-the-counter. a Scale: 1 = not at all confident; 2 = minimally confident; 3 = moderately confident; 4 = very confident; 5 = extremely confident. b Repeated ANOVA was performed to assess difference in student confidence throughout the same period. c Pair-wise comparison with Bonferronic Correction was performed to assess difference between two time points in the study period. d p < 0.05 for statistically difference between pre-communication and post-PBA. e p < 0.05 for statistically difference between pre-communication and post-MWH lab.
Counsel a patient on lifestyle modification Counsel a patient on OTC products Counsel a patient on prescription medicationsd Counsel a patient on devicesd,e Use teach-back method to assess patient understandingd Use open-ended questions to assess patient understanding Explain a follow-up plan to a patient
Item
Table 3 Mean (standard deviation) for student confidence in patient communication (n = 72).a,b,c
3.31 3.03 3.21 3.01 3.44 3.29 3.30
(0.60) (0.67) (0.61) (0.66) (0.61) (0.70) (0.62)
Post-PBA
1.000 0.832 1.000 1.000 1.000 0.069 1.000
p-value (Pre-MWH lab and post-MWH lab)
B. Deng et al.
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
7
primary literature (e.g., journal article) to answer medication questions electronic drug references (e.g., Micromedex) to answer medication questionsd,e evidence-based clinical guidelines (e.g., IDSA) to answer medication questions reliable online resources (e.g., CDC, FDA) to answer medication questions package insert to answer medication questionsd,f
d
2.92 (0.63) 3.39 (0.58) 2.92 (0.64) 3.06(0.66) 3.22 (0.69)
Pre-communication 2.97 3.42 2.96 3.00 3.28
(0.66) (0.58) (0.67) (0.64) (0.61)
Pre-MWH lab 3.06 3.56 3.06 3.13 3.43
(0.64) (0.51) (0.64) (0.67) (0.59)
Post-MWH lab 3.10 3.58 2.99 3.18 3.38
(0.64) (0.51) (0.64) (0.62) (0.55)
Pre-PBA
3.15 3.51 3.10 3.15 3.39
(0.71) (0.53) (0.68) (0.64) (0.65)
Post-PBA
1.000 0.114 1.000 0.488 0.153
p-value (Pre-MWH lab and Post MWH lab)
MWH: men's and women's health; PBA: problem-based assessment; IDSA: Infectious Diseases Society of America; CDC: Centers for Disease Control; FDA: Food and Drug Administration. a Scale: 1 = not at all confident; 2 = minimally confident; 3 = moderately confident; 4 = very confident; 5 = extremely confident. b Repeated ANOVA was performed to assess difference in student confidence throughout the same period. c Pair-wise comparison with Bonferronic Correction was performed to assess difference between two time points in the study period. d p < 0.05 for statistically difference between pre-communication and post-PBA. e p < 0.05 for statistically difference between pre-MWH lab and post-PBA. f p < 0.05 for statistically difference between pre-communication and post-MWH lab.
Use Use Use Use Use
Item
Table 4 Mean (standard deviation) for student confidence in drug-related problem resolution (n = 72).a,b,c
B. Deng et al.
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
B. Deng et al.
Table 5 Student confidence and predicted grades on PBA (n = 72).a Questions
Pearson correlation coefficient
p-value
I feel confident going into PBAb I feel confident that I will perform well on PBAb I feel anxious when I am taking a PBA PBA is a stressful experience The TOSCE prepared me for the PBAb
0.197 0.353 −0.072 −0.117 0.177
0.029 <0.001 0.424 0.193 0.048
PBA: problem-based assessment; TOSCE: teaching objective structured clinical examination. a Two-tailed Pearson test was performed to establish correlation between student confidence and their predicted grades on PBA. b p < 0.05 for positive relationship between student confidence and their predicted grades on PBA.
the MWH laboratory was beneficial and favorably contributed to their performance on the PBA (p = 0.048). Students’ confidence in preparing for and taking the PBA was positively reflected on their actual performance on the PBA, where the average score was 82% (high = 100%, low = 70%, standard deviation ± 8.17%) and 59.3% of students achieved a grade of ≥80% on the PBA. Discussion TOSCE and student confidence in summative assessment To the authors’ knowledge, this is the first prospective study describing TOSCE implementation and changes in student confidence over multiple laboratory sessions. The goal of a TOSCE is to provide students with formative feedback that addresses both their strengths and weaknesses as well as prepare students for summative assessments, such as OSCEs.14 This study was structured similarly to a standard OSCE format, with an exception of the length of each TOSCE session (three minutes feedback) and personnel composition (pharmacy faculty, residents, and fellows). Students indicated their confidence improved after TOSCE implementation, although the improvements were not statistically significant. One potential explanation was that students already rated themselves as moderately confident prior to TOSCE implementation in the MWH laboratory. The incremental improvement (3.2 on a 5-point scale) in the post-MWH survey indicated that a TOSCE was effective in developing their confidence in core competency areas but was not sufficient to improve their confidence to very confident (4 on a 5-point scale) with only one TOSCE session. The results from this study support those in the literature that a TOSCE has the potential to enhance student confidence and their readiness for summative assessments,15 even though a sufficient amount of TOSCE experiences are needed to see significant change in student's confidence level (such as advancing from moderately confident to very confident). In our study, student confidence increased immediately after completing the TOSCE. However, confidence was not sustained for the duration of the semester as student confidence decreased prior to the PBA for most survey items. One possible explanation was the gap of time between the MWH laboratory and the PBA. Since there is a three week gap between the MWH laboratory and the PBA, the confidence gained from the TOSCE experience may not have carried over to the PBA. Thus, increasing the number of TOSCEs during this time period may be necessary to effectively sustain student confidence over the course of the semester. PBA is a summative assessment that is needed to objectively measure learners’ performance in multiple competency areas. In this study, students described their confidence as moderately confident (3.3 on a 5-point Likert scale) after taking the PBA, which was achieved in a highly stressful summative assessment environment. The confidence in this study was comparable to confidence (3 or 4 on a 5-point Likert scale) achieved in a less stressful problem-based learning assessment.12 This result suggests that summative assessments, such as the PBA, are valuable tools that can achieve similar outcomes in student confidence as those seen in formative assessments. TOSCE and student performance in summative assessment Numerous studies have described the relationships between formative assessments and student grades on OSCE, although very few studies address the relationship between student confidence and their performance on summative assessments. In our study, students felt confident that they would perform well on the PBAs. Students also agreed that the feedback they received from the TOSCE prepared them for the PBA. Similar results were identified in a study conducted in a large medical school in England, where students reported that formative assessment was a good preparation for summative assessment.13 In particular, 46.6% and 85% of second-year medical students passed their formative and summative OSCEs respectively.13 In our study, 100% of students passed their PBAs and 59.3% of them achieved a grade of 80% or higher, which positively correlated with their confidence on predicting their performance on the PBA. The results from this study support previous findings that formative assessments (TOSCEs) have an important role in building student confidence and preparing them for high-stake summative assessments, such as PBA and OSCE.13 Limitations This study has several limitations. First, there was no comparator group. Since this was a pilot study and all participants in this study were third-year pharmacy students, the true degree of confidence change is unclear. In future studies, students from first and 8
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
B. Deng et al.
second didactic years will also be included so the study can further evaluate if the same degree of confidence change can be replicated in students with different didactic backgrounds. Although 134 students completed at least one survey, only 72 students (54%) completed all five surveys administered throughout the study period, despite it being a requirement for the laboratory course. This may be due to the small point value allocated for this assignment. Since it only accounted for a very small percentage of student grades, some students may have chosen to accept the consequence of losing these points instead of completing all five surveys. In future studies, increasing incentives for completing all five surveys may improve survey completion rate and expand overall sample size. Another reason that may explain the low survey response rate is survey fatigue, given that five surveys were administered in a semester. In particular, multiple back-to-back surveys were given in a relatively short period of time, which may negatively attribute to low survey response rate in this study.20 In future studies, decreasing the number of surveys administered in a short amount of time or decreasing back-to-back surveys may increase survey response rate. Another challenge encountered over the course of this study was students’ prior experience with high-stakes summative assessments. Since these students may already have had unpleasant feelings with summative assessments and developed mechanisms to cope with these feelings, it may have diminished the benefits of TOSCEs on student confidence with high-stakes summative assessments. One final challenge encountered in this study was the content difference between the TOSCE and the PBA. The TOSCE was implemented in the MWH laboratory, which primarily focused on male and female related disease states (e.g., benign prostatic hypertrophy, menopause, contraception). In contrast, common primary care chronic disease states (e.g., diabetes, hypertension, hyperlipidemia) were evaluated in the PBA. Due to the content difference, student confidence in the MWH laboratory may not affect students’ performance in the PBA. However, some of the skills emphasized in the MWH laboratory, such as clinical skills delivery, interpersonal communication, and drug-related resolution skills, are similar to and transferrable across different laboratories, which may minimize the difference in student confidence between these laboratory experiences. Lastly, social desirability bias may not be completely ruled out since study results were based on student self-reported responses. To limit the potential impact of social desirability bias, student responses were not provided to the study team until grades were posted at the end of the semester. In addition, students were graded on completion of all five surveys instead of the content of students’ responses. Conclusion A TOSCE is a formative assessment that has the potential to improve student confidence in core competency areas. In our study, students indicated that they were moderately confident in core competency areas after TOSCE implementation, which was higher than pre-TOSCE confidence in most cases. Additional studies are needed to assess long-term sustainability across multiple laboratory activities over the semester and throughout the curriculum. Implementing multiple TOSCEs over the course of a semester or shortening the time interval between TOSCE activities and PBAs may improve student confidence prior to taking high-stake summative assessments. Although students report increased stress when taking summative assessments, a combination of summative and formative assessments may enhance students’ confidence throughout their didactic and experiential education. Conflict of interest None. Disclosure(s) None. Appendix 1. Study administration timeline TOSCE = teaching objective structured clinical examination.
9
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
B. Deng et al.
Supplementary materials Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.cptl.2018.11.009. References 1. Salih MRM, Baidi Bahari M, Syed Sulaimam SA, Sarriff A, Yaseen SN, Oktavia Sari Y. Pharmacy student perceptions and feedback on the modified objective structured clinical examination. Pharm Educ. 2010;10(2):165–172. 2. Austin Z, O'Byrne C, Pugsley J, Munoz LQ. Development and validation processes for an objective structured clinical examination (OSCE) for entry-to-practice certification in pharmacy: the Canadian experience. Am J Pharm Educ. 2003;67(3) https://doi.org/10.5688/aj7005119. 3. Bond R, Cone C. Improving student confidence in skill performance in a pharmaceutical care lab setting. Pharm Educ. 2012;12(1):20–24. 4. Accreditation Standards and Key Elements for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree (“Standards 2016”). Accreditation Council for Pharmacy Education. Published February 2015. https://www.acpe-accredit.org/pdf/Standards2016FINAL.pdf. Accessed 6 November 2018. 5. Bandura A. Perceived self-efficacy in cognitive development and functioning. Educ Psychol. 1993;28(2):117–148. 6. Urteaga EM, Attridge RL, Tovar JM, Witte AP. Evaluation of clinical and communication skills of pharmacy students and pharmacists with an objective structured clinical examination. Am J Pharm Educ. 2015;79(8) https://doi.org/10.5688/ajpe798122. 7. Thomas MC, Macias-Moriarity LZ. Student knowledge and confidence in an elective clinical toxicology course using active-learning techniques. Am J Pharm Educ. 2014;78(5) https://doi.org/10.5688/ajpe78595. 8. Sturpe DA. Objective structured clinical examinations in doctor of pharmacy programs in the United States. Am J Pharm Educ. 2010;74(8) https://doi.org/10. 5688/aj7408148. 9. Shirwaikar A. Objective structured clinical examination (OSCE) in pharmacy education - a trend. Pharm Pract (Granada). 2015;13(4) https://doi.org/10.18549/ PharmPract.2015.04.627. 10. Sloan DA, Donnelly MB, Schwartz RW, Strodel WE. The objective structured clinical examination. The new gold standard for evaluating postgraduate clinical performance. Ann Surg. 1995;222(6):735–742. 11. Allen R, Heard J, Savidge M, Bittergle J, Cantrell M, Huffmaster T. Surveying students’ attitudes during the OSCE. Adv Heal Sci Educ. 1998;3(3):197–206. 12. Sherman JJ, Riche DM, Stover KR. Physical assessment experience in a problem-based learning course. Am J Pharm Educ. 2011;75(8) https://doi.org/10.5688/ ajpe758156. 13. Chisnall B, Vince T, Hall S, Tribe R. Evaluation of outcomes of a formative objective structured clinical examination for second-year UK medical students. Int J Med Educ. 2015;6:76–83. 14. Brazeau C, Boyd L, Crosson J. Changing an existing OSCE to a teaching tool: the making of a TOSCE. Acad Med. 2002;77(9):932. 15. Gums TH, Kleppinger EL, Urick BY. Outcomes of individualized formative assessments in a pharmacy skills laboratory. Am J Pharm Educ. 2014;78(9) https://doi. org/10.5688/ajpe789166. 16. PharmD Curriculum. Purdue College of Pharmacy. Published 2017. https://www.pharmacy.purdue.edu/current-students/resources/curriculum. Accessed 6 November 2018. 17. Detailed Course Information. Purdue University. https://selfservice.mypurdue.purdue.edu/prod/bzwsrch.p_catalog_detail?term=current&subject=PHRM& cnbr=86100. Accessed 6 November 2018. 18. Qualtrics [comupter program]. Version 09.16. Provo, Utah: Qualtrics; 2017. 19. IBM SPSS Statistics (for Windows) [computer program]. Version 23.0. Armonk, NY: IBM Corporation; 2017. 20. Porter S, Whitcomb M, Weitzer W. Multiple surveys of students and survey fatigue. New Dir Inst Res. 2004;2004(121):63–73. https://doi.org/10.1002/ir.101.
10
Contents lists available at ScienceDirect
Currents in Pharmacy Teaching and Learning journal homepage: www.elsevier.com/locate/cptl
Research Note
Impact of a teaching objective structured clinical examination (TOSCE) on student confidence in a pharmacy skills laboratory Bin Denga, Norman E Fenn IIIb, Kimberly S. Plakec, a b c
⁎
California Northstate University College of Pharmacy, 9700 West Taron Drive, Elk Grove, CA 95757, United States The University of Texas at Tyler, Department of Clinical Sciences, 3900 University Blvd, Tyler, TX 75799, United States Purdue University College of Pharmacy, Department of Pharmacy Practice, 575 Stadium Mall Drive, West Lafayette, IN 47907, United States
A R T IC LE I N F O
ABS TRA CT
Keywords: Teaching objective structured clinical examination Pharmacy skills laboratory Student confidence Interpersonal communication skills Clinical skills delivery Medication-related problem resolution
Objective: To describe the relationship between the use of a teaching objective structured clinical examination (TOSCE) and student confidence in a pharmacy skills laboratory. Methods: To develop students’ abilities in providing recommendations and patient education on selected drug products, third-year pharmacy students participated in a TOSCE in a men's and women's health laboratory. A 19-item, 5-point Likert scale survey was administered longitudinally to assess student confidence (1 = not all confident to 5 = extremely confident). An additional seven items (1 = strongly disagree to 5 = strongly agree) were added to the original 19-item survey to measure changes in confidence and attitude before and after a problem-based assessment. Results: Seventy-two students completed all five surveys administered throughout the semester. Students’ confidence improved in multiple areas within the clinical skills and interpersonal communication skills categories. Students also indicated their confidence in using primary literature, electronic drug references, and package inserts to answer medication-related questions improved after the TOSCE implementation. Students agreed that the feedback they received from the TOSCE was beneficial to their performance on a problem-based assessment. Conclusions: A TOSCE is a formative assessment that has the potential to improve student confidence in core competency areas immediately after an activity. However, its benefit may be short-term when a time gap exists. Further study is needed to assess long-term sustainability in student confidence over multiple laboratory activities throughout a semester.
Introduction An objective structured clinical examination (OSCE) is an assessment method used to evaluate learners’ abilities to perform various clinical and technical tasks.1 In an OSCE, learners rotate through a series of timed interactive and non-interactive stations.2 In an interactive station, a learner performs a task with a standardized patient or an individual who receives specific training on portraying a patient with a specific medical condition. The learner's interaction is observed and assessed by a facilitator who is either a faculty member, pharmacy resident, or a pharmacy fellow using a standardized assessment called an analytical checklist.2 At a noninteractive station, the learner provides a written response to a task or problem without direct observation from a faculty member.2 Confidence is essential in ensuring students have the ability and willingness to effectively perform clinical tasks in experiential
⁎
Corresponding author. E-mail addresses: [email protected] (B. Deng), [email protected] (N.E. Fenn), [email protected] (K.S. Plake).
https://doi.org/10.1016/j.cptl.2018.11.009
1877-1297/ © 2018 Elsevier Inc. All rights reserved.
Please cite this article as: Deng, B., Currents in Pharmacy Teaching and Learning, https://doi.org/10.1016/j.cptl.2018.11.009
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
B. Deng et al.
practice settings. Social cognitive theory defines the role of confidence as belief in one's own ability, and this can be developed by creating self-directed lifelong learners as required by the Accreditation Council for Pharmacy Education Accreditation Standards.3–5 Students who have self-confidence perform better and have a positive attitude when solving complex problems.5 In a study conducted by Bandura,5 school children were given difficult mathematical questions and separated into three ability levels (high, medium, and low) as related to perceived self-confidence or self-doubt. At each level, children with perceived self-confidence spent more time solving wrong questions and had higher problem resolution rate as compared to children with self-doubt.5 The same theory can also applies to pharmacy education, where self-confidence can be developed through skill utilization in both didactic and experiential education. In the didactic curriculum, students acquire clinical skills through classroom discussion and laboratory application. In experiential education, students continue to build their self-confidence by applying clinical skills learned in the classroom to different pharmacy practice settings. In order to successfully develop students’ clinical confidence and prepare them for experiential education, it is necessary for them to build confidence in applying these skills throughout the didactic curriculum. The pharmaceutical care laboratory provides students with opportunities to practice clinical skills in a small group setting and to receive frequent feedback from faculty instructors.3 OSCEs are often implemented in the pharmaceutical care laboratory curriculum to assess students’ confidence and competence to perform specific tasks in simulated clinical settings. In one study, second-, third-, and fourth-year pharmacy students completed an OSCE as part of their course requirement.6 Their performance was compared to six licensed pharmacists who completed the same OSCE as the students. Although licensed pharmacists scored higher in communication and clinical skills, students improved significantly in these two areas over time. Eighty-six percent of students expressed confidence that they would be able to perform these types of clinical activities as licensed pharmacists.6 In a similar study, students’ confidence and knowledge scores improved significantly after completion of an active-learning course that focused on student-centered activities such as student-led presentations and peer evaluations.7 These findings suggested that active-learning strategies and summative assessments, such as OSCEs, can evaluate student confidence and mastery of subject matters over time.6 OSCEs have been used to evaluate learners’ competencies in multiple health professions entry examinations, including the United States Medical Licensing Examination, the Canadian Pharmacist Qualification Examination, and the Medical Council of Canada Qualifying Examination.2,8,9 In comparison to written multiple-choice examination, an OSCE allows assessment of multiple dimensions, including interpersonal skills, problem-solving abilities, and clinical decision-making abilities.10 However, students have also reported higher levels of stress and anxiety in completing OSCEs in comparison to traditional multiple-choice examinations.11 In one study, 37% of students reported they are more anxious about OSCEs than traditional multiple choice examinations.11 Some potential approaches to address stress and anxiety are problem-based assessment (PBA) and teaching OSCE (TOSCE), which both are formative assessments. PBA is a nationally recognized learning methodology where students are assigned to a small group with a faculty facilitator.12 In one study, groups of seven or eight students were placed with a faculty facilitator for a one-hour formative learning experience in a physical assessment course.12 At the conclusion of this learning experience, students rated their confidence in performing physical assessment skills as a 3 or 4 on a 5-point Likert scale.12 Unlike PBA, TOSCE is an individual formative assessment where an evaluator provides performance feedback to a learner at the conclusion of his or her performance.13 This feedback can then be used by the learner to prepare for future summative OSCE.13 In the literature, several approaches to using TOSCEs have been reported.13–15 In one study, students were given 10 min to perform a physical assessment of a standardized patient under the observation of a faculty facilitator and multiple student peers. Immediately following the assessment, faculty and student peers provided performance feedback to the student completing the TOSCE.14 In another approach, a student was given 10 min to counsel a standardized patient on an ophthalmic prescription. After the counseling component, a clinical pharmacist spent four minutes providing individualized verbal feedback on the student's clinical knowledge and communication technique.15 In both studies, students and faculty alike were positive in its potential to improve student performance in summative assessments.14–15 A study conducted in the United Kingdom demonstrated improvement in students’ grades with summative OSCE after a formative OSCE (i.e. TOSCE) was implemented.13 A subsequent formative OSCE in 2013 utilized the same content in a different format, demonstrating improvement in overall station scores and communication grades regardless of confounding variables (e.g., curriculum format, pharmacy coursework, Grade Point Average).15 Students viewed the TOSCE as a positive experience to prepare them for summative assessments as well as a valuable opportunity to discuss and receive constructive feedback on their strengths and weaknesses from faculty members and student peers.13–15 Despite students’ positive reviews of their experiences with TOSCEs, there is no literature describing a TOSCE and its impact on student confidence in their clinical decision-making skills and interpersonal communication skills over the course of multiple laboratory sessions. Therefore, the purpose of this study was to examine the relationship between TOSCE implementation and student confidence in three competency areas (clinical skills delivery, interpersonal communication, and problem resolution) and to explore its influence on student performance on summative problem-based assessment (PBA). Methods This study was conducted at the Purdue University College of Pharmacy and was deemed exempt by the institutional review board. All third-year professional students (n = 150) participated in this study as part of a one-credit hour Professional Program Laboratory (PPL).16 In PPL, students apply didactic clinical knowledge in simulated clinical situations, such as formulating pharmacotherapy plans, counseling patients on medications and devices, and communicating recommendations with providers.17 As part of the college's assessment strategy, the summative PBA is completed each semester to evaluate student competency and progress. Students must pass each PBA with a score of at least 80% to advance through the PPL curriculum. Each topic in the PPL curriculum 2
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
B. Deng et al.
lasts one week (five days), and there are 30 students in each laboratory group. Students were required to complete five surveys throughout the semester to receive full participation credit for this course and were informed that their survey responses would have no impact on their course grades. Men's and women's health (MWH) laboratory description The goals of the MWH laboratory were to recommend and counsel a patient on appropriate men's and women's nonprescription and prescription products, use drug references to answer specific questions from patients and providers, and communicate medication recommendations with patients and providers. Students spent 35 min at each of the four stations: men's product counseling, women's product counseling, drug-information (DI) scavenger hunt, and clinical oral case. In the men's and women's product counseling sessions, students were divided into pairs and assigned to a facilitator. One student played the role of the patient, and the other student played the role of the pharmacist. These roles were reversed for the other session. The student pharmacist was given 5 min to counsel the patient on a men's or women's product. In the DI scavenger hunt, students were asked to identify solutions to a series of clinical questions using a variety of drug references, such as primary literature, electronic drug references, evidence-based clinical guidelines, and package inserts. The facilitator provided answers to clinical questions and reviewed the appropriate use of drug references after the laboratory session. TOSCE description and implementation A TOSCE was implemented as a part of the MWH laboratory as two substations of the clinical oral case station. One substation focused on male-related disease states, such as erectile dysfunction and benign prostatic hypertrophy, while the other substation focused on female-related disease states, such as menopause and contraception. One facilitator (clinical pharmacy faculty, pharmacy residents, or pharmacy fellows) and one standardized patient (either a hired actor or a fourth year pharmacy student) was assigned to each TOSCE station. Detailed instruction was distributed to all facilitators (clinical pharmacy faculty, pharmacy residents, pharmacy fellows, and standardized patients) one week prior to the TOSCE implementation. Instructions were provided on the structure and length of the activity (5 min student case presentation with 3 min of verbal feedback) and on the method for providing feedback on the analytical checklist for each patient case. Students also received a brief description and procedure for all four stations in the MWH laboratory one week prior to the first laboratory session. Just prior to the TOSCE, each student was given a case scenario and allowed 5 min to formulate patient-specific recommendations using electronic (e.g., journal articles, Micromedex®, clinical guidelines) and printed (e.g., drug package insert) drug references before interacting with the standardized patient. The student was allowed to bring his or her personal computer to access these online drug references, and the course instructors provided the student with sample drug products and package inserts. The student was then given 5 min to interview a standardized patient to elicit a symptom analysis and counsel the patient on a recommended product. The facilitator used a case-specific analytical checklist (23 items: patient interview = 7 items, pharmacotherapy and patient counseling = 7 items, and communication = 9 items) to evaluate students’ performance on clinical decision-making skills, problemsolving skills, and interpersonal skills. The facilitator then provided 3 min of verbal feedback on these competency areas after the counseling session. PBA description The PBA took place three weeks after the MWH laboratory. This PBA activity simulated a routine student-preceptor interaction on an advanced pharmacy practice experience (APPE). Each student was assigned a patient case with two common chronic disease states (anticoagulation, hypertension, diabetes, hyperlipidemia, or heart failure). The student had 30 min to review the assigned patient case and formulate evidence-based recommendations based on the patient's disease presentation using a teaching electronic medical record and various electronic and printed drug references. After case preparation, each student had 7 min to present therapy recommendations and counseling points to a faculty facilitator, who played the role of an APPE preceptor. The faculty facilitators used a case-specific analytical checklist (21 items: pharmacotherapy and counseling = 18 items, and communication = 3 items) to evaluate students’ performance and provide verbal feedback on communication and verbal case presentation delivery. To minimize academic dishonesty, faculty facilitators were instructed not to provide verbal feedback on the quality of the care plan (pharmacotherapy and counseling points) or inform the student of their performance on the PBA. Survey structure and administration A 19-item survey was constructed based on learning objectives from PPL and contained four sections: student demographics, clinical skills delivery, interpersonal communication skills, and drug-related problem resolution. The survey was developed and administered to students using Qualtrics online.18 Students responded to each item using a 5-point Likert scale (1 = not at all confident to 5 = extremely confident). Student demographic information was collected in the first survey administration. To minimize social desirability, the study team did not access survey responses until the conclusion of the semester after assignment of grades was completed. The course coordinator, who was not part of the study team, monitored students’ progress in survey completion and assigned participation credit as appropriate. A timeline for survey administration is provided in Appendix 1. In summary, the first survey (Survey 1) was administered to students prior to the communication laboratory to establish students’ baseline confidence prior 3
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
B. Deng et al.
Table 1 Survey completion rate for the TOSCE study (n = 150). Survey number
Survey name
Number of student-completed surveys
Survey completion rate (%)
1 2 3 4 5
Pre-communication Pre-MWH lab Post-MWH lab Pre-PBA Post-PBA
136 139 95 128 127
90.7 92.7 63.3 85.3 84.7
TOSCE: teaching objective structured clinical examination; MWH: men's and women's health; PBA: problem-based assessment.
to intervention. The same survey, without demographic questions, was administered two weeks after the communication laboratory immediately prior to the MWH laboratory (Survey 2) as well as just after the MWH laboratory (Survey 3) to measure student confidence changes before and after TOSCE implementation. An additional seven items (1 = strongly disagree to 5 = strongly agree) were added to the original 19-item survey to measure students’ confidence and attitudes specifically towards the PBA, and this version was administered before (Survey 4) and after (Survey 5) the PBA. Survey 4 took place three weeks after Survey 3. All analyses were performed using SPSS.19 An a priori level of statistical significance was established at α = 0.05. Descriptive statistics were performed for all items. Repeated measures ANOVA tests were used to identify statistical significance for each survey item throughout the study period. If a statistical difference was identified, pair-wise comparisons with Bonferroni correction were then performed. Finally, Pearson correlation analysis between student confidence and their performance was conducted. Results All third-year pharmacy students (n = 150) participated in the TOSCE as part of their course requirements. There were 134 students (89%) who completed at least one of the study's surveys. Of these, 72 students (54%) completed all five surveys administered throughout the study period, and their responses were included in the analysis. Table 1 summarizes the response rates for each survey. The mean age was 24 years (range 21–45), with the majority being Caucasian (71%) and female (63%). A total of 100 students (75%) were currently employed in a pharmacy setting, such as community, hospital, or independent pharmacy. Approximately 80% of the respondents reported previous experience in community pharmacy practice. There was no statistical difference in age, gender, race, employment status, or employment settings. Clinical skills delivery Student confidence improved in ability to evaluate appropriate drug therapy for chronic MWH disorders from before to after the MWH laboratory (p = 0.048, Table 2). Students also reported non-statistically significant improvements in confidence for the following: ability to gather relevant patient data; identify, resolve, and prioritize drug-related problems; and evaluate appropriate treatment options in managing acute men's and women's health disorders. Students reported that their confidence improved in prioritizing drug-related issues (p = 0.046) and managing chronic men's and women's health-related disorders (p = 0.004) between the pre-communication and pre-MWH laboratories. Students’ confidence also improved in their ability to evaluate appropriate treatment options in managing acute (p = 0.008) and chronic (p = 0.001) MWH disorders and in developing patient-specific monitoring plans (p = 0.001) between the pre-communication and post-MWH laboratories. Interpersonal communication Students reported non-statistically significant improvement in confidence to counsel a patient on lifestyle and medications and utilize appropriate techniques to assess a patient's understanding of the counseling session from before to after the MWH laboratory (Table 3). Students also showed improved confidence in their ability to counsel a patient on different MWH devices between the precommunication and post-MWH laboratories (p = 0.036). Drug-related problem resolution Student responses indicated that confidence improved in all areas measured in this competency, although these improvements were not statistically significant (Table 4). The change in student confidence in using package inserts to answer medication questions was statistically significant (p = 0.023) between the pre-communication and post-MWH laboratories. In addition, there was a statistically significant difference in student confidence in using electronic drug references (p = 0.046) between the pre-MWH laboratory and after the PBA. Correlation between student confidence and predicted performance on PBA Students felt confident in their preparation for the PBA (p = 0.029) and believed they performed well on the assessment (p < 0.001) despite not being aware of their grade (Table 5). Students also agreed that the feedback they received from the TOSCE in 4
3.07 2.79 2.83 2.99 2.93 2.86 2.50 2.75 3.25
Obtain focused and relevant patient history using the SCHOLAR methodd Identify drug-related problems (e.g., inappropriate therapy, drug-drug interactions)d Resolve drug-related problems (e.g., inappropriate therapy, drug-drug interactions)d Prioritize drug-related issuesd,f,g Identify OTC drug therapyd Evaluate prescription drug therapy options for a chronic disease state (e.g., BPH)d,e,f,g Evaluate prescription drug therapy options for an acute disease state (e.g., UTI)d,e,f Develop a patient monitoring pland,e Recommend lifestyle modifications
5
3.17 2.92 2.90 3.03 3.00 2.90 2.69 2.88 3.27
(0.44) (0.47) (0.55) (0.62) (0.61) (0.60) (0.63) (0.64) (0.60)
Pre-MWH lab 3.21 2.93 2.99 3.15 3.07 3.10 2.90 3.01 3.30
(0.47) (0.51) (0.53) (0.59) (0.54) (0.60) (0.65) (0.64) (0.58)
Post- MWH lab 3.10 2.90 2.94 3.07 3.01 3.07 2.76 2.98 3.29
(0.45) (0.56) (0.48) (0.56) (0.61) (0.45) (0.64) (0.56) (0.53)
Pre- PBA
3.28 3.03 3.06 3.24 3.20 3.13 2.89 3.07 3.26
(0.54) (0.56) (0.56) (0.57) (0.73) (0.60) (0.68) (0.70) (0.67)
Post-PBA
1.000 1.000 1.000 0.716 1.000 0.048 0.081 0.404 0.062
p-value (pre- and post-MWH Lab)
SCHOLAR: symptoms, characteristics, history, onset, location, aggravating factors, remitting factors; MWH: men's and women's health; PBA: problem-based assessment; OTC: over-the-counter; BPH: benign prostatic hypertrophy; UTI: urinary tract infection. a Scale: 1 = not at all confident; 2 = minimally confident; 3 = moderately confident; 4 = very confident; 5 = extremely confident. b Repeated ANOVA was performed to assess difference in student confidence throughout the same period. c Pair-wise comparison with Bonferronic Correction was performed to assess difference between two time points in the study period. d p < 0.05 for statistically difference between pre-communication and post-PBA. e p < 0.05 for statistically difference between pre-communication and post-MWH lab. f p < 0.05 for statistically difference between pre-communication and pre-PBA. g p < 0.05 for statistically difference between pre-MWH lab and post-PBA.
(0.48) (0.53) (0.57) (0.65) (0.68) (0.54) (0.68) (0.62) (0.61)
Pre-communication
Item
Table 2 Mean (standard deviation) for student confidence in clinical skills delivery (n = 72).a,b,c
B. Deng et al.
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
d
6
3.11 2.96 2.99 2.74 3.22 3.15 3.14
(0.64) (0.66) (0.52) (0.60) (0.61) (0.66) (0.53)
Pre-communication 3.15 3.00 3.07 2.82 3.29 3.18 3.14
(0.67) (0.65) (0.56) (0.63) (0.64) (0.66) (0.68)
Pre-MWH lab 3.19 3.13 3.11 2.93 3.31 3.28 3.14
(0.59) (0.64) (0.58) (0.63) (0.63) (0.54) (0.59)
Post-MWH lab 3.18 3.01 3.06 2.93 3.32 3.35 3.24
(0.56) (0.62) (0.49) (0.63) (0.62) (0.57) (0.60)
Pre-PBA
MWH: men's and women's health; PBA: problem-based assessment; OTC: over-the-counter. a Scale: 1 = not at all confident; 2 = minimally confident; 3 = moderately confident; 4 = very confident; 5 = extremely confident. b Repeated ANOVA was performed to assess difference in student confidence throughout the same period. c Pair-wise comparison with Bonferronic Correction was performed to assess difference between two time points in the study period. d p < 0.05 for statistically difference between pre-communication and post-PBA. e p < 0.05 for statistically difference between pre-communication and post-MWH lab.
Counsel a patient on lifestyle modification Counsel a patient on OTC products Counsel a patient on prescription medicationsd Counsel a patient on devicesd,e Use teach-back method to assess patient understandingd Use open-ended questions to assess patient understanding Explain a follow-up plan to a patient
Item
Table 3 Mean (standard deviation) for student confidence in patient communication (n = 72).a,b,c
3.31 3.03 3.21 3.01 3.44 3.29 3.30
(0.60) (0.67) (0.61) (0.66) (0.61) (0.70) (0.62)
Post-PBA
1.000 0.832 1.000 1.000 1.000 0.069 1.000
p-value (Pre-MWH lab and post-MWH lab)
B. Deng et al.
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
7
primary literature (e.g., journal article) to answer medication questions electronic drug references (e.g., Micromedex) to answer medication questionsd,e evidence-based clinical guidelines (e.g., IDSA) to answer medication questions reliable online resources (e.g., CDC, FDA) to answer medication questions package insert to answer medication questionsd,f
d
2.92 (0.63) 3.39 (0.58) 2.92 (0.64) 3.06(0.66) 3.22 (0.69)
Pre-communication 2.97 3.42 2.96 3.00 3.28
(0.66) (0.58) (0.67) (0.64) (0.61)
Pre-MWH lab 3.06 3.56 3.06 3.13 3.43
(0.64) (0.51) (0.64) (0.67) (0.59)
Post-MWH lab 3.10 3.58 2.99 3.18 3.38
(0.64) (0.51) (0.64) (0.62) (0.55)
Pre-PBA
3.15 3.51 3.10 3.15 3.39
(0.71) (0.53) (0.68) (0.64) (0.65)
Post-PBA
1.000 0.114 1.000 0.488 0.153
p-value (Pre-MWH lab and Post MWH lab)
MWH: men's and women's health; PBA: problem-based assessment; IDSA: Infectious Diseases Society of America; CDC: Centers for Disease Control; FDA: Food and Drug Administration. a Scale: 1 = not at all confident; 2 = minimally confident; 3 = moderately confident; 4 = very confident; 5 = extremely confident. b Repeated ANOVA was performed to assess difference in student confidence throughout the same period. c Pair-wise comparison with Bonferronic Correction was performed to assess difference between two time points in the study period. d p < 0.05 for statistically difference between pre-communication and post-PBA. e p < 0.05 for statistically difference between pre-MWH lab and post-PBA. f p < 0.05 for statistically difference between pre-communication and post-MWH lab.
Use Use Use Use Use
Item
Table 4 Mean (standard deviation) for student confidence in drug-related problem resolution (n = 72).a,b,c
B. Deng et al.
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
B. Deng et al.
Table 5 Student confidence and predicted grades on PBA (n = 72).a Questions
Pearson correlation coefficient
p-value
I feel confident going into PBAb I feel confident that I will perform well on PBAb I feel anxious when I am taking a PBA PBA is a stressful experience The TOSCE prepared me for the PBAb
0.197 0.353 −0.072 −0.117 0.177
0.029 <0.001 0.424 0.193 0.048
PBA: problem-based assessment; TOSCE: teaching objective structured clinical examination. a Two-tailed Pearson test was performed to establish correlation between student confidence and their predicted grades on PBA. b p < 0.05 for positive relationship between student confidence and their predicted grades on PBA.
the MWH laboratory was beneficial and favorably contributed to their performance on the PBA (p = 0.048). Students’ confidence in preparing for and taking the PBA was positively reflected on their actual performance on the PBA, where the average score was 82% (high = 100%, low = 70%, standard deviation ± 8.17%) and 59.3% of students achieved a grade of ≥80% on the PBA. Discussion TOSCE and student confidence in summative assessment To the authors’ knowledge, this is the first prospective study describing TOSCE implementation and changes in student confidence over multiple laboratory sessions. The goal of a TOSCE is to provide students with formative feedback that addresses both their strengths and weaknesses as well as prepare students for summative assessments, such as OSCEs.14 This study was structured similarly to a standard OSCE format, with an exception of the length of each TOSCE session (three minutes feedback) and personnel composition (pharmacy faculty, residents, and fellows). Students indicated their confidence improved after TOSCE implementation, although the improvements were not statistically significant. One potential explanation was that students already rated themselves as moderately confident prior to TOSCE implementation in the MWH laboratory. The incremental improvement (3.2 on a 5-point scale) in the post-MWH survey indicated that a TOSCE was effective in developing their confidence in core competency areas but was not sufficient to improve their confidence to very confident (4 on a 5-point scale) with only one TOSCE session. The results from this study support those in the literature that a TOSCE has the potential to enhance student confidence and their readiness for summative assessments,15 even though a sufficient amount of TOSCE experiences are needed to see significant change in student's confidence level (such as advancing from moderately confident to very confident). In our study, student confidence increased immediately after completing the TOSCE. However, confidence was not sustained for the duration of the semester as student confidence decreased prior to the PBA for most survey items. One possible explanation was the gap of time between the MWH laboratory and the PBA. Since there is a three week gap between the MWH laboratory and the PBA, the confidence gained from the TOSCE experience may not have carried over to the PBA. Thus, increasing the number of TOSCEs during this time period may be necessary to effectively sustain student confidence over the course of the semester. PBA is a summative assessment that is needed to objectively measure learners’ performance in multiple competency areas. In this study, students described their confidence as moderately confident (3.3 on a 5-point Likert scale) after taking the PBA, which was achieved in a highly stressful summative assessment environment. The confidence in this study was comparable to confidence (3 or 4 on a 5-point Likert scale) achieved in a less stressful problem-based learning assessment.12 This result suggests that summative assessments, such as the PBA, are valuable tools that can achieve similar outcomes in student confidence as those seen in formative assessments. TOSCE and student performance in summative assessment Numerous studies have described the relationships between formative assessments and student grades on OSCE, although very few studies address the relationship between student confidence and their performance on summative assessments. In our study, students felt confident that they would perform well on the PBAs. Students also agreed that the feedback they received from the TOSCE prepared them for the PBA. Similar results were identified in a study conducted in a large medical school in England, where students reported that formative assessment was a good preparation for summative assessment.13 In particular, 46.6% and 85% of second-year medical students passed their formative and summative OSCEs respectively.13 In our study, 100% of students passed their PBAs and 59.3% of them achieved a grade of 80% or higher, which positively correlated with their confidence on predicting their performance on the PBA. The results from this study support previous findings that formative assessments (TOSCEs) have an important role in building student confidence and preparing them for high-stake summative assessments, such as PBA and OSCE.13 Limitations This study has several limitations. First, there was no comparator group. Since this was a pilot study and all participants in this study were third-year pharmacy students, the true degree of confidence change is unclear. In future studies, students from first and 8
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
B. Deng et al.
second didactic years will also be included so the study can further evaluate if the same degree of confidence change can be replicated in students with different didactic backgrounds. Although 134 students completed at least one survey, only 72 students (54%) completed all five surveys administered throughout the study period, despite it being a requirement for the laboratory course. This may be due to the small point value allocated for this assignment. Since it only accounted for a very small percentage of student grades, some students may have chosen to accept the consequence of losing these points instead of completing all five surveys. In future studies, increasing incentives for completing all five surveys may improve survey completion rate and expand overall sample size. Another reason that may explain the low survey response rate is survey fatigue, given that five surveys were administered in a semester. In particular, multiple back-to-back surveys were given in a relatively short period of time, which may negatively attribute to low survey response rate in this study.20 In future studies, decreasing the number of surveys administered in a short amount of time or decreasing back-to-back surveys may increase survey response rate. Another challenge encountered over the course of this study was students’ prior experience with high-stakes summative assessments. Since these students may already have had unpleasant feelings with summative assessments and developed mechanisms to cope with these feelings, it may have diminished the benefits of TOSCEs on student confidence with high-stakes summative assessments. One final challenge encountered in this study was the content difference between the TOSCE and the PBA. The TOSCE was implemented in the MWH laboratory, which primarily focused on male and female related disease states (e.g., benign prostatic hypertrophy, menopause, contraception). In contrast, common primary care chronic disease states (e.g., diabetes, hypertension, hyperlipidemia) were evaluated in the PBA. Due to the content difference, student confidence in the MWH laboratory may not affect students’ performance in the PBA. However, some of the skills emphasized in the MWH laboratory, such as clinical skills delivery, interpersonal communication, and drug-related resolution skills, are similar to and transferrable across different laboratories, which may minimize the difference in student confidence between these laboratory experiences. Lastly, social desirability bias may not be completely ruled out since study results were based on student self-reported responses. To limit the potential impact of social desirability bias, student responses were not provided to the study team until grades were posted at the end of the semester. In addition, students were graded on completion of all five surveys instead of the content of students’ responses. Conclusion A TOSCE is a formative assessment that has the potential to improve student confidence in core competency areas. In our study, students indicated that they were moderately confident in core competency areas after TOSCE implementation, which was higher than pre-TOSCE confidence in most cases. Additional studies are needed to assess long-term sustainability across multiple laboratory activities over the semester and throughout the curriculum. Implementing multiple TOSCEs over the course of a semester or shortening the time interval between TOSCE activities and PBAs may improve student confidence prior to taking high-stake summative assessments. Although students report increased stress when taking summative assessments, a combination of summative and formative assessments may enhance students’ confidence throughout their didactic and experiential education. Conflict of interest None. Disclosure(s) None. Appendix 1. Study administration timeline TOSCE = teaching objective structured clinical examination.
9
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
B. Deng et al.
Supplementary materials Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.cptl.2018.11.009. References 1. Salih MRM, Baidi Bahari M, Syed Sulaimam SA, Sarriff A, Yaseen SN, Oktavia Sari Y. Pharmacy student perceptions and feedback on the modified objective structured clinical examination. Pharm Educ. 2010;10(2):165–172. 2. Austin Z, O'Byrne C, Pugsley J, Munoz LQ. Development and validation processes for an objective structured clinical examination (OSCE) for entry-to-practice certification in pharmacy: the Canadian experience. Am J Pharm Educ. 2003;67(3) https://doi.org/10.5688/aj7005119. 3. Bond R, Cone C. Improving student confidence in skill performance in a pharmaceutical care lab setting. Pharm Educ. 2012;12(1):20–24. 4. Accreditation Standards and Key Elements for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree (“Standards 2016”). Accreditation Council for Pharmacy Education. Published February 2015. https://www.acpe-accredit.org/pdf/Standards2016FINAL.pdf. Accessed 6 November 2018. 5. Bandura A. Perceived self-efficacy in cognitive development and functioning. Educ Psychol. 1993;28(2):117–148. 6. Urteaga EM, Attridge RL, Tovar JM, Witte AP. Evaluation of clinical and communication skills of pharmacy students and pharmacists with an objective structured clinical examination. Am J Pharm Educ. 2015;79(8) https://doi.org/10.5688/ajpe798122. 7. Thomas MC, Macias-Moriarity LZ. Student knowledge and confidence in an elective clinical toxicology course using active-learning techniques. Am J Pharm Educ. 2014;78(5) https://doi.org/10.5688/ajpe78595. 8. Sturpe DA. Objective structured clinical examinations in doctor of pharmacy programs in the United States. Am J Pharm Educ. 2010;74(8) https://doi.org/10. 5688/aj7408148. 9. Shirwaikar A. Objective structured clinical examination (OSCE) in pharmacy education - a trend. Pharm Pract (Granada). 2015;13(4) https://doi.org/10.18549/ PharmPract.2015.04.627. 10. Sloan DA, Donnelly MB, Schwartz RW, Strodel WE. The objective structured clinical examination. The new gold standard for evaluating postgraduate clinical performance. Ann Surg. 1995;222(6):735–742. 11. Allen R, Heard J, Savidge M, Bittergle J, Cantrell M, Huffmaster T. Surveying students’ attitudes during the OSCE. Adv Heal Sci Educ. 1998;3(3):197–206. 12. Sherman JJ, Riche DM, Stover KR. Physical assessment experience in a problem-based learning course. Am J Pharm Educ. 2011;75(8) https://doi.org/10.5688/ ajpe758156. 13. Chisnall B, Vince T, Hall S, Tribe R. Evaluation of outcomes of a formative objective structured clinical examination for second-year UK medical students. Int J Med Educ. 2015;6:76–83. 14. Brazeau C, Boyd L, Crosson J. Changing an existing OSCE to a teaching tool: the making of a TOSCE. Acad Med. 2002;77(9):932. 15. Gums TH, Kleppinger EL, Urick BY. Outcomes of individualized formative assessments in a pharmacy skills laboratory. Am J Pharm Educ. 2014;78(9) https://doi. org/10.5688/ajpe789166. 16. PharmD Curriculum. Purdue College of Pharmacy. Published 2017. https://www.pharmacy.purdue.edu/current-students/resources/curriculum. Accessed 6 November 2018. 17. Detailed Course Information. Purdue University. https://selfservice.mypurdue.purdue.edu/prod/bzwsrch.p_catalog_detail?term=current&subject=PHRM& cnbr=86100. Accessed 6 November 2018. 18. Qualtrics [comupter program]. Version 09.16. Provo, Utah: Qualtrics; 2017. 19. IBM SPSS Statistics (for Windows) [computer program]. Version 23.0. Armonk, NY: IBM Corporation; 2017. 20. Porter S, Whitcomb M, Weitzer W. Multiple surveys of students and survey fatigue. New Dir Inst Res. 2004;2004(121):63–73. https://doi.org/10.1002/ir.101.
10