- Email: [email protected]
Efficacy of formative evaluation using a focus group for a large classroom setting in an accelerated pharmacy program
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Contents lists available at ScienceDirect
Currents in Pharmacy Teaching and Learning journal homepage: www.elsevier.com/locate/cptl
Experiences in Teaching and Learning
Efficacy of formative evaluation using a focus group for a large classroom setting in an accelerated pharmacy program Shaun Nolette, Alyssa Nguyen, David Kogan, Catherine Oswald, Alana Whittaker, ⁎ Arup Chakraborty Roseman University of Health Sciences, College of Pharmacy, 11 Sunset Way, Henderson, NV 89014, United States
AR TI CLE I NF O
AB S T R A CT
Keywords: Communication Focus group Formative evaluation Large classroom Time management
Background and purpose: Formative evaluation is a process utilized to improve communication between students and faculty. This evaluation method allows the ability to address pertinent issues in a timely manner; however, implementation of formative evaluation can be a challenge, especially in a large classroom setting. Using mediated formative evaluation, the purpose of this study is to determine if a student based focus group is a viable option to improve efficacy of communication between an instructor and students as well as time management in a large classroom setting. Educational activity and setting: Out of 140 total students, six students were selected to form a focus group – one from each of six total sections of the classroom. Each focus group representative was responsible for collecting all the questions from students of their corresponding sections and submitting them to the instructor two to three times a day. Responses from the instructor were either passed back to pertinent students by the focus group representatives or addressed directly with students by the instructor. This study was conducted using a fifteenquestion survey after the focus group model was utilized for one month. A printed copy of the survey was distributed in the class by student investigators. Questions were of varying types, including Likert scale, yes/no, and open-ended response. Findings: One hundred forty surveys were administered, and 90 complete responses were collected. Surveys showed that 93.3% of students found that use of the focus group made them more likely to ask questions for understanding. The surveys also showed 95.5% of students found utilizing the focus group for questions allowed for better understanding of difficult concepts. General open-ended answer portions of the survey showed that most students found the focus group allowed them to ask questions more easily since they did not feel intimidated by asking in front of the whole class. No correlation was found between demographic characteristics and survey responses. This may demonstrate that students in similar large classroom settings may respond in a similar fashion. Discussion and summary: An overwhelmingly positive result suggests focus groups may be a valuable addition to communication routes currently employed in classrooms. The focus group offers an alternate pathway to improve student understanding in larger lecture classroom environments and may encourage more students to seek answers to questions in a timely manner without breaking classroom flow.
⁎
Corresponding author. E-mail addresses: [email protected] (S. Nolette), [email protected] (A. Nguyen), [email protected] (D. Kogan), [email protected] (C. Oswald), [email protected] (A. Whittaker), [email protected] (A. Chakraborty). http://dx.doi.org/10.1016/j.cptl.2017.03.004
1877-1297/ © 2017 Elsevier Inc. All rights reserved.
Please cite this article as: Nolette, S., Currents in Pharmacy Teaching and Learning (2017), http://dx.doi.org/10.1016/j.cptl.2017.03.004
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
S. Nolette et al.
Background and purpose Young and Shaw1 proposed six major dimensions of effective teaching: subject value, student motivation, comfortable learning atmosphere, subject organization, effective communication, and concern for student learning. Effective communication is the key to determining the success of the other five dimensions. Studies describe that effective communication can be better achieved through formative evaluation rather than summative evaluation.2–4 The goal of formative evaluation is to monitor student learning in real time to provide ongoing feedback that can be used by instructors to improve and fine tune their teaching environment. Formative evaluations can identify gaps between what students understand and what the instructor expects them to know, or the expected learning objectives for a course. With frequent targeted feedback, the instructor can offer feedback to help students close these gaps prior to the assessment that may promote metacognition and possibly reduce anxiety. In this study, we employed a student-based focus group model as a formative teaching evaluation tool and measured its effectiveness. Formative teaching evaluation in a large classroom is often challenging, because there is not a one-size-fits-all approach to collect and address the feedback from all students in a timely manner. Roseman University of Health Sciences, College of Pharmacy (COP) utilizes a “block” system of curricular design, which provides students with the opportunity to study only one content area at a time. The program does not award students traditional letter grades and uses a pass and fail system. It is an accelerated (three-year) doctor of pharmacy (PharmD) program with a high-stakes 90% passing rate requirement for all courses. Current communication methods in the college of pharmacy include: students asking questions before, during and after class; e-mailing, calling or texting an instructor; and visiting an instructor during office hours. However, it is difficult to determine if any of these methods can be used effectively to implement formative evaluation within a large classroom setting.5 The term “focus group” is defined as “a demographically diverse group of people assembled to participate in a guided discussion about a particular product before it is launched, or to provide ongoing feedback on a political campaign, television series, etc”.6 The concept of focus group in a classroom setting is not new.7–9 In 2002, the University of Kentucky College of Pharmacy (COP) created a Student Liaison Committee (SLC), a structured program of student focus groups to improve communication in the classroom. Their study concluded that a focus group was “…an effective way of providing feedback to faculty members and students about the classroom learning environment”. They stated: “Student opinions about the effectiveness of the focus groups were fairly positive, with 59–87% agreeing that the process allowed students to communicate effectively with faculty members”.10 Although their focus group (SLC) collected student feedback every day, SLC discussions and feedback were only addressed after each set of block examinations. This approach delayed responses to students and limited response periods to a total of three sessions per semester. Our study was designed differently and sought to provide feedback channeled through focus groups back to students on a daily basis, thus including the focus group as part of a formative evaluation process for the instructor. In this study we evaluated whether a focus group can be used for formative evaluation, and if this form of interaction improves barriers to effective communication within a large classroom setting. Educational activity and setting This study was conducted during the first professional year of pharmacy school in a biochemistry course. In the Roseman University of Health Sciences, College of Pharmacy (COP) block-based model, students attend six hours of didactic teaching and active learning daily. The instructor running this course has done so for seven years and was the sole faculty member teaching daily for fifteen days of the block. The instructor was available for one additional hour each lecture day to address student questions oneon-one or in a small group independent of the focus group. The pharmacy classroom in this study is composed of 140 students. Seats are divided into six wedges, 25–40 students per wedge, making a circular classroom design where the instructor is in the middle. This survey-based study utilized a focus group consisting of one student from each wedge for a total of six students. The focus group size was limited to ensure all members were able to voice feedback to the instructor in a timely manner. Three students within three specific wedges were elected by their classmates, and three students from the three remaining wedges were selected by the instructor. These student representatives comprised the focus group and were tasked to receive questions, comments, and concerns provided by other students within their wedge during hourly ten-minute breaks and relay them to the instructor throughout the six-hour teaching day during break times. The focus group was also asked to approach fellow students proactively to receive feedback. The instructor was updated with any concerns and/or questions from students two or three times daily. Focus group representatives corresponded with the instructor routinely in person, and less frequently via e-mail. All responses to focus group questions were conveyed either individually to student(s) outside of classroom time, or the instructor addressed the issue with the whole classroom depending on the nature of the concern. Also, the teacher often prepared additional reading or practice materials based on focus group feedback and made these available to students via e-mail. After a period of 15 six-hour days, wherein the focus group typically met two to three times daily with the instructor, an Institutional Review Board (IRB) approved survey with informed consent was conducted. The 15-question survey collected demographic data and assessed students’ comfort level asking questions based on material presented in class. Using Likert scale, yes/no, and open-ended responses, students rated their comfort level asking questions in a variety of ways and provided insight as to what made them feel comfortable to do so. Mann-Whitney U and Kruskal-Wallis tests were performed to determine if demographic variables impacted the survey answers given. To objectively determine success of the focus group in regard to student learning, the same ten course learning outcomes were assessed over a three-year period using 30 multiple choice exam questions with similar levels of difficulty (10 questions per year). Group 1 (batch 2010) did not utilize the focus group method, while groups 2 (batch 2011) and 3 (batch 2012) did. The percentage of correctly answered exam questions per 140-student cohort was evaluated using Student's t2
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
S. Nolette et al.
50
First choice Second choice Third choice
40
%
30 20 10 0 In class
After class
e-mail
Focus group
Fig. 1. Preferred methods of communications by students based on a survey conducted in 2014. Only first, second and third options are shown in the figure.
test. Similarly, summative teaching evaluations of questions related to communication and classroom management were compared between the same years. The survey results were analyzed using descriptive statistics with SPSSv20. In all analyses, a p-value of ≤0.05 was accepted as statistically significant. Findings In 2014, 140 students in the class received the survey and 90 students completed the survey (64% response rate). Of those students who completed the survey, 32% were male and 68% were female with 81% of students 18–30 years of age. In terms of the highest degree obtained, 13% of responders achieved high school diplomas, 17% achieved an associate's degree, 66% achieved a bachelor's degree, and 4% achieved a master's degree. In general, the students’ perception of the focus group efficacy was not affected by their gender, age, or level of education. Students were given options to ask questions during lecture, via e-mail, face-to-face at the end of class, through open office hours, and through focus groups. The top three preferred methods of communication by students were asking questions directly in class or after class (70%), through e-mail (8.4%), or with the focus group (7%) (Fig. 1). Although a majority of the student responders (63.7%) asked questions during or after class one or more times over the course of the 15 days, around 44% of the students used the focus group to communicate with the instructor, followed by e-mail (33%) during this time period (Fig. 2). The overall positive student perception of the focus group was stronger than initial expectations. A large majority of students (> 80%) thought using focus groups to channel their questions allowed these questions to be answered adequately by the instructor and helped them to understand difficult concepts. The majority of the students (60%) believed that focus groups should be utilized by other faculty (Fig. 3). Student test question scores increased each year the focus group method was employed when compared to the non-focus group year (increases of 5.7% in batch 2011 and 6.9% in batch 2012 as compared to batch 2010). Over this three-year span, scores increased from an average of 88.6% correctly answered in 2010 to 94.3% (p=0.065) in 2011 and 95.5% (p=0.039) in 2012 (Fig. 4). Summative teaching evaluations were assessed for improvements in two areas. Two questions were selected from the summative evaluation tool assessing communication and classroom management: 1) The instructor effectively communicated his/her subject matter to the students and 2) The instructor used class time well. Two other questions were selected for contrast: 3) The instructor was concerned with whether or not the students learned the material and 4) The instructor made me feel free to ask questions. These questions were answered using a five-point Likert scale. Questions 3 and 4 were selected because they addressed whether the instructor was sincere enough to communicate effectively with the students. There was an improvement observed in the summative teaching evaluation scores for questions 1 and 2 (Fig. 5). Before the use of focus groups (2010), the average score for question 1 was 3.16 (median 3), which improved to an average of 4.16 (median 4) in 2011 and an average of 4.49 (median 5) in 2012. Similarly, the average score for question 2 was 3.93 and median was 4 in 2010, which improved to average 4.36 and median 4 in 2011, and an average of 4.6 and median 5 in 2012. While the scores for questions 3 and 4 remained consistent, the focus group-mediated formative 70
63.7%
60
%
50
43.9%
40
33.1%
30 20 10 0 Questions asked:
Classroom
Focus group
e-mail
Fig. 2. Avenues by which students asked questions in the classroom based on a survey conducted in 2014.
3
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
S. Nolette et al.
Fig. 3. Students’ perception of utilization of focus group in formative evaluation based on a survey conducted in 2014. Four questions were asked in the survey to assess the perception.
feedback improved classroom communication as well as classroom time management to a greater extent than demonstrating instructor's concern for learning and encouraging freedom to ask questions. Out of 90 students that completed the survey, 69 (76.6%) students answered an optional open-ended question regarding what deterred them from asking questions during class. A sizable portion (34.6%) of responders felt that asking questions during class disrupts the flow of teaching, while 17.3% identified themselves as too shy or embarrassed. Notably, 3% of respondents indicated that they did not feel adequately prepared to ask a question. Discussion The Roseman University of Health Sciences, College of Pharmacy (COP) is a three-year accelerated program, whereas the majority of the other pharmacy programs in the United States are four-year programs.11 To circumvent the time limitation with this accelerated approach, a block system was adopted in which students learn one subject at a time, for six hours per day, five days per week; they are assessed every alternate week. Effective communication between students and teachers is a key element in establishing successful learning. Time constraints and number of students in the classroom make effective learning challenging, particularly in a fast-paced environment. Modern communication methods like e-mail are useful, but they have limitations. E-mail messages can be written or accessed after class, but replying in a timely manner can be a challenge if the class size is large and if there are multiple questions to address. Audience response devices or platforms can be used to ask questions in real time, but they do not necessarily reveal when students are struggling with certain concepts. An alternative form of interaction is to communicate directly with the instructor during and/or after class for clarification. In this regard, having a large class size with a heterogeneous population can make addressing each question difficult and time consuming. When time is limited in the classroom and there is a large number of students, it is difficult for one instructor to address all questions during that allotted classroom time. Other limiting factors include a desire to not disrupt the flow of class and student timidity to pose questions in front of the whole class. After a long day of class, students might forget the exact question that they had in mind or are too mentally exhausted to continue additional discussion. This study shows that focus group-mediated formative evaluation may be a complementary approach to the existing communication methods and is supported with both subjective and objective student data. The focus group contained a blend of students representing
% of correct answer
100 80 60 40 20 0 2010
2011
2012
Fig. 4. Enhancement of students’ performance in the assessment before and after implementation of focus group. * indicated for statistical significance.
4
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
S. Nolette et al.
Fig. 5. Improvement in the summative teaching evaluation before and after implementation of focus group.
the heterogeneous nature of the classroom. Because students tend to select the most vocal person in a group, three of the representatives were elected by students and the teacher selected the other three with the intent to create a diverse student demographic within the focus group. We assessed the measurable benefits of formative evaluation using a focus group model. We found a significant improvement in year-over-year testing data from student performance (Fig. 4). We also compared the year-over-year summative teaching evaluations of the instructor. Interestingly, question scores pertaining to effective communication and classroom management were improved after the introduction of a focus group. These improvements were markedly higher than improvements noted in the instructor's augmented concern for student learning or encouragement to ask more questions (Fig. 5). Summary Based on the findings of this study, the focus group model appears to be a useful method for communication between instructors and students and may aid in effective classroom time management, particularly in large classroom settings. Instructors may consider this communication method in addition to other types of communication as a viable option to collect questions and provide formative feedback in a timely fashion. Students may also learn better when this communication style is employed; however, additional studies across multiple types of classrooms and subjects may help to confirm these results. Conflict of interest None declared. Source of support None. Acknowledgement The authors thank Kristi Singer for critically reviewing the manuscript. Appendix A. Supporting information Supplementary data associated with this article can be found in the online version at http://dx.doi.org/10.1016/j.cptl.2017.03. 004. References 1. 2. 3. 4. 5. 6.
Young S, Shaw DG. Profiles of effective college and university teachers. J High Educ. 1999;70(6):670–686. Bloom BS, Hastings JT, Madaus GF. Handbook of Summative and Formative Evaluation of Student Learning. New York, NY: McGraw-Hill; 1971. Fowlera P, Wilford B. Formative feedback in the clinical practice setting: what are the perceptions of student radiographers? Radiography. 2016;22(1):e16–e24. Aasen A. A teacher's guide to giving effective feedback. J Best Teach Pract. 2015;2(2):11–12. Flagg BN. Formative Evaluation for Educational Technologies. New York, NY: Routledge; 2013. Powell RA, Single HM. Focus groups. Int J Qual Health Care. 1996;8(5):499–504.
5
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
S. Nolette et al. 7. 8. 9. 10. 11.
Stewart DW, Shamdasani PN. Focus Groups: Theory and Practice. 3rd ed. Thousand Oaks, CA: Sage Publications; 2015. Krueger RA. Focus Groups: A Practical Guide for Applied Research. Singapore: Sage Publications; 1994. Kevern J, Webb C. Focus groups as a tool for critical social research in nurse education. Nurse Educ Today. 2001;21(4):323–333. Bhavsar V, Bird E, Anderson H. Pharmacy student focus groups for formative evaluation of the learning environment. Am J Pharm Educ. 2007;71(2):Article 22. American Association of Colleges of Pharmacy. Pharmacy school admission requirements. Table 1: PharmD programs; Updated 2015. Available at: 〈http://www. aacp.org/resources/student/pharmacyforyou/admissions/admissionrequirements/Documents/Table%201.pdf〉. Accessed May 30, 2017.
6
Contents lists available at ScienceDirect
Currents in Pharmacy Teaching and Learning journal homepage: www.elsevier.com/locate/cptl
Experiences in Teaching and Learning
Efficacy of formative evaluation using a focus group for a large classroom setting in an accelerated pharmacy program Shaun Nolette, Alyssa Nguyen, David Kogan, Catherine Oswald, Alana Whittaker, ⁎ Arup Chakraborty Roseman University of Health Sciences, College of Pharmacy, 11 Sunset Way, Henderson, NV 89014, United States
AR TI CLE I NF O
AB S T R A CT
Keywords: Communication Focus group Formative evaluation Large classroom Time management
Background and purpose: Formative evaluation is a process utilized to improve communication between students and faculty. This evaluation method allows the ability to address pertinent issues in a timely manner; however, implementation of formative evaluation can be a challenge, especially in a large classroom setting. Using mediated formative evaluation, the purpose of this study is to determine if a student based focus group is a viable option to improve efficacy of communication between an instructor and students as well as time management in a large classroom setting. Educational activity and setting: Out of 140 total students, six students were selected to form a focus group – one from each of six total sections of the classroom. Each focus group representative was responsible for collecting all the questions from students of their corresponding sections and submitting them to the instructor two to three times a day. Responses from the instructor were either passed back to pertinent students by the focus group representatives or addressed directly with students by the instructor. This study was conducted using a fifteenquestion survey after the focus group model was utilized for one month. A printed copy of the survey was distributed in the class by student investigators. Questions were of varying types, including Likert scale, yes/no, and open-ended response. Findings: One hundred forty surveys were administered, and 90 complete responses were collected. Surveys showed that 93.3% of students found that use of the focus group made them more likely to ask questions for understanding. The surveys also showed 95.5% of students found utilizing the focus group for questions allowed for better understanding of difficult concepts. General open-ended answer portions of the survey showed that most students found the focus group allowed them to ask questions more easily since they did not feel intimidated by asking in front of the whole class. No correlation was found between demographic characteristics and survey responses. This may demonstrate that students in similar large classroom settings may respond in a similar fashion. Discussion and summary: An overwhelmingly positive result suggests focus groups may be a valuable addition to communication routes currently employed in classrooms. The focus group offers an alternate pathway to improve student understanding in larger lecture classroom environments and may encourage more students to seek answers to questions in a timely manner without breaking classroom flow.
⁎
Corresponding author. E-mail addresses: [email protected] (S. Nolette), [email protected] (A. Nguyen), [email protected] (D. Kogan), [email protected] (C. Oswald), [email protected] (A. Whittaker), [email protected] (A. Chakraborty). http://dx.doi.org/10.1016/j.cptl.2017.03.004
1877-1297/ © 2017 Elsevier Inc. All rights reserved.
Please cite this article as: Nolette, S., Currents in Pharmacy Teaching and Learning (2017), http://dx.doi.org/10.1016/j.cptl.2017.03.004
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
S. Nolette et al.
Background and purpose Young and Shaw1 proposed six major dimensions of effective teaching: subject value, student motivation, comfortable learning atmosphere, subject organization, effective communication, and concern for student learning. Effective communication is the key to determining the success of the other five dimensions. Studies describe that effective communication can be better achieved through formative evaluation rather than summative evaluation.2–4 The goal of formative evaluation is to monitor student learning in real time to provide ongoing feedback that can be used by instructors to improve and fine tune their teaching environment. Formative evaluations can identify gaps between what students understand and what the instructor expects them to know, or the expected learning objectives for a course. With frequent targeted feedback, the instructor can offer feedback to help students close these gaps prior to the assessment that may promote metacognition and possibly reduce anxiety. In this study, we employed a student-based focus group model as a formative teaching evaluation tool and measured its effectiveness. Formative teaching evaluation in a large classroom is often challenging, because there is not a one-size-fits-all approach to collect and address the feedback from all students in a timely manner. Roseman University of Health Sciences, College of Pharmacy (COP) utilizes a “block” system of curricular design, which provides students with the opportunity to study only one content area at a time. The program does not award students traditional letter grades and uses a pass and fail system. It is an accelerated (three-year) doctor of pharmacy (PharmD) program with a high-stakes 90% passing rate requirement for all courses. Current communication methods in the college of pharmacy include: students asking questions before, during and after class; e-mailing, calling or texting an instructor; and visiting an instructor during office hours. However, it is difficult to determine if any of these methods can be used effectively to implement formative evaluation within a large classroom setting.5 The term “focus group” is defined as “a demographically diverse group of people assembled to participate in a guided discussion about a particular product before it is launched, or to provide ongoing feedback on a political campaign, television series, etc”.6 The concept of focus group in a classroom setting is not new.7–9 In 2002, the University of Kentucky College of Pharmacy (COP) created a Student Liaison Committee (SLC), a structured program of student focus groups to improve communication in the classroom. Their study concluded that a focus group was “…an effective way of providing feedback to faculty members and students about the classroom learning environment”. They stated: “Student opinions about the effectiveness of the focus groups were fairly positive, with 59–87% agreeing that the process allowed students to communicate effectively with faculty members”.10 Although their focus group (SLC) collected student feedback every day, SLC discussions and feedback were only addressed after each set of block examinations. This approach delayed responses to students and limited response periods to a total of three sessions per semester. Our study was designed differently and sought to provide feedback channeled through focus groups back to students on a daily basis, thus including the focus group as part of a formative evaluation process for the instructor. In this study we evaluated whether a focus group can be used for formative evaluation, and if this form of interaction improves barriers to effective communication within a large classroom setting. Educational activity and setting This study was conducted during the first professional year of pharmacy school in a biochemistry course. In the Roseman University of Health Sciences, College of Pharmacy (COP) block-based model, students attend six hours of didactic teaching and active learning daily. The instructor running this course has done so for seven years and was the sole faculty member teaching daily for fifteen days of the block. The instructor was available for one additional hour each lecture day to address student questions oneon-one or in a small group independent of the focus group. The pharmacy classroom in this study is composed of 140 students. Seats are divided into six wedges, 25–40 students per wedge, making a circular classroom design where the instructor is in the middle. This survey-based study utilized a focus group consisting of one student from each wedge for a total of six students. The focus group size was limited to ensure all members were able to voice feedback to the instructor in a timely manner. Three students within three specific wedges were elected by their classmates, and three students from the three remaining wedges were selected by the instructor. These student representatives comprised the focus group and were tasked to receive questions, comments, and concerns provided by other students within their wedge during hourly ten-minute breaks and relay them to the instructor throughout the six-hour teaching day during break times. The focus group was also asked to approach fellow students proactively to receive feedback. The instructor was updated with any concerns and/or questions from students two or three times daily. Focus group representatives corresponded with the instructor routinely in person, and less frequently via e-mail. All responses to focus group questions were conveyed either individually to student(s) outside of classroom time, or the instructor addressed the issue with the whole classroom depending on the nature of the concern. Also, the teacher often prepared additional reading or practice materials based on focus group feedback and made these available to students via e-mail. After a period of 15 six-hour days, wherein the focus group typically met two to three times daily with the instructor, an Institutional Review Board (IRB) approved survey with informed consent was conducted. The 15-question survey collected demographic data and assessed students’ comfort level asking questions based on material presented in class. Using Likert scale, yes/no, and open-ended responses, students rated their comfort level asking questions in a variety of ways and provided insight as to what made them feel comfortable to do so. Mann-Whitney U and Kruskal-Wallis tests were performed to determine if demographic variables impacted the survey answers given. To objectively determine success of the focus group in regard to student learning, the same ten course learning outcomes were assessed over a three-year period using 30 multiple choice exam questions with similar levels of difficulty (10 questions per year). Group 1 (batch 2010) did not utilize the focus group method, while groups 2 (batch 2011) and 3 (batch 2012) did. The percentage of correctly answered exam questions per 140-student cohort was evaluated using Student's t2
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
S. Nolette et al.
50
First choice Second choice Third choice
40
%
30 20 10 0 In class
After class
Focus group
Fig. 1. Preferred methods of communications by students based on a survey conducted in 2014. Only first, second and third options are shown in the figure.
test. Similarly, summative teaching evaluations of questions related to communication and classroom management were compared between the same years. The survey results were analyzed using descriptive statistics with SPSSv20. In all analyses, a p-value of ≤0.05 was accepted as statistically significant. Findings In 2014, 140 students in the class received the survey and 90 students completed the survey (64% response rate). Of those students who completed the survey, 32% were male and 68% were female with 81% of students 18–30 years of age. In terms of the highest degree obtained, 13% of responders achieved high school diplomas, 17% achieved an associate's degree, 66% achieved a bachelor's degree, and 4% achieved a master's degree. In general, the students’ perception of the focus group efficacy was not affected by their gender, age, or level of education. Students were given options to ask questions during lecture, via e-mail, face-to-face at the end of class, through open office hours, and through focus groups. The top three preferred methods of communication by students were asking questions directly in class or after class (70%), through e-mail (8.4%), or with the focus group (7%) (Fig. 1). Although a majority of the student responders (63.7%) asked questions during or after class one or more times over the course of the 15 days, around 44% of the students used the focus group to communicate with the instructor, followed by e-mail (33%) during this time period (Fig. 2). The overall positive student perception of the focus group was stronger than initial expectations. A large majority of students (> 80%) thought using focus groups to channel their questions allowed these questions to be answered adequately by the instructor and helped them to understand difficult concepts. The majority of the students (60%) believed that focus groups should be utilized by other faculty (Fig. 3). Student test question scores increased each year the focus group method was employed when compared to the non-focus group year (increases of 5.7% in batch 2011 and 6.9% in batch 2012 as compared to batch 2010). Over this three-year span, scores increased from an average of 88.6% correctly answered in 2010 to 94.3% (p=0.065) in 2011 and 95.5% (p=0.039) in 2012 (Fig. 4). Summative teaching evaluations were assessed for improvements in two areas. Two questions were selected from the summative evaluation tool assessing communication and classroom management: 1) The instructor effectively communicated his/her subject matter to the students and 2) The instructor used class time well. Two other questions were selected for contrast: 3) The instructor was concerned with whether or not the students learned the material and 4) The instructor made me feel free to ask questions. These questions were answered using a five-point Likert scale. Questions 3 and 4 were selected because they addressed whether the instructor was sincere enough to communicate effectively with the students. There was an improvement observed in the summative teaching evaluation scores for questions 1 and 2 (Fig. 5). Before the use of focus groups (2010), the average score for question 1 was 3.16 (median 3), which improved to an average of 4.16 (median 4) in 2011 and an average of 4.49 (median 5) in 2012. Similarly, the average score for question 2 was 3.93 and median was 4 in 2010, which improved to average 4.36 and median 4 in 2011, and an average of 4.6 and median 5 in 2012. While the scores for questions 3 and 4 remained consistent, the focus group-mediated formative 70
63.7%
60
%
50
43.9%
40
33.1%
30 20 10 0 Questions asked:
Classroom
Focus group
Fig. 2. Avenues by which students asked questions in the classroom based on a survey conducted in 2014.
3
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
S. Nolette et al.
Fig. 3. Students’ perception of utilization of focus group in formative evaluation based on a survey conducted in 2014. Four questions were asked in the survey to assess the perception.
feedback improved classroom communication as well as classroom time management to a greater extent than demonstrating instructor's concern for learning and encouraging freedom to ask questions. Out of 90 students that completed the survey, 69 (76.6%) students answered an optional open-ended question regarding what deterred them from asking questions during class. A sizable portion (34.6%) of responders felt that asking questions during class disrupts the flow of teaching, while 17.3% identified themselves as too shy or embarrassed. Notably, 3% of respondents indicated that they did not feel adequately prepared to ask a question. Discussion The Roseman University of Health Sciences, College of Pharmacy (COP) is a three-year accelerated program, whereas the majority of the other pharmacy programs in the United States are four-year programs.11 To circumvent the time limitation with this accelerated approach, a block system was adopted in which students learn one subject at a time, for six hours per day, five days per week; they are assessed every alternate week. Effective communication between students and teachers is a key element in establishing successful learning. Time constraints and number of students in the classroom make effective learning challenging, particularly in a fast-paced environment. Modern communication methods like e-mail are useful, but they have limitations. E-mail messages can be written or accessed after class, but replying in a timely manner can be a challenge if the class size is large and if there are multiple questions to address. Audience response devices or platforms can be used to ask questions in real time, but they do not necessarily reveal when students are struggling with certain concepts. An alternative form of interaction is to communicate directly with the instructor during and/or after class for clarification. In this regard, having a large class size with a heterogeneous population can make addressing each question difficult and time consuming. When time is limited in the classroom and there is a large number of students, it is difficult for one instructor to address all questions during that allotted classroom time. Other limiting factors include a desire to not disrupt the flow of class and student timidity to pose questions in front of the whole class. After a long day of class, students might forget the exact question that they had in mind or are too mentally exhausted to continue additional discussion. This study shows that focus group-mediated formative evaluation may be a complementary approach to the existing communication methods and is supported with both subjective and objective student data. The focus group contained a blend of students representing
% of correct answer
100 80 60 40 20 0 2010
2011
2012
Fig. 4. Enhancement of students’ performance in the assessment before and after implementation of focus group. * indicated for statistical significance.
4
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
S. Nolette et al.
Fig. 5. Improvement in the summative teaching evaluation before and after implementation of focus group.
the heterogeneous nature of the classroom. Because students tend to select the most vocal person in a group, three of the representatives were elected by students and the teacher selected the other three with the intent to create a diverse student demographic within the focus group. We assessed the measurable benefits of formative evaluation using a focus group model. We found a significant improvement in year-over-year testing data from student performance (Fig. 4). We also compared the year-over-year summative teaching evaluations of the instructor. Interestingly, question scores pertaining to effective communication and classroom management were improved after the introduction of a focus group. These improvements were markedly higher than improvements noted in the instructor's augmented concern for student learning or encouragement to ask more questions (Fig. 5). Summary Based on the findings of this study, the focus group model appears to be a useful method for communication between instructors and students and may aid in effective classroom time management, particularly in large classroom settings. Instructors may consider this communication method in addition to other types of communication as a viable option to collect questions and provide formative feedback in a timely fashion. Students may also learn better when this communication style is employed; however, additional studies across multiple types of classrooms and subjects may help to confirm these results. Conflict of interest None declared. Source of support None. Acknowledgement The authors thank Kristi Singer for critically reviewing the manuscript. Appendix A. Supporting information Supplementary data associated with this article can be found in the online version at http://dx.doi.org/10.1016/j.cptl.2017.03. 004. References 1. 2. 3. 4. 5. 6.
Young S, Shaw DG. Profiles of effective college and university teachers. J High Educ. 1999;70(6):670–686. Bloom BS, Hastings JT, Madaus GF. Handbook of Summative and Formative Evaluation of Student Learning. New York, NY: McGraw-Hill; 1971. Fowlera P, Wilford B. Formative feedback in the clinical practice setting: what are the perceptions of student radiographers? Radiography. 2016;22(1):e16–e24. Aasen A. A teacher's guide to giving effective feedback. J Best Teach Pract. 2015;2(2):11–12. Flagg BN. Formative Evaluation for Educational Technologies. New York, NY: Routledge; 2013. Powell RA, Single HM. Focus groups. Int J Qual Health Care. 1996;8(5):499–504.
5
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
S. Nolette et al. 7. 8. 9. 10. 11.
Stewart DW, Shamdasani PN. Focus Groups: Theory and Practice. 3rd ed. Thousand Oaks, CA: Sage Publications; 2015. Krueger RA. Focus Groups: A Practical Guide for Applied Research. Singapore: Sage Publications; 1994. Kevern J, Webb C. Focus groups as a tool for critical social research in nurse education. Nurse Educ Today. 2001;21(4):323–333. Bhavsar V, Bird E, Anderson H. Pharmacy student focus groups for formative evaluation of the learning environment. Am J Pharm Educ. 2007;71(2):Article 22. American Association of Colleges of Pharmacy. Pharmacy school admission requirements. Table 1: PharmD programs; Updated 2015. Available at: 〈http://www. aacp.org/resources/student/pharmacyforyou/admissions/admissionrequirements/Documents/Table%201.pdf〉. Accessed May 30, 2017.
6