- Email: [email protected]
Assessing student perceived outcomes for a Methods Television (MTV) video project in a research methodology class
Available online at www.sciencedirect.com
Currents in Pharmacy Teaching and Learning 2 (2010) 211–217 http://www.pharmacyteaching.com
Assessing student perceived outcomes for a Methods Television (MTV) video project in a research methodology class Terri L. Warholak, PhDa, Adrienne M. Gilligan, BSa,*, Jason T. Hurwitz, PhDb, Marion Slack, PhDa, Christina A. Spivey, PhDa, Marie Chisholm-Burns, PharmD, MPH, FCCP, FASHPa b
a Department of Pharmacy Practice and Science, The University of Arizona College of Pharmacy, Tucson, AZ Center for Health Outcomes & PharmacoEconomic Research, The University of Arizona College of Pharmacy, Tucson, AZ
Abstract Objectives: To (1) assess the outcomes of making a Methods Television (MTV) video on student perceptions, and (2) evaluate how students learn best in a research methods class. Methods: Pharmacy students enrolled in a research methods class (n ⫽ 93) were given the option to create an educational MTV video. Teams (n ⫽ 15) were randomly assigned a research methodology topic and three objectives for the video. All students in attendance on the last day (n ⫽ 88) completed an attitudinal questionnaire designed to assess students’ perceptions regarding the class and MTV video projects. Results: Students indicated that multiple activities helped them learn the most in class (i.e., in-class assignments, homework); 51% stated that videos helped improve their knowledge of research methods and 52% indicated it was entertaining. Conclusion: More than half of the students indicated that the MTV videos helped with their understanding of research methods, an indication that the project goal was reached. © 2010 Elsevier Inc. All rights reserved. Keywords: Research methodology; Retrospective prettest-posttest; Statistics; Pharmacy education
Pharmacists must have the skills to evaluate the validity of methodologies and statistics used in current medical literature to support their clinical decisions. If they cannot evaluate and assimilate new medical literature, then there is a risk that they will provide inappropriate or outdated recommendations to patients and providers alike. Given this dynamic environment, knowledge of statistical and methodological concepts is increasingly important for pharmacy students as they transition into pharmacy practice. However, many students do not perceive the importance of research methods and statistics coursework; they find it uninteresting, inapplicable, or difficult.1– 6 These perceptions hinder
* Corresponding author: Adrienne M. Gilligan, BS, Graduate Student, Department of Pharmacy Practice and Science, The University of Arizona College of Pharmacy, 1295 N. Martin, PO Box 210202, Tucson, Arizona 85721. E-mail address: [email protected].
learning and represent a significant hurdle for faculty who strive to engage students in the classroom. Different methods to teach statistics and research methodology to students vary in their effectiveness given students’ individual learning styles.7,8 Many current pharmacy students are accustomed to watching fast-paced, multipleformat content delivery in the form of television, music videos, internet, and video games.9 Regardless of discipline, pharmacy educators struggle to develop mechanisms to keep students engaged in learning. Lasso found that students who grew up with music videos and video games relate better to a screen than to a blackboard and identified four main features of effective teaching: clarity, organization, stimulation of interest and student engagement, and classroom climate.10 Wilson and Gerber have named those born from 1982–2003 the “Millennials,” an idealistic generation that strives to be special and unique, as well as teamoriented.11 The authors listed four items that an instructor
1877-1297/10/$ – see front matter © 2010 Elsevier Inc. All rights reserved. doi:10.1016/j.cptl.2010.07.006
212
T. L. Warholak et al. / Currents in Pharmacy Teaching and Learning 2 (2010) 211–217
can do to keep the Millenial generation engaged in the classroom: (1) strive for greater clarity in course structure, assignments, and grading expectations; (2) provide significant opportunities for student initiative, participation, and choice; (3) incorporate stress-reduction mechanisms; and (4) engage students in a significant, courselong conversation on the ethical dimensions of taking a college class.11 Most survey-level textbooks have extensive and, at times, creative coverage of research methods and statistics concepts, but abstract discussion about the steps involved in research endeavors do not always elicit student involvement with the material; in contrast, research suggests that the use of music is an effective stimulus to enhance learning in the classroom.12 Activity-based learning experiences are also more likely to increase student engagement with course material.9,10,12 In particular, small group activities promote students’ mathematical and scientific thinking.13 Bluestone studied an activity-based, hands-on module used for the research methods and statistics unit of an introductory psychology course.14 Retrospective pretest-posttest questionnaire results showed that the process helped students understand the material and apply the concepts they learned to other aspects of their professional and personal life.14 Active learning has also been shown to be beneficial in pharmacy education.15-19 According to the Accreditation Council for Pharmacy Education (ACPE) competencies, Guideline 23.4 states that “the college or school should implement strategies and programs to broaden the professional horizons of students in areas such as scientific inquiry, scholarly concern for the profession, the relevance and value of research, and postgraduate education and training through guest lecturers, participation in curricular and extracurricular activities.”20 This course was designed to help students understand the relevance and value of research to their career in pharmacy, as well as the importance of understanding research methodology regardless of research or practice setting. Section I part C of the Professional Practice–Based Outcomes in the Center for Advancement of Pharmacy Education (CAPE) states that pharmacy students should be able to “interpret and evaluate pharmaceutical data and related information needed to prevent or resolve medicationrelated problems or to respond to information requests.”21 Examples from this section that pertain to research methodology include: (1) interpret statistical data; (2) evaluate research studies; (3) determine the validity of the patient-prescriber relationship; and (4) make reasonable assumptions and/or draw reasonable conclusions when data are incomplete. Bloom’s taxonomy of learning contains three domains: (1) cognitive; (2) affective; and (3) psychomotor.22 This project focused on the cognitive domain of learning that contains six levels of understanding, with knowledge at the lowest level, followed by comprehension, application, analysis, synthesis, and evaluation.22 With the creation of the
educational music videos, instructors wanted students to reach the application level in this research methods course, titled Drug Literature Evaluation-Research Design Considerations. The student projects started with learning the material (knowledge and comprehension) to recite in their video. The videos were then provided as a learning tool for assignments and exams (application). The purpose of this study was to assess the studentperceived outcomes of educational music video creation and to evaluate how students learn best in a research methods class. The music video assignment was termed the “methods television (MTV) video project” as a play on the Music Television (MTV) network channels. This instructional innovation is unique in that, to the authors’ knowledge, this is the only didactic research methodology and statistics course offered at any of the ACPE-accredited schools of pharmacy that offered students an option to learn through music video creation. This study evaluated the impact of the MTV video project on student learning and how students learned best in a research methods class.
Design Educational environment Drug Literature Evaluation–Research Design Considerations is the second required course in a required threecourse sequence. The first course in the sequence, Statistical Methods, focuses on biostatistics; the second, Research Design Considerations, focuses on research methodology and study design; and the third, Drug Literature Evaluation, applies the statistics and methods that students learned in the first two courses to the evaluation of medical literature and writing concise responses to drug information questions. The purpose of Drug Literature Evaluation–Research Design Considerations, offered in the second professional year, is to provide students with the skills needed to apply the principles of research design, statistical methods, and ethical dimensions to evaluate the medical literature and assess research reports and proposals. The overall goal is for students to accept the responsibility to prepare for a professional lifetime of evaluation and decision-making in the practice setting. This course is focused more on interpretation than on computation. From a more global perspective, the purpose of this course is to provide an applicationsoriented foundation that will allow students to comprehend and critically evaluate the statistical methodologies used in scientific literature. Specific topics included: understanding survey research, reliability, validity, sampling, p-values, selecting the appropriate statistical tests, research design, the scientific method, and basic epidemiology. Each class session contained the following components: (1) a 20 –30 minute mini lecture; (2) an in-class activity that gave the students the opportunity to practice new skills or check their understanding of information; and (3) a debriefing period to discuss the activity and answers.
T. L. Warholak et al. / Currents in Pharmacy Teaching and Learning 2 (2010) 211–217
Pedagogy/andragogy Instructors introduced the MTV extra credit project to students on the first day of class. Students had the option to produce an MTV video for extra credit. The MTV projects had to meet the following four criteria to receive credit: (1) minimum length of 1.75 minutes; (2) contain video for the entire length of the clip; (3) contain audio for the entire length of the clip; and (4) cover three objectives from the assigned lecture. Instructors presented each video in class to ensure that every student had an opportunity to view/learn from it, and they also posted each video on the class website, so that students could download them to use for study purposes. Assessment methods At the end of the 2008 Fall term, the six investigators (i.e., 4 class instructors, 1 graduate teaching assistant, and 1 researcher with expertise in educational evaluation) used a standard evaluation form to independently judge each of the 15 MTV videos. They evaluated each video on its organization, appearance, creativity, accuracy, clarity, and effort using a scale of 0 – 4 (poor to excellent), for a possible total of 24 points. All scores were added together to select the MTV video winners. All students who produced the optional video received 20 extra credit points (added to the 400 possible class points). Those students who produced the best videos, as judged by the investigators, received an additional 10 extra credit points. Expected outcomes The objective and desired outcome of the MTV video project was to make class more engaging, thereby improving students’ attitudes toward and knowledge of research methodology. Content Students who wished to participate in the MTV video project self-selected a team of up to four members. Each team informed the course coordinator of their intention to participate. Once all teams were selected, the coordinator and the teaching assistant randomly assigned a different lecture to each team. Teams were required to address at least three of the objectives listed for a given lecture in their video. If the assigned lecture had more than three objectives, students could select the ones they wanted to address. For lectures with less than three objectives, students worked with the coordinator to identify subpoints to address in the video. Each video was due two weeks after the lecture was presented in class. Examples of video titles, lyrics, and hyperlinks to the actual videos are shown in Fig. 1. Evaluation and assessment During the final class of the semester, all 88 students in attendance (95% of those enrolled) consented to complete
213
an attitudinal questionnaire about their opinions on the research methodology class and the MTV video assignments. Of the 88 students who completed the questionnaire, 63.9% were female, and 73.5% were 21–25 years of age. This questionnaire was designed as a retrospective pretestposttest of student perceptions with regard to research methods and their reaction to the MTV video projects, including whether they thought the video project was an effective learning tool. The questionnaire contained 15 open-ended items designed to solicit self-reported knowledge and attitudes associated with the methods class overall, and the MTV video assignment in particular along with 17 Likertscale response items. Two 4-point Likert response scales were used: (1) one ranging from “Weak” to “Very Good”; and (2) one ranging from “Disagree” to “Agree”. All results are presented as the percent of respondents. A retrospective pretest-posttest was used for evaluation because the method has demonstrated its usefulness in supporting validity when used to obtain attitudinal responses before and after a designated situation (in this case, research methods class).23–27 Retrospective pretest-posttest is defined as a “self-report during the course or at the end of treatment that measured subjects’ recall of how they were functioning before program outset.”23 This method helps to limit response-shift bias, a phenomenon that may occur when the internal construct changes as a result of the experience.23,24 The first portion of the student questionnaire (items 1–5) addressed students’ perception of their research methodology competencies before and after taking the research methods class. Note that although this particular set of items addressed the class as a whole, the authors feel that the MTV video projects were an important part of the overall learning experience because one video was viewed during every class period. As indicated in Table 1, many students rated their overall attitudes toward research methods as “Weak” or “Fair,” which increased to “Good” or “Very Good” after taking the class. After class completion, the number of students who rated their ability to evaluate research methods (i.e., item 1) increased almost fivefold. Similar large increases occurred in self-reported knowledge of research methods, motivation to learn research methods, awareness of the impact of research methods on pharmacy practice, and overall understanding of research methods. Student participation in the optional MTV video project was high; 65% of students participated and all of the videos were shown in class. More students downloaded the videos than expected; 57.8% of respondents downloaded one or more MTV videos and 36.5% downloaded five or more videos from the class website. Overall class grades were high; 51% of students in the class received a final grade of A, whereas 44% received a B. More students who completed the video project received grades of A, as expected, given the extra credit points awarded for their participation. The second portion of the student questionnaire (items 6 and 7; included in Table 2) addressed students’ perceptions of the importance of learning research methodology and the
214
T. L. Warholak et al. / Currents in Pharmacy Teaching and Learning 2 (2010) 211–217
“The Statistician” sample lyrics
“Office Stats” sample lyrics
“What is the concept of variability? Differences within a group. Define Probability and make it real easy! The chance an event will occur.
It’s observational You have no control You can’t divide You can’t decide
I really don’t get it; what is an alpha? It is like a False Alarm. Sectional OK, that makes sense; what about Beta? A Missed Opportunity!”
Observational Research Design Prospective, Retrospective, Cross
Please visit the following link for the full video:
Please visit the following link for full video:
http://www.youtube.com/watch?v=rCiAV305pxw
Let’s break it down Epidemiologic, Database, Case Studies
http://video.google.com/videosearch?q=office+stats&emb=0&aq=f#
Fig. 1. Sample pictures and lyrics from the 2 winning videos.
ability to assess studies before and after completion of the research methods class. Most respondents indicated that they “Somewhat Agreed” or “Agreed” more with each of these statements after taking the research methods class. The third portion of the student questionnaire (items 8 –15; also included in Table 2) addressed students’ perceptions of the impact of the MTV video projects before and after completion of the research methods class. At first, results for the MTV video projects appeared to be less than
optimal. Evaluation of the change in percent agreement for each question before and after the research methods class indicates that the impact or usefulness of the videos was less than the students expected in some instances (items 9, 11, 12, 13, and 15), whereas it remained steady for item 10 (“I am likely to remember research methods better because of the MTV video project”) and increased only for item 14 (“I would recommend the MTV assignment when the instructor teaches this class again”). Table 2 provides these results.
Table 1 Students overall attitudes toward research methods before and after completing the course
1. 2. 3. 4.
My ability to evaluate research methods My knowledge of research methods My motivation to learn research methods My awareness of the impact of research methods on pharmacy practice 5. My overall understanding of research methods
%
%
Before N⫽85
After N⫽86
Weak
Fair
Good
Very good
Weak
Fair
Good
Very good
42.3 37.7 51.8
41.2 49.4 40.0
16.5 12.9 8.2
0.0 0.0 0.0
0.0 1.2 23.3
20.9 18.6 46.4
74.4 70.9 25.6
4.7 9.3 4.7
21.2 37.6
58.8 47.1
12.9 15.3
7.1 0.0
3.5 2.3
14.0 22.1
59.2 67.5
23.3 8.1
T. L. Warholak et al. / Currents in Pharmacy Teaching and Learning 2 (2010) 211–217
215
Table 2 Student perceptions of the impact of research methodology class and the MTV videos
I feel. . . 6. It is important for me (as a future pharmacist) to have an understanding of research methods. (n ⫽ 86) 7. It is important for pharmacists to evaluate methods of drug studies. (n ⫽ 86) 8. Creating an MTV video will/did help me gain a better understanding of research methods. (n ⫽ 86) 9. Watching the MTV videos others created will/did help me gain a better understanding of research methods. (n ⫽ 86) 10. I am likely to remember research methods better because of the MTV project. (n ⫽ 85) 11. The MTV videos were a good studying tool. (n ⫽ 85) 12. The MTV videos increased my awareness of what I did not understand in the course. (n ⫽ 86) 13. The MTV videos helped me to better understand course material. (n ⫽ 86) 14. I would recommend the MTV assignment when the instructor teaches this course again. (n ⫽ 86) 15. Activities like the MTV assignment would be useful in other courses. (n ⫽ 86)
%
%
Before
After Somewhat disagree
Somewhat agree
Agree
Somewhat agree
Agree
7.0
14.0
58.0
21.0
3.5
5.8
34.9
55.8
5.8
9.3
41.9
43.0
2.3
5.8
26.7
65.2
24.4
29.1
41.8
4.7
22.1
31.2
32.7
14.0
30.2
31.4
33.7
4.7
34.9
39.5
24.4
1.2
28.4
30.5
32.9
8.2
29.4
29.4
27.1
14.1
31.8
35.2
27.1
5.9
37.6
37.7
17.6
7.1
37.1
33.7
28.0
1.2
44.2
33.7
19.8
2.3
31.4
33.7
30.2
4.7
37.1
32.6
23.3
7.0
23.3
23.2
30.2
23.3
22.1
15.1
29.1
33.7
23.3
28.0
35.9
12.8
29.1
24.4
27.9
18.6
Disagree
However, upon closer inspection of the data, it became apparent that those who created MTV videos felt that they benefited more than those who only watched the videos. The hypothesis that students who participated in the project would rate their perceived value of the project higher, on the average, than nonparticipants was evaluated using the Mann-Whitney U test. The average posttest ratings of the perceived value of the project (i.e., items 9 –15, n ⫽ 85 students), and then the ranks were evaluated for differences. Evaluation ended at item 15 because these were the open-ended items that pertained specifically to the MTV video. Differences were significant in the expected direction, z ⫽ –2.48, p ⬍ 0.05. Students who created a video had an average rank of 47.87, whereas students who did not create a video had an average rank of 34.07. Figure 2 shows the distributions of the average posttest ratings by both groups on the perceived value of the project for the two groups. On average, the mean posttest scores for items 9 –15 are 2.30 (SD ⫽ 0.81) for participants, and 1.86 (SD ⫽ 0.70) for nonparticipants.
Disagree
Somewhat disagree
When asked if the MTV videos helped to improve their knowledge of research methods, 51% of questionnaire respondents indicated they found the videos to be helpful (item 21). This project was supposed to be fun and entertaining, but also enhance learning. One student stated, “The stats video project was a great opportunity to reinforce information learned in the classroom. It was also the most fun I have had in pharmacy school completing an assignment, and I am sure that the rest of the group would agree.” Members of the group believed that the extra credit assignment would help them retain what they learned. “Because of this project,” says another student, “the word ‘epidemiology’ will never look the same.” Representative student comments about the videos included: “It appeared that those who made the video understood their subjects well;” “Making it forced me to study and understand my own section;” “They get stuck in your head! A good way to remember facts;” and “They were a blast to watch—my favorite part of the beginning of each class.” Planned changes will hopefully improve this in future cohorts. Again, respon-
216
T. L. Warholak et al. / Currents in Pharmacy Teaching and Learning 2 (2010) 211–217 Upper 75%ile Middle 50%ile Median Lower 25%ile
Fig. 2. Distributions of average posttest scores regarding the perceived value of the video project for participating and nonparticipating students.
and answers to solidify the concepts. Thus, the answers are consistent with the given responses to item 27 (“What type of material do you learn best?”), in which 72% of students indicated that they found concrete material easier to learn than abstract material. When asked how they would classify their learning style (item 28), only 30% indicated a preference for reading and writing, with the remainder indicating the visual, aural, kinesthetic, and visual or aural combination categories. This supports the need for alternative approaches such as the MTV video projects in a class that focuses on learning abstract research methodological concepts. The open-ended items of the questionnaire addressed students’ perceived outcomes toward the MTV video project, what activities helped them learn most in class, and what types of learning styles they preferred and what kind of learners they are. This information will be useful in planning future MTV assignments.
Limitations dents who created a video found the MTV video project more useful. When asked what activity in the course they learned the most from (item 26), the largest proportion respondents (32%) indicated that they learned from a variety of activities (Fig. 3). The next most frequent responses were in-class assignments and the combination of homework and some other activity. In-class assignments were completed each class period and were intended to allow students to practice the knowledge and skills covered in the readings and lecture. For example, after the reliability lecture, the in-class activity included a scientific abstract and a series of multiple-choice questions. Students worked with partners to answer the questions. Once the teams answered the questions, the instructor led the class in a discussion of the questions
Fig. 3. Student Report of Course Activity that Led to the Greatest Learning.
Overall, the MTV video project had four limitations that future assignments will address. First, some students thought that the MTV video project was not an appropriate assignment for introverts. Although instructors announced to the class that team members were not required to be on camera or to sing (one of the videos that tied for first place contained talking rather than singing), this announcement could have also been made clearer (e.g., in writing) to prevent this misunderstanding. Future assignments can suggest roles for those with introverted personalities. A second limitation of this video project was that many of the lyrics were interesting or creative, but because of sound quality issues and poor intonation, the lyrics in some videos were difficult to understand. In the future, students will be required to distribute or display the lyrics to their video at the time of the class presentation. Instructors could highlight how this assignment could target multiple learning styles when the assignment is introduced in the future. For example, providing lyrics could help those students who learn best from written materials. Another limitation of this video project was inappropriate dress. A few students either tried to emulate actual MTV network videos, or targeted video lyrics, costumes, and tone (e.g., skimpy dress, inappropriate language) to appeal to peers who they mistakenly thought would judge the videos. In the future, the MTV video assignment instructions will include a list of the evaluators and the criteria on which the videos will be evaluated so students can design videos accordingly. In addition, there will be explicit guidelines on language and dress. A final limitation to this study was that student’s grades were unable to be added into the analysis due to procedural restrictions. Future projects in this area may be able to take into account the student’s grades, so that investigators can determine whether creation of the MTV
T. L. Warholak et al. / Currents in Pharmacy Teaching and Learning 2 (2010) 211–217
video actually had a significant impact on test scores and knowledge. Conclusions The instructors thought the MTV video projects were successful. The data indicated that students who participated in the MTV video projects reported improved attitudes toward; knowledge and understanding of research methods. In addition, the MTV video projects resulted in unintended development of students’ technological competencies, such as video production/editing and uploading to the Internet. Based on the results, it appears that the perceived outcomes of the students in this research methodology class shifted in a positive direction. References 1. Connor FA, McCown SM, Roskos-Ewaldsen B. Unique challenges in teaching undergraduate statistics. Teach Psychol 1998;25:40 –2. 2. Connor JM. Making statistics come alive: Using space and students’ bodies to illustrate statistical concepts. Teach Psychol 2003;30:141–3. 3. Giesbrecht N, Sell Y, Scialfa C, et al. Essential topics in introductory statistics and methodology courses. Teach Psychol 1997;24:242– 6. 4. Hewett TT, Porpora DV. A case study report on integrating statistics, problem-based learning, and computerized data analysis. Behav Res Methods Instrum Comput 1999;31:244 –51. 5. Lanier ML. A pedagogical aid for linking methodological and statistical courses. Journal of Criminal Justice Education 2002; 13:155–71. 6. Piotrowski C, Bagui SC, Hemasina R. Development of a measure on statistics anxiety in graduate-level psychology students. Journal on Instructional Psychology 2002;29:97–100. 7. Bluestone C. Feature films as a teaching tool. College Teaching 2000;48:141– 6. 8. Perry NN, Huss MT, McAuliff BD, et al. An active-learning approach to teaching the undergraduate psychology and law course. Teach Psychol 1996;23:76 – 81. 9. MacGilchrist M, Buttress M. Transforming teaching and learning. Educational Management Administration & Leadership 2005;35:589 –90. 10. Lasso R. Technology in the classroom: Bridging the gap with the MTV Generation. 2001. Conference proceeding. Conference on new ideas for experienced teachers: We teach but do they learn? June 9 –13, 2001. Alberta, Canada.
217
11. Wilson M, Gerber L. How generational theory can improve teaching: Strategies for working with the “Millenials.” Currents in Teaching and Learning 2008;1:29 – 44. 12. Stadler MA. Demonstrating scientific reasoning. Teach Psychol 1998;25:205– 6. 13. Springer L, Stanne ME, Donovan S. Effects of small-group learning on undergraduates in science, mathematics, engineering, and technology: A meta-analysis. Rev Educ Res 1999;69: 21–51. 14. Bluestone C. Infusing active learning into the research methods unit. College Teaching 2007;55:91–5. 15. Gonyeau M, Dascoli M. Assessing classroom engagement utilizing student perceptions of faculty attributes and teaching techniques. Paper presented at the annual meeting of the American Association of Colleges of Pharmacy, Sheraton San Diego Hotel & Marina, San Diego, CA, July 5, 2006. 16. Jackson TL. Application of quality assurance principles: Teaching medication error reduction skills in a “real world Environment. Am J Pharm Educ 2004;68:Article 17. 17. Slain D, Abate M, Hodges BM, et al. An interactive response system to promote active learning in the Doctor of Pharmacy curriculum. Am J Pharm Educ 2004;68:Article 117. 18. Epstein RM, Cole DR, Gawinski BA, et al. How students learn from community-based preceptors. Arch Fam Med 1998;7:149 –54. 19. Reddy IK. Implementation of a pharmaceutics course in a large class through active learning using quick-thinks and case-based learning. Am J Pharm Educ 2000;64:348 –55. 20. ACPE Board of Directors and Staff. Accreditation Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree. Chicago: Accreditation Council for Pharmacy Education, 2006. 21. AACP Center for the Advancement of Pharmaceutical Education (CAPE) Advisory Panel on Educational Outcomes. Outcomes. Chicago: Accreditation Council for Pharmacy Education, 1998. 22. Bloom BS, ed. Taxonomy of Educational Objectives. The Classification of Educational Goals. Handbook 1: Cognitive Domain. New York: McKay, 1956. 23. Aiken LS, West SG. Invalidity of true experiments: Self-report pretest biases. Eval Rev 1991;14:374 –90. 24. Sprangers M, Hoogstraten J. Pre testing effects in retrospective pretest-posttest designs. J Appl Psychol 1989;74:265–72. 25. Howard GS. Response-shift bias: A problem in evaluating interventions with pre/post self-reports. Eval Rev 1980;4:93–106. 26. Skeff KM, Stratos GA, Bergen MR. Evaluation of a medical faculty development program: A comparison of traditional pre/post and retrospective pre/post self-assessment ratings. Evaluation & The Health Professions 1992;15:350 – 66. 27. Bray JH, Howard GS. Methodological considerations in the evaluation of a teacher- training program. J Educ Psychol 1980;72:62–70.
Currents in Pharmacy Teaching and Learning 2 (2010) 211–217 http://www.pharmacyteaching.com
Assessing student perceived outcomes for a Methods Television (MTV) video project in a research methodology class Terri L. Warholak, PhDa, Adrienne M. Gilligan, BSa,*, Jason T. Hurwitz, PhDb, Marion Slack, PhDa, Christina A. Spivey, PhDa, Marie Chisholm-Burns, PharmD, MPH, FCCP, FASHPa b
a Department of Pharmacy Practice and Science, The University of Arizona College of Pharmacy, Tucson, AZ Center for Health Outcomes & PharmacoEconomic Research, The University of Arizona College of Pharmacy, Tucson, AZ
Abstract Objectives: To (1) assess the outcomes of making a Methods Television (MTV) video on student perceptions, and (2) evaluate how students learn best in a research methods class. Methods: Pharmacy students enrolled in a research methods class (n ⫽ 93) were given the option to create an educational MTV video. Teams (n ⫽ 15) were randomly assigned a research methodology topic and three objectives for the video. All students in attendance on the last day (n ⫽ 88) completed an attitudinal questionnaire designed to assess students’ perceptions regarding the class and MTV video projects. Results: Students indicated that multiple activities helped them learn the most in class (i.e., in-class assignments, homework); 51% stated that videos helped improve their knowledge of research methods and 52% indicated it was entertaining. Conclusion: More than half of the students indicated that the MTV videos helped with their understanding of research methods, an indication that the project goal was reached. © 2010 Elsevier Inc. All rights reserved. Keywords: Research methodology; Retrospective prettest-posttest; Statistics; Pharmacy education
Pharmacists must have the skills to evaluate the validity of methodologies and statistics used in current medical literature to support their clinical decisions. If they cannot evaluate and assimilate new medical literature, then there is a risk that they will provide inappropriate or outdated recommendations to patients and providers alike. Given this dynamic environment, knowledge of statistical and methodological concepts is increasingly important for pharmacy students as they transition into pharmacy practice. However, many students do not perceive the importance of research methods and statistics coursework; they find it uninteresting, inapplicable, or difficult.1– 6 These perceptions hinder
* Corresponding author: Adrienne M. Gilligan, BS, Graduate Student, Department of Pharmacy Practice and Science, The University of Arizona College of Pharmacy, 1295 N. Martin, PO Box 210202, Tucson, Arizona 85721. E-mail address: [email protected].
learning and represent a significant hurdle for faculty who strive to engage students in the classroom. Different methods to teach statistics and research methodology to students vary in their effectiveness given students’ individual learning styles.7,8 Many current pharmacy students are accustomed to watching fast-paced, multipleformat content delivery in the form of television, music videos, internet, and video games.9 Regardless of discipline, pharmacy educators struggle to develop mechanisms to keep students engaged in learning. Lasso found that students who grew up with music videos and video games relate better to a screen than to a blackboard and identified four main features of effective teaching: clarity, organization, stimulation of interest and student engagement, and classroom climate.10 Wilson and Gerber have named those born from 1982–2003 the “Millennials,” an idealistic generation that strives to be special and unique, as well as teamoriented.11 The authors listed four items that an instructor
1877-1297/10/$ – see front matter © 2010 Elsevier Inc. All rights reserved. doi:10.1016/j.cptl.2010.07.006
212
T. L. Warholak et al. / Currents in Pharmacy Teaching and Learning 2 (2010) 211–217
can do to keep the Millenial generation engaged in the classroom: (1) strive for greater clarity in course structure, assignments, and grading expectations; (2) provide significant opportunities for student initiative, participation, and choice; (3) incorporate stress-reduction mechanisms; and (4) engage students in a significant, courselong conversation on the ethical dimensions of taking a college class.11 Most survey-level textbooks have extensive and, at times, creative coverage of research methods and statistics concepts, but abstract discussion about the steps involved in research endeavors do not always elicit student involvement with the material; in contrast, research suggests that the use of music is an effective stimulus to enhance learning in the classroom.12 Activity-based learning experiences are also more likely to increase student engagement with course material.9,10,12 In particular, small group activities promote students’ mathematical and scientific thinking.13 Bluestone studied an activity-based, hands-on module used for the research methods and statistics unit of an introductory psychology course.14 Retrospective pretest-posttest questionnaire results showed that the process helped students understand the material and apply the concepts they learned to other aspects of their professional and personal life.14 Active learning has also been shown to be beneficial in pharmacy education.15-19 According to the Accreditation Council for Pharmacy Education (ACPE) competencies, Guideline 23.4 states that “the college or school should implement strategies and programs to broaden the professional horizons of students in areas such as scientific inquiry, scholarly concern for the profession, the relevance and value of research, and postgraduate education and training through guest lecturers, participation in curricular and extracurricular activities.”20 This course was designed to help students understand the relevance and value of research to their career in pharmacy, as well as the importance of understanding research methodology regardless of research or practice setting. Section I part C of the Professional Practice–Based Outcomes in the Center for Advancement of Pharmacy Education (CAPE) states that pharmacy students should be able to “interpret and evaluate pharmaceutical data and related information needed to prevent or resolve medicationrelated problems or to respond to information requests.”21 Examples from this section that pertain to research methodology include: (1) interpret statistical data; (2) evaluate research studies; (3) determine the validity of the patient-prescriber relationship; and (4) make reasonable assumptions and/or draw reasonable conclusions when data are incomplete. Bloom’s taxonomy of learning contains three domains: (1) cognitive; (2) affective; and (3) psychomotor.22 This project focused on the cognitive domain of learning that contains six levels of understanding, with knowledge at the lowest level, followed by comprehension, application, analysis, synthesis, and evaluation.22 With the creation of the
educational music videos, instructors wanted students to reach the application level in this research methods course, titled Drug Literature Evaluation-Research Design Considerations. The student projects started with learning the material (knowledge and comprehension) to recite in their video. The videos were then provided as a learning tool for assignments and exams (application). The purpose of this study was to assess the studentperceived outcomes of educational music video creation and to evaluate how students learn best in a research methods class. The music video assignment was termed the “methods television (MTV) video project” as a play on the Music Television (MTV) network channels. This instructional innovation is unique in that, to the authors’ knowledge, this is the only didactic research methodology and statistics course offered at any of the ACPE-accredited schools of pharmacy that offered students an option to learn through music video creation. This study evaluated the impact of the MTV video project on student learning and how students learned best in a research methods class.
Design Educational environment Drug Literature Evaluation–Research Design Considerations is the second required course in a required threecourse sequence. The first course in the sequence, Statistical Methods, focuses on biostatistics; the second, Research Design Considerations, focuses on research methodology and study design; and the third, Drug Literature Evaluation, applies the statistics and methods that students learned in the first two courses to the evaluation of medical literature and writing concise responses to drug information questions. The purpose of Drug Literature Evaluation–Research Design Considerations, offered in the second professional year, is to provide students with the skills needed to apply the principles of research design, statistical methods, and ethical dimensions to evaluate the medical literature and assess research reports and proposals. The overall goal is for students to accept the responsibility to prepare for a professional lifetime of evaluation and decision-making in the practice setting. This course is focused more on interpretation than on computation. From a more global perspective, the purpose of this course is to provide an applicationsoriented foundation that will allow students to comprehend and critically evaluate the statistical methodologies used in scientific literature. Specific topics included: understanding survey research, reliability, validity, sampling, p-values, selecting the appropriate statistical tests, research design, the scientific method, and basic epidemiology. Each class session contained the following components: (1) a 20 –30 minute mini lecture; (2) an in-class activity that gave the students the opportunity to practice new skills or check their understanding of information; and (3) a debriefing period to discuss the activity and answers.
T. L. Warholak et al. / Currents in Pharmacy Teaching and Learning 2 (2010) 211–217
Pedagogy/andragogy Instructors introduced the MTV extra credit project to students on the first day of class. Students had the option to produce an MTV video for extra credit. The MTV projects had to meet the following four criteria to receive credit: (1) minimum length of 1.75 minutes; (2) contain video for the entire length of the clip; (3) contain audio for the entire length of the clip; and (4) cover three objectives from the assigned lecture. Instructors presented each video in class to ensure that every student had an opportunity to view/learn from it, and they also posted each video on the class website, so that students could download them to use for study purposes. Assessment methods At the end of the 2008 Fall term, the six investigators (i.e., 4 class instructors, 1 graduate teaching assistant, and 1 researcher with expertise in educational evaluation) used a standard evaluation form to independently judge each of the 15 MTV videos. They evaluated each video on its organization, appearance, creativity, accuracy, clarity, and effort using a scale of 0 – 4 (poor to excellent), for a possible total of 24 points. All scores were added together to select the MTV video winners. All students who produced the optional video received 20 extra credit points (added to the 400 possible class points). Those students who produced the best videos, as judged by the investigators, received an additional 10 extra credit points. Expected outcomes The objective and desired outcome of the MTV video project was to make class more engaging, thereby improving students’ attitudes toward and knowledge of research methodology. Content Students who wished to participate in the MTV video project self-selected a team of up to four members. Each team informed the course coordinator of their intention to participate. Once all teams were selected, the coordinator and the teaching assistant randomly assigned a different lecture to each team. Teams were required to address at least three of the objectives listed for a given lecture in their video. If the assigned lecture had more than three objectives, students could select the ones they wanted to address. For lectures with less than three objectives, students worked with the coordinator to identify subpoints to address in the video. Each video was due two weeks after the lecture was presented in class. Examples of video titles, lyrics, and hyperlinks to the actual videos are shown in Fig. 1. Evaluation and assessment During the final class of the semester, all 88 students in attendance (95% of those enrolled) consented to complete
213
an attitudinal questionnaire about their opinions on the research methodology class and the MTV video assignments. Of the 88 students who completed the questionnaire, 63.9% were female, and 73.5% were 21–25 years of age. This questionnaire was designed as a retrospective pretestposttest of student perceptions with regard to research methods and their reaction to the MTV video projects, including whether they thought the video project was an effective learning tool. The questionnaire contained 15 open-ended items designed to solicit self-reported knowledge and attitudes associated with the methods class overall, and the MTV video assignment in particular along with 17 Likertscale response items. Two 4-point Likert response scales were used: (1) one ranging from “Weak” to “Very Good”; and (2) one ranging from “Disagree” to “Agree”. All results are presented as the percent of respondents. A retrospective pretest-posttest was used for evaluation because the method has demonstrated its usefulness in supporting validity when used to obtain attitudinal responses before and after a designated situation (in this case, research methods class).23–27 Retrospective pretest-posttest is defined as a “self-report during the course or at the end of treatment that measured subjects’ recall of how they were functioning before program outset.”23 This method helps to limit response-shift bias, a phenomenon that may occur when the internal construct changes as a result of the experience.23,24 The first portion of the student questionnaire (items 1–5) addressed students’ perception of their research methodology competencies before and after taking the research methods class. Note that although this particular set of items addressed the class as a whole, the authors feel that the MTV video projects were an important part of the overall learning experience because one video was viewed during every class period. As indicated in Table 1, many students rated their overall attitudes toward research methods as “Weak” or “Fair,” which increased to “Good” or “Very Good” after taking the class. After class completion, the number of students who rated their ability to evaluate research methods (i.e., item 1) increased almost fivefold. Similar large increases occurred in self-reported knowledge of research methods, motivation to learn research methods, awareness of the impact of research methods on pharmacy practice, and overall understanding of research methods. Student participation in the optional MTV video project was high; 65% of students participated and all of the videos were shown in class. More students downloaded the videos than expected; 57.8% of respondents downloaded one or more MTV videos and 36.5% downloaded five or more videos from the class website. Overall class grades were high; 51% of students in the class received a final grade of A, whereas 44% received a B. More students who completed the video project received grades of A, as expected, given the extra credit points awarded for their participation. The second portion of the student questionnaire (items 6 and 7; included in Table 2) addressed students’ perceptions of the importance of learning research methodology and the
214
T. L. Warholak et al. / Currents in Pharmacy Teaching and Learning 2 (2010) 211–217
“The Statistician” sample lyrics
“Office Stats” sample lyrics
“What is the concept of variability? Differences within a group. Define Probability and make it real easy! The chance an event will occur.
It’s observational You have no control You can’t divide You can’t decide
I really don’t get it; what is an alpha? It is like a False Alarm. Sectional OK, that makes sense; what about Beta? A Missed Opportunity!”
Observational Research Design Prospective, Retrospective, Cross
Please visit the following link for the full video:
Please visit the following link for full video:
http://www.youtube.com/watch?v=rCiAV305pxw
Let’s break it down Epidemiologic, Database, Case Studies
http://video.google.com/videosearch?q=office+stats&emb=0&aq=f#
Fig. 1. Sample pictures and lyrics from the 2 winning videos.
ability to assess studies before and after completion of the research methods class. Most respondents indicated that they “Somewhat Agreed” or “Agreed” more with each of these statements after taking the research methods class. The third portion of the student questionnaire (items 8 –15; also included in Table 2) addressed students’ perceptions of the impact of the MTV video projects before and after completion of the research methods class. At first, results for the MTV video projects appeared to be less than
optimal. Evaluation of the change in percent agreement for each question before and after the research methods class indicates that the impact or usefulness of the videos was less than the students expected in some instances (items 9, 11, 12, 13, and 15), whereas it remained steady for item 10 (“I am likely to remember research methods better because of the MTV video project”) and increased only for item 14 (“I would recommend the MTV assignment when the instructor teaches this class again”). Table 2 provides these results.
Table 1 Students overall attitudes toward research methods before and after completing the course
1. 2. 3. 4.
My ability to evaluate research methods My knowledge of research methods My motivation to learn research methods My awareness of the impact of research methods on pharmacy practice 5. My overall understanding of research methods
%
%
Before N⫽85
After N⫽86
Weak
Fair
Good
Very good
Weak
Fair
Good
Very good
42.3 37.7 51.8
41.2 49.4 40.0
16.5 12.9 8.2
0.0 0.0 0.0
0.0 1.2 23.3
20.9 18.6 46.4
74.4 70.9 25.6
4.7 9.3 4.7
21.2 37.6
58.8 47.1
12.9 15.3
7.1 0.0
3.5 2.3
14.0 22.1
59.2 67.5
23.3 8.1
T. L. Warholak et al. / Currents in Pharmacy Teaching and Learning 2 (2010) 211–217
215
Table 2 Student perceptions of the impact of research methodology class and the MTV videos
I feel. . . 6. It is important for me (as a future pharmacist) to have an understanding of research methods. (n ⫽ 86) 7. It is important for pharmacists to evaluate methods of drug studies. (n ⫽ 86) 8. Creating an MTV video will/did help me gain a better understanding of research methods. (n ⫽ 86) 9. Watching the MTV videos others created will/did help me gain a better understanding of research methods. (n ⫽ 86) 10. I am likely to remember research methods better because of the MTV project. (n ⫽ 85) 11. The MTV videos were a good studying tool. (n ⫽ 85) 12. The MTV videos increased my awareness of what I did not understand in the course. (n ⫽ 86) 13. The MTV videos helped me to better understand course material. (n ⫽ 86) 14. I would recommend the MTV assignment when the instructor teaches this course again. (n ⫽ 86) 15. Activities like the MTV assignment would be useful in other courses. (n ⫽ 86)
%
%
Before
After Somewhat disagree
Somewhat agree
Agree
Somewhat agree
Agree
7.0
14.0
58.0
21.0
3.5
5.8
34.9
55.8
5.8
9.3
41.9
43.0
2.3
5.8
26.7
65.2
24.4
29.1
41.8
4.7
22.1
31.2
32.7
14.0
30.2
31.4
33.7
4.7
34.9
39.5
24.4
1.2
28.4
30.5
32.9
8.2
29.4
29.4
27.1
14.1
31.8
35.2
27.1
5.9
37.6
37.7
17.6
7.1
37.1
33.7
28.0
1.2
44.2
33.7
19.8
2.3
31.4
33.7
30.2
4.7
37.1
32.6
23.3
7.0
23.3
23.2
30.2
23.3
22.1
15.1
29.1
33.7
23.3
28.0
35.9
12.8
29.1
24.4
27.9
18.6
Disagree
However, upon closer inspection of the data, it became apparent that those who created MTV videos felt that they benefited more than those who only watched the videos. The hypothesis that students who participated in the project would rate their perceived value of the project higher, on the average, than nonparticipants was evaluated using the Mann-Whitney U test. The average posttest ratings of the perceived value of the project (i.e., items 9 –15, n ⫽ 85 students), and then the ranks were evaluated for differences. Evaluation ended at item 15 because these were the open-ended items that pertained specifically to the MTV video. Differences were significant in the expected direction, z ⫽ –2.48, p ⬍ 0.05. Students who created a video had an average rank of 47.87, whereas students who did not create a video had an average rank of 34.07. Figure 2 shows the distributions of the average posttest ratings by both groups on the perceived value of the project for the two groups. On average, the mean posttest scores for items 9 –15 are 2.30 (SD ⫽ 0.81) for participants, and 1.86 (SD ⫽ 0.70) for nonparticipants.
Disagree
Somewhat disagree
When asked if the MTV videos helped to improve their knowledge of research methods, 51% of questionnaire respondents indicated they found the videos to be helpful (item 21). This project was supposed to be fun and entertaining, but also enhance learning. One student stated, “The stats video project was a great opportunity to reinforce information learned in the classroom. It was also the most fun I have had in pharmacy school completing an assignment, and I am sure that the rest of the group would agree.” Members of the group believed that the extra credit assignment would help them retain what they learned. “Because of this project,” says another student, “the word ‘epidemiology’ will never look the same.” Representative student comments about the videos included: “It appeared that those who made the video understood their subjects well;” “Making it forced me to study and understand my own section;” “They get stuck in your head! A good way to remember facts;” and “They were a blast to watch—my favorite part of the beginning of each class.” Planned changes will hopefully improve this in future cohorts. Again, respon-
216
T. L. Warholak et al. / Currents in Pharmacy Teaching and Learning 2 (2010) 211–217 Upper 75%ile Middle 50%ile Median Lower 25%ile
Fig. 2. Distributions of average posttest scores regarding the perceived value of the video project for participating and nonparticipating students.
and answers to solidify the concepts. Thus, the answers are consistent with the given responses to item 27 (“What type of material do you learn best?”), in which 72% of students indicated that they found concrete material easier to learn than abstract material. When asked how they would classify their learning style (item 28), only 30% indicated a preference for reading and writing, with the remainder indicating the visual, aural, kinesthetic, and visual or aural combination categories. This supports the need for alternative approaches such as the MTV video projects in a class that focuses on learning abstract research methodological concepts. The open-ended items of the questionnaire addressed students’ perceived outcomes toward the MTV video project, what activities helped them learn most in class, and what types of learning styles they preferred and what kind of learners they are. This information will be useful in planning future MTV assignments.
Limitations dents who created a video found the MTV video project more useful. When asked what activity in the course they learned the most from (item 26), the largest proportion respondents (32%) indicated that they learned from a variety of activities (Fig. 3). The next most frequent responses were in-class assignments and the combination of homework and some other activity. In-class assignments were completed each class period and were intended to allow students to practice the knowledge and skills covered in the readings and lecture. For example, after the reliability lecture, the in-class activity included a scientific abstract and a series of multiple-choice questions. Students worked with partners to answer the questions. Once the teams answered the questions, the instructor led the class in a discussion of the questions
Fig. 3. Student Report of Course Activity that Led to the Greatest Learning.
Overall, the MTV video project had four limitations that future assignments will address. First, some students thought that the MTV video project was not an appropriate assignment for introverts. Although instructors announced to the class that team members were not required to be on camera or to sing (one of the videos that tied for first place contained talking rather than singing), this announcement could have also been made clearer (e.g., in writing) to prevent this misunderstanding. Future assignments can suggest roles for those with introverted personalities. A second limitation of this video project was that many of the lyrics were interesting or creative, but because of sound quality issues and poor intonation, the lyrics in some videos were difficult to understand. In the future, students will be required to distribute or display the lyrics to their video at the time of the class presentation. Instructors could highlight how this assignment could target multiple learning styles when the assignment is introduced in the future. For example, providing lyrics could help those students who learn best from written materials. Another limitation of this video project was inappropriate dress. A few students either tried to emulate actual MTV network videos, or targeted video lyrics, costumes, and tone (e.g., skimpy dress, inappropriate language) to appeal to peers who they mistakenly thought would judge the videos. In the future, the MTV video assignment instructions will include a list of the evaluators and the criteria on which the videos will be evaluated so students can design videos accordingly. In addition, there will be explicit guidelines on language and dress. A final limitation to this study was that student’s grades were unable to be added into the analysis due to procedural restrictions. Future projects in this area may be able to take into account the student’s grades, so that investigators can determine whether creation of the MTV
T. L. Warholak et al. / Currents in Pharmacy Teaching and Learning 2 (2010) 211–217
video actually had a significant impact on test scores and knowledge. Conclusions The instructors thought the MTV video projects were successful. The data indicated that students who participated in the MTV video projects reported improved attitudes toward; knowledge and understanding of research methods. In addition, the MTV video projects resulted in unintended development of students’ technological competencies, such as video production/editing and uploading to the Internet. Based on the results, it appears that the perceived outcomes of the students in this research methodology class shifted in a positive direction. References 1. Connor FA, McCown SM, Roskos-Ewaldsen B. Unique challenges in teaching undergraduate statistics. Teach Psychol 1998;25:40 –2. 2. Connor JM. Making statistics come alive: Using space and students’ bodies to illustrate statistical concepts. Teach Psychol 2003;30:141–3. 3. Giesbrecht N, Sell Y, Scialfa C, et al. Essential topics in introductory statistics and methodology courses. Teach Psychol 1997;24:242– 6. 4. Hewett TT, Porpora DV. A case study report on integrating statistics, problem-based learning, and computerized data analysis. Behav Res Methods Instrum Comput 1999;31:244 –51. 5. Lanier ML. A pedagogical aid for linking methodological and statistical courses. Journal of Criminal Justice Education 2002; 13:155–71. 6. Piotrowski C, Bagui SC, Hemasina R. Development of a measure on statistics anxiety in graduate-level psychology students. Journal on Instructional Psychology 2002;29:97–100. 7. Bluestone C. Feature films as a teaching tool. College Teaching 2000;48:141– 6. 8. Perry NN, Huss MT, McAuliff BD, et al. An active-learning approach to teaching the undergraduate psychology and law course. Teach Psychol 1996;23:76 – 81. 9. MacGilchrist M, Buttress M. Transforming teaching and learning. Educational Management Administration & Leadership 2005;35:589 –90. 10. Lasso R. Technology in the classroom: Bridging the gap with the MTV Generation. 2001. Conference proceeding. Conference on new ideas for experienced teachers: We teach but do they learn? June 9 –13, 2001. Alberta, Canada.
217
11. Wilson M, Gerber L. How generational theory can improve teaching: Strategies for working with the “Millenials.” Currents in Teaching and Learning 2008;1:29 – 44. 12. Stadler MA. Demonstrating scientific reasoning. Teach Psychol 1998;25:205– 6. 13. Springer L, Stanne ME, Donovan S. Effects of small-group learning on undergraduates in science, mathematics, engineering, and technology: A meta-analysis. Rev Educ Res 1999;69: 21–51. 14. Bluestone C. Infusing active learning into the research methods unit. College Teaching 2007;55:91–5. 15. Gonyeau M, Dascoli M. Assessing classroom engagement utilizing student perceptions of faculty attributes and teaching techniques. Paper presented at the annual meeting of the American Association of Colleges of Pharmacy, Sheraton San Diego Hotel & Marina, San Diego, CA, July 5, 2006. 16. Jackson TL. Application of quality assurance principles: Teaching medication error reduction skills in a “real world Environment. Am J Pharm Educ 2004;68:Article 17. 17. Slain D, Abate M, Hodges BM, et al. An interactive response system to promote active learning in the Doctor of Pharmacy curriculum. Am J Pharm Educ 2004;68:Article 117. 18. Epstein RM, Cole DR, Gawinski BA, et al. How students learn from community-based preceptors. Arch Fam Med 1998;7:149 –54. 19. Reddy IK. Implementation of a pharmaceutics course in a large class through active learning using quick-thinks and case-based learning. Am J Pharm Educ 2000;64:348 –55. 20. ACPE Board of Directors and Staff. Accreditation Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree. Chicago: Accreditation Council for Pharmacy Education, 2006. 21. AACP Center for the Advancement of Pharmaceutical Education (CAPE) Advisory Panel on Educational Outcomes. Outcomes. Chicago: Accreditation Council for Pharmacy Education, 1998. 22. Bloom BS, ed. Taxonomy of Educational Objectives. The Classification of Educational Goals. Handbook 1: Cognitive Domain. New York: McKay, 1956. 23. Aiken LS, West SG. Invalidity of true experiments: Self-report pretest biases. Eval Rev 1991;14:374 –90. 24. Sprangers M, Hoogstraten J. Pre testing effects in retrospective pretest-posttest designs. J Appl Psychol 1989;74:265–72. 25. Howard GS. Response-shift bias: A problem in evaluating interventions with pre/post self-reports. Eval Rev 1980;4:93–106. 26. Skeff KM, Stratos GA, Bergen MR. Evaluation of a medical faculty development program: A comparison of traditional pre/post and retrospective pre/post self-assessment ratings. Evaluation & The Health Professions 1992;15:350 – 66. 27. Bray JH, Howard GS. Methodological considerations in the evaluation of a teacher- training program. J Educ Psychol 1980;72:62–70.