- Email: [email protected]
Mixed methods: Expanding research methodologies in pharmacy education
Available online at www.sciencedirect.com
Currents in Pharmacy Teaching and Learning 8 (2016) 715–721
Methodology Matters
http://www.pharmacyteaching.com
Mixed methods: Expanding research methodologies in pharmacy education Jacqueline E. McLaughlin, PhD, MS*, Antonio A. Bush, PhD, MS, Jacqueline M. Zeeman, PharmD Division of Practice Advancement and Clinical Education, UNC Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC
Abstract Mixed methods research combines quantitative and qualitative research to address a research question. Researchers engaging in mixed methods can establish a more complete understanding of an issue, triangulate findings, develop a complementary picture, identify trends, and validate results. Using mixed methods research may provide a more accurate and comprehensive reflection of educational experiences and outcomes across the span of pharmacy, including admissions, experiential education, and faculty development. Expanding our research toolkit to include multiple approaches for collecting, analyzing, and applying quantitative and qualitative data are likely to strengthen educational research in pharmacy education and broaden our impact across the health professions. After reviewing this article, the reader should be able to define mixed methods research, provide recommendations for its use, identify applications for pharmacy education, and describe the implications and limitations of the mixed methods approach. r 2016 Elsevier Inc. All rights reserved.
Keywords: Research methods; Mixed methods; Qualitative research; Quantitative research; Triangulation; Research design
Our Situation, issue, or problem A growing body of literature in health care and education point to the challenges of providing students with opportunities to develop critical workplace skills (e.g., adaptability, collaboration, communication, and initiative).1–3 In many cases, these skills are best taught and assessed in the context of experiential education; however, researchers interested in examining student development during experiential education are faced with a number of challenges, including small sample sizes (e.g., n o 30) and * Corresponding author: Jacqueline E. McLaughlin, PhD, MS, Division of Practice Advancement and Clinical Education, UNC Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599. E-mail: [email protected] http://dx.doi.org/10.1016/j.cptl.2016.06.015 1877-1297/r 2016 Elsevier Inc. All rights reserved.
complex construct measurement. As an example, my research team was recently tasked with developing a new experiential education program and evaluating its impact on the professional identity development of students. In higher education, identity development is articulated by a number of distinct theories and can include complex constructs that are not easily reduced to a single metric or survey item such as developing competence, moving through autonomy toward interdependence, developing mature interpersonal relationships, and developing purpose.4 Further, some experiential education theories articulate the role of reflection as a key element of creating knowledge.5 While quantitative data may be useful in this situation, quantitative data alone will not provide the depth and richness of data (e.g., reflection) necessary to fully understand student development. Along the same lines,
716
J.E. McLaughlin et al. / Currents in Pharmacy Teaching and Learning 8 (2016) 715–721
qualitative data may not be amenable to systematic comparisons or hypothesis testing (e.g., limited by smaller sample sizes and analyzed qualitatively) and may fail to generate the type of statistical feedback desired by experiential educators to inform practice moving forward. For these reasons, we needed the distinct advantages of quantitative and qualitative methods and chose to employ both approaches to conduct a mixed methods study. Methodological literature review In general, mixed methods research can be defined as the combination of both quantitative and qualitative research efforts to address one research question (Table 1).6–8 The combination of these two approaches can answer a specific research question more comprehensively, increase the tools available to conduct a research study, and broaden the scope of research possible.7,9 Through integration of quantitative and qualitative research, investigators are able to maximize the strengths and minimize the weaknesses of each individual approach.6,7 The sequence and dominance of quantitative and qualitative efforts may vary among mixed method approaches; however, the integration of the mixed methods results, even if they initially appear to disagree, can strengthen the overall research design to overcome the respective limitations of the individual quantitative and qualitative designs.6,10 To better understand mixed methods design, it might be helpful to review quantitative and qualitative research in more detail. Quantitative research utilizes numerical data to explain a particular phenomenon, such as performance, attitudes, or
engagement.11,12 It originates from positivism, which is the idea that a single reality or truth can be discovered through the objective study of numeric data and variables. A wide range of sources can be utilized to collect numeric data including surveys, structured interviews, observations, performance indicators and assessments, institutional information, and national databases.13 In some cases, these data may be collected as qualitative data (e.g., interviews) and reduced to numeric quantities for the purposes of analysis. Quantitative data are analyzed using statistical methods that describe variables (i.e., descriptive statistics) and draw inferences of a sample to the population at large (i.e., inferential statistics).11–13 Quantitative methods use two main research design approaches: experimental design and non-experimental design.11 Experimental design involves the random assignment of participants to either the experimental group or the control group to test a hypothesis under controlled conditions.11 Quasi-experimental designs are utilized when random assignment is not feasible but continue to test a hypothesis under controlled conditions.11 In contrast, nonexperimental design (e.g., observational study) does not involve the manipulation of study variables nor is the setting controlled. In pharmacy education, research designs vary widely and include the quantitative research traditions of epidemiologic, correlational, and psychometric studies in addition to randomized control. Statistical methods for analyzing quantitative data can range in complexity from simple correlations to multilevel modeling.13 Although health care researchers have traditionally employed quantitative methods, qualitative approaches have
Table 1 Comparison of research approaches Quantitative research
Qualitative research
Mixed methods research
Understand a specific population or phenomenon Text, pictures, video, audio, or other multimedia
Examine a question from a quantitative and qualitative perspective Both quantitative and qualitative data types
Observations, interviews, focus groups, documents
Uses a mix of qualitative and quantitative data sources
Coding and document analysis
Triangulation; integrating data analysis methods
Quality criteria
Determine relationships and describe variables; test hypotheses Numeric or reduced to numeric quantities for purposes of analysis Surveys, records, learning assessments (e.g. tests), measurements Statistical and numerical analyses (descriptive and inferential statistics) Validity, reliability, objectivity
Common applications
Description; generalization
Credibility, dependability, conformability, transferability. Theme identification; theory development; case analysis
Limitations
Reduced power for small sample sizes; difficult to measure complex constructs; lack of deep, rich description
Some combination of quantitative and qualitative criteria Triangulation, complementarity, development initiation, expansion Lack of resources; requires skills in both approaches; publication word limitations hindering thorough method and results descriptions
Purpose
Data characteristics
Common data sources
Data analysis
Lack of statistical power and generalizability; time intensive; potentially misunderstood by audience due to lack of training
J.E. McLaughlin et al. / Currents in Pharmacy Teaching and Learning 8 (2016) 715–721
been shown to complement or serve as a viable alternative for statistical methods that may not fully address the complexities associated with health behavior, organizations, practice, and policy.14 Qualitative research can be defined as a method that emphasizes studying people and places in a natural setting and interpreting phenomena or experiences in a way that is most consistent with participant understanding.15,16 It originates from constructivism, which is the idea that groups construct knowledge for one another and that there are as many realities as there are social constructions. Key characteristics of the qualitative research process include16–19 (1) multiple sources of data are collected and analyzed to identify themes, (2) the research emphasizes the participants' perspectives and subjective views of the phenomena being studied, (3) the research design can be fluid since the data collection process may change or shift when entering the field, (4) the research methods should include a saturation process in which data are collected and reviewed until new information does not add to the understanding of the issues being studied, and (5) the researcher recognizes that multiple interpretations of an issue may emerge from various stakeholders (e.g., subjectivity of the researcher, participants, and reader). Qualitative research is commonly used to explore the complex understanding of an issue, empower individuals to share their stories, de-emphasize power relationships, understand the context or setting in which participants face challenges, follow-up on quantitative findings, develop theories, and when a quantitative approach is impractical or inappropriate.14,16 This approach reduces the distance between the researcher and participant and provides depth about contexts, values, opinions, and behaviors that is often missing in quantitative research. Five traditional qualitative designs are narrative research, phenomenology, grounded theory, ethnography, and case study (Table 2).
717
Critical aspects of research include implementing an appropriate research design, collecting useful and accurate data, using the right analysis tools, and interpreting the results correctly. Integrating qualitative and quantitative research can optimize the strengths and address the limitations of each approach and provides a pragmatic alternative to two distinct paradigms that drive the design and implementation of educational research (i.e., positivism and constructivism). The purpose of this article is to describe mixed methods research as a means of expanding the scope and impact of educational research in pharmacy. The following sections provide recommendations for its use, identify applications for pharmacy education, and describe the implications and limitations of the mixed methods approach.
Our recommendations and their applications In the situation described at the start of this article (i.e., my research team was recently tasked with developing a new experiential education program and evaluating its impact on the professional identity development of students), mixed methods were used for a number of reasons. First, our sample size was relatively small (about 20 students). This meant that while we planned to collect valuable and relevant information in the form of survey items (e.g., self-efficacy), there was a risk that the study was underpowered for statistical analyses. As such, we recognized during the design of our study that qualitative data could supplement, provide insight into, and potentially support our limited quantitative findings (i.e., triangulation). Second, identity development is a complex educational construct that is not easily reduced to items measured on a survey instrument or questionnaire. For this reason, we asked students to turn in reflection statements that provided rich qualitative data about their experiences in the program. Third, this study was implemented as a pilot with plans to scale the program out to a much larger number of students in the subsequent academic year. While we needed data for the purposes of the identity development research, we also
Table 2 Five qualitative approaches and their characteristicsa Approaches to qualitative research Characteristics
Narrative research
Phenomenology
Grounded theory
Ethnography
Case study
Focus
The life of an individual
Developing a theory
A culture-sharing group
Unit of analysis
One or more individuals
A group that shares the same culture
Clandinin and Connelly37
A process, action, or interaction involving many individuals Charmaz39
In-depth description and analysis of a case(s) An event, program, activity, group, or individual
Resources
The essence of the experience or phenomenon Several individuals that have shared the experience Moustakas38
Atkinson et al.40
Stake41 and Yin46
a
Adapted from Creswell et al.16
718
J.E. McLaughlin et al. / Currents in Pharmacy Teaching and Learning 8 (2016) 715–721
needed to collect program evaluation data about the practical and logistical elements of the experience to inform future iterations. These evaluation data were also best suited for a mixed methods approach, with survey items (i.e., quantitative data) and open-ended text survey boxes (i.e., qualitative data) used to collect feedback from students at the conclusion of the program. We recommend that any researcher using mixed methods consider the following criteria to guide the design and implementation of a study: (1) timing, (2) weighting, (3) mixing, and (4) theorizing.20 Specifically, the researcher must determine the timing of data collection, which may be sequential (i.e., in phases) or concurrent (i.e., at the same time). It may be problematic, for example, to collect quantitative and qualitative data sequentially due to scheduling and/or other limiting factors. The weighting (or priority) that the researcher gives to the qualitative and quantitative approach may be equally or unequally distributed depending on a variety of factors, including researcher interests, research team skills, and the study audience. Mixing describes the extent to and the point at which the researcher merges the qualitative and quantitative data. Mixed data may be connected to inform a second phase of a study, integrated by transforming qualitative data into quantitative data (e.g., word counts, theme counts, and descriptive data), or used to support the findings of one another. Theorizing involves the process of using a theory to frame or guide the research design. Theoretical frameworks are commonly used in educational research to inform the research questions, participants involved, data collection, and implications.13 Further, we recommend that the quality indicators for quantitative and qualitative approaches be taken into consideration when designing a mixed methods study.20 Validity, reliability, and objectivity are generally used as criteria for determining the quality of quantitative research. Validity reflects the extent to which a measure actually represents what it claims to measure, reliability concerns the degree to which a finding is stable and consistent, and objectivity indicates the extent to which the instrument being used is resistant to influence by the beliefs and biases of researchers. Related but distinct primary criteria used to evaluate the trustworthiness of qualitative research include credibility, dependability, conformability, transferability, and authenticity.18,21,22 Credibility reflects confidence of the findings and can be promoted by prolonged and persistent engagement in the field, sharing data and interpretations of the data with participants for confirmation (i.e., member checking), discussing the data with critical colleagues (i.e., peer debriefing), and using multiple sources to verify the data (i.e., triangulation).18,21,22 Dependability refers to the stability or repeatability of the research process.22 Since qualitative research can be fluid, any changes in methods during the study should be tracked using extensive notes to demonstrate the quality of the research.22,23 Conformability involves minimizing the researcher's subjective influence on the research and can
be encouraged by providing a transparent description of the study methods, triangulating the data, and practicing reflexivity.21,22 Transferability concerns the extent to which the research findings are applicable in other contexts21,22 and can be enhanced through the use of rich, in-depth descriptions (e.g., time, place, and context) and/or using multiple case studies. Authenticity reflects the extent to which participants' experiences are objectively reported. O'Brien et al.24 defines additional standards for the conduct and reporting of qualitative research in the health professions. The criteria used to determine the quality of quantitative and qualitative research, though distinct and not without contention, have been firmly established. However, these criteria become more complex when two different sets of procedures are merged to conduct a single study. In recent years, interest in identifying and addressing quality criteria in mixed methods research has grown25,26 and researchers may find these new mixed methods criteria useful. At the very least, it is recommended that researchers design and implement mixed methods research with consideration to the quality criteria for both approaches. For example, indicators for validity and reliability could be examined and reported with quantitative data while methods for promoting credibility and conformability could be described with qualitative data. Applications: A growing body of literature demonstrates the use of mixed methods in pharmacy and health professions education. Examples in pharmacy education include studies of experiential education, academic help-seeking behavior, and classroom experiences.29–31 Johnson et al.32 for example, used mixed methods research to determine experiential learning opportunities and identify barriers to rotation sites. In this study, quantitative data were used to build a multivariate logistic regression model while qualitative data from faculty interviews were used to further elucidate site placement factors. Mixed methods were also employed by Khanova et al.33 to examine student perceptions of a flipped pharmacotherapy course. Precourse and post-course survey data were collected, analyzed, and further explored through qualitative themes derived from text responses on the survey. As another example, Pinelli et al.29 examined health system pharmacy practice outcomes associated with student pharmacists involved in medication reconciliation. Quantitative data were used to describe patient characteristics and student productivity statistics while qualitative analysis was used to examine pharmacist perceptions of student contributions to the health care system. Clearly, a wide range of research contexts and problems in pharmacy education can be approached with a mixed methods design. From admissions to experiential education to alumni engagement, mixed methods provides unique opportunities and approaches for designing studies that more accurately and comprehensively reflect educational environments and outcomes. Research questions most suitable for mixed method approaches are those in which a quantitative or qualitative approach alone are inadequate to develop a complete understanding of the research question.20 Greene et al.27 identified five specific applications for mixed methods research—(1)
J.E. McLaughlin et al. / Currents in Pharmacy Teaching and Learning 8 (2016) 715–721
triangulation, (2) complementarity, (3) development, (4) initiation, and (5) expansion. Triangulation increases validity by seeking to converge and confirm results from quantitative and qualitative methods.9,27 In our situation, the qualitative findings converged with the quantitative findings to suggest that the program contributed to development of professional identity. Complementarity uses the research from one method to enhance, illustrate, or further clarify the results of the other, which can increase interpretability and validity of the individual methods.27 Development refers to the utilization of one method's results to inform the other method.27 For example, focus group interviews (or qualitative methods) may help to inform the development of a questionnaire (or quantitative methods) to be administered to the community.9 The primary purpose of initiation is to uncover inconsistencies and contradictions within one research method to influence the research question or results of the other research method.27 Finally, expansion seeks to extend the breadth and depth of inquiry by using different research methods to inquire about different components of the research.27 Further, mixed methods research presents a unique opportunity to test a hypothesis among a large group while adding information about the individual participant.20 Wagner et al.,28 for example, utilized a mixed methods approach in pharmacy practice research regarding drug using populations and noted that they “achieved a more nuanced understanding of the issues than may have been achieved through the use of a single method.” Since small samples may be underpowered (i.e., unable to detect a statistical difference), quantitative data can provide important numeric
719
information while qualitative data can provide rich, in-depth information that enables the researcher to more fully understand the studied phenomenon. Further, activities and outcomes associated with teaching and learning can often embody important social, cultural, and educational values that influence the way in which data are understood and interpreted. As such, complex constructs that are not easily measurable may benefit from a mixed methods approach that enables triangulation of statistical findings with important contextual and value-based themes.
Potential impact Pharmacy educators are uniquely positioned to examine critical research questions that are not only of importance in pharmacy and pharmacy education, but of increasing importance and interest to larger audiences within health professions and higher education. Expanding research toolkits to include approaches to collecting, analyzing, and applying quantitative and qualitative data are likely to strengthen educational research in pharmacy education, contribute to associated research in other disciplines, and broaden our impact across the health professions. Researchers utilizing mixed methods can establish a more complete understanding of a problem, develop a complementary picture by comparing and validating results, identify trends, and examine the process or experience in addition to the outcome.34 Box 1 provides additional readings for the interested reader.
Box 1 Recommended Resources Journal articles • Johnson RB, Onwuegbuzie AJ. Mixed Methods Research: A Research Paradigm Whose Time Has Come. Educ Res. 2004;33(7):14-26. http://dx.doi.org/10.3102/0013189X033007014.42 o This article provides a review of mixed methods research and an eight step process to explain mixed methods research. • Onwuegbuzie AJ, Johnson RB. The Validity Issue in Mixed Research. University of South Alabama. Res Sch. 2006;13(1):48-63.25 o In recognizing that validy discussions in mixed methods research are in their infancy, this article introduces a framework termed legitimation to evaluate mixed methods research. • Small ML. How to Conduct a Mixed Methods Study: Recent Trends in a Rapidly Growing Literature. Annu Rev Sociol. 2011;37(1):5786. http://dx.doi.org/10.1146/annurev.soc.012809.102657.43 o This articles provides reviews and critiques of general approaches to mixed methods research. Books • Creswell, JW. Research design: Qualitative, quantitative, and mixed methods approaches. Sage Publications, 2013.12 o This book provides in depth analysis on the frameworks, processes, and approaches utilized in designing qualitative, quantitative, and mixed methods research. • McMillan JH. Educational Research: Fundamentals for the Consumer. 6th edition. New York: Pearson; 2012.44 o This book introduces basic research principles for conducting and evaluating research and includes several useful aids and author reflections. • Teddlie C, Tashakkori A. Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. Sage Publications; 2009.45 o This book details all aspects of mixed methods research from research design to analysis and reporting.
720
J.E. McLaughlin et al. / Currents in Pharmacy Teaching and Learning 8 (2016) 715–721
In addition, mixed methods research provides additional mechanisms for conducting assessment and evaluation intended for programmatic improvement. In an era of increasing accountability at all levels of higher education, colleges and schools of pharmacy must take measurable steps to ensure that processes are in place to better utilize data to inform planning. Since educational environments often preclude experimental design and include complex constructs and phenomena, mixed methods can provide assessment professionals with diverse strategies and data sources for understanding outcomes at a local level. This might include course evaluations that use Likert-scale items and open-ended text boxes, student feedback in the form of surveys and focus groups, and curriculum mapping that includes syllabus review and faculty surveys. Limitations: Despite the apparent benefits of mixed methods research, there are challenges and limitations associated with its implementation. Perhaps the biggest challenge is acquiring the necessary knowledge and skill for both quantitative and qualitative paradigms, which may include design, execution, data collection, analysis, and interpretation.9,20 Collaboration with a team of researchers possessing the required range of expertise can strengthen mixed methods research9,20; however, it may be worth noting that the expansion of the research team may consequently cause the research to require more time and resources to complete.9,35 Additionally, quantitative and qualitative timelines may not always align and thus result in one component of the mixed methods research design completing prior to the other research method, therefore inhibiting the seamless integration of findings.10 In addition to the time and resources required to execute mixed methods research, communicating the breadth and depth of findings from mixed methods may pose unique challenges.6,36 Publishing mixed methods research often requires creative ways to present methods and results effectively while adhering to length restrictions imposed on authors.10 Because of this, some authors elect to publish their quantitative methods and results separate from their qualitative methods and results, thus negating the impact of the mixed methods approach.6,10 When disseminating research, consideration should be given to the critical value of the combination of quantitative and qualitative methods as two types of data and methodologies that complement one another and effectively overcome their respective limitations.6,7
Conclusion Mixed methods research can equip pharmacy educators to expand the scope and reach of their research. It has implications across the full span of pharmacy education and provides unique opportunities for designing studies that more accurately and comprehensively address questions of interest. Blending quantitative and qualitative methods can
expand our understanding of a phenomenon, help us triangulate findings, and generate complementary results. Expanding our research toolkits to include mixed methods can inform efforts to conduct assessment, evaluation, and research in pharmacy education and is likely to broaden our impact across the health professions. Conflict of interest None. References 1. Roth MT, Mumper RJ, Singleton SF, et al. A renaissance in pharmacy education at the University of North Carolina at Chapel Hill. N C Med J. 2011;75(1):48–52. 2. Irby DM, Cooke M, O'Brien BC. Calls for reform of medical education by the Carnegie Foundation for the advancement of teaching: 1910 and 2010. Acad Med. 2010;85(2):220–227. 3. Speedie MK, Baldwin JN, Carter RA, Raehl CL, Yanchick VA, Maine LL. Cultivating habits of mind in the scholarly pharmacy clinician: report of the 2011–12 Argus Commission. Am J Pharm Educ. 2012;76(6):S3. 4. Chickering AW, Reisser L. Education and Identity. Vol 2.; 1993. 5. Kolb DA. Experiential Learning: Experience as The Source of Learning and Development. Prentice Hall, Inc; 1984, 20–38. 6. Creswell J, Klassen AC, Plano V, Smith KC. Best Practices for Mixed Methods Research in the Health Sciences. Bethesda, MD: National Institutes of Health; 2011. 7. Hadi MA, Alldred DP, Closs SJ, Briggs M. Mixed-methods research in pharmacy practice: basics and beyond (part 1). Int J Pharm Pract. 2013;21(5):341–345. 8. Tashakkori A, Creswell JW. Editorial: the new era of mixed methods. J Mix Methods Res. 2007;1(1):3–7. 9. Hadi MA, Alldred DP, Closs SJ, Briggs M. Mixed-methods research in pharmacy practice: recommendations for quality reporting. Part 2. Int J Pharm Pract. 2014;22(1):96–100. 10. Bryman A. Barriers to integrating quantitative and qualitative research. J Mix Methods Res. 2007;1(1):8–22. 11. Muijs D. Doing Quantitative Research in Education With SPSS, vol 27. SAGE Publications; 2005. 12. Creswell JW. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. SAGE Publications; 2013. 13. McLaughlin JE, Dean MJ, Mumper RJ, Blouin RA, Roth MT. A roadmap for educational research in pharmacy. Am J Pharm Educ. 2013;77(10): Article 218. 14. Bourgeault I, De Vries RG, Dingwall R. The SAGE Handbook of Qualitative Methods in Health Research [electronic resource]. 2010. 15. Denzin N, Lincoln YS. The SAGE Handbook of Qualitative Research. Thousand Oaks, CA: SAGE Publications; 2005. 16. Creswell JW. Qualitative Inquiry and Research Design: Choosing Among Five Approaches. Thousand Oaks, CA: SAGE Publications; 2007. 17. LeCompte MD, Schensul JJ. Designing and Conducting Ethnographic Research. Altamira: Rowman; 2010. 18. Marshall C, Rossman GB. Designing Qualitative Research. SAGE Publications; 2014.
J.E. McLaughlin et al. / Currents in Pharmacy Teaching and Learning 8 (2016) 715–721 19. Hatch J. Doing Qualitative Research in Education Settings. SUNY Press; 2002. 20. Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: SAGE Publications; 2011. 21. Cohen D, Crabtree B. Qualitative research guidelines project. Robert Wood Johnson Foundation. Available at: 〈http://www.qualres.org/ HomeLinc-3684.html〉. Published 2006. Accessed June 1, 2016. 22. Mertens D. Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods. Thousand Oaks, CA: SAGE Publications; 2010. 23. Guba EG, Lincoln YS. Fourth Generation Evaluation. Thousand Oaks, CA: SAGE Publications; 1989. 24. O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research. Acad Med. 2014;89(9):1245–1251. 25. Onwuegbuzie AJ, Johnson RB. The validity issue in mixed research. Res Sch. 2006;13(1):48–63. 26. Bergman MM. Advances in mixed methods research: Theories and applications. Thousand Oaks, CA: SAGE Publications; 2008. 27. Greene JC, Caracelli VJ, Graham WF. Toward a conceptual framework for mixed-method evaluation designs. Educ Eval Policy Anal. 1989;11(3):255–274. 28. Wagner KD, Davidson PJ, Pollini RA, Strathdee SA, Washburn R, Palinkas LA. Reconciling incongruous qualitative and quantitative findings in mixed methods research: Exemplars from research with drug using populations. Int J Drug Policy. 2012;23(1):54–61. 29. Pinelli NR, McLaughlin JE, Chen S-L, et al. Improved organizational outcomes associated with incorporation of early clinical experiences for second-year student pharmacists at an academic medical center. J Pharm Pract. 2015. http://dx. doi.org/10.1177/0897190015585765. 30. McLaughlin JE, Amerine LB, Chen S-L, et al. Early clinical experiences for second-year student pharmacists at an academic medical center. Am J Pharm Educ. 2015;79(9): Article 139. 31. Payakachat N, Gubbins PO, Ragland D, et al. Academic helpseeking behavior among student pharmacists. Am J Pharm Educ. 2013;77(1): Article 7.
721
32. Johnson CL, Crawford SY, Lin S-J, Salmon JW, Smith MM. Neighborhood geographical factors and the presence of advanced community pharmacy practice sites in Greater Chicago. Am J Pharm Educ. 2009;73(1): Article 8. 33. Khanova J, McLaughlin JE, Rhoney DH, Roth MT, Harris S. Student perceptions of a flipped pharmacotherapy course. Am J Pharm Educ. 2015;79(9): Article 140. 34. Plano Clark VL. The adoption and practice of mixed methods: U.S. trends in federally funded health-related research. Qual Inq. 2010;16(6):428–440. 35. Ivankova NV. Using mixed-methods sequential explanatory design: from theory to practice. Field Methods. 2006;18(1):3–20. 36. Stange KC, Crabtree BF, Miller WL. Publishing multimethod research. Ann Fam Med. 2006;4(4):292–294. 37. Clandinin DJ, Connelly FM. Narrative Inquiry: Experience and Story in Qualitative Research. Jossey-Bass; 2004. 38. Moustakas C. Phenomenological Research Methods. Thousand Oaks, CA: SAGE Publications; 1994. 39. Charmaz K. Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. Thousand Oaks, CA: SAGE Publications; 2006. 40. Atkinson P, Delamont S, Coffey A. Key Themes in Qualitative Research: Continuities and Changes. Altamira: Rowman; 2004. 41. Stake RE. The Art of Case Study Research. Thousand Oaks, CA: SAGE Publications; 1995. 42. Johnson RB, Onwuegbuzie AJ. Mixed methods research: a research paradigm whose time has come. Educ Res. 2004;33(7): 14–26. 43. Small ML. How to conduct a mixed methods study: recent trends in a rapidly growing literature. Annu Rev Sociol. 2011;37(1):57–86. 44. McMillan J. Educational Research: Fundamentals for the Consumer. New York, NY: Harper Collins College Publishers; 1996. 45. Teddlie C, Tashakkori A. Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. Thousand Oaks, CA: SAGE Publications; 2009. 46. Yin RK. Case Study Research: Design and Methods. Thousand Oaks, CA: SAGE Publications; 2013.
Currents in Pharmacy Teaching and Learning 8 (2016) 715–721
Methodology Matters
http://www.pharmacyteaching.com
Mixed methods: Expanding research methodologies in pharmacy education Jacqueline E. McLaughlin, PhD, MS*, Antonio A. Bush, PhD, MS, Jacqueline M. Zeeman, PharmD Division of Practice Advancement and Clinical Education, UNC Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC
Abstract Mixed methods research combines quantitative and qualitative research to address a research question. Researchers engaging in mixed methods can establish a more complete understanding of an issue, triangulate findings, develop a complementary picture, identify trends, and validate results. Using mixed methods research may provide a more accurate and comprehensive reflection of educational experiences and outcomes across the span of pharmacy, including admissions, experiential education, and faculty development. Expanding our research toolkit to include multiple approaches for collecting, analyzing, and applying quantitative and qualitative data are likely to strengthen educational research in pharmacy education and broaden our impact across the health professions. After reviewing this article, the reader should be able to define mixed methods research, provide recommendations for its use, identify applications for pharmacy education, and describe the implications and limitations of the mixed methods approach. r 2016 Elsevier Inc. All rights reserved.
Keywords: Research methods; Mixed methods; Qualitative research; Quantitative research; Triangulation; Research design
Our Situation, issue, or problem A growing body of literature in health care and education point to the challenges of providing students with opportunities to develop critical workplace skills (e.g., adaptability, collaboration, communication, and initiative).1–3 In many cases, these skills are best taught and assessed in the context of experiential education; however, researchers interested in examining student development during experiential education are faced with a number of challenges, including small sample sizes (e.g., n o 30) and * Corresponding author: Jacqueline E. McLaughlin, PhD, MS, Division of Practice Advancement and Clinical Education, UNC Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599. E-mail: [email protected] http://dx.doi.org/10.1016/j.cptl.2016.06.015 1877-1297/r 2016 Elsevier Inc. All rights reserved.
complex construct measurement. As an example, my research team was recently tasked with developing a new experiential education program and evaluating its impact on the professional identity development of students. In higher education, identity development is articulated by a number of distinct theories and can include complex constructs that are not easily reduced to a single metric or survey item such as developing competence, moving through autonomy toward interdependence, developing mature interpersonal relationships, and developing purpose.4 Further, some experiential education theories articulate the role of reflection as a key element of creating knowledge.5 While quantitative data may be useful in this situation, quantitative data alone will not provide the depth and richness of data (e.g., reflection) necessary to fully understand student development. Along the same lines,
716
J.E. McLaughlin et al. / Currents in Pharmacy Teaching and Learning 8 (2016) 715–721
qualitative data may not be amenable to systematic comparisons or hypothesis testing (e.g., limited by smaller sample sizes and analyzed qualitatively) and may fail to generate the type of statistical feedback desired by experiential educators to inform practice moving forward. For these reasons, we needed the distinct advantages of quantitative and qualitative methods and chose to employ both approaches to conduct a mixed methods study. Methodological literature review In general, mixed methods research can be defined as the combination of both quantitative and qualitative research efforts to address one research question (Table 1).6–8 The combination of these two approaches can answer a specific research question more comprehensively, increase the tools available to conduct a research study, and broaden the scope of research possible.7,9 Through integration of quantitative and qualitative research, investigators are able to maximize the strengths and minimize the weaknesses of each individual approach.6,7 The sequence and dominance of quantitative and qualitative efforts may vary among mixed method approaches; however, the integration of the mixed methods results, even if they initially appear to disagree, can strengthen the overall research design to overcome the respective limitations of the individual quantitative and qualitative designs.6,10 To better understand mixed methods design, it might be helpful to review quantitative and qualitative research in more detail. Quantitative research utilizes numerical data to explain a particular phenomenon, such as performance, attitudes, or
engagement.11,12 It originates from positivism, which is the idea that a single reality or truth can be discovered through the objective study of numeric data and variables. A wide range of sources can be utilized to collect numeric data including surveys, structured interviews, observations, performance indicators and assessments, institutional information, and national databases.13 In some cases, these data may be collected as qualitative data (e.g., interviews) and reduced to numeric quantities for the purposes of analysis. Quantitative data are analyzed using statistical methods that describe variables (i.e., descriptive statistics) and draw inferences of a sample to the population at large (i.e., inferential statistics).11–13 Quantitative methods use two main research design approaches: experimental design and non-experimental design.11 Experimental design involves the random assignment of participants to either the experimental group or the control group to test a hypothesis under controlled conditions.11 Quasi-experimental designs are utilized when random assignment is not feasible but continue to test a hypothesis under controlled conditions.11 In contrast, nonexperimental design (e.g., observational study) does not involve the manipulation of study variables nor is the setting controlled. In pharmacy education, research designs vary widely and include the quantitative research traditions of epidemiologic, correlational, and psychometric studies in addition to randomized control. Statistical methods for analyzing quantitative data can range in complexity from simple correlations to multilevel modeling.13 Although health care researchers have traditionally employed quantitative methods, qualitative approaches have
Table 1 Comparison of research approaches Quantitative research
Qualitative research
Mixed methods research
Understand a specific population or phenomenon Text, pictures, video, audio, or other multimedia
Examine a question from a quantitative and qualitative perspective Both quantitative and qualitative data types
Observations, interviews, focus groups, documents
Uses a mix of qualitative and quantitative data sources
Coding and document analysis
Triangulation; integrating data analysis methods
Quality criteria
Determine relationships and describe variables; test hypotheses Numeric or reduced to numeric quantities for purposes of analysis Surveys, records, learning assessments (e.g. tests), measurements Statistical and numerical analyses (descriptive and inferential statistics) Validity, reliability, objectivity
Common applications
Description; generalization
Credibility, dependability, conformability, transferability. Theme identification; theory development; case analysis
Limitations
Reduced power for small sample sizes; difficult to measure complex constructs; lack of deep, rich description
Some combination of quantitative and qualitative criteria Triangulation, complementarity, development initiation, expansion Lack of resources; requires skills in both approaches; publication word limitations hindering thorough method and results descriptions
Purpose
Data characteristics
Common data sources
Data analysis
Lack of statistical power and generalizability; time intensive; potentially misunderstood by audience due to lack of training
J.E. McLaughlin et al. / Currents in Pharmacy Teaching and Learning 8 (2016) 715–721
been shown to complement or serve as a viable alternative for statistical methods that may not fully address the complexities associated with health behavior, organizations, practice, and policy.14 Qualitative research can be defined as a method that emphasizes studying people and places in a natural setting and interpreting phenomena or experiences in a way that is most consistent with participant understanding.15,16 It originates from constructivism, which is the idea that groups construct knowledge for one another and that there are as many realities as there are social constructions. Key characteristics of the qualitative research process include16–19 (1) multiple sources of data are collected and analyzed to identify themes, (2) the research emphasizes the participants' perspectives and subjective views of the phenomena being studied, (3) the research design can be fluid since the data collection process may change or shift when entering the field, (4) the research methods should include a saturation process in which data are collected and reviewed until new information does not add to the understanding of the issues being studied, and (5) the researcher recognizes that multiple interpretations of an issue may emerge from various stakeholders (e.g., subjectivity of the researcher, participants, and reader). Qualitative research is commonly used to explore the complex understanding of an issue, empower individuals to share their stories, de-emphasize power relationships, understand the context or setting in which participants face challenges, follow-up on quantitative findings, develop theories, and when a quantitative approach is impractical or inappropriate.14,16 This approach reduces the distance between the researcher and participant and provides depth about contexts, values, opinions, and behaviors that is often missing in quantitative research. Five traditional qualitative designs are narrative research, phenomenology, grounded theory, ethnography, and case study (Table 2).
717
Critical aspects of research include implementing an appropriate research design, collecting useful and accurate data, using the right analysis tools, and interpreting the results correctly. Integrating qualitative and quantitative research can optimize the strengths and address the limitations of each approach and provides a pragmatic alternative to two distinct paradigms that drive the design and implementation of educational research (i.e., positivism and constructivism). The purpose of this article is to describe mixed methods research as a means of expanding the scope and impact of educational research in pharmacy. The following sections provide recommendations for its use, identify applications for pharmacy education, and describe the implications and limitations of the mixed methods approach.
Our recommendations and their applications In the situation described at the start of this article (i.e., my research team was recently tasked with developing a new experiential education program and evaluating its impact on the professional identity development of students), mixed methods were used for a number of reasons. First, our sample size was relatively small (about 20 students). This meant that while we planned to collect valuable and relevant information in the form of survey items (e.g., self-efficacy), there was a risk that the study was underpowered for statistical analyses. As such, we recognized during the design of our study that qualitative data could supplement, provide insight into, and potentially support our limited quantitative findings (i.e., triangulation). Second, identity development is a complex educational construct that is not easily reduced to items measured on a survey instrument or questionnaire. For this reason, we asked students to turn in reflection statements that provided rich qualitative data about their experiences in the program. Third, this study was implemented as a pilot with plans to scale the program out to a much larger number of students in the subsequent academic year. While we needed data for the purposes of the identity development research, we also
Table 2 Five qualitative approaches and their characteristicsa Approaches to qualitative research Characteristics
Narrative research
Phenomenology
Grounded theory
Ethnography
Case study
Focus
The life of an individual
Developing a theory
A culture-sharing group
Unit of analysis
One or more individuals
A group that shares the same culture
Clandinin and Connelly37
A process, action, or interaction involving many individuals Charmaz39
In-depth description and analysis of a case(s) An event, program, activity, group, or individual
Resources
The essence of the experience or phenomenon Several individuals that have shared the experience Moustakas38
Atkinson et al.40
Stake41 and Yin46
a
Adapted from Creswell et al.16
718
J.E. McLaughlin et al. / Currents in Pharmacy Teaching and Learning 8 (2016) 715–721
needed to collect program evaluation data about the practical and logistical elements of the experience to inform future iterations. These evaluation data were also best suited for a mixed methods approach, with survey items (i.e., quantitative data) and open-ended text survey boxes (i.e., qualitative data) used to collect feedback from students at the conclusion of the program. We recommend that any researcher using mixed methods consider the following criteria to guide the design and implementation of a study: (1) timing, (2) weighting, (3) mixing, and (4) theorizing.20 Specifically, the researcher must determine the timing of data collection, which may be sequential (i.e., in phases) or concurrent (i.e., at the same time). It may be problematic, for example, to collect quantitative and qualitative data sequentially due to scheduling and/or other limiting factors. The weighting (or priority) that the researcher gives to the qualitative and quantitative approach may be equally or unequally distributed depending on a variety of factors, including researcher interests, research team skills, and the study audience. Mixing describes the extent to and the point at which the researcher merges the qualitative and quantitative data. Mixed data may be connected to inform a second phase of a study, integrated by transforming qualitative data into quantitative data (e.g., word counts, theme counts, and descriptive data), or used to support the findings of one another. Theorizing involves the process of using a theory to frame or guide the research design. Theoretical frameworks are commonly used in educational research to inform the research questions, participants involved, data collection, and implications.13 Further, we recommend that the quality indicators for quantitative and qualitative approaches be taken into consideration when designing a mixed methods study.20 Validity, reliability, and objectivity are generally used as criteria for determining the quality of quantitative research. Validity reflects the extent to which a measure actually represents what it claims to measure, reliability concerns the degree to which a finding is stable and consistent, and objectivity indicates the extent to which the instrument being used is resistant to influence by the beliefs and biases of researchers. Related but distinct primary criteria used to evaluate the trustworthiness of qualitative research include credibility, dependability, conformability, transferability, and authenticity.18,21,22 Credibility reflects confidence of the findings and can be promoted by prolonged and persistent engagement in the field, sharing data and interpretations of the data with participants for confirmation (i.e., member checking), discussing the data with critical colleagues (i.e., peer debriefing), and using multiple sources to verify the data (i.e., triangulation).18,21,22 Dependability refers to the stability or repeatability of the research process.22 Since qualitative research can be fluid, any changes in methods during the study should be tracked using extensive notes to demonstrate the quality of the research.22,23 Conformability involves minimizing the researcher's subjective influence on the research and can
be encouraged by providing a transparent description of the study methods, triangulating the data, and practicing reflexivity.21,22 Transferability concerns the extent to which the research findings are applicable in other contexts21,22 and can be enhanced through the use of rich, in-depth descriptions (e.g., time, place, and context) and/or using multiple case studies. Authenticity reflects the extent to which participants' experiences are objectively reported. O'Brien et al.24 defines additional standards for the conduct and reporting of qualitative research in the health professions. The criteria used to determine the quality of quantitative and qualitative research, though distinct and not without contention, have been firmly established. However, these criteria become more complex when two different sets of procedures are merged to conduct a single study. In recent years, interest in identifying and addressing quality criteria in mixed methods research has grown25,26 and researchers may find these new mixed methods criteria useful. At the very least, it is recommended that researchers design and implement mixed methods research with consideration to the quality criteria for both approaches. For example, indicators for validity and reliability could be examined and reported with quantitative data while methods for promoting credibility and conformability could be described with qualitative data. Applications: A growing body of literature demonstrates the use of mixed methods in pharmacy and health professions education. Examples in pharmacy education include studies of experiential education, academic help-seeking behavior, and classroom experiences.29–31 Johnson et al.32 for example, used mixed methods research to determine experiential learning opportunities and identify barriers to rotation sites. In this study, quantitative data were used to build a multivariate logistic regression model while qualitative data from faculty interviews were used to further elucidate site placement factors. Mixed methods were also employed by Khanova et al.33 to examine student perceptions of a flipped pharmacotherapy course. Precourse and post-course survey data were collected, analyzed, and further explored through qualitative themes derived from text responses on the survey. As another example, Pinelli et al.29 examined health system pharmacy practice outcomes associated with student pharmacists involved in medication reconciliation. Quantitative data were used to describe patient characteristics and student productivity statistics while qualitative analysis was used to examine pharmacist perceptions of student contributions to the health care system. Clearly, a wide range of research contexts and problems in pharmacy education can be approached with a mixed methods design. From admissions to experiential education to alumni engagement, mixed methods provides unique opportunities and approaches for designing studies that more accurately and comprehensively reflect educational environments and outcomes. Research questions most suitable for mixed method approaches are those in which a quantitative or qualitative approach alone are inadequate to develop a complete understanding of the research question.20 Greene et al.27 identified five specific applications for mixed methods research—(1)
J.E. McLaughlin et al. / Currents in Pharmacy Teaching and Learning 8 (2016) 715–721
triangulation, (2) complementarity, (3) development, (4) initiation, and (5) expansion. Triangulation increases validity by seeking to converge and confirm results from quantitative and qualitative methods.9,27 In our situation, the qualitative findings converged with the quantitative findings to suggest that the program contributed to development of professional identity. Complementarity uses the research from one method to enhance, illustrate, or further clarify the results of the other, which can increase interpretability and validity of the individual methods.27 Development refers to the utilization of one method's results to inform the other method.27 For example, focus group interviews (or qualitative methods) may help to inform the development of a questionnaire (or quantitative methods) to be administered to the community.9 The primary purpose of initiation is to uncover inconsistencies and contradictions within one research method to influence the research question or results of the other research method.27 Finally, expansion seeks to extend the breadth and depth of inquiry by using different research methods to inquire about different components of the research.27 Further, mixed methods research presents a unique opportunity to test a hypothesis among a large group while adding information about the individual participant.20 Wagner et al.,28 for example, utilized a mixed methods approach in pharmacy practice research regarding drug using populations and noted that they “achieved a more nuanced understanding of the issues than may have been achieved through the use of a single method.” Since small samples may be underpowered (i.e., unable to detect a statistical difference), quantitative data can provide important numeric
719
information while qualitative data can provide rich, in-depth information that enables the researcher to more fully understand the studied phenomenon. Further, activities and outcomes associated with teaching and learning can often embody important social, cultural, and educational values that influence the way in which data are understood and interpreted. As such, complex constructs that are not easily measurable may benefit from a mixed methods approach that enables triangulation of statistical findings with important contextual and value-based themes.
Potential impact Pharmacy educators are uniquely positioned to examine critical research questions that are not only of importance in pharmacy and pharmacy education, but of increasing importance and interest to larger audiences within health professions and higher education. Expanding research toolkits to include approaches to collecting, analyzing, and applying quantitative and qualitative data are likely to strengthen educational research in pharmacy education, contribute to associated research in other disciplines, and broaden our impact across the health professions. Researchers utilizing mixed methods can establish a more complete understanding of a problem, develop a complementary picture by comparing and validating results, identify trends, and examine the process or experience in addition to the outcome.34 Box 1 provides additional readings for the interested reader.
Box 1 Recommended Resources Journal articles • Johnson RB, Onwuegbuzie AJ. Mixed Methods Research: A Research Paradigm Whose Time Has Come. Educ Res. 2004;33(7):14-26. http://dx.doi.org/10.3102/0013189X033007014.42 o This article provides a review of mixed methods research and an eight step process to explain mixed methods research. • Onwuegbuzie AJ, Johnson RB. The Validity Issue in Mixed Research. University of South Alabama. Res Sch. 2006;13(1):48-63.25 o In recognizing that validy discussions in mixed methods research are in their infancy, this article introduces a framework termed legitimation to evaluate mixed methods research. • Small ML. How to Conduct a Mixed Methods Study: Recent Trends in a Rapidly Growing Literature. Annu Rev Sociol. 2011;37(1):5786. http://dx.doi.org/10.1146/annurev.soc.012809.102657.43 o This articles provides reviews and critiques of general approaches to mixed methods research. Books • Creswell, JW. Research design: Qualitative, quantitative, and mixed methods approaches. Sage Publications, 2013.12 o This book provides in depth analysis on the frameworks, processes, and approaches utilized in designing qualitative, quantitative, and mixed methods research. • McMillan JH. Educational Research: Fundamentals for the Consumer. 6th edition. New York: Pearson; 2012.44 o This book introduces basic research principles for conducting and evaluating research and includes several useful aids and author reflections. • Teddlie C, Tashakkori A. Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. Sage Publications; 2009.45 o This book details all aspects of mixed methods research from research design to analysis and reporting.
720
J.E. McLaughlin et al. / Currents in Pharmacy Teaching and Learning 8 (2016) 715–721
In addition, mixed methods research provides additional mechanisms for conducting assessment and evaluation intended for programmatic improvement. In an era of increasing accountability at all levels of higher education, colleges and schools of pharmacy must take measurable steps to ensure that processes are in place to better utilize data to inform planning. Since educational environments often preclude experimental design and include complex constructs and phenomena, mixed methods can provide assessment professionals with diverse strategies and data sources for understanding outcomes at a local level. This might include course evaluations that use Likert-scale items and open-ended text boxes, student feedback in the form of surveys and focus groups, and curriculum mapping that includes syllabus review and faculty surveys. Limitations: Despite the apparent benefits of mixed methods research, there are challenges and limitations associated with its implementation. Perhaps the biggest challenge is acquiring the necessary knowledge and skill for both quantitative and qualitative paradigms, which may include design, execution, data collection, analysis, and interpretation.9,20 Collaboration with a team of researchers possessing the required range of expertise can strengthen mixed methods research9,20; however, it may be worth noting that the expansion of the research team may consequently cause the research to require more time and resources to complete.9,35 Additionally, quantitative and qualitative timelines may not always align and thus result in one component of the mixed methods research design completing prior to the other research method, therefore inhibiting the seamless integration of findings.10 In addition to the time and resources required to execute mixed methods research, communicating the breadth and depth of findings from mixed methods may pose unique challenges.6,36 Publishing mixed methods research often requires creative ways to present methods and results effectively while adhering to length restrictions imposed on authors.10 Because of this, some authors elect to publish their quantitative methods and results separate from their qualitative methods and results, thus negating the impact of the mixed methods approach.6,10 When disseminating research, consideration should be given to the critical value of the combination of quantitative and qualitative methods as two types of data and methodologies that complement one another and effectively overcome their respective limitations.6,7
Conclusion Mixed methods research can equip pharmacy educators to expand the scope and reach of their research. It has implications across the full span of pharmacy education and provides unique opportunities for designing studies that more accurately and comprehensively address questions of interest. Blending quantitative and qualitative methods can
expand our understanding of a phenomenon, help us triangulate findings, and generate complementary results. Expanding our research toolkits to include mixed methods can inform efforts to conduct assessment, evaluation, and research in pharmacy education and is likely to broaden our impact across the health professions. Conflict of interest None. References 1. Roth MT, Mumper RJ, Singleton SF, et al. A renaissance in pharmacy education at the University of North Carolina at Chapel Hill. N C Med J. 2011;75(1):48–52. 2. Irby DM, Cooke M, O'Brien BC. Calls for reform of medical education by the Carnegie Foundation for the advancement of teaching: 1910 and 2010. Acad Med. 2010;85(2):220–227. 3. Speedie MK, Baldwin JN, Carter RA, Raehl CL, Yanchick VA, Maine LL. Cultivating habits of mind in the scholarly pharmacy clinician: report of the 2011–12 Argus Commission. Am J Pharm Educ. 2012;76(6):S3. 4. Chickering AW, Reisser L. Education and Identity. Vol 2.; 1993. 5. Kolb DA. Experiential Learning: Experience as The Source of Learning and Development. Prentice Hall, Inc; 1984, 20–38. 6. Creswell J, Klassen AC, Plano V, Smith KC. Best Practices for Mixed Methods Research in the Health Sciences. Bethesda, MD: National Institutes of Health; 2011. 7. Hadi MA, Alldred DP, Closs SJ, Briggs M. Mixed-methods research in pharmacy practice: basics and beyond (part 1). Int J Pharm Pract. 2013;21(5):341–345. 8. Tashakkori A, Creswell JW. Editorial: the new era of mixed methods. J Mix Methods Res. 2007;1(1):3–7. 9. Hadi MA, Alldred DP, Closs SJ, Briggs M. Mixed-methods research in pharmacy practice: recommendations for quality reporting. Part 2. Int J Pharm Pract. 2014;22(1):96–100. 10. Bryman A. Barriers to integrating quantitative and qualitative research. J Mix Methods Res. 2007;1(1):8–22. 11. Muijs D. Doing Quantitative Research in Education With SPSS, vol 27. SAGE Publications; 2005. 12. Creswell JW. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. SAGE Publications; 2013. 13. McLaughlin JE, Dean MJ, Mumper RJ, Blouin RA, Roth MT. A roadmap for educational research in pharmacy. Am J Pharm Educ. 2013;77(10): Article 218. 14. Bourgeault I, De Vries RG, Dingwall R. The SAGE Handbook of Qualitative Methods in Health Research [electronic resource]. 2010. 15. Denzin N, Lincoln YS. The SAGE Handbook of Qualitative Research. Thousand Oaks, CA: SAGE Publications; 2005. 16. Creswell JW. Qualitative Inquiry and Research Design: Choosing Among Five Approaches. Thousand Oaks, CA: SAGE Publications; 2007. 17. LeCompte MD, Schensul JJ. Designing and Conducting Ethnographic Research. Altamira: Rowman; 2010. 18. Marshall C, Rossman GB. Designing Qualitative Research. SAGE Publications; 2014.
J.E. McLaughlin et al. / Currents in Pharmacy Teaching and Learning 8 (2016) 715–721 19. Hatch J. Doing Qualitative Research in Education Settings. SUNY Press; 2002. 20. Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: SAGE Publications; 2011. 21. Cohen D, Crabtree B. Qualitative research guidelines project. Robert Wood Johnson Foundation. Available at: 〈http://www.qualres.org/ HomeLinc-3684.html〉. Published 2006. Accessed June 1, 2016. 22. Mertens D. Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods. Thousand Oaks, CA: SAGE Publications; 2010. 23. Guba EG, Lincoln YS. Fourth Generation Evaluation. Thousand Oaks, CA: SAGE Publications; 1989. 24. O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research. Acad Med. 2014;89(9):1245–1251. 25. Onwuegbuzie AJ, Johnson RB. The validity issue in mixed research. Res Sch. 2006;13(1):48–63. 26. Bergman MM. Advances in mixed methods research: Theories and applications. Thousand Oaks, CA: SAGE Publications; 2008. 27. Greene JC, Caracelli VJ, Graham WF. Toward a conceptual framework for mixed-method evaluation designs. Educ Eval Policy Anal. 1989;11(3):255–274. 28. Wagner KD, Davidson PJ, Pollini RA, Strathdee SA, Washburn R, Palinkas LA. Reconciling incongruous qualitative and quantitative findings in mixed methods research: Exemplars from research with drug using populations. Int J Drug Policy. 2012;23(1):54–61. 29. Pinelli NR, McLaughlin JE, Chen S-L, et al. Improved organizational outcomes associated with incorporation of early clinical experiences for second-year student pharmacists at an academic medical center. J Pharm Pract. 2015. http://dx. doi.org/10.1177/0897190015585765. 30. McLaughlin JE, Amerine LB, Chen S-L, et al. Early clinical experiences for second-year student pharmacists at an academic medical center. Am J Pharm Educ. 2015;79(9): Article 139. 31. Payakachat N, Gubbins PO, Ragland D, et al. Academic helpseeking behavior among student pharmacists. Am J Pharm Educ. 2013;77(1): Article 7.
721
32. Johnson CL, Crawford SY, Lin S-J, Salmon JW, Smith MM. Neighborhood geographical factors and the presence of advanced community pharmacy practice sites in Greater Chicago. Am J Pharm Educ. 2009;73(1): Article 8. 33. Khanova J, McLaughlin JE, Rhoney DH, Roth MT, Harris S. Student perceptions of a flipped pharmacotherapy course. Am J Pharm Educ. 2015;79(9): Article 140. 34. Plano Clark VL. The adoption and practice of mixed methods: U.S. trends in federally funded health-related research. Qual Inq. 2010;16(6):428–440. 35. Ivankova NV. Using mixed-methods sequential explanatory design: from theory to practice. Field Methods. 2006;18(1):3–20. 36. Stange KC, Crabtree BF, Miller WL. Publishing multimethod research. Ann Fam Med. 2006;4(4):292–294. 37. Clandinin DJ, Connelly FM. Narrative Inquiry: Experience and Story in Qualitative Research. Jossey-Bass; 2004. 38. Moustakas C. Phenomenological Research Methods. Thousand Oaks, CA: SAGE Publications; 1994. 39. Charmaz K. Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. Thousand Oaks, CA: SAGE Publications; 2006. 40. Atkinson P, Delamont S, Coffey A. Key Themes in Qualitative Research: Continuities and Changes. Altamira: Rowman; 2004. 41. Stake RE. The Art of Case Study Research. Thousand Oaks, CA: SAGE Publications; 1995. 42. Johnson RB, Onwuegbuzie AJ. Mixed methods research: a research paradigm whose time has come. Educ Res. 2004;33(7): 14–26. 43. Small ML. How to conduct a mixed methods study: recent trends in a rapidly growing literature. Annu Rev Sociol. 2011;37(1):57–86. 44. McMillan J. Educational Research: Fundamentals for the Consumer. New York, NY: Harper Collins College Publishers; 1996. 45. Teddlie C, Tashakkori A. Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. Thousand Oaks, CA: SAGE Publications; 2009. 46. Yin RK. Case Study Research: Design and Methods. Thousand Oaks, CA: SAGE Publications; 2013.