- Email: [email protected]
Design-based research: Connecting theory and practice in pharmacy educational intervention research
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
Contents lists available at ScienceDirect
Currents in Pharmacy Teaching and Learning journal homepage: www.elsevier.com/locate/cptl
Methodology Matters
Design-based research: Connecting theory and practice in pharmacy educational intervention research
T
Michael D. Wolcotta,b, Nikki G. Lobczowskia,b, Kayley Lyonsa,b, ⁎ Jacqueline E. McLaughlina, a b
University of North Carolina, Eshelman School of Pharmacy, 329 Beard Hall, Chapel Hill, NC 27599, United States University of North Carolina, School of Education, 3500 Peabody Hall, Chapel Hill, NC 27599, United States
ARTICLE INFO
ABSTRACT
Keywords: Research methods Research design Design-based research Mixed methods Collaboration
Our situation: Interventional research in pharmacy education includes the study of complex challenges that can be difficult to navigate. Design-based research (DBR), is a systematic and iterative approach to interventional research that is attentive to the practical and theoretical contributions to education. Practical contributions include the creation of novel solutions to complex problems that improve learning while theoretical contributions include refining our understanding of context-specific learning. In this paper, we describe how we addressed challenges associated with student collaboration in pharmacy education by applying DBR to bridge theory and practice. Methodological literature review: DBR is characterized as authentic, contextually aware, collaborative, theoretically focused, methodologically diverse, practical, iterative, and operation-oriented. DBR includes three iterative phases: (1) analysis and exploration, (2) design and construction, and (3) evaluation and reflection. Our recommendations and their applications: To integrate DBR into interventional research, scholars should work collaboratively with diverse teams of experts. DBR also requires extensive planning, a toolkit of expansive research methodologies, and attention to practical and theoretical considerations. Finally, scholars should share their work as often as possible and engage in creative exercises to promote innovative solutions to challenges in education. Potential impact: DBR offers an approach to generate practical, theoretical, and scholarly contributions to pharmacy education research. In summary, DBR can aid pharmacy educational scholars by using a flexible, iterative, and systematic process to generate novel and creative solutions to complex problems.
Our situation Pharmacy educational research is at a precipice—there is significant discourse about a need to advance pharmacy education by incorporating evidence-based educational practices.1,2 Consequently, there is substantial interest in conducting high-quality educational intervention research intended to inform practice while producing meaningful outcomes for our learners.3,4 In the health professions, however, we tend to focus on if our interventions influenced learning outcomes—what is missing is a better understanding of how our interventions have influenced learning as this can inform the design of more effective interventions in the
⁎
Corresponding author. E-mail address: [email protected] (J.E. McLaughlin).
https://doi.org/10.1016/j.cptl.2018.12.002
1877-1297/ © 2018 Elsevier Inc. All rights reserved.
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
future.5 Design-based research (DBR) is an iterative approach to designing, implementing, evaluating, and improving educational interventions.6 DBR can help scholars better understand various design elements of an intervention and how they influence learning experiences and outcomes and address this gap in pharmacy education scholarship.7 For example, questions asked when using DBR might include: (a) what is happening between the point when a learner enters a learning experience and when they exit?, (b) how do learners interact with the materials and information we give them?, (c) how can the learning process be optimized?, or (d) how do our findings build on our understanding of learning in this context? Specifically, the aim of DBR is to enhance educational intervention research through practical contributions (i.e. generating novel solutions to complex problems and improving learning through high-quality interventions) and theoretical contributions (i.e. refining our understanding of how individuals learn in a specific context and offering strategies to optimize learning) to the literature. DBR is particularly relevant to pharmacy education as it can aid education scholars in measuring the effectiveness of interventions in addition to advancing our understanding of how learning occurs within pharmacy settings. The iterative nature of DBR emphasizes how research can continuously build on itself and articulates how future research is explicitly informed from previous research. Our current approaches tend to focus exclusively on measuring intervention effectiveness though changes in surveys, questionnaires, test scores, or other readily accessible data, which can be insufficient.8 Although these methods are valuable, this approach provides limited information for interventional research because it lacks a deeper investigation into why we are or are not witnessing a change in learning in pharmacy practice. Consider, for example, the common challenge that we addressed in pharmacy education: how to effectively promote, measure, and improve learner collaboration (Box 1). Our previous approach to this challenge was often generic and lacked sufficient depth to understand what contributed to improvements in learning and how. Our simplistic approach would often include identifying strategies to promote collaboration based on empirical evidence and anecdotal expectations of learner collaboration in a course. Typically, our approach would not include collaboration with educational experts or students to generate novel solutions to the challenge. In addition, we were often tempted to use only surveys at the beginning and end of the course to measure the intervention's effectiveness based on faculty and student perceptions about collaboration or the efficacy of certain strategies introduced through our study. We noticed that if the study showed a statistically significant improvement in perceptions of collaboration, it would be difficult to ascertain how the strategies contributed effectively to improve collaboration or why perceptions changed. Alternatively, if the study showed no or minimal change, we would conclude the study was unsuccessful, not disseminate the findings, and try a different approach next year without learning from participants how the intervention could be improved to be more successful in the future. For example, students may have been disengaged because they did not know the strategies were intended to improve collaboration or the faculty members may not have implemented the strategies appropriately. DBR offers an approach to address the issues we witnessed in our scholarship. We found DBR could improve our research by supporting the creation of novel solutions to complex challenges, promoting collaboration with experts in education, and helping us understand how the intervention worked to impact learning outcomes. Overall, DBR has improved the quality and refinement of our interventions based on its iterative and systematic approach to conducting educational intervention research. In summary, this article outlines how DBR can be used to advance interventional pharmacy education research. Specifically, we describe throughout the article how we applied DBR to address the challenge of improving learner collaboration. The purpose of this article is to introduce pharmacy scholars to DBR, outline common components of the DBR process, and offer recommendations on how pharmacy scholars can integrate DBR into their research practice. We strongly encourage individuals interested in DBR to consult additional resources (Box 2) to obtain a more in-depth discussion of the DBR approach, the methodologies associated with it, and additional practical tips for the process. Methodological literature review DBR first originated as an approach to integrate design experiments to address theoretical questions about the nature of learning in real-life contexts.9,10 It has evolved into a research approach defined by eight characteristics.6,7,10–15 1 Authentic: The hallmark of DBR is that it takes place in real-life learning settings to capture how learning normally occurs instead of studying learning in a laboratory setting. 2 Contextually aware: Unlike experimental studies, in DBR scholars aim to identify, not necessarily control, all the variables that may influence outcomes; effects related to the student, teacher, and institution are described as they are expected to matter in realworld education practice and can be used to understand future research directions. 3 Collaborative: DBR recognizes the importance of working with others (e.g., learners, educators, scholars) who share expertise in fields such as design, education, and the learning context to ensure optimal design of interventions. 4 Theoretically focused: The goal of DBR is to create interventions aligned with our understanding of learning theories as well as using the results to support, derive, or refine theories as they are applied in the setting of interest. 5 Methodologically diverse: DBR incorporates a pragmatic approach to research design that incorporates both qualitative and quantitative data collection strategies (i.e. mixed methods) to gain a more holistic perspective of learning in real-life contexts. 6 Practical: Interventions created through DBR studies are sensitive to the learner's needs to address a specific problem while attending to constraints in the specified context such as limitations in resources and technology. 7 Iterative: DBR is a systematic process that has distinct cycles of analysis, creation, and evaluation to continuously improve 310
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Fig. 1. Sample model of iterations in DBR (adapted from McKenny & van den Akker).16 DBR = design-based research.
interventions and refine theories based on the findings at each stage. As opposed to waiting to test an intervention until it is perfect, DBR scholars test their designs early and often. Fig. 1 exemplifies how the cycles can vary throughout the research process, each informed by the previous findings. 8 Operational: The aim of DBR is to create and understand interventions so that they can be implemented in other settings and have a broader impact in educational research. The combination of these characteristics is what makes DBR unique; attention to each element promotes higher quality educational research that connects theory and interventions in real-life settings. Table 1 shows how these characteristics can be applied to the case example of studying collaboration among student pharmacists. These characteristics create the foundation for the overarching model of DBR that promotes the systematic approach to educational intervention research. DBR is similar to participatory action research in that both approaches identify real-world problems and generate ideas to improve conditions or educational practices.17 However, DBR can be distinguished from participatory action research by the use, development, and testing of either learning or implementation theories. Compared to the action research process, the DBR process includes an initial step to identify and apply education theory from previous literature to the proposed educational design. Then DBR researchers test the theoretically-guided interventions in order to impact practice and contribute to theory development. Also, DBR aims to integrate stakeholders as researchers and designers in the process.18 Teachers or clinicians, for example, in the DBR process are encouraged to engage in the research and design process to facilitate implementation of the intervention and ensure that the Table 1 Applying characteristics of DBR to the example case study on student collaboration. Characteristic
Example
Authentic Contextually aware
We studied existing collaboration and implementing the intervention within an ongoing course. We aimed to characterize how students interacted with the Collabucate application created by the research team, what the student and teacher needs were, and how collaborative learning took place throughout the course. We recruited learners, as well as experts in research methodologies, educational research, technology development, and pharmacy education. Our intervention design was based on the theory of socially-shared regulation of learning, which describes how individuals coconstruct knowledge.18 We included observation data from video recordings, qualitative data from focus groups and think-aloud protocols, and quantitative data from surveys and learning analytics/responses on the mobile application. Our intervention design and research were constrained by time and budget limits while adhering to instructor requests without disrupting the course. We involved at least three prototype iterations to constantly improve design. Our goal was to make the intervention sufficiently broad to implement in multiple courses and other health professions education programs.
Collaborative Theoretically focused Methodologically diverse Practical Iterative Operational
DBR = design-based research. 311
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Fig. 2. Model of design-based research (adopted from McKenny and Reeves).13
intervention includes the various perspectives of stakeholder groups. The focus on these components—theory and the role of practitioners in the research process—makes DBR distinct from other research approaches. A model of DBR DBR includes a compilation of methodologies that can be integrated into a systematic process to address complex challenges in practice settings. An important hallmark of DBR is that although it takes place in complex settings, it still relies on a systematic process to make informed decisions—even though it is inspired by creativity and novel solutions. There have been several models to describe DBR such as the one shown in Fig. 2 by McKenney and Reeves.13 This model of DBR includes three main phases that inform theory and practice in a flexible process. Of note, the model portrays DBR as a linear process; however, the approach is highly iterative and you may not necessarily complete each phase sequentially or in great depth before moving on to a different stage.13 Table 2 outlines how we applied each of these phases to the design and implementation of the research study on student collaboration. Analysis and exploration: In this phase, the goal is to define the problem, identify potential causes, explore the setting, and learn about the needs and wishes of the stakeholders. Analysis is focused on orienting the researchers to the setting, conducting cursory investigations, and often includes a literature review or discussion with subject matter experts. The exploration element includes a deeper dive into the setting, developing a better understanding of the stakeholders, and identifying potential constraints or considerations that may inform the solution to the challenge. It often includes interviews, observations, site visits, as well as meetings with stakeholders and research collaborators. Table 2 Applying the DBR model to the example case study on student collaboration. Phase
Example
Analysis and exploration
We conducted interviews with instructors and focus groups with students focused on problems in collaborative learning. Overall findings suggest that students lacked a collaboration skill set and were apprehensive to resolve issues in group work. Further, instructors valued collaboration skills but did not know how to teach them. We met to determine all the possible solutions to help students and instructors with this problem. Next, we rated each solution on its feasibility and impact. Finally, we took the solution that was the most feasible and impactful and created a prototype (i.e. the Collabucate application). We then tested the prototype with students, integrating their feedback into future versions. We conducted a pilot study with six groups of four to five pharmacy students, collecting video observations of their weekly group meetings and log data from the app. At the end of the semester, we also collected focus group data, having students describe successes, challenges, and areas for improvement with regards to the app. We then analyzed this data and identified changes for future iterations of the app. We began using Collabucate with six student groups and have since increased to class-wide implementation (150 students). Future implementations will include different pharmacy courses (i.e. including different instructors, content, students, course structure) and different (i.e. non-pharmacy) contexts. We started with three design principles from the literature and have derived another design principle. This research has helped shape our theoretical understanding of how groups regulation their thinking, motivation, and emotions, as well as the tensions and interactions that emerge in this type of collaborative learning environment. We started with a minimally viable working prototype of the Collabucate app. We now have a functional application that has been through two iterations and implementations. Perhaps our most important addition is a group page that prompts students to discuss the results. Looking toward the future, we have plans to create an integrate a teacher dashboard and better personalize certain aspects of the app to include new challenges that we did not originally account for but have been identified in our research.
Design and construction Evaluation and reflection
Implementation and spread Theoretical understanding Maturing intervention
DBR = design-based research. 312
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Design and construction: In this phase, the goal is to use a systematic process to generate potential solutions to the problem identified in the preceding phase. This phase is focused on creating ideas, models, and thoughtful approaches while balancing the needs, wishes, and constraints of the research project. Design includes generating, exploring, and mapping potential solutions; whereas, construction involves the physical creation of prototypes that embody the design ideas. In this phase, it is important to be aware of how people may best learn in the context of interest as that can inform what may or may not work within the setting. In addition, all design decisions should be justified and documented during each iteration. Evaluation and reflection: In this phase, the goal is to conduct empirical testing using the intervention that has been created or refined from previous cycles. Evaluation typically includes pilot testing of or integration of the intervention into a practice setting—this can include the target user or a proxy depending on the feasibility and availability. Evaluation of the intervention often involves observations, interviews, focus groups, and other strategies to collect data about how individuals interacted with the intervention, areas of success, and opportunities for improvement. These findings are intended to inform reflection, which includes a critical analysis of the results compared to the expected based on previous empirical work. Implementation and spread: Implementation and spread span all phases of the McKenney and Reeves13 DBR model. In this model, implementation refers to the process when the intervention is set in motion. Typically, this process occurs at a small, local level with pilot testing and close collaboration with educators and practitioners who will interact with and help improve the intervention. The goal is to create an environment that supports the use of the intervention and the collection of useful data and feedback. As the intervention develops, the next goal is to promote the implementation and spread of the intervention to a more expansive setting or different context. In summary, implementation and spread is related to the strategies used to share the ideas and results, the context and surrounding systems that support or hinder this process, and the connections between the individuals involved to promote the intervention. Theoretical understanding and maturing intervention: Each phase is also intended to inform the design of the intervention and our understanding of the theories that describe how people learn in practice, which both support the research. Typically, these insights are obtained after a full cycle of all three phases, which then inform subsequent cycles, as shown in Fig. 1. It is possible small observations or insights may inform other stages; however, the primary focus of the process is to make more substantial contributions at the end of the cycle. The end of a cycle often marks the end of a “study” based on our conventional expectations. The theoretical understanding and maturing intervention are critical because an expected outcome of the DBR process is to make practical contributions (i.e. create novel solutions to complex problems) and theoretical contributions (i.e. refine our understanding of how people learn). Our recommendations and their applications DBR can be a complex undertaking but can have numerous benefits—we outline six recommendations for DBR in pharmacy education. For each recommendation we offer ideas in how these recommendations can be applied to various pharmacy educational research topics. In Table 3, we outline how these recommendations guided our research on student collaboration. Recommendation 1: It takes a team to generate high-quality educational research—include students, clinicians, educational experts, designers, and other stakeholders as soon as possible in the research process. A team is necessary to attend to the expectations of DBR.20,21 Using a team also promotes diversity of experiences that can generate creative ideas and solutions. Collaboration can only be effective, however, if all team members have a voice and there is a safe, supportive environment to contribute.20 Scholars should be open to the opinions and recommendations of others, especially the end-users (i.e. learners), as their perspective can be critical. This, for example, could be pertinent if you are interested in designing online modules as part of a flipped classroom.22 In this case it is important to bring on experts in instructional and technological design who can help create these resources while clinicians provide the clinical expertise. In addition, getting insights from students about the length of the videos or how they would like to interact with the modules can aid the development process. It can save time and frustration in the long run if relationships are built with those who have diverse skill sets, experiences, and knowledge. Include these individuals and their perspectives as early as possible in the research process. Often this is an important consideration in the planning process and integrating the right team members early in the process can help identify and avoid problems before they become more substantial. Recommendation 2: Plan each step of the research process and have realistic expectations—high quality research takes time, patience, and iteration. DBR is a systematic yet creative approach to addressing practical needs in education. DBR projects often span several years and numerous iterations. The best approach is to plan out the first phase of the research objectives and map out future modifications, understanding that these ideas are highly susceptible to change based on the initial findings. Planning helps the research team manage the expectations of the entire research process: design, data collection, analysis, modifications, and iteration. Mapping out the research process helps to visualize the scope of the project and ensures that initiatives are properly aligned at each stage. In addition, this helps to prevent overzealous scholars from addressing too many questions or changing too many components at once. Consider, for example, designing an objective structured clinical examination (OSCE) intended to track a learner's progress through the course. The OSCE should be developed to include key components and mapped back to course or program expectations at the time the assessment is delivered.22,23 The plan should include how the data will be used to modify future OSCEs if something does not go as planned.24 New information is obtained through the research, which then leads to modifications. However, avoid making too many changes at once or else it can be difficult to identify which change was responsible for an outcome.13,21 When creating the plan, draw from the experiences and expertise of the team by integrating unique ideas to address the problem at hand. 313
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Table 3 Applying the recommendations to the example case study on student collaboration. Recommendation
Example
1: Create a team of diverse experts
While studying student collaboration, there was an idea to build a mobile application that could be used as a medium for students to evaluate their group performance and receive strategies for improvement. For our project, there were several partnerships with the UNC School of Education and the Center for Innovation in Pharmacy Simulation at UNC Eshelman School of Pharmacy to support the development and refinement of prototypes. In general, creating a network through the school helped build a foundation for future work and generated a host of potential resources. Throughout this process, we have made it a priority to document all decisions regarding design, research, and implementation. This has allowed us to organize what we have done, how well it was executed, and changes that we want to make in future iterations. When possible, we like to share this information with participants and stakeholders (e.g., students, educators, administrators, and faculty) to get their feedback. In this project, we have collected data from the app (i.e. log data), video observations, focus groups, and cognitive interviews. We have used mostly qualitative methods in exploratory studies designed to better understand how these groups work and learn together. These included the development of three codebooks and triangulation between the different forms of data. We have also conducted mixed methods analyses, integrating quantitative log data with qualitative coding of the focus groups to better understand the students’ experiences when using the app. With regards to our approach to studying collaboration, this included a broad investigation of learning about student interactions in the classroom, areas of potential improvement, and obtaining a thorough understanding of the literature on collaboration. We started with exploring the opportunities and barriers of current collaborative learning practices at our school of pharmacy. Then we searched for theories of how collaborative learning occurs, how to teach collaborative learning, and how to design a web-based tool to improve collaborative learning. Primarily, our research was guided by theories of socially shared regulation of learning, or how groups collectively monitor and control their learning, and three design principles for collaborative tools.19 We have presented our work at various national and international education conferences (i.e. European Association of Research on Learning and Instruction, American Education Research Association) and have several manuscripts in draft with one under review. We often have brainstorming sessions in which we write down as many ideas as we can in a short period of time (e.g., one minute). We write each idea on a separate post-it note and put them on a wall. Our one rule is that there are no bad ideas. We also enjoy going through all the ideas and laughing at the ones that are silly or completely unrealistic! Even some of the ones that seem off the mark, though, have inspired us to form new ideas that we have since integrated into the app.
2: Plan each step but be adaptive
3: Build a toolkit of methodologies
4: Be attentive to practice and theory
5: Share your work often 6: Embrace creativity and have fun
UNC = University of North Carolina.
Recommendation 3: Build a toolkit of diverse research methodologies to measure outcomes and inform future work. DBR can be messy—studying complex phenomenon in practice requires a mixture of research methodologies that are aligned with the questions and anticipated outcomes.21 DBR often collects a substantial amount of data from surveys, focus groups, interviews, observations, video recordings, artifacts created by participants, log data generated from various technologies, and more. Be sure to have a strategy in place to store all this data with an understanding that not all of it will be used to address each research question. These data are used to triangulate findings and ensure the results are consistent based on the various methodologies. This is an important feature of research on the admissions or resident selection process, for example. In each case, a high-stakes decision must be made; therefore, multiple data sources are used to evaluate the individual and create a holistic picture of their ability. In general, it would be inappropriate to base this decision on a letter of recommendation or grade point average alone. Instead a combination of methodologies (e.g., interviews, observations, tests) is used; each approach can target a specific aim or address a particular question based on the needs of the scholar or other stakeholders. The next step then includes sharing this information with others to help them accomplish similar goals. Recommendation 4: Be attentive to the practical considerations and educational theory at all phases of research. This is an important element of DBR—the goal of DBR is to improve educational practices by using our understanding of how we learn in various contexts. Of note, the focus on either element does not have to be equal. In general, most education scholarship in the health professions is intended to inform education practice. Our hope is to encourage greater attention to theory in educational research. Consider, for example, if you wanted to develop a preceptor development program, it would be important to understand the constraints that limit practitioner engagement with the current program (e.g., practice), while being cognizant of strategies on how to motivate adult learners (e.g., theory). In general, there is a dynamic equilibrium between both practice and theory, so it is advantageous to be attentive to the contributions of each to create more robust research. To accomplish this, it may be best to bring in the support of other educational experts who are well versed in education theory or DBR from a school of education. Recommendation 5: Share your work using multiple strategies through various opportunities. DBR can produce large amounts of data and the line demarcating when a study is complete may not always be clear. As a result, knowing when to share your findings can be challenging. Each iteration or cycle may be considered a study for all intents and purposes—this can be a good rule of thumb.21,25 With the conclusion of each cycle, the scholars should reconvene to decide how to disseminate the findings.26,27 Scholars should be willing to share interim findings through posters and presentations; this can spark interest in the project and notify others of your initiatives. Potential conferences can include those of the American Association of Colleges of Pharmacy, the American Educational Research Association, the International Conference of the Learning Sciences, and the European Association of Research on Learning and Instruction. Manuscript publication is most appropriate when an intervention is ready for large-scale 314
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
dissemination. Journals that have a preference to share DBR research findings can include the Journal of the Learning Sciences, Educational Design Research, and Educational Researcher. Recommendation 6: Embrace creativity in the design process and have fun in the research process. Lastly, research in general is a significant undertaking that can be emotionally and physically taxing. An interest in education design and research is often driven by a desire to improve learning and patient care—do not lose sight of that vision in the process. DBR promotes creativity to generate novel solutions to complex problems. In this process, use strategies that encourage collaboration, generate unique ideas, and have fun with one another as that can support creative solutions and insights. Consider, for example, how creativity has been sparked in hackathons intended to solve clinical challenges such as improving patient adherence to medications.28,29 Creativity in educational research can be beneficial to address challenges such as promoting interprofessional education or designing effective co-curricular activities for learners. Scholars must think of creative approaches to bring together groups and engage them in meaningful learning activities and creative problem-solving processes can help. Potential impact DBR represents a dynamic interventional research approach that aims to improve education practices with the use of iterative design and testing that is tailored to the context and informed by theory. We suggest DBR can be a valuable approach for pharmacy education scholars as they embark on creating meaningful learning inventions. Specifically, DBR draws attention to the practical and theoretical contributions of interventional research in a comprehensive and systematic way. This helps scholars address complex challenges of learning in pharmacy contexts while collaborating with and including the perspectives of relevant stakeholders.30,31 Practical contributions of DBR The practical contributions are often more readily visible when using DBR compared to the theoretical contributions. In general, DBR helps to create a solution to a problem, which typically results in a tangible component or outcome. The type of practical contributions can vary greatly depending on the scope of the project, the type of intervention developed, and the field to which DBR is being applied. Most practical contributions can be categorized into four groups: products, processes, programs, and policies.13 1 Products are the resources and materials that are used to by the scholars, learners, or educators to support learning and instruction (e.g,. a modeling software to teach pharmacokinetics, a video to describe the mechanism of action of a drug). 2 Processes includes strategies, sequences, or tactics that are used to support learning and instruction (e.g., strategies to help students prepare for an exam, instructional approaches on how to flip a classroom). 3 Programs include a combination of products and processes into a unit that is intended to meet an educational goal (e.g., a continuing education series, a preceptor development program). 4 Policies include elements that signify a commitment of intent regardless of how strict that commitment may be (e.g., protocols for a faculty evaluation, a guideline for grading a SOAP note). It is important to understand that the intervention created from a DBR approach can serve dual roles. For instance, DBR conducted through interventions focuses on describing and explaining phenomena; the intervention functions as a medium to facilitate the investigation, which centers on the phenomenon. Refining the intervention may not be the primary focus of the research; instead, the goal is to better understand why something is happening in practice and the intervention assists with answering that question. Conversely, DBR conducted on interventions is often used to predict and prescribe how to manipulate certain phenomena using the intervention. In this case, the intervention becomes the primary focus and the goal is often to use the intervention as a tool to achieve a targeted outcome. In this process, more may be uncovered about the phenomenon, but that may not be a substantial area of focus. In addition, the practical contributions can be scaled differently in DBR studies depending on the desired scope and impact. A common goal in DBR is to continuously scale-up the intervention to be applicable in various settings and have a broader impact; however, the scholar may be interested in containing the intervention locally based on the predication that it may be effective only in one setting such as within the classroom or in practice. The decision is at the discretion of the scholar with an understanding of how this may impact the generalizability of the findings. In general, if the intervention is not scaled-up it is difficult to infer the intervention will have a similar impact in other settings. Lastly, the practical contributions of DBR also include benefits to scholars and participants that may not be directly impacted by the intervention. For example, partnering with administrators, practitioners, or patients may offer a benefit by refining their expertise, developing an appreciation for others’ perspectives, or other attributes that can be just as significant as the tangible outcome. High-quality DBR should continue to inspire those who encounter it and can have many downstream effects that are not explicit outcomes of the project. Theoretical contributions of DBR It can be difficult to understand how theories of learning effectively integrate into educational research. In the health professions, it is not common practice to explicate the theories that inform the design of an intervention or to describe the relationship to the findings.30 In educational research, theories are how we explain real-world phenomena, which are supported by empirical evidence.11 In 315
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Table 4 Practical and theoretical contributions of the example case study on student collaboration. Contribution
Example
Theoretical
Local Theory • To help students in our pharmacy entrepreneurship course regulate their thinking, motivation, emotions, and behavior, our intervention should provide strategies and prompt discussions during the groups’ second weekly meeting. Design Principles • Increase awareness of learning processes at individual, peer, and group levels19 • Support sharing of these learning processes19 • Encourage understanding and utilization of regulatory processes • Explicitly instruct on collaborative learning strategies Products • An app to help students identify salient group challenges and learn “when”, “why”, and “how” to target them. Process • Strategies that help students overcome obstacles during collaborative learning. Program • In the future, we hope to supplement the app with curricular resources that instructors can embed within their courses. Policies • Future policies may include explicit instruction of collaboration skills in health professions courses.
Practical
general, theories provide a model or framework to describe how and why we observe certain phenomena. More importantly, theories can be helpful in predicting outcomes and they can be used to guide the design of our interventions to optimize their effects. Theories, therefore, are critical because they “help us make sense of the world”.13 In the context of DBR, we use theories to help design the intervention based on what we currently know about learning and the setting of interest. We also use DBR to test the theory and determine which components are salient or should be modified based on the findings. Most often, DBR studies provide insights into theories that are restricted to a specific context. These findings may help describe or explain a phenomenon in these settings or they can provide data that confirm predictions and offer recommendations for future intervention designs. These insights that guide intervention design in DBR are often called design principles, which are a list of strategies that should be considered when creating interventions to address similar problems in similar settings. The level of impact these insights have is highly dependent on the setting of the DBR study. In summary, DBR research can provide varied contributions to theory depending on the context, the extent of the intervention, the research questions, and a host of other factors.13 Overall, DBR uses theory as an input to create interventions and it produces theoretical understanding as an output though the interpretation of the results. An example of the practical and theoretical contributions of the DBR study on student collaboration is available in Table 4. Scholarly contributions In addition, using DBR has the potential to improve the quality of educational research and make more frequent and valuable contributions to the literature. Often the focus in educational research is either exclusively related to the assessment of interventions in practice or the development of theory. The scope of DBR is expansive enough to effectively incorporate both elements as it explores how and why educational interventions have an impact while being cognizant of the challenges in design and implementation in realworld settings. Moreover, educational research can be complicated, proving difficult for educators and scholars. With DBR, scholars can capture their experiences in the design process and recognize the value in the trials and tribulations that contribute to the final product. Disseminating information about these challenges can help other scholars as they develop interventions and conduct research; the additional insight from others can help scholars be more efficient in their approach. Scholars and educators are unaware of the potential for DBR to transform their practices; DBR offers a framework that supports a rational, data-driven process to explicate practical implications of educational research. When conducting DBR, however, be advised of some expected challenges and limitations.10 First, the DBR process can sometimes appear chaotic or disorganized as it often includes rapid prototyping and troubleshooting with new ideas that may not be fully formed. Rarely does the initial design idea become the final designed product or process. Engaging in DBR requires an open mindset to manage these expectations when it is first implemented. In addition, the DBR process is complex and requires extensive planning, time, and resources. Individuals interested in DBR are strongly encouraged to learn more about the methodologies it can include, to view additional published examples of this type of research, and to partner with researchers who have engaged in DBR previously. Also, many DBR designs produce large amounts of data which, at times, are left unanalyzed due to the urgency of the next upcoming DBR implementation or lack of resources.10 Lastly, DBR limits the ability to compare across designs since the intervention is purposefully improved after each DBR cycle. In summary, DBR encompasses a systematic, iterative approach to develop, implement, and test innovative educational interventions that address complex, real-world problems while advancing and refining educational theories to inform future research.13 With an understanding of the elements that comprise DBR, the next step is to consider how DBR can be used in future educational research to inform practice. Additional resources on the DBR process are included in Box 2. 316
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Conclusions DBR can aid pharmacy educational scholars by using a flexible, iterative, and systematic process to generate novel and creative solutions to complex problems. We offer recommendations on how practitioners can learn more about DBR, effectively integrate DBR into research processes, and promote positive change. Conflict of interest None. Disclosures None. Box 1 Design-based research case study. Problem: Pharmacy students need to co-construct knowledge during team-based projects and develop an understanding of strategies for overcoming teamwork obstacles. Currently, students are failing to do so, as only course content, not the collaboration process, is taught, supported, and assessed. Purpose: Based on interview and observation data from students and instructors, we think students need an algorithmbased technology that assists students to diagnose current obstacles and then suggest corresponding group-level regulation strategies. Plan: Who: Second year pharmacy students in an entrepreneurship course What: Develop, test, and implement a technology-enhanced learning tool (Collabucate – a mobile application) to promote effective collaboration among student pharmacists When: Spring 2016 – present Where: University of North Carolina Eshelman School of Pharmacy Why: Each student team faces a wide variety of obstacles and there is no personalized coaching to help them identify new strategies for overcoming collaboration obstacles. How: Students complete questions in the app individually each week and identify personal challenges related to their skills to, motivation for, and feelings about collaboration. The app compiles scores from the group to identify the salient challenge and offers strategies for the students to implement. The app monitors progress over time. Sample Research Questions: What are students’ experiences while using the Collabucate app? How and to what extent do students employ strategies learned through the Collabucate app? What tensions do students face in student project groups? How do students’ experiences and perceptions about collaboration change during the course? Box 2 Design-based research resources. The DBR approach: Anderson T, Shattuck J. Design-based research: a decade of progress in education research? Educ Res. 2012;41(1):16–25. Brown A. Design experiments: theoretical and methodological challenges in creating complex interventions. J Learn Sci. 1992;2(2):141–178. Collins A, Joseph D, Bielaczyc K. Design research: theoretical and methodological issues. J Learn Sci. 2004;13(1):15–42. The Design-Based Research Collective. Design-based research: an emerging paradigm for educational inquiry. Educ Res. 2003;32(1):5–8. Dolmans DH, Tigelaar D. Building bridges between theory and practice in medical education using a design-based research approach: AMEE Guide No. 60. Med Teach. 2012;34(1):1–10. McKenney S, Reeves TC. Conducting Educational Design Research. 1st ed. New York, NY: Routledge; 2012. Svihla V, Reeve R, eds. Design as Scholarship: Case Studies from the Learning Sciences. New York, NY: Routledge; 2016. Van den Akker J, Gravemeijer K, McKenney S, Nieveen N, eds. Educational Design Research. 1st ed. New York, NY: Routledge; 2006.
317
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Supplementary materials Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.cptl.2018.12.002. References 1. Bauman JL, Ascione FJ, Brueggemeier RW, et al. Maintaining pharmacy education's research focus as the academy expands. Am J Pharm Educ. 2012;76(8) https:// doi.org/10.5688/ajpe768144. 2. DiPiro J. The 21st century Abraham Flexner. Am J Pharm Educ. 2008;72(4) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2576418/pdf/ajpe79.pdf Accessed 23 December 2018. 3. McLaughlin JE, Dean MJ, Mumper RJ, Blouin RA, Roth MT. A roadmap for educational research in pharmacy. Am J Pharm Educ. 2013;77(10) https://doi.org/10. 5688/ajpe7710218. 4. Persky AM, Romanelli F. Insights, pearls, and guidance on successfully producing and publishing educational research. Am J Pharm Educ. 2016;80(5) https://doi. org/10.5688/ajpe80575. 5. Cook DA, Bordage G, Schmidt HG. Description, justification and clarification: a framework for classifying the purposes of research in medical education. Med Educ. 2008;42(2):128–133. 6. The Design-Based Research Collective. Design-based research: an emerging paradigm for educational inquiry. Educ Res. 2003;32(1):5–8. 7. Dolmans DH, Tigelaar D. Building bridges between theory and practice in medical education using a design-based research approach: AMEE Guide No. 60. Med Teach. 2012;34(1):1–10. 8. Norman G. How bad is medical educational research anyway? Adv Health Sci Educ. 2007;12:1–5. 9. Brown A. Design experiments: theoretical and methodological challenges in creating complex interventions. J Learn Sci. 1992;2(2):141–178. 10. Collins A, Joseph D, Bielaczyc K. Design research: theoretical and methodological issues. J Learn Sci. 2004;13(1):15–42. 11. Barab S. Design-based research: a methodological toolkit for engineering change. In: Sawyer RK, ed. The Cambridge Handbook of the Learning Sciences. 2nd ed New York, NY: Cambridge University Press; 2014:151–170. 12. Barab S, Squire K. Design-based research: putting a stake in the ground. J Learn Sci. 2004;13(1):1–14. 13. McKenney S, Reeves TC. Conducting Educational Design Research. 1st ed. New York, NY: Routledge; 2012. 14. Kelly A. Design research in education: yes, but is it methodological? J Learn Sci. 2004;13(1):115–128. 15. Kelly AE. The role of design in educational research. Educ Res. 2003;32(1):3–4. 16. McKenney S, van den Akker J. Computer-based support for curriculum designers: a case of developmental research. Educ Technol Res Dev. 2005;53(2):41–66. 17. Reeves TC, Herrington J, Oliver R. Design research: a socially responsible approach to instructional technology research in higher education. J Comput High Educ. 2005;16(2):97–116. 18. Wang F, Hannafin MJ. Design-based research and technology-enhanced learning environments. Educ Technol Res Dev. 2005;53(4):5–23. 19. Järvelä S, Kirschner PA, Panadero E, et al. Enhancing socially shared regulation in collaborative learning groups: designing for CSCL regulation tools. Educ Technol Res Dev. 2015;63(1):125–142. 20. Edelson DC. Design research: what we learn when we engage in design. J Learn Sci. 2002;11(1):105–121. 21. Svihla V, Reeve R. Design as Scholarship: Case Studies from the Learning Sciences. New York, NY: Routledge; 2016. 22. McLaughlin JE, Roth MT, Glatt DM, et al. The flipped classroom: a course redesign to foster learning and engagement in a health professions school. Acad Med. 2014;89(2):236–243. 23. Khan KZ, Ramachandran S, Gaunt K, Pushkar P. The objective structured clinical examination (OSCE): AMEE Guide No. 81. Part I: an historical and theoretical perspective. Med Teach. 2013;35(9):e1437–e1446. 24. Khan KZ, Gaunt K, Ramachandran S, Pushkar P. The objective structured clinical examination (OSCE): AMEE Guide No. 81. Part II: organization and administration. Med Teach. 2013;35(9):e1447–e1463. 25. Easterday MW, Lewis DR, Gerber EM. Design-based research process: problems, phases, and applications. Paper Presented at: The International Conference of Learning Sciences. Boulder, CO; 2014 June 24. 26. Cobb P, Confrey J, diSessa A, Lehrer R, Schauble L. Design experiments in educational research. Educ Res. 2003;32(1):9–13. 27. Van den Akker J, Gravemeijer K, McKenney S, Nieveen N. Educational Design Research. 1st ed. New York, NY: Routledge; 2006. 28. Silver JK, Binder DS, Zubcevik N, Zafonte RD. Healthcare hackathons provide educational and innovation opportunities: a case study and best practice recommendations. J Med Syst. 2016;40(7) 10.1007s10916-019-0532-3. 29. Olson KR, Walsh M, Garg P, et al. Health hackathons: theatre or substance? A survey assessment of outcomes from healthcare-focused hackathons in three countries. BMJ Innov. 2017;3(1):37–44. 30. Bradley G. The crisis in educational research: a pragmatic approach. Eur Educ Res J. 2003;2(2):296–308. 31. Albert M, Hodges B, Regehr G. Research in medical education: balancing service and science. Adv Health Sci Educ Theory Pract. 2007;12(1):103–115.
318
Contents lists available at ScienceDirect
Currents in Pharmacy Teaching and Learning journal homepage: www.elsevier.com/locate/cptl
Methodology Matters
Design-based research: Connecting theory and practice in pharmacy educational intervention research
T
Michael D. Wolcotta,b, Nikki G. Lobczowskia,b, Kayley Lyonsa,b, ⁎ Jacqueline E. McLaughlina, a b
University of North Carolina, Eshelman School of Pharmacy, 329 Beard Hall, Chapel Hill, NC 27599, United States University of North Carolina, School of Education, 3500 Peabody Hall, Chapel Hill, NC 27599, United States
ARTICLE INFO
ABSTRACT
Keywords: Research methods Research design Design-based research Mixed methods Collaboration
Our situation: Interventional research in pharmacy education includes the study of complex challenges that can be difficult to navigate. Design-based research (DBR), is a systematic and iterative approach to interventional research that is attentive to the practical and theoretical contributions to education. Practical contributions include the creation of novel solutions to complex problems that improve learning while theoretical contributions include refining our understanding of context-specific learning. In this paper, we describe how we addressed challenges associated with student collaboration in pharmacy education by applying DBR to bridge theory and practice. Methodological literature review: DBR is characterized as authentic, contextually aware, collaborative, theoretically focused, methodologically diverse, practical, iterative, and operation-oriented. DBR includes three iterative phases: (1) analysis and exploration, (2) design and construction, and (3) evaluation and reflection. Our recommendations and their applications: To integrate DBR into interventional research, scholars should work collaboratively with diverse teams of experts. DBR also requires extensive planning, a toolkit of expansive research methodologies, and attention to practical and theoretical considerations. Finally, scholars should share their work as often as possible and engage in creative exercises to promote innovative solutions to challenges in education. Potential impact: DBR offers an approach to generate practical, theoretical, and scholarly contributions to pharmacy education research. In summary, DBR can aid pharmacy educational scholars by using a flexible, iterative, and systematic process to generate novel and creative solutions to complex problems.
Our situation Pharmacy educational research is at a precipice—there is significant discourse about a need to advance pharmacy education by incorporating evidence-based educational practices.1,2 Consequently, there is substantial interest in conducting high-quality educational intervention research intended to inform practice while producing meaningful outcomes for our learners.3,4 In the health professions, however, we tend to focus on if our interventions influenced learning outcomes—what is missing is a better understanding of how our interventions have influenced learning as this can inform the design of more effective interventions in the
⁎
Corresponding author. E-mail address: [email protected] (J.E. McLaughlin).
https://doi.org/10.1016/j.cptl.2018.12.002
1877-1297/ © 2018 Elsevier Inc. All rights reserved.
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
future.5 Design-based research (DBR) is an iterative approach to designing, implementing, evaluating, and improving educational interventions.6 DBR can help scholars better understand various design elements of an intervention and how they influence learning experiences and outcomes and address this gap in pharmacy education scholarship.7 For example, questions asked when using DBR might include: (a) what is happening between the point when a learner enters a learning experience and when they exit?, (b) how do learners interact with the materials and information we give them?, (c) how can the learning process be optimized?, or (d) how do our findings build on our understanding of learning in this context? Specifically, the aim of DBR is to enhance educational intervention research through practical contributions (i.e. generating novel solutions to complex problems and improving learning through high-quality interventions) and theoretical contributions (i.e. refining our understanding of how individuals learn in a specific context and offering strategies to optimize learning) to the literature. DBR is particularly relevant to pharmacy education as it can aid education scholars in measuring the effectiveness of interventions in addition to advancing our understanding of how learning occurs within pharmacy settings. The iterative nature of DBR emphasizes how research can continuously build on itself and articulates how future research is explicitly informed from previous research. Our current approaches tend to focus exclusively on measuring intervention effectiveness though changes in surveys, questionnaires, test scores, or other readily accessible data, which can be insufficient.8 Although these methods are valuable, this approach provides limited information for interventional research because it lacks a deeper investigation into why we are or are not witnessing a change in learning in pharmacy practice. Consider, for example, the common challenge that we addressed in pharmacy education: how to effectively promote, measure, and improve learner collaboration (Box 1). Our previous approach to this challenge was often generic and lacked sufficient depth to understand what contributed to improvements in learning and how. Our simplistic approach would often include identifying strategies to promote collaboration based on empirical evidence and anecdotal expectations of learner collaboration in a course. Typically, our approach would not include collaboration with educational experts or students to generate novel solutions to the challenge. In addition, we were often tempted to use only surveys at the beginning and end of the course to measure the intervention's effectiveness based on faculty and student perceptions about collaboration or the efficacy of certain strategies introduced through our study. We noticed that if the study showed a statistically significant improvement in perceptions of collaboration, it would be difficult to ascertain how the strategies contributed effectively to improve collaboration or why perceptions changed. Alternatively, if the study showed no or minimal change, we would conclude the study was unsuccessful, not disseminate the findings, and try a different approach next year without learning from participants how the intervention could be improved to be more successful in the future. For example, students may have been disengaged because they did not know the strategies were intended to improve collaboration or the faculty members may not have implemented the strategies appropriately. DBR offers an approach to address the issues we witnessed in our scholarship. We found DBR could improve our research by supporting the creation of novel solutions to complex challenges, promoting collaboration with experts in education, and helping us understand how the intervention worked to impact learning outcomes. Overall, DBR has improved the quality and refinement of our interventions based on its iterative and systematic approach to conducting educational intervention research. In summary, this article outlines how DBR can be used to advance interventional pharmacy education research. Specifically, we describe throughout the article how we applied DBR to address the challenge of improving learner collaboration. The purpose of this article is to introduce pharmacy scholars to DBR, outline common components of the DBR process, and offer recommendations on how pharmacy scholars can integrate DBR into their research practice. We strongly encourage individuals interested in DBR to consult additional resources (Box 2) to obtain a more in-depth discussion of the DBR approach, the methodologies associated with it, and additional practical tips for the process. Methodological literature review DBR first originated as an approach to integrate design experiments to address theoretical questions about the nature of learning in real-life contexts.9,10 It has evolved into a research approach defined by eight characteristics.6,7,10–15 1 Authentic: The hallmark of DBR is that it takes place in real-life learning settings to capture how learning normally occurs instead of studying learning in a laboratory setting. 2 Contextually aware: Unlike experimental studies, in DBR scholars aim to identify, not necessarily control, all the variables that may influence outcomes; effects related to the student, teacher, and institution are described as they are expected to matter in realworld education practice and can be used to understand future research directions. 3 Collaborative: DBR recognizes the importance of working with others (e.g., learners, educators, scholars) who share expertise in fields such as design, education, and the learning context to ensure optimal design of interventions. 4 Theoretically focused: The goal of DBR is to create interventions aligned with our understanding of learning theories as well as using the results to support, derive, or refine theories as they are applied in the setting of interest. 5 Methodologically diverse: DBR incorporates a pragmatic approach to research design that incorporates both qualitative and quantitative data collection strategies (i.e. mixed methods) to gain a more holistic perspective of learning in real-life contexts. 6 Practical: Interventions created through DBR studies are sensitive to the learner's needs to address a specific problem while attending to constraints in the specified context such as limitations in resources and technology. 7 Iterative: DBR is a systematic process that has distinct cycles of analysis, creation, and evaluation to continuously improve 310
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Fig. 1. Sample model of iterations in DBR (adapted from McKenny & van den Akker).16 DBR = design-based research.
interventions and refine theories based on the findings at each stage. As opposed to waiting to test an intervention until it is perfect, DBR scholars test their designs early and often. Fig. 1 exemplifies how the cycles can vary throughout the research process, each informed by the previous findings. 8 Operational: The aim of DBR is to create and understand interventions so that they can be implemented in other settings and have a broader impact in educational research. The combination of these characteristics is what makes DBR unique; attention to each element promotes higher quality educational research that connects theory and interventions in real-life settings. Table 1 shows how these characteristics can be applied to the case example of studying collaboration among student pharmacists. These characteristics create the foundation for the overarching model of DBR that promotes the systematic approach to educational intervention research. DBR is similar to participatory action research in that both approaches identify real-world problems and generate ideas to improve conditions or educational practices.17 However, DBR can be distinguished from participatory action research by the use, development, and testing of either learning or implementation theories. Compared to the action research process, the DBR process includes an initial step to identify and apply education theory from previous literature to the proposed educational design. Then DBR researchers test the theoretically-guided interventions in order to impact practice and contribute to theory development. Also, DBR aims to integrate stakeholders as researchers and designers in the process.18 Teachers or clinicians, for example, in the DBR process are encouraged to engage in the research and design process to facilitate implementation of the intervention and ensure that the Table 1 Applying characteristics of DBR to the example case study on student collaboration. Characteristic
Example
Authentic Contextually aware
We studied existing collaboration and implementing the intervention within an ongoing course. We aimed to characterize how students interacted with the Collabucate application created by the research team, what the student and teacher needs were, and how collaborative learning took place throughout the course. We recruited learners, as well as experts in research methodologies, educational research, technology development, and pharmacy education. Our intervention design was based on the theory of socially-shared regulation of learning, which describes how individuals coconstruct knowledge.18 We included observation data from video recordings, qualitative data from focus groups and think-aloud protocols, and quantitative data from surveys and learning analytics/responses on the mobile application. Our intervention design and research were constrained by time and budget limits while adhering to instructor requests without disrupting the course. We involved at least three prototype iterations to constantly improve design. Our goal was to make the intervention sufficiently broad to implement in multiple courses and other health professions education programs.
Collaborative Theoretically focused Methodologically diverse Practical Iterative Operational
DBR = design-based research. 311
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Fig. 2. Model of design-based research (adopted from McKenny and Reeves).13
intervention includes the various perspectives of stakeholder groups. The focus on these components—theory and the role of practitioners in the research process—makes DBR distinct from other research approaches. A model of DBR DBR includes a compilation of methodologies that can be integrated into a systematic process to address complex challenges in practice settings. An important hallmark of DBR is that although it takes place in complex settings, it still relies on a systematic process to make informed decisions—even though it is inspired by creativity and novel solutions. There have been several models to describe DBR such as the one shown in Fig. 2 by McKenney and Reeves.13 This model of DBR includes three main phases that inform theory and practice in a flexible process. Of note, the model portrays DBR as a linear process; however, the approach is highly iterative and you may not necessarily complete each phase sequentially or in great depth before moving on to a different stage.13 Table 2 outlines how we applied each of these phases to the design and implementation of the research study on student collaboration. Analysis and exploration: In this phase, the goal is to define the problem, identify potential causes, explore the setting, and learn about the needs and wishes of the stakeholders. Analysis is focused on orienting the researchers to the setting, conducting cursory investigations, and often includes a literature review or discussion with subject matter experts. The exploration element includes a deeper dive into the setting, developing a better understanding of the stakeholders, and identifying potential constraints or considerations that may inform the solution to the challenge. It often includes interviews, observations, site visits, as well as meetings with stakeholders and research collaborators. Table 2 Applying the DBR model to the example case study on student collaboration. Phase
Example
Analysis and exploration
We conducted interviews with instructors and focus groups with students focused on problems in collaborative learning. Overall findings suggest that students lacked a collaboration skill set and were apprehensive to resolve issues in group work. Further, instructors valued collaboration skills but did not know how to teach them. We met to determine all the possible solutions to help students and instructors with this problem. Next, we rated each solution on its feasibility and impact. Finally, we took the solution that was the most feasible and impactful and created a prototype (i.e. the Collabucate application). We then tested the prototype with students, integrating their feedback into future versions. We conducted a pilot study with six groups of four to five pharmacy students, collecting video observations of their weekly group meetings and log data from the app. At the end of the semester, we also collected focus group data, having students describe successes, challenges, and areas for improvement with regards to the app. We then analyzed this data and identified changes for future iterations of the app. We began using Collabucate with six student groups and have since increased to class-wide implementation (150 students). Future implementations will include different pharmacy courses (i.e. including different instructors, content, students, course structure) and different (i.e. non-pharmacy) contexts. We started with three design principles from the literature and have derived another design principle. This research has helped shape our theoretical understanding of how groups regulation their thinking, motivation, and emotions, as well as the tensions and interactions that emerge in this type of collaborative learning environment. We started with a minimally viable working prototype of the Collabucate app. We now have a functional application that has been through two iterations and implementations. Perhaps our most important addition is a group page that prompts students to discuss the results. Looking toward the future, we have plans to create an integrate a teacher dashboard and better personalize certain aspects of the app to include new challenges that we did not originally account for but have been identified in our research.
Design and construction Evaluation and reflection
Implementation and spread Theoretical understanding Maturing intervention
DBR = design-based research. 312
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Design and construction: In this phase, the goal is to use a systematic process to generate potential solutions to the problem identified in the preceding phase. This phase is focused on creating ideas, models, and thoughtful approaches while balancing the needs, wishes, and constraints of the research project. Design includes generating, exploring, and mapping potential solutions; whereas, construction involves the physical creation of prototypes that embody the design ideas. In this phase, it is important to be aware of how people may best learn in the context of interest as that can inform what may or may not work within the setting. In addition, all design decisions should be justified and documented during each iteration. Evaluation and reflection: In this phase, the goal is to conduct empirical testing using the intervention that has been created or refined from previous cycles. Evaluation typically includes pilot testing of or integration of the intervention into a practice setting—this can include the target user or a proxy depending on the feasibility and availability. Evaluation of the intervention often involves observations, interviews, focus groups, and other strategies to collect data about how individuals interacted with the intervention, areas of success, and opportunities for improvement. These findings are intended to inform reflection, which includes a critical analysis of the results compared to the expected based on previous empirical work. Implementation and spread: Implementation and spread span all phases of the McKenney and Reeves13 DBR model. In this model, implementation refers to the process when the intervention is set in motion. Typically, this process occurs at a small, local level with pilot testing and close collaboration with educators and practitioners who will interact with and help improve the intervention. The goal is to create an environment that supports the use of the intervention and the collection of useful data and feedback. As the intervention develops, the next goal is to promote the implementation and spread of the intervention to a more expansive setting or different context. In summary, implementation and spread is related to the strategies used to share the ideas and results, the context and surrounding systems that support or hinder this process, and the connections between the individuals involved to promote the intervention. Theoretical understanding and maturing intervention: Each phase is also intended to inform the design of the intervention and our understanding of the theories that describe how people learn in practice, which both support the research. Typically, these insights are obtained after a full cycle of all three phases, which then inform subsequent cycles, as shown in Fig. 1. It is possible small observations or insights may inform other stages; however, the primary focus of the process is to make more substantial contributions at the end of the cycle. The end of a cycle often marks the end of a “study” based on our conventional expectations. The theoretical understanding and maturing intervention are critical because an expected outcome of the DBR process is to make practical contributions (i.e. create novel solutions to complex problems) and theoretical contributions (i.e. refine our understanding of how people learn). Our recommendations and their applications DBR can be a complex undertaking but can have numerous benefits—we outline six recommendations for DBR in pharmacy education. For each recommendation we offer ideas in how these recommendations can be applied to various pharmacy educational research topics. In Table 3, we outline how these recommendations guided our research on student collaboration. Recommendation 1: It takes a team to generate high-quality educational research—include students, clinicians, educational experts, designers, and other stakeholders as soon as possible in the research process. A team is necessary to attend to the expectations of DBR.20,21 Using a team also promotes diversity of experiences that can generate creative ideas and solutions. Collaboration can only be effective, however, if all team members have a voice and there is a safe, supportive environment to contribute.20 Scholars should be open to the opinions and recommendations of others, especially the end-users (i.e. learners), as their perspective can be critical. This, for example, could be pertinent if you are interested in designing online modules as part of a flipped classroom.22 In this case it is important to bring on experts in instructional and technological design who can help create these resources while clinicians provide the clinical expertise. In addition, getting insights from students about the length of the videos or how they would like to interact with the modules can aid the development process. It can save time and frustration in the long run if relationships are built with those who have diverse skill sets, experiences, and knowledge. Include these individuals and their perspectives as early as possible in the research process. Often this is an important consideration in the planning process and integrating the right team members early in the process can help identify and avoid problems before they become more substantial. Recommendation 2: Plan each step of the research process and have realistic expectations—high quality research takes time, patience, and iteration. DBR is a systematic yet creative approach to addressing practical needs in education. DBR projects often span several years and numerous iterations. The best approach is to plan out the first phase of the research objectives and map out future modifications, understanding that these ideas are highly susceptible to change based on the initial findings. Planning helps the research team manage the expectations of the entire research process: design, data collection, analysis, modifications, and iteration. Mapping out the research process helps to visualize the scope of the project and ensures that initiatives are properly aligned at each stage. In addition, this helps to prevent overzealous scholars from addressing too many questions or changing too many components at once. Consider, for example, designing an objective structured clinical examination (OSCE) intended to track a learner's progress through the course. The OSCE should be developed to include key components and mapped back to course or program expectations at the time the assessment is delivered.22,23 The plan should include how the data will be used to modify future OSCEs if something does not go as planned.24 New information is obtained through the research, which then leads to modifications. However, avoid making too many changes at once or else it can be difficult to identify which change was responsible for an outcome.13,21 When creating the plan, draw from the experiences and expertise of the team by integrating unique ideas to address the problem at hand. 313
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Table 3 Applying the recommendations to the example case study on student collaboration. Recommendation
Example
1: Create a team of diverse experts
While studying student collaboration, there was an idea to build a mobile application that could be used as a medium for students to evaluate their group performance and receive strategies for improvement. For our project, there were several partnerships with the UNC School of Education and the Center for Innovation in Pharmacy Simulation at UNC Eshelman School of Pharmacy to support the development and refinement of prototypes. In general, creating a network through the school helped build a foundation for future work and generated a host of potential resources. Throughout this process, we have made it a priority to document all decisions regarding design, research, and implementation. This has allowed us to organize what we have done, how well it was executed, and changes that we want to make in future iterations. When possible, we like to share this information with participants and stakeholders (e.g., students, educators, administrators, and faculty) to get their feedback. In this project, we have collected data from the app (i.e. log data), video observations, focus groups, and cognitive interviews. We have used mostly qualitative methods in exploratory studies designed to better understand how these groups work and learn together. These included the development of three codebooks and triangulation between the different forms of data. We have also conducted mixed methods analyses, integrating quantitative log data with qualitative coding of the focus groups to better understand the students’ experiences when using the app. With regards to our approach to studying collaboration, this included a broad investigation of learning about student interactions in the classroom, areas of potential improvement, and obtaining a thorough understanding of the literature on collaboration. We started with exploring the opportunities and barriers of current collaborative learning practices at our school of pharmacy. Then we searched for theories of how collaborative learning occurs, how to teach collaborative learning, and how to design a web-based tool to improve collaborative learning. Primarily, our research was guided by theories of socially shared regulation of learning, or how groups collectively monitor and control their learning, and three design principles for collaborative tools.19 We have presented our work at various national and international education conferences (i.e. European Association of Research on Learning and Instruction, American Education Research Association) and have several manuscripts in draft with one under review. We often have brainstorming sessions in which we write down as many ideas as we can in a short period of time (e.g., one minute). We write each idea on a separate post-it note and put them on a wall. Our one rule is that there are no bad ideas. We also enjoy going through all the ideas and laughing at the ones that are silly or completely unrealistic! Even some of the ones that seem off the mark, though, have inspired us to form new ideas that we have since integrated into the app.
2: Plan each step but be adaptive
3: Build a toolkit of methodologies
4: Be attentive to practice and theory
5: Share your work often 6: Embrace creativity and have fun
UNC = University of North Carolina.
Recommendation 3: Build a toolkit of diverse research methodologies to measure outcomes and inform future work. DBR can be messy—studying complex phenomenon in practice requires a mixture of research methodologies that are aligned with the questions and anticipated outcomes.21 DBR often collects a substantial amount of data from surveys, focus groups, interviews, observations, video recordings, artifacts created by participants, log data generated from various technologies, and more. Be sure to have a strategy in place to store all this data with an understanding that not all of it will be used to address each research question. These data are used to triangulate findings and ensure the results are consistent based on the various methodologies. This is an important feature of research on the admissions or resident selection process, for example. In each case, a high-stakes decision must be made; therefore, multiple data sources are used to evaluate the individual and create a holistic picture of their ability. In general, it would be inappropriate to base this decision on a letter of recommendation or grade point average alone. Instead a combination of methodologies (e.g., interviews, observations, tests) is used; each approach can target a specific aim or address a particular question based on the needs of the scholar or other stakeholders. The next step then includes sharing this information with others to help them accomplish similar goals. Recommendation 4: Be attentive to the practical considerations and educational theory at all phases of research. This is an important element of DBR—the goal of DBR is to improve educational practices by using our understanding of how we learn in various contexts. Of note, the focus on either element does not have to be equal. In general, most education scholarship in the health professions is intended to inform education practice. Our hope is to encourage greater attention to theory in educational research. Consider, for example, if you wanted to develop a preceptor development program, it would be important to understand the constraints that limit practitioner engagement with the current program (e.g., practice), while being cognizant of strategies on how to motivate adult learners (e.g., theory). In general, there is a dynamic equilibrium between both practice and theory, so it is advantageous to be attentive to the contributions of each to create more robust research. To accomplish this, it may be best to bring in the support of other educational experts who are well versed in education theory or DBR from a school of education. Recommendation 5: Share your work using multiple strategies through various opportunities. DBR can produce large amounts of data and the line demarcating when a study is complete may not always be clear. As a result, knowing when to share your findings can be challenging. Each iteration or cycle may be considered a study for all intents and purposes—this can be a good rule of thumb.21,25 With the conclusion of each cycle, the scholars should reconvene to decide how to disseminate the findings.26,27 Scholars should be willing to share interim findings through posters and presentations; this can spark interest in the project and notify others of your initiatives. Potential conferences can include those of the American Association of Colleges of Pharmacy, the American Educational Research Association, the International Conference of the Learning Sciences, and the European Association of Research on Learning and Instruction. Manuscript publication is most appropriate when an intervention is ready for large-scale 314
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
dissemination. Journals that have a preference to share DBR research findings can include the Journal of the Learning Sciences, Educational Design Research, and Educational Researcher. Recommendation 6: Embrace creativity in the design process and have fun in the research process. Lastly, research in general is a significant undertaking that can be emotionally and physically taxing. An interest in education design and research is often driven by a desire to improve learning and patient care—do not lose sight of that vision in the process. DBR promotes creativity to generate novel solutions to complex problems. In this process, use strategies that encourage collaboration, generate unique ideas, and have fun with one another as that can support creative solutions and insights. Consider, for example, how creativity has been sparked in hackathons intended to solve clinical challenges such as improving patient adherence to medications.28,29 Creativity in educational research can be beneficial to address challenges such as promoting interprofessional education or designing effective co-curricular activities for learners. Scholars must think of creative approaches to bring together groups and engage them in meaningful learning activities and creative problem-solving processes can help. Potential impact DBR represents a dynamic interventional research approach that aims to improve education practices with the use of iterative design and testing that is tailored to the context and informed by theory. We suggest DBR can be a valuable approach for pharmacy education scholars as they embark on creating meaningful learning inventions. Specifically, DBR draws attention to the practical and theoretical contributions of interventional research in a comprehensive and systematic way. This helps scholars address complex challenges of learning in pharmacy contexts while collaborating with and including the perspectives of relevant stakeholders.30,31 Practical contributions of DBR The practical contributions are often more readily visible when using DBR compared to the theoretical contributions. In general, DBR helps to create a solution to a problem, which typically results in a tangible component or outcome. The type of practical contributions can vary greatly depending on the scope of the project, the type of intervention developed, and the field to which DBR is being applied. Most practical contributions can be categorized into four groups: products, processes, programs, and policies.13 1 Products are the resources and materials that are used to by the scholars, learners, or educators to support learning and instruction (e.g,. a modeling software to teach pharmacokinetics, a video to describe the mechanism of action of a drug). 2 Processes includes strategies, sequences, or tactics that are used to support learning and instruction (e.g., strategies to help students prepare for an exam, instructional approaches on how to flip a classroom). 3 Programs include a combination of products and processes into a unit that is intended to meet an educational goal (e.g., a continuing education series, a preceptor development program). 4 Policies include elements that signify a commitment of intent regardless of how strict that commitment may be (e.g., protocols for a faculty evaluation, a guideline for grading a SOAP note). It is important to understand that the intervention created from a DBR approach can serve dual roles. For instance, DBR conducted through interventions focuses on describing and explaining phenomena; the intervention functions as a medium to facilitate the investigation, which centers on the phenomenon. Refining the intervention may not be the primary focus of the research; instead, the goal is to better understand why something is happening in practice and the intervention assists with answering that question. Conversely, DBR conducted on interventions is often used to predict and prescribe how to manipulate certain phenomena using the intervention. In this case, the intervention becomes the primary focus and the goal is often to use the intervention as a tool to achieve a targeted outcome. In this process, more may be uncovered about the phenomenon, but that may not be a substantial area of focus. In addition, the practical contributions can be scaled differently in DBR studies depending on the desired scope and impact. A common goal in DBR is to continuously scale-up the intervention to be applicable in various settings and have a broader impact; however, the scholar may be interested in containing the intervention locally based on the predication that it may be effective only in one setting such as within the classroom or in practice. The decision is at the discretion of the scholar with an understanding of how this may impact the generalizability of the findings. In general, if the intervention is not scaled-up it is difficult to infer the intervention will have a similar impact in other settings. Lastly, the practical contributions of DBR also include benefits to scholars and participants that may not be directly impacted by the intervention. For example, partnering with administrators, practitioners, or patients may offer a benefit by refining their expertise, developing an appreciation for others’ perspectives, or other attributes that can be just as significant as the tangible outcome. High-quality DBR should continue to inspire those who encounter it and can have many downstream effects that are not explicit outcomes of the project. Theoretical contributions of DBR It can be difficult to understand how theories of learning effectively integrate into educational research. In the health professions, it is not common practice to explicate the theories that inform the design of an intervention or to describe the relationship to the findings.30 In educational research, theories are how we explain real-world phenomena, which are supported by empirical evidence.11 In 315
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Table 4 Practical and theoretical contributions of the example case study on student collaboration. Contribution
Example
Theoretical
Local Theory • To help students in our pharmacy entrepreneurship course regulate their thinking, motivation, emotions, and behavior, our intervention should provide strategies and prompt discussions during the groups’ second weekly meeting. Design Principles • Increase awareness of learning processes at individual, peer, and group levels19 • Support sharing of these learning processes19 • Encourage understanding and utilization of regulatory processes • Explicitly instruct on collaborative learning strategies Products • An app to help students identify salient group challenges and learn “when”, “why”, and “how” to target them. Process • Strategies that help students overcome obstacles during collaborative learning. Program • In the future, we hope to supplement the app with curricular resources that instructors can embed within their courses. Policies • Future policies may include explicit instruction of collaboration skills in health professions courses.
Practical
general, theories provide a model or framework to describe how and why we observe certain phenomena. More importantly, theories can be helpful in predicting outcomes and they can be used to guide the design of our interventions to optimize their effects. Theories, therefore, are critical because they “help us make sense of the world”.13 In the context of DBR, we use theories to help design the intervention based on what we currently know about learning and the setting of interest. We also use DBR to test the theory and determine which components are salient or should be modified based on the findings. Most often, DBR studies provide insights into theories that are restricted to a specific context. These findings may help describe or explain a phenomenon in these settings or they can provide data that confirm predictions and offer recommendations for future intervention designs. These insights that guide intervention design in DBR are often called design principles, which are a list of strategies that should be considered when creating interventions to address similar problems in similar settings. The level of impact these insights have is highly dependent on the setting of the DBR study. In summary, DBR research can provide varied contributions to theory depending on the context, the extent of the intervention, the research questions, and a host of other factors.13 Overall, DBR uses theory as an input to create interventions and it produces theoretical understanding as an output though the interpretation of the results. An example of the practical and theoretical contributions of the DBR study on student collaboration is available in Table 4. Scholarly contributions In addition, using DBR has the potential to improve the quality of educational research and make more frequent and valuable contributions to the literature. Often the focus in educational research is either exclusively related to the assessment of interventions in practice or the development of theory. The scope of DBR is expansive enough to effectively incorporate both elements as it explores how and why educational interventions have an impact while being cognizant of the challenges in design and implementation in realworld settings. Moreover, educational research can be complicated, proving difficult for educators and scholars. With DBR, scholars can capture their experiences in the design process and recognize the value in the trials and tribulations that contribute to the final product. Disseminating information about these challenges can help other scholars as they develop interventions and conduct research; the additional insight from others can help scholars be more efficient in their approach. Scholars and educators are unaware of the potential for DBR to transform their practices; DBR offers a framework that supports a rational, data-driven process to explicate practical implications of educational research. When conducting DBR, however, be advised of some expected challenges and limitations.10 First, the DBR process can sometimes appear chaotic or disorganized as it often includes rapid prototyping and troubleshooting with new ideas that may not be fully formed. Rarely does the initial design idea become the final designed product or process. Engaging in DBR requires an open mindset to manage these expectations when it is first implemented. In addition, the DBR process is complex and requires extensive planning, time, and resources. Individuals interested in DBR are strongly encouraged to learn more about the methodologies it can include, to view additional published examples of this type of research, and to partner with researchers who have engaged in DBR previously. Also, many DBR designs produce large amounts of data which, at times, are left unanalyzed due to the urgency of the next upcoming DBR implementation or lack of resources.10 Lastly, DBR limits the ability to compare across designs since the intervention is purposefully improved after each DBR cycle. In summary, DBR encompasses a systematic, iterative approach to develop, implement, and test innovative educational interventions that address complex, real-world problems while advancing and refining educational theories to inform future research.13 With an understanding of the elements that comprise DBR, the next step is to consider how DBR can be used in future educational research to inform practice. Additional resources on the DBR process are included in Box 2. 316
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Conclusions DBR can aid pharmacy educational scholars by using a flexible, iterative, and systematic process to generate novel and creative solutions to complex problems. We offer recommendations on how practitioners can learn more about DBR, effectively integrate DBR into research processes, and promote positive change. Conflict of interest None. Disclosures None. Box 1 Design-based research case study. Problem: Pharmacy students need to co-construct knowledge during team-based projects and develop an understanding of strategies for overcoming teamwork obstacles. Currently, students are failing to do so, as only course content, not the collaboration process, is taught, supported, and assessed. Purpose: Based on interview and observation data from students and instructors, we think students need an algorithmbased technology that assists students to diagnose current obstacles and then suggest corresponding group-level regulation strategies. Plan: Who: Second year pharmacy students in an entrepreneurship course What: Develop, test, and implement a technology-enhanced learning tool (Collabucate – a mobile application) to promote effective collaboration among student pharmacists When: Spring 2016 – present Where: University of North Carolina Eshelman School of Pharmacy Why: Each student team faces a wide variety of obstacles and there is no personalized coaching to help them identify new strategies for overcoming collaboration obstacles. How: Students complete questions in the app individually each week and identify personal challenges related to their skills to, motivation for, and feelings about collaboration. The app compiles scores from the group to identify the salient challenge and offers strategies for the students to implement. The app monitors progress over time. Sample Research Questions: What are students’ experiences while using the Collabucate app? How and to what extent do students employ strategies learned through the Collabucate app? What tensions do students face in student project groups? How do students’ experiences and perceptions about collaboration change during the course? Box 2 Design-based research resources. The DBR approach: Anderson T, Shattuck J. Design-based research: a decade of progress in education research? Educ Res. 2012;41(1):16–25. Brown A. Design experiments: theoretical and methodological challenges in creating complex interventions. J Learn Sci. 1992;2(2):141–178. Collins A, Joseph D, Bielaczyc K. Design research: theoretical and methodological issues. J Learn Sci. 2004;13(1):15–42. The Design-Based Research Collective. Design-based research: an emerging paradigm for educational inquiry. Educ Res. 2003;32(1):5–8. Dolmans DH, Tigelaar D. Building bridges between theory and practice in medical education using a design-based research approach: AMEE Guide No. 60. Med Teach. 2012;34(1):1–10. McKenney S, Reeves TC. Conducting Educational Design Research. 1st ed. New York, NY: Routledge; 2012. Svihla V, Reeve R, eds. Design as Scholarship: Case Studies from the Learning Sciences. New York, NY: Routledge; 2016. Van den Akker J, Gravemeijer K, McKenney S, Nieveen N, eds. Educational Design Research. 1st ed. New York, NY: Routledge; 2006.
317
Currents in Pharmacy Teaching and Learning 11 (2019) 309–318
M.D. Wolcott, et al.
Supplementary materials Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.cptl.2018.12.002. References 1. Bauman JL, Ascione FJ, Brueggemeier RW, et al. Maintaining pharmacy education's research focus as the academy expands. Am J Pharm Educ. 2012;76(8) https:// doi.org/10.5688/ajpe768144. 2. DiPiro J. The 21st century Abraham Flexner. Am J Pharm Educ. 2008;72(4) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2576418/pdf/ajpe79.pdf Accessed 23 December 2018. 3. McLaughlin JE, Dean MJ, Mumper RJ, Blouin RA, Roth MT. A roadmap for educational research in pharmacy. Am J Pharm Educ. 2013;77(10) https://doi.org/10. 5688/ajpe7710218. 4. Persky AM, Romanelli F. Insights, pearls, and guidance on successfully producing and publishing educational research. Am J Pharm Educ. 2016;80(5) https://doi. org/10.5688/ajpe80575. 5. Cook DA, Bordage G, Schmidt HG. Description, justification and clarification: a framework for classifying the purposes of research in medical education. Med Educ. 2008;42(2):128–133. 6. The Design-Based Research Collective. Design-based research: an emerging paradigm for educational inquiry. Educ Res. 2003;32(1):5–8. 7. Dolmans DH, Tigelaar D. Building bridges between theory and practice in medical education using a design-based research approach: AMEE Guide No. 60. Med Teach. 2012;34(1):1–10. 8. Norman G. How bad is medical educational research anyway? Adv Health Sci Educ. 2007;12:1–5. 9. Brown A. Design experiments: theoretical and methodological challenges in creating complex interventions. J Learn Sci. 1992;2(2):141–178. 10. Collins A, Joseph D, Bielaczyc K. Design research: theoretical and methodological issues. J Learn Sci. 2004;13(1):15–42. 11. Barab S. Design-based research: a methodological toolkit for engineering change. In: Sawyer RK, ed. The Cambridge Handbook of the Learning Sciences. 2nd ed New York, NY: Cambridge University Press; 2014:151–170. 12. Barab S, Squire K. Design-based research: putting a stake in the ground. J Learn Sci. 2004;13(1):1–14. 13. McKenney S, Reeves TC. Conducting Educational Design Research. 1st ed. New York, NY: Routledge; 2012. 14. Kelly A. Design research in education: yes, but is it methodological? J Learn Sci. 2004;13(1):115–128. 15. Kelly AE. The role of design in educational research. Educ Res. 2003;32(1):3–4. 16. McKenney S, van den Akker J. Computer-based support for curriculum designers: a case of developmental research. Educ Technol Res Dev. 2005;53(2):41–66. 17. Reeves TC, Herrington J, Oliver R. Design research: a socially responsible approach to instructional technology research in higher education. J Comput High Educ. 2005;16(2):97–116. 18. Wang F, Hannafin MJ. Design-based research and technology-enhanced learning environments. Educ Technol Res Dev. 2005;53(4):5–23. 19. Järvelä S, Kirschner PA, Panadero E, et al. Enhancing socially shared regulation in collaborative learning groups: designing for CSCL regulation tools. Educ Technol Res Dev. 2015;63(1):125–142. 20. Edelson DC. Design research: what we learn when we engage in design. J Learn Sci. 2002;11(1):105–121. 21. Svihla V, Reeve R. Design as Scholarship: Case Studies from the Learning Sciences. New York, NY: Routledge; 2016. 22. McLaughlin JE, Roth MT, Glatt DM, et al. The flipped classroom: a course redesign to foster learning and engagement in a health professions school. Acad Med. 2014;89(2):236–243. 23. Khan KZ, Ramachandran S, Gaunt K, Pushkar P. The objective structured clinical examination (OSCE): AMEE Guide No. 81. Part I: an historical and theoretical perspective. Med Teach. 2013;35(9):e1437–e1446. 24. Khan KZ, Gaunt K, Ramachandran S, Pushkar P. The objective structured clinical examination (OSCE): AMEE Guide No. 81. Part II: organization and administration. Med Teach. 2013;35(9):e1447–e1463. 25. Easterday MW, Lewis DR, Gerber EM. Design-based research process: problems, phases, and applications. Paper Presented at: The International Conference of Learning Sciences. Boulder, CO; 2014 June 24. 26. Cobb P, Confrey J, diSessa A, Lehrer R, Schauble L. Design experiments in educational research. Educ Res. 2003;32(1):9–13. 27. Van den Akker J, Gravemeijer K, McKenney S, Nieveen N. Educational Design Research. 1st ed. New York, NY: Routledge; 2006. 28. Silver JK, Binder DS, Zubcevik N, Zafonte RD. Healthcare hackathons provide educational and innovation opportunities: a case study and best practice recommendations. J Med Syst. 2016;40(7) 10.1007s10916-019-0532-3. 29. Olson KR, Walsh M, Garg P, et al. Health hackathons: theatre or substance? A survey assessment of outcomes from healthcare-focused hackathons in three countries. BMJ Innov. 2017;3(1):37–44. 30. Bradley G. The crisis in educational research: a pragmatic approach. Eur Educ Res J. 2003;2(2):296–308. 31. Albert M, Hodges B, Regehr G. Research in medical education: balancing service and science. Adv Health Sci Educ Theory Pract. 2007;12(1):103–115.
318