A practice-based tool for engaging stakeholders in future research: a synthesis of current practices

Journal of Clinical Epidemiology 66 (2013) 666e674

A practice-based tool for engaging stakeholders in future research: a synthesis of current practices Jeanne-Marie Guisea,b,c,*, Christen O’Haired, Melissa McPheeterse, Carole Mostb, Lia LaBrantf, Kathy Leee, Erika K. Barth Cottrella,b, Elaine Grahamb b

a Division of Maternal Fetal Medicine, Department of Obstetrics and Gynecology, Oregon Health & Science University, Portland, OR, USA Department of Medical Informatics and Clinical Epidemiology, Oregon Evidence-based Practice Center, Oregon Health and Science University, Mail Code: BICC, 3181 SW Sam Jackson Park Road, Portland, OR, 97239-3098, USA c Department of Public Health & Preventive Medicine, Oregon Health & Science University, Mail Code: CB 669, 3181 S.W. Sam Jackson Park Road, Portland, OR, 97239-3098, USA d School of Nursing, Oregon Health & Science University, 3455 SW US Veterans Hospital Road, Portland, OR, 97239-2941, USA e Vanderbilt Evidence-based Practice Center, Vanderbilt University, 2525 West End Avenue, Suite 600, Nashville, TN, 37203-1738, USA f Department of Obstetrics and Gynecology, Cedars Sinai, 8700 Beverly Boulevard, Los Angeles, CA 90048, USA

Accepted 23 December 2012; Published online 13 March 2013

Abstract Objective: A major goal of patient-centered outcomes and comparative effectiveness research is to increase the involvement of stakeholders throughout the research process to provide relevant and immediately actionable information. In this report, we review the current practices for engaging stakeholders in prioritizing research. Study Design and Setting: To evaluate the range of approaches to stakeholder engagement, we reviewed the relevant literature and conducted semistructured interviews with (1) leading research organizations in the United States, Canada, and the United Kingdom; and (2) eight Evidence-based Practice Centers that engage stakeholders in comparative effectiveness research. Results: We identified 56 articles related to stakeholder engagement in research prioritization. Studies and research organizations interviewed frequently used mixed methods approaches combining in-person venues with structured ranking or voting processes such as Delphi. EPCs similarly used group web/conference calls combined with Delphi ranking or voting. Research organizations reported difficulties engaging the public and policy makers, and EPCs reported challenges engaging federal stakeholders. Conclusion: Explicit and consistent use of terminology about stakeholders was absent. In-person techniques were useful to generate ideas and clarify issues, and quantitative methods were important in the prioritization of research. Recommendations for effective stakeholder engagement and a reporting checklist were developed from the accumulation of findings. Ó 2013 Elsevier Inc. All rights reserved. Keywords: Evidence-based medicine; Qualitative research; Comparative effectiveness research; Consumer participation; Delphi technique; Research design

1. Introduction Clinicians, patients, policy makers, and other stakeholders regularly face medical decisions in complex areas that have little or no published scientific evidence [1]. To inform this process and ensure a patient-centered research agenda in the United States, national health policy and Disclaimer: The authors of this document are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the US Department of Health and Human Services. * Corresponding author. Division of Maternal Fetal Medicine, Department of Obstetrics and Gynecology, Mail Code: L-466, 3181 SW Sam Jackson Park Road, Portland, OR 97239-3098, USA. Tel.: þ503-4942101; fax: þ503-346-6815. E-mail address: [email protected] (J.-M. Guise). 0895-4356/$ - see front matter Ó 2013 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jclinepi.2012.12.010

research organizations have initiated patient-centered outcomes research and comparative effectiveness research to inform and improve health care [2,3]. Consumer involvement has been shown to positively affect research by increasing its relevance to patients, families, and communities [4]. However, the best methods to engage a wide range of stakeholders in prioritizing research are largely unknown. To support a new future research needs initiative [5] by the Agency for Healthcare Research and Quality (AHRQ), we undertook a project to describe stakeholder engagement methods used by a range of academic institutions, practice-based research networks, nonprofit organizations, and other organizations devoted to health care issues on local and national levels in the United States, Canada, and Europe.

J.-M. Guise et al. / Journal of Clinical Epidemiology 66 (2013) 666e674

What is new?  Increasing emphasis is placed on engaging stakeholders in research.  Consistent and explicit terminology to describe stakeholders and engagement methods is essential.  In-person techniques are useful to explain topics and to clarify and develop a deeper understanding of stakeholders’ perspectives.  Quantitative methods such as voting, survey, and Delphi technique are useful for prioritization.  We developed a checklist of practices and reporting in stakeholder engagement to promote consistency and advance the field.

AHRQ created the Evidence-based Practice Centers (EPCs) in 1997 to conduct research reviews for the Effective Health Care Program. Housed at major universities, medical centers, and research institutions in the United States and Canada, the EPCs synthesize existing scientific literature about important health care topics and promote evidence-based practice and decision making. EPCs apply rigorous systematic review methods to produce systematic evidence reviews and comparative effectiveness reviews on medications, devices, and other health care services, with the goal of helping patients, physicians, and policy makers make better decisions about treatments. In addition to synthesizing the evidence, the reviews identify the gaps in evidence that limited the ability to answer the systematic review questions. AHRQ has worked with a formal, broadbased stakeholder group at the agency level, and stakeholder engagement is welcome throughout the phases of the EPC systematic review process (Fig. 1). In 2010, AHRQ expanded its support for EPCs to work with various stakeholders to further develop and prioritize the future research needed by decision makers. To develop guidance for this effort, AHRQ commissioned the Oregon and the Vanderbilt EPCs to conduct a collaborative multidimensional project to outline best practices for engaging stakeholders in defining and prioritizing research needs.

2. Methods This project consisted of three complementary activities that occurred between May and September 2010, as shown in Fig. 2. We searched the published literature for methods of stakeholder engagement to set research priorities (phase I), conducted interviews with leading research organizations in the United States, Canada, and the United Kingdom known to engage stakeholders in their work (Phase II), and interviewed EPCs that engaged stakeholders for their future

667

research pilot studies in 2010 (phase III). The project protocol was submitted to the relevant Institutional Review Boards, which determined that phases I and II did not meet the definition of human subject research per 45 Code of Federal Regulations (CFR) 46.102 and that the phase III study was exempt per 45 CFR 46.101. 2.1. Phase I: Literature summary We searched PubMed/MEDLINE (OVID), PsychINFO (OVID), CINAHL (EBSCO), Cochrane Consumer Group, LocatorPlus (National Library of Medicine catalog), and Sociological Abstracts (Cambridge Scientific Abstracts) from inception through September 2010. Search terms included both keywords and subject headings: *consumer participation, exploded patient participation, stakeholder*, consumer*, citizen*, policy maker, policymaker*, policy makers, research, *Health Services Research, *CommunityBased Participatory Research, *Comparative Effectiveness Research, exp *Biomedical Research, exp *Translational Research, priorit*, research agenda, research agendas,* and English language. We reviewed relevant literature to identify methods and processes used to engage stakeholders in identifying and prioritizing research. 2.2. Phase II: Interviews with leading research and policy making organizations Organizations (Table 1) known to have expertise engaging a broad range of stakeholders (i.e., consumers, clinicians, and policy makers) were invited to participate in interviews between June and July 2010. Each key informant (KI) completed an ‘‘EPC Conflict of Interest Disclosure Form’’ before being interviewed and no conflicts disclosed precluded participation for any of the invited organizations. Two investigators interviewed KIs about their experiences engaging stakeholders, using a semistructured interview guide that included questions regarding experiences engaging stakeholders, the types of stakeholders engaged, the methods used to engage them, the processes used to set research priorities, and the impact of stakeholder engagement. All telephone interviews were digitally recorded and transcribed, and the transcripts were reviewed to identify and code themes. Four additional investigators who did not participate in interviews independently reviewed transcripts to identify themes. All investigators (six: two interviewers and four additional investigators) met to discuss themes and resolve discrepancies. 2.3. Phase III: EPC interviews We evaluated EPC experiences of engaging stakeholders in two ways: (1) we reviewed EPC proposals to assess the methods proposed by EPCs to engage stakeholders in future research needs development and (2) we conducted semistructured interviews with EPC directors and staff at

668

J.-M. Guise et al. / Journal of Clinical Epidemiology 66 (2013) 666e674

AHRQ Effective Health Care Program

Program Phase

Topic Development

Topic Refinement

Research Review

Research Needs Development

Report Translation and Dissemination

Generate nominations relevant to “realworld” health care decisions

Clarify nomination and decisional dilemma

Clarify and refine key questions to accurately reflect the decisional issue

Provide clinical & methodological input to EPC’s protocol

Provide input to develop and prioritize research gaps

Test and provide input on summary guide development

Engagement Opportunities

Topic Generation

Engagement Purpose

Points of Stakeholder Engagement for Systematic Reviews

Nominate topics via the Web

Clarify nomination intent

Provide “real world” input from provider, patient, and policy perspectives

Frame clinical context of nomination Provide input on clinical logic

Provide public comment via Web

Provide methodological & clinical input as a TEP member

Provide input via peer review

Provide public comment via Web

Participate in discussions to describe research gaps and prioritize research needs

Participate in focus groups

Participate in individual interviews

Provide expert review of summary guides

Fig. 1. AHRQ stakeholder engagement. Source: Scientific Resource Center for AHRQ Effective Health Care, Oregon Evidence-based Practice Center. AHRQ, Agency for Healthcare Research and Quality; EPC, Evidence-based Practice Center; TEP, technical expert panel.

the conclusion of the pilots to identify lessons learned about the stakeholder engagement process. We obtained copies of each EPC’s protocol for the Future Research Needs pilot projects and extracted the information specific to methods for stakeholder engagement to prioritize future research. We then sent letters of inquiry to the directors of the eight EPCs that had received American Recovery and Reinvestment Act funding for future research needs activities conducted as pilot projects, asking them to participate in brief, semistructured interviews to discuss their experience of engaging stakeholders in those future research needs pilot projects. Each EPC identified the individual that they felt could best represent the experience of their EPC to participate in the interview. The interviews were conducted by one investigator with a research assistant present and recorded. A list of potential questions was developed to guide the conversation. The discussions took place between August 19 and September 21, 2010.

3. Results 3.1. Phase I: Literature summary After the exclusion of editorials, commentaries, and studies that did not provide data for stakeholder engagement, we identified 56 articles that provided information about methods for engaging stakeholders specifically for the purpose of prioritizing future research (Fig. 3). The literature lacked consistent terminology and definitions for stakeholders and lacked consistency in processes for

stakeholder engagement. For the purposes of this report, stakeholders are individuals or organizations who have an interest, personal or professional, in the topic. We developed operational definitions of the major categories of stakeholders based on definitions, terms, and stakeholder categories reported by the researchers (Table 2). 3.2. The a priori identification of relevant stakeholder groups An important step to ensure that the intended stakeholders are engaged is to list the relevant stakeholder groups before initiating recruitment. This step makes it possible to monitor and maintain the intended spectrum and distribution of stakeholders during recruitment and analysis. Once stakeholder groups were identified, snowball sampling was one of the most commonly reported methods used to identify pertinent stakeholders [6e12]. This method is thought to be particularly useful to identify members of special populations that are more difficult to locate, such as people with rare conditions. Because people may be more likely to recommend other individuals who are similar to themselves, it is possible that bias could be introduced with this method. Once stakeholders were selected, letters and e-mail were the most common methods used to invite stakeholders to participate. 3.3. Engaging stakeholders to prioritize research Engaging stakeholders in the identification and prioritization of research is intended to make the research more

J.-M. Guise et al. / Journal of Clinical Epidemiology 66 (2013) 666e674 Phase III EPC interviews (Vanderbilt EPC)

Phase II Key Informant Interviews (Oregon EPC)

Phase I Literature Summary (Oregon EPC)

669

Review of EPC Protocols • Stakeholder engagement • Future research needs pilot projects

Key Informant Recruitment and Interview Guide • Identification and email invitation • Conflict of interest determination • Developed semi-structured interview guide

Invitations and EPC Interview Guide • Developed semi-structured interview guide • Emailed invitations to EPCs and stakeholders

Literature Review to Identify Methods for: • Identification of stakeholder groups • Selection of representative stakeholders • Recruitment of stakeholders • Engagement of stakeholders to prioritize research

Key Informant Interviews

EPC Interviews • Obtained consent to record interviews • Interviewed 8 ARRA-funded EPCs

Synthesis

Analysis of Key Informant Interviews

Literature Search PsychINFO (OVID), CINAHL (EBSCO), Cochrane Consumer Group, LocatorPlus (NLM catalog), Sociological Abstracts (CSA) • Selected key articles

engagement to prioritize research • Tradeoffs of methods

consent to record interviews • 2 investigators conducted interviews • 13 interviews with 15 key informants (1-2 key informants per interview)

Analysis of EPC Interviews • Transcribed interviews • Project lead coded themes • Collated and reviewed notes from interviews

• 2 investigators independently coded themes • 4 other investigators reviewed transcripts to identify additional themes • Collated and reviewed results

Cumulative Synthesis

Fig. 2. Project overview. EPC, Evidence-based Practice Center; NLM, National Library of Medicine; CSA, Cambridge Scientific Abstracts; ARRA, American Recovery and Reinvestment Act.

relevant to everyday care and to facilitate more rapid uptake of results. Table 3 depicts the methods researchers used among published studies to engage stakeholders in generating and prioritizing topics for future research [13]. Investigators commonly used in-person methods such as focus groups, forums, and conferences, either alone or in combination with other methods for the initial engagement of stakeholders. In-person methods are advantageous

because they allow for interaction to clarify participants’ questions, and they elicit a deeper understanding of viewpoints. When priorities or rankings are desired objectives, structured methods such as such as voting, Delphi, questionnaires, or other explicit rating processes were used, with the Delphi technique being the most common. The Delphi technique involves administering a series of two to five questionnaires sequentially to gain consensus.

Table 1. Research organization experts interviewed about stakeholder engagement and research prioritization Types of stakeholders engaged Key informant 1 2 3 4 5 6 7 8 9 10 11 12 13

Organization type

Location

Research funder Research organization Academic institution Research funder Research organization Research organization Policy maker Research organization Government organization Academic institution Research organization Research organization Research organization

United States United Kingdom Canada United States United States United States United States United States United States United States United States Canada United States

Consumer

Clinician

O O

O

O O O O O O

O O O O O O O

O

O

Policy maker

Researcher

O O O O O

O O O O O O O O O O O O O

O O O O O O

Research funder

Manufacturer O

O O

O O O

O

670

J.-M. Guise et al. / Journal of Clinical Epidemiology 66 (2013) 666e674 152 abstracts selected

20 Excluded • 4 Not health research • 1 Not developed country • 1 Conceptual, theoretical, or modeling • 14 No data (e.g. editorial, instructional)

Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research. 2009.

133 full-text articles assessed for stakeholder engagement 66 Excluded • 7 Not developed country • 16 Conceptual, theoretical, or modeling • 43 No data (e.g., editorial, instructional)

67 articles engaged stakeholders

11 articles engaged stakeholders for other reasons

56 articles engaged stakeholders to prioritize research

Fig. 3. Results from literature search.

Advantages of the Delphi method include: (1) scalability for small and large groups of people, (2) ability to administer across a wide geographical area, (3) ability to arrive at a final ranked list, (4) stakeholders may be more comfortable expressing their opinions on paper (less public), and (5) the method helps to ensure that every participant’s priorities are known. Overall, a mixed methods approach may be optimal because the in-person discussions allow for explanations and clarification of concepts, whereas questionnaires, Delphi, or voting allow for quantitative analyses (either totals for all individuals or weighted for groupings).

3.4. Phase II: Interviews with leading research and policy making organizations Thirteen semistructured telephone interviews were conducted with 15 KIs in the United States, Canada, and the United Kingdom. Even among individuals with extensive experience engaging stakeholders, stakeholder terminology was unclear or inconsistently applied. As stated in one interview, ‘‘When we started the project, we had to define who the stakeholder was . and that in itself really proved to be quite challenging.’’ Informants reported particular challenges in recruiting the ‘‘public’’ and policy

Table 2. Stakeholder groups Stakeholder group Consumer (public, patient, and caregiver) Clinician (health professional) Policy maker

Researcher

Research funder Insurer/payer

Manufacturer

Description An individual or advocacy group representing individuals who have a health condition, use health care services, and/or who are members of the community Health care or public health provider (e.g., academic, rural/frontier, and community) and medical and/or public health organizations An individual or organization who is involved in health care policy (e.g., local, state, provincial, and federal legislators and staff)  Medical organizations  Governmental organizations (e.g., Veteran’s Affairs, Agency for Healthcare Research and Quality, and so on) An individual who conducts and/or facilitates research activities in:  Basic, translational, clinical sciences; research methodology, public health or health services, and systematic reviews A public or private organization that funds research (e.g., National Institute of Health, Susan G. Komen Foundation, and American Cancer Society) An organization or agency that pays for health-related goods and services (e.g., Blue Cross Blue Shield, Medicaid, and Medicare) or a business group that pays for health insurance (e.g., employers and government) A business group that produces health-related items (e.g., pharmaceuticals and medical devices)

J.-M. Guise et al. / Journal of Clinical Epidemiology 66 (2013) 666e674

671

Table 3. Literature results: methods used to engage stakeholders and prioritize research Engagement

Prioritization

Engagement and prioritization

Symposia/ Nominal One-to-one Town workshops/ group (in-person or Focus Citizens’ Stakeholder group telephone)a,b groups juries meetings conferences/forums Consensus Voting technique Consumers Clinicians Policy makers Manufacturers Researchers Research funders

4 3 4 4 2

15 7 6 2 11 2

4

2 1 1

7

1

6 3

7

9 7 7 3 9 3

2 1

2 4 3 2 4 1

One-time priority questionnaire

Delphi technique

9 8 5

5 9 5

5 2

4

Abbreviation: AHRQ, Agency for Healthcare Research and Quality. a Numbers of studies (see AHRQ research report [12] for table with cited references). b Blank cells indicate no studies found.

makers. Regarding the ‘‘public,’’ informants mentioned that they may get industry participants rather than the lay consumers they intended. Informants mentioned that policymakers often ignored their contact attempts or would refer the invitation to someone lower in the hierarchy resulting in recruitment of ‘‘a midlevel decision maker who is less likely to have influence over policy than the person we invited.’’ Media, Internet, leveraging of personal contacts, mailing, and cold calling were commonly reported techniques for recruiting stakeholders among organizations. Table 4 presents highlights from KI interviews. The organizations reported using e-mail, conference calls or Webinars, focus groups, citizens’ juries, workshops conferences, and the modified Delphi technique, with the two most common being one-to-one meetings (in-person or by phone) and working group/ committee/council meetings combined with Delphi or ranking when quantitative rankings were wanted. According to KIs, the advantages of in-person meetings included: (1) ensuring a period of focused stakeholder attention to the subject, (2) fostering relationships between stakeholders, and (3) promoting a better understanding of each other’s needs and priorities through discussion.

3.5. Phase III: EPC interviews Representatives from eight EPCs were interviewed regarding their experiences engaging stakeholders during their future research needs pilot projects. Six of eight EPCs considered the involvement of stakeholders to be of value in identifying research gaps, with caveats noted about being restricted on the number of stakeholders they were allowed to contact and questions about whether the group was appropriately representative. EPCs included between 6 and 30 stakeholders for their research gaps pilot projects. Target groups for stakeholder engagement included primary researchers, methodologists, patient/ consumer advocates, policy makers, funders, payers, industry (for devices), and patients themselves. All EPCs considered their stakeholder panels to be well balanced in terms of representing different constituencies. However, several questioned how ‘‘representative’’ should be defined, absent empirical evidence about numbers, and breadth of stakeholders required for optimal input. Although all EPCs felt that patients should be included as stakeholders in other EPC processes, several questioned whether patients would contribute effectively to

Table 4. Key informant interview highlights Barriers to stakeholder engagement  Lack of time on the part of stakeholders (busy)  Lack of release time and compensation for members of the public  Researcher need for quick response (time frame too short for community to weigh in)  Stakeholder needs not met in previous engagement Factors in successful engagement  Engage stakeholders early in the process  Before engagement, clearly detail expectations (e.g., anticipated commitment of time and types of activities)  Maintaining ongoing relationships is important to building trust and credibility  Provide opportunities for people to ask questions before meetings (particularly for consumers or others who are unfamiliar with research)  Premeeting information materials promote comfort with the topic and enable informed discussion  Premeeting ‘‘icebreakers,’’ especially when engaging stakeholders with differing experiences/perspectives  Having someone with similar training as the stakeholder can be helpfuldparticularly thought to be important for clinicians  No stakeholder’s comment is considered superior or inferior to another perspective  Followup presentation of results is important to stakeholders  Be clear about the stakeholder roleddo not expect community members to do academic duties  Be sensitive to the time constraints of all stakeholders Facilitators can be useful for stakeholder engagement  Trained/neutral facilitator encourages participation and focuses discussion at in-person meetings  Facilitator knowledgeable of various stakeholder environments (cultural understanding)  Skilled facilitator ensures all stakeholders are heard and manages dominance issues

672

J.-M. Guise et al. / Journal of Clinical Epidemiology 66 (2013) 666e674

Table 5. Checklist for reporting stakeholder engagement Process Engagement purpose

Stakeholder orientation information and materials

Stakeholder recruitment

Methods of engagement

Prioritization methods

Analysis and synthesis

Conclusion

Tasks  State purpose of stakeholder engagement (e.g., topic identification, prioritization, and so on) and desired results or outcomes (affects selection of recruitment, engagement, and prioritization methods)  Provide orientation to the program (e.g., effective health care and systematic evidence reviews)  Communicate scope, aims, and desired product/ outcome of engagement  Delineate stakeholder roles, responsibilities, and expectations  Describe extent of commitment, including methods of engagement (e.g., single Webinar on a specific date), frequency of activities, and project schedule  Explain compensation plan, if any (e.g., travel reimbursement, honorarium, or other compensation)  Inform of conflict of interest disclosure requirements  Provide contact information for project team (e.g., e-mail address and phone number of investigator and research staff contact)  Develop a priori list of relevant stakeholder groups, including numbers of stakeholders for each group  Identify specific individuals within each stakeholder group to contact  Plan one or more methods of contact (e.g., write e-mail scripts or telephone dialog to convey expectations of stakeholders)  Plan recruitment followup (e.g., repeat e-mails or return calls for refusals or nonrespondents)  Contact individual stakeholders and document results of individual responses for calculation of overall response rates  Monitor distribution of stakeholder groups; add individuals to recruitment as necessary to ensure target stakeholder groups are included  Assess and identify engagement methods and criteria for selection for each stakeholder group based on: B Purpose of engagement B Characteristics of stakeholders to be engaged (e.g., geographic locations, numbers of stakeholders, range of perspectives, existing relationships among stakeholders, familiarity with research area or research prioritization generally, and technological capacity) B Context of engagement (e.g., staff, time, budget, technological capacity, and complexity of research area)  Provide criteria for prioritization if previously established (by the sponsor or by previous engagement activity)  Document individual or group development of criteria for prioritization if not previously established  Document, review, synthesize and report stakeholder research priorities B Overall and by stakeholder group B Weighting of input if applicable (e.g., do two votes from one organization count as one or two?)  Plan for review and validation of results by stakeholders to ensure that their intended priorities are accurately communicated  Distribute final product or report to stakeholders  Express appreciation for participation and contribution

Reporting checklist  State purpose of stakeholder engagement

 Report conflicts of interest and disposition  Report any compensation

 State the types stakeholders targeted with rationale a priori  Report methods used to identify, sample, and contact each stakeholder group  Document recruitment dates  Report distribution of stakeholders planned and recruited

 Report engagement method(s) and describe process for the identification of research topics  Document number and types of engagements  Report participation rates and distribution in each engagement

 Document prioritization criteria when applicable  Describe methods of prioritization of research topics

 Provide assessment of stakeholder distribution  Provide details of synthesis and analytic methods including weighting of stakeholder group’s responses and calculation of participation rates  Provide results both overall and by stakeholder group  Verify priorities with stakeholders  Provide prioritized list of future research topics (ideally after public posting) (Continued )

J.-M. Guise et al. / Journal of Clinical Epidemiology 66 (2013) 666e674

673

Table 5. Continued Process

Tasks

Reporting checklist

 Gather input from stakeholders on engagement experience to evaluate the process  Advise stakeholders of opportunities for future participation and any plans for future contact (if relationship will be maintained)

identifying and specifying research gaps. Two of the EPCs felt strongly that patients did not have a role in this particular process, one felt strongly that the patient participation in their project changed the outcome in a positive way, and the remainder were unsure about whether patients should be included. Those EPCs that felt that patient stakeholders did not have a role in specifying future research needs indicated that this process in particular was intended to capture the perspective of researchers and required their methodological expertise. The particular work of identifying well-specified research questions and approaches to answering them was, in their view, in the realm of trained researchers, with the understanding that patients should have been engaged in the systematic review process that led to the initial identification of research gaps. Patients in this way would have ‘‘set the stage’’ with relevant research focus. EPCs most commonly used group conference calls along with e-mail and Web-based prioritization through ranking exercises or Delphi technique, and one presented a Webinar to bring stakeholders up to speed on the topic. EPCs identified challenges that may differentially impact certain stakeholders such as: Timing. When projects have short timelines or immediate start dates, it can be difficult to coordinate stakeholder participation given different work schedules and timing issues; this could result in an inability to engage some stakeholders. Engaging federal partners. Several EPCs identified barriers to engaging funders, particularly federal funders. EPCs reported that federal funders sometimes required high levels of clearance or approval to participate, which was ultimately time prohibitive.

4. Discussion National attention has recently focused on engaging patients, clinicians, and policy makers to create a more patient-centered research agenda that is relevant to everyday practice and directly actionable. Stakeholder involvement may increase a researcher’s understanding of the decisional dilemmas facing the public, thus increasing the relevancy of the research. Several themes emerged from the literature and interviews with research organizations and EPCs:

 When possible, summarize lessons learned to promote continuous improvement

 Engage stakeholders early in the process;  Be attentive to building relationships and maintaining credibility and trust between the investigative team and stakeholders to ensure successful long-term relationships and opportunities for multiple engagements;  Establish credibility with stakeholders by consistently producing and disseminating high-quality project products over the life of the project or beyond;  Use multiple methods (i.e., in-person meeting and voting to obtain both details of thought and ranking of research priorities);  Allow the time needed for engagement of all relevant stakeholder groups (for instance, consumers may require more time to recruit and engage than other groups);  Document all stakeholder input, including audio or video recordings, that can be transcribed;  For group discussions (in-person or online) employ a skilled, neutral facilitator to ensure balanced participation and focused discussions;  Plan strategies for managing disruptive or dominating stakeholders and for resolving conflicts that may arise;  If prioritizing broad or complex research topics, pare down the list in advance (i.e., through electronic vetting) or through sequential processes to ease stakeholder workload and focus attention on key topics;  Conduct icebreaker sessions at in-person activities whenever stakeholders have diverse backgrounds or are not already acquainted;  Provide brief, easy-to-understand informational materials at the beginning of engagement and before any in-person meetings for complex initiatives or for stakeholders who are unfamiliar with the engagement process or the topic; and  Offer opportunities for clarification of research topics (discussion or question-and-answer session). If conducting an in-person meeting, also offer these opportunities before the meeting. Results from the literature scan and interviews were synthesized into the above recommendations for stakeholder engagement to prioritize future research needs, and incorporated into a proposed reporting checklist for EPCs (Table 5). These recommendations were developed to enhance consistency and transparency in the methods used for stakeholder engagement, and ultimately these items

674

J.-M. Guise et al. / Journal of Clinical Epidemiology 66 (2013) 666e674

may also be used when conducting systematic reviews that incorporate stakeholder engagement. Our analysis was limited to the description of best practices, and did not prospectively or retrospectively compare the various methods for their impact on research processes (extended timelines, improved products), impact on research groups (requirements for additional training, time, and so on), or costs (additional effort). As recommended by the Institute of Medicine, AHRQ, the National Institutes of Health, and the Patient-Centered Outcomes Research Institute, the activity of engaging stakeholders in research is rapidly expanding. Although knowledge is increasing on best practices, questions still remain about how to consistently identify stakeholders, evaluate the process, and disseminate results and reports from these activities. This report provides a summary of common practices in stakeholder engagement that can inform researchers wishing to engage stakeholders and similarly inform reviewers of such reports.

[2]

[3]

[4] [5]

[6]

[7]

[8]

[9]

Acknowledgments [10]

The authors extend their deepest appreciation for the contributions of the key informants and Evidence-based Practice Center participants who agreed to be interviewed for the project. This project was funded under Contract Nos. 290-2007-10057-I (Oregon Evidence-based Practice Center) and 290-2007-10065-I (Vanderbilt Evidence-based Practice Center) from the Agency for Healthcare Research and Quality, US Department of Health and Human Services.

[11]

[12]

[13]

References [1] IOM Roundtable on Evidence-based Medicine. Learning What Works Best: The Nation’s Need for Evidence on Comparative Effectiveness in Health Care: An Issue Overview. In: Institute of Medicine (US) Roundtable on Value & Science-Driven Health Care. Learning What Works: Infrastructure Required for Comparative Effectiveness Research: Workshop Summary. Washington (DC): National Acade-

mies Press (US); 2011. Appendix A. Available from: http://www. ncbi.nlm.nih.gov/books/NBK64784/. Accessed February 14, 2013. Institute of Medicine. Initial national priorities for comparative effectiveness research. Washington, DC: The National Academies Press; 2009. Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. Washington, DC: US Department of Health & Human Services; 2009. Staley K. Exploring impact: public involvement in NHS, public health and social care research. Eastleigh, UK: INVOLVE; 2009. Chang S, Carey T, Kato E, Guise J-M, Sanders G. Identifying research needs for improving health care. Ann Intern Med 2012;157: 439e45. Jinks C, Ong BN, O’Neill TJ. The Keele community knee pain forum: action research to engage with stakeholders about the prevention of knee pain and disability. BMC Musculoskelet Disord 2009;10:85. Niner S, Pirkis J, Krysinska K, Robinson J, Dudley M, Schindeler E, et al. Research priorities in suicide prevention: a qualitative study of stakeholders’ views. Advances in Mental Health 2009;8(1):48e56. Griffiths KM, Jorm AF, Christensen H, Medway J, Dear KB. Research priorities in mental health, Part 2: an evaluation of the current research effort against stakeholders’ priorities. Aust N Z J Psychiatry 2002;36(3):327e39. Gould N, Huxley P, Tew J. Finding a direction for social research in mental health: establishing priorities and developing capacity. J Soc Work 2007;7(2):179e96. Sharan P, Gallo C, Gureje O, Lamberte E, Mari JJ, Mazzotti G, et al. Mental health research priorities in low- and middle-income countries of Africa, Asia, Latin America and the Caribbean. Br J Psychiatry 2009;195:354e63. Sibbald SL, Singer PA, Upshur R, Martin DK. Priority setting: what constitutes success? A conceptual framework for successful priority setting. BMC Health Serv Res 2009;9:43. Schur CL, Berk ML, Silver LE, Yegian JM, O’Grady MJ. Connecting the ivory tower to main street: setting research priorities for real-world impact. Health Aff (Millwood) 2009;28(5): w886e99. O’Haire C, McPheeters M, Nakamoto E, LaBrant L, Most C, Lee K, et al. Engaging stakeholders to identify and prioritize future research needs. Methods Future Research Needs Report No. 4. (Prepared by the Oregon Evidence-based Practice Center and the Vanderbilt Evidence-based Practice Center under Contract No. 290-200710057-I.) AHRQ Publication No. 11-EHC044-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2011. Available at www.effectivehealthcare.ahrq.gov/reports/final.cfm. Accessed February 14, 2013.