- Email: [email protected]
Scaling Out a 4-H Healthy Meeting Initiative: Challenges in Implementation and Comprehensive Evaluation
Perspective Scaling Out a 4-H Healthy Meeting Initiative: Challenges in Implementation and Comprehensive Evaluation Laura E. Balis, PhD1; Samantha M. Harden, PhD2 ABSTRACT The national Cooperative Extension System delivers a number of nutrition and physical activity interventions, but it is unknown how an intervention may translate from 1 state within the system to another. Using the reach, effectiveness, adoption, implementation, maintenance framework for program evaluation can improve intervention scale-out. Adoption is a key dimension of reach, effectiveness, adoption, implementation, maintenance: if delivery personnel do not deliver an intervention, it can have no impacts on health. Here, differences are discussed regarding adoption rates between state Extension systems when scaling out a 4-H healthy meeting intervention. This experience provides suggestions for improved scale-out of Extension programs, including state-specific adaptation and pragmatic data collection. Key Words: adoption, Cooperative Extension, physical activity, RE-AIM framework, scale-out (J Nutr Educ Behav. 2019; 51:1020−1024.) Accepted May 7, 2019.
INTRODUCTION The national Cooperative Extension System1 connects research to practice through land-grant universities in each state and territory. With its broad reach2 and strong history of improving agricultural practices,3 Extension is positioned to tackle the nation’s biggest challenges. Following the release of the National Framework for Health and Wellness in 2014,4 chronic disease prevention became an Extension priority area. Extension delivers a number of nutrition and physical activity interventions, however, scaling out (ie, implementing an intervention in a new setting and to a new population)5 from 1 state system to another remains challenging.6,7 Part of this challenge is that Extension struggles with program evaluation. Extension programs are often evaluated with a posttest only (generally directed toward program satisfaction) or no evaluation beyond reporting attendance.8,9 This results in 2 translational issues: (1) programs are developed in 1 state and adopted in
1
another before full evaluation efforts are completed or reported, and (2) the degree to which programs are feasible for delivery outside their original state is unknown. To scale out Extension programs better, comprehensive program evaluation is important. Recent publications10−12 recommended planning and evaluating Extension programs through the reach, effectiveness, adoption, implementation, maintenance (RE-AIM) framework, which goes beyond measuring who participates in a program (reach) and the primary outcomes (effectiveness) to include systems-level dimensions (adoption, implementation, and maintenance), to assess whether an intervention works in the real world.10 Extension has traditionally focused heavily on reach (specifically, the number of individuals participating in a program), because educators are mandated to track the number of contacts, but the other RE-AIM dimensions were not emphasized. Without strong adoption rates
Extension, University of Wyoming, Lander, WY Department of Human Nutrition, Foods, and Exercise, Virginia Tech, Blacksburg, VA Conflict of Interest Disclosure: The authors have not stated any conflicts of interest. Address for correspondence: Laura Balis, PhD, Extension, University of Wyoming, 130 Eugene St, Lander, WY 82520; E-mail: [email protected] Ó 2019 Society for Nutrition Education and Behavior. Published by Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.jneb.2019.05.288
2
1020
(ie, the number, proportion, and representativeness of delivery agents willing to initiate a program),10 the reach and effectiveness of programs cannot be fully achieved. One Extension target audience is youths. The 4-H program, the nation’s largest youth development program, is delivered through Extension and reaches nearly 6 million youths/yr.13 The 4-H program stands for Head, Heart, Hands, and Health.13 This widely recognized 4-H branch of Extension offers a number of youth programming types: in-school and after-school programs, camps, and community clubs.13 Whereas 4-H programs are often strong in the head, heart, and hands (through engaging in project learning, leadership, and service),14−16 many state 4-H systems struggle to address contemporary health challenges such as poor dietary quality and sedentary time.17 4-H is of particular interest because program offerings are educational in nature and youths are inspired to learn by doing.13 In this way, health behaviors that are observed during 4-H programming may extend into other facets of the youths’ lives. However, before 2015, there were no structured programs in place to help 4-H volunteers and staff integrate the fourth H of health in meeting practices.18 One intervention developed to assist 4-H programs with promoting healthy habits is the
Journal of Nutrition Education and Behavior Volume 51, Number 8, 2019
Journal of Nutrition Education and Behavior Volume 51, Number 8, 2019 4th H for Health Challenge (here called the Challenge), an evidenceinformed intervention developed through a participatory approach between researchers and 4-H volunteer leaders and administrators in Maine, New Hampshire, and Massachusetts.18 The Challenge is designed to help club leaders implement healthy meeting practices: serving only water as the beverage, serving a fruit or vegetable as the snack, and including at least 15 minutes of moderate to vigorous physical activity.18 The Challenge was implemented in 16 states (according to a search of each state’s Extension Web site conducted for this perspective). Although a 30% adoption rate within the nation seems promising, it is currently unknown how many countylevel educators and volunteers implement the Challenge when it is adopted by state Extension systems. The purposes of this perspective are to provide suggestions for improved scale-out using findings regarding differences in Challenge adoption rates among state Extension systems, and to discuss challenges and results of pragmatic data collection.
DISCUSSION Setting To elucidate lessons in the process of adopting an evidence-based program in a new state, Wyoming was used. In Wyoming, 26 4-H educators managed county 4-H programs and supervised approximately 1,677 adult volunteers. The Challenge was introduced to Wyoming by an Extension health educator in an effort to bridge health promotion efforts across Extension program areas. Notably, no funding or administrative support was provided to adopt the Challenge; that is, the educator had the opportunity to champion the program, but it was not mandated or part of a research trial.
Assessing Need As part of a pragmatic research study, all 4-H educators and volunteers were invited to complete an online survey to collect baseline demographic information and determine whether there was a need to implement the
Challenge. The purposes of the survey were to ascertain demographic information about the educators and volunteers and understand their current meeting behaviors (snack options and physical activity opportunities). About 30% of educators and 10% of volunteers responded to the survey. Their responses indicated that many Extension practitioners were offering water during meetings, but few served fruit and vegetable snacks most or every time, and 35% responded that physical activity was included most or every time. Similar to other research, it is likely that those most interested in healthy meetings responded to the survey, yet the need remained to implement healthy meeting initiatives.
Promoting the Challenge To see how the Challenge would look in Wyoming, 2 educators agreed to serve as program champions and pilot-test the Challenge for 3 months. One of these educators then provided testimony about the program when the next 4-H program year started. The Challenge was promoted to 4-H educators through several methods. A brief (approximately 15-minute) presentation was given to all 4-H educators at the annual statewide team meeting. E-mails with details about the Challenge were sent to all 4-H educators, and information was posted to the state 4-H Web site for educators to access. Educators then recruited volunteer club leaders through 4-H newsletters and achievement night presentations.
Determining Adoption Rates After educators recruited volunteers to participate in the Challenge, a survey was administered to determine how many educators and volunteers adopted the Challenge and whether those who adopted were representative of the overall population. This survey also contained standard demographic items and organization information. Unfortunately, no 4-H educators completed the pre-program survey and only 17 4-H volunteers (eg, 22% of baseline respondents and 1% of all volunteers in Wyoming) completed the
Balis and Harden
1021
survey indicating they planned to adopt the Challenge.
Challenges in Scaling-Out The researchers applied an evidencebased approach to translation: needs assessment, pilot-testing, and establishment of a program champion, and then full-scale integration. Yet, in Wyoming, at a basic level, the Challenge failed. Limited responses (30% of educators and 10% of volunteers) were received and few eligible 4-H practitioners (no educators and 1% of volunteers) adopted the Challenge. This is surprising because there were high county administrator adoption rates (82%) in a comprehensive evaluation of the Challenge in 3 states (Maine, New Hampshire, and Massachusetts).19 Notably, volunteer and educator adoption rates were not calculated in these states.19 The Challenge was a promising intervention for translation, but enthusiasm and success were not seen in Wyoming. The researchers offer a perspective on why this happened, as well as some potential solutions.
IMPLICATIONS FOR RESEARCH AND PRACTICE As Extension continues its efforts to deliver evidence-based programs and show broad impacts, an understanding of how to scale out programs from 1 state to another will be helpful. Without a strategic effort to assess and improve adoption rates, new initiatives such as the Challenge fail to gain traction and become adopted within their intended delivery systems.5,20 This can lead to implementation and de-implementation of new initiatives that, if they were successfully implemented, might have led to changes in the desired behaviors and public health benefits. The difficulties in scaling out the Challenge can provide useful lessons for both practitioners and researchers. These can be categorized according to the need for state-specific adaptations and the issues faced when conducting research without funding or staff resources.
1022
Journal of Nutrition Education and Behavior Volume 51, Number 8, 2019
Balis and Harden
Need for State-Specific Adaptations Extension is a complex system with organizational structures that vary by state.3,21 For example, some state Extension systems have administrators located on campus as well as in district and county offices, whereas some have administrators located only on campus.21 In addition, in some states, educators (also called agents) have the autonomy to develop or select programs to deliver, whereas in others they must select programs from a preapproved list.22 In the New England states, where the Challenge was developed, it is possible that county-level staff, who had administrative roles, experienced fewer barriers to implementing the Challenge than did 4-H educators in Wyoming. Different strategies may be needed in each state to meet the needs of different organizational structures. In Wyoming, although 4H educators expressed interest in the Challenge when it was presented at their team meeting and agreed to pilot-test the Challenge, no educators completed the pre-program survey indicating that they planned to adopt the Challenge. Although 4-H educators held meetings for their programs (eg, junior leaders), 4-H volunteer club leaders likely held more meetings, and therefore had more opportunities to provide beverages and snacks and lead physical activities. In the future, efforts to improve Challenge adoption rates may need to target both educators (by encouraging them to promote
the Challenge to their volunteers) and volunteers. Future research could test implementation strategies (ie, methods to improve program adoption and implementation, such as training or incentives)23 at various levels to increase adoption rates. The table (adapted from Powell et al24) details examples of implementation strategies to be used during the planning phase of an intervention to increase adoption. In addition to targeting implementation strategies to levels appropriate for each state, state-specific adaptations may be needed to scale out the Challenge.5 For example, in the New England states, pins were used as an incentive to complete the Challenge. In Wyoming, when the Challenge was initially presented to 4-H educators, they indicated that pins were not widely used and would not serve as an effective incentive. Limited research has been done on scaling out Extension interventions to other states; however, results suggest that place-based adaptations can improve adoption rates.6,25−27 Although the Challenge was developed through a participatory approach, in the future, a state-specific integrated research−practice partnership might be used to adapt Extension programs to additional states before implementation.25,27,28
Need for Pragmatic Research It is unknown whether the barriers to adoption in Wyoming were state-specific (ie, the Challenge should have been adapted in Wyoming before
being introduced) or whether program adoption rates are also low in other states. It is possible that the different adoption rates between Wyoming and the 3 New England states resulted from the difference between a funded, researcher-initiated approach and an educator-initiated pragmatic one. Similar strategies were used in both studies (eg, promoting the Challenge through interactive presentations and offering technical assistance).19 However, in Wyoming, there was not strong support from the state 4-H office, which may have contributed to low adoption rates. Because other states implementing the Challenge also may not be part of funded research studies, future research could assess whether the Challenge was adopted and maintained in these systems. Pragmatic research approaches (those that address specific, practice-based needs and questions; consider contextual factors, resources, and relationships; and use measures that are feasible and lead to action) are recommended for this work.29 As part of this pragmatic approach, perceptions of both those who adopt the Challenge and those who choose not to adopt it might be assessed to understand barriers better.27 In the New England Challenge evaluation, volunteers who successfully implemented the Challenge were interviewed to assess implementation, but perceptions of those who were not deemed successful were not captured.19 Perceptions of nonadopters are important because they can be used to adapt the program for future iterations.27
Table. Examples of Planning Phase Implementation Strategies to Increase Adoption Implementation Strategy Assess for readiness and identify barriers Develop a formal implementation blueprint Conduct local consensus discussions; build a coalition Involve clients and families
Target Level
Example
Administrators, Educators
Survey administrators and educators to assess degree of readiness and potential barriers/facilitators Collaborative to create a plan including staff responsibilities and funding Determine whether the Challenge is appropriate to meet needs; divide responsibilities Include throughout the process and provide training on the Challenge Announce that the Challenge will be implemented; document the commitment in writing
Administrators, Educators Educators, Volunteers Volunteers, Youth, Parents
Mandate change; obtain formal commitments
Administrators 24
Note: Adapted from Powell et al.
Journal of Nutrition Education and Behavior Volume 51, Number 8, 2019 There were some limitations to this work. The sample size for the pre-program survey was small; however, these results are important to understand adoption rates and intervene to improve them. In addition, in pragmatic settings such as Extension, large sample sizes are not always possible. The total number of volunteers (N = 1,677) included both project volunteers (who might not have had an influence over snack and physical activity practices) and club leaders. A more accurate denominator for reach would include only club leaders, but it was unavailable. In the future, better efforts to monitor who adopts the Challenge (eg, through registering on a Web site) are needed to capture data on those who may adopt the Challenge but not complete the survey. To determine whether interventions are successfully implemented in the real world, it is important to capture adoption rates. This perspective highlights that state-specific adaptations may be needed when scaling out programs, and adoption rates might be assessed within each system when implementing a new program. Therefore, the perspective of these authors is that future work could include pragmatic evaluations to capture adoption rates through unfunded, practitionerdeveloped studies, rather than only those that are funded and initiated by researchers.
ACKNOWLEDGMENTS
4.
5.
6.
7.
8.
9.
10.
The authors would like to thank Hannah Kesterson for her assistance.
REFERENCES 1. US Department of Agriculture. Extension. https://nifa.usda.gov/extension. Accessed April 17, 2019. 2. Battelle. 2015 Battelle Study: analysis of the value of Family & Consumer Sciences Extension in the North Central Region. Columbus, OH: Battelle; 2015. http://www.nccea.org/multistate-activities/fcs-battelle-report2015/. Accessed July 5, 2018. 3. Rasmussen W. Taking the University to the People: Seventy-Five Years of Cooperative
11.
12.
13.
14.
Extension. West Lafayette, IN: Purdue University Press; 2002. Braun B, Bruns K, Cronk L, et al. Cooperative Extension’s National Framework for Health and Wellness. http://www. aplu.org/members/commissions/foodenvironment-and-renewable-resources/ CFERR_Library/national-frameworkfor-health-and-wellness/file. Accessed May 6, 2019. Aarons G, Sklar M, Bustanski B, Benbow N, Hendricks Brown C. “Scalingout” evidence-based interventions to new populations or new health care delivery systems. Implementation Sci. 2017;12:111. Balis L, Strayer T, Ramalingam N, Harden S. Beginning with the end in mind: contextual considerations for scaling-out a community-based intervention. Front Public Health. 2018; 6:357. Harden SM, Ramalingam N, Breig S, Estabrooks P. Walk this way: our perspectives on challenges and opportunities for Extension statewide walking promotion programs. J Nutr Educ Behav. 2019;51:636–643. Balis L, Strayer T, Ramalingam N, Wilson M, Harden S. Open-access physical activity programs for older adults: a pragmatic and systematic review. Gerontologist. 2019;59:e268– e278. Davis G, Baral R, Strayer T, Serrano E. Using pre- and post-survey instruments in interventions: determining the random response benchmark and its implications for measuring effectiveness. Public Health Nutr. 2018;21: 1043–1047. Glasgow R, Vogt T, Boles S. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;9: 1322–1327. Balis L, John D, Harden S. Beyond evaluation: using the RE-AIM framework for program planning in Extension. J Ext. 2019;57:2TOT1. Downey L, Peterson D, Donaldson J, Hardman A. An Extension application of the RE-AIM evaluation framework. J Ext. 2017;55:3IAW1. National 4-H Council. What is 4-H? https://4-h.org/about/what-is-4-h/. Accessed April 17, 2019. Holmgren L, Reid C. 4-H and FFA livestock projects: life skills gained and
Balis and Harden
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
1023
knowledge learned. J Youth Dev. 2007;2:1–9. Lewis SR, Murphy TH, Baker M. The impact of the 4-H program on Nevada public school youth. J Ext. 2009;47: 3RIB3. Garton MS, Miltenberger M, Pruett B. Does 4-H camp influence life skill and leadership development? J Ext. 2007; 45:4FEA4. Frank KL, Donaldson JL, Toman J, Moody T. Moving beyond the program: incorporating healthy living behaviors throughout 4-H. J Ext. 2014; 52:4COM2. Folta S, Koomas A, Metayer N, et al. Engaging stakeholders from volunteerled out-of-school time programs in the dissemination of guiding principles for healthy snacking and physical activity. Prev Chronic Dis. 2015;12:E225. Economos C, Anzman-Frasca S, Koomas A, et al. Dissemination of healthy kids out of school principles for obesity prevention: a RE-AIM analysis. Prev Med. 2019;119:37–43. Brownson R, Jones E. Bridging the gap: translating research into policy and practice. Prev Med. 2009;49:313– 315. Franz NK, Townson L. The nature of complex organizations: the case of Cooperative Extension. New Dir Eval. 2008;12:5–14. Strayer T III, Balis LE, Harden SM. Partnering for successful dissemination: how to improve public health with the National Cooperative Extension System [published online ahead of print June 4, 2019]. J Public Health Manag Pract. https://doi.org/10.1097/ PHH.0000000000001025. Proctor E, Powell B, McMillen J. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139. Powell BJ, McMillen JC, Proctor EK, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2013;69:123–157. Harden S, Johnson SB, Almeida F, Estabrooks P. Improving physical activity program adoption using integrated research−practice partnerships: an effectiveness-implementation trial. Transl Behav Med. 2017;7: 28–38. Wilson M, Strayer T, Davis R, Harden SM. Informed adaptations of a
1024
Balis and Harden
strength-training program through a research−practice partnership. Front Public Health. 2018;6:58. 27. Johnson SB, Harden S, Estabrooks P. Uptake of evidence-based physical activity programs: comparing perceptions of adopters and nonadopters.
Journal of Nutrition Education and Behavior Volume 51, Number 8, 2019 Transl Behav Med. 2016;6: 629–637. 28. Wilson ML, Strayer TE, Davis R, Harden S. Use of an integrated researchpractice partnership to improve outcomes of a community-based strength-training intervention for older adults: reach and
effect of Lifelong Improvements through Fitness Together (LIFT). Int J Environ Res Public Health. 2018;15:E237. 29. Glasgow R. What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Educ Behav. 2013;40:257–265.
Great Educational Materials (GEMs) are brief descriptions of innovative and useful approaches to nutrition education and behavior. GEMs include a description of the teaching technique, activity, or material; implementation procedures; and evidence of usefulness or impact. Submit your next GEM to JNEB. Questions? Review detailed author instructions online (www.jneb.org) or send email to [email protected].
INTRODUCTION The national Cooperative Extension System1 connects research to practice through land-grant universities in each state and territory. With its broad reach2 and strong history of improving agricultural practices,3 Extension is positioned to tackle the nation’s biggest challenges. Following the release of the National Framework for Health and Wellness in 2014,4 chronic disease prevention became an Extension priority area. Extension delivers a number of nutrition and physical activity interventions, however, scaling out (ie, implementing an intervention in a new setting and to a new population)5 from 1 state system to another remains challenging.6,7 Part of this challenge is that Extension struggles with program evaluation. Extension programs are often evaluated with a posttest only (generally directed toward program satisfaction) or no evaluation beyond reporting attendance.8,9 This results in 2 translational issues: (1) programs are developed in 1 state and adopted in
1
another before full evaluation efforts are completed or reported, and (2) the degree to which programs are feasible for delivery outside their original state is unknown. To scale out Extension programs better, comprehensive program evaluation is important. Recent publications10−12 recommended planning and evaluating Extension programs through the reach, effectiveness, adoption, implementation, maintenance (RE-AIM) framework, which goes beyond measuring who participates in a program (reach) and the primary outcomes (effectiveness) to include systems-level dimensions (adoption, implementation, and maintenance), to assess whether an intervention works in the real world.10 Extension has traditionally focused heavily on reach (specifically, the number of individuals participating in a program), because educators are mandated to track the number of contacts, but the other RE-AIM dimensions were not emphasized. Without strong adoption rates
Extension, University of Wyoming, Lander, WY Department of Human Nutrition, Foods, and Exercise, Virginia Tech, Blacksburg, VA Conflict of Interest Disclosure: The authors have not stated any conflicts of interest. Address for correspondence: Laura Balis, PhD, Extension, University of Wyoming, 130 Eugene St, Lander, WY 82520; E-mail: [email protected] Ó 2019 Society for Nutrition Education and Behavior. Published by Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.jneb.2019.05.288
2
1020
(ie, the number, proportion, and representativeness of delivery agents willing to initiate a program),10 the reach and effectiveness of programs cannot be fully achieved. One Extension target audience is youths. The 4-H program, the nation’s largest youth development program, is delivered through Extension and reaches nearly 6 million youths/yr.13 The 4-H program stands for Head, Heart, Hands, and Health.13 This widely recognized 4-H branch of Extension offers a number of youth programming types: in-school and after-school programs, camps, and community clubs.13 Whereas 4-H programs are often strong in the head, heart, and hands (through engaging in project learning, leadership, and service),14−16 many state 4-H systems struggle to address contemporary health challenges such as poor dietary quality and sedentary time.17 4-H is of particular interest because program offerings are educational in nature and youths are inspired to learn by doing.13 In this way, health behaviors that are observed during 4-H programming may extend into other facets of the youths’ lives. However, before 2015, there were no structured programs in place to help 4-H volunteers and staff integrate the fourth H of health in meeting practices.18 One intervention developed to assist 4-H programs with promoting healthy habits is the
Journal of Nutrition Education and Behavior Volume 51, Number 8, 2019
Journal of Nutrition Education and Behavior Volume 51, Number 8, 2019 4th H for Health Challenge (here called the Challenge), an evidenceinformed intervention developed through a participatory approach between researchers and 4-H volunteer leaders and administrators in Maine, New Hampshire, and Massachusetts.18 The Challenge is designed to help club leaders implement healthy meeting practices: serving only water as the beverage, serving a fruit or vegetable as the snack, and including at least 15 minutes of moderate to vigorous physical activity.18 The Challenge was implemented in 16 states (according to a search of each state’s Extension Web site conducted for this perspective). Although a 30% adoption rate within the nation seems promising, it is currently unknown how many countylevel educators and volunteers implement the Challenge when it is adopted by state Extension systems. The purposes of this perspective are to provide suggestions for improved scale-out using findings regarding differences in Challenge adoption rates among state Extension systems, and to discuss challenges and results of pragmatic data collection.
DISCUSSION Setting To elucidate lessons in the process of adopting an evidence-based program in a new state, Wyoming was used. In Wyoming, 26 4-H educators managed county 4-H programs and supervised approximately 1,677 adult volunteers. The Challenge was introduced to Wyoming by an Extension health educator in an effort to bridge health promotion efforts across Extension program areas. Notably, no funding or administrative support was provided to adopt the Challenge; that is, the educator had the opportunity to champion the program, but it was not mandated or part of a research trial.
Assessing Need As part of a pragmatic research study, all 4-H educators and volunteers were invited to complete an online survey to collect baseline demographic information and determine whether there was a need to implement the
Challenge. The purposes of the survey were to ascertain demographic information about the educators and volunteers and understand their current meeting behaviors (snack options and physical activity opportunities). About 30% of educators and 10% of volunteers responded to the survey. Their responses indicated that many Extension practitioners were offering water during meetings, but few served fruit and vegetable snacks most or every time, and 35% responded that physical activity was included most or every time. Similar to other research, it is likely that those most interested in healthy meetings responded to the survey, yet the need remained to implement healthy meeting initiatives.
Promoting the Challenge To see how the Challenge would look in Wyoming, 2 educators agreed to serve as program champions and pilot-test the Challenge for 3 months. One of these educators then provided testimony about the program when the next 4-H program year started. The Challenge was promoted to 4-H educators through several methods. A brief (approximately 15-minute) presentation was given to all 4-H educators at the annual statewide team meeting. E-mails with details about the Challenge were sent to all 4-H educators, and information was posted to the state 4-H Web site for educators to access. Educators then recruited volunteer club leaders through 4-H newsletters and achievement night presentations.
Determining Adoption Rates After educators recruited volunteers to participate in the Challenge, a survey was administered to determine how many educators and volunteers adopted the Challenge and whether those who adopted were representative of the overall population. This survey also contained standard demographic items and organization information. Unfortunately, no 4-H educators completed the pre-program survey and only 17 4-H volunteers (eg, 22% of baseline respondents and 1% of all volunteers in Wyoming) completed the
Balis and Harden
1021
survey indicating they planned to adopt the Challenge.
Challenges in Scaling-Out The researchers applied an evidencebased approach to translation: needs assessment, pilot-testing, and establishment of a program champion, and then full-scale integration. Yet, in Wyoming, at a basic level, the Challenge failed. Limited responses (30% of educators and 10% of volunteers) were received and few eligible 4-H practitioners (no educators and 1% of volunteers) adopted the Challenge. This is surprising because there were high county administrator adoption rates (82%) in a comprehensive evaluation of the Challenge in 3 states (Maine, New Hampshire, and Massachusetts).19 Notably, volunteer and educator adoption rates were not calculated in these states.19 The Challenge was a promising intervention for translation, but enthusiasm and success were not seen in Wyoming. The researchers offer a perspective on why this happened, as well as some potential solutions.
IMPLICATIONS FOR RESEARCH AND PRACTICE As Extension continues its efforts to deliver evidence-based programs and show broad impacts, an understanding of how to scale out programs from 1 state to another will be helpful. Without a strategic effort to assess and improve adoption rates, new initiatives such as the Challenge fail to gain traction and become adopted within their intended delivery systems.5,20 This can lead to implementation and de-implementation of new initiatives that, if they were successfully implemented, might have led to changes in the desired behaviors and public health benefits. The difficulties in scaling out the Challenge can provide useful lessons for both practitioners and researchers. These can be categorized according to the need for state-specific adaptations and the issues faced when conducting research without funding or staff resources.
1022
Journal of Nutrition Education and Behavior Volume 51, Number 8, 2019
Balis and Harden
Need for State-Specific Adaptations Extension is a complex system with organizational structures that vary by state.3,21 For example, some state Extension systems have administrators located on campus as well as in district and county offices, whereas some have administrators located only on campus.21 In addition, in some states, educators (also called agents) have the autonomy to develop or select programs to deliver, whereas in others they must select programs from a preapproved list.22 In the New England states, where the Challenge was developed, it is possible that county-level staff, who had administrative roles, experienced fewer barriers to implementing the Challenge than did 4-H educators in Wyoming. Different strategies may be needed in each state to meet the needs of different organizational structures. In Wyoming, although 4H educators expressed interest in the Challenge when it was presented at their team meeting and agreed to pilot-test the Challenge, no educators completed the pre-program survey indicating that they planned to adopt the Challenge. Although 4-H educators held meetings for their programs (eg, junior leaders), 4-H volunteer club leaders likely held more meetings, and therefore had more opportunities to provide beverages and snacks and lead physical activities. In the future, efforts to improve Challenge adoption rates may need to target both educators (by encouraging them to promote
the Challenge to their volunteers) and volunteers. Future research could test implementation strategies (ie, methods to improve program adoption and implementation, such as training or incentives)23 at various levels to increase adoption rates. The table (adapted from Powell et al24) details examples of implementation strategies to be used during the planning phase of an intervention to increase adoption. In addition to targeting implementation strategies to levels appropriate for each state, state-specific adaptations may be needed to scale out the Challenge.5 For example, in the New England states, pins were used as an incentive to complete the Challenge. In Wyoming, when the Challenge was initially presented to 4-H educators, they indicated that pins were not widely used and would not serve as an effective incentive. Limited research has been done on scaling out Extension interventions to other states; however, results suggest that place-based adaptations can improve adoption rates.6,25−27 Although the Challenge was developed through a participatory approach, in the future, a state-specific integrated research−practice partnership might be used to adapt Extension programs to additional states before implementation.25,27,28
Need for Pragmatic Research It is unknown whether the barriers to adoption in Wyoming were state-specific (ie, the Challenge should have been adapted in Wyoming before
being introduced) or whether program adoption rates are also low in other states. It is possible that the different adoption rates between Wyoming and the 3 New England states resulted from the difference between a funded, researcher-initiated approach and an educator-initiated pragmatic one. Similar strategies were used in both studies (eg, promoting the Challenge through interactive presentations and offering technical assistance).19 However, in Wyoming, there was not strong support from the state 4-H office, which may have contributed to low adoption rates. Because other states implementing the Challenge also may not be part of funded research studies, future research could assess whether the Challenge was adopted and maintained in these systems. Pragmatic research approaches (those that address specific, practice-based needs and questions; consider contextual factors, resources, and relationships; and use measures that are feasible and lead to action) are recommended for this work.29 As part of this pragmatic approach, perceptions of both those who adopt the Challenge and those who choose not to adopt it might be assessed to understand barriers better.27 In the New England Challenge evaluation, volunteers who successfully implemented the Challenge were interviewed to assess implementation, but perceptions of those who were not deemed successful were not captured.19 Perceptions of nonadopters are important because they can be used to adapt the program for future iterations.27
Table. Examples of Planning Phase Implementation Strategies to Increase Adoption Implementation Strategy Assess for readiness and identify barriers Develop a formal implementation blueprint Conduct local consensus discussions; build a coalition Involve clients and families
Target Level
Example
Administrators, Educators
Survey administrators and educators to assess degree of readiness and potential barriers/facilitators Collaborative to create a plan including staff responsibilities and funding Determine whether the Challenge is appropriate to meet needs; divide responsibilities Include throughout the process and provide training on the Challenge Announce that the Challenge will be implemented; document the commitment in writing
Administrators, Educators Educators, Volunteers Volunteers, Youth, Parents
Mandate change; obtain formal commitments
Administrators 24
Note: Adapted from Powell et al.
Journal of Nutrition Education and Behavior Volume 51, Number 8, 2019 There were some limitations to this work. The sample size for the pre-program survey was small; however, these results are important to understand adoption rates and intervene to improve them. In addition, in pragmatic settings such as Extension, large sample sizes are not always possible. The total number of volunteers (N = 1,677) included both project volunteers (who might not have had an influence over snack and physical activity practices) and club leaders. A more accurate denominator for reach would include only club leaders, but it was unavailable. In the future, better efforts to monitor who adopts the Challenge (eg, through registering on a Web site) are needed to capture data on those who may adopt the Challenge but not complete the survey. To determine whether interventions are successfully implemented in the real world, it is important to capture adoption rates. This perspective highlights that state-specific adaptations may be needed when scaling out programs, and adoption rates might be assessed within each system when implementing a new program. Therefore, the perspective of these authors is that future work could include pragmatic evaluations to capture adoption rates through unfunded, practitionerdeveloped studies, rather than only those that are funded and initiated by researchers.
ACKNOWLEDGMENTS
4.
5.
6.
7.
8.
9.
10.
The authors would like to thank Hannah Kesterson for her assistance.
REFERENCES 1. US Department of Agriculture. Extension. https://nifa.usda.gov/extension. Accessed April 17, 2019. 2. Battelle. 2015 Battelle Study: analysis of the value of Family & Consumer Sciences Extension in the North Central Region. Columbus, OH: Battelle; 2015. http://www.nccea.org/multistate-activities/fcs-battelle-report2015/. Accessed July 5, 2018. 3. Rasmussen W. Taking the University to the People: Seventy-Five Years of Cooperative
11.
12.
13.
14.
Extension. West Lafayette, IN: Purdue University Press; 2002. Braun B, Bruns K, Cronk L, et al. Cooperative Extension’s National Framework for Health and Wellness. http://www. aplu.org/members/commissions/foodenvironment-and-renewable-resources/ CFERR_Library/national-frameworkfor-health-and-wellness/file. Accessed May 6, 2019. Aarons G, Sklar M, Bustanski B, Benbow N, Hendricks Brown C. “Scalingout” evidence-based interventions to new populations or new health care delivery systems. Implementation Sci. 2017;12:111. Balis L, Strayer T, Ramalingam N, Harden S. Beginning with the end in mind: contextual considerations for scaling-out a community-based intervention. Front Public Health. 2018; 6:357. Harden SM, Ramalingam N, Breig S, Estabrooks P. Walk this way: our perspectives on challenges and opportunities for Extension statewide walking promotion programs. J Nutr Educ Behav. 2019;51:636–643. Balis L, Strayer T, Ramalingam N, Wilson M, Harden S. Open-access physical activity programs for older adults: a pragmatic and systematic review. Gerontologist. 2019;59:e268– e278. Davis G, Baral R, Strayer T, Serrano E. Using pre- and post-survey instruments in interventions: determining the random response benchmark and its implications for measuring effectiveness. Public Health Nutr. 2018;21: 1043–1047. Glasgow R, Vogt T, Boles S. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;9: 1322–1327. Balis L, John D, Harden S. Beyond evaluation: using the RE-AIM framework for program planning in Extension. J Ext. 2019;57:2TOT1. Downey L, Peterson D, Donaldson J, Hardman A. An Extension application of the RE-AIM evaluation framework. J Ext. 2017;55:3IAW1. National 4-H Council. What is 4-H? https://4-h.org/about/what-is-4-h/. Accessed April 17, 2019. Holmgren L, Reid C. 4-H and FFA livestock projects: life skills gained and
Balis and Harden
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
1023
knowledge learned. J Youth Dev. 2007;2:1–9. Lewis SR, Murphy TH, Baker M. The impact of the 4-H program on Nevada public school youth. J Ext. 2009;47: 3RIB3. Garton MS, Miltenberger M, Pruett B. Does 4-H camp influence life skill and leadership development? J Ext. 2007; 45:4FEA4. Frank KL, Donaldson JL, Toman J, Moody T. Moving beyond the program: incorporating healthy living behaviors throughout 4-H. J Ext. 2014; 52:4COM2. Folta S, Koomas A, Metayer N, et al. Engaging stakeholders from volunteerled out-of-school time programs in the dissemination of guiding principles for healthy snacking and physical activity. Prev Chronic Dis. 2015;12:E225. Economos C, Anzman-Frasca S, Koomas A, et al. Dissemination of healthy kids out of school principles for obesity prevention: a RE-AIM analysis. Prev Med. 2019;119:37–43. Brownson R, Jones E. Bridging the gap: translating research into policy and practice. Prev Med. 2009;49:313– 315. Franz NK, Townson L. The nature of complex organizations: the case of Cooperative Extension. New Dir Eval. 2008;12:5–14. Strayer T III, Balis LE, Harden SM. Partnering for successful dissemination: how to improve public health with the National Cooperative Extension System [published online ahead of print June 4, 2019]. J Public Health Manag Pract. https://doi.org/10.1097/ PHH.0000000000001025. Proctor E, Powell B, McMillen J. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139. Powell BJ, McMillen JC, Proctor EK, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2013;69:123–157. Harden S, Johnson SB, Almeida F, Estabrooks P. Improving physical activity program adoption using integrated research−practice partnerships: an effectiveness-implementation trial. Transl Behav Med. 2017;7: 28–38. Wilson M, Strayer T, Davis R, Harden SM. Informed adaptations of a
1024
Balis and Harden
strength-training program through a research−practice partnership. Front Public Health. 2018;6:58. 27. Johnson SB, Harden S, Estabrooks P. Uptake of evidence-based physical activity programs: comparing perceptions of adopters and nonadopters.
Journal of Nutrition Education and Behavior Volume 51, Number 8, 2019 Transl Behav Med. 2016;6: 629–637. 28. Wilson ML, Strayer TE, Davis R, Harden S. Use of an integrated researchpractice partnership to improve outcomes of a community-based strength-training intervention for older adults: reach and
effect of Lifelong Improvements through Fitness Together (LIFT). Int J Environ Res Public Health. 2018;15:E237. 29. Glasgow R. What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Educ Behav. 2013;40:257–265.
Great Educational Materials (GEMs) are brief descriptions of innovative and useful approaches to nutrition education and behavior. GEMs include a description of the teaching technique, activity, or material; implementation procedures; and evidence of usefulness or impact. Submit your next GEM to JNEB. Questions? Review detailed author instructions online (www.jneb.org) or send email to [email protected].