- Email: [email protected]
What Is Your Nutrition Program Missing? Finding Answers with the Guide for Effective Nutrition Interventions and Education (GENIE)
FROM THE ACADEMY
What Is Your Nutrition Program Missing? Finding Answers with the Guide for Effective Nutrition Interventions and Education (GENIE) Jenica K. Abram, MPH, RDN, LDN; Rosa K. Hand, MS, RDN, LD; J. Scott Parrott, PhD; Katie Brown, EdD, RDN; Paula J. Ziegler, PhD, RDN, CFCS; Alison L. Steiber, PhD, RDN
N
UTRITION EDUCATION PROgrams exist for many settings, formats, and audiences.1-5 Many show limited or shortlived impact on health-related behaviors. 2 Other programs conducted are never reported in scientific literature and fail to report outcomes and process measures.1,5 Although programmatic variety, publication bias, and inconsistent quality can make the comparison of nutrition education programs difficult, it is important to identify commonalities of successful nutrition education programs.6-10 A number of published reviews have identified elements of nutrition education that lead to successful outcomes.6-11 Nutrition education practitioners can design more effective interventions by incorporating these elements into their programs.10 The Guide for Effective Nutrition Interventions and Education (GENIE) is a checklist of evidence-based quality indicators for nutrition education programs developed to help users design, modify, or compare effective nutrition education programs. This simple to use online resource was created jointly by the Academy of Nutrition and Dietetics’ Research, International and Scientific Affairs team, and the Academy Foundation. The work, which took place in 2013 and 2014, was supported by an educational grant from the ConAgra Foods Foundation, which included funding for a full-time research fellow to coordinate the previously described validation and this literature review. GENIE contains nine categories with 35 quality criteria and is available online 2212-2672/Copyright ª 2015 by the Academy of Nutrition and Dietetics. http://dx.doi.org/10.1016/j.jand.2014.08.024 Available online 25 October 2014
122
to guide the development of nutrition education programs, or to evaluate program plans or proposals. GENIE has been validated across a wide range of nutrition education program proposals,12 giving this tool the unique ability to compare disparate programs. This article describes a literature review in which GENIE was used to assess and compare published program descriptions and to: 1) identify qualities of nutrition education programs commonly present or absent in both high-scoring and lower-scoring interventions when assessed with GENIE; 2) provide examples of high-scoring programs in each category; and 3) determine the relationship between GENIE score and program outcomes.
THE REVIEW PROCESS A systematic review process was developed to identify relevant published nutrition education programs for review. Searches were performed within the National Institutes of Health PubMed database between July and September 2013, and were limited to clinical trials, evaluation, and validation study article types with human subjects. A list of terms was created before initiating the search process and the following Medical Subject Headings (MeSH terms) were selected: health education, nutrition therapy, health promotion, nutrition, and schools. Articles were excluded if the full text was unavailable in English; the program goal was to test the effects of a specific diet regimen, medication, or policy as opposed to a nutrition education program; or if no quantitative results were reported. Duplicate programs and those that were funded through proposals included in the reliability and validity testing of GENIE
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
were also excluded. The search process is described in the Figure. Articles were screened at two points during the process—once after reviewing the article abstracts and a second time after reviewing the full text of the articles. In cases where multiple articles were published about a single program, only the most recent article was assessed. When a program’s protocol was published separately from the most recent results, both articles were referenced to assess the program. In total, 117 articles were selected for assessment, representing a total of 102 unique nutrition education programs. All screening was conducted by the first author. Information about each program was organized via a data collection tool designed specifically for this study using an online survey platform (Survey Monkey Inc). Data of interest included:
presence of GENIE’s 35 quality criteria and information about program participants (ie, number, age, and disease state); format and content (ie, instructional techniques, group size, dose, frequency, duration, curricula, and retention); stated goals and outcomes (ie, knowledge-based, behavioral, and physiological); and any notes on bias or methodological concerns.
Outcomes were divided into three levels: 1.
2.
knowledge-based (such as disease state knowledge, macronutrient knowledge, and cooking knowledge); behavior (such as food choices, eating behaviors, and physical activity); and
ª 2015 by the Academy of Nutrition and Dietetics.
FROM THE ACADEMY
Figure. A systematic review process to identify relevant published nutrition education programs for review using the Guide for Effective Nutrition Interventions and Education (GENIE).
January 2015 Volume 115 Number 1
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
123
FROM THE ACADEMY Table 1. Percent of programs achieving GENIEa quality criteria for programs in the top 25% of GENIE scores, bottom 25% of GENIE scores, and total for all 102 programs reviewed
Quality criteria
Total
Top scoring quartile
Bottom scoring quartile
% achieved! Category 1: Program Description and Importance 1.1 Provides evidence of feasibility
94.1
100
96
1.2 Describes why it is well timed and/or novel
89.2
100
84
1.3 Defines the target group and need (think about health inequities)
97.1
100
88
1.4 Justifies that the target group will benefit from the program or intervention, based on related research, best-practice examples, or a needs assessment
96.1
100
92
2.1 Promotes healthy eating behaviors
99
100
96
2.2 Includes nutrition-related goals that address proximal outcomes
30.4
48
20
100
100
100
99
100
96
3.1 Uses research or best-practice examples to show how a model or framework integrates with 46.1 the program goal
68
28
3.2 Includes partnerships with other groups and explains how these partners aid the program
21.6
48
0
3.3 Meets the needs of the target group
99
100
96
3.4 Addresses external influences on food and eating
58.8
80
12
4.1 Describes an appropriate setting for target group
86.3
100
76
4.2 Describes realistic recruitment and retention of participants
97.1
100
88
Category 2: Program Goal
2.3 Includes nutrition-related goals that address intermediate or distal outcomes If weight is the outcome, including appropriate physical activity in goals is encouraged. 2.4 Includes measurable goals Category 3: Program Framework
Category 4: Program Setting, Recruitment, and Retention
Category 5: Program Methods 5.1 Includes several techniques to promote learning
81.4
92
64
5.2 Includes several techniques to motivate participants
61.8
88
20
5.3 Includes several techniques to promote nutrition behavior change
91.2
96
80
5.4 Explains why the planned teaching time and dose are adequate/fitting using related research or best-practice examples as support
11.8
24
4
100
96
100
72
84
24
96
32
100
Category 6: Program Content 6.1 Relates to program goals
99
6.2 Is based on best-practice examples or related research with citations from relevant research 88.2 or government/health society guidance 6.3 Is supported by experts or key informants
52
Category 7: Program Materials 62.7 7.1 The program cites and explains that the materials have social and cultural relevance, including: language, reading level, food likes and dislikes, household status, food/ diet needs, interests, age/development stage matched, learning style, and/or format Category 8: Evaluation 8.1 Includes measurement tools that address program goals
87.3
92
8.2 Includes measurement tools that are reliable, valid, and chosen based on related research or best practice
93.1
100
96.3
(continued on next page) 124
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
January 2015 Volume 115 Number 1
FROM THE ACADEMY Table 1. Percent of programs achieving GENIEa quality criteria for programs in the top 25% of GENIE scores, bottom 25% of GENIE scores, and total for all 102 programs reviewed (continued)
Quality criteria
Total
Top scoring quartile
Bottom scoring quartile
% achieved! 8.3 Evaluation method is cited, based on related research, best practice or includes pilot testing 86.3 to support its use
100
100
8.4 Includes process evaluation to check that the program is implemented as planned
37.3
64
44.4
8.5 Includes outcome evaluation measures for proximal goals
27.5
40
29.6
8.6 Includes outcome and/or impact evaluation measures for intermediate or distal goals
99
8.7 Evaluates outcome/impact at multiple time points
37.3
48
8.8 Includes an appropriate analysis plan
97.1
96
100
100 44.4 100
Category 9: Sustainability 9.1 Addresses the potential for the program to continue
49
9.2 Indicates evidence of prior/current successful programs
78.4
84
24
100
60
9.3 Describes shared roles and duties of program partners
13.7
36
4
9.4 Implies potential for broader reach, replication, and growth
51
84
24
24
4
9.5 Addresses the collective program impact within the community and/or among program partners
7.8
a
GENIE¼Guide for Effective Nutrition Interventions and Education.
3.
physiological (such as weight change, body mass index, lipid concentration, and blood pressure).
These outcome categories were established a priori as part of the GENIE development process.12 The first author extracted the data and completed the GENIE assessment for each of the 102 unique programs assessed. Programs were grouped into quartiles based on overall GENIE score. The maximum GENIE score was 35. The fourth (upper) quartile contained 25 programs, all with a GENIE score of 27 or higher. The first (lowest) quartile also contained 25 programs, all with a GENIE score of 21 or lower. For each quartile and the dataset as a whole, we calculated the prevalence of each GENIE criteria (percent of articles with the criteria present). To examine the relationship between the presence of quality criteria and positive outcomes, the number of outcomes achieved was calculated as a proportion of outcomes measured for each program. Achieved outcomes were outcomes that the program authors considered successful, because either they met a specified goal or reached statistical significance compared to baseline values or a control January 2015 Volume 115 Number 1
group. This proportion was calculated for each reviewed program separately for knowledge-based, behavioral, and physiological outcomes, and cumulatively for all outcomes. The proportions of outcomes achieved were compared to individual GENIE category scores and overall GENIE scores for each program using Spearman’s correlation. The raw number of outcomes measured and the raw number of outcomes achieved for each program was also compared to individual GENIE category scores and overall GENIE scores for each program using Spearman’s correlation. c2 Tests were used to look for associations between these proportions and all GENIE categories collectively. All analyses were conducted in SPSS software, version 20 (2011, SPSS IBM).
and high-scoring programs, and other program qualities were unique to highscoring programs within our sample. Table 1 illustrates the prevalence of GENIE’s quality criteria among programs in the upper quartile of overall scores, programs in the lower quartile of overall scores, and all programs reviewed. To illustrate the relationship between individual categories and overall GENIE score, nine programs were selected to represent an outstanding example of each GENIE category. For consistency, all example programs tracked weight change as an outcome and only those with GENIE scores in the top 25% were selected. These programs are presented in Table 2 and provided as examples in the text.
Categories 1, 2, and 4 PROGRAM COMPARISON USING GENIE Certain GENIE criteria were more likely to be found in high-scoring programs than low-scoring programs. It was expected that high-scoring GENIE programs would achieve more quality criteria than their lower-scoring counterparts. However, some program qualities were common in both low-
Within GENIE categories 1 (Program Description and Importance) and 4 (Program Setting, Recruitment, and Retention), nearly all quality criteria were achieved by all programs. A similar trend was seen within GENIE category 2 (Program Goal). However, criterion 2.2 (“includes nutrition-related goals that address proximal outcomes”) was rarely present in either high- or low-scoring programs. Within the context of GENIE,
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
125
FROM THE ACADEMY proximal outcomes were defined as knowledge-related outcomes. Because solely knowledge-focused nutrition education programs have not been found to be effective in producing behavior change,23 programs destined for publication may fail to set goals related to knowledge. In programs where knowledge-related goals were defined, knowledge acquisition was frequently examined as a mediator to explain any behavioral changes that may have been observed.13,14,21 For example, although weight change was the primary outcome in Turner-McGrievy and colleagues’14 podcasting intervention, knowledge acquisition was also measured. The authors suggest that improved levels of participant engagement within the intervention group and greater weight loss could be partly attributed to increased weight-control knowledge.14
Category 3 Within GENIE category 3 (Program Framework), there were particularly large disparities between high- and low-scoring programs in achievement of criterion 3.2 (“includes partnerships with other groups and explains how these partners aid the program”) and criterion 3.4 (addresses external influence on food and eating”). Successful partnerships require coordination and foresight, as demonstrated by Glasgow and Toobert, who incorporated a variety of community resources in their diabetes self-management intervention.15 Eighty percent of high-scoring programs addressed external influences on food and eating, and only 12% of lower-scoring programs included this characteristic. Addressing this criterion requires a more multifaceted approach, which might include many different intervention techniques, strategies, and resources. Therefore, programs fulfilling only the most commonly achieved criteria may lack the dimensions to achieve this criterion.
Category 5 Within GENIE category 5 (Instructional Methods), there was a 68-percentagepoint difference between high- and low-scoring programs in the likelihood of achieving criterion 5.2 (“includes several techniques to motivate 126
participants”). Criterion 5.3 (“strategies to promote learning and strategies to promote behavior change”) was commonly found in both high- and low-scoring programs. Motivational strategies can be just as important as strategies to promote behavior change, but appear to be more commonly overlooked, perhaps because research has not clearly shown how these strategies should be put into practice.24 However, motivating participants is vital for empowerment and maintaining engagement.25 Kristal and colleagues motivated participants to lower their dietary fat intake through interactive activities, self-assessments, and social support.17 Only 11.8% of all programs met criterion 5.4 (“explains why the planned teaching time and dose are adequate/fitting using related research or best practice examples as support”). Information concerning the doseeresponse relationship with regard to nutrition education appears limited to the guideline of 50 hours needed to produce behavior change found by Connell and colleagues in their review of classroom health education, which was completed nearly 30 years ago.1,26 Therefore, many program developers may not seek out evidence to support the timing of their program. However, for nutrition education to promote positive health outcomes, more research is needed on how much programming is needed to result in the desired behavior change.
Categories 6 and 7 Large differences were also observed between programs in the upper quartile and programs in the lower quartile within GENIE category 6 (Program Content) for criterion 6.3 and for criterion 7.1. In general, most reviewed programs had appropriate content and materials, but less than a quarter of low-scoring programs had support from experts or key informants for their content and less than a third cite that the materials used in the program had social and cultural relevance to participants. These results show that including appropriate materials is far more common in higher-scoring programs, but also may indicate that published programs do not thoroughly describe and support their choice of materials and content. Johnston and colleagues’ intervention to promote
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
weight loss among Mexican-American children proved to be an exception by using culturally correct recipes, bilingual written information, and an age appropriate token economy with prizes to encourage engagement.20
Category 8 Within GENIE category 8 (Evaluation), there were three cases where criteria were found roughly half as often in low-scoring programs than they were in high-scoring programs: criteria 8.4 (“includes process evaluation to check that the program is implemented as planned”), 8.5 (“includes outcome evaluation measures for proximal goals”), and 8.7 (“evaluates outcome/impact at multiple time points”). Evaluation can be expensive, time consuming, and frequently planned only after sufficient programming funds have been secured.8 Given that GENIE criterion 2.2 (“setting proximal goals”) was often missing, it is not surprising that GENIE criterion 8.5 (“measuring proximal outcomes”) is infrequent. However, one must remember that knowledge may be measured, but not prioritized for publication. This was the case in Kristal and colleagues’ intervention for breast cancer survivors where nutrition education was targeted, but participant knowledge was not reported in the published article.17
Category 9 Most criteria within GENIE category 9 (Sustainability) were not observed in the reviewed programs with the following exception, criterion 9.2 (“indicates evidences of prior/current successful programs”). The lowest percent achievement out of all criteria was observed for 9.5 (“addresses the collective impact within the community and/or among program partners”). Although the percent of programs achieving criteria within category 9 tended to be low overall, higherscoring programs appeared to include these criteria far more often. The inclusion of partners and other community organizations in criteria 9.3 and 9.5 were frequently missing programmatic elements in published programs. Criterion 9.4 (“implies potential for broader reach, replication, and growth”) was also far more likely to be found in high-scoring programs than January 2015 Volume 115 Number 1
FROM THE ACADEMY low-scoring programs. For example, Salinardi and colleagues provide an excellent model for sustainability by incorporating long-lasting environmental changes as part of a worksite intervention.22 Planning for sustainability may promote program designs that are better integrated with and supported by community partners, easily adapted to different settings, and financially stable. These features that promote sustainability may not only increase the likelihood of a program continuing, but also create a stronger program in other ways.1,7,18,27
GENIE CATEGORY EXAMPLES Table 2 was developed by selecting programs within the upper quartile for overall GENIE score that tracked weight change as an outcome. Nine programs were chosen to illustrate examples of how criteria within each of GENIE’s categories were fulfilled.
Relationship between GENIE Score and Program Outcome When the proportion of knowledgebased outcomes achieved was compared to GENIE category scores, positive significant associations were observed between outcomes and scores on category 2 (Program Goal) and 8 (Evaluation). A negative correlation was observed between the proportion of knowledge-based outcomes achieved and category 3 (Program Framework) score. No meaningful relationships were observed between the proportion of behavioral or physiological outcomes and GENIE category scores. No meaningful relationships were observed between either the raw number of outcomes measured or the raw number of outcomes achieved and GENIE scores for individual categories or total GENIE scores. Overall, the nutrition education programs included in this review tended to achieve high GENIE scores. There was a relatively small difference separating programs scoring in the 25th percentile and those in the 75th percentile. Given the maximum overall GENIE score of 35, we might expect the 25th and 75th percentile to fall at roughly 9 and 26, respectively, given an even distribution across all possible GENIE scores. The higher scores observed in this study (25th percentile¼21; 75th percentile¼27) are indicative of a January 2015 Volume 115 Number 1
negatively skewed score set, meaning that GENIE scores tended to be more common in the upper range. A likely explanation for this score distribution is that nutrition education programs that result in peer-reviewed publications tend to be of a higher quality than program proposals in general, due to multiple types of publication bias. Although, rejection of a manuscript is not considered a common cause of publication bias,28 many nutrition education programs intended for publication do not move through the publication process due to poor programming, poor results, or poor reporting.28,29 These programs are destined to remain in what is commonly referred to as the “file drawer” and may never be formulated into a manuscript, submitted, considered, or accepted by a journal.29 Some programs intended for publication are never accepted by a journal because the program failed to achieve statistically significant results.28 Programs conducted by those who do not consider themselves part of the scientific community may never be considered for publication, regardless of results.5 In general, programs that are published in peer-reviewed literature are more likely to have statistically significant results and be viewed as more important by the investigators, particularly for those programs of an observational nature.28 Another possible explanation for the relatively high GENIE scores is that GENIE was designed to assess nutrition education program plans and proposals rather than published articles. Program plans and proposals have inherent differences. For example, published programs may be more likely to contain the criteria within category 1 because this information is commonly required for the introduction section of many journals. In fact, our review found that nearly all programs contained these criteria. Our review also found that nearly all programs included the criteria within category 4. These findings support the notion that nutrition education without an appropriate setting, recruitment, and retention plan may not be able to secure and sustain enough participants to support a program deemed viable for publication. Issues of publication bias and inherent differences between published nutrition education manuscripts
and program proposals may also explain why GENIE category or overall scores failed to be a reliable predictor of achieved health outcomes. GENIE was designed to help nutrition education program developers assess program plans or proposals,12 and the use of GENIE in this context represents retrofitting the tool to examine published programs. A retrospective analysis of published programs was undertaken for reasons of feasibility, cost, and ease. Published articles are in the public domain and, therefore, far easier to obtain than program proposals. Program proposals are rarely made public and would require extensive coordination with proposal authors, organizations, and/or funders. In addition, programs that do not require external funding may not have been developed from a formal written plan or proposal. However, future work with GENIE should seek to evaluate program proposals and obtain outcomes in order to establish another measure of the tool’s criterion validity.
CONCLUSIONS When assessed using GENIE, highscoring nutrition programs were found to possess quality characteristics different than those of low-scoring programs. Prevalent program characteristics common in all included programs were thorough program descriptions, appropriate goals with matching content that addressed target audience needs, and realistic recruitment strategies and statistical analysis. Program characteristics most likely to be found only in higher-scoring programs were research-based program content and rigorous evaluation methodology, techniques to motivate participants, consideration of external influences on participant behavior, and partnerships with other organizations to promote sustainability. Some exceptions were seen that can be explained by the intrinsic differences between published nutrition education program articles and program proposals. Nutrition education developers can set their programs apart by including quality elements commonly found missing in published programs. All assessed programs tended to score higher than expected within a normal distribution when assessed using GENIE.
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
127
FROM THE ACADEMY Table 2. Exemplar programs highlighting each of the Guide for Effective Nutrition Interventions and Education’s nine categories Author(s) of study, year Participants and design
GENIEa total score and category example
Stahre et al, 200713
Category 1: Program Description and Importance n¼1 GENIE score: 31 Evaluation retention: 59% Justify the need for program among those who Adult women who work as or for child-care providers, body mass index >30; group sessions work with children and why a cognitive with lectures, homework, and discussion; control behavioral approach is appropriate; evidence that similar programs have been successful; new group received program similar to standard effort to look at longer-term weight loss weight-control programs used on occupation health setting; researcher-designed curricula
Turner-McGrievy et al, 200914
Category 2: Program Goal n¼41 GENIE score: 29 Evaluation retention: 0.85 Trial emerging technology of podcasting to Adult individual intervention; Social Cognitive promote weight loss using behavioral theory; Theory podcast; control group received popular includes knowledge goals and physiologic goals podcast emphasizing stimulus control and (weight and body mass index); clear comparison positive body image; researcher designed of podcast programs podcasts
Glasgow and Toobert, 200015
Category 3: Program Framework n¼68 GENIE Score: 28 Evaluation retention: 85% Social-cognitive environmental intervention; multiAdult individual and group intervention, first group visit 60 to 120 min; tailored feedback via level model; modified RE-AIMb framework for computer program, follow-up phone calls, evaluation; integration of community resources; community resources; partial research design meets needs of participants; addresses curriculaeresearcher-designed computer environmental influences on eating and health program; American Heart Association recipes
Gold et al, 200716
Category 4: Program Setting, Recruitment, and n¼40 Retention Plan Program retention: 65% GENIE score: 28 Adults, individual and group; online behavioral weight loss with assignments, “meetings,” and Intervention offered online; cited as common and convenient mode of obtaining health individual feedback; VTrim (researcher-designed information; participants recruited locally, curricula) compared with commercial program technology check performed, encouragement ediets.com provided, attrition on par with control and similar programs
Kristal et al, 199717
n¼57 Program retention: 77% Adult individual and group intervention 6 weekly individual sessions; 10 monthly group session; additional individual session at 6 mo; based on Women’s Health Trial
Booth et al, 200818
Category 6: Program Content n¼53 GENIE score: 27 Program retention: 73% Step guidelines based on National Heart Adults, individual intervention; Foundation of Australia guidelines, dietary website for personal physical activity and diet tracking; personalized feedback; message forum advice based on best practice for weight loss (decrease saturated fat, increase fruits and for feedback from researchers/fellow vegetables); contact matches program goals participants; evaluation completed in person with researchers; researcher-designed curricula (continued on next page)
128
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
Category 5: Instructional Methods GENIE score: 31 Includes educational and behavior change skills development objectives; provides social support to promote new dietary habits; feedback, reinforcement, and problem solving; emphasis on interactive activities and self-assessment
January 2015 Volume 115 Number 1
FROM THE ACADEMY Table 2. Exemplar programs highlighting each of the Guide for Effective Nutrition Interventions and Education’s nine categories (continued) Author(s) of study, year Participants and design
GENIEa total score and category example
Category 7: Program Materials Johnston et al, 201019 n¼37 GENIE score: 32 Johnston et al, 200720 Evaluation retention: 93% Materials age appropriate for audience; childChildren (primary) caregivers (secondary); appropriate approach to healthy eating and individual and group; physical activity; caregiver materials culturally Mexican-American children at charter school; not appropriate; all communications in English and differentiated by weight status, but only Spanish; child-appropriate token economy used overweight/obese children included in the to encourage behaviors analysis; one nutrition education and four physical activity classes/week in group; biweekly quizzes; rewards for meeting goals; 1-on-1 education for not meeting goals; monthly education with parents/caregivers, including extended family; mostly researcher designed curricula based on Go, Slow, Whoa concept Englert et al, 200721
Category 8: Evaluation n¼1,517 GENIE score: 33 Program retention: 97% Adult group intervention; CHIP (Coronary Health Assessments completed at multiple time points Improvement Program); intensive series of group (including post-intervention); process evaluation used to explore mediating variables; appropriate, classes with readings, lectures, question and validated measurement tools answer sessions, and hands-on workshops
Salinardi et al, 201322
Category 9: Sustainability n¼40 GENIE score: 30 Program retention: 89% Program emphasizes environmental changes; Adult; group worksite intervention, 4 sites; group partnership with worksites; opportunities for all program with weekly and biweekly sessions; maintenance sessions for 6 mo afterward monthly; employees to participate; promotes healthy monthly newsletters and seminars for workers not worksite and supportive weight-loss participating in the intervention; partial researcher environment; worksites requested program continue designed curricula based on The “I” Diet (book) modified to emphasize higher fiber
a
GENIE¼Guide for Effective Nutrition Interventions and Education. RE-AIM¼Reach Effectiveness Adoption Implementation Maintenance.
b
This can be attributed to multiple types of publication bias, which greatly limit the number of nutrition education programs that are formally presented to the public through peer-reviewed publication. These issues were also likely to contribute to the finding that GENIE could not be proven as a reliable predictor of positive outcomes when used to assess published articles. Although this study design was used for reasons of practicality, GENIE was not intended to assess published programs. Rather, it was created and validated for assessing program proposals. A prospective analysis whereby GENIE is used to score program proposals and these scores are later compared to program outcomes would be a more accurate test of the January 2015 Volume 115 Number 1
potential of GENIE’s scores to predict positive health outcomes.
References
obesity: Systematic review with metaanalysis. Pediatrics. 2012;130(6):e1647e1671. 5.
Sims LS. Chapter 1 Introduction. J Nutr Educ. 1995;27(6):284-286.
1.
Blue CL, Black DR. Synthesis of intervention research to modify physical activity and dietary behaviors. Res Theory Nurs Pract An Int J. 2005;19(1):25-61.
6.
Gillespie AH, Brun JK. Trends and challenges for nutrition education research. J Nutr Educ. 1992;24(5):222-226.
2.
Desroches S, Lapointe A, Ratté S, Gravel K, Légaré F, Turcotte S. Interventions to enhance adherence to dietary advice for preventing and managing chronic diseases in adults. Cochrane database Syst Rev. 2013;2(2):CD008722.
7.
Contento I. Review of nutrition education research in the Journal of Nutrition Education and Behavior, 1998 to 2007. J Nutr Educ Behav. 2008;40(6): 331-340.
8.
3.
Kwok TCY, Lam LCW, Sea MMM, Goggins W, Woo J. A randomized controlled trial of dietetic interventions to prevent cognitive decline in old age hostel residents. Eur J Clin Nutr. 2012;66(10): 1135-1140.
Lytle L, Achterberg C. Changing the diet of America’s children: What works and why? J Nutr Educ. 1995;27(5):250-260.
9.
Spahn JM, Reeves RS, Keim KS, et al. State of the evidence regarding behavior change theories and strategies in nutrition counseling to facilitate health and food behavior change. J Am Diet Assoc. 2010;110(6):879-891.
4.
Ho M, Garnett SP, Baur L, et al. Effectiveness of lifestyle interventions in child
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
129
FROM THE ACADEMY 10.
11.
12.
13.
14.
15.
Nation M, Crusto C, Wandersman A, et al. What works in prevention: Principles of effective prevention programs. Am Psychol. 2003;58(6-7):449-456. Greidanus IDA, Contento S. The relationship between the ability to solve practical nutrition problems in an adult population and piagetian cognitive levels. J Nutr Educ. 1989;21(2):73-81.
16.
17.
Abram J, Colby S. GENIE: Your nutrition education wishes have been granted. Presented at: 2014 Food & Nutrition Conference & Expo; October 18-21, 2014; Atlanta, GA. Stahre L, Tärnell B, Håkanson C-E, Hällström T. A randomized controlled trial of two weight-reducing short-term group treatment programs for obesity with an 18-month follow-up. Int J Behav Med. 2007;14(1):48-55. Turner-McGrievy GM, Campbell MK, Tate DF, Truesdale KP, Bowling JM, Crosby L. Pounds Off Digitally study: A randomized podcasting weight-loss intervention. Am J Prev Med. 2009;37(4): 263-269. Glasgow RE, Toobert DJ. Brief, computerassisted diabetes dietary self-management counseling: Effects on behavior, physiologic outcomes, and quality of life. Med Care. 2000;38(11):1062-1073.
Gold BC, Burke S, Pintauro S, Buzzell P, Harvey-Berino J. Weight loss on the web: A pilot study comparing a structured behavioral intervention to a commercial program. Obesity (Silver Spring). 2007; 15(1):155-164. Kristal AR, Shattuck AL, Bowen DJ, Sponzo RW, Nixon DW. Feasibility of using volunteer research staff to deliver and evaluate a low-fat dietary intervention: The American Cancer Society Breast Cancer Dietary Intervention Project. Cancer Epidemiol Biomarkers Prev. 1997;6(6): 459-467.
The Rockford CHIP. Prev Med. 2007;44(6): 513-519. 22.
Salinardi TC, Batra P, Roberts SB, et al. Lifestyle intervention reduces body weight and improves cardiometabolic risk factors in worksites. Am J Clin Nutr. 2013;97(4):667-676.
23.
Contento IR. Nutrition education: Linking research, theory, and practice. Asia Pac J Clin Nutr. 2008;17(suppl 1):176-179.
24.
Abusabha R, Peacock J, Achterberg C. How to make nutrition education more meaningful through facilitated group discussions. J Am Diet Assoc. 1999;99(1):72-76.
25.
Rusness BA. Striving for empowerment through nutrition education. J Am Diet Assoc. 1993;93(1):78-79.
18.
Booth AO, Nowson CA, Matters H. Evaluation of an interactive, internet-based weight loss program: A pilot study. Health Educ Res. 2008;23(3):371-381.
19.
Johnston CA, Tyler C, Fullerton G, et al. Effects of a school-based weight maintenance program for Mexican-American children: Results at 2 years. Obesity (Silver Spring). 2010;18(3):542-547.
26.
Connell DB, Turner RR, Mason EF. Summary of findings of the School Health Education Evaluation: Health promotion effectiveness, implementation, and costs. J Sch Health. 1985;55(8):316-321.
20.
Johnston CA, Tyler C, McFarlin BK, et al. Weight loss in overweight Mexican American children: A randomized, controlled trial. Pediatrics. 2007;120(6): e1450-e1457.
27.
Sims LS. Chapter 9: Conclusions. J Nutr Educ. 1995;27(6):355-364.
28.
Easterbrook P, Gopalan R, Berlin J, Matthews D. Publication bias in clinical research. Lancet. 1991;337(8746):867-872.
29.
Scargle JD. Publication bias (The “FileDrawer Problem”) in scientific inference. J Sci Explor. 1999;14(1):91-106.
21.
Englert HS, Diehl HA, Greenlaw RL, Willich SN, Aldana S. The effect of a community-based coronary risk reduction:
AUTHOR INFORMATION J. K. Abram is project coordinator, Dietetics Practice Based Research Network, Academy of Nutrition and Dietetics, Chicago, IL; at the time of the study, she was a ConAgra Foods Foundation Nutrition Education Research Fellow, Academy of Nutrition and Dietetics, Chicago, IL. R. K. Hand is a senior manager, Dietetics Practice Based Research Network, K. Brown is national education director, P. J. Ziegler is senior director, research and evidence analysis, and A. L. Steiber is chief science officer, Academy of Nutrition and Dietetics, Chicago, IL. J. S. Parrott is an associate professor, Rutgers University, Newark, NJ. Address correspondence to: Rosa K. Hand, MS, RDN, LD, Dietetics Practice Based Research Network, Academy of Nutrition and Dietetics, 120 S Riverside Plaza, Suite 2000, Chicago, IL 60606-6995. E-mail: [email protected]
STATEMENT OF POTENTIAL CONFLICT OF INTEREST No potential conflict of interest was reported by the authors.
FUNDING/SUPPORT Funding was provided by the ConAgra Foods Foundation through an educational grant to the Academy of Nutrition and Dietetics Foundation, which included salary support for the investigators. The ConAgra Foods Foundation was not involved in the collection, analysis or interpretation of data.
ACKNOWLEDGEMENTS The authors would like to acknowledge the contribution of Brian Oliver, MLIS Medical Librarian at the Academy of Nutrition and Dietetics, for his guidance in organizing our literature search and obtaining manuscripts for review.
130
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
January 2015 Volume 115 Number 1
What Is Your Nutrition Program Missing? Finding Answers with the Guide for Effective Nutrition Interventions and Education (GENIE) Jenica K. Abram, MPH, RDN, LDN; Rosa K. Hand, MS, RDN, LD; J. Scott Parrott, PhD; Katie Brown, EdD, RDN; Paula J. Ziegler, PhD, RDN, CFCS; Alison L. Steiber, PhD, RDN
N
UTRITION EDUCATION PROgrams exist for many settings, formats, and audiences.1-5 Many show limited or shortlived impact on health-related behaviors. 2 Other programs conducted are never reported in scientific literature and fail to report outcomes and process measures.1,5 Although programmatic variety, publication bias, and inconsistent quality can make the comparison of nutrition education programs difficult, it is important to identify commonalities of successful nutrition education programs.6-10 A number of published reviews have identified elements of nutrition education that lead to successful outcomes.6-11 Nutrition education practitioners can design more effective interventions by incorporating these elements into their programs.10 The Guide for Effective Nutrition Interventions and Education (GENIE) is a checklist of evidence-based quality indicators for nutrition education programs developed to help users design, modify, or compare effective nutrition education programs. This simple to use online resource was created jointly by the Academy of Nutrition and Dietetics’ Research, International and Scientific Affairs team, and the Academy Foundation. The work, which took place in 2013 and 2014, was supported by an educational grant from the ConAgra Foods Foundation, which included funding for a full-time research fellow to coordinate the previously described validation and this literature review. GENIE contains nine categories with 35 quality criteria and is available online 2212-2672/Copyright ª 2015 by the Academy of Nutrition and Dietetics. http://dx.doi.org/10.1016/j.jand.2014.08.024 Available online 25 October 2014
122
to guide the development of nutrition education programs, or to evaluate program plans or proposals. GENIE has been validated across a wide range of nutrition education program proposals,12 giving this tool the unique ability to compare disparate programs. This article describes a literature review in which GENIE was used to assess and compare published program descriptions and to: 1) identify qualities of nutrition education programs commonly present or absent in both high-scoring and lower-scoring interventions when assessed with GENIE; 2) provide examples of high-scoring programs in each category; and 3) determine the relationship between GENIE score and program outcomes.
THE REVIEW PROCESS A systematic review process was developed to identify relevant published nutrition education programs for review. Searches were performed within the National Institutes of Health PubMed database between July and September 2013, and were limited to clinical trials, evaluation, and validation study article types with human subjects. A list of terms was created before initiating the search process and the following Medical Subject Headings (MeSH terms) were selected: health education, nutrition therapy, health promotion, nutrition, and schools. Articles were excluded if the full text was unavailable in English; the program goal was to test the effects of a specific diet regimen, medication, or policy as opposed to a nutrition education program; or if no quantitative results were reported. Duplicate programs and those that were funded through proposals included in the reliability and validity testing of GENIE
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
were also excluded. The search process is described in the Figure. Articles were screened at two points during the process—once after reviewing the article abstracts and a second time after reviewing the full text of the articles. In cases where multiple articles were published about a single program, only the most recent article was assessed. When a program’s protocol was published separately from the most recent results, both articles were referenced to assess the program. In total, 117 articles were selected for assessment, representing a total of 102 unique nutrition education programs. All screening was conducted by the first author. Information about each program was organized via a data collection tool designed specifically for this study using an online survey platform (Survey Monkey Inc). Data of interest included:
presence of GENIE’s 35 quality criteria and information about program participants (ie, number, age, and disease state); format and content (ie, instructional techniques, group size, dose, frequency, duration, curricula, and retention); stated goals and outcomes (ie, knowledge-based, behavioral, and physiological); and any notes on bias or methodological concerns.
Outcomes were divided into three levels: 1.
2.
knowledge-based (such as disease state knowledge, macronutrient knowledge, and cooking knowledge); behavior (such as food choices, eating behaviors, and physical activity); and
ª 2015 by the Academy of Nutrition and Dietetics.
FROM THE ACADEMY
Figure. A systematic review process to identify relevant published nutrition education programs for review using the Guide for Effective Nutrition Interventions and Education (GENIE).
January 2015 Volume 115 Number 1
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
123
FROM THE ACADEMY Table 1. Percent of programs achieving GENIEa quality criteria for programs in the top 25% of GENIE scores, bottom 25% of GENIE scores, and total for all 102 programs reviewed
Quality criteria
Total
Top scoring quartile
Bottom scoring quartile
% achieved! Category 1: Program Description and Importance 1.1 Provides evidence of feasibility
94.1
100
96
1.2 Describes why it is well timed and/or novel
89.2
100
84
1.3 Defines the target group and need (think about health inequities)
97.1
100
88
1.4 Justifies that the target group will benefit from the program or intervention, based on related research, best-practice examples, or a needs assessment
96.1
100
92
2.1 Promotes healthy eating behaviors
99
100
96
2.2 Includes nutrition-related goals that address proximal outcomes
30.4
48
20
100
100
100
99
100
96
3.1 Uses research or best-practice examples to show how a model or framework integrates with 46.1 the program goal
68
28
3.2 Includes partnerships with other groups and explains how these partners aid the program
21.6
48
0
3.3 Meets the needs of the target group
99
100
96
3.4 Addresses external influences on food and eating
58.8
80
12
4.1 Describes an appropriate setting for target group
86.3
100
76
4.2 Describes realistic recruitment and retention of participants
97.1
100
88
Category 2: Program Goal
2.3 Includes nutrition-related goals that address intermediate or distal outcomes If weight is the outcome, including appropriate physical activity in goals is encouraged. 2.4 Includes measurable goals Category 3: Program Framework
Category 4: Program Setting, Recruitment, and Retention
Category 5: Program Methods 5.1 Includes several techniques to promote learning
81.4
92
64
5.2 Includes several techniques to motivate participants
61.8
88
20
5.3 Includes several techniques to promote nutrition behavior change
91.2
96
80
5.4 Explains why the planned teaching time and dose are adequate/fitting using related research or best-practice examples as support
11.8
24
4
100
96
100
72
84
24
96
32
100
Category 6: Program Content 6.1 Relates to program goals
99
6.2 Is based on best-practice examples or related research with citations from relevant research 88.2 or government/health society guidance 6.3 Is supported by experts or key informants
52
Category 7: Program Materials 62.7 7.1 The program cites and explains that the materials have social and cultural relevance, including: language, reading level, food likes and dislikes, household status, food/ diet needs, interests, age/development stage matched, learning style, and/or format Category 8: Evaluation 8.1 Includes measurement tools that address program goals
87.3
92
8.2 Includes measurement tools that are reliable, valid, and chosen based on related research or best practice
93.1
100
96.3
(continued on next page) 124
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
January 2015 Volume 115 Number 1
FROM THE ACADEMY Table 1. Percent of programs achieving GENIEa quality criteria for programs in the top 25% of GENIE scores, bottom 25% of GENIE scores, and total for all 102 programs reviewed (continued)
Quality criteria
Total
Top scoring quartile
Bottom scoring quartile
% achieved! 8.3 Evaluation method is cited, based on related research, best practice or includes pilot testing 86.3 to support its use
100
100
8.4 Includes process evaluation to check that the program is implemented as planned
37.3
64
44.4
8.5 Includes outcome evaluation measures for proximal goals
27.5
40
29.6
8.6 Includes outcome and/or impact evaluation measures for intermediate or distal goals
99
8.7 Evaluates outcome/impact at multiple time points
37.3
48
8.8 Includes an appropriate analysis plan
97.1
96
100
100 44.4 100
Category 9: Sustainability 9.1 Addresses the potential for the program to continue
49
9.2 Indicates evidence of prior/current successful programs
78.4
84
24
100
60
9.3 Describes shared roles and duties of program partners
13.7
36
4
9.4 Implies potential for broader reach, replication, and growth
51
84
24
24
4
9.5 Addresses the collective program impact within the community and/or among program partners
7.8
a
GENIE¼Guide for Effective Nutrition Interventions and Education.
3.
physiological (such as weight change, body mass index, lipid concentration, and blood pressure).
These outcome categories were established a priori as part of the GENIE development process.12 The first author extracted the data and completed the GENIE assessment for each of the 102 unique programs assessed. Programs were grouped into quartiles based on overall GENIE score. The maximum GENIE score was 35. The fourth (upper) quartile contained 25 programs, all with a GENIE score of 27 or higher. The first (lowest) quartile also contained 25 programs, all with a GENIE score of 21 or lower. For each quartile and the dataset as a whole, we calculated the prevalence of each GENIE criteria (percent of articles with the criteria present). To examine the relationship between the presence of quality criteria and positive outcomes, the number of outcomes achieved was calculated as a proportion of outcomes measured for each program. Achieved outcomes were outcomes that the program authors considered successful, because either they met a specified goal or reached statistical significance compared to baseline values or a control January 2015 Volume 115 Number 1
group. This proportion was calculated for each reviewed program separately for knowledge-based, behavioral, and physiological outcomes, and cumulatively for all outcomes. The proportions of outcomes achieved were compared to individual GENIE category scores and overall GENIE scores for each program using Spearman’s correlation. The raw number of outcomes measured and the raw number of outcomes achieved for each program was also compared to individual GENIE category scores and overall GENIE scores for each program using Spearman’s correlation. c2 Tests were used to look for associations between these proportions and all GENIE categories collectively. All analyses were conducted in SPSS software, version 20 (2011, SPSS IBM).
and high-scoring programs, and other program qualities were unique to highscoring programs within our sample. Table 1 illustrates the prevalence of GENIE’s quality criteria among programs in the upper quartile of overall scores, programs in the lower quartile of overall scores, and all programs reviewed. To illustrate the relationship between individual categories and overall GENIE score, nine programs were selected to represent an outstanding example of each GENIE category. For consistency, all example programs tracked weight change as an outcome and only those with GENIE scores in the top 25% were selected. These programs are presented in Table 2 and provided as examples in the text.
Categories 1, 2, and 4 PROGRAM COMPARISON USING GENIE Certain GENIE criteria were more likely to be found in high-scoring programs than low-scoring programs. It was expected that high-scoring GENIE programs would achieve more quality criteria than their lower-scoring counterparts. However, some program qualities were common in both low-
Within GENIE categories 1 (Program Description and Importance) and 4 (Program Setting, Recruitment, and Retention), nearly all quality criteria were achieved by all programs. A similar trend was seen within GENIE category 2 (Program Goal). However, criterion 2.2 (“includes nutrition-related goals that address proximal outcomes”) was rarely present in either high- or low-scoring programs. Within the context of GENIE,
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
125
FROM THE ACADEMY proximal outcomes were defined as knowledge-related outcomes. Because solely knowledge-focused nutrition education programs have not been found to be effective in producing behavior change,23 programs destined for publication may fail to set goals related to knowledge. In programs where knowledge-related goals were defined, knowledge acquisition was frequently examined as a mediator to explain any behavioral changes that may have been observed.13,14,21 For example, although weight change was the primary outcome in Turner-McGrievy and colleagues’14 podcasting intervention, knowledge acquisition was also measured. The authors suggest that improved levels of participant engagement within the intervention group and greater weight loss could be partly attributed to increased weight-control knowledge.14
Category 3 Within GENIE category 3 (Program Framework), there were particularly large disparities between high- and low-scoring programs in achievement of criterion 3.2 (“includes partnerships with other groups and explains how these partners aid the program”) and criterion 3.4 (addresses external influence on food and eating”). Successful partnerships require coordination and foresight, as demonstrated by Glasgow and Toobert, who incorporated a variety of community resources in their diabetes self-management intervention.15 Eighty percent of high-scoring programs addressed external influences on food and eating, and only 12% of lower-scoring programs included this characteristic. Addressing this criterion requires a more multifaceted approach, which might include many different intervention techniques, strategies, and resources. Therefore, programs fulfilling only the most commonly achieved criteria may lack the dimensions to achieve this criterion.
Category 5 Within GENIE category 5 (Instructional Methods), there was a 68-percentagepoint difference between high- and low-scoring programs in the likelihood of achieving criterion 5.2 (“includes several techniques to motivate 126
participants”). Criterion 5.3 (“strategies to promote learning and strategies to promote behavior change”) was commonly found in both high- and low-scoring programs. Motivational strategies can be just as important as strategies to promote behavior change, but appear to be more commonly overlooked, perhaps because research has not clearly shown how these strategies should be put into practice.24 However, motivating participants is vital for empowerment and maintaining engagement.25 Kristal and colleagues motivated participants to lower their dietary fat intake through interactive activities, self-assessments, and social support.17 Only 11.8% of all programs met criterion 5.4 (“explains why the planned teaching time and dose are adequate/fitting using related research or best practice examples as support”). Information concerning the doseeresponse relationship with regard to nutrition education appears limited to the guideline of 50 hours needed to produce behavior change found by Connell and colleagues in their review of classroom health education, which was completed nearly 30 years ago.1,26 Therefore, many program developers may not seek out evidence to support the timing of their program. However, for nutrition education to promote positive health outcomes, more research is needed on how much programming is needed to result in the desired behavior change.
Categories 6 and 7 Large differences were also observed between programs in the upper quartile and programs in the lower quartile within GENIE category 6 (Program Content) for criterion 6.3 and for criterion 7.1. In general, most reviewed programs had appropriate content and materials, but less than a quarter of low-scoring programs had support from experts or key informants for their content and less than a third cite that the materials used in the program had social and cultural relevance to participants. These results show that including appropriate materials is far more common in higher-scoring programs, but also may indicate that published programs do not thoroughly describe and support their choice of materials and content. Johnston and colleagues’ intervention to promote
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
weight loss among Mexican-American children proved to be an exception by using culturally correct recipes, bilingual written information, and an age appropriate token economy with prizes to encourage engagement.20
Category 8 Within GENIE category 8 (Evaluation), there were three cases where criteria were found roughly half as often in low-scoring programs than they were in high-scoring programs: criteria 8.4 (“includes process evaluation to check that the program is implemented as planned”), 8.5 (“includes outcome evaluation measures for proximal goals”), and 8.7 (“evaluates outcome/impact at multiple time points”). Evaluation can be expensive, time consuming, and frequently planned only after sufficient programming funds have been secured.8 Given that GENIE criterion 2.2 (“setting proximal goals”) was often missing, it is not surprising that GENIE criterion 8.5 (“measuring proximal outcomes”) is infrequent. However, one must remember that knowledge may be measured, but not prioritized for publication. This was the case in Kristal and colleagues’ intervention for breast cancer survivors where nutrition education was targeted, but participant knowledge was not reported in the published article.17
Category 9 Most criteria within GENIE category 9 (Sustainability) were not observed in the reviewed programs with the following exception, criterion 9.2 (“indicates evidences of prior/current successful programs”). The lowest percent achievement out of all criteria was observed for 9.5 (“addresses the collective impact within the community and/or among program partners”). Although the percent of programs achieving criteria within category 9 tended to be low overall, higherscoring programs appeared to include these criteria far more often. The inclusion of partners and other community organizations in criteria 9.3 and 9.5 were frequently missing programmatic elements in published programs. Criterion 9.4 (“implies potential for broader reach, replication, and growth”) was also far more likely to be found in high-scoring programs than January 2015 Volume 115 Number 1
FROM THE ACADEMY low-scoring programs. For example, Salinardi and colleagues provide an excellent model for sustainability by incorporating long-lasting environmental changes as part of a worksite intervention.22 Planning for sustainability may promote program designs that are better integrated with and supported by community partners, easily adapted to different settings, and financially stable. These features that promote sustainability may not only increase the likelihood of a program continuing, but also create a stronger program in other ways.1,7,18,27
GENIE CATEGORY EXAMPLES Table 2 was developed by selecting programs within the upper quartile for overall GENIE score that tracked weight change as an outcome. Nine programs were chosen to illustrate examples of how criteria within each of GENIE’s categories were fulfilled.
Relationship between GENIE Score and Program Outcome When the proportion of knowledgebased outcomes achieved was compared to GENIE category scores, positive significant associations were observed between outcomes and scores on category 2 (Program Goal) and 8 (Evaluation). A negative correlation was observed between the proportion of knowledge-based outcomes achieved and category 3 (Program Framework) score. No meaningful relationships were observed between the proportion of behavioral or physiological outcomes and GENIE category scores. No meaningful relationships were observed between either the raw number of outcomes measured or the raw number of outcomes achieved and GENIE scores for individual categories or total GENIE scores. Overall, the nutrition education programs included in this review tended to achieve high GENIE scores. There was a relatively small difference separating programs scoring in the 25th percentile and those in the 75th percentile. Given the maximum overall GENIE score of 35, we might expect the 25th and 75th percentile to fall at roughly 9 and 26, respectively, given an even distribution across all possible GENIE scores. The higher scores observed in this study (25th percentile¼21; 75th percentile¼27) are indicative of a January 2015 Volume 115 Number 1
negatively skewed score set, meaning that GENIE scores tended to be more common in the upper range. A likely explanation for this score distribution is that nutrition education programs that result in peer-reviewed publications tend to be of a higher quality than program proposals in general, due to multiple types of publication bias. Although, rejection of a manuscript is not considered a common cause of publication bias,28 many nutrition education programs intended for publication do not move through the publication process due to poor programming, poor results, or poor reporting.28,29 These programs are destined to remain in what is commonly referred to as the “file drawer” and may never be formulated into a manuscript, submitted, considered, or accepted by a journal.29 Some programs intended for publication are never accepted by a journal because the program failed to achieve statistically significant results.28 Programs conducted by those who do not consider themselves part of the scientific community may never be considered for publication, regardless of results.5 In general, programs that are published in peer-reviewed literature are more likely to have statistically significant results and be viewed as more important by the investigators, particularly for those programs of an observational nature.28 Another possible explanation for the relatively high GENIE scores is that GENIE was designed to assess nutrition education program plans and proposals rather than published articles. Program plans and proposals have inherent differences. For example, published programs may be more likely to contain the criteria within category 1 because this information is commonly required for the introduction section of many journals. In fact, our review found that nearly all programs contained these criteria. Our review also found that nearly all programs included the criteria within category 4. These findings support the notion that nutrition education without an appropriate setting, recruitment, and retention plan may not be able to secure and sustain enough participants to support a program deemed viable for publication. Issues of publication bias and inherent differences between published nutrition education manuscripts
and program proposals may also explain why GENIE category or overall scores failed to be a reliable predictor of achieved health outcomes. GENIE was designed to help nutrition education program developers assess program plans or proposals,12 and the use of GENIE in this context represents retrofitting the tool to examine published programs. A retrospective analysis of published programs was undertaken for reasons of feasibility, cost, and ease. Published articles are in the public domain and, therefore, far easier to obtain than program proposals. Program proposals are rarely made public and would require extensive coordination with proposal authors, organizations, and/or funders. In addition, programs that do not require external funding may not have been developed from a formal written plan or proposal. However, future work with GENIE should seek to evaluate program proposals and obtain outcomes in order to establish another measure of the tool’s criterion validity.
CONCLUSIONS When assessed using GENIE, highscoring nutrition programs were found to possess quality characteristics different than those of low-scoring programs. Prevalent program characteristics common in all included programs were thorough program descriptions, appropriate goals with matching content that addressed target audience needs, and realistic recruitment strategies and statistical analysis. Program characteristics most likely to be found only in higher-scoring programs were research-based program content and rigorous evaluation methodology, techniques to motivate participants, consideration of external influences on participant behavior, and partnerships with other organizations to promote sustainability. Some exceptions were seen that can be explained by the intrinsic differences between published nutrition education program articles and program proposals. Nutrition education developers can set their programs apart by including quality elements commonly found missing in published programs. All assessed programs tended to score higher than expected within a normal distribution when assessed using GENIE.
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
127
FROM THE ACADEMY Table 2. Exemplar programs highlighting each of the Guide for Effective Nutrition Interventions and Education’s nine categories Author(s) of study, year Participants and design
GENIEa total score and category example
Stahre et al, 200713
Category 1: Program Description and Importance n¼1 GENIE score: 31 Evaluation retention: 59% Justify the need for program among those who Adult women who work as or for child-care providers, body mass index >30; group sessions work with children and why a cognitive with lectures, homework, and discussion; control behavioral approach is appropriate; evidence that similar programs have been successful; new group received program similar to standard effort to look at longer-term weight loss weight-control programs used on occupation health setting; researcher-designed curricula
Turner-McGrievy et al, 200914
Category 2: Program Goal n¼41 GENIE score: 29 Evaluation retention: 0.85 Trial emerging technology of podcasting to Adult individual intervention; Social Cognitive promote weight loss using behavioral theory; Theory podcast; control group received popular includes knowledge goals and physiologic goals podcast emphasizing stimulus control and (weight and body mass index); clear comparison positive body image; researcher designed of podcast programs podcasts
Glasgow and Toobert, 200015
Category 3: Program Framework n¼68 GENIE Score: 28 Evaluation retention: 85% Social-cognitive environmental intervention; multiAdult individual and group intervention, first group visit 60 to 120 min; tailored feedback via level model; modified RE-AIMb framework for computer program, follow-up phone calls, evaluation; integration of community resources; community resources; partial research design meets needs of participants; addresses curriculaeresearcher-designed computer environmental influences on eating and health program; American Heart Association recipes
Gold et al, 200716
Category 4: Program Setting, Recruitment, and n¼40 Retention Plan Program retention: 65% GENIE score: 28 Adults, individual and group; online behavioral weight loss with assignments, “meetings,” and Intervention offered online; cited as common and convenient mode of obtaining health individual feedback; VTrim (researcher-designed information; participants recruited locally, curricula) compared with commercial program technology check performed, encouragement ediets.com provided, attrition on par with control and similar programs
Kristal et al, 199717
n¼57 Program retention: 77% Adult individual and group intervention 6 weekly individual sessions; 10 monthly group session; additional individual session at 6 mo; based on Women’s Health Trial
Booth et al, 200818
Category 6: Program Content n¼53 GENIE score: 27 Program retention: 73% Step guidelines based on National Heart Adults, individual intervention; Foundation of Australia guidelines, dietary website for personal physical activity and diet tracking; personalized feedback; message forum advice based on best practice for weight loss (decrease saturated fat, increase fruits and for feedback from researchers/fellow vegetables); contact matches program goals participants; evaluation completed in person with researchers; researcher-designed curricula (continued on next page)
128
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
Category 5: Instructional Methods GENIE score: 31 Includes educational and behavior change skills development objectives; provides social support to promote new dietary habits; feedback, reinforcement, and problem solving; emphasis on interactive activities and self-assessment
January 2015 Volume 115 Number 1
FROM THE ACADEMY Table 2. Exemplar programs highlighting each of the Guide for Effective Nutrition Interventions and Education’s nine categories (continued) Author(s) of study, year Participants and design
GENIEa total score and category example
Category 7: Program Materials Johnston et al, 201019 n¼37 GENIE score: 32 Johnston et al, 200720 Evaluation retention: 93% Materials age appropriate for audience; childChildren (primary) caregivers (secondary); appropriate approach to healthy eating and individual and group; physical activity; caregiver materials culturally Mexican-American children at charter school; not appropriate; all communications in English and differentiated by weight status, but only Spanish; child-appropriate token economy used overweight/obese children included in the to encourage behaviors analysis; one nutrition education and four physical activity classes/week in group; biweekly quizzes; rewards for meeting goals; 1-on-1 education for not meeting goals; monthly education with parents/caregivers, including extended family; mostly researcher designed curricula based on Go, Slow, Whoa concept Englert et al, 200721
Category 8: Evaluation n¼1,517 GENIE score: 33 Program retention: 97% Adult group intervention; CHIP (Coronary Health Assessments completed at multiple time points Improvement Program); intensive series of group (including post-intervention); process evaluation used to explore mediating variables; appropriate, classes with readings, lectures, question and validated measurement tools answer sessions, and hands-on workshops
Salinardi et al, 201322
Category 9: Sustainability n¼40 GENIE score: 30 Program retention: 89% Program emphasizes environmental changes; Adult; group worksite intervention, 4 sites; group partnership with worksites; opportunities for all program with weekly and biweekly sessions; maintenance sessions for 6 mo afterward monthly; employees to participate; promotes healthy monthly newsletters and seminars for workers not worksite and supportive weight-loss participating in the intervention; partial researcher environment; worksites requested program continue designed curricula based on The “I” Diet (book) modified to emphasize higher fiber
a
GENIE¼Guide for Effective Nutrition Interventions and Education. RE-AIM¼Reach Effectiveness Adoption Implementation Maintenance.
b
This can be attributed to multiple types of publication bias, which greatly limit the number of nutrition education programs that are formally presented to the public through peer-reviewed publication. These issues were also likely to contribute to the finding that GENIE could not be proven as a reliable predictor of positive outcomes when used to assess published articles. Although this study design was used for reasons of practicality, GENIE was not intended to assess published programs. Rather, it was created and validated for assessing program proposals. A prospective analysis whereby GENIE is used to score program proposals and these scores are later compared to program outcomes would be a more accurate test of the January 2015 Volume 115 Number 1
potential of GENIE’s scores to predict positive health outcomes.
References
obesity: Systematic review with metaanalysis. Pediatrics. 2012;130(6):e1647e1671. 5.
Sims LS. Chapter 1 Introduction. J Nutr Educ. 1995;27(6):284-286.
1.
Blue CL, Black DR. Synthesis of intervention research to modify physical activity and dietary behaviors. Res Theory Nurs Pract An Int J. 2005;19(1):25-61.
6.
Gillespie AH, Brun JK. Trends and challenges for nutrition education research. J Nutr Educ. 1992;24(5):222-226.
2.
Desroches S, Lapointe A, Ratté S, Gravel K, Légaré F, Turcotte S. Interventions to enhance adherence to dietary advice for preventing and managing chronic diseases in adults. Cochrane database Syst Rev. 2013;2(2):CD008722.
7.
Contento I. Review of nutrition education research in the Journal of Nutrition Education and Behavior, 1998 to 2007. J Nutr Educ Behav. 2008;40(6): 331-340.
8.
3.
Kwok TCY, Lam LCW, Sea MMM, Goggins W, Woo J. A randomized controlled trial of dietetic interventions to prevent cognitive decline in old age hostel residents. Eur J Clin Nutr. 2012;66(10): 1135-1140.
Lytle L, Achterberg C. Changing the diet of America’s children: What works and why? J Nutr Educ. 1995;27(5):250-260.
9.
Spahn JM, Reeves RS, Keim KS, et al. State of the evidence regarding behavior change theories and strategies in nutrition counseling to facilitate health and food behavior change. J Am Diet Assoc. 2010;110(6):879-891.
4.
Ho M, Garnett SP, Baur L, et al. Effectiveness of lifestyle interventions in child
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
129
FROM THE ACADEMY 10.
11.
12.
13.
14.
15.
Nation M, Crusto C, Wandersman A, et al. What works in prevention: Principles of effective prevention programs. Am Psychol. 2003;58(6-7):449-456. Greidanus IDA, Contento S. The relationship between the ability to solve practical nutrition problems in an adult population and piagetian cognitive levels. J Nutr Educ. 1989;21(2):73-81.
16.
17.
Abram J, Colby S. GENIE: Your nutrition education wishes have been granted. Presented at: 2014 Food & Nutrition Conference & Expo; October 18-21, 2014; Atlanta, GA. Stahre L, Tärnell B, Håkanson C-E, Hällström T. A randomized controlled trial of two weight-reducing short-term group treatment programs for obesity with an 18-month follow-up. Int J Behav Med. 2007;14(1):48-55. Turner-McGrievy GM, Campbell MK, Tate DF, Truesdale KP, Bowling JM, Crosby L. Pounds Off Digitally study: A randomized podcasting weight-loss intervention. Am J Prev Med. 2009;37(4): 263-269. Glasgow RE, Toobert DJ. Brief, computerassisted diabetes dietary self-management counseling: Effects on behavior, physiologic outcomes, and quality of life. Med Care. 2000;38(11):1062-1073.
Gold BC, Burke S, Pintauro S, Buzzell P, Harvey-Berino J. Weight loss on the web: A pilot study comparing a structured behavioral intervention to a commercial program. Obesity (Silver Spring). 2007; 15(1):155-164. Kristal AR, Shattuck AL, Bowen DJ, Sponzo RW, Nixon DW. Feasibility of using volunteer research staff to deliver and evaluate a low-fat dietary intervention: The American Cancer Society Breast Cancer Dietary Intervention Project. Cancer Epidemiol Biomarkers Prev. 1997;6(6): 459-467.
The Rockford CHIP. Prev Med. 2007;44(6): 513-519. 22.
Salinardi TC, Batra P, Roberts SB, et al. Lifestyle intervention reduces body weight and improves cardiometabolic risk factors in worksites. Am J Clin Nutr. 2013;97(4):667-676.
23.
Contento IR. Nutrition education: Linking research, theory, and practice. Asia Pac J Clin Nutr. 2008;17(suppl 1):176-179.
24.
Abusabha R, Peacock J, Achterberg C. How to make nutrition education more meaningful through facilitated group discussions. J Am Diet Assoc. 1999;99(1):72-76.
25.
Rusness BA. Striving for empowerment through nutrition education. J Am Diet Assoc. 1993;93(1):78-79.
18.
Booth AO, Nowson CA, Matters H. Evaluation of an interactive, internet-based weight loss program: A pilot study. Health Educ Res. 2008;23(3):371-381.
19.
Johnston CA, Tyler C, Fullerton G, et al. Effects of a school-based weight maintenance program for Mexican-American children: Results at 2 years. Obesity (Silver Spring). 2010;18(3):542-547.
26.
Connell DB, Turner RR, Mason EF. Summary of findings of the School Health Education Evaluation: Health promotion effectiveness, implementation, and costs. J Sch Health. 1985;55(8):316-321.
20.
Johnston CA, Tyler C, McFarlin BK, et al. Weight loss in overweight Mexican American children: A randomized, controlled trial. Pediatrics. 2007;120(6): e1450-e1457.
27.
Sims LS. Chapter 9: Conclusions. J Nutr Educ. 1995;27(6):355-364.
28.
Easterbrook P, Gopalan R, Berlin J, Matthews D. Publication bias in clinical research. Lancet. 1991;337(8746):867-872.
29.
Scargle JD. Publication bias (The “FileDrawer Problem”) in scientific inference. J Sci Explor. 1999;14(1):91-106.
21.
Englert HS, Diehl HA, Greenlaw RL, Willich SN, Aldana S. The effect of a community-based coronary risk reduction:
AUTHOR INFORMATION J. K. Abram is project coordinator, Dietetics Practice Based Research Network, Academy of Nutrition and Dietetics, Chicago, IL; at the time of the study, she was a ConAgra Foods Foundation Nutrition Education Research Fellow, Academy of Nutrition and Dietetics, Chicago, IL. R. K. Hand is a senior manager, Dietetics Practice Based Research Network, K. Brown is national education director, P. J. Ziegler is senior director, research and evidence analysis, and A. L. Steiber is chief science officer, Academy of Nutrition and Dietetics, Chicago, IL. J. S. Parrott is an associate professor, Rutgers University, Newark, NJ. Address correspondence to: Rosa K. Hand, MS, RDN, LD, Dietetics Practice Based Research Network, Academy of Nutrition and Dietetics, 120 S Riverside Plaza, Suite 2000, Chicago, IL 60606-6995. E-mail: [email protected]
STATEMENT OF POTENTIAL CONFLICT OF INTEREST No potential conflict of interest was reported by the authors.
FUNDING/SUPPORT Funding was provided by the ConAgra Foods Foundation through an educational grant to the Academy of Nutrition and Dietetics Foundation, which included salary support for the investigators. The ConAgra Foods Foundation was not involved in the collection, analysis or interpretation of data.
ACKNOWLEDGEMENTS The authors would like to acknowledge the contribution of Brian Oliver, MLIS Medical Librarian at the Academy of Nutrition and Dietetics, for his guidance in organizing our literature search and obtaining manuscripts for review.
130
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
January 2015 Volume 115 Number 1