- Email: [email protected]
Short-Term Effectiveness of an Outcomes Research Training Curriculum Within a Coordinated Program
RESEARCH Research and Professional Briefs
Short-Term Effectiveness of an Outcomes Research Training Curriculum Within a Coordinated Program CATHERINE A. PETERSON, PhD, RD; JANET E. HAYS-KIMMONS, MS, RD; JAMES S. COLE, PhD
ABSTRACT The fourth edition of the Commission on Accreditation for Dietetics Education Standards of Education mandated outcomes research training. Our objective was to determine the short-term effectiveness (⬍5 years) of the outcomes research training curriculum in the Coordinated Program in Dietetics (CP) at the University of MissouriColumbia, which exceeds these minimum standards. Toward this end, a survey tool was administered to University of Missouri-Columbia CP graduates before the implementation of the fourth edition of Standards of Education and to University of Missouri-Columbia CP graduates with two semesters of outcomes research training; graduates of two other CPs from different universities from the same years were also surveyed. Graduates who went through CPs after implementation of the fourth edition of the Standards of Education showed notable improvement in attitudes, interest, and participation in most of the outcomes research skills queried, compared with those who completed their education under the earlier standards; knowledge was only slightly improved. Of these areas, only enhancements in attitudes and interest were consistently more pronounced in food and nutrition professionals trained using the University of MissouriColumbia’s outcomes research curriculum compared with other CP graduates. Both the Commission on Accreditation for Dietetics Education and program directors can use the findings described herein to further assess and develop the research competencies and/or to strengthen their programs. J Am Diet Assoc. 2008;108:120-124.
C. A. Peterson is an assistant professor, Dietetics Program, Department of Nutritional Sciences, University of Missouri-Columbia, Columbia, MO. J. E. Hays-Kimmons is a clinical dietitian, Boone Hospital, Columbia, MO. J. S. Cole is a research analyst, Center for Postsecondary Research, Indiana University, Bloomington. Address correspondence to: Catherine A. Peterson, PhD, RD, University of Missouri-Columbia, 106 McKee Hall, Columbia, MO 65211. E-mail: petersonca@ missouri.edu Copyright © 2008 by the American Dietetic Association. 0002-8223/08/10801-0004$34.00/0 doi: 10.1016/j.jada.2007.10.009
120
Journal of the AMERICAN DIETETIC ASSOCIATION
I
t is well established that research, including outcomes research, is essential to the dietetics profession and that dietetics education is the starting point for appreciating this relationship (1-10). Toward this end, in 1997 the Commission on Accreditation for Dietetics Education Standards of Education (11) began to include “foundation knowledge and skills” as well as competency statements related to outcomes research and research training. The current 2000 Commission on Accreditation for Dietetics Education Eligibility Requirements and Accreditation Standards retains these requirements (12). The researchrelated foundation knowledge and skills the Commission on Accreditation for Dietetics Education outlined include knowledge of interpretation of current research, needs assessment, basic statistics, and quality improvement, whereas the research-related competency statements include “interpret and incorporate new scientific knowledge . . .” and “develop and measure outcomes for . . . practice.” The outcomes research training curriculum in the Coordinated Program in Dietetics (CP) at the University of Missouri-Columbia surpasses the minimum standards the Commission on Accreditation for Dietetics Education established in Eligibility Requirements and Accreditation Standards by providing dietetics students with supervised practice experience in designing, conducting, and presenting outcomes-based research projects. This unique two-semester outcomes research training curriculum, described in detail in the Journal (3), has for several years used a collaborative team to prepare entry-level food and nutrition professionals to execute a research plan. The objective of this study was to determine the short-term (⬍5 years postgraduation) effectiveness of this outcomes research training curriculum. Our hypothesis was that incorporating an outcomes research training curriculum within the CP would improve attitudes toward, knowledge of, and interest and participation in outcomes research among food and nutrition professionals who complete the curriculum. METHODS Subject Selection The University of Missouri-Columbia dietetics program is a 2-year CP. Subjects from two University of MissouriColumbia graduation classes were selected for participation in this 2001 study: one class graduated before the initiation of the fourth edition of The Standards of Education was published and had no formal training in outcomes research (n⫽17); the other class was enrolled in
© 2008 by the American Dietetic Association
the two-semester research curriculum (n⫽15) (3). Both classes were under the mentorship of the same dietetics faculty except for one professor who introduced the outcomes research curriculum. Graduates of two other CPs from the same 2 years were also included. Using the American Dietetic Association Directory of Dietetics Programs (13), the two CPs were selected based on their similarity to University of Missouri-Columbia’s program using the following criteria: similar class size, minimum grade point average of 2.5 for acceptance, Committee on Dietetics Education Area Two designation, and a medical nutrition therapy/clinical emphasis. Both institutions agreed to participate by providing their alumni mailing lists for the specified years and were designated Program X (n⫽24) or Program Y (n⫽29) to ensure confidentiality. Human subject approval was obtained through the University of Missouri-Columbia Campus Institutional Review Board. Study Design and Survey Tool A survey was used to assess attitudes toward, knowledge of, and interest and participation in outcomes research of CP graduates from the three different programs, before (pre-outcomes research) and after (post-outcomes research) the initiation of the fourth edition of The Standards of Education; as well as to assess the effectiveness of the two-semester outcomes research curriculum at the University of Missouri-Columbia. The 20-item survey was developed by the authors in collaboration with the Assessment Resource Center at University of Missouri-Columbia and adapted from previously developed instruments (4,5,14,15). Content and format were based upon established guidelines (5,16-18). Multiple-choice or dichotomous questions were used to obtain information on education, employment, and research training/experience and involvement; Likert scales, in accordance with Peterson’s guidelines, were used to measure participants’ attitudes, preferences, or confidence levels regarding research/outcomes research as it relates to dietetics in general as well as in their own careers (16). A subset of survey questions was designed to measure each of four content areas (ie, attitudes, knowledge, interest, and participation) related to specific research skills and to allow for cross-comparisons using a technique based on the studies of Carson and Hedl (19) and Ma and Contento (20). Face and content validity were assessed for content, clarity, readability, and comprehensiveness through a pilot test conducted on local practicing registered dietitians (n⫽8). The survey was then slightly modified (wording and question placement) according to feedback. A chance to win (via blinded, random drawing) payment of the registration fee for the American Dietetic Association 2001 Food & Nutrition Conference & Expo was included in the mailing to increase the response rate of eligible graduates. Statistics The Statistical Package for the Social Sciences (version 10.0.5, 1999, SPSS Inc, Chicago, IL) was used for data compilation and statistical analysis. Pearson’s 2 was
used to compare graduation years (pre-outcomes research vs post-outcomes research graduates) and the three programs (ie, University of Missouri-Columbia, Program X, Program Y); as well as to assess responses to individual questions selected a priori within each content area (attitudes, knowledge, interest, and participation). Variable frequencies were computed to describe the data. Because responses between Programs X and Y were not significantly different, data were collapsed; thus, University of Missouri-Columbia post-outcomes research graduates were compared with “other” post-outcomes research graduates to evaluate the effectiveness of University of Missouri-Columbia’s outcomes research curriculum; the other programs were meant to serve as representatives of CP curriculums in general. RESULTS AND DISCUSSION We evaluated the short-term effectiveness of an outcomes research curriculum at University of Missouri-Columbia by attempting to survey a total of 85 food and nutrition professionals from six graduation classes, representing two different editions of the Committee on Dietetics Education education standards and three different CPs. Owing to one missing address and 11 returned surveys marked “undeliverable” by the US Postal Service, there were a total of 73 deliverable surveys. Fifty-seven usable surveys were received; thus, the overall response rate was 78% (57 of 73). There were no significant differences in response rates among the three programs or between years of graduation. The data demonstrate notable improvements in attitudes, interest, and participation in several of the outcomes research and research-related skills of food and nutrition professionals who graduated after the initiation of the fourth edition of The Standards of Education (11); improvements in knowledge were less notable. Among these four content areas, only enhancements in attitude and interest were more pronounced in food and nutrition professionals trained using the University of MissouriColumbia outcomes research curriculum; thus, the data only partially support our hypothesis. Although the main focus of this analysis was to assess the effectiveness of a specific curriculum, the results are also useful in assessing the strengths of the researchrelated foundation skills and competencies required in current dietetics education. CP Graduates Critical of Research Training and Preparedness Fifty-two of 57 respondents stated that the CP from which they graduated “included instruction or training” on the research process, yet only 32 out of 57 believed their educational training “actually prepared” them for participating in research. Ninety-seven percent (28 of 29) of the post-outcomes research graduates vs 82% (23 of 28) of pre-outcomes research graduates reported that the CP from which they graduated included instruction or training on the outcomes research process (P⫽0.176). All postoutcomes research graduates (29 of 29) and nearly all pre-outcomes research graduates (26 of 28) reported learning about outcomes research via coursework as opposed to other modes.
January 2008 ● Journal of the AMERICAN DIETETIC ASSOCIATION
121
Table 1. Responses of pre-outcomes researcha (Pre-OR) vs post-outcomes researchb (Post-OR) Coordinated Program in Dietetics graduates to questions regarding various research skillsc Knowledged
Intereste
Pre-OR Post-OR Research skill Read and critically evaluate a research article Conduct a comprehensive literature review Write a research proposal Identify research problems Define research objectives Obtain approval from an HSC/ IRBg committee Collect data for research Use statistics to analyze data Present research at conference Write a journal article Collect data for quality improvement
Participationf
Pre-OR Post-OR
Attitudee
Pre-OR Post-OR
Pre-OR
Post-OR
n
%
n
%
n
%
n
%
n
%
n
%
n
21
75
27
93
3.51
22
79
26
90
1.32
15
54
24
83
5.62*
27
96
28
97
0.003
18
64
24
83
2.51
16
57
22
76
2.25
11
39
13
45
0.179
21
78
23
79
0.005
10
36
17
59
3.00
12
43
20
72
5.11*
2
7
11
36
6.79**
17
63
25
86
4.03*
19
68
24
83
1.71
20
71
25
86
1.87
8
29
17
59
5.22**
21
78
28
97
4.51*
23
82
25
86
0.177
20
71
27
93
4.63*
19
68
21
72
0.141
27
96
29
100
1.09
6
21
5
17
0.160
12
43
19
66
2.95
1
4
5
18
2.99
16
56
16
55
0.001
19
68
24
83
1.71
18
64
27
93
7.12**
9
32
14
46
1.20
25
89
28
97
1.24
9
32
12
41
0.522
13
46
20
69
2.97
14
50
17
59
0.427
24
85
26
90
0.255
16
57
9
31
3.94*
15
54
19
66
0.845
5
18
3
10
0.667
21
74
25
86
1.30
7
25
2
7
3.51
13
46
16
57
0.644
2
7
1
3
0.380
19
67
21
72
0.218
19
68
21
72
0.141
20
70
27
93
4.67*
18
63
18
62
0.005
28
100
26
90
2.95
2
2
2
%
n
%
2
a
Graduates from coordinated programs before outcomes research became a component of dietetics’ standards of education (n⫽25-28). Graduates from coordinated programs after outcomes research became a component of dietetics’ standards of education (n⫽26-29). The group with the greater percentage value is bold; shaded indicates significantly different at P⬍0.05. d Respondents answering “accomplished” or “adequate.” e Respondents answering “highly” or “somewhat” interested. f Respondents answering “yes.” g HSC/IRB⫽ Human Subjects Committee/Institutional Research Board. *P⬍0.05. **P⬍0.01. b c
Our results show that the majority of CP graduates believe they are at least “adequately” trained for most of the specific research skills discussed (Tables 1 and 2). However, graduates reported deficiencies (ie, percentage “accomplished” or “adequate” responses ⬍50%) in knowledge of obtaining approval from a Human Subjects Committee/Institutional Review Board, writing a journal article, and using statistics to analyze data. This latter finding is surprising given that all respondents reported completing at least one statistics course, with one third completing two. Remarkably, more post-outcomes research graduates reported being “in need of more training” in the above three skills areas than pre-outcomes research graduates (Table 1), perhaps because pre-outcomes research graduates believe they have gained adequate research experience in the workplace or graduate school, or because
122
January 2008 Volume 108 Number 1
post-outcomes research graduates are more aware of the increasing emphasis on research within the profession. Improvement in CP Graduates’ Interest in Research Using a scale from one to five, where 1⫽strongly disagree and 5⫽strongly agree, respondents were asked to rate their interest level in participating in outcomes data collection because they see the benefits in doing so. Respondents indicated they see the benefits of participating in outcomes research (mean 4.32). By comparison, only 68% (39 of 57) of total respondents agreed or strongly agreed that food and nutrition professionals need to be involved in outcomes research. Within the interest content area, post-outcomes research graduates had a greater percentage than preoutcomes research graduates of “highly interested” or “somewhat interested” responses to all 11 skill questions;
Table 2. Responses of University of Missouri-Columbia post-outcomes researcha vs other post-outcomes researchb Coordinated Program in Dietetics graduates to questions regarding various research skillsc Knowledged Other Research skill Read and critically evaluate a research article Conduct a comprehensive literature review Write a research proposal Identify research problems Define research objectives Obtain approval from an HSC/ IRBg committee Collect data for research Use statistics to analyze data Present research at conference Write a journal article Collect data for quality improvement
Intereste
UMC
Other
n
%
n
%
n
12
87
15
100
2.01
12
12
87
14
97
0.333
7
47
11
71
11
80
13
12
86
3
Participationf
UMC
Other
Attitudee
UMC
Other
n
%
n
%
n
%
n
87
14
93
0.299
12
87
12
79
0.333
13
11
80
10
71
0.291
6
47
6
43
0.042
1.83
8
60
13
86
2.40
3
20
8
54
86
0.166
12
87
13
86
0.006
7
53
8
13
86
0.006
14
100
13
86
2.30
7
67
20
2
14
0.166
9
67
9
64
0.018
2
12
86
12
79
0.333
14
100
13
86
2.30
7
53
4
29
1.83
10
73
9
64
3
20
6
43
1.79
7
47
13
1
7
1
7
0.003
5
40
9
67
12
79
0.514
13
93
2
%
2
2
%
UMC
2
n
%
93
15
100
0.967
10
71
13
86
0.848
3.48
10
73
15
100
4.33*
64
0.358
13
93
15
100
0.967
8
64
3.16
14
100
15
100
—
13
4
23
0.451
4
27
13
86
10.21**
6
47
7
46
0.001
13
93
15
100
0.967
0.277
8
60
8
57
0.024
12
87
14
93
0.299
86
4.89*
2
13
1
7
0.299
10
73
15
100
4.33*
12
77
3.88*
0
0
1
7
0.292
9
67
12
79
0.514
14
92
0.011
7
53
11
71
1.01
11
80
15
100
3.12
a
Graduates of the University of Missouri-Columbia coordinated program after outcomes research as a component of dietetics’ standards of education and with a two-semester outcomes research curriculum (n⫽12-14). b Graduates from other coordinated programs after outcomes research as a component of dietetics’ standards of education and without a two-semester outcomes research curriculum (n⫽13-15). c The group with the greater percentage value is bold; shaded indicates significantly different at P ⬍.05. d For knowledge, data are shown as number of respondents answering “accomplished” or “adequate”; percentage in parentheses. e For interest, data are shown as number of respondents answering “highly” or “somewhat” interested; percentage in parentheses. f For attitude and participation, data are shown as percentage of respondents answering “yes”; percentage in parentheses. g HSC/IRB⫽Human Subjects Committee/Institutional Research Board. *P⬍0.05. **P⬍0.01.
however, only four were significant: interest in writing a research proposal, defining research objectives, collecting data for research, and collecting data for quality improvement research (Table 1). Matched to other post-outcomes research graduates, University of Missouri-Columbia post-outcomes research graduates showed significantly greater interest in two specific research skills: presenting research at a conference and writing a journal article (Table 2). Taken together, these results suggest that dietetics education’s increased emphasis on research is creating greater interest in the basic steps of research, such as writing a proposal and collecting data. The University of Missouri-Columbia outcomes research curriculum appears to extend this advancement by also creating interest in communicating research findings. This result is encouraging because documenting the effectiveness of dietetics services is paramount to the profession (21).
Some Disparities in CP Graduates’ Attitudes vs Participation in Outcomes Research Post-outcomes research graduates displayed some improvements in attitudes and participation (Table 1). In addition, with few exceptions, graduates’ improved attitudes toward specific research skills generally correlated with improved participation in specific research activities (Table 1). This trend was not observed when comparing the University of Missouri-Columbia post-outcomes research with the other post-outcomes research graduates. Only research skills within the attitude content area showed significant improvement among University of Missouri-Columbia post-outcomes research graduates; there were no significant differences within the participation content area (Table 2). Only 33% of all respondents (19 of 57) reported ever being involved with an outcomes research project. This moderate level of reported outcomes research participa-
January 2008 ● Journal of the AMERICAN DIETETIC ASSOCIATION
123
tion is consistent with findings from a survey of food and nutrition professionals conducted by the Clinical Nutrition Management Dietetic Practice Group (2), in which even clinicians (n⫽153), with a vested interest in assessment of outcomes and evidence-based practice, reported only 27% participation in outcomes research (2). The main limitation of this study is that the effect of either changes in the Standards of Education (11) in general or Universty of Missouri-Columbia’s outcomes research curriculum—specifically on research participation— could not be adequately measured because graduates had been in the workplace for only a few years and, presumably, had not reached positions in which outcomes-based research and/or development of evidencebased protocols are a significant component of their responsibilities. Other content areas (attitude, knowledge, and interest) were used as surrogates to evaluate the curriculum’s potential influence on future participation. Post-outcomes research graduates’ more positive attitude toward research suggests that these graduates may participate in more research in the future. Further, Universty of Missouri-Columbia post-outcomes research graduates demonstrated a markedly significant improvement in attitudes toward two research skills (Table 1) not shown within the attitudes content area in the pre- vs post-outcomes research graduates comparison (Table 1): obtaining approval from Human Subjects Committee/Institutional Review Board and presenting at a research conference. These additional attitudinal improvements in graduates trained via University of Missouri-Columbia’s outcomes research curriculum may offer added potential for increased future participation in research by food and nutrition professionals. More research is needed to confirm and expand on these data. The work described here should be considered a pilot study examining the changes made in dietetics education curricula in recent years and evaluating the influence on students of implementing an outcomes research curriculum. CONCLUSIONS Within the content areas of attitude, knowledge, interest, and participation, post-outcomes research graduates demonstrate improvement over pre-outcomes research graduates. These findings provide support for the effectiveness of the current eligibility requirements and accreditation standards. In light of this study and past curriculum experience, the following points may strengthen research training of future food and nutrition professionals: ●
●
Experience with the application of statistics better prepares graduates for involvement in research than increasing the number of required statistics courses. Students should be intimately involved in data collection, analysis, and interpretation. The ethics of working with human subjects is a necessary component of research training. We suggest using resources offered by an institution’s Institutional Review Board (eg, online training and certification) to
124
January 2008 Volume 108 Number 1
●
help students learn the intricacies of human subject protection. Providing students with opportunities to write a complete manuscript and deliver a scientific seminar can empower graduates with the confidence and experience necessary to be a successful researcher. A valuable reference for publishing nutrition research can be found in a recent series of articles in the Journal, representing efforts by a working group convened within the Board of Editors to support the American Dietetic Association’s research mission (21).
This study was supported, in part, by the Margaret Flynn Research Award, University of Missouri, Columbia, MO. References 1. Fitz P, Winkler MF. Education, research, and practice: Bridging the gap. J Am Diet Assoc. 1989;89:1676-1679. 2. Gardner JK, Rall LC, Peterson CA. Lack of multidisciplinary collaboration is a barrier to outcomes research. J Am Diet Assoc. 2002;102: 65-71. 3. Hays JE, Peterson CA. Use of an outcomes research collaborative training curriculum to enhance entry-level dietitians’ and established professionals’ self-reported understanding of research. J Am Diet Assoc. 2003;103:77-84. 4. Hynak-Hankinson MT, Martin S, Wirth J. Research competencies in the dietetics curricula. J Am Diet Assoc. 1997;97(suppl 2):S102-S106. 5. Monsen ER, Cheney CL. Research methods in nutrition and dietetics: Design, data analysis, and presentation. J Am Diet Assoc. 1988;88: 1047-1065. 6. Monsen ER. Forces for research. J Am Diet Assoc. 1993;93:981-985. 7. Rebovich EJ, Wodarski LA, Hurley RS, Rasor-Greenhalgh S, Stombaugh I. A university-community model for the integration of nutrition research, practice, and education. J Am Diet Assoc. 1994;94:179182. 8. Rinke WJ, Berry MW. Integrating research into clinical practice: A model and call for action. J Am Diet Assoc. 1987;87:159-161. 9. Sims LS, Simko MD. Applying research methods in nutrition and dietetics: Embodiment of the profession’s backbone. J Am Diet Assoc. 1988;88:1045-1046. 10. Wyse BW. The backbone of our profession. J Am Diet Assoc. 1987;87: 1394-1396. 11. Commission on Accreditation/Approval for Dietetics Education. Accreditation/Approval Manual for Dietetics Education. Chicago, IL: The American Dietetic Association; 1997. 12. Commission on Accreditation for Dietetics Education. Accreditation Handbook. Chicago, IL: The American Dietetic Association; 2002. 13. Directory of Dietetics Programs: 2000-2001. Chicago, IL: The American Dietetic Association; 2000. 14. Gilbride JA, Parks SC, Palakurthi RR. Attitudes and opinions of dietetics professionals toward cost-benefit and cost-effectiveness analyses. J Am Diet Assoc. 1994;94:386-389. 15. Neumark-Sztainer D, Story M, Harris T. Beliefs and attitudes about obesity among teachers and school healthcare providers working with adolescents. J Nutr Educ. 1999;31:3-9. 16. Peterson RA. Constructing Effective Questionnaires. Thousand Oaks, CA: Sage; 2000. 17. Miller CK, Achterberg CL. Reliability and validity of a nutrition and food-label knowledge test for women with type 2 diabetes mellitus. J Nutr Educ. 2000;32:43-48. 18. Dillman DA. Mail and Telephone Surveys—The Total Design Method. New York, NY: John Wiley and Sons; 1978. 19. Carson JS, Hedl JJ. Smart Shoppers tours: Outcome evaluation. J Nutr Educ. 1998;30:13-22. 20. Ma FC, Contento IR. Development and formative evaluation of a nutrition education curriculum aimed at reducing fat intake in Taiwan elementary students.J Nutr Educ. 1997;29:237-243. 21. Boushey C, Harris J, Bruemmer B, Archer SL, Van HL. Publishing nutrition research: A review of study design, statistical analyses, and other key elements of manuscript preparation, Part 1. J Am Diet Assoc. 2006;106:89-96.
Short-Term Effectiveness of an Outcomes Research Training Curriculum Within a Coordinated Program CATHERINE A. PETERSON, PhD, RD; JANET E. HAYS-KIMMONS, MS, RD; JAMES S. COLE, PhD
ABSTRACT The fourth edition of the Commission on Accreditation for Dietetics Education Standards of Education mandated outcomes research training. Our objective was to determine the short-term effectiveness (⬍5 years) of the outcomes research training curriculum in the Coordinated Program in Dietetics (CP) at the University of MissouriColumbia, which exceeds these minimum standards. Toward this end, a survey tool was administered to University of Missouri-Columbia CP graduates before the implementation of the fourth edition of Standards of Education and to University of Missouri-Columbia CP graduates with two semesters of outcomes research training; graduates of two other CPs from different universities from the same years were also surveyed. Graduates who went through CPs after implementation of the fourth edition of the Standards of Education showed notable improvement in attitudes, interest, and participation in most of the outcomes research skills queried, compared with those who completed their education under the earlier standards; knowledge was only slightly improved. Of these areas, only enhancements in attitudes and interest were consistently more pronounced in food and nutrition professionals trained using the University of MissouriColumbia’s outcomes research curriculum compared with other CP graduates. Both the Commission on Accreditation for Dietetics Education and program directors can use the findings described herein to further assess and develop the research competencies and/or to strengthen their programs. J Am Diet Assoc. 2008;108:120-124.
C. A. Peterson is an assistant professor, Dietetics Program, Department of Nutritional Sciences, University of Missouri-Columbia, Columbia, MO. J. E. Hays-Kimmons is a clinical dietitian, Boone Hospital, Columbia, MO. J. S. Cole is a research analyst, Center for Postsecondary Research, Indiana University, Bloomington. Address correspondence to: Catherine A. Peterson, PhD, RD, University of Missouri-Columbia, 106 McKee Hall, Columbia, MO 65211. E-mail: petersonca@ missouri.edu Copyright © 2008 by the American Dietetic Association. 0002-8223/08/10801-0004$34.00/0 doi: 10.1016/j.jada.2007.10.009
120
Journal of the AMERICAN DIETETIC ASSOCIATION
I
t is well established that research, including outcomes research, is essential to the dietetics profession and that dietetics education is the starting point for appreciating this relationship (1-10). Toward this end, in 1997 the Commission on Accreditation for Dietetics Education Standards of Education (11) began to include “foundation knowledge and skills” as well as competency statements related to outcomes research and research training. The current 2000 Commission on Accreditation for Dietetics Education Eligibility Requirements and Accreditation Standards retains these requirements (12). The researchrelated foundation knowledge and skills the Commission on Accreditation for Dietetics Education outlined include knowledge of interpretation of current research, needs assessment, basic statistics, and quality improvement, whereas the research-related competency statements include “interpret and incorporate new scientific knowledge . . .” and “develop and measure outcomes for . . . practice.” The outcomes research training curriculum in the Coordinated Program in Dietetics (CP) at the University of Missouri-Columbia surpasses the minimum standards the Commission on Accreditation for Dietetics Education established in Eligibility Requirements and Accreditation Standards by providing dietetics students with supervised practice experience in designing, conducting, and presenting outcomes-based research projects. This unique two-semester outcomes research training curriculum, described in detail in the Journal (3), has for several years used a collaborative team to prepare entry-level food and nutrition professionals to execute a research plan. The objective of this study was to determine the short-term (⬍5 years postgraduation) effectiveness of this outcomes research training curriculum. Our hypothesis was that incorporating an outcomes research training curriculum within the CP would improve attitudes toward, knowledge of, and interest and participation in outcomes research among food and nutrition professionals who complete the curriculum. METHODS Subject Selection The University of Missouri-Columbia dietetics program is a 2-year CP. Subjects from two University of MissouriColumbia graduation classes were selected for participation in this 2001 study: one class graduated before the initiation of the fourth edition of The Standards of Education was published and had no formal training in outcomes research (n⫽17); the other class was enrolled in
© 2008 by the American Dietetic Association
the two-semester research curriculum (n⫽15) (3). Both classes were under the mentorship of the same dietetics faculty except for one professor who introduced the outcomes research curriculum. Graduates of two other CPs from the same 2 years were also included. Using the American Dietetic Association Directory of Dietetics Programs (13), the two CPs were selected based on their similarity to University of Missouri-Columbia’s program using the following criteria: similar class size, minimum grade point average of 2.5 for acceptance, Committee on Dietetics Education Area Two designation, and a medical nutrition therapy/clinical emphasis. Both institutions agreed to participate by providing their alumni mailing lists for the specified years and were designated Program X (n⫽24) or Program Y (n⫽29) to ensure confidentiality. Human subject approval was obtained through the University of Missouri-Columbia Campus Institutional Review Board. Study Design and Survey Tool A survey was used to assess attitudes toward, knowledge of, and interest and participation in outcomes research of CP graduates from the three different programs, before (pre-outcomes research) and after (post-outcomes research) the initiation of the fourth edition of The Standards of Education; as well as to assess the effectiveness of the two-semester outcomes research curriculum at the University of Missouri-Columbia. The 20-item survey was developed by the authors in collaboration with the Assessment Resource Center at University of Missouri-Columbia and adapted from previously developed instruments (4,5,14,15). Content and format were based upon established guidelines (5,16-18). Multiple-choice or dichotomous questions were used to obtain information on education, employment, and research training/experience and involvement; Likert scales, in accordance with Peterson’s guidelines, were used to measure participants’ attitudes, preferences, or confidence levels regarding research/outcomes research as it relates to dietetics in general as well as in their own careers (16). A subset of survey questions was designed to measure each of four content areas (ie, attitudes, knowledge, interest, and participation) related to specific research skills and to allow for cross-comparisons using a technique based on the studies of Carson and Hedl (19) and Ma and Contento (20). Face and content validity were assessed for content, clarity, readability, and comprehensiveness through a pilot test conducted on local practicing registered dietitians (n⫽8). The survey was then slightly modified (wording and question placement) according to feedback. A chance to win (via blinded, random drawing) payment of the registration fee for the American Dietetic Association 2001 Food & Nutrition Conference & Expo was included in the mailing to increase the response rate of eligible graduates. Statistics The Statistical Package for the Social Sciences (version 10.0.5, 1999, SPSS Inc, Chicago, IL) was used for data compilation and statistical analysis. Pearson’s 2 was
used to compare graduation years (pre-outcomes research vs post-outcomes research graduates) and the three programs (ie, University of Missouri-Columbia, Program X, Program Y); as well as to assess responses to individual questions selected a priori within each content area (attitudes, knowledge, interest, and participation). Variable frequencies were computed to describe the data. Because responses between Programs X and Y were not significantly different, data were collapsed; thus, University of Missouri-Columbia post-outcomes research graduates were compared with “other” post-outcomes research graduates to evaluate the effectiveness of University of Missouri-Columbia’s outcomes research curriculum; the other programs were meant to serve as representatives of CP curriculums in general. RESULTS AND DISCUSSION We evaluated the short-term effectiveness of an outcomes research curriculum at University of Missouri-Columbia by attempting to survey a total of 85 food and nutrition professionals from six graduation classes, representing two different editions of the Committee on Dietetics Education education standards and three different CPs. Owing to one missing address and 11 returned surveys marked “undeliverable” by the US Postal Service, there were a total of 73 deliverable surveys. Fifty-seven usable surveys were received; thus, the overall response rate was 78% (57 of 73). There were no significant differences in response rates among the three programs or between years of graduation. The data demonstrate notable improvements in attitudes, interest, and participation in several of the outcomes research and research-related skills of food and nutrition professionals who graduated after the initiation of the fourth edition of The Standards of Education (11); improvements in knowledge were less notable. Among these four content areas, only enhancements in attitude and interest were more pronounced in food and nutrition professionals trained using the University of MissouriColumbia outcomes research curriculum; thus, the data only partially support our hypothesis. Although the main focus of this analysis was to assess the effectiveness of a specific curriculum, the results are also useful in assessing the strengths of the researchrelated foundation skills and competencies required in current dietetics education. CP Graduates Critical of Research Training and Preparedness Fifty-two of 57 respondents stated that the CP from which they graduated “included instruction or training” on the research process, yet only 32 out of 57 believed their educational training “actually prepared” them for participating in research. Ninety-seven percent (28 of 29) of the post-outcomes research graduates vs 82% (23 of 28) of pre-outcomes research graduates reported that the CP from which they graduated included instruction or training on the outcomes research process (P⫽0.176). All postoutcomes research graduates (29 of 29) and nearly all pre-outcomes research graduates (26 of 28) reported learning about outcomes research via coursework as opposed to other modes.
January 2008 ● Journal of the AMERICAN DIETETIC ASSOCIATION
121
Table 1. Responses of pre-outcomes researcha (Pre-OR) vs post-outcomes researchb (Post-OR) Coordinated Program in Dietetics graduates to questions regarding various research skillsc Knowledged
Intereste
Pre-OR Post-OR Research skill Read and critically evaluate a research article Conduct a comprehensive literature review Write a research proposal Identify research problems Define research objectives Obtain approval from an HSC/ IRBg committee Collect data for research Use statistics to analyze data Present research at conference Write a journal article Collect data for quality improvement
Participationf
Pre-OR Post-OR
Attitudee
Pre-OR Post-OR
Pre-OR
Post-OR
n
%
n
%
n
%
n
%
n
%
n
%
n
21
75
27
93
3.51
22
79
26
90
1.32
15
54
24
83
5.62*
27
96
28
97
0.003
18
64
24
83
2.51
16
57
22
76
2.25
11
39
13
45
0.179
21
78
23
79
0.005
10
36
17
59
3.00
12
43
20
72
5.11*
2
7
11
36
6.79**
17
63
25
86
4.03*
19
68
24
83
1.71
20
71
25
86
1.87
8
29
17
59
5.22**
21
78
28
97
4.51*
23
82
25
86
0.177
20
71
27
93
4.63*
19
68
21
72
0.141
27
96
29
100
1.09
6
21
5
17
0.160
12
43
19
66
2.95
1
4
5
18
2.99
16
56
16
55
0.001
19
68
24
83
1.71
18
64
27
93
7.12**
9
32
14
46
1.20
25
89
28
97
1.24
9
32
12
41
0.522
13
46
20
69
2.97
14
50
17
59
0.427
24
85
26
90
0.255
16
57
9
31
3.94*
15
54
19
66
0.845
5
18
3
10
0.667
21
74
25
86
1.30
7
25
2
7
3.51
13
46
16
57
0.644
2
7
1
3
0.380
19
67
21
72
0.218
19
68
21
72
0.141
20
70
27
93
4.67*
18
63
18
62
0.005
28
100
26
90
2.95
2
2
2
%
n
%
2
a
Graduates from coordinated programs before outcomes research became a component of dietetics’ standards of education (n⫽25-28). Graduates from coordinated programs after outcomes research became a component of dietetics’ standards of education (n⫽26-29). The group with the greater percentage value is bold; shaded indicates significantly different at P⬍0.05. d Respondents answering “accomplished” or “adequate.” e Respondents answering “highly” or “somewhat” interested. f Respondents answering “yes.” g HSC/IRB⫽ Human Subjects Committee/Institutional Research Board. *P⬍0.05. **P⬍0.01. b c
Our results show that the majority of CP graduates believe they are at least “adequately” trained for most of the specific research skills discussed (Tables 1 and 2). However, graduates reported deficiencies (ie, percentage “accomplished” or “adequate” responses ⬍50%) in knowledge of obtaining approval from a Human Subjects Committee/Institutional Review Board, writing a journal article, and using statistics to analyze data. This latter finding is surprising given that all respondents reported completing at least one statistics course, with one third completing two. Remarkably, more post-outcomes research graduates reported being “in need of more training” in the above three skills areas than pre-outcomes research graduates (Table 1), perhaps because pre-outcomes research graduates believe they have gained adequate research experience in the workplace or graduate school, or because
122
January 2008 Volume 108 Number 1
post-outcomes research graduates are more aware of the increasing emphasis on research within the profession. Improvement in CP Graduates’ Interest in Research Using a scale from one to five, where 1⫽strongly disagree and 5⫽strongly agree, respondents were asked to rate their interest level in participating in outcomes data collection because they see the benefits in doing so. Respondents indicated they see the benefits of participating in outcomes research (mean 4.32). By comparison, only 68% (39 of 57) of total respondents agreed or strongly agreed that food and nutrition professionals need to be involved in outcomes research. Within the interest content area, post-outcomes research graduates had a greater percentage than preoutcomes research graduates of “highly interested” or “somewhat interested” responses to all 11 skill questions;
Table 2. Responses of University of Missouri-Columbia post-outcomes researcha vs other post-outcomes researchb Coordinated Program in Dietetics graduates to questions regarding various research skillsc Knowledged Other Research skill Read and critically evaluate a research article Conduct a comprehensive literature review Write a research proposal Identify research problems Define research objectives Obtain approval from an HSC/ IRBg committee Collect data for research Use statistics to analyze data Present research at conference Write a journal article Collect data for quality improvement
Intereste
UMC
Other
n
%
n
%
n
12
87
15
100
2.01
12
12
87
14
97
0.333
7
47
11
71
11
80
13
12
86
3
Participationf
UMC
Other
Attitudee
UMC
Other
n
%
n
%
n
%
n
87
14
93
0.299
12
87
12
79
0.333
13
11
80
10
71
0.291
6
47
6
43
0.042
1.83
8
60
13
86
2.40
3
20
8
54
86
0.166
12
87
13
86
0.006
7
53
8
13
86
0.006
14
100
13
86
2.30
7
67
20
2
14
0.166
9
67
9
64
0.018
2
12
86
12
79
0.333
14
100
13
86
2.30
7
53
4
29
1.83
10
73
9
64
3
20
6
43
1.79
7
47
13
1
7
1
7
0.003
5
40
9
67
12
79
0.514
13
93
2
%
2
2
%
UMC
2
n
%
93
15
100
0.967
10
71
13
86
0.848
3.48
10
73
15
100
4.33*
64
0.358
13
93
15
100
0.967
8
64
3.16
14
100
15
100
—
13
4
23
0.451
4
27
13
86
10.21**
6
47
7
46
0.001
13
93
15
100
0.967
0.277
8
60
8
57
0.024
12
87
14
93
0.299
86
4.89*
2
13
1
7
0.299
10
73
15
100
4.33*
12
77
3.88*
0
0
1
7
0.292
9
67
12
79
0.514
14
92
0.011
7
53
11
71
1.01
11
80
15
100
3.12
a
Graduates of the University of Missouri-Columbia coordinated program after outcomes research as a component of dietetics’ standards of education and with a two-semester outcomes research curriculum (n⫽12-14). b Graduates from other coordinated programs after outcomes research as a component of dietetics’ standards of education and without a two-semester outcomes research curriculum (n⫽13-15). c The group with the greater percentage value is bold; shaded indicates significantly different at P ⬍.05. d For knowledge, data are shown as number of respondents answering “accomplished” or “adequate”; percentage in parentheses. e For interest, data are shown as number of respondents answering “highly” or “somewhat” interested; percentage in parentheses. f For attitude and participation, data are shown as percentage of respondents answering “yes”; percentage in parentheses. g HSC/IRB⫽Human Subjects Committee/Institutional Research Board. *P⬍0.05. **P⬍0.01.
however, only four were significant: interest in writing a research proposal, defining research objectives, collecting data for research, and collecting data for quality improvement research (Table 1). Matched to other post-outcomes research graduates, University of Missouri-Columbia post-outcomes research graduates showed significantly greater interest in two specific research skills: presenting research at a conference and writing a journal article (Table 2). Taken together, these results suggest that dietetics education’s increased emphasis on research is creating greater interest in the basic steps of research, such as writing a proposal and collecting data. The University of Missouri-Columbia outcomes research curriculum appears to extend this advancement by also creating interest in communicating research findings. This result is encouraging because documenting the effectiveness of dietetics services is paramount to the profession (21).
Some Disparities in CP Graduates’ Attitudes vs Participation in Outcomes Research Post-outcomes research graduates displayed some improvements in attitudes and participation (Table 1). In addition, with few exceptions, graduates’ improved attitudes toward specific research skills generally correlated with improved participation in specific research activities (Table 1). This trend was not observed when comparing the University of Missouri-Columbia post-outcomes research with the other post-outcomes research graduates. Only research skills within the attitude content area showed significant improvement among University of Missouri-Columbia post-outcomes research graduates; there were no significant differences within the participation content area (Table 2). Only 33% of all respondents (19 of 57) reported ever being involved with an outcomes research project. This moderate level of reported outcomes research participa-
January 2008 ● Journal of the AMERICAN DIETETIC ASSOCIATION
123
tion is consistent with findings from a survey of food and nutrition professionals conducted by the Clinical Nutrition Management Dietetic Practice Group (2), in which even clinicians (n⫽153), with a vested interest in assessment of outcomes and evidence-based practice, reported only 27% participation in outcomes research (2). The main limitation of this study is that the effect of either changes in the Standards of Education (11) in general or Universty of Missouri-Columbia’s outcomes research curriculum—specifically on research participation— could not be adequately measured because graduates had been in the workplace for only a few years and, presumably, had not reached positions in which outcomes-based research and/or development of evidencebased protocols are a significant component of their responsibilities. Other content areas (attitude, knowledge, and interest) were used as surrogates to evaluate the curriculum’s potential influence on future participation. Post-outcomes research graduates’ more positive attitude toward research suggests that these graduates may participate in more research in the future. Further, Universty of Missouri-Columbia post-outcomes research graduates demonstrated a markedly significant improvement in attitudes toward two research skills (Table 1) not shown within the attitudes content area in the pre- vs post-outcomes research graduates comparison (Table 1): obtaining approval from Human Subjects Committee/Institutional Review Board and presenting at a research conference. These additional attitudinal improvements in graduates trained via University of Missouri-Columbia’s outcomes research curriculum may offer added potential for increased future participation in research by food and nutrition professionals. More research is needed to confirm and expand on these data. The work described here should be considered a pilot study examining the changes made in dietetics education curricula in recent years and evaluating the influence on students of implementing an outcomes research curriculum. CONCLUSIONS Within the content areas of attitude, knowledge, interest, and participation, post-outcomes research graduates demonstrate improvement over pre-outcomes research graduates. These findings provide support for the effectiveness of the current eligibility requirements and accreditation standards. In light of this study and past curriculum experience, the following points may strengthen research training of future food and nutrition professionals: ●
●
Experience with the application of statistics better prepares graduates for involvement in research than increasing the number of required statistics courses. Students should be intimately involved in data collection, analysis, and interpretation. The ethics of working with human subjects is a necessary component of research training. We suggest using resources offered by an institution’s Institutional Review Board (eg, online training and certification) to
124
January 2008 Volume 108 Number 1
●
help students learn the intricacies of human subject protection. Providing students with opportunities to write a complete manuscript and deliver a scientific seminar can empower graduates with the confidence and experience necessary to be a successful researcher. A valuable reference for publishing nutrition research can be found in a recent series of articles in the Journal, representing efforts by a working group convened within the Board of Editors to support the American Dietetic Association’s research mission (21).
This study was supported, in part, by the Margaret Flynn Research Award, University of Missouri, Columbia, MO. References 1. Fitz P, Winkler MF. Education, research, and practice: Bridging the gap. J Am Diet Assoc. 1989;89:1676-1679. 2. Gardner JK, Rall LC, Peterson CA. Lack of multidisciplinary collaboration is a barrier to outcomes research. J Am Diet Assoc. 2002;102: 65-71. 3. Hays JE, Peterson CA. Use of an outcomes research collaborative training curriculum to enhance entry-level dietitians’ and established professionals’ self-reported understanding of research. J Am Diet Assoc. 2003;103:77-84. 4. Hynak-Hankinson MT, Martin S, Wirth J. Research competencies in the dietetics curricula. J Am Diet Assoc. 1997;97(suppl 2):S102-S106. 5. Monsen ER, Cheney CL. Research methods in nutrition and dietetics: Design, data analysis, and presentation. J Am Diet Assoc. 1988;88: 1047-1065. 6. Monsen ER. Forces for research. J Am Diet Assoc. 1993;93:981-985. 7. Rebovich EJ, Wodarski LA, Hurley RS, Rasor-Greenhalgh S, Stombaugh I. A university-community model for the integration of nutrition research, practice, and education. J Am Diet Assoc. 1994;94:179182. 8. Rinke WJ, Berry MW. Integrating research into clinical practice: A model and call for action. J Am Diet Assoc. 1987;87:159-161. 9. Sims LS, Simko MD. Applying research methods in nutrition and dietetics: Embodiment of the profession’s backbone. J Am Diet Assoc. 1988;88:1045-1046. 10. Wyse BW. The backbone of our profession. J Am Diet Assoc. 1987;87: 1394-1396. 11. Commission on Accreditation/Approval for Dietetics Education. Accreditation/Approval Manual for Dietetics Education. Chicago, IL: The American Dietetic Association; 1997. 12. Commission on Accreditation for Dietetics Education. Accreditation Handbook. Chicago, IL: The American Dietetic Association; 2002. 13. Directory of Dietetics Programs: 2000-2001. Chicago, IL: The American Dietetic Association; 2000. 14. Gilbride JA, Parks SC, Palakurthi RR. Attitudes and opinions of dietetics professionals toward cost-benefit and cost-effectiveness analyses. J Am Diet Assoc. 1994;94:386-389. 15. Neumark-Sztainer D, Story M, Harris T. Beliefs and attitudes about obesity among teachers and school healthcare providers working with adolescents. J Nutr Educ. 1999;31:3-9. 16. Peterson RA. Constructing Effective Questionnaires. Thousand Oaks, CA: Sage; 2000. 17. Miller CK, Achterberg CL. Reliability and validity of a nutrition and food-label knowledge test for women with type 2 diabetes mellitus. J Nutr Educ. 2000;32:43-48. 18. Dillman DA. Mail and Telephone Surveys—The Total Design Method. New York, NY: John Wiley and Sons; 1978. 19. Carson JS, Hedl JJ. Smart Shoppers tours: Outcome evaluation. J Nutr Educ. 1998;30:13-22. 20. Ma FC, Contento IR. Development and formative evaluation of a nutrition education curriculum aimed at reducing fat intake in Taiwan elementary students.J Nutr Educ. 1997;29:237-243. 21. Boushey C, Harris J, Bruemmer B, Archer SL, Van HL. Publishing nutrition research: A review of study design, statistical analyses, and other key elements of manuscript preparation, Part 1. J Am Diet Assoc. 2006;106:89-96.