Reliability and Validity of Nutrition Knowledge, Social-Psychological Factors, and Food Label Use Scales from the 1995 Diet and Health Knowledge Survey

R E S E A R C H A RT I C L E

Reliability and Validity of Nutrition Knowledge, Social-Psychological Factors, and Food Label Use Scales from the 1995 Diet and Health Knowledge Survey S AO R I O B AYA S H I , M S, R D ; 1 L E O N A R D J. B I A N C H I , P H D ; 2 W O N O. S O N G , P H D, M P H , R D 1 1

Department of Food Science and Human Nutrition, 2Department of Counseling, Educational Psychology and Special Education, Michigan State University, East Lansing, Michigan

INTRODUCTION

ABSTRACT

Understanding factors associated with dietary behaviors helps nutr ition educators formulate educational theories about dietary behaviors and design effective nutrition education progr a m s .1 , 2 M a ny studies re p o rted that dietary behaviors are influenced by factors such as demographics,3-6 lifestyle and health factors,7,8 nutrition knowledge,3,9-14 and social-psychological factors.3,9,13-19 Because nutrition intervention programs are more likely to be effective when they a re theory based, studies have attempted to use socialpsychological theories to explain how social-psychological factors influence dietary behaviors.4-6,8,9,14-18 For instance, Contento and Murphy used the Health Belief Model to differentiate those who made desirable changes in their diets from those who did not. 5 Shepherd and Stockley used the Fishbein and Ajzen attitude model to explain variability in individuals’ nutrition knowledge, attitudes, and fat consumption.9 However, these results were not always consistent. Some issues that may explain inconsistent results across these studies are as follows: (1) the measurements used differed across studies20; (2) the number of constructs included in these studies varied or some of the constructs in different theories overlapped,21,22 suggesting that more than one theory should be incorporated to explain complex dietary behaviors23; (3) studies used small numbers or certain types of subjects (eg, graduate students and undergraduate students),15 and the results cannot be generalized for other population groups; and (4) the reliability and validity of meas u rements we re not identified, suggesting possibl e attenuation of the results.20 Use of measurements with unknown reliability and validity would make it impossible for study investigators to know whether their results are, indeed, true or biased by their measurements.Although there are many different ways of testing the reliability and validity of measurements (ie, test–retest,Cronbach α, content validity, construct validity), some studies in nutrition education did not test their meas u rements prior to use. 2 0 , 2 4 , 2 5 These different tests are equally important, however, to ensure the reliability and validity of the measurement.

Objective: To test the reliability and validity of scales on nutrition knowledge, social-psychological factors, and use of food labels developed from the 1995 Diet and Health Knowledge Survey (DHKS) questions. Design: The 1995 DHKS questions within a section were pooled together as a scale and their reliability and validity were examined. Participants: US adults (≥ 20 years) in the 1995 DHKS who responded to questions selected for this study (n = 1196). Variables: Nutrition knowledge about the diet-disease relationship and nutrient content of products, perceived barriers and benefits of food labels, perceived ease of understanding food labels, food label use, and importance of healthful eating. Analysis: Scales validity, Cronbach α, item total correlation, α if the item was deleted, and discriminant,convergence, and correspondence validity. Results: Scales on perceived ease of understanding the food label,benefits of using food labels, food label use, and importance of healthful eating were reliable (Cronbach α = .78, . 8 2 ,. 9 1 , and .82, respectively) and valid. Conclusion and Implica tions: Accurate findings and interpretation of survey data depend on the use of reliable and valid instruments. This study identified the scales in the DHKS that can substantiate the conclusion on which effective nutrition education strategies should be established. KEY WORDS: Diet and Health Knowledge Survey, reliability, validity, nutrition labeling (J Nutr Educ Behav. 2003;35:83-92.)

Address for correspondence: Won O. S o n g ,P h D, MPH, RD, Department of Food Science and Human Nutrition, College of Human Ecology, Michigan State University, East Lansing, MI 48824-1224; Tel: (517) 355-7690; Fax: (517) 432-7753; E - m a i l :s o n g @ m s u . e d u . ©2003 SOCIETY FOR NUTRITION EDUCATION

83

84

Obayashi et al/RELIABILITY AND VALIDITY OF SELECTED SCALES FROM THE DHKS

A larger, population-representative sample not only is beneficial for its statistical power, it also allows the generalizability of re s u l t s .T h u s ,d eveloping reliable and valid measurements for a larger, more population-representative sample is critical for studying dietary behaviors and nutrition policy making. The Continuing Survey of Food Intakes by Individuals (CSFII) and its follow-up, the Diet and Health Knowledge Survey (DHKS), have been conducted by the US Department of A gri c u l t u re (USDA) as a part of the National Nutrition Monitoring and Related Research Program.26 The CSFII/DHKS have collected data on dietary intakes, nutrition knowledge, and social-psychological factors of the US population.The results of the CSFII, for example, have been used for the development of several national nutrition objectives, such as Healthy People 2000. Behavioral and social-psychological questions in these surveys a l l ow re s e a rc h e rs to explore theorized re l a t i o n s h i p s b e t ween these f a c t o rs influencing dietary behav i o rs . 2 7 Another advantage of the CSFII is the Healthy Eating Index (HEI), which was developed by the Center for Nutrition Policy and Promotion (CNPP) in the USDA based on the CSFII results, with the purpose of characterizing the overall diet quality of these part i c i p a n t s . Researchers traditionally evaluated food or nutrient intakes s e p a r a t e l y, failing to evaluate va r ious aspects of dietary i n t a ke simu l t a n e o u s l y. T h u s , use of the CSFII/DHKS allows us to understand the relationship between va ri o u s factors and healthful eating. Finally, changes in these variables over the years can be monitored as long as the survey items are compatible across the ye a rs . Because of its large, population-representative sample size with a wide range of nu t ri t i o n - related questions, m a ny re s e a rc h e rs have used these surveys in their studies. 12,28,29 Few studies have reported the reliability and validity of questions in these surveys. Sapp and Jensen reported that reliability estimates of nutrition knowledge and diet-health awareness questions in the DHKS (1989-1991) ranged from modest to high.The correspondence validity of these questions with dietary intakes was low, whereas the discriminant and convergent validity of these questions were confirmed. On the other hand, this study did not examine the reliability and validity of social-psychological questions or food label–related questions.The 1990 Nutrition Labeling Education Act (NLEA; effective May 1994) has promoted food labels as a national nutrition education tool for consumers to make more healthful food choices. 31,32 Thus, to examine the effectiveness of food labels on nutrition knowledge, attitudes, and dietary intake of the US population is important. Since 1994, studies have evaluated consumers ’k n ow ledge, attitudes, and use of the food label and their relationship with dietary intakes.33-36 However, only a few studies examined a nationally representative population.33 In addition, the reliability and validity of these measurements were not always stated.

The original 1995 DHKS separated 144 questions (ie, nutrition knowledge, social-psychological fa c t o rs , use of food labels) into 42 sections. Questions within the same section shared the same response form a t . H oweve r, it is unknown whether these questions within the same section are intercorrelated each other, and if they are, what characteristics the questions were measuring.Social-psychological questions in the DHKS were developed based on socialpsychological theories.37 Colavito and Guthrie stated that questions on food labels were developed from the Roger’s Model of Diffusion of Innovation and questions on pesticides were developed from the Health Belief Model.37 However, little documentation is available regarding other DHKS questions in relation to other social-psychological theories. Thus, the objective of our study was to test the reliability and validity of nutrition knowledge, social-psychological factors (as a component of various social-psychological theories), and food label use measures developed from the 1995 DHKS questions in relation to social-psychological theori e s .T h e 1995 DHKS was chosen instead of the DHKS 1994-1996 combined data set because we plan to use constructs developed for future studies to evaluate the immediate impacts of the enactment of the NLEA on consumers’ nutrition knowle d g e, nu t ri t i o n - related social-psychological fa c t o rs , a n d dietary behaviors.

METHODS Data and Subjects Data were collected as part of the CSFII/DHKS 1995, collected from mid-January of 1995 to mid-January of 1996.A stratified, multistage area probability sampling method was used. The population was stratified based on geographic location,degree of urbanization, and socioeconomic considerations. The low-income population was oversampled to generate reliable inferences of the US low-income population based on adequate sample size.This complex sampling method allows investigators to obtain a nationally representative sample of noninstitutionalized people residing in the United States more efficiently and for less cost than simpler methods. Sampling weights should be applied in statistical analyses to generate population-representative estimates to adjust for oversampling, va ri a ble probabilities of selections, and response rates. In addition, specialized techniques and statistical software such as SUDAAN and WesVar PC should be used to estimate variance and to conduct inferential statistics accurately.26 In CSFII 1995, 2 nonconsecutive days of dietary intake information for individuals of all ages were collected by trained interv i ewe rs using the 24-hour dietary re c a l l method. The Center for Nutrition Policy and Promotion (CNPP) in the USDA released the HEI data set, which was developed from the CSFII 1995, to evaluate food intake in

Journal of Nutrition Education and Behavior Volume 35 Number 2

terms of overall diet quality. 38 The HEI total score represents the overall diet quality by summing 10 subscores of dietary intake (total fat, saturated fa t ,c h o l e s t e ro l ,s o d i u m , 5 food groups of the Food Guide Pyramid, and variety in their diets), a healthful diet that would reduce the risk of chronic diseases.39 We decided to use a total HEI score as well as HEI subscores to characterize individual’s dietary intake because (1) some DHKS questions are disease or nutrient specific and others are related to a healthful or nutritious diet; (2) the goal for the nutrition community is improving not only certain food/nutrient intake but also the overall diet quality of the US population; and (3) it is important to develop reliable and valid constructs on nutrition knowledge, social-psychological factors, and the use of food labels associated with the overall diet quality. The DHKS 1995, a follow-up of the 1995 CSFII, interviewed subjects regarding their nutrition knowledge (42 questions),social-psychological factors (ie, belief, attitudes: 52 questions), and use of food labels (23 questions). Only one adult was selected per household for the DHKS. We included subjects who participated on the CSFII/DHKS aged 20 years and older (n = 1966).After excluding subjects with missing data, 1196 participants were included (61% of the original). Development of Scales As described earlier, the original DHKS separated question items into sections. Questions within the same section were pooled together as a scale measure after the following procedures: (1) the content (domain) of questions within each section was examined by content validity (if more than one domain was identified within a section, more than one measure was developed from the same section); (2) the reliability of scales developed from these questions was examined by Cronbach α, item total correlation (ITC), and α if the item was deleted; (3) the discriminant validity of measures was examined based on the information from prior studies; (4) the convergence validity was determined based on the information of prior studies; and (5) correspondent validity was examined based on the information from prior studies. Detailed information on these procedures follows. Scoring system. A score for each scale was obtained by summing the responses to questions within each scale. For example, there were 7 questions inquiring about diet-disease relationships.The respondent received 2 points for a correct answer and 1 point for an incorrect answer (the response choice [2 or 1] was used instead of 1 and 0 because it is required for SUDAAN operation). 40 Thus, the score of this construct would range from 7 (not knowledgeable) to 14 (very knowledgeable). Similarly, the scores of constructs on social-psychological factors and the use of food labels were obtained by summing the scales (eg, 4 = strongly agree to 1 = strongly disagree) for survey items.

March • April 2003

85

Reliability Tests Reliability is a measurement of the extent to which a test yields the same result on repeated administration when all other fa c t o rs (ie, subjects and test timing) are held the same.20,30 A method of internal consistency used to assess the reliability of questions we pooled together as a construct was Cronbach α. Cronbach α ranges from r = 0 to 1, with r = .7 or greater considered as sufficiently reliable.30,41 Methods such as ITC and α if the item was deleted were used to measure the reliability of the item. ITC assessed the extent of the relationship between each question item with the rest of the items within the same construct.42 It is a measurement of how closely each item relates to the overall score. Differences in α va l u e s ,m e a s u red with and without potential score survey elements, reflect differences in score reliability and are used to decide which items should be included or excluded to obtain desirable reliability.42 Validity Tests Testing the validity of questions or constructs ensures the ability of questions or constructs to measure the characteristic purported to be measured.24,30 Content,discriminant,convergent, and correspondence validity were tested in this study. Content validity. Content validity measures the extent to which the items within a construct represent the domain of the characteristics,30 such as nutrition knowledge on diet-disease re l a t i o n s h i p s . It is, in general, d e t e rm i n e d through the agreement of experts in the field. The review panel consisted of the principal investigator, 5 graduate students, and 3 faculty members in the Department of Food Science and Human Nutrition at Michigan State University, all of whom have nutrition and public health expertise. They individually reviewed to determine if questions represent various aspects that could make up the domain under i nve s t i g a t i o n . For example, food labels contain a list of ingredients, nutrition panel, health claims, and short nutrient phrases. The DHKS 1995 questions were reviewed to evaluate if these aspects were included to make up a construct on food labels. References, including past studies, nutrition textbooks, and social-psychological textbooks, were used during this process.3-16,19-23,30,31,35,43-49 In addition, interpretability was assessed if there was ambiguous wording in questions. A number of questions were discarded until there was agreement. Discriminant validity. Discriminant validity examines the ability of a measure to discriminate between groups of respondents based on the information from prior studies. 30 Theories were established based on the review of literature. For example, studies reported that the factors associated with having better nu t rition knowledge are being female,16,30 being highly educated, 30 having health problems such as high blood cholesterol,30 being on a choles-

86

Obayashi et al/RELIABILITY AND VALIDITY OF SELECTED SCALES FROM THE DHKS

terol-lowering diet,30 and being at a higher income level.30 Factors identified in relation to positive attitudes toward healthful dietary behaviors were being female,50 h av i n g higher income, 10 and being older.10 Factors reported to be associated with perceived benefits of using food labels were being female,36 being older,36 following a normal or low - fa t diet,36 having elevated blood cholesterol level,36 and having elevated blood pressure.36 Perceived ease of understanding the food label was reported to be associated with higher educational background and younger subjects. 36 Factors reported to be associated with perceived barriers to using food labels were being older and being less educated. 48 Identified fa c t o rs associated with the use of food label i n f o rmation we re being female,3 5 , 3 6 being highly educated,35 being younger than 35 years old,35 and following a normal or low-fat diet.36 C o nve r gence va l i d i t y. C o nvergent validity expects moderate correlation between related constructs because these constructs measure theoretically related characteristics.30 Previously, the correlation between nutrition knowledge about diet-disease relationships and nutrient content between products was reported (.44, .50, and .44 for 1989, 1990, and 1991, respectively).30 Thus, this relationship was examined in our study.

pooled correlation was used to test the convergent and correspondence validity using the SAS Version 8.These procedures were necessary to develop constructs that can be used for the US representative population. RESULTS Characteristics of the Subjects The majority of respondents in the weighted sample were 20 to 50 years old (68%), were not impoverished (87%), did not have high blood cholesterol (85%), and had more than a high school education (56%) (Table 1). Reliability Tests Cronbach α’s for the scales for perceived benefit of using food labels, perceived ease of understanding food labels, perceived benefits of using food labels, and use of food labels were higher than .7 (.82, . 7 8 ,. 8 2 , and .91, respectively; Table 2). The rest of the Cronbach α’s were less than .70.

Table 1.

C o rrespondence va l i d i t y. C o r respondence va l i d i t y examined whether constructs correlate significantly with theoretically related behaviors.30 Five sets of correlation, which were reported in prior studies, were examined: (1) the correlation between nutrition knowledge about dietdisease relationships and the HEI, 5 1 (2) the corre l a t i o n between nutrition knowledge on nutrient content between products and the HEI,51 (3) the correlation between the construct on perceived benefits of using the food label and food label use,52 (4) the correlation between the construct on use of food labels and the HEI, 33 (5) the correlation b e t ween the construct on use of food labels and fa t c o n s u m p t i o n .6 Statistical Analyses SUDAAN was used to do the statistical analyses to take into account the study design effects (ie, cluster sample). 40 All statistical analyses used the sampling weights provided in the 1995 DHKS to generalize our results to the US population. To test the reliability of constructs and items, the pooled correlation matrix for the questions and the variance were obtained using SAS Version 8 and SUDAAN 8.0.0, respectively.53 The pooled correlation matri x ,s t a n d a rd deviation, mean, and weighted sample size were transported into the SPSS 7.5 to obtain ITC, α if the item was deleted, and Cronbach α.42 The discriminant validity test was conducted through the t test and the Wald F test using SUDAAN 8.0.0. The

Characteristics of the Subjects*

Variable

Unweighted Sample (n)

Weighted Sample (%)

Gender Males

538

44.5

Females

658

55.5

20-34 yr

198

32.4

35-50 yr

345

35.4

51-64 yr

362

19.4

65+ yr

291

12.7

< High school

207

10.8

High school graduate

406

32.8

> High school

583

56.4

≤ 130% PT

237

13.1

> 130% PT

959

86.9

Age

Education

Income

High blood cholesterol Yes

232

15.1

No

964

84.9

Low-fat diet Yes

137

9.3

No

1059

90.7

High blood pressure Yes

125

6.1

No

1071

93.9

*n = 1196 (raw); n = 122,957,735 (weighted). PT indicates poverty threshold.

Journal of Nutrition Education and Behavior Volume 35 Number 2 Table 2.

March • April 2003

87

Summary of the Results of Reliability Tests for Knowledge, Social-Psychological, and Behavior Measures in the DHKS 1995* Cronbach

Characteristics

ITC

Deleted †

Knowledge of nutrient contents between products§

‡

.46

Cronbach ITC

Deleted †

The list of ingredients

.39

.78

Characteristics Perceived ease of understanding food labels

.78

Which has more saturated fat: liver or T-bone steak?

.15

.45

Which has more saturated fat: butter or margar ine?

A short phrase such as “low fat” or “light” or “good source of fiber”

.42

.77

.07

.48

The number of calories in a serving

.57

.74

Which has more saturated fat: egg white or egg yolk?

.21

.43

The number of calories from fat in a ser ving

.65

.72

.44

The number of g or mg of nutrients such as fat and sodium/serving

.63

.73

.42

The percentage of the daily value for each n utrient

.58

.74

A description such as “lean” or “extra lean” on meats

.33

.79

Which has more saturated fat: skim milk or whole milk? Which has more fat: regular hamburger or ground round?

.21 .22

Which has more fat: loin pork chops or pork spare r ibs?

.26

.41

Which has more fat: hot dogs or ham?

.17

.44

Which has more fat: peanuts or popcorn?

Perceived importance of practicing healthy dietary habits

.24

.41

Use salt or sodium only in moderation

.43

.82

Which has more fat: yogurt or sour cream?

.18

.44

Choose a diet low in saturated fat

.66

.79

Which has more fat: porterhouse steak or round steak?

.2

.43

Knowledge of diet-disease relationships Eating too much fat causes health problems Not eating enough fiber causes health problems Eating too much salt or sodium causes health problems Not eating enough calcium causes health problems

.65 .35 .47 .39 .51

.62

.82

Choose a diet with plenty of fruits and vegetables

.62

.8

Use sugars only in moderation

.46

.81

Choose a diet with adequate fiber

.63

.8

Eat a variety of f oods

.46

.81

Maintain a healthy weight

.57

.8

Choose a diet low in fat

.64

.8

.57

Choose a diet low in cholesterol

.56

.8

.61

Choose a diet with plenty of breads, cereals, rice, and pasta

.37

.82

.55

Eat at least 2 servings of dairy products daily

.28

.83

The list of ingredients

.54

.91

Eating too much cholesterol causes health problems

.36

.62

Eating too much sugar causes health problems

.24

.66

Being ov erweight causes health problems

The short phrases such as “low fat” or “light” or “good source of fiber”

.49

.91

.36

.62

The nutrition panel that tells the amount of calories, protein, fat, etc in a serving of the food

.69

.9

The information about the size of a s e rv i n g

.49

.91

A statement that describes how nutrients or foods and health problems are related

.59

.91

Perceived barriers to using the food label The nutrition inf ormation on food labels is hard to interpret Reading food labels takes more time than I can spend

.33 .33

NA NA

Perceived benefits of using the food label The nutrition information on food labels is useful to me Reading food labels makes it easier to choose foods

Use of food labels

.48

.82 .66 .59

.77 .8

.91

Information about calories

.68

.91

Information about salt or sodium

.66

.91

Information about total fat

.77

.9

Information about saturated fat

.78

.9

Information about cholesterol

.71

.9

.59

.91

When I use food labels, I make better food choices

.71

.74

Information about vitamins or minerals

Using food labels to choose foods is better than just relying on my own knowledge about what is in them

Information about fiber

.68

.91

.62

.79

Information about sugars

.66

.91

*Weight, v ariance estimation unit, and variance estimation stratum provided in the 1995 DHKS were used. † Alpha if item deleted. ‡ r ≥ 0.7 considered sufficiently reliable. § Underlined ones are the correct responses. DHKS indicates Diet and Health Knowledge Survey; ITC = item total correlation.

‡

88

Obayashi et al/RELIABILITY AND VALIDITY OF SELECTED SCALES FROM THE DHKS

Validity Tests Content validity. Questions agreed on by fewer than 3 expert reviewers were excluded. Seven domains were identified from the DHKS 1995 questions. Two nu t ri t i o n knowledge domains, nutrient content between products and diet-disease relationships, were treated as behavioral capability constructs from the Social Cognitive Theory.54 The four social-psychological domains were perceived barriers to using food labels from the Health Belief Model,54 a decisional balance construct from the Tr a n s t h e o re t i c a l Model,54 or a complexity construct from the Diffusion of Innovations Theory54; perceived benefits from the Health Belief Model54; perceived ease of understanding food labels as a proxy of a self-efficacy construct from the Transtheoretical Model and the Social Cog n i t ive T h e o ry5 4 ; a n d importance of a healthful diet as an evaluation construct from the Theory of Reasoned Action.54 One behavior construct was the use of food labels. D i s c riminant va l i d i t y. Although there we re some exceptions, 70% of our results corresponded to the findings from previous studies (our 19 findings matched with 24 findings reported in prior studies; see discriminant validity under the Method section) (Table 3). For example, as Sapp and Jensen reported, a statistically significant difference in the knowledge construct on diet-disease relationships was found between females and males. 30 Females scored higher than males (13.17 ± 2.77 versus 12.88 ± 2.77 at P = .03; effect size = .2).As expected, those who were impove ri s h e d were less knowledgeable about nutrient content between p roducts than those who we re not impove ri s h e d (16.59 ± 7.26 versus 17.48 ± 2.77 at P < .001; effect size = .3).30 They were also less knowledgeable about diet-disease relationships than those who were not impoverished (12.37 ± 6.57 versus 13.14 ± 1.73 at P < .001; effect size = .4). As anticipated, individuals with a higher education background found the information on the food label easier to understand than those with less education, 13.32 ± 12.79 versus 15.33 ± 7.26 at P < .001; effect size = .3).36 As expected, older subjects were more likely to agree with the importance of healthful dietary behaviors than younger subjects (eg, 20 to 34 year old versus over 65 year old; 36.19 ± 15.91 versus 39.03 ± 12.1, respectively; effect size = .4).10 As expected, females used food labels more frequently than males (40.02 ± 14.52 versus 36.45 ± 17.98 at P < .001; effect size = .3).36 As expected, those who had an elevated blood cholesterol level perceived greater benefits of using food labels than those who had a normal blood cholesterol level 13.36 ± 6.22 versus 12.62 ± 5.19 at P = .001; effect size = .2).36 On the other hand, unlike studies reported,16,30 there was no statistically significant difference in the knowledge construct on the nutrient content between products in our

study. Unexpectedly, no statistically significant difference was seen in perceived benefits of using the food label between those who had high blood pressure and those who had normal blood pressure.36 Another unexpected result of our study was a statistically insignificant difference in food label use by education level.36 C o nve r gent va l i d i t y. The correlation between the nutrition knowledge construct on diet-disease relationships and the nutrient content between products was significant (r = .2 at P < .0001). C o rrespondence va l i d i t y. The correlation betwe e n nutrition knowledge about diet-disease relationships and the total HEI was 0.2 at P < .0001 (data not shown).The correlation between nutrition knowledge about the nutrient content between products and the total HEI was .1 at P < .001. For the construct on perceived benefits of using the food label and use of food labels, the correlation was .6 at P < .0001. The correlation between the construct on food label use and the total HEI was .2 at P < .0001. Finally, the correlation between food label use and the sum of the HEI subscores for fat was statistically not significant.

DISCUSSION Our study examined the reliability and validity of 7 scales developed from the 1995 DHKS (2 nutrition knowledge, 4 social-psychological factors, and 1 use of food label). No scale for nutrition knowledge had a reliability estimate above .70. Sapp and Jensen reported the reliability estimates of the nutrition knowledge and diet-health awareness measures as less than .70 and above .70, respectively.30 These two measures represent very similar domains to our 2 nu t ri t i o n knowledge scales. Our small reliability estimates of nutrition knowledge may be partially attributable to the small number of questions included in our study. Nunnally and Bernstein stated that “a major way to make tests more reliable is to make them longer.”41 Sapp and Jensen’s nutrition knowledge measure and diet-health awareness measure had a larger number of question items (23 and 27, respectively) than ours (10 and 7 for the measure for the nutrient content between p roducts and the measure of diet-disease re l a t i o n s h i p s , respectively).30 The same argument can be used for our measure of perceived barriers to using food labels (created from 2 question items), resulting in a low reliability estimate (.49). However, recognizing that one of Sapp and Jensen’s measures had a low reliability, regardless of the number of questions, there might be another reason for our scales not obtaining high reliability.30 Frazao and Cleveland reported that the diet-disease awareness was higher for cholesterol than fat.12 Thus, our diet-disease awareness questions regarding differ-

Journal of Nutrition Education and Behavior Volume 35 Number 2 Table 3.

March • April 2003

89

Discriminant Validity of Knowledge, Social-Psychological, and Behavior Measures in 1995 DHKS* Knowledge

Construct

K1†

K2‡

Social-Psychological P1 §

P2 i

P3 ¶

Behavior P4#

B1**

Gender Male

12.88 (2.77)

12.4 (8.30)

36.60 (11.76)

36.45 (17.98)

Female

13.17 (2.77)

12.99 (4.50)

38.27 (9.34)

40.02 (14.52)

P value†† and effect size‡‡

.03 and .2

.03 and .1

< .001 and .2

< .001 and .3

Age (yr) §§ 20-34

36.19 (15.91)

35-50

37.63 (9.34)

51-64

38.58 (10.72)

65+

39.03 (12.10)

P value †† and effect size‡‡

< .001 and .4

Income ≤ 130% PT

16.59 (7.26)

12.37 (6.57)

> 130% PT

17.48 (2.77)

13.14 (1.73)

P value †† and effect size ‡‡

< .001 and .3

< .001 and .4

Education (yr)i i < High school

16.59 (8.99)

12.64 (4.50)

5.66 (5.53)

High school graduate

17.34 (4.15)

12.81 (3.46)

5.14 (2.07)

13.32 (12.79) 15.17 (10.72)

> High school

17.53 (3.11)

13.26 (2.42)

4.74 (2.77)

15.33 (7.26)

P valuei and effect size j

< .001 and .3

< .001 and .4

< .001 and .4

< .001 and .3

Yes

17.9 (3.80)

13.37 (2.42)

13.36 (6.22)

No

17.27 (3.11)

12.98 (2.07)

12.62 (5.19)

< .001 and .4

.001 and .2

Elevated cholesterol

P value†† and effect size‡‡ < .001 and .2 Low-fat/-cholesterol diet Yes

18.01 (6.22)

13.59 (3.11)

14.22 (7.26)

45.16 (17.29)

No

17.3 (2.77)

12.99 (1.73)

12.58 (4.84)

37.74 (12.45)

< .001 and .3

< .001 and .4

< .001 and .7

P value†† and effect size‡‡ < .001 and .3

*Weight, variance estimation unit, and variance estimation stratum provided in the 1995 DHKS were used.Mean and standard deviation (in parentheses) are shown. Only the results, which were validated based on prior studies, are shown. † K1 = knowledge on nutrient content between products (10 less knowledgeable, 20 very kno wledgeable). ‡ K2 = knowledge on diet-disease relationships (7 less knowledgeable, 14 very kno wledgeable). § P1 = perceived barriers to using food labels (2 strongly disagree, 8 strongly agree). i P2 = perceived benefits of using food labels (4 strongly disagree, 16 strongly agree). ¶ P3 = perceived ease of understanding food labels (7 not easy, 21 very easy). # P4 = importance of healthful diet (11 not very important, 44 very impor tant). **B1 = use of food label (13 not at all, 52 often). ††Listed only P < .05. ‡‡Effect size = (mean1-mean2)/standard deviation. 0.2 = small, 0.5 = medium, 0.8 = large. §§20-34 years is the reference for the Wald F test. i i< High school is the reference for the Wald F test. DHKS indicates Diet and Health Knowledge Survey ;P T, federal poverty threshold.

ent dietary intakes might be uncorrelated from each other, even though most of these dietary intakes are related to chronic diseases. The convergence validity of our 2 nutrition knowledge measures was weaker (r = .2) than what Sapp and Jensen reported (r = .4 to .5).30 Although two knowledge measures in our study represent the same domains as in their study,

questions used in the 2 studies were not identical.We suspect that the difference in convergent validity between these 2 studies was attributable to the differences in quest i o n s . Another possible reason for the weak conve r g e n t validity of our measures may be the low reliability of our c o n s t ru c t s .D i s c riminant validity of our 2 nutrition knowledge measures was supported.

90

Obayashi et al/RELIABILITY AND VALIDITY OF SELECTED SCALES FROM THE DHKS

Scales on social-psychological factors and the use of food labels received reliability estimates above .7, suggesting that they are reliable. Most of these constructs partially s u p p o rted discriminant validity (some of our findings matched with findings from previous studies but not othe rs ) .10,16,30,35,36,48,50 Again, we suspect that the differences between our results and those from previous studies were attributable to different questions across studies. Few studies, especially regarding new food labels, were available for our discriminant validity tests. Also, there were not enough questions in the DHKS that could be used for testing the c o nvergence validity of our measure s . Ve ry few studies reported convergent validity of nutrition knowledge constructs, making our interpretation of the results difficult. 20,30 Thus, additional studies regarding constructs on social-psychological factors and food label use are needed to determine support for our measures and document the discriminant and convergent validity. The correspondence validity of our knowledge and social-psychological measures was weak, with the exception of a measure of perceived benefits of using food labels. Sloan pointed out that over 90% of her respondents could identify good sources of vitamin C but that less than 10% could answer how much orange juice would meet their requirement for vitamin C.55 This means that her respondents might not consume an adequate amount of vitamin C despite their familiarity with good sources of vitamin C because they do not know how much they need to cons u m e. T h u s , even though our knowledge measures are important for our subjects to eat healthfully, there might be additional knowledge influences on overall diet quality, resulting in weak correspondence validity of these constructs in relation to the HEI. Similarly, even though our participants reported reading food labels, various factors may impact their eating behav i o rs . Some studies indicate that nutrition knowledge does not influence dietary behaviors directly but rather indirectly through attitudes, intentions, or perceived threats to health.5,30 Many social-psychological factors in our study were also subcomponents of va r ious social-cog n i t ive theories that indirectly affect dietary behav i o rs .T h u s , further investigation is needed to determine whether simultaneous use of these measures may better explain individuals’ dietary behaviors or use of food labels. With high reliability and partial support for validity, we conclude that the constructs on perceived benefits of using food labels, ease of understanding the food label, importance of healthful eating, and the use of food labels are reliable and valid (Table 4). One limitation of our study is the reduction of ori gi n a l sample size. Although the sample size was large enough to conduct statistical tests, we had to eliminate a large number of subjects to include many question items we were interested in. Caution should be taken because our results may have underestimated the subjects who had an education less than high school and may have overestimated the subjects

who had more than high school education compared with the actual US population. In addition, use of the HEI might have over- or underestimated our results. The correspondence validity for the two nutrition knowledge measures was confirmed based on the findings by Va ri yam and Blayl o c k . 5 1 A busabha and colleagues re p o rted that effective s t r a t e gies to reduce dietary fat among 65 fre e - l iv i n g adults included increasing summer fruits, vegetables, and grains along with decreasing cooking fa t . 5 6 In CSFII 1994-1996, most of the HEI subscores for the US population averaged around 6 and 7, except for fruit and milk i n t a ke s . 3 9 H oweve r, it is still possible for individuals to obtain a high HEI score even though some of his/her subscores are low. Thus, caution should be taken in using our measures.

IMPLICATIONS FOR RESEARCH AND PRACTICE On completion of this research on the validity and reliability of the DHKS scale measures, we are able to establish with confidence the positive association among perceived benefits of using food labels, actual use of food labels, and the overall diet quality. Reliable and valid measures are prerequisite to obtaining accurate information to derive useful conclusions on complex individual dietary behavior in relation to other associated factors. Measures to characterize complex issues such as nutrition knowledge and socialpsychological factors in turn require multiple questions to cover different domains within a measure. Large-scale surveys are further constrained with the number of questions in the survey to balance validity, reliability, and burdens on respondents. Although it is not unusual to have marginal reliability and validity of nutrition knowledge measures, as in the 1995 DHKS, researchers should be aware of the risks associated with drawing inferences or planning programs based on the findings. Our study clearly supports the confidence for nutrition educators in using the question items included in the DHKS for selected measures and the importance of checking the validity and reliability of survey questionnaires.

ACKNOWLEDGMENTS The authors would like to acknowledge Dr. Stephen Sapp from the Department of Sociology at Iowa State University for his guidance in developing the study method used in this article. We also would like to acknowledge J.M. Kerver, Dr. Eun Ju Yang, and E.R. Meier from the Department of Food Science and Human Nutrition at Michigan State University for their input on developing measures and support.

Journal of Nutrition Education and Behavior Volume 35 Number 2 Table 4.

March • April 2003

91

Summary of Reliability and Validity Results* Validity Reliability †

Content

Discriminant‡

Convergent

K 1§

0.46

Yes

4/6

Weak

Weak

K2 ||

0.65

Yes

5/5

Weak

Weak

Scales

Correspondence

P1 ¶

0.48

Yes

1/2

NA

NA

P2 #

0.82

Yes

3/5

NA

Moderate NA

P3**

0.78

Yes

1/2

NA

P4 ††

0.82

Yes

3/3

NA

NA

B1 ‡‡

0.91

Yes

2/4

NA

Weak

NA = not applicable. *Yes = valid. † Cronbach α. ‡ Number of findings matched with the findings in prior studies/number of findings in prior studies. § K1 = knowledge of nutrient content between products (10 less knowledgeable, 20 very knowledgeable). ||K2 = knowledge of diet-disease relationships (7 less knowledgeable, 14 very kno wledgeable). ¶ P1 = perceived barriers from using food labels (2 strongly disagree, 8 strongly agree). # P2 = perceived benefits of using food labels (4 strongly disagree, 16 strongly agree). **P3 = perceived ease to understand food labels (7 not easy, 21 very easy). †† P4 = importance of healthful diet (11 not very important, 44 very impor tant). ‡‡ B1 = use of food label (13 not at all, 52 often).

REFERENCES 1. Glanz K. Nutrition for risk factor reduction and patient education: a review. Prev Med. 1985;14:721-752. 2. Thomas PR, ed. Improving America’s Diet and Health: From Recommendations to Action.Wa s h i n g t o n ,D C : National Academy Pre s s ;1 9 9 1 . 3. Morton JF, Guthrie JF. Diet-related knowledge, attitudes, and practices of low-income individuals with children in the household. Fam Econ Nutr Rev. 1997;10:2-15. 4. Smith AM, Owen N. Association of social status and health-related beliefs with dietary fat and fiber densities. Prev Med. 1992;21:735-745. 5. Contento IR, Murphy BM. Psycho-social factors differentiating people who reported making desirable changes in their diets from those who did not. J Nutr Educ. 1990;22:6-14. 6. Kristal AR, Hedderson MM. P re d i c t o rs of self-initiated, healthful dietary change. J Am Diet Assoc. 2001;101:762-766. 7. Smith MJ, Johnson RK,Wang MQ.The association between smoking and the diet and health attitudes, awareness, and knowledge of lowincome parents. Fam Econ Nutr Rev. 1997;10:16-25. 8. Tuorila H, Pangborn RM. Prediction of reported consumption of selected fat-containing foods. Appetite. 1988;11:81-95. 9. Shepherd R, Stockley L. Nutrition knowledge, attitudes, and fat consumption. J Am Diet Assoc. 1987;87:615-619. 10. Jensen HH, Kesavan T, Johnson SR. Measuring the impact of health awareness on food demand. Rev Agric Econ. 1992;14:299-312. 11. Carlson KA, Gould BW. The role of health and nutrition factors in food demand analysis. West J Agric Econ. 1991;16:21-35. 12. Frazao B, Cleveland L. Diet-health awareness about fat and cholesterol—only a start. Food Rev. 1994;17:15-22. 13. Va ri yam JN, Blaylock J, Smallwood DM. Modeling nutrition knowledge, attitudes, and diet-disease awareness: the case of dietary fiber. Stat Med. 1996;15:23-35.

14. Colavito EA, Guthrie JF, Webb RE. Relationship of diet-health attitudes and nutrition knowledge of household meal planners to the fat and fiber intakes of meal planners and pre s c h o o l e rs . J Nutr Educ. 1996;28:321-328. 15. Saunders RP, Rahilly SA. Influences on intention to reduce dietary intake of fat and sugar. J Nutr Educ. 1990;22:169-176. 16. Glanz K, Kristal A R ,S o rensen G, Palombo R, Heimendinger J, Probart C. Development and validation of measures of psychological factors influencing fat-and fiber-related dietary behavior. Prev Med. 1993; 22:373-387. 17. Towler G, Shepherd R.Application of Fishbein and Ajzen’s expectancyvalue model to understanding fat intake. Appetite. 1992;18:15-27. 18. Stafleu A, Graaf C, Staveren W. Attitudes towards high-fat foods and their low-fat alternatives: reliability and relationship with fat intake. Appetite. 1994;22:183-96. 19. Kristal AR, Patterson RE, Glanz K, et al. Psychosocial correlates of healthful diets: baseline results from the working well study. Prev Med. 1995;24:221-228. 20. Axelson ML, Brinberg D. The measurement and conceptualization of nutrition knowledge. J Nutr Educ. 1992;24:239-246. 21. Rosenstock IM, Strecher VJ, Becker MH. Social learning theory and the health belief model. Health Educ Q. 1988;15:175-183. 22. Janz N, Becker M.The health belief model: a decade later. Health Educ Q. 1984;11:1-47. 23. Cummings KM, B e c ker MH, Maili MC. B ri n ging the models together: an empirical approach to combining va ri a bles used to explain health actions. J Behav Med. 1980;3:123-145. 24. Parmenter K,Wardle J. Evaluation and design of nutrition knowledge measures. J Nutr Educ. 2000;32:269-277. 25. Contento IR, Randell JS, Basch CE. Review and analysis of evaluation measures used in nutrition education intervention research. J Nutr Educ Behav. 2002;34:2-25.

92

Obayashi et al/RELIABILITY AND VALIDITY OF SELECTED SCALES FROM THE DHKS

26. US Dept of Agriculture, Agricultural Research Service. 1995 Continuing Survey of Food Intakes by Individuals and 1995 Diet and Health Knowledge Survey, Springfield,Va: National Technical Information Serv i c e ;1 9 9 5 . Accession No. PB97-500789. 27. Guthrie JF. Quantitative nutrition education research:approaches,findi n g s ,o u t l o o k . J Nutr. 1994;124:1813S-1819S. 28. Lino M, Guthrie JF. Food expenditures, shopping behavior, and diet quality of families maintained by single mothers and married couples. Fam Econ Rev. 1994;7:9-21. 29. Smallwood DM, Blaylock JR. Fiber: not enough of a good thing. Food Rev. 1994;17:23-29. 30. Sapp SG, Jensen HH. Reliability and validity of nutrition knowledge and diet-health awareness tests developed from 1989-1991 Diet and Health Knowledge Surveys. J Nutr Educ. 1997;29:63-72. 31. Nutrition Labeling and Education Act of 1990, Publ L No. 101-535, 104 Stat 2353. 32. Geiger CJ, Wyse BW, Parent CR, Hansen RG. Review of nutrition labeling formats. J Am Diet Assoc. 1991;91:808-812. 33. Perez-Escamilla R, Haldeman L. Food label use modifies association of income with dietary quality. J Nutr. 2002;132:768-772. 34. Levy AS Fein SB. Consumers’ ability to perform tasks using nutrition labels. J Nutr Educ. 1998;30:210-217. 35. Neuhouser ML, Kristal AR, Patterson RE. Use of food nutrition labels is associated with lower fat intake. J Am Diet Assoc. 1999;99:45-50,53. 36. Kreuter MW, Brennan LK, Scharff DP, Lukwago SN. Do nutrition label readers eat healthier diets? Behavioral correlates of adults’ use of food labels. Am J Prev Med. 1997;13:277-283. 37. Colavito E, Guthrie JF. USDA’s new Diet and Health Knowledge Survey: how can it be used for theory-based research? J Assoc Study Food Soc. 1996;1(1):13-22. 38. The Center of Nutrition Policy and Promotion (CNPP), the United States Department of Agriculture (USDA).The USDA Healthy Eating I n d e x . Ava i l a ble at: http://www.usda/gov/cnpp/hei9496data.htm. Accessed November 26, 2001. 39. Browman SA, Lino M, Gerrior SA, Basiotis PP. The Healthy Eating Index: 1994-96. USDA Center for Nutrition Policy and Promotion; 1998. 40. Research Triangle Institute. SUDAAN User’s Manual, Release 8.0. Research Triangle Pa r k ,N C :R e s e a rch Triangle Institute; 2001.

41. Nunnally JC, Bernstein IH. Psychometric Theory. 3rd ed. New Yo r k ,N Y: M c G r aw - H i l l ,I n c. ;1 9 9 4 . 42. SPSS Professional Statistics 7.5. Chicago, Ill: SPSS Inc. ;1 9 9 7 . 43. Guthrie JF, Fox JJ, Cleveland LE,Welsh S.Who uses nutrition labeling, and what effects does label use have on diet quality? J Nutr Educ. 1995;27:163-172. 44. Fullmer S, Geiger CJ, Parent CRM. Consumers’ knowledge, understanding, and attitudes toward health claims on food labels. J Am Diet Assoc. 1991;91:166-171. 45. Bender MM, Derby BM. Prevalence of reading nutrition and ingredient information on food labels among adult A m e ri c a n s :1 9 8 2 - 1 9 8 8 . J Nutr Educ. 1992;24:292-297. 46. Michel PM, Korslund MK, Finan A, Johnson J. Food label reading habits of WIC clients. J Nutr Educ. 1994;26:146-148. 47. Hrovat KB, Harris KZ, Leach AD, Russell BS, Harris BV, Sprecher DL. The new food label, type of fat and consumer choice. Arch Fam Med. 1994;3:690-695. 48. Kristal A R ,L evy L, Patterson RE, Li SS,White E.Trends in food label use associated with new nutrition labeling regulations. J Public Health. 1998;88:1212-1215. 49. Pulter D, Frazao E. Diet/health concerns about fat intake. Food Rev. 1991;14(1):16-20. 50. Sheiham A, Marmot M, Rawson D, Ruck N. Food values: health and d i e t .I n : Jowell R,Witherspoon S, Brook L, eds. British Social Attitudes. The 1987 Report. L o n d o n :G ower; 1987:95-113. 51. Va ri yam JN, Blaylock J. Unlocking the mystery between nu t ri t i o n knowledge and diet quality. Food Rev. 1998;21(2):21-28. 52. Marietta AB, Welshimer KJ, Anderson SL. Knowledge, attitudes, and behaviors of college students regarding the 1990 Nutrition Labeling Education Act food labels. J Am Diet Assoc. 1999;99:445-449. 53. SAS Institute Inc. SAS OnlineDoc®,Version 8. Cary, NC: SAS Institute I n c. ;1 9 9 9 . 54. Glanz K, Lewis FM, Rimer BK, eds. Health Behavior and Health Education:Theory, Research, and Practice. 2nd ed. San Francisco, Calif: JosseyBass Publ i s h e rs ;1 9 9 6 . 55. Sloan AE. Educating a nutrition-wise public. J Nutr Educ. 1987;19:303. 56. Abusabha R, Hsieh K-H, Achterberg C. Dietary fat reduction strategies used by a group of adults aged 50 years and older. J Am Diet Assoc. 2001;101:1024-1030.