- Email: [email protected]
Agreement between a brief mailed screener and an in-depth telephone survey: Observations from the Fresh Start study
RESEARCH Research and Professional Briefs
Agreement between a Brief Mailed Screener and an In-Depth Telephone Survey: Observations from the Fresh Start Study DENISE CLUTTER SNYDER, MS, RD; RICHARD SLOANE, MPH; DAVID LOBACH, MD, PhD; ISAAC LIPKUS, PhD; ELIZABETH CLIPP, PhD, RN; WILLIAM E. KRAUS, MD; WENDY DEMARK-WAHNEFRIED, PhD, RD
ABSTRACT Brief screening instruments can roughly characterize individual behavior and target those most in need of change. However, the level of agreement between abbreviated and full-scale instruments is often unknown. We determined agreement between a brief screener and an in-depth survey for assessing eligibility into a randomized controlled trial to improve lifestyle behaviors among cancer survivors who consumed diets with ⱖ30% total energy from fat or fewer than five servings of fruits and vegetables per day, and/or who exercised ⬍150 min/wk. Responses of 203 subjects to mailed screeners, which included scales from the National Cancer Institute (NCI) Percent Energy from Fat Screener and 5-A-Day trials, and one item on exercise, were compared with data from telephone interviews using the Diet History Questionnaire and the 7-Day Physical Activity Recall. Moderate correlations and fair agreement existed between screener and survey for intakes of fat [r⫽0.54 (P⬍.0001)/kappa statistic (ks)⫽0.35] and fruits and vegetables [r⫽0.50 (P⬍.0001)/ks⫽0.32], whereas agreement was low for exercise (ks⫽0.15). NCI and 5-A-Day screening instruments perform relatively well in targeting cancer survivors most in need of dietary change. Decisions to use brief
D. C. Snyder is a clinical research coordinator, Department of Surgery; R. Sloane is a biostatistician, Center for the Study of Aging and Human Development; D. Lobach is chief, Division of Clinical Medical Informatics, and associate professor of community and family medicine; I. Lipkus is associate professor of psychiatry; E. Clipp is a professor of nursing and medicine; W. E. Kraus is associate professor of medicine; and W. Demark-Wahnefried is director, Program of Cancer Prevention, Detection and Control Research, and associate professor of surgery, Duke University Medical Center, Durham, NC. E. Clipp is also associate director of research, Geriatric Research Education and Clinical Center, VA Medical Center, Durham, NC. Address correspondence to: Denise Clutter Snyder, MS, RD, Clinical Research Coordinator, Department of Surgery, Duke University Medical Center, Box 3003, Durham, NC 27710. E-mail: [email protected] Copyright © 2004 by the American Dietetic Association. 0002-8223/04/10410-0008$30.00/0 doi: 10.1016/j.jada.2004.07.024
© 2004 by the American Dietetic Association
screening instruments should be based on available resources and tolerance for misclassification. J Am Diet Assoc. 2004;104:1593-1596.
F
resh Start is a randomized controlled trial testing whether a home-based program of interactive mailed materials that provide continuously updated tailored information will decrease fat intake and increase fruit and vegetable consumption and exercise among breast and prostate cancer survivors (1). As the survivor population continues to grow and is at risk for cardiovascular disease, diabetes, and secondary cancers (2-4), more health behavior interventions are warranted. However, because a large proportion of cancer survivors may already practice healthful behaviors (5), approaches are needed to identify individuals with the greatest need so that interventions target people who can benefit the most. Brief screening instruments are used to minimize respondent burden and quickly identify individuals most in need of behavioral change (6-10). Although some studies have found that brief screening instruments are prone to underreporting and misclassification (7,8,11,12), others suggest that screeners perform relatively well, albeit most of this work has been performed in healthy populations and not among cancer survivors explicitly (6,9,10,13,14). The purpose of this study was to examine the agreement between a brief mailed screener and an in-depth telephone survey in determining eligibility for participation in a home-based diet and exercise intervention study for newly diagnosed breast and prostate cancer survivors. METHODS The overall methods for this randomized controlled trial are reported elsewhere (1). Briefly, a mailed screening instrument was used to expediently assess factors that would exclude participation in the trial (ie, medical or physical conditions, such as angina, wheelchair use, or warfarin use, that would preclude adherence to unsupervised exercise or a diet high in fruits and vegetables). From August 2002 to January 2003, the screener was mailed with a study invitation letter, consent forms, and a prepaid return envelope to 649 individuals who had been diagnosed with early-stage breast or prostate cancer within the previous 9 months. Given that the Fresh Start trial targets survivors most
Journal of the AMERICAN DIETETIC ASSOCIATION
1593
Table 1. Estimates of intakes of fat, fruits and vegetables, and exercise reported by mailed screener and telephone survey Mailed Screener Women (Nⴝ127) Fat measure Mean % fat (SDb) % ⱖ30% fat Fruits and vegetables measure Mean servings per day (SD) % fewer than five per day Exercise measure Mean min/wk (SD) % ⬍150 min/wk a
Men (Nⴝ76)
Telephone Survey Total (Nⴝ203)
Women (Nⴝ127)
Men (Nⴝ76)
NCIa Percent Energy Fat Screener (13) 32 (4.7) 35 (4.7) 33 (5.6) 66 78 71
Diet History Questionnaire (18) 36 (6.9) 37 (6.6) 81 88
Scales from the 5 A Day (16) 4.9 (2.4) 4.6 (2.5) 52 49 Single item
4.8 (2.4) 51
43
37
Diet History Questionnaire (18) 5.8 (2.7) 5.7 (2.9) 43 50 7-Day Physical Activity Recall (19) 54 (97.7) 102 (174.2) 86 75
29
Total (Nⴝ203) 37 (6.8) 83 5.7 (2.7) 45 72 (133.3) 82
NCI⫽National Cancer Institute. SD⫽standard deviation.
b
in need of lifestyle change, eligibility criteria also require the practice of two out of three of the following behaviors: (a) ⱖ30% total energy intake from fat, (b) fewer than five daily servings of fruits and vegetables, or (c) ⬍150 minutes of exercise weekly. To determine whether a mailed screener could provide a sensitive and less burdensome means of discriminating eligibility than the use of fullscale instruments, we appended the screener with the following: (a) the National Cancer Institute (NCI) Percent Energy from Fat screening items (13) with use of the NCI scoring algorithm (15); (b) standard scales from the 5-ADay trials to estimate fruit and vegetable intake (16); and (c) a single yes/no question to assess exercise (“In an average week, did you get at least 2.5 hours of exercise?”) based on the physical activity guidelines of the American College of Sports Medicine and the Centers for Disease Control and Prevention (17). Subjects were asked to base their responses on their usual dietary and exercise behavior during the 12 months before diagnosis. The screener was a pencil-and-paper format designed to be completed in 10 minutes. Data from screeners were compared with data from telephone surveys, which used the Diet History Questionnaire (18) to estimate percent energy from fat and fruit and vegetable intake, and the 7-Day Physical Activity Recall (19) to assess the amount of moderate-to-vigorous exercise per week. To reduce respondent burden using the Diet History Questionnaire, medium portions were assumed and data were analyzed using Diet*Calc (version 1.2.1, 2003, National Cancer Institute, Applied Research Program, Bethesda, MD) to estimate percent energy from fat and servings of fruits and vegetables. On average, telephone surveys were completed within 6 weeks of receiving the mailed screener. Statistical analyses were performed using the Statistical Analysis System software (version 8.02, 2001, SAS Institute, Cary, NC). Pearson correlation coefficients were generated for percent energy from fat and servings of fruits and vegetables between the screener and survey. Kappa statistics were used to determine chance-adjusted agreement in eligibility status between the screener and survey. The resulting kappa values were compared with
1594
October 2004 Volume 104 Number 10
cut points proposed by Fleiss (20), where kappa values ⬍0.40 are considered to represent poor agreement, those between 0.40 and 0.75 are fair to good, and those ⬎0.75 suggest excellent agreement. RESULTS AND DISCUSSION Two hundred sixty-nine subjects submitted completed screeners and consent forms (41% response rate). Of these subjects, 203 had no medical or physical exclusions and completed the baseline telephone survey. The mean age of the participants was 59 years (range⫽31 to 82 years). The study sample was predominantly female (63%), white (87%), well educated with at least a college degree (60%), and had an income over $60,000 per year (52%). Eighty-two percent of subjects resided in North Carolina. Participants, when compared with the 446 nonparticipants, were more likely to be white (P⬍.0001) and younger (P⫽.02), but did not differ by sex. Table 1 presents estimates of percent energy from fat, fruit and vegetable servings, and exercise measured by the mailed screener and telephone survey. Results of the NCI Percent Energy from Fat Screener showed that mean reported consumption was 35% among men with prostate cancer and 32% among women with breast cancer, whereas mean percent energy from fat generated from the Diet History Questionnaire was 37% and 36%, respectively. These percentages exceeded those previously reported by Thompson and colleagues among healthy men and women, in which the percent of energy from fat using the screener was 32% and 30%, respectively, and the survey was 30% and 29%, respectively (13). Recently, the NCI developed a revised algorithm estimating dietary fat using the Percent Energy from Fat Screener to incorporate information about portion size, age, and sex-specific regression coefficients (15). Using the revised algorithm, mean percent fat was significantly lower for both prostate cancer survivors (34%) and breast cancer survivors (31%) (overall P⫽.005); however, levels still remained above those reported by Thompson and colleagues (13). Roughly half of subjects did not eat five or more serv-
Table 2. Agreement between mailed screener and telephone survey for fat, fruits and vegetables, and exercise measures Survey, >30% Calories Fat
Fat, kappa statisticⴝ0.35
Yes
No
Total
Screener, >30% calories fat Yes No Total
134 35 169 (83%)
12 22 34 (17%)
146 (71%) 57 (29%) 203
Survey, Fewer than Five Servings per Day
Fruits and vegetables, kappa statisticⴝ0.32
Yes
No
Total
Screener, fewer than five servings per day Yes No Total
63 29 92 (45%)
40 71 111 (55%)
103 (51%) 100 (49%) 203
Survey, <150 min/wk
Exercise, kappa statisticⴝ0.15
Yes
No
Total
Screener, <150 min/wk Yes No Total
71 95 166 (82%)
5 32 37 (18%)
76 (37%) 127 (62%) 203
ings of fruits and vegetables per day according to either the screener or the Diet History Questionnaire. Estimated mean daily servings of fruits and vegetables were lower with the screener (4.8) than with the Diet History Questionnaire (5.7). These results are consistent with those of a previous study showing subjects underestimating fruit and vegetable consumption using a screener compared with a food frequency questionnaire and 24hour recalls (8). Although few subjects reported that they were sedentary (ie, exercising ⬍150 min/wk) according to the single-item measure, the majority were classified as such using the 7-Day Physical Activity Recall. If this single question was used to determine eligibility for exercise in this trial, 45% more subjects (37% vs 82%) would have been screened out. Pearson correlations between the screener and survey were highly significant for percent total energy from fat (r⫽0.54, n⫽203, P⬍.0001) and for daily servings of fruits and vegetables (r⫽0.50, n⫽203, P⬍.0001). Classification agreement between the screener and survey, as estimated by the kappa statistic, was 0.35 for percent total energy from fat, 0.32 for daily servings of fruits and vegetables, and 0.15 for exercise. Eligibility status agreement between screener and survey are listed in boldfaced type in Table 2, whereas discordant classification is listed in regular type. Crude agreement between screener and survey was 77% for fat,
66% for fruits and vegetables, and only 51% for exercise. Overall eligibility for Fresh Start (practicing at least two of three behaviors that determine eligibility) was 55% using the screener and 80% using the survey. Given the expense of case identification and recruitment for the Fresh Start trial, we decided to determine eligibility via the in-depth survey. Although burdensome, significantly more individuals were deemed eligible to participate in the study. Studies aimed at unlimited and nonclinical populations would likely make different decisions. This study had the following limitations: (a) survey mode differed between the screener (written, self-administered) and the survey (telephone, interviewer-assisted); (b) the exercise screening item was not validated, and the time reference (12 months vs 1 week) differed between the two instruments; and (c) our response rate was quite low and likely related to the fact that surveys were submitted by individuals interested in participating in the 2-year diet and exercise study. Thus, results are based on a sample of predominantly white, North Carolinian, highly educated breast and prostate cancer survivors, and may not be generalizable to healthy populations or to individuals with other health conditions. Nor may it apply to the broader population of cancer survivors, ie, those with other cancers or who have later-stage breast or prostate cancer, and who are more likely to be ethnically diverse or of lower socioeconomic status (21). This is relevant because previous studies suggest that concordance between instruments is influenced by socioeconomic factors (22,23), with better agreement typically manifested among higher socioeconomic groups. Therefore, the level of agreement that we report is likely to be higher than that found in other populations. CONCLUSIONS Results of this study among breast and prostate cancer survivors suggest that although brief screening instruments show higher frequencies of healthful behavior practice than do full-scale instruments, moderate correlations and fair concordance exist between data obtained using NCI and 5-A-Day dietary screening instruments and full-scale surveys. Decisions to use these validated screeners should be based on tolerance for some misclassification and on carefully weighing available resources (ie, patient pool, subject and staffing burden, and time). This research was supported by NIH/NCI/AG no. 1R01CA/ AG81191-01A2 (W. D. W.), NINR 1P20NR07795-01 (E. C.), and the Susan G. Komen Foundation. References 1. Demark-Wahnefried W, Clipp EC, McBride C, Lobach DF, Lipkus I, Peterson B, Snyder DC, Sloane R, Arbanas J, Kraus WE. Design of FRESH START: A randomized trial of exercise and diet among cancer survivors. Med Sci Sports Exerc. 2003;35:415-424. 2. Brown BW, Brauner C, Minnotte M. Noncancer deaths in white adult cancer patients. J Natl Cancer Inst. 1993;85:979-997. 3. Edwards BK, Howe HL, Ries LA, Thun MJ, Rosenberg HM, Yancik R, Wingo PA, Jemal A, Feigal EG. Annual report to the nation on the status of cancer,
Journal of the AMERICAN DIETETIC ASSOCIATION
1595
4.
5.
6. 7.
8.
9. 10.
11.
12.
13.
1596
1973-1999, featuring implications of age and aging on US cancer burden. Cancer. 2002;94:2766-2792. Hewitt M, Breen N, Devesa S. Cancer prevalence and survivorship issues: Analyses of the 1992 National Health Interview Survey. J Natl Cancer Inst. 1999; 91:1480-1486. Demark-Wahnefried W, Peterson B, McBride C, Lipkus I, Clipp E. Current health behaviors and readiness to pursue life-style changes among men and women diagnosed with early stage prostate and breast carcinomas. Cancer. 2000;88:674-684. Block G, Clifford C, Naughton MD, Henderson M, McAdams M. A brief dietary screen for high fat intake. J Nutr Educ. 1989;21:199-207. Neuhouser ML, Kristal AR, McLerran D, Patterson RE, Atkinson J. Validity of short food frequency questionnaires used in cancer chemoprevention trials: Results from the Prostate Cancer Prevention Trial. Cancer Epidemiol Biomarkers Prev. 1999;8:721-725. Thompson FE, Kipnis V, Subar AF, Krebs-Smith SM, Kahle LL, Midthune D, Potischman N, Schatzkin A. Evaluation of 2 brief instruments and a food-frequency questionnaire to estimate daily number of servings of fruits and vegetables. Am J Clin Nutr. 2000;71:1503-1510. Schectman KB, Barzilai B, Rost K, Fisher EB. Measuring physical activity with a single question. Am J Public Health. 1991;81:771-773. Weiss TW, Slater CH, Green LW, Kennedy VC, Albright DL, Wun C-C. The validity of single-item, selfassessment questions as measures of adult physical activity. J Clin Epidemiol. 1990;43:1123-1129. Macera CA, Ham SA, Jones DA, Kimsey DK, Ainsworth BE, Neff LJ. Limitations on the use of a single screening question to measure sedentary behavior. Am J Public Health. 2001;91:2010-2012. Caan B, Coates A, Schaffer D. Variations in sensitivity, specificity, and predictive value of a dietary fat screener modified from Block et al. J Am Diet Assoc. 1995;95:564-568. Thompson FE, Kipnis V, Subar AF, Schatzkin A, Potischman N, Kahle LL, McNutt S. Performance of a short instrument to estimate usual dietary intake of percent calories from fat. Eur J Clin Nutr. 1998; 52(suppl 2):S63.
October 2004 Volume 104 Number 10
14. Rifas-Shiman SL, Willett WC, Lobb R, Kotch J, Dart C, Gillman MW. PrimeScreen, a brief dietary screening tool: Reproducibility and comparability with both a longer food frequency questionnaire and biomarkers. Public Health Nutr. 2001;4:249-254. 15. Percent Energy from Fat Screener: Scoring Procedures. National Institutes of Health, Applied Research Program, National Cancer Institute; 2003. Available at: http://riskfactor.cancer.gov/diet/screeners/ fat/scoring.html. Accessed August 31, 2004. 16. Thompson B, Demark-Wahnefried W, Taylor G, McClelland JW, Stables G, Havas S, Ziding F, Topor M, Heimendinger J, Reynolds KD, Cohen N. Baseline fruit and vegetable intake among adults in seven 5 A Day study centers located in diverse geographic areas. J Am Diet Assoc. 1999;99:1241-1248. 17. American College of Sports Medicine Position Stand. The recommended quantity and quality of exercise for developing and maintaining cardiorespiratory and muscular fitness, and flexibility in healthy adults. Med Sci Sports Exercise. 1998;30:975-991. 18. Diet History Questionnaire, Version 1.0. National Institutes of Health, Applied Research Program, National Cancer Institute; 2002. Available at: http:// riskfactor.cancer.gov/DHQ/forms/. Accessed August 31, 2004. 19. Sallis JF, Haskell W, Wood P, Fortmann SP, Rogers T, Blair SN, Paffenbarger RS Jr. Physical activity assessment methodology in the Five-City Project. Am J Epidemiol. 1985;121:91-106. 20. Fleiss JL. Statistical Methods for Rates and Proportions. 2nd ed. New York, NY: Wiley; 1981. 21. Hunter CP. Epidemiology, stage at diagnosis, and tumor biology of breast carcinoma in multiracial and multiethnic populations. Cancer. 2000;88:1193-1202. 22. Kristal AR, Feng Z, Coates RJ, Oberman A, George V. Associations of race/ethnicity, education, and dietary intervention with the validity and reliability of a food frequency questionnaire: The Women’s Health Trial Feasibility Study in minority populations. Am J Epidemiol. 1998;146:856-869. 23. Coates RJ, Monteilh CP. Assessments of food-frequency questionnaires in minority populations. Am J Clin Nutr. 1997;65:S1108-S1115.
Agreement between a Brief Mailed Screener and an In-Depth Telephone Survey: Observations from the Fresh Start Study DENISE CLUTTER SNYDER, MS, RD; RICHARD SLOANE, MPH; DAVID LOBACH, MD, PhD; ISAAC LIPKUS, PhD; ELIZABETH CLIPP, PhD, RN; WILLIAM E. KRAUS, MD; WENDY DEMARK-WAHNEFRIED, PhD, RD
ABSTRACT Brief screening instruments can roughly characterize individual behavior and target those most in need of change. However, the level of agreement between abbreviated and full-scale instruments is often unknown. We determined agreement between a brief screener and an in-depth survey for assessing eligibility into a randomized controlled trial to improve lifestyle behaviors among cancer survivors who consumed diets with ⱖ30% total energy from fat or fewer than five servings of fruits and vegetables per day, and/or who exercised ⬍150 min/wk. Responses of 203 subjects to mailed screeners, which included scales from the National Cancer Institute (NCI) Percent Energy from Fat Screener and 5-A-Day trials, and one item on exercise, were compared with data from telephone interviews using the Diet History Questionnaire and the 7-Day Physical Activity Recall. Moderate correlations and fair agreement existed between screener and survey for intakes of fat [r⫽0.54 (P⬍.0001)/kappa statistic (ks)⫽0.35] and fruits and vegetables [r⫽0.50 (P⬍.0001)/ks⫽0.32], whereas agreement was low for exercise (ks⫽0.15). NCI and 5-A-Day screening instruments perform relatively well in targeting cancer survivors most in need of dietary change. Decisions to use brief
D. C. Snyder is a clinical research coordinator, Department of Surgery; R. Sloane is a biostatistician, Center for the Study of Aging and Human Development; D. Lobach is chief, Division of Clinical Medical Informatics, and associate professor of community and family medicine; I. Lipkus is associate professor of psychiatry; E. Clipp is a professor of nursing and medicine; W. E. Kraus is associate professor of medicine; and W. Demark-Wahnefried is director, Program of Cancer Prevention, Detection and Control Research, and associate professor of surgery, Duke University Medical Center, Durham, NC. E. Clipp is also associate director of research, Geriatric Research Education and Clinical Center, VA Medical Center, Durham, NC. Address correspondence to: Denise Clutter Snyder, MS, RD, Clinical Research Coordinator, Department of Surgery, Duke University Medical Center, Box 3003, Durham, NC 27710. E-mail: [email protected] Copyright © 2004 by the American Dietetic Association. 0002-8223/04/10410-0008$30.00/0 doi: 10.1016/j.jada.2004.07.024
© 2004 by the American Dietetic Association
screening instruments should be based on available resources and tolerance for misclassification. J Am Diet Assoc. 2004;104:1593-1596.
F
resh Start is a randomized controlled trial testing whether a home-based program of interactive mailed materials that provide continuously updated tailored information will decrease fat intake and increase fruit and vegetable consumption and exercise among breast and prostate cancer survivors (1). As the survivor population continues to grow and is at risk for cardiovascular disease, diabetes, and secondary cancers (2-4), more health behavior interventions are warranted. However, because a large proportion of cancer survivors may already practice healthful behaviors (5), approaches are needed to identify individuals with the greatest need so that interventions target people who can benefit the most. Brief screening instruments are used to minimize respondent burden and quickly identify individuals most in need of behavioral change (6-10). Although some studies have found that brief screening instruments are prone to underreporting and misclassification (7,8,11,12), others suggest that screeners perform relatively well, albeit most of this work has been performed in healthy populations and not among cancer survivors explicitly (6,9,10,13,14). The purpose of this study was to examine the agreement between a brief mailed screener and an in-depth telephone survey in determining eligibility for participation in a home-based diet and exercise intervention study for newly diagnosed breast and prostate cancer survivors. METHODS The overall methods for this randomized controlled trial are reported elsewhere (1). Briefly, a mailed screening instrument was used to expediently assess factors that would exclude participation in the trial (ie, medical or physical conditions, such as angina, wheelchair use, or warfarin use, that would preclude adherence to unsupervised exercise or a diet high in fruits and vegetables). From August 2002 to January 2003, the screener was mailed with a study invitation letter, consent forms, and a prepaid return envelope to 649 individuals who had been diagnosed with early-stage breast or prostate cancer within the previous 9 months. Given that the Fresh Start trial targets survivors most
Journal of the AMERICAN DIETETIC ASSOCIATION
1593
Table 1. Estimates of intakes of fat, fruits and vegetables, and exercise reported by mailed screener and telephone survey Mailed Screener Women (Nⴝ127) Fat measure Mean % fat (SDb) % ⱖ30% fat Fruits and vegetables measure Mean servings per day (SD) % fewer than five per day Exercise measure Mean min/wk (SD) % ⬍150 min/wk a
Men (Nⴝ76)
Telephone Survey Total (Nⴝ203)
Women (Nⴝ127)
Men (Nⴝ76)
NCIa Percent Energy Fat Screener (13) 32 (4.7) 35 (4.7) 33 (5.6) 66 78 71
Diet History Questionnaire (18) 36 (6.9) 37 (6.6) 81 88
Scales from the 5 A Day (16) 4.9 (2.4) 4.6 (2.5) 52 49 Single item
4.8 (2.4) 51
43
37
Diet History Questionnaire (18) 5.8 (2.7) 5.7 (2.9) 43 50 7-Day Physical Activity Recall (19) 54 (97.7) 102 (174.2) 86 75
29
Total (Nⴝ203) 37 (6.8) 83 5.7 (2.7) 45 72 (133.3) 82
NCI⫽National Cancer Institute. SD⫽standard deviation.
b
in need of lifestyle change, eligibility criteria also require the practice of two out of three of the following behaviors: (a) ⱖ30% total energy intake from fat, (b) fewer than five daily servings of fruits and vegetables, or (c) ⬍150 minutes of exercise weekly. To determine whether a mailed screener could provide a sensitive and less burdensome means of discriminating eligibility than the use of fullscale instruments, we appended the screener with the following: (a) the National Cancer Institute (NCI) Percent Energy from Fat screening items (13) with use of the NCI scoring algorithm (15); (b) standard scales from the 5-ADay trials to estimate fruit and vegetable intake (16); and (c) a single yes/no question to assess exercise (“In an average week, did you get at least 2.5 hours of exercise?”) based on the physical activity guidelines of the American College of Sports Medicine and the Centers for Disease Control and Prevention (17). Subjects were asked to base their responses on their usual dietary and exercise behavior during the 12 months before diagnosis. The screener was a pencil-and-paper format designed to be completed in 10 minutes. Data from screeners were compared with data from telephone surveys, which used the Diet History Questionnaire (18) to estimate percent energy from fat and fruit and vegetable intake, and the 7-Day Physical Activity Recall (19) to assess the amount of moderate-to-vigorous exercise per week. To reduce respondent burden using the Diet History Questionnaire, medium portions were assumed and data were analyzed using Diet*Calc (version 1.2.1, 2003, National Cancer Institute, Applied Research Program, Bethesda, MD) to estimate percent energy from fat and servings of fruits and vegetables. On average, telephone surveys were completed within 6 weeks of receiving the mailed screener. Statistical analyses were performed using the Statistical Analysis System software (version 8.02, 2001, SAS Institute, Cary, NC). Pearson correlation coefficients were generated for percent energy from fat and servings of fruits and vegetables between the screener and survey. Kappa statistics were used to determine chance-adjusted agreement in eligibility status between the screener and survey. The resulting kappa values were compared with
1594
October 2004 Volume 104 Number 10
cut points proposed by Fleiss (20), where kappa values ⬍0.40 are considered to represent poor agreement, those between 0.40 and 0.75 are fair to good, and those ⬎0.75 suggest excellent agreement. RESULTS AND DISCUSSION Two hundred sixty-nine subjects submitted completed screeners and consent forms (41% response rate). Of these subjects, 203 had no medical or physical exclusions and completed the baseline telephone survey. The mean age of the participants was 59 years (range⫽31 to 82 years). The study sample was predominantly female (63%), white (87%), well educated with at least a college degree (60%), and had an income over $60,000 per year (52%). Eighty-two percent of subjects resided in North Carolina. Participants, when compared with the 446 nonparticipants, were more likely to be white (P⬍.0001) and younger (P⫽.02), but did not differ by sex. Table 1 presents estimates of percent energy from fat, fruit and vegetable servings, and exercise measured by the mailed screener and telephone survey. Results of the NCI Percent Energy from Fat Screener showed that mean reported consumption was 35% among men with prostate cancer and 32% among women with breast cancer, whereas mean percent energy from fat generated from the Diet History Questionnaire was 37% and 36%, respectively. These percentages exceeded those previously reported by Thompson and colleagues among healthy men and women, in which the percent of energy from fat using the screener was 32% and 30%, respectively, and the survey was 30% and 29%, respectively (13). Recently, the NCI developed a revised algorithm estimating dietary fat using the Percent Energy from Fat Screener to incorporate information about portion size, age, and sex-specific regression coefficients (15). Using the revised algorithm, mean percent fat was significantly lower for both prostate cancer survivors (34%) and breast cancer survivors (31%) (overall P⫽.005); however, levels still remained above those reported by Thompson and colleagues (13). Roughly half of subjects did not eat five or more serv-
Table 2. Agreement between mailed screener and telephone survey for fat, fruits and vegetables, and exercise measures Survey, >30% Calories Fat
Fat, kappa statisticⴝ0.35
Yes
No
Total
Screener, >30% calories fat Yes No Total
134 35 169 (83%)
12 22 34 (17%)
146 (71%) 57 (29%) 203
Survey, Fewer than Five Servings per Day
Fruits and vegetables, kappa statisticⴝ0.32
Yes
No
Total
Screener, fewer than five servings per day Yes No Total
63 29 92 (45%)
40 71 111 (55%)
103 (51%) 100 (49%) 203
Survey, <150 min/wk
Exercise, kappa statisticⴝ0.15
Yes
No
Total
Screener, <150 min/wk Yes No Total
71 95 166 (82%)
5 32 37 (18%)
76 (37%) 127 (62%) 203
ings of fruits and vegetables per day according to either the screener or the Diet History Questionnaire. Estimated mean daily servings of fruits and vegetables were lower with the screener (4.8) than with the Diet History Questionnaire (5.7). These results are consistent with those of a previous study showing subjects underestimating fruit and vegetable consumption using a screener compared with a food frequency questionnaire and 24hour recalls (8). Although few subjects reported that they were sedentary (ie, exercising ⬍150 min/wk) according to the single-item measure, the majority were classified as such using the 7-Day Physical Activity Recall. If this single question was used to determine eligibility for exercise in this trial, 45% more subjects (37% vs 82%) would have been screened out. Pearson correlations between the screener and survey were highly significant for percent total energy from fat (r⫽0.54, n⫽203, P⬍.0001) and for daily servings of fruits and vegetables (r⫽0.50, n⫽203, P⬍.0001). Classification agreement between the screener and survey, as estimated by the kappa statistic, was 0.35 for percent total energy from fat, 0.32 for daily servings of fruits and vegetables, and 0.15 for exercise. Eligibility status agreement between screener and survey are listed in boldfaced type in Table 2, whereas discordant classification is listed in regular type. Crude agreement between screener and survey was 77% for fat,
66% for fruits and vegetables, and only 51% for exercise. Overall eligibility for Fresh Start (practicing at least two of three behaviors that determine eligibility) was 55% using the screener and 80% using the survey. Given the expense of case identification and recruitment for the Fresh Start trial, we decided to determine eligibility via the in-depth survey. Although burdensome, significantly more individuals were deemed eligible to participate in the study. Studies aimed at unlimited and nonclinical populations would likely make different decisions. This study had the following limitations: (a) survey mode differed between the screener (written, self-administered) and the survey (telephone, interviewer-assisted); (b) the exercise screening item was not validated, and the time reference (12 months vs 1 week) differed between the two instruments; and (c) our response rate was quite low and likely related to the fact that surveys were submitted by individuals interested in participating in the 2-year diet and exercise study. Thus, results are based on a sample of predominantly white, North Carolinian, highly educated breast and prostate cancer survivors, and may not be generalizable to healthy populations or to individuals with other health conditions. Nor may it apply to the broader population of cancer survivors, ie, those with other cancers or who have later-stage breast or prostate cancer, and who are more likely to be ethnically diverse or of lower socioeconomic status (21). This is relevant because previous studies suggest that concordance between instruments is influenced by socioeconomic factors (22,23), with better agreement typically manifested among higher socioeconomic groups. Therefore, the level of agreement that we report is likely to be higher than that found in other populations. CONCLUSIONS Results of this study among breast and prostate cancer survivors suggest that although brief screening instruments show higher frequencies of healthful behavior practice than do full-scale instruments, moderate correlations and fair concordance exist between data obtained using NCI and 5-A-Day dietary screening instruments and full-scale surveys. Decisions to use these validated screeners should be based on tolerance for some misclassification and on carefully weighing available resources (ie, patient pool, subject and staffing burden, and time). This research was supported by NIH/NCI/AG no. 1R01CA/ AG81191-01A2 (W. D. W.), NINR 1P20NR07795-01 (E. C.), and the Susan G. Komen Foundation. References 1. Demark-Wahnefried W, Clipp EC, McBride C, Lobach DF, Lipkus I, Peterson B, Snyder DC, Sloane R, Arbanas J, Kraus WE. Design of FRESH START: A randomized trial of exercise and diet among cancer survivors. Med Sci Sports Exerc. 2003;35:415-424. 2. Brown BW, Brauner C, Minnotte M. Noncancer deaths in white adult cancer patients. J Natl Cancer Inst. 1993;85:979-997. 3. Edwards BK, Howe HL, Ries LA, Thun MJ, Rosenberg HM, Yancik R, Wingo PA, Jemal A, Feigal EG. Annual report to the nation on the status of cancer,
Journal of the AMERICAN DIETETIC ASSOCIATION
1595
4.
5.
6. 7.
8.
9. 10.
11.
12.
13.
1596
1973-1999, featuring implications of age and aging on US cancer burden. Cancer. 2002;94:2766-2792. Hewitt M, Breen N, Devesa S. Cancer prevalence and survivorship issues: Analyses of the 1992 National Health Interview Survey. J Natl Cancer Inst. 1999; 91:1480-1486. Demark-Wahnefried W, Peterson B, McBride C, Lipkus I, Clipp E. Current health behaviors and readiness to pursue life-style changes among men and women diagnosed with early stage prostate and breast carcinomas. Cancer. 2000;88:674-684. Block G, Clifford C, Naughton MD, Henderson M, McAdams M. A brief dietary screen for high fat intake. J Nutr Educ. 1989;21:199-207. Neuhouser ML, Kristal AR, McLerran D, Patterson RE, Atkinson J. Validity of short food frequency questionnaires used in cancer chemoprevention trials: Results from the Prostate Cancer Prevention Trial. Cancer Epidemiol Biomarkers Prev. 1999;8:721-725. Thompson FE, Kipnis V, Subar AF, Krebs-Smith SM, Kahle LL, Midthune D, Potischman N, Schatzkin A. Evaluation of 2 brief instruments and a food-frequency questionnaire to estimate daily number of servings of fruits and vegetables. Am J Clin Nutr. 2000;71:1503-1510. Schectman KB, Barzilai B, Rost K, Fisher EB. Measuring physical activity with a single question. Am J Public Health. 1991;81:771-773. Weiss TW, Slater CH, Green LW, Kennedy VC, Albright DL, Wun C-C. The validity of single-item, selfassessment questions as measures of adult physical activity. J Clin Epidemiol. 1990;43:1123-1129. Macera CA, Ham SA, Jones DA, Kimsey DK, Ainsworth BE, Neff LJ. Limitations on the use of a single screening question to measure sedentary behavior. Am J Public Health. 2001;91:2010-2012. Caan B, Coates A, Schaffer D. Variations in sensitivity, specificity, and predictive value of a dietary fat screener modified from Block et al. J Am Diet Assoc. 1995;95:564-568. Thompson FE, Kipnis V, Subar AF, Schatzkin A, Potischman N, Kahle LL, McNutt S. Performance of a short instrument to estimate usual dietary intake of percent calories from fat. Eur J Clin Nutr. 1998; 52(suppl 2):S63.
October 2004 Volume 104 Number 10
14. Rifas-Shiman SL, Willett WC, Lobb R, Kotch J, Dart C, Gillman MW. PrimeScreen, a brief dietary screening tool: Reproducibility and comparability with both a longer food frequency questionnaire and biomarkers. Public Health Nutr. 2001;4:249-254. 15. Percent Energy from Fat Screener: Scoring Procedures. National Institutes of Health, Applied Research Program, National Cancer Institute; 2003. Available at: http://riskfactor.cancer.gov/diet/screeners/ fat/scoring.html. Accessed August 31, 2004. 16. Thompson B, Demark-Wahnefried W, Taylor G, McClelland JW, Stables G, Havas S, Ziding F, Topor M, Heimendinger J, Reynolds KD, Cohen N. Baseline fruit and vegetable intake among adults in seven 5 A Day study centers located in diverse geographic areas. J Am Diet Assoc. 1999;99:1241-1248. 17. American College of Sports Medicine Position Stand. The recommended quantity and quality of exercise for developing and maintaining cardiorespiratory and muscular fitness, and flexibility in healthy adults. Med Sci Sports Exercise. 1998;30:975-991. 18. Diet History Questionnaire, Version 1.0. National Institutes of Health, Applied Research Program, National Cancer Institute; 2002. Available at: http:// riskfactor.cancer.gov/DHQ/forms/. Accessed August 31, 2004. 19. Sallis JF, Haskell W, Wood P, Fortmann SP, Rogers T, Blair SN, Paffenbarger RS Jr. Physical activity assessment methodology in the Five-City Project. Am J Epidemiol. 1985;121:91-106. 20. Fleiss JL. Statistical Methods for Rates and Proportions. 2nd ed. New York, NY: Wiley; 1981. 21. Hunter CP. Epidemiology, stage at diagnosis, and tumor biology of breast carcinoma in multiracial and multiethnic populations. Cancer. 2000;88:1193-1202. 22. Kristal AR, Feng Z, Coates RJ, Oberman A, George V. Associations of race/ethnicity, education, and dietary intervention with the validity and reliability of a food frequency questionnaire: The Women’s Health Trial Feasibility Study in minority populations. Am J Epidemiol. 1998;146:856-869. 23. Coates RJ, Monteilh CP. Assessments of food-frequency questionnaires in minority populations. Am J Clin Nutr. 1997;65:S1108-S1115.