- Email: [email protected]
Content analysis of web-based norovirus education materials targeting consumers who handle food: An assessment of alignment and readability
Food Control 65 (2016) 32e36
Contents lists available at ScienceDirect
Food Control journal homepage: www.elsevier.com/locate/foodcont
Content analysis of web-based norovirus education materials targeting consumers who handle food: An assessment of alignment and readability Hillary Evans a, Morgan G. Chao a, Cortney M. Leone a, Michael Finney b, Angela Fraser a, * a b
Department of Food, Nutrition, and Packaging Sciences, Clemson University, Clemson, SC, USA Department of Mathematical Sciences, Clemson University, Clemson, SC, USA
a r t i c l e i n f o
a b s t r a c t
Article history: Received 27 August 2015 Received in revised form 29 December 2015 Accepted 3 January 2016 Available online 8 January 2016
Human noroviruses sicken 19e21 million people in the U.S. each year, suggesting the need for education. The World Wide Web is an accessible source of information about how to prevent a norovirus infection but the accuracy and readability of information targeting consumers is unknown. The aim of the study was to evaluate the alignment and readability of web-based norovirus education materials targeting consumers using CDC evidence-based guidelines and Microsoft Word readability formulas. The search yielded 60 artifacts. Most did not address duration of handwashing (83%), use of hand sanitizers (83%), type of drying devices (92%), or avoidance of bare-hand contact with ready-to-eat foods (97%). Less than half (n ¼ 29) recommended minimizing contact with sick persons. Two-thirds of the artifacts (n ¼ 40) also did not mention the recommended concentration of sodium hypochlorite (bleach) solution to be used to disinfect surfaces contaminated with noroviruses. The mean Flesch Reading Ease score was 47.75 (score of >70 is easy to read), and the mean Flesch-Kincaid Grade Level was 10.36 so documents were written at a 10th grade level. The alignment and readability of web-based educational materials about noroviruses must be improved as knowledge is a prerequisite to application of behaviors that can prevent one from becoming infected with noroviruses. © 2016 Elsevier Ltd. All rights reserved.
Keywords: Human noroviruses Content analysis Consumer education Food safety Readability
1. Introduction Worldwide, human noroviruses are the leading cause of acute gastroenteritis. In the U.S., an estimated 21 million individuals (1 in 14) are sickened each year by this group of viruses resulting in $777 million in healthcare costs (Hall et al., 2013). Most reported outbreaks are attributed to person-to-person contact (69%) followed by the consumption of contaminated food (23%) making noroviruses the most common cause of foodborne disease in the U.S. (Hall, Wikswo, Pringle, Gould, & Parashar, 2014; Scallan et al., 2011). Results from a nationally representative survey of 1051 U.S. adults revealed that many consumers are not aware of noroviruses and have limited knowledge about how to prevent a norovirus infection. Forty-seven percent (47%) of respondents reported awareness of noroviruses with 85% having heard the terms ‘‘cruise
* Corresponding author. Department of Food, Nutrition, and Packaging Sciences, 206 Poole Agriculture Center, Clemson University, Clemson, SC 29634, USA. E-mail address: [email protected] (A. Fraser). http://dx.doi.org/10.1016/j.foodcont.2016.01.003 0956-7135/© 2016 Elsevier Ltd. All rights reserved.
ship virus,’’ ‘‘stomach bug,’’ or ‘‘stomach flu,’’ which are all commonly used to describe noroviruses (Cates, Kosa, Brophy, Hall, & Fraser, 2015). Less than one-third (341) correctly answered 50% or more of 22 true-and-false questions, suggesting consumers also have limited knowledge about how to prevent a norovirus infection. These survey findings support the need to educate consumers about noroviruses as knowledge is a prerequisite to application of necessary behaviors to prevent illness (DiClemente et al., 1989; Fisher & Fisher, 1992). The World Wide Web (hereafter referred to as the Web) is one way to reach consumers with information about how to prevent a norovirus infection. Eighty-seven percent (87%) of U.S. adults use the internet, and of those, 72 percent use the Web to seek health information (Pew Research Center, 2014). While the internet has accelerated the speed and spread of information, it also has a downside e allowing the dissemination and diffusion of information without checks and balances, peer review, and/or fact checking. Several studies that evaluated health information available online reported a lack of quality information characterized by inaccurate content and/or inappropriate design and usability
H. Evans et al. / Food Control 65 (2016) 32e36
(Berland et al., 2001; Irwin et al., 2011; Scullard, Peacock, & Davies, 2010). Moreover, a systematic review of web-based medical information further supported the above-mentioned authors' conclusions that there is a lack of high-quality sites on the Web that offer complete and accurate information on various health topics (Eysenbach, Powell, Kuss, & Sa, 2002). One problem with inaccurate or misleading health information being so accessible on the Web is that the user of this information might implement recommendations that have no effect on preventing the targeted disease or condition or even worse one might implement recommendations that cause harm (Ilic, 2010). Another potential problem is the readability of the text. If text is not easy to read, its usability can be diminished. To illustrate how wide ranging this problem could be, one needs to simply read the National Assessment of Adult Literacy (NAAL), released in 2006 by the U.S. Department of Education. The authors report that 30 million U.S. adults struggle with basic reading tasks, which represents nearly 10% of the U.S. population (Kutner, Greenberg, Jin, & Paulsen, 2006). Within the context of health information, Kessels (2003) pointed out that 40e80% of information provided to patients by health professionals is forgotten immediately because when text is too difficult to read, the reader could become frustrated hence they stop reading, hence its purpose is not achieved e improving knowledge. Thus, experts recommend documents about health, medicine, or safety be written at the 5th grade level (DuBay, 2004). These authors assert that if consumer education materials are aligned with evidence-based guidelines and are easy to read, the reader is more likely to engage in behaviors known to prevent a norovirus infection. Thus, the aim of the study was to evaluate the alignment and readability of Web-based consumer education materials addressing noroviruses using evidence-based guidance documents and Microsoft Word readability formulas. Two research questions guided the study: 1) Are Web-based norovirus materials targeting consumers who handle food aligned with the three prevention strategies recommended in CDC guidelines? and 2) Are materials written at the recommended grade level for healthrelated documents (5th grade level)? The study findings can be used to inform the revision or creation of consumer education materials that focus on preventing a norovirus infection. 2. Methods To be included in the sample, eligible educational materials (referred to as artifacts hereafter) were published between January 1, 2011 and May 23, 2013 and: 1) target adult consumers, 2) pertain to preparing food in the home, and 3) be available via Advanced Search of Google.com. The start date of the search was chosen because in January 2011, Scallan et al. (2011) reported that noroviruses causes 58% of foodborne disease in the U.S. Artifacts directed toward children and those formatted as blogs, news articles, theses, dissertations, research articles, Wikipedia entries, question/answer sites, forums, and continuing education training materials were deleted. A Google.com Advanced Search was conducted using the following search string: “norovirus” AND “food handling” AND “food preparation.” Food-related terms were chosen as it was assumed this would narrow the findings to artifacts that address preventing norovirus infections related to food preparation in the home. Artifacts were sorted by relevance, and two team members screened each artifact based on inclusion criteria. All eligible artifacts were downloaded as Portable Document Format (PDF) files or HyperText Markup Language (HTML) files. A coding manual comprised of 71 items divided into four topic areas was created. The topic areas included: 1) identifying information, 2) format (including readability statistics), 3) content
33
(including food safety hazards), and 4) prevention strategies. The items in Table 1 assess three prevention strategies outlined in the Centers for Disease Control and Prevention (CDC) guidelines for preventing and managing a norovirus infection: 1) hand hygiene, 2) exclusion and isolation of sick individuals, and 3) environmental sanitation (Hall et al., 2011). While these guidelines were designed for use in health care and not home settings, at the time of the analysis it was believed they were the best source of evidencebased information to prevent norovirus infections. A corresponding SurveyMonkey® (SurveyMonkey Inc., Paolo Alto, CA) instrument served as the coding sheet. Four trained coders independently conducted a pilot-test of the coding manual. Inconsistencies or ambiguities found in the manual during piloting were corrected before analysis began. Two trained coders then independently reviewed each artifact. Responses were entered into the SurveyMonkey® instrument. All responses were exported to an Excel spreadsheet. A third coder reconciled disagreements between coders. To address alignment with the CDC recommendations, each artifact was assigned a total score, and sub-scores across the three disease management guidance documents were assessed: hand hygiene, isolation and exclusion of sick persons, and environmental sanitation. The maximum possible quality scores for each topic area were 7, 2, and 5, respectively. Response frequencies as well as mean scores, standard deviations, and ranges were calculated using SAS 9.3 (SAS Institute, Inc., Cary, NC). To assess readability, two formulas were used e Flesch Reading Ease and Flesch-Kincaid Grade Level e both available in Microsoft Word. Flesch Reading Ease scores range from 0 to 100, with a lower score indicating that a document is more difficult to read than one with a higher score. A score of 70 or above is classified as ‘easy’ and is written at the grade school level. A score of 60e70 is ‘standard’ and is written at the high school level. A score of 60 or below is ‘difficult.’ The Flesch-Kincaid Grade Level uses mean sentence and word length to determine grade level between grades 3 and 12 (D'Alessandro, Kingsley, & Johnson-West, 2001). 3. Results The Google Advanced Search yielded 826 results. After opening and viewing the first 292 results (sorted by relevance) the following was displayed on the results listing screen: “In order to show you the most relevant results, some entries very similar to the 292 already displayed were omitted. If you like, you can repeat the search with the omitted results included.” The search was repeated and it was determined that all results were duplicates of those already viewed. These 292 results were screened using the three eligibility criteria, yielding 74 artifacts which were then downloaded. After a second screening, 14 artifacts were removed due to page unavailability, duplicate, or wrong target audience. A total of 60 artifacts were included in the sample and analyzed. 3.1. Research question 1: alignment with prevention strategies The total mean score across all three prevention strategies was low e 5.2 of 14 points e suggesting artifacts were not aligned with CDC guidelines (Table 2). All (N ¼ 60; 100%) artifacts addressed at least one of seven components of hand hygiene that were assessed but the mean sub-score for hand hygiene was low, 2.3 of 7 points (SD ¼ 0.77). Most (83%) did not address length of handwashing, type of drying devices (92%), avoiding bare-hand contact with ready-to-eat foods (97%), or use of hand sanitizers (83%). Of the artifacts that mentioned exclusion and isolation of sick individuals (78%), the mean score was 1.2 of 2. In nearly all artifacts (72%) sick individuals were discouraged from preparing food for others, but
34
H. Evans et al. / Food Control 65 (2016) 32e36
Table 1 Total scores and sub-scores assessing alignment of norovirus education materials on the Web with the three prevention strategies outlined in the Centers for Disease Control and Prevention disease management guidelines. Prevention strategy
Hand Hygiene Isolation and Exclusion of Sick Persons Environmental Sanitation Total
Artifactsa n
%
60 47 29 60
100% 78% 48% 100%
Mean score
Range (Min-Max)
Standard deviation
2.3b 1.2c 1.7d 5.2e
1e4 0e2 0e5 1e11
0.77 0.78 1.9 2.4
a
Artifacts refer to individual items included in our sample of norovirus education materials on the Web. The maximum possible score an artifact could receive by fully aligning with the CDC disease management guidelines for hand hygiene was 7; no artifacts were fully aligned as evidenced by the maximum score received being 4. c The maximum possible score an artifact could receive by fully aligning with the CDC disease management guidelines for isolation and exclusion of sick persons was 2. d The maximum possible score an artifact could receive by fully aligning with the CDC disease management guidelines for environmental sanitation was 5. e The maximum possible score an artifact could receive by fully aligning with all three of the prevention strategies outlined in the CDC disease management guidelines was 14; no artifacts were fully aligned as evidenced by the maximum score received being 11. b
Table 2 Scoring key for determining alignment of prevention strategies with Centers for Disease Control and Prevention disease management guidelines and/or the FDA 2013 Food Code. Coding manual question by topic area
Accurate recommendation
Hand Hygiene Is hand-washing stated? What is the duration for handwashing?
Source of recommendation Score
Yes CDC Guidelines Between 10 and 15 seconds or 20 seconds or greatera FDA 2013 Food Code CDC Guidelines What type of drying device is recommended? Paper towels FDA 2013 Food Code Mechanical dryer Are hand sanitizers mentioned? Yes CDC Guidelines Are hand sanitizers stated to be an acceptable alternative for hand-washing? No CDC Guidelines Is avoiding bare-hand contact with ready-to-eat foods mentioned? Yes CDC Guidelines Maximum Possible Score Isolation and Exclusion of Sick Individuals Is minimizing contact with persons when they are sick mentioned? Yes CDC Guidelines Are sick persons discouraged from preparing food for others? Yes CDC Guidelines Maximum Possible Score Environmental Sanitation Are there recommendations for cleaning vomit? Yes CDC Guidelines Are there recommendations for cleaning fecal matter? Yes CDC Guidelines Is it mentioned that bleach solutions must be freshly prepared? Yes CDC Guidelines Is a concentration of bleach solution suggested? Yes CDC Guidelines Is a method/procedure for cleaning vomit or fecal matter provided? Yes CDC Guidelines Maximum Possible Score
1 1 1 1 1 1 7 1 1 2 1 1 1 1 1 5
a If an artifact recommended washing hands for either 10e15 seconds e or any length of time within that range (recommended by the FDA 2013 Food Code) or 20 seconds or greater (recommended by the CDC Guidelines), the artifact was given a score of 1.
only 48% recommended minimizing contact with sick persons. Less than half (48%) addressed at least one aspect of environmental sanitation with the mean score 1.7 of 5. 3.2. Research question 2: readability The mean Flesch Reading Ease for the sample was 47.6, indicating artifacts were difficult to read as the score is <60 (D'Alessandro et al., 2001). The mean Flesch-Kincaid Grade Level was 10.4, which is more than five grade levels higher than the recommended target grade level (5th grade) for health-related materials (D'Alessandro et al., 2001). 4. Discussion In the analysis of consumer education materials, information was found to not be aligned (either omitted and/or inconsistent) with CDC guidelines suggesting consumers might not have sufficient information needed to prevent a norovirus infection. One obvious reason for the incompleteness and/or omission of information is the authors of materials in the sample might not have known about the CDC prevention guidelines, published in March
2011 in Morbidity Mortality Weekly Report (MMWR). While MMWR is in the public domain, it might not be a known or commonly used source of information for those who author consumer education materials. Thus, some authors might have used information from other sources for which the evidence base is unknown or possibly incorrect. This illustrates the need to create (and subsequently use) evidence-based guidelines as the underpinning of consumer education materials. Food safety educators could use as a model approach the evidence-based clinical practice guidelines commonly used in health care. Such guidelines minimize intelligent guesswork as they can inform practice for individual healthcare professionals in the face of mounting evidence often presented in inaccessible journals. Moreover, the process of using evidence-based guidelines would also encourage food safety educators to actively seek the best evidence to inform practices rather than maintaining a “this is how it's always been done,” mindset. However, it is important to note that even within these evidence-based guidelines, the evidence base contains gaps. One example centers around vomit and fecal matter clean-up, which none of the artifacts in the sample addressed. To disrupt the transmission of noroviruses, it is well known that vomitus and feces must be properly cleaned up then the contaminated area
H. Evans et al. / Food Control 65 (2016) 32e36
disinfected. But, the actual distance surrounding the vomit/fecal episode that one needs to disinfect is unknown as the evidence to support a geographic radius is inconclusive. The only published evidence available suggests vomit can be deposited up to 25 feet from its point source but these findings are based on one epidemiological study and one laboratory study (Evans et al., 2002; Booth, 2014). When an author is faced with incomplete information, such as not knowing how far from an episode to disinfect, they might use whatever information is available (25 feet), even if it is not grounded in science, or one might simply omit a recommendation (provide no distance) (Kardes, Posavac, Cronley, & Herr, 2008; Sanbonmatsu, Kardes, & Herr, 1992; Simmons & Lynch, 1991). It is important to minimize missing information as that could lead to poor implementation of practices, emphasizing the need for guidelines to evolve as new knowledge becomes available. To the contrary, sometimes the evidence base was complete but the information presented was not. Case in point, many materials recommended using sodium hypochlorite (bleach) solutions to treat surfaces, which is correct, but most did not state a concentration of the disinfection solution, which has clearly been established (Belliot, Lavaux, Souihel, Agnello, & Pothier, 2008; Gulati, Allwood, Hedberg, & Goyal, 2001; Jimenez & Chiang, 2006). One reason this detail was omitted could be that some authors might have presented motivational messages (emotional messages), which tend to be briefer than procedural messages (task-oriented messages), which tend to provide more details. It is important to note that message type was not analyzed as part of the study so this is an unproven assertion. Another reason for omitting information could be that some authors might have viewed recommended practices as impractical or unnecessary to implement in a home setting. One example of this was minimizing contact with sick persons, which was addressed in less than half of the artifacts in the sample. It makes sense that in a regulated food establishment workers cannot work while ill as a means to disrupt the transmission of noroviruses and other pathogens through direct (person-to-person) as well as indirect (contact with contaminated surfaces and food) routes. However, recommending that one not contact sick persons in a home setting might not be practical. One might not be able to isolate themselves from others if they are a primary caregiver for one or more household members or simply because of the physical constraints of the dwelling. Thus, more practical recommendations are needed, such as not preparing foods while sick, using disposal paper towels to dry hands instead of cloth towels, and routine cleaning and disinfecting of all high-touch surfaces while sick. Thus, information must be practical as well as accurate. One more issue that needs to be addressed is the use of government regulations in the absence of consumer-targeted regulations. This appears to be a sensible approach as the presumption is most regulations are grounded in science. However, one must exercise caution as regulations might not always be based on the most current evidence. Take for example the use of alcohol-based hand sanitizers as an alternative for hand washing, which the U.S. Food Code prohibits, and which most artifacts in the sample did not mention. In 2002 the CDC released Guidelines for Hand Hygiene in Healthcare (Boyce & Pittet, 2002) but in May 2003 the FDA prepared a written response, clearly stating the CDC guidelines could not be applied to foodservice establishments (FDA, 2003). The logic was that (1) pathogens commonly transmitted by hands in health-care settings differ from those transmitted in foodservice settings; (2) use of alcohol-based hand rubs in place of hand-washing has not been shown to reduce important foodborne pathogens, such as noroviruses, on food worker hands; and (3) types and levels of soil on the hands of health care workers differ from that on the hands of foodservice and retail food handlers. Much has been learned about
35
hand hygiene since the FDA response was published over ten years ago. For example, some studies suggest that some hand sanitizer formulations may be effective against noroviruses while others are not, so the assertion that hand rubs cannot reduce noroviruses is not necessarily true. Obviously, this particular situation is complex, and in the interest of providing a clear message, it may be unclear whether to include recommendations regarding alcohol-based hand sanitizers or to rely on government regulations. The conservative approach would be to omit the recommendation; whereas, some authors might have decided to provide the knowledge under the impression that recommending some hand cleaning behavior is better than no recommendation. Finally, along with lack of alignment with CDC guidelines, the artifacts in the sample were also not easy to read – written at a high school grade level and not at the 5th grade level as suggested. Simplifying information by writing it at a 5th grade level or lower can increase comprehension, a precursor to knowledge implementation. Presenting too much information about noroviruses and its epidemiology and using complicated words can decrease one's understanding (comprehension) of educational materials. Thus, the focus of norovirus education materials should be on how to prevent a norovirus infection and not extraneous facts about noroviruses. Even when exposed to accurate and complete information, if one cannot understand it, they cannot act on it. Educational materials are only as beneficial as the knowledge gained as a result of being able to read them.
4.1. Limitations Due to the ever changing nature of the Web, artifacts sampled during the study could change as time progresses so new materials that are better aligned with CDC guidance documents and that are easy to read might be available. Likewise, knowledge also evolves. In late 2014, the CDC published on their website five messages to help consumers prevent a norovirus infections: 1) practice proper hand hygiene; 2) take care in the kitchen; 3) do not prepare food while infected; 4) clean and disinfect contaminated surfaces; and 5) wash laundry thoroughly (CDC, 2015). However, coding and analysis had already been completed so kitchen and laundry practices were not assessed within the study, demonstrating the challenge of evaluating web-based information. Further, to analyze the readability of the artifacts in the sample, two readability formulas available through Microsoft Word were used. This method is widely used, but also not comprehensive, as it only considers factors such as word and sentence length so does not take into consideration comprehension.
5. Conclusion These authors assert that there is a need to revise and possibly create new materials about how to prevent a norovirus infection. Consumer education, if approached properly, could effectively reduce the burden of illness attributed to noroviruses. To be effective, consumer-targeted materials must be aligned with evidence-based guidelines. However, it is important that these guidelines be routinely evaluated to be certain that they are based on the best evidence that is available. Moreover, practices included in such guidelines must also be practical to be implemented in a home setting and must be presented in a manner that is easy to read. Finally, future research should assess materials using the CDC Clear Communication Index, a set of 20 items that can be used to develop as well as assess public communication products (CDC, 2014).
36
H. Evans et al. / Food Control 65 (2016) 32e36
Acknowledgments This work was funded through a grant from the U.S. Department of Agriculture, National Institute of Food and Agriculture, Agriculture and Food Research Initiative, “Building Capacity to Control Viral Foodborne Disease: A Translational, Multidisciplinary Approach,” Grant No. 2011-68003-30395. Appendix A. Supplementary data Supplementary data related to this article can be found at http:// dx.doi.org/10.1016/j.foodcont.2016.01.003. References Belliot, G., Lavaux, A., Souihel, D., Agnello, D., & Pothier, P. (2008). Use of murine norovirus as a surrogate to evaluate resistance of huma norovirus to disinfectants. Applied and Environmental Microbiology, 74(10), 3315e3318. Berland, G. K., Elliott, M. N., Morales, L. S., Algazy, J. I., Kravitz, R. L., Broder, M. S., et al. (2001). Health information on the internet: accessibility, quality, and readability in English and Spanish. Journal of the American Medical Association, 285(20), 2612e2621. Booth, C. M. (2014). Vomiting Larry: a simulated vomiting system for assessing environmental contamination from projectile vomiting related to norovirus infection. Journal of Infection Prevention, 15(5), 176e180. Boyce, J. M., & Pittet, D. (2002). Guideline for hand hygiene in health-care settings: recommendations of the healthcare infection control practices advisory committee and the HICPAC/SHEA/APIC/IDSA hand Hygiene task force. Morbidity and Mortality Weekly Report, 51(16), 1e45. Cates, S. C., Kosa, K. M., Brophy, J. E., Hall, A. J., & Fraser, A. (2015). Consumer education needed on norovirus prevention and control: findings from a nationally representative survey of U.S. adults. Journal of Food Protection, 78(3), 484e490. Centers for Disease Control and Prevention. (2014). The CDC clear communication index Accessed 27 August 2015. Available at: http://www.cdc.gov/ccindex/. Centers for Disease Control and Prevention. (2015). Preventing norovirus infection Accessed 29 October 2015. Available at: http://www.cdc.gov/norovirus/ preventing-infection.html. D'Alessandro, D. M., Kingsley, P., & Johnson-West, J. (2001). The readability of pediatric patient education materials on the World Wide Web. Archives of Pediatrics & Adolescent Medicine, 155(7), 807e812. DiClemente, R. J., Pies, C. A., Stoller, E. J., Straits, C., Olivia, G. E., Haskin, J., et al. (1989). Evaluation of school-based AIDS education curricula in San Francisco. The Journal of Sex Research, 26(2), 188e198. DuBay, W. H. (2004). The principles of readability. Costa Mesa, CA: Impact Information. Evans, M. R., Meldrum, R., Lane, W., Gardner, D., Ribeiro, C. D., Gallimore, C. I., et al. (2002). An outbreak of viral gastroenteritis following environmental contamination at a concert hall. Epidemiology and Infection, 129, 355e360. Eysenbach, G., Powell, J., Kuss, O., & Sa, E.-R. (2002). Empirical studies assessing the quality of health information for consumers on the World Wide Web: a
systematic review. Journal of the American Medical Association, 287(20), 2691e2700. Fisher, J. D., & Fisher, W. A. (1992). Changing AIDS risk behavior. Psychological Bulletin, 3, 455e474. Gulati, B. R., Allwood, P. B., Hedberg, C. W., & Goyal, S. M. (2001). Efficacy of commonly used disinfectants for the inactivation of calicivirus on strawberry, lettuce, and a food-contact surface. Journal of Food Protection, 64(9), 1430e1434. ~ aduy, P. A., Vinje , J., & Hall, A. J., Lopman, B. A., Payne, D. C., Patel, M. M., Gastan Parashar, U. D. (2013). norovirus disease in the United States. Emerging Infectious Diseases, 19(8), 1198e1205. , J., Lopman, B., Park, G. W., Yen, C., Gregoricus, N., & Parashar, U. Hall, A. J., Vinje (2011). Updated norovirus outbreak management and disease prevention guidance documents. Morbidity and Mortality Weekly Report, 60(3), 1e15. Hall, A. J., Wikswo, M. E., Pringle, K., Gould, L. H., & Parashar, U. D. (2014). Vital signs: foodborne norovirus outbreakse United States, 2009-2012. Morbidity and Mortality Weekly Report, 63(22), 491e495. Ilic, D. (2010). The role of the internet on patient knowledge management, education, and decision-making. Telemedicine and e-Health, 16(6), 664e669. Irwin, J. Y., Thyvalikakath, T., Spallek, H., Wali, T., Kerr, A. R., & Schleyer, T. (2011). English and Spanish oral cancer information on the internet: a pilot surface quality and content evaluation of oral cancer web sites. Journal of Public Health Dentistry, 71(2), 106e116. Jimenez, L., & Chiang, M. (2006). Virucidal activity of a quaternary ammonium compound disinfectant against feline calicivirus: a surrogate for norovirus. American Journal of Infection Control, 34, 269e273. Kardes, F. R., Posavac, S. S., Cronley, M. L., & Herr, P. M. (2008). Consumer inference. In C. P. Haugtvedt, P. M. Herr, & F. Kardes (Eds.), Handbook of consumer psychology. Mahwah, NJ: Lawrence Erlbaum Associates. Kessels, R. P. C. (2003). Patients' memory for medical information. Journal of the Royal Society of Medicine, 96(5), 219e222. Kutner, M., Greenberg, E., Jin, Y., & Paulsen, C. (2006). The health literacy of America's adults results from the 2003 National Assessment of Adult Literacy (NCES 2006e483). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Pew Research Center. (2014). Health fact sheet Accessed 26 August 2015. Available at: http://www.pewinternet.org/fact-sheets/health-fact-sheet/. Sanbonmatsu, D. M., Kardes, F. R., & Herr, P. M. (1992). The role of prior knowledge and missing information in multiattribute evaluation. Organizational Behavior and Human Decision Processes, 51(1), 76e91. Scallan, E., Hoeksttra, R. M., Angulo, F. J., Tauxe, R. V., Widdowson, M.-A., Roy, S. L., et al. (2011). Foodborne illness acquired in the United States e major pathogens. Emerging Infectious Diseases, 17(1), 16e22. Scullard, P., Peacock, C., & Davies, P. (2010). Googling children's health: reliability of medical advice on the internet. Archives of Disease in Childhood, 95(8), 580e582. Simmons, C. J., & Lynch, J. G. (1991). Inference effects without inference making e effects of missing information on discounting and use of presented information. Journal of Consumer Research, 17(4), 477e491. U.S. Food and Drug Administration. (2003). FDA fact sheet on hand Hygiene in retail and foodservice establishments Accessed 21 May 2012. Available at: http://www. fda.gov/Food/FoodSafety/RetailFoodProtection/ IndustryandRegulatoryAssistanceandTrainingResources/ucm135577.htm. U.S. Food and Drug Administration. (2013). Food code (DHHS publication no. PB2013110462). Alexandria, VA: U.S. Department of Commerce Technology Administration.
Contents lists available at ScienceDirect
Food Control journal homepage: www.elsevier.com/locate/foodcont
Content analysis of web-based norovirus education materials targeting consumers who handle food: An assessment of alignment and readability Hillary Evans a, Morgan G. Chao a, Cortney M. Leone a, Michael Finney b, Angela Fraser a, * a b
Department of Food, Nutrition, and Packaging Sciences, Clemson University, Clemson, SC, USA Department of Mathematical Sciences, Clemson University, Clemson, SC, USA
a r t i c l e i n f o
a b s t r a c t
Article history: Received 27 August 2015 Received in revised form 29 December 2015 Accepted 3 January 2016 Available online 8 January 2016
Human noroviruses sicken 19e21 million people in the U.S. each year, suggesting the need for education. The World Wide Web is an accessible source of information about how to prevent a norovirus infection but the accuracy and readability of information targeting consumers is unknown. The aim of the study was to evaluate the alignment and readability of web-based norovirus education materials targeting consumers using CDC evidence-based guidelines and Microsoft Word readability formulas. The search yielded 60 artifacts. Most did not address duration of handwashing (83%), use of hand sanitizers (83%), type of drying devices (92%), or avoidance of bare-hand contact with ready-to-eat foods (97%). Less than half (n ¼ 29) recommended minimizing contact with sick persons. Two-thirds of the artifacts (n ¼ 40) also did not mention the recommended concentration of sodium hypochlorite (bleach) solution to be used to disinfect surfaces contaminated with noroviruses. The mean Flesch Reading Ease score was 47.75 (score of >70 is easy to read), and the mean Flesch-Kincaid Grade Level was 10.36 so documents were written at a 10th grade level. The alignment and readability of web-based educational materials about noroviruses must be improved as knowledge is a prerequisite to application of behaviors that can prevent one from becoming infected with noroviruses. © 2016 Elsevier Ltd. All rights reserved.
Keywords: Human noroviruses Content analysis Consumer education Food safety Readability
1. Introduction Worldwide, human noroviruses are the leading cause of acute gastroenteritis. In the U.S., an estimated 21 million individuals (1 in 14) are sickened each year by this group of viruses resulting in $777 million in healthcare costs (Hall et al., 2013). Most reported outbreaks are attributed to person-to-person contact (69%) followed by the consumption of contaminated food (23%) making noroviruses the most common cause of foodborne disease in the U.S. (Hall, Wikswo, Pringle, Gould, & Parashar, 2014; Scallan et al., 2011). Results from a nationally representative survey of 1051 U.S. adults revealed that many consumers are not aware of noroviruses and have limited knowledge about how to prevent a norovirus infection. Forty-seven percent (47%) of respondents reported awareness of noroviruses with 85% having heard the terms ‘‘cruise
* Corresponding author. Department of Food, Nutrition, and Packaging Sciences, 206 Poole Agriculture Center, Clemson University, Clemson, SC 29634, USA. E-mail address: [email protected] (A. Fraser). http://dx.doi.org/10.1016/j.foodcont.2016.01.003 0956-7135/© 2016 Elsevier Ltd. All rights reserved.
ship virus,’’ ‘‘stomach bug,’’ or ‘‘stomach flu,’’ which are all commonly used to describe noroviruses (Cates, Kosa, Brophy, Hall, & Fraser, 2015). Less than one-third (341) correctly answered 50% or more of 22 true-and-false questions, suggesting consumers also have limited knowledge about how to prevent a norovirus infection. These survey findings support the need to educate consumers about noroviruses as knowledge is a prerequisite to application of necessary behaviors to prevent illness (DiClemente et al., 1989; Fisher & Fisher, 1992). The World Wide Web (hereafter referred to as the Web) is one way to reach consumers with information about how to prevent a norovirus infection. Eighty-seven percent (87%) of U.S. adults use the internet, and of those, 72 percent use the Web to seek health information (Pew Research Center, 2014). While the internet has accelerated the speed and spread of information, it also has a downside e allowing the dissemination and diffusion of information without checks and balances, peer review, and/or fact checking. Several studies that evaluated health information available online reported a lack of quality information characterized by inaccurate content and/or inappropriate design and usability
H. Evans et al. / Food Control 65 (2016) 32e36
(Berland et al., 2001; Irwin et al., 2011; Scullard, Peacock, & Davies, 2010). Moreover, a systematic review of web-based medical information further supported the above-mentioned authors' conclusions that there is a lack of high-quality sites on the Web that offer complete and accurate information on various health topics (Eysenbach, Powell, Kuss, & Sa, 2002). One problem with inaccurate or misleading health information being so accessible on the Web is that the user of this information might implement recommendations that have no effect on preventing the targeted disease or condition or even worse one might implement recommendations that cause harm (Ilic, 2010). Another potential problem is the readability of the text. If text is not easy to read, its usability can be diminished. To illustrate how wide ranging this problem could be, one needs to simply read the National Assessment of Adult Literacy (NAAL), released in 2006 by the U.S. Department of Education. The authors report that 30 million U.S. adults struggle with basic reading tasks, which represents nearly 10% of the U.S. population (Kutner, Greenberg, Jin, & Paulsen, 2006). Within the context of health information, Kessels (2003) pointed out that 40e80% of information provided to patients by health professionals is forgotten immediately because when text is too difficult to read, the reader could become frustrated hence they stop reading, hence its purpose is not achieved e improving knowledge. Thus, experts recommend documents about health, medicine, or safety be written at the 5th grade level (DuBay, 2004). These authors assert that if consumer education materials are aligned with evidence-based guidelines and are easy to read, the reader is more likely to engage in behaviors known to prevent a norovirus infection. Thus, the aim of the study was to evaluate the alignment and readability of Web-based consumer education materials addressing noroviruses using evidence-based guidance documents and Microsoft Word readability formulas. Two research questions guided the study: 1) Are Web-based norovirus materials targeting consumers who handle food aligned with the three prevention strategies recommended in CDC guidelines? and 2) Are materials written at the recommended grade level for healthrelated documents (5th grade level)? The study findings can be used to inform the revision or creation of consumer education materials that focus on preventing a norovirus infection. 2. Methods To be included in the sample, eligible educational materials (referred to as artifacts hereafter) were published between January 1, 2011 and May 23, 2013 and: 1) target adult consumers, 2) pertain to preparing food in the home, and 3) be available via Advanced Search of Google.com. The start date of the search was chosen because in January 2011, Scallan et al. (2011) reported that noroviruses causes 58% of foodborne disease in the U.S. Artifacts directed toward children and those formatted as blogs, news articles, theses, dissertations, research articles, Wikipedia entries, question/answer sites, forums, and continuing education training materials were deleted. A Google.com Advanced Search was conducted using the following search string: “norovirus” AND “food handling” AND “food preparation.” Food-related terms were chosen as it was assumed this would narrow the findings to artifacts that address preventing norovirus infections related to food preparation in the home. Artifacts were sorted by relevance, and two team members screened each artifact based on inclusion criteria. All eligible artifacts were downloaded as Portable Document Format (PDF) files or HyperText Markup Language (HTML) files. A coding manual comprised of 71 items divided into four topic areas was created. The topic areas included: 1) identifying information, 2) format (including readability statistics), 3) content
33
(including food safety hazards), and 4) prevention strategies. The items in Table 1 assess three prevention strategies outlined in the Centers for Disease Control and Prevention (CDC) guidelines for preventing and managing a norovirus infection: 1) hand hygiene, 2) exclusion and isolation of sick individuals, and 3) environmental sanitation (Hall et al., 2011). While these guidelines were designed for use in health care and not home settings, at the time of the analysis it was believed they were the best source of evidencebased information to prevent norovirus infections. A corresponding SurveyMonkey® (SurveyMonkey Inc., Paolo Alto, CA) instrument served as the coding sheet. Four trained coders independently conducted a pilot-test of the coding manual. Inconsistencies or ambiguities found in the manual during piloting were corrected before analysis began. Two trained coders then independently reviewed each artifact. Responses were entered into the SurveyMonkey® instrument. All responses were exported to an Excel spreadsheet. A third coder reconciled disagreements between coders. To address alignment with the CDC recommendations, each artifact was assigned a total score, and sub-scores across the three disease management guidance documents were assessed: hand hygiene, isolation and exclusion of sick persons, and environmental sanitation. The maximum possible quality scores for each topic area were 7, 2, and 5, respectively. Response frequencies as well as mean scores, standard deviations, and ranges were calculated using SAS 9.3 (SAS Institute, Inc., Cary, NC). To assess readability, two formulas were used e Flesch Reading Ease and Flesch-Kincaid Grade Level e both available in Microsoft Word. Flesch Reading Ease scores range from 0 to 100, with a lower score indicating that a document is more difficult to read than one with a higher score. A score of 70 or above is classified as ‘easy’ and is written at the grade school level. A score of 60e70 is ‘standard’ and is written at the high school level. A score of 60 or below is ‘difficult.’ The Flesch-Kincaid Grade Level uses mean sentence and word length to determine grade level between grades 3 and 12 (D'Alessandro, Kingsley, & Johnson-West, 2001). 3. Results The Google Advanced Search yielded 826 results. After opening and viewing the first 292 results (sorted by relevance) the following was displayed on the results listing screen: “In order to show you the most relevant results, some entries very similar to the 292 already displayed were omitted. If you like, you can repeat the search with the omitted results included.” The search was repeated and it was determined that all results were duplicates of those already viewed. These 292 results were screened using the three eligibility criteria, yielding 74 artifacts which were then downloaded. After a second screening, 14 artifacts were removed due to page unavailability, duplicate, or wrong target audience. A total of 60 artifacts were included in the sample and analyzed. 3.1. Research question 1: alignment with prevention strategies The total mean score across all three prevention strategies was low e 5.2 of 14 points e suggesting artifacts were not aligned with CDC guidelines (Table 2). All (N ¼ 60; 100%) artifacts addressed at least one of seven components of hand hygiene that were assessed but the mean sub-score for hand hygiene was low, 2.3 of 7 points (SD ¼ 0.77). Most (83%) did not address length of handwashing, type of drying devices (92%), avoiding bare-hand contact with ready-to-eat foods (97%), or use of hand sanitizers (83%). Of the artifacts that mentioned exclusion and isolation of sick individuals (78%), the mean score was 1.2 of 2. In nearly all artifacts (72%) sick individuals were discouraged from preparing food for others, but
34
H. Evans et al. / Food Control 65 (2016) 32e36
Table 1 Total scores and sub-scores assessing alignment of norovirus education materials on the Web with the three prevention strategies outlined in the Centers for Disease Control and Prevention disease management guidelines. Prevention strategy
Hand Hygiene Isolation and Exclusion of Sick Persons Environmental Sanitation Total
Artifactsa n
%
60 47 29 60
100% 78% 48% 100%
Mean score
Range (Min-Max)
Standard deviation
2.3b 1.2c 1.7d 5.2e
1e4 0e2 0e5 1e11
0.77 0.78 1.9 2.4
a
Artifacts refer to individual items included in our sample of norovirus education materials on the Web. The maximum possible score an artifact could receive by fully aligning with the CDC disease management guidelines for hand hygiene was 7; no artifacts were fully aligned as evidenced by the maximum score received being 4. c The maximum possible score an artifact could receive by fully aligning with the CDC disease management guidelines for isolation and exclusion of sick persons was 2. d The maximum possible score an artifact could receive by fully aligning with the CDC disease management guidelines for environmental sanitation was 5. e The maximum possible score an artifact could receive by fully aligning with all three of the prevention strategies outlined in the CDC disease management guidelines was 14; no artifacts were fully aligned as evidenced by the maximum score received being 11. b
Table 2 Scoring key for determining alignment of prevention strategies with Centers for Disease Control and Prevention disease management guidelines and/or the FDA 2013 Food Code. Coding manual question by topic area
Accurate recommendation
Hand Hygiene Is hand-washing stated? What is the duration for handwashing?
Source of recommendation Score
Yes CDC Guidelines Between 10 and 15 seconds or 20 seconds or greatera FDA 2013 Food Code CDC Guidelines What type of drying device is recommended? Paper towels FDA 2013 Food Code Mechanical dryer Are hand sanitizers mentioned? Yes CDC Guidelines Are hand sanitizers stated to be an acceptable alternative for hand-washing? No CDC Guidelines Is avoiding bare-hand contact with ready-to-eat foods mentioned? Yes CDC Guidelines Maximum Possible Score Isolation and Exclusion of Sick Individuals Is minimizing contact with persons when they are sick mentioned? Yes CDC Guidelines Are sick persons discouraged from preparing food for others? Yes CDC Guidelines Maximum Possible Score Environmental Sanitation Are there recommendations for cleaning vomit? Yes CDC Guidelines Are there recommendations for cleaning fecal matter? Yes CDC Guidelines Is it mentioned that bleach solutions must be freshly prepared? Yes CDC Guidelines Is a concentration of bleach solution suggested? Yes CDC Guidelines Is a method/procedure for cleaning vomit or fecal matter provided? Yes CDC Guidelines Maximum Possible Score
1 1 1 1 1 1 7 1 1 2 1 1 1 1 1 5
a If an artifact recommended washing hands for either 10e15 seconds e or any length of time within that range (recommended by the FDA 2013 Food Code) or 20 seconds or greater (recommended by the CDC Guidelines), the artifact was given a score of 1.
only 48% recommended minimizing contact with sick persons. Less than half (48%) addressed at least one aspect of environmental sanitation with the mean score 1.7 of 5. 3.2. Research question 2: readability The mean Flesch Reading Ease for the sample was 47.6, indicating artifacts were difficult to read as the score is <60 (D'Alessandro et al., 2001). The mean Flesch-Kincaid Grade Level was 10.4, which is more than five grade levels higher than the recommended target grade level (5th grade) for health-related materials (D'Alessandro et al., 2001). 4. Discussion In the analysis of consumer education materials, information was found to not be aligned (either omitted and/or inconsistent) with CDC guidelines suggesting consumers might not have sufficient information needed to prevent a norovirus infection. One obvious reason for the incompleteness and/or omission of information is the authors of materials in the sample might not have known about the CDC prevention guidelines, published in March
2011 in Morbidity Mortality Weekly Report (MMWR). While MMWR is in the public domain, it might not be a known or commonly used source of information for those who author consumer education materials. Thus, some authors might have used information from other sources for which the evidence base is unknown or possibly incorrect. This illustrates the need to create (and subsequently use) evidence-based guidelines as the underpinning of consumer education materials. Food safety educators could use as a model approach the evidence-based clinical practice guidelines commonly used in health care. Such guidelines minimize intelligent guesswork as they can inform practice for individual healthcare professionals in the face of mounting evidence often presented in inaccessible journals. Moreover, the process of using evidence-based guidelines would also encourage food safety educators to actively seek the best evidence to inform practices rather than maintaining a “this is how it's always been done,” mindset. However, it is important to note that even within these evidence-based guidelines, the evidence base contains gaps. One example centers around vomit and fecal matter clean-up, which none of the artifacts in the sample addressed. To disrupt the transmission of noroviruses, it is well known that vomitus and feces must be properly cleaned up then the contaminated area
H. Evans et al. / Food Control 65 (2016) 32e36
disinfected. But, the actual distance surrounding the vomit/fecal episode that one needs to disinfect is unknown as the evidence to support a geographic radius is inconclusive. The only published evidence available suggests vomit can be deposited up to 25 feet from its point source but these findings are based on one epidemiological study and one laboratory study (Evans et al., 2002; Booth, 2014). When an author is faced with incomplete information, such as not knowing how far from an episode to disinfect, they might use whatever information is available (25 feet), even if it is not grounded in science, or one might simply omit a recommendation (provide no distance) (Kardes, Posavac, Cronley, & Herr, 2008; Sanbonmatsu, Kardes, & Herr, 1992; Simmons & Lynch, 1991). It is important to minimize missing information as that could lead to poor implementation of practices, emphasizing the need for guidelines to evolve as new knowledge becomes available. To the contrary, sometimes the evidence base was complete but the information presented was not. Case in point, many materials recommended using sodium hypochlorite (bleach) solutions to treat surfaces, which is correct, but most did not state a concentration of the disinfection solution, which has clearly been established (Belliot, Lavaux, Souihel, Agnello, & Pothier, 2008; Gulati, Allwood, Hedberg, & Goyal, 2001; Jimenez & Chiang, 2006). One reason this detail was omitted could be that some authors might have presented motivational messages (emotional messages), which tend to be briefer than procedural messages (task-oriented messages), which tend to provide more details. It is important to note that message type was not analyzed as part of the study so this is an unproven assertion. Another reason for omitting information could be that some authors might have viewed recommended practices as impractical or unnecessary to implement in a home setting. One example of this was minimizing contact with sick persons, which was addressed in less than half of the artifacts in the sample. It makes sense that in a regulated food establishment workers cannot work while ill as a means to disrupt the transmission of noroviruses and other pathogens through direct (person-to-person) as well as indirect (contact with contaminated surfaces and food) routes. However, recommending that one not contact sick persons in a home setting might not be practical. One might not be able to isolate themselves from others if they are a primary caregiver for one or more household members or simply because of the physical constraints of the dwelling. Thus, more practical recommendations are needed, such as not preparing foods while sick, using disposal paper towels to dry hands instead of cloth towels, and routine cleaning and disinfecting of all high-touch surfaces while sick. Thus, information must be practical as well as accurate. One more issue that needs to be addressed is the use of government regulations in the absence of consumer-targeted regulations. This appears to be a sensible approach as the presumption is most regulations are grounded in science. However, one must exercise caution as regulations might not always be based on the most current evidence. Take for example the use of alcohol-based hand sanitizers as an alternative for hand washing, which the U.S. Food Code prohibits, and which most artifacts in the sample did not mention. In 2002 the CDC released Guidelines for Hand Hygiene in Healthcare (Boyce & Pittet, 2002) but in May 2003 the FDA prepared a written response, clearly stating the CDC guidelines could not be applied to foodservice establishments (FDA, 2003). The logic was that (1) pathogens commonly transmitted by hands in health-care settings differ from those transmitted in foodservice settings; (2) use of alcohol-based hand rubs in place of hand-washing has not been shown to reduce important foodborne pathogens, such as noroviruses, on food worker hands; and (3) types and levels of soil on the hands of health care workers differ from that on the hands of foodservice and retail food handlers. Much has been learned about
35
hand hygiene since the FDA response was published over ten years ago. For example, some studies suggest that some hand sanitizer formulations may be effective against noroviruses while others are not, so the assertion that hand rubs cannot reduce noroviruses is not necessarily true. Obviously, this particular situation is complex, and in the interest of providing a clear message, it may be unclear whether to include recommendations regarding alcohol-based hand sanitizers or to rely on government regulations. The conservative approach would be to omit the recommendation; whereas, some authors might have decided to provide the knowledge under the impression that recommending some hand cleaning behavior is better than no recommendation. Finally, along with lack of alignment with CDC guidelines, the artifacts in the sample were also not easy to read – written at a high school grade level and not at the 5th grade level as suggested. Simplifying information by writing it at a 5th grade level or lower can increase comprehension, a precursor to knowledge implementation. Presenting too much information about noroviruses and its epidemiology and using complicated words can decrease one's understanding (comprehension) of educational materials. Thus, the focus of norovirus education materials should be on how to prevent a norovirus infection and not extraneous facts about noroviruses. Even when exposed to accurate and complete information, if one cannot understand it, they cannot act on it. Educational materials are only as beneficial as the knowledge gained as a result of being able to read them.
4.1. Limitations Due to the ever changing nature of the Web, artifacts sampled during the study could change as time progresses so new materials that are better aligned with CDC guidance documents and that are easy to read might be available. Likewise, knowledge also evolves. In late 2014, the CDC published on their website five messages to help consumers prevent a norovirus infections: 1) practice proper hand hygiene; 2) take care in the kitchen; 3) do not prepare food while infected; 4) clean and disinfect contaminated surfaces; and 5) wash laundry thoroughly (CDC, 2015). However, coding and analysis had already been completed so kitchen and laundry practices were not assessed within the study, demonstrating the challenge of evaluating web-based information. Further, to analyze the readability of the artifacts in the sample, two readability formulas available through Microsoft Word were used. This method is widely used, but also not comprehensive, as it only considers factors such as word and sentence length so does not take into consideration comprehension.
5. Conclusion These authors assert that there is a need to revise and possibly create new materials about how to prevent a norovirus infection. Consumer education, if approached properly, could effectively reduce the burden of illness attributed to noroviruses. To be effective, consumer-targeted materials must be aligned with evidence-based guidelines. However, it is important that these guidelines be routinely evaluated to be certain that they are based on the best evidence that is available. Moreover, practices included in such guidelines must also be practical to be implemented in a home setting and must be presented in a manner that is easy to read. Finally, future research should assess materials using the CDC Clear Communication Index, a set of 20 items that can be used to develop as well as assess public communication products (CDC, 2014).
36
H. Evans et al. / Food Control 65 (2016) 32e36
Acknowledgments This work was funded through a grant from the U.S. Department of Agriculture, National Institute of Food and Agriculture, Agriculture and Food Research Initiative, “Building Capacity to Control Viral Foodborne Disease: A Translational, Multidisciplinary Approach,” Grant No. 2011-68003-30395. Appendix A. Supplementary data Supplementary data related to this article can be found at http:// dx.doi.org/10.1016/j.foodcont.2016.01.003. References Belliot, G., Lavaux, A., Souihel, D., Agnello, D., & Pothier, P. (2008). Use of murine norovirus as a surrogate to evaluate resistance of huma norovirus to disinfectants. Applied and Environmental Microbiology, 74(10), 3315e3318. Berland, G. K., Elliott, M. N., Morales, L. S., Algazy, J. I., Kravitz, R. L., Broder, M. S., et al. (2001). Health information on the internet: accessibility, quality, and readability in English and Spanish. Journal of the American Medical Association, 285(20), 2612e2621. Booth, C. M. (2014). Vomiting Larry: a simulated vomiting system for assessing environmental contamination from projectile vomiting related to norovirus infection. Journal of Infection Prevention, 15(5), 176e180. Boyce, J. M., & Pittet, D. (2002). Guideline for hand hygiene in health-care settings: recommendations of the healthcare infection control practices advisory committee and the HICPAC/SHEA/APIC/IDSA hand Hygiene task force. Morbidity and Mortality Weekly Report, 51(16), 1e45. Cates, S. C., Kosa, K. M., Brophy, J. E., Hall, A. J., & Fraser, A. (2015). Consumer education needed on norovirus prevention and control: findings from a nationally representative survey of U.S. adults. Journal of Food Protection, 78(3), 484e490. Centers for Disease Control and Prevention. (2014). The CDC clear communication index Accessed 27 August 2015. Available at: http://www.cdc.gov/ccindex/. Centers for Disease Control and Prevention. (2015). Preventing norovirus infection Accessed 29 October 2015. Available at: http://www.cdc.gov/norovirus/ preventing-infection.html. D'Alessandro, D. M., Kingsley, P., & Johnson-West, J. (2001). The readability of pediatric patient education materials on the World Wide Web. Archives of Pediatrics & Adolescent Medicine, 155(7), 807e812. DiClemente, R. J., Pies, C. A., Stoller, E. J., Straits, C., Olivia, G. E., Haskin, J., et al. (1989). Evaluation of school-based AIDS education curricula in San Francisco. The Journal of Sex Research, 26(2), 188e198. DuBay, W. H. (2004). The principles of readability. Costa Mesa, CA: Impact Information. Evans, M. R., Meldrum, R., Lane, W., Gardner, D., Ribeiro, C. D., Gallimore, C. I., et al. (2002). An outbreak of viral gastroenteritis following environmental contamination at a concert hall. Epidemiology and Infection, 129, 355e360. Eysenbach, G., Powell, J., Kuss, O., & Sa, E.-R. (2002). Empirical studies assessing the quality of health information for consumers on the World Wide Web: a
systematic review. Journal of the American Medical Association, 287(20), 2691e2700. Fisher, J. D., & Fisher, W. A. (1992). Changing AIDS risk behavior. Psychological Bulletin, 3, 455e474. Gulati, B. R., Allwood, P. B., Hedberg, C. W., & Goyal, S. M. (2001). Efficacy of commonly used disinfectants for the inactivation of calicivirus on strawberry, lettuce, and a food-contact surface. Journal of Food Protection, 64(9), 1430e1434. ~ aduy, P. A., Vinje , J., & Hall, A. J., Lopman, B. A., Payne, D. C., Patel, M. M., Gastan Parashar, U. D. (2013). norovirus disease in the United States. Emerging Infectious Diseases, 19(8), 1198e1205. , J., Lopman, B., Park, G. W., Yen, C., Gregoricus, N., & Parashar, U. Hall, A. J., Vinje (2011). Updated norovirus outbreak management and disease prevention guidance documents. Morbidity and Mortality Weekly Report, 60(3), 1e15. Hall, A. J., Wikswo, M. E., Pringle, K., Gould, L. H., & Parashar, U. D. (2014). Vital signs: foodborne norovirus outbreakse United States, 2009-2012. Morbidity and Mortality Weekly Report, 63(22), 491e495. Ilic, D. (2010). The role of the internet on patient knowledge management, education, and decision-making. Telemedicine and e-Health, 16(6), 664e669. Irwin, J. Y., Thyvalikakath, T., Spallek, H., Wali, T., Kerr, A. R., & Schleyer, T. (2011). English and Spanish oral cancer information on the internet: a pilot surface quality and content evaluation of oral cancer web sites. Journal of Public Health Dentistry, 71(2), 106e116. Jimenez, L., & Chiang, M. (2006). Virucidal activity of a quaternary ammonium compound disinfectant against feline calicivirus: a surrogate for norovirus. American Journal of Infection Control, 34, 269e273. Kardes, F. R., Posavac, S. S., Cronley, M. L., & Herr, P. M. (2008). Consumer inference. In C. P. Haugtvedt, P. M. Herr, & F. Kardes (Eds.), Handbook of consumer psychology. Mahwah, NJ: Lawrence Erlbaum Associates. Kessels, R. P. C. (2003). Patients' memory for medical information. Journal of the Royal Society of Medicine, 96(5), 219e222. Kutner, M., Greenberg, E., Jin, Y., & Paulsen, C. (2006). The health literacy of America's adults results from the 2003 National Assessment of Adult Literacy (NCES 2006e483). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Pew Research Center. (2014). Health fact sheet Accessed 26 August 2015. Available at: http://www.pewinternet.org/fact-sheets/health-fact-sheet/. Sanbonmatsu, D. M., Kardes, F. R., & Herr, P. M. (1992). The role of prior knowledge and missing information in multiattribute evaluation. Organizational Behavior and Human Decision Processes, 51(1), 76e91. Scallan, E., Hoeksttra, R. M., Angulo, F. J., Tauxe, R. V., Widdowson, M.-A., Roy, S. L., et al. (2011). Foodborne illness acquired in the United States e major pathogens. Emerging Infectious Diseases, 17(1), 16e22. Scullard, P., Peacock, C., & Davies, P. (2010). Googling children's health: reliability of medical advice on the internet. Archives of Disease in Childhood, 95(8), 580e582. Simmons, C. J., & Lynch, J. G. (1991). Inference effects without inference making e effects of missing information on discounting and use of presented information. Journal of Consumer Research, 17(4), 477e491. U.S. Food and Drug Administration. (2003). FDA fact sheet on hand Hygiene in retail and foodservice establishments Accessed 21 May 2012. Available at: http://www. fda.gov/Food/FoodSafety/RetailFoodProtection/ IndustryandRegulatoryAssistanceandTrainingResources/ucm135577.htm. U.S. Food and Drug Administration. (2013). Food code (DHHS publication no. PB2013110462). Alexandria, VA: U.S. Department of Commerce Technology Administration.