nutrigenetics research

Evaluation and Program Planning 46 (2014) 103–114

Contents lists available at ScienceDirect

Evaluation and Program Planning journal homepage: www.elsevier.com/locate/evalprogplan

Evaluation of a cartoon-based knowledge dissemination intervention on scientific and ethical challenges raised by nutrigenomics/nutrigenetics research Darquise Lafrenie`re, Thierry Hurlimann, Vincent Menuz 1, Be´atrice Godard * OMICS-ETHICS Research Group, Bioethics Programs, Department of Social and Preventive Medicine, School of Public Health, University of Montreal, PO Box 6128, succ. centre-ville, Montreal, Quebec H3C 3J7, Canada

A R T I C L E I N F O

A B S T R A C T

Article history: Received 25 June 2013 Received in revised form 2 June 2014 Accepted 9 June 2014 Available online 14 June 2014

Context: The push for knowledge translation on the part of health research funding agencies is significant in Canada, and many strategies have been adopted to promote the conversion of knowledge into action. In recent years, an increasing number of health researchers have been studying arts-based interventions to transform knowledge into action. This article reports on the results of an online questionnaire aimed at evaluating the effectiveness of a knowledge dissemination intervention (KDI) conveying findings from a study on the scientific and ethical challenges raised by nutrigenomics-nutrigenetics (NGx) research. The KDI was based on the use of four Web pages combining original, interactive cartoon-like illustrations accompanied by text to disseminate findings to Canadian Research Ethics Boards members, as well as to NGx researchers and researchers in ethics worldwide. Methods: Between May and October 2012, the links to the Web pages were sent in a personal email to target audience members, one thematic Web page at a time. On each thematic Web page, members of the target audience were invited to answer nine evaluation questions assessing the effectiveness of the KDI on four criteria, (i) acquisition of knowledge; (ii) change in initial understanding; (iii) generation of questions from the findings; and (iv) intent to change own practice. Findings: Response rate was low; results indicate that: (i) content of the four Web pages did not bring new knowledge to a majority of the respondents, (ii) initial understanding of the findings did not change for a majority of NGx researchers and a minority of ethics respondents, (iii) although the KDI did raise questions for respondents, it did not move them to change their practice. Conclusions: While target end-users may not feel that they actually learned from the KDI, it seems that the findings conveyed encouraged reflection and raised useful and valuable questions for them. Moreover, the evaluation of the KDI proved to be useful to gain knowledge about our target audiences’ views since respondents’ comments allowed us to improve our understanding of the disseminated knowledge as well as to modify (and hopefully improve) the content of the Web pages used for dissemination. ß 2014 Elsevier Ltd. All rights reserved.

Keywords: Information dissemination Knowledge Attitude Practice Nutrigenetics Nutrigenomics

1. Introduction Increasingly, health researchers are required by funding agencies to develop knowledge translation (KT) plans that include activities meant to disseminate their research findings among

* Corresponding author. Tel.: +1 514 343 6111x44036; fax: +1 514 343 2371. E-mail addresses: [email protected] (D. Lafrenie`re), [email protected] (T. Hurlimann), [email protected] (V. Menuz), [email protected] (B. Godard). 1 Present address: NeoHumanitas, CH-1200 Geneva, Switzerland. http://dx.doi.org/10.1016/j.evalprogplan.2014.06.002 0149-7189/ß 2014 Elsevier Ltd. All rights reserved.

various audiences. The Canadian Institutes of Health Research (CIHR), the major funding agency for health research in Canada, has proposed the most widely used definition for KT: ‘‘Knowledge translation is a dynamic and iterative process that includes the synthesis, dissemination, exchange and ethically-sound application of knowledge to improve health, provide more effective health services and products, and strengthen the health care system.’’2 There are many ways to put research into practice and many KT strategies have been proposed to increase knowledge, change attitudes or practice (Scott et al., 2012) in clinical, government and

2

About knowledge translation & commercialization – CIHR (accessed 19.11.13).

104

D. Lafrenie`re et al. / Evaluation and Program Planning 46 (2014) 103–114

research settings. In the last few years, the arts have emerged as an innovative avenue for disseminating scholarly findings (Bombard, Cox, & Semaka, 2011; Boxall & Ralph, 2009; Lafrenie`re & Cox, 2012a,b). If photographs (Lafrenie`re, Hurlimann, Menuz, & Godard, 2012; Teti, Murray, Johnson, & Binson, 2012; Vaughn, RojasGuyler, & Howell, 2008), and, to a lesser extent, films and videos (Parsons & Lavery, 2012; Rich, Lamola, & Woods, 2006; Shapiro, Tomasa, & Koff, 2009) have been explored as a visual way of presenting study results, cartoons and other forms of drawings have rarely been used for disseminating health research findings (Bartlett, 2012; Guillemin, 2004). The main objective of this article is to present the results of the evaluation of a knowledge dissemination intervention (hereafter: ‘‘KDI’’) based on the use of cartoon-like visuals, accompanied by text, and aimed at disseminating the results of a study on the scientific and ethical challenges raised by nutrigenomics-nutrigenetics (hereafter referred to as ‘‘NGx’’).

2. Background NGx – a field of research that studies the interactions between genome/gene(s) and nutrition, and their impact on health and wellbeing3 – has stimulated great expectations for future beneficial applications in public health and individuals. For instance, NGx research anticipates the prevention of chronic diseases, such as cancer and diabetes, through personalized dietary interventions, engineered/modified foods that respond to consumers’ specific genetic characteristics, and contribution to better public health (Godard and Hurlimann, 2009; Ronteltap & van Trijp, 2007). Yet, the potential achievability of such promises is not without socio-ethical considerations that challenge NGx development and implementation. Studying ethical issues relevant to personalized health interventions, spanning from clinical discoveries/innovation to publication/translation of clinical -omics research data is crucial because it can influence the development of personalized health interventions and help avoid predictable pitfalls, thus ensuring an effective and ethical application of NGx in the laboratory, in the clinic and in the evidence-based development of science policy. For that purpose, the OMICS-ETHICS Research Group4 based at the University of Montreal, Canada, conducted a research project aimed at laying an empirical foundation that could discern and anticipate the socioethical issues associated with NGx research and its potential applications (hereafter ‘‘NGx project’’).5 In the first phase of this project, an extensive analysis of 173 NGx clinical studies published between 1998 and 2007 inclusively was carried out.6 It highlighted both scientific challenges and significant ethical concerns raised by the geographical location of NGx clinical studies, by the selection of participants in NGx clinical studies, by the methodological limitations encountered in NGx research, as well as by the publication of study results and their interpretation. These study results were published in peer-reviewed journals (Hurlimann, Stenne, Menuz, & Godard, 2011; Hurlimann et al., 2014; Stenne, Hurlimann, & Godard, 2012; Stenne, Hurlimann, & Godard, 2013) and are presented and summarized in the final report of the project, accessible online.7 The findings on which this article focuses are also reported in the next section. 3 See http://www.omics-ethics.org/en/what-is-nutrigenomics (accessed 18.4.13). 4 See information about the OMICS-ETHICS Research Group on their Web site: http://www.omics-ethics.org/en (accessed 21.11.13). 5 See http://omics-ethics.org/en/research-projects-nutrigenomics (accessed 18.4.13). 6 The methodology used is available online: http://www.omics-ethics.org/docs/ news/Methodo-NGx-Project-Omics-Ethics.pdf (accessed 18.4.13). 7 See final report available online: http://omics-ethics.org/en/NGX-researchproject-results (accessed 19.11.13).

In order to disseminate these findings beyond the usual channels (i.e., scientific literature and conferences), a sub-study of the NGx project (hereafter referred to as ‘‘KDI study’’) was developed and aimed at designing, implementing and evaluating a knowledge dissemination intervention (KDI) among researchers in NGx and in ethics, as well as members of research ethics boards/ institutional research boards (REBs/IRBs). The present article describes the development of this KDI and focuses on the results of the study aimed at evaluating its effectiveness. 3. Development of the KDI and methods 3.1. Literature review Prior to developing the knowledge dissemination intervention (KDI), the members of the OMICS-ETHICS research group performed a literature review (Lafrenie`re, Menuz, Hurlimann, & Godard, 2013) to gain information on the types of KDIs used in applied research, and the evaluation of their effectiveness. It shed light on the variety of interventions, contexts, and actors involved, and highlighted the complexity of implementing KDIs. No articles described a KDI with research findings and target audiences similar to those in this KDI study. Therefore, the review did not provide information that could be transformed instantly into a readily implementable KDI for the communication of the findings of the NGx project. 3.2. Knowledge to be disseminated (message) As mentioned above, a detailed content analysis of 173 NGx clinical studies was performed in the NGx project. Various data were extracted from these publications, such as participants’ geographical location; participants’ particulars, such as race, ethnicity, origin, nationality, ancestry, age, sex, comorbidities, and any other available data linked to participants’ description, as well as any exclusion and inclusion criteria reported by the authors. The following elements were also extracted from the publications of our sample: all authors’ statements or comments about the potential or actual impact of genetic variations linked to ethnicity, and all limitations of the study results explicitly acknowledged and reported by the authors.8 While the results of this analysis do not question the validity or relevance of NGx research, they deserved wide dissemination among stakeholders in the field of NGx, as they raise both scientific and ethical concerns, in four pivotal themes: 1. A majority of the NGx clinical studies in our sample focus on ‘‘white’’ or ‘‘Caucasian’’ populations and that there is little coherence in the way the terms race/ethnicity/ancestry/origin and even nationality are used to describe sample populations. Information about the participants’ origin could be found in 124 publications (72%), thus it could not be determined for 49 studies (28%) of our sample. Yet, ethnicity matters in NGx research as it is of the utmost importance to consider the influence of acculturation on diet and health, and the impact of genetic admixture in populations and genomic variability on research results. In such circumstances, can it be inferred to what populations study findings can apply if the ethnicity of participants is not explicitly reported in publications? Moreover, what can be the impact of a lack of representation of ethnic minorities in NGx clinical research, in terms of fairness and equity? (http://omics-ethics.org/en/nutrigenomics-ethnicity; accessed April 2013) 8 A complete description of the methodology used is available online: http:// www.omics-ethics.org/docs/news/Methodo-NGx-Project-Omics-Ethics.pdf (accessed 18.4.13).

D. Lafrenie`re et al. / Evaluation and Program Planning 46 (2014) 103–114

2. A majority of the NGx clinical studies in our sample focus on middle-aged populations. Less than 10 studies in our whole sample (n = 173) focused on people <20 years old and no study focused on people 75 years old. Age matters in NGx research as it is of the utmost importance to consider the variations of gene expression and of dietary needs across life span. In such circumstances, does a lack of representation of the elderly and minors in NGx research limit the possibilities to generalize study findings to these populations? Is this an issue in terms of justice and equity? (http://omics-ethics.org/en/nutrigenomicsage; accessed April 2013) 3. Most studies in our sample were conducted in Europe (42%) and North America (39%). The other studies were conducted with participants in Asia (17%) and Central America (5%). No NGx study in our sample enrolled participants located in Africa, South America and Oceania. Yet, in order to achieve its goals and promote optimal health on a global scale, NGx research should also be conducted in developing countries, with local participants and local foodstuffs. Indeed, the usefulness and validity of research results obtained solely in industrialized countries may be limited to these countries. Is this an issue in terms of justice and equity? (http://omics-ethics.org/en/nutrigenomics-location; accessed April 2013) 4. One of the goals of the NGx project was to investigate whether scientific articles could contribute to feeding a biohype phenomenon – defined in the present article as ‘‘a phenomenon in which there is a gap between the expectations associated with the development of a genetically derived technology/field of research and the practical applications that may realistically result from it’’ (Stenne et al., 2012, p. 286). Our analysis show that a minority (9%) of the publications in our sample contained explicit claims about broader implications of research findings, i.e., clinical applications extrapolated from study results, despite significant methodological limitations. Moreover, all these limitations were not always reported in these publications. While these results show that published NGx clinical studies do not appear as a primary source of biohype, they deserve wide dissemination among stakeholders in the field of NGx, in order to raise their awareness about the potential risks associated with the underreporting of methodological limitations and premature claims about the benefits of NGx in scientific publications. (http://omics-ethics.org/en/nutrigenomics-biohype; accessed April 2013).

3.3. Target audiences Two main target audiences were identified. First, researchers play a critical role in the selection of participants in NGx clinical studies as well as in the interpretation and publication of study results. They are at the forefront of the development of NGx, and, as such, constituted a crucial target audience for the communication of results pertaining to the scientific and ethical challenges raised by NGx clinical research. Secondly, the NGx project sought to identify potential ethical issues in NGx research. Thus, researchers who study ethics in the field of genetics/genomics and members of research ethics boards (REBs), who may be called upon to review NGx protocols were also considered as a relevant target audience. Five-hundred seventy-six (576) researchers in NGx were identified as follows: they were (a) first or corresponding authors in NGx clinical studies published between 1998 to 2011 inclusively, (b) members of societies, associations and other research groups identified as NGx experts through an Internet Google search using the keywords ‘‘nutrigenetics’’ or ‘‘nutrigenomics’’ (e.g., International Society of Nutrigenetics and Nutrigenomics, NuGo, Nutrigenomics Consortium Netherlands, Nutrigenomik

105

Deutschland), or their own website; (c) selected from among the speakers of the 4th Congress of the International Society of Nutrigenetics and Nutrigenomics; or (d) first or corresponding authors of articles on gene and food/nutrient interactions published in the Journal of Nutrigenetics and Nutrigenomics; or (e) corresponding authors of articles in the journals Gene and Nutrition and Molecular Nutrition and Food Research, identified through a search with the keywords ‘‘nutrigenetics’’ or ‘‘nutrigenomics’’ within these 2 journals. Ethics researchers were identified using ‘‘genetics’’ or ‘‘genomics’’ and ‘‘ethics’’ as keywords in Internet searches. Members of REBs were identified through the Canadian Association of Research Ethics Boards (CAREB). We limited our search to Canada for this specific subset of our target audience because it would be very difficult to identify members of REBs abroad; each jurisdiction has its own system of ethical review of research. This could limit the generalization of our results. When members’ names were not publicly available on a REB’s website, coordinators were contacted and asked to forward KDI information (i.e., Web hyperlinks) to their members. Overall, 294 ethics researchers or REB members were identified. Members of the target audience, whether active in NGx or in ethics, were geographically dispersed and worked in various research areas. 3.4. Media A strategy based on the use of four Web pages combining original, interactive cartoon-like illustrations accompanied by text was selected as a means to convey findings from the NGx study to both target audiences. Empirical evidence on the impact of visuals is relatively scarce in the social sciences (Joffe, 2008), but some studies have been conducted to assess their effectiveness in health disciplines, although not as a method for disseminating research findings. While some researchers concluded that the use of visuals had little impact on attitudes, beliefs or acquisition of knowledge (Gillies, Stork, & Bretman, 1990), more recent studies showed that visuals were more effective than text to convey a message (Delp & Jones, 2006; Gallopel-Morvan, Gabriel, Le Gall-Ely, Rieunier, & Urien, 2011; Hammond, Fong, Mcdonald, Brown, & Cameron, 2006; Houts, Witmerb, Egethc, Loscalzoa, & Zabora, 2001; Tait, VoepelLewis, Zikmund-Fisher, & Fagerlin, 2010). Various reasons are provided in the literature to explain why visuals might be an effective means for conveying information. Some authors claim that the power of visual messages comes from their ability to convey an immediate, visceral understanding, in ways that conventional texts cannot. They argue that reading text and viewing images activate different information processing systems within the brain (Green & Myers, 2010). Some people seem to be better at processing words, others at processing images (Mayer & Massa, 2003). Still other authors consider that visuals allow for the crossing of cultural barriers, an understanding both of the complexity of individual decision-making and organizational influences on the one hand, and of the universality of dilemmas and values, on the other (Gerde & Foster, 2008). Visuals appeal to emotions whereas textual/verbal materials are more likely to lead to a more rational, logical and linear pathway of thought. Visual materials serve as an intermediary, as the reader can attend to emotionally charged topics in an environment removed from them (Yang, 2013). This emotive quality of visuals makes them more vivid and memorable (Joffe, 2008). Visuals employ a language that is almost universally understood (Sones, 1944), not age- (McAllister, 1992), numeracy-, nor literacy-dependent (Roberts, Ghiassi, & Partridge, 2008). Cartoons, more specifically, are effective means of communication because they are visually engaging (Bartlett, 2012), can assist in clarifying thinking and initiating dialogue about various ideas and beliefs (Lockyer & Pickering, 2009), as well as being useful to illustrate and challenge cultural norms

D. Lafrenie`re et al. / Evaluation and Program Planning 46 (2014) 103–114

106

(Prosser, 1998). Thus, cartoons as a representation method was considered a valuable option for conveying findings from the NGx project. Cartoons had to be coupled with a channel of communication that could reach the target audience right where it was. The Web seemed to be the best solution to achieve this objective, although some points would have to be carefully thought-through. Graphics (e.g., drawings, photos, cartoons) play a key role in enhancing the appearance of Web pages and getting visitors’ attention (Noiwan & Norcio, 2006). Virtual visitors usually look at the graphics of a Web page before even reading the text (Lin, Yeh, & Wei, 2013). However, the visual density of a Web page needs to be well-balanced to ascertain its overall aesthetics (Saade & Otrakji, 2007). Aesthetics is important as it impacts directly on appeal and thus on the visitors’ feelings about the Web page (Cheng & Patterson, 2007; Ou & Sia, 2010). The need to take into account visual density limits the space available to provide detailed textual explanations. In addition to the aesthetic aspect of the design of the Web pages, other elements had to be considered. For instance, previous research has shown that long pages can be disorienting since they require lengthy scrolling and remembering information that is already off-screen (Price & Price, 2009). Memory seems to be reinforced by the fact that the information stays visible as the virtual visitor reads through the page (Price & Price, 2009). Thus, it is recommended to organize the message within the layout of a printed page, using concise and well-organized text (Kilian, 2009; Price & Price, 2009). With these recommendations in mind, the KDI was developed. Each Web page featured a cartoon illustrating one of the main themes highlighted in the research results, i.e., ethnicity, age, geographical location, and biohype.9 The cartoons were created by one of the research team members (Vincent Menuz) who had previously published this form of art. The approach adopted in the design of the Web pages was one where the cartoons were subordinate to the text and served merely to illustrate the written material (McCloud, 1993). Therefore, the meaning was carried in the verbal structure, and the images were used mainly to catch the reader’s attention, add emotional resonance, and illustrate points to consider that were also described in the text. When moving the cursor over megaphone-shaped icons located at various positions on the cartoon, information about the NGx project results on that theme popped up on the same Web page (pop-ups on ‘‘mouseover’’). Below the cartoon, a ‘‘Food for thought’’ component was included and presented the potential scientific and ethical issues associated with the findings on that page’s theme, as described into the section b above, ‘‘Knowledge to be disseminated’’. This section was followed by a short survey, further described below in the Evaluation section, aimed at assessing the effectiveness of our KDI. The Web pages, including questions, were pre-tested regarding format and content with one group of scientists and another one of ethicists prior to undertaking the evaluation of the KDI. 4. Evaluation of the KDI and methods 4.1. Dissemination process Following approval by the University of Montreal Health Research Ethics Board, the link to the Web pages was sent in a personal email to the professional address of the target audience members, one thematic Web page at a time. This strategy was based on the assumption that the recipients might not read everything if knowledge to be disseminated arrived all at once. The first email was sent on May 23, 2012, using MailChimp, an email list manager. An ad hoc verification among people who were not 9

See http://omics-ethics.org/en/NGX-research-index (accessed 18.4.13).

part of our sample suggested that some of the targeted participants might not have received this first email due to a technical problem. Thereafter, links to the following three Web pages were sent to each recipient using standard email, on June 20, September 10, and October 3, 2012. The dates of mailing were mainly dictated by the Canadian university calendar (e.g., no email was sent during summer break in July and August). The first and second mailings were followed by a reminder two weeks later. No reminder was sent for the last two Web pages in order to avoid annoying recipients, since they had already received four emails for the first two Web pages, and they were to receive two more for the last two Web pages. The emails briefly described the study and the KDI, with the last three including the previous links. Recipients were asked to answer the evaluation questions contained on the Web page. The four Web pages were ultimately posted on the research team’s website (OMICS-ETHICS), but only upon completion of the KDI study. 4.2. Evaluation of the effectiveness of the KDI Each Web page included a series of nine questions that the members of the two target audiences were invited to fill out (Table 1). The first four questions were designed to assess the effectiveness of the KDI according to four criteria: (i) acquisition of knowledge; (ii) change in initial understanding; (iii) generation of questions from the findings, and (iv) intent to change one’s own practice. These questions were based on the four questions identified in the Guiding Arts-Based Research Assessment (GABRA) framework developed by Lafrenie`re and Cox (2012a,b) to assess the quality and effectiveness of arts-based works used to represent research findings. The fifth question sought to evaluate whether participants considered that the means of dissemination used (i.e., interactive Web pages with cartoons and texts) was effective in conveying the results of the NGx project to them. These five questions were closed, ‘‘Yes/No questions’’, with a space for comments after each one to provide respondents with an opportunity to comment without any restrictions. The last four questions collected information on the demographic characteristics of the respondents, i.e., current status/position, age group, gender, and field of work. The responses to the questions were anonymous and IP addresses of the respondents were not used nor communicated to the research team by the webmaster. Average time to complete the questionnaire was approximately five minutes. 4.3. Data analysis The data analysis performed in the evaluation of the KDI used mixed methods, quantitative and qualitative. As per the quantitative analysis, the online questionnaire included a limited number of questions. The answers to the close yes/no/somewhat questions were transferred to an Excel spreadsheet and analyzed. As regards qualitative analysis, open-ended statements were also transferred to a separate Excel spreadsheet and analyzed for content. A directed approach to qualitative content analysis (Hsieh & Shannon, 2005) was used given that questions were pre-determined and inspired by the Guiding Arts-Based Research Assessment meta-framework (Lafrenie`re & Cox, 2012a,b). According to this framework, an artistic form of representation of study results is effective if it contributes to acquisition of knowledge, generates emotions, raises questions and moves to change initial understanding or practice. In accordance with this qualitative approach, excerpts were selected to illustrate the broad range of comments provided by respondents and their contributions were semi-quantified when reported (Anderson, 2010).

D. Lafrenie`re et al. / Evaluation and Program Planning 46 (2014) 103–114

107

Table 1 Questions for the evaluation of the cartoon-based knowledge dissemination intervention. Ethnicity

Age

Q1. Prior to reading this Web page. . . . . . did you know that NGx research focused on . . . did you know that NGx research enrolled White/Caucasians in such middle-aged populations in such proportions? proportions?

Geographical location

Biohype

. . . did you know that NGx research was concentrated in such proportions in Europe and North America?

. . . did you know that NGx research contained explicit statements that attributed a clinical significance to study results in such proportions?

. . . concentration of NGx research in Europe and North America change your initial understanding about the impact of geographical location of NGx research on justice and equity?

. . . extrapolation of study results and the reporting of potential benefits associated with the development of NGx in scientific publications change your initial understanding about the impact of biohype in NGx research on the responsibility of researchers?

. . . concentration of NGx research in Europe and North America raise questions or comments to you?

. . . extrapolation of study results and the reporting of potential benefits associated with the development of NGx in scientific publications raise questions or comments to you?

. . . concentration of NGx research in Europe and North America change anything in your practice?

. . . extrapolation of study results and the reporting of potential benefits associated with the development of NGx in scientific publications change anything in your practice?

&Yes &No &Somewhat Comments/Suggestions: ______________________ Q2. Does this finding about. . . . . . under-representation of ethnic minorities in NGx research change your initial understanding about the impact of lack of representation of ethnic minorities on justice and equity in NGx research?

. . . over-representation of middle-aged populations in NGx research change your initial understanding about the impact of lack of representation of the elderly or minors on justice and equity in NGx research?

&Yes &No &Somewhat Comments/Suggestions: ______________________ Q3. Does this finding about. . . . . . underrepresentation of ethnic minorities in NGx research raise questions or comments to you?

. . . underrepresentation of the elderly and minors in NGx research raise questions or comments to you?

&Yes (please comment) &No &Somewhat Comments/Suggestions: ______________________ Q4. Will this information about. . . . . . underrepresentation of ethnic minorities in NGx research change anything in your practice?

Q5.

Q6.

Q7.

Q8.

Q9.

. . . underrepresentation of the elderly and minors in NGx research change anything in your practice?

&Yes &No &Somewhat Comments/Suggestions: ______________________ Did this format (e.g., illustration and texts on the Internet) convey our research findings on NGx research effectively to you? &Yes &No &Somewhat &Comments/Suggestions: ______________________ What is your current position/status? &PhD student &Postdoctoral fellow &Assistant Professor/Researcher &Associate Professor/Researcher &Professor/Researcher &MD &Other Specify: ______________________ To which age group do you belong to? &30 and under &31–40 &41–50 &51–60 &61 and over What is your gender? &Female &Male In what field do you work? &Nutrigenomics/nutrigenetics/nutrition &Ethics &Other Specify: ______________________

D. Lafrenie`re et al. / Evaluation and Program Planning 46 (2014) 103–114

108

Table 3 Response rates.

5. Results The majority of respondents were female professors/researchers between 31 and 50 years of age who worked in the field of NGx. Table 2 provides detailed information on the demographic profile of respondents for each Web page. The details are provided for each thematic page since respondents can differ for each page (i.e., one individual might have answered the questionnaire for the Ethnicity Web page, but not for the three other themes; another might have answered three out of the four thematic questionnaires). 5.1. Response rates There are many ways of defining and calculating response rates to Web surveys (Eysenbach, 2004; Shih & Fan, 2008). In this article, the definition and methodology promoted by Eysenbach were chosen. Accordingly, the response rate in our study was computed as the ratio of those who agreed to participate in the KDI divided by unique, first survey page visitors. Table 3 shows (A) the number of individuals who received an invitation to participate in the KDI; (B) the number of recipients of the invitation who actually accessed the Web pages; (C) the number of recipients who answered the evaluation questions for each Web page: ethnicity, age, geographical location or biohype; and (D) the response rate, which was arrived at by dividing the number of recipients who answered the evaluation questions for each Web page (C) by the number of recipients who accessed that Web page (B). The number of respondents who answered the evaluation questions went from 43 for the first Web page, to 35 for the second, nine for the third, and up slightly to 13 for the last page. 5.2. Results of the KDI effectiveness questions for the four Web pages (Q1–Q5) Results for each question aimed at assessing the effectiveness of the KDI (Questions 1–5, Table 1) are presented separately for each Web page and for each category of the target audience, namely

Table 2 Respondents’ characteristics. Web Page Web Page about about Ethnicity Age (n) (n)

Web Page about Geographical location (n)

Web Page about Biohype (n)

Current position/status PhD student 4 Postdoctoral fellow 3 Assist. Professor/Researcher 7 Assoc. Professor/Researcher 8 Professor/Researcher 12 MD 4 Other 5

2 6 2 4 12 2 7

1 2 3 0 0 1 2

0 1 2 1 5 1 3

Age group 30 and under 31–40 41–50 51–60 61 and over

5 12 14 8 4

2 9 10 11 3

1 5 2 1 0

1 3 3 3 3

Gender Female Male

26 17

18 17

5 4

6 7

32

18

5

9

5 6

6 11

3 1

2 2

Field of work Nutrigenomics/ nutrigenetics/nutrition Ethics Other

A Researchers and Research Ethics Board members contacted (n)a

B Recipients who accessed the Web pages (n)

C Respondents (n)

D Response rate (C/B) (%)

Email #1 NGxb Ethicsc Total

(Ethnicity) 23/05/2012 576 122 (21.2%) 294 59 (20.1%) 870 181 (20.1%)

31 12 43

25.4% 20.3% 23.8%

Email #2 NGx Ethics Total

(Age) 20/06/2012 567 277 844

25 10 35

29.1% 16.4% 23.8%

Email #3 NGx Ethics Total

(Location) 10/09/2012 561 36 (6.4%) 278 19 (6.8%) 839 55 (6.6%)

6 3 9

16.7% 15.8% 16.4%

Email #4 NGx Ethics Total

(Biohype) 03/10/2012 559 24 (4.3%) 274 15 (5.5%) 833 39 (4.7%)

10 3 13

41.7% 20.0% 33.3%

86 (15.2%) 61 (22.0%) 147 (17.4%)

a The number of potential respondents contacted varied for each Web page given bounces, individuals who unsubscribed from our list of recipients, leaves (e.g., sabbatical and maternity leaves), etc. b NGx: Researchers in NGx. c Ethics: Researchers in ethics or REB members.

NGx researchers (hereafter ‘‘NGx respondents’’), and ethics researchers as well as members of REBs (hereafter ‘‘ethics respondents’’). It must be noted that each aspect of the effectiveness of the KDI (i.e., (i) acquisition of knowledge – Q1; (ii) change in initial understanding – Q2; (iii) generation of questions on the findings – Q3; and (iv) intent to change own practice – Q4) was assessed not only by analysing the answers provided to each respective question separately, but also by taking into account all respondents’ explicit comments in their answers to any question of the survey. The first evaluation question aimed at assessing whether respondents acquired new knowledge after being exposed to the KDI (Table 4, Q1). A majority of the respondents considered that they already knew about the information provided on the Web pages. More than three quarters of the NGx respondents stated that they were familiar with the findings displayed on each Web page. Also, there were significantly more NGx than ethics respondents who knew about the results contained in the KDI (at least three times more). However, some of the latter explicitly stated that the findings confirmed what they thought intuitively. This is what I thought, but your publication is confirming it. (Ethics respondent/Ethnicity thematic Web page) [Translated from French to English] I did not ‘‘know’’ it [NGx research] was [geographically] concentrated but I supposed it was. (Ethics respondent/Geographic location thematic Web page) The second question (Table 5, Q2) aimed to evaluate changes in the initial understanding of the respondents regarding the research findings for each theme. A minority of NGx respondents considered that the thematic Web pages changed their initial understanding of the matters addressed within these pages. Conversely, a majority of respondents in the ethics group answered that their initial understanding had

D. Lafrenie`re et al. / Evaluation and Program Planning 46 (2014) 103–114

109

Table 4 Question 1 – Acquisition of knowledge. Age

Ethnicity

Geographical location

Biohype

Q1. Prior to reading this Web page, did you know that. . .

NGx Ethics Total

. . . NGx research enrolled White/ Caucasians in such proportions?

. . . NGx research focused on middle-aged populations in such proportions?

. . . NGx research was concentrated in such proportions in Europe and North America?

. . . that NGx research contained explicit statements that attributed a clinical significance to study results in such proportions?

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

25 6 31

6 6 12

31 12 43

20 5 25

5 5 10

25 10 35

6 2 8

0 1 1

6 3 9

9 1 10

1 2 3

10 3 13

been modified for three out of the four themes: ethnicity, age, and geographical location. Overall, the information provided on the age of the research participants in NGx studies and on the potential impact of an underrepresentation of children and the elderly in NGx research was the theme that had the most impact on changing understanding. Eighteen out of 35 respondents indicated that the results modified their understanding on this issue. Very few respondents specified how their initial understanding changed thanks to the KDI. However, one respondent mentioned that the content of the ethnicity Web page ‘‘. . . heighten[ed]. . . [his] concerns about the validity of the studies’’ (Ethics respondent/Ethnicity thematic Web page). The third question (Table 6, Q3) sought to determine whether the information provided on the Web pages raised questions among respondents on the themes covered in the KDI. The content of the Web pages raised questions for a majority of NGx respondents, except for the ‘‘geographical location’’ Web page, for which responses were equally divided. All ethics respondents answered that the Web pages addressing age, geographical location and biohype raised questions for them. Nine out of 12 answered the same for the Web page on ethnicity. The Web page pertaining to the age of participants in NGx research generated questions among the greatest number of both NGx and ethics respondents: 28 out of 35 participants. All respondents had the opportunity to explicitly specify the questions that the Web page content raised for them. Yet, a minority of respondents actually did note their questions. Explicit reporting came first and foremost from NGx researchers and mostly pertained to ethnicity, geographical location, and biohype themes. While ethics respondents stated in a larger number that the findings raised questions for them, they were the ones who asked the fewest questions. The Web page addressing issues raised by ethnicity in NGx research made some NGx respondents

explicitly wonder to what extent NGx study results could be applied to people from all ethnic backgrounds or whether on-line personalized dietary recommendations would be appropriate for ‘‘non-white’’ customers, such as this particular respondent: Very much so! There are more and more personalized dietary recommendations services available on-line (e.g., Siona, 23 and me) that raise ethical questions, such as ethnicity. Even if the gene-nutriment interaction has been replicated in several studies and seem to be credible, how should one interpret results for a customer that is not ‘White’? Should we suggest to him up-front not to go through testing? (NGx respondent/Ethnicity thematic Web page) [Translated from French to English] One respondent asked if ethical considerations could be the reason why NGx researchers do not always clearly report the ethnic background of the research participants in their studies: This raises the question whether there is a perception in the NGx scientific community that ethnicity (similar to race) should not be reported for ethical reasons (e.g., discrimination, stigmatization). (Ethics respondent/Ethnicity thematic Web page) The results about the age of participants in NGx research generated a question about the underrepresentation of children. An NGx respondent wondered whether the literature review such as performed in the NGx project in order to identify NGx clinical studies did not exclude studies in which children were actually recruited but that were labelled differently than NGx:

Table 5 Question 2 – Change in initial understanding. Age

Geographical location

Biohype

. . . overrepresentation of White/ Caucasian populations in NGx research change your initial understanding about the impact of lack of representation of ethnic minorities on justice and equity in NGx research?

. . . overrepresentation of middleaged populations in NGx research change your initial understanding about the impact of lack of representation of the elderly or minors on justice and equity in NGx research?

. . . concentration of NGx research in Europe and North America change your initial understanding about the impact of geographical location of NGx research on justice and equity?

. . . extrapolation of study results and the reporting of potential benefits associated with the development of NGx in scientific publications change your initial understanding about the impact of biohype in NGx research on the responsibility of researchers?

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

7 8 15

24 4 28

31 12 43

11 7 18

14 3 17

25 10 35

1 3 4

5 0 5

6 3 9

4 0 4

6 3 9

10 3 13

Ethnicity Q2. Does this finding about. . .

NGx Ethics Total

D. Lafrenie`re et al. / Evaluation and Program Planning 46 (2014) 103–114

110

Table 6 Question 3 – Generation of questions from the findings. Age

Geographical location

Biohype

. . .underrepresentation of ethnic minorities in NGx research raise questions or comments to you?

. . .underrepresentation of the elderly and minors in NGx research raise questions or comments to you?

. . .concentration of NGx research in Europe and North America raise questions or comments to you?

. . .extrapolation of study results and the reporting of potential benefits associated with the development of NGx in scientific publications raise questions or comments to you?

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

22 9 31

9 3 12

31 12 43

18 10 28

7 0 7

25 10 35

3 3 6

3 0 3

6 3 9

6 3 9

4 0 4

10 3 13

Ethnicity Q3. Does this finding about. . .

NGx Ethics Total

Nutrigenomics studies are usually directed to health maintenance, disease development, for that this is the right age group. Comparable studies in youngsters will rather be labelled nutritional development studies and such. I agree that the NGx label for these would make sense but I am not convinced they don’t exist. (NGx respondent/Age thematic Web page) The relevance of conducting NGx studies in certain parts of the world where basic nutritional needs are not met was also questioned: Not sure how nutrigenomics will be useful in developing countries since the problems are related to meeting their basic needs (rather than optimizing health). Should the research/intervention budget in developing countries be spent on nutrigenomics or for meeting their basic nutritional needs?

impact them. One REB member stated that he would pay more attention and require justifications when reviewing exclusion or inclusion criteria in NGx clinical research protocols. Another respondent working in the field of ethics stressed that he would change his approach to reading and reviewing journal articles in the future. An NGx researcher expressed his intention to recruit ‘‘a good mix of participants in terms of demographics’’ in his future studies. Some respondents claimed that actions had already been taken to change practices and solve some of the issues that were identified in the Web pages. They emphasized that those issues were not unique to NGx research. Referring to the underrepresentation of children and of the elderly in NGx research, one respondent notably mentioned: Yes, unfortunately that is the case for almost all types of health research. We are working to try to make that less the case!

(NGx respondent/Geographical location thematic Web

(NGx respondent/Ethnicity thematic Web page)

page) Another NGx researcher raised questions about the standards that should be implemented to avoid biohype and the risks generated by premature or exaggerated claims about the potential benefits of NGx: What are the standards to ensure that reported claims are not overstated? (NGx respondent/Biohype thematic Web page) The fourth question (Table 7, Q4) aimed to find out if respondents intended to change their practice after being exposed to the KDI. The majority of NGx and ethics respondents did not intend to change anything in their practice following the KDI. Only three respondents explicitly mentioned how some of the findings would

Another respondent seemed to feel that the authors of the KDI were blaming NGx researchers for the scientific and ethical issues reported on the Web page on ethnicity (e.g., underrepresentation of ethnic minorities in research) and stressed the fact that changes in practices were already underway: We are already working on it! You make us (nutrigenomics researchers) out to be ethnoculturally prejudice [sic] or something–many of us are doing the best we can–it takes years to start a new study sometimes! (NGx respondent/Ethnicity thematic Web page) Some respondents suggested arguments and explanations as to why current research practices did not change. One respondent

Table 7 Question 4 – Intent to change own practice. Age

Geographical location

Biohype

. . . underrepresentation of ethnic minorities in NGx research change anything in your practice?

. . . underrepresentation of the elderly and minors in NGx research change anything in your practice?

. . . concentration of NGx research in Europe and North America change anything in your practice?

. . . extrapolation of study results and the reporting of potential benefits associated with the development of NGx in scientific publications change anything in your practice?

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

11 4 15

20 8 28

31 12 43

10 3 13

15 7 22

25 10 35

1 1 2

5 2 7

6 3 9

2 0 2

8 3 11

10 3 13

Ethnicity Q4. Will this information about. . .

NGx Ethics Total

D. Lafrenie`re et al. / Evaluation and Program Planning 46 (2014) 103–114

discussed the results provided on the Web page about ethnicity as follows: Lack of representation causing loophole on the impact/effectiveness on the minority group, but on the other hand whenever a study was conducted on minority group often there was an accusation that quickly flair negative prejudice reactions that they have been unfairly exposed on a test/clinical trial; and in certain cases there is a risk result can easily be linked to a small population group and may cause a negative consequence for that population.

111

world likely resulted from the non-publication of NGx studies conducted in these areas. According to this researcher, the situation would correct by itself over time: The bias is probably to a large extent caused by NGX studies from other continents not being published. So, more a publication than a research bias. That will probably self-correct over time. (NGx respondent–NGx/Ethnicity thematic Web page)

(Ethics respondent/Ethnicity thematic Web page) Other respondents explained why they could not change their practices. Financial limitations or commercial interests, as well as limitations linked to certain methodological study designs were mentioned as obstacles to actual changes in the practice of NGx research, as was the difficulty to include ethnic minorities in research carried out in ethnically homogenous regions. Similarly, practical considerations linked to the geographical location of NGx studies and the demographic characteristics of their populations were invoked. For instance, this researcher explained that he could not recruit people aged 75 years or more in a country where a majority did not reach that age: In [name of country removed] where our study was based, the average longevity of the population is less than in the western countries. (NGx respondent/Age thematic Web page) The scientific rationale of including children or old people in NGx research protocols focusing on diseases that mainly affect middle-aged populations was also questioned by respondents who mentioned that sometimes, there is no point in changing practices by including different age groups: The focus is on disease (diabetes and metabolic syndrome) development for that this [i.e., middle-aged populations] is precisely the right age groups.

Two respondents explicitly expressed their disagreement with the results presented on the biohype thematic Web page and their interpretation. One referred to the article published by the research team and reporting the detailed results and complete methodology summarized on the ‘‘biohype’’ Web page: in this respondent’s viewpoint, the number of publications (16 out of 173) that contained explicit claims about clinical NGx applications based on the extrapolation of study findings was not significant, and such claims were legitimate: [. . .] the reason why the research is done is precisely to look for clinical significance so the presence of these statements [i.e., claims about potential benefits of NGx in scientific publications] is not a surprise. (Ethics respondent/Biohype thematic Web page) Another NGx respondent stated that it should be left to the research community only to interpret claims made in scientific publications, and that the way such claims could be used by media was irrelevant: [. . .] it surely must be expected – claims will be made. The research papers are not aimed at lay people – scientists read the claims and can read/assess the quality of the research and decide about the strength of the claims. The fact that they are sometimes picked up by the media is irrelevant. (NGx respondent/Biohype thematic Web page)

(NGx respondent/Age thematic Web page)

I do heart disease nutrigenomics research - I will never be able to use children. And I get in trouble if I use adults that are too old because of genetic survival issues. . .As I said, there can be issues with researching the young (i.e., if you do CVD research, well kids don’t have heart attacks) and also the very old (people with the genes of interest have already passed on or are too ill to study).

Finally, the fifth question (Table 8, Q5) sought to inquire if the media (Web pages) and format (cartoon-like illustrations) used to communicate the results of the NGx project were perceived as being effective by the respondents. A large majority of both the NGx and ethics respondents thought that the means used in this KDI (Web pages with cartoons and text) were effective to communicate the results of our NGx project. Some respondents, both in NGx and ethics, explicitly commented on the positive aspects associated with the use of a cartoon-based KDI, notably stressing that the Web pages were ‘‘entertaining and stimulating’’, like this one:

(NGx respondent/Age thematic Web page) Finally, a few respondents questioned some of the NGx project results presented in this KDI, and, consequently, the need or the necessity to change practices. A respondent commented on the Web page about the location of NGx clinical studies and suggested that the apparent absence of NGx research in certain parts of the

It is appealing and fun. It certainly can encourage participation. I think this interactive amusing illustrations with short sentences and relevant information increase participant self-perception of competency regarding the information presented. (Ethics respondent/Ethnicity thematic Web page)

Table 8 Question 5 – Effectiveness in conveying research findings. Ethnicity

Age

Geographical location

Biohype

Q5. Did this format (e.g., illustration and texts on the Internet) convey our research findings on NGx research effectively to you?

NGx Ethics Total

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

Yes (n)

No (n)

Total (n)

27 11 38

4 1 5

31 12 43

24 10 34

1 0 1

25 10 35

4 3 7

2 0 2

6 3 9

8 3 11

2 0 2

10 3 13

D. Lafrenie`re et al. / Evaluation and Program Planning 46 (2014) 103–114

112

Nonetheless, a respondent nuanced the above comment: Clever and useful way to deliver information, but as with all ‘sound bites’ it doesn’t tell the whole story. (NGx respondent/Ethnicity thematic Web page) 6. Discussion There is a dearth of publications reporting on the evaluation of the effectiveness of KDIs in the knowledge translation (Colquhoun, Letts, Law, MacDermid, & Missiuna, 2010; LaRocca, Yost, Dobbins, Cliska, & Butt, 2012; McLean et al., 2012) and arts-based health research literature (Lafrenie`re and Cox, 2012a,b; Parsons, Heus, Moravac 2013). However, researchers who decide to evaluate the effectiveness of their KDI encounter several challenges, be they conceptual, methodological (Bhattacharyya, Estey and Zwarenstein 2011; Holmes, Scarrow, & Schellenberg, 2012) or financial (Joseph, 2013; Nixon et al., 2012). The advancement of knowledge in the field of KT makes the complexity of KDIs more and more apparent (Strauss, Tetroe and Graham 2009), which in turn requires increasingly complex evaluations (Sargeant et al., 2011). Another difficulty associated with the evaluation of KDIs refers to low response rates (Weymann, Harter and Dirmaier 2013; Sauermann & Roach, 2013; Wyatt, 2000). Response rates to Web surveys vary widely and depend on a variety of factors (e.g., characteristics of the audience, purpose of the survey, perceived benefit from participating in survey, incentive, length, convenience). The range is especially wide when different audiences are targeted. For instance, in a study where the audience included college and university students, health educators, employees, consumers, librarians, U.S. military and spouses, Schonlau, Fricker, & Elliot (2002) obtained a response rate that varied from 7–44 percent. In a meta-analysis study where the audiences were also broad, including college students, professionals, employees, and the general public, Shih and Fan (2008) obtained a range of 7–88 percent. In another study surveying ethics committee members, clinical investigators and sponsors, Dainesi and Goldbaum (2012) came to a response rate between 21 and 45%. As shown in Table 3, the response rate to our KDI evaluation ranked near the lowest limit of this range. Nevertheless, our findings highlight important points reflecting the difficulties that may occur in the development and evaluation of a KDI. They also shed light on the potential benefits that can result from KDIs for those who disseminate research results. 6.1. Effectiveness of the KDI carried out Questions included on the Web pages were designed to assess the effectiveness of the KDI on four criteria (i) acquisition of knowledge; (ii) change in initial understanding; (iii) generation of questions from the findings; and (iv) intent to change own practice. According to NGx respondents, our KDI was not effective in bringing new knowledge to them: a majority of them stated that they already knew about the findings provided in the four Web pages (see Table 4). Conversely, at least half of the ethics respondents stated that they acquired new knowledge, except for the findings described in the ‘‘geographical location’’ thematic Web page. These results may not be surprising since the knowledge disseminated in this KDI originated from an analysis of clinical NGx studies and thus described facts and practices that may be better known and understood by NGx researchers themselves. As regards the second criterion (‘‘Change in initial understanding’’), NGx respondents also stated in a greater proportion than ethics respondents that the Web pages did not change their understanding about the topics covered in the KDI (see Table 5).

On the other hand, a large majority of both NGx and ethics respondents stated that the information provided in the four Web pages actually raised questions for them (third criterion: ‘‘Generation of questions from the findings’’, see Table 6). In this respect, the KDI was successful in generating thoughts and interrogations. Yet, while several respondents acknowledged the relevance of the findings disseminated using this KDI, most of the respondents, both in NGx and ethics, mentioned that they would not change their practice (fourth criterion: ‘‘Intent to change own practice’’, see Table 7). Some respondents explicitly drew our attention to the actual and concrete obstacles that were preventing them from changing their own practice. In this respect, were notably mentioned, the practical impossibility of increasing ethnic diversity in research when studied cohorts are recruited in local, ethnically homogenous populations, and the potential absence of a scientific rationale or the difficulty of increasing the representation of certain age categories in research on diseases with middle-age onset. These findings should be put into perspective with the current KT paradigm, according to which knowledge from research follows a linear process and is ready to be implemented in practice immediately after dissemination (Bowen & Graham, 2013). Our results confirm that this paradigm does not coincide with what is experienced by target audiences in practice, where real barriers exist and can prevent systematic and immediate knowledge uptake and changes in end-user practices. 6.2. Effectiveness and challenges pertaining to the format and message In our study, both NGx and ethics respondents agreed that the format (cartoon-like illustrations with text) used in the present KDI had been effective in communicating the findings of the NGx project to them. However, this format required some trade-offs, in particular with regard to space. As the use of Internet and of cartoon-like illustrations requires succinct and quickly consultable information (Petrovici, 2013), it was necessary to significantly summarize the NGx project findings. This process was demanding as it was difficult to reduce the message to its very essence while remaining precise and faithful to the actual results. Such a process notably led to the misinterpretation of the findings provided in the ‘‘geographical location’’ thematic Web page: a respondent understood that there were no NGx studies conducted in Oceania. Yet, this conclusion goes beyond the message that was to be disseminated, as it does not take into account the limitations of the NGx project (e.g., studies identified on PubMed only, published during a limited period of time) as well as the potential existence of unpublished NGx studies conducted in this geographical region. Similarly, another respondent felt that NGx researchers were blamed for the scientific and ethical issues reported on the ethnicity-related Web page. This comment suggests that the message was not sufficiently explicit in regard to the objective of our KDI, which was not to question the validity of NGx research, nor to argue that NGx research could be raising more concerns than other fields of research. Yet, it must be stressed that some respondents understood the limitations of the format adopted and explicitly mentioned that the content displayed ‘‘doesn’t tell the whole story’’ or suggested that the detailed results of the NGx project should have been available through a clickable link on each Web page. Another challenge associated with the message related to the two categories of targeted audiences which required that the message be adapted to both NGx and ethics publics. It appeared that respondents working in ethics were less familiar with the scientific background of NGx, which likely had an impact on the rate of their participation in the evaluation of the KDI. Finally, as to the visuals, since the cartoons could be seen by people with

D. Lafrenie`re et al. / Evaluation and Program Planning 46 (2014) 103–114

113

potentially different values and beliefs, it was difficult to foresee how the viewers would interpret them. For instance, in a cartoon illustrating the underrepresentation of ethnic minorities, how to represent ‘‘ethnic minorities’’ without perpetuating stereotypes? How to represent diversity without falling into cliche´s?

Conflicts of interest

6.3. Benefits gained from the KDI study

This work was funded by a grant from the Canadian Institutes of Health Research (CIHR) (Grant #238582). The authors would like to acknowledge the work of Franc¸ois Brouillet from Cre´ations Touche´ who created the Web pages and administered the evaluation of the KDI.

As stressed by Ronteltap and colleagues, in order to achieve a successful implementation of new technologies such as NGx, it is of the utmost importance to take into account all stakeholders’ views on the potential ethical or scientific challenges that they may raise (Ronteltap & van Trijp, 2007; Ronteltap, van Trijp, & Renes, 2008). This KDI endeavoured to serve such a purpose. Firstly, the results indicate that the present KDI may generate concrete questions about the potential ethical and scientific challenges of NGx within the targeted audiences. This seems to us an important step in a fruitful debate about, and substantive reflection on, the issues that were addressed in the NGx project. Secondly, a major benefit gained from the evaluation of this KDI was the feedback received from the respondents. Such a benefit was already described by Nixon et al. (2012) who, after completing a study on an HIV prevention programme, used an interactive KDI to return their results to their research participants: this interaction provided valuable insights that improved the researchers’ understanding of their study results. Similarly, the nuances and reactions expressed by the respondents to our survey contributed to deepening our understanding not only of the NGx project results, but also about how their dissemination could be improved. The respondents’ comments that were described in all the previous sections were thus seriously taken into consideration and changes were brought to our KDI Web pages. In particular, a new Web page explicitly presents the goals of the NGx project and addresses the dissemination of its results,10 with clickable links to the publications or to the summary of the publications in which detailed results are available on each Web page, and various text modifications were added to ease visitors’ access and improve their understanding, as well as to avoid any potential misinterpretations of the disseminated results.

7. Conclusion Various difficulties were encountered in the design, implementation and evaluation of the present KDI. Very little guidance is available in the literature – either about KT or arts-based research – to help design, implement, or assess a KDI with visuals and text on the Internet. The evaluation of a KDI itself presents numerous challenges, starting with participation rates in such a process. Tailoring a message that would be appropriate and effective for different target audiences, with various characteristics and needs, was another difficulty. The results presented in this article show that while targeted end-users of a KDI may not feel that they actually learn anything from a KDI, the latter may nevertheless encourage reflection and raise useful and valuable questions for them. Most importantly, this KDI proved to be beneficial for the interpretation of the NGx project results. Indeed, respondents’ comments and questions allowed us to improve our understanding of these results and, hopefully, the content of the Web pages aimed at disseminating them. This is one of the greatest outcomes of the KDI. In addition of disseminating findings, the results from the evaluation of the KDI suggest that the format used (cartoon-like illustrations with text) could also be a valuable platform for the field to debate and discuss complex issues. 10

http://omics-ethics.org/en/NGX-research-index (accessed 18.4.13).

The authors declare that they have no competing interests. Acknowledgements

References Anderson, C. (2010). Presenting and evaluating qualitative research. American Journal of Pharmaceutical Education, 74(8), 1–7. Bartlett, R. (2012). Playing with meaning: Using cartoons to disseminate research findings. Qualitative Research, 1–14. http://qrj.sagepub.com/content/early/2012/ 07/24/1468794112451037 Bhattacharyya, O. K., Estey, E. A., & Zwarenstein, M. (2011). Methodologies to evaluate the effectiveness of knowledge translation interventions: A primer for researchers and health care managers. Journal of Clinical Epidemiology, 64(1), 32–40. Bombard, Y., Cox, S. M., & Semaka, A. (2011). When they hear what we say: Ethical challenges in presenting findings to the Huntington disease community. Journal of Empirical Research on Human Research Ethics, 6(3), 47–54. Bowen, S. J., & Graham, I. D. (2013). From knowledge translation to engaged scholarship: Promoting research relevance and utilization. Archives of Physical Medicine and Rehabilitation, 94(Suppl. 1), S3–S8. Boxall, K., & Ralph, S. (2009). Research ethics and the use of visuals in research with people with intellectual disability. Journal of Intellectual & Developmental Disability, 34, 45–54. Cheng, H. I., & Patterson, P. E. (2007). Iconic hyperlinks on e-commerce websites. Applied Ergonomics, 38(1), 65–69. Colquhoun, H. L., Letts, L. J., Law, M. C., MacDermid, J. C., & Missiuna, C. A. (2010). A scoping review of the use of theory in studies of knowledge translation. Canadian Journal of Occupational Therapy, 77(5), 270–279. Dainesi, S. M., & Goldbaum, M. (2012). E-survey with researchers, members of ethics committees and sponsors of clinical research in Brazil: An emerging methodology for scientific research. Revista Brasileira de Epidemiologia, 15(4), 705–713. Delp, C., & Jones, J. (2006). Communicating information to patients: The use of cartoon illustrations to improve comprehension of instructions. Academic Emergency Medicine, 3(3), 264–270. Eysenbach, G. (2004). Improving the quality of web surveys: The Checklist for Reporting Results of Internet E-Surveys (CHERRIES). Journal of Medical Internet Research, 6(3), e34. Gallopel-Morvan, K., Gabriel, P., Le Gall-Ely, M., Rieunier, S., & Urien, B. (2011). The use of visual warnings in social marketing: The case of tobacco. Journal of Business Research, 64, 7–11. Gerde, V. W., & Foster, R. S. (2008). X-men ethics: Using comic books to teach business ethics. Journal of Business Ethics, 77, 245–258. Gillies, P. A., Stork, A., & Bretman, M. (1990). Streetwize UK: A controlled trial of an AIDS education comic. Health Education Research, 5(1), 27–33. Godard, B., & Hurlimann, T. (2009). Nutrigenomics for Global Health: ethical challenges for underserved populations. Current pharmacogenomics and personalized medicine, 7, 205–214. Green, M. J., & Myers, K. R. (2010). Graphic medicine: Use of comics in medical education and patient care. BMJ, 340, 574–577. Guillemin, M. (2004). Understanding illness: Using drawings as a research method. Qualitative Health Research, 14(2), 272–289. Hammond, D., Fong, G. T., Mcdonald, P. W., Brown, K. S., & Cameron, R. (2006). Showing leads to doing: Graphic cigarette warning labels are an effective public health policy. The European Journal of Public Health, 16(2), 223–224. Holmes, B., Scarrow, G., & Schellenberg, M. (2012). Translating evidence into practice: The role of health research funders. Implementation Science, 7, 39. Houts, P. S., Witmerb, J. T., Egethc, H. E., Loscalzoa, M. J., & Zabora, J. R. (2001). Using pictographs to enhance recall of spoken medical instructions II. Patient Education and Counseling, 43, 231–242. Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis? Qualitative Health Research, 15(9), 1277–1288. Hurlimann, T., Menuz, V., Graham, J., Robitaille, J., Vohl, M. C., & Godard, B. (2014). Risks of nutrigenomics and nutrigenetics? What scientists say. Genes & Nutrition, 9, 330. Hurlimann, T., Stenne, R., Menuz, V., & Godard, B. (2011). Inclusion and exclusion in nutrigenetics clinical research: Ethical and scientific challenges. Journal of Nutrigenetics and Nutrigenomics, 4(6), 322–343. Joffe, H. (2008). The power of visual material: Persuasion, emotion and identification. Diogenes, 55(84), 84–93. Joseph, K. S. (2013). Fundamentals of knowledge translation. Journal of Obstetrics and Gynaecology Canada, 35(1), 73–77. Kilian, C. (2009). Writing for the web. North Vancouver: Self-Counsel Press. Lafrenie`re, D., & Cox, S. M. (2012a). Means of knowledge dissemination: Are the cafe´ scientifique and the artistic performance equally effective? Sociology Mind, 2(2), 191–199.

114

D. Lafrenie`re et al. / Evaluation and Program Planning 46 (2014) 103–114

Lafrenie`re, D., & Cox, S. M. (2012b). «If you can call it a poem »: Towards a framework for the assessment of arts-based works. Qualitative Research, 13(3), 316–334. Lafrenie`re, D., Hurlimann, T., Menuz, V., & Godard, B. (2012). Health research: Ethics and the use of arts-based methods in knowledge translation processes. International Journal of the Creative Arts in Interdisciplinary Practice, 11(1). http://www.ijcaip.com/archives/IJCAIP-11-paper3.html Lafrenie`re, D., Menuz, V., Hurlimann, T., & Godard, B. (2013). Knowledge dissemination interventions: A literature review. SAGE Open. LaRocca, R., Yost, J., Dobbins, M., Cliska, D., & Butt, M. (2012). The effectiveness of knowledge translation strategies used in public health: A systematic review. BMC Public Health, 12, 751. Lin, Y. C., Yeh, C. H., & Wei, C. C. (2013). How will the use of graphics affect visual aesthetics? A user-centered approach for web page design. International Journal of Human-Computer Studies, 71, 217–227. Lockyer, S., & Pickering, M. (2009). Beyond a joke: The limits of humour. Basingstoke: Palgrave Macmillan. Mayer, R. E., & Massa, L. J. (2003). Three facets of visual and verbal learners: Cognitive ability, cognitive style, and learning preference. Journal of Educational Psychology, 95(4), 833–846. McAllister, M. P. (1992). Comics and AIDS. Journal of Popular Culture, 1–24. McCloud, S. (1993). Understanding comics: The invisible art. New York: Harper Collins. McLean, R. K. D., Graham, I. D., Bosompra, K., Choudhry, y., Coen, S. E., & MacLeod, M. (2012). Understanding the performance and impact of public knowledge translation funding interventions: Protocol for an evaluation of Canadian Institutes of Health Research knowledge translation funding programs. Implementation Science, 7, 57. Nixon, S. A., Casale, M., Flicker, S., & Rogan, M. (2012). Applying the principles of knowledge translation and exchange to inform dissemination of HIV survey results to adolescent participants in South Africa. Health Promotion International. http:// dx.doi.org/10.1093/heapro/dar094 Noiwan, J., & Norcio, A. F. (2006). Cultural differences on attention and perceived usability: Investigating colour combinations of animated graphics. International Journal of Human-Computer Studies, 64(2), 103–122. Ou, C. X., & Sia, C. L. (2010). Consumer trust and distrust: An issue of website design. International Journal of Human-Computer Studies, 68, 913–934. Parsons, J. A., & Lavery, J. V. (2012). Brokered dialogue: A new research method for controversial health and social issues. BMC Medical Research Methodology, 12, 92– 101. Parsons, J. A., Heus, L., & Moravac, C. (2013). Seeing voices of health disparity: Evaluating arts projects as influence processes. Evaluation and Program Planning, 36, 165–171. Petrovici, I. (2013). Hermeneutical aspects concerning the aesthetics of the web design. European Journal of Science and Technology, 9(1), 133–142. Price, J., & Price, L. (2009). Hot text: Web writing that works. Berkeley: New Riders. Prosser, J. (1998). The visual representation of school culture. Communication & Cognition, 31(2/3), 243–264. Rich, M., Lamola, S., & Woods, E. R. (2006). Effects of creating visual illness narratives on quality of life with asthma: A pilot intervention study. Journal of Adolescent Health, 38, 748–752. Roberts, N., Ghiassi, R., & Partridge, M. R. (2008). Health literacy in COPD. International Journal of COPD, 3(4), 499–507. Ronteltap, A., van Trijp, J. C. M., & Renes, R. J. (2008). Making nutrigenomics work: Integrating expert stakeholder opinions and consumer preferences. Trends in Food Science & Technology, 19(22), 390–398. Ronteltap, A., & van Trijp, H. (2007). Consumer acceptance of personalised nutrition. Genes & Nutrition, 2(1), 85–87. Saade, G., & Otrakji, A. (2007). First impressions last a lifetime: Effect of interface type on disorientation and cognitive load. Computers in Human Behavior, 23, 525–535. Sargeant, J., Borduas, F., Sales, A., Klein, D., Lynn, B., & Stenerson, H. (2011). CPD and KT: Models used and opportunities for synergy. Journal of Continuing Education in the Health Professions, 31(3), 167–173. Sauermann, H., & Roach, M. (2013). Increasing web survey response rates in innovation research: An experimental study of static and dynamic contact design features. Research Policy, 42, 273–286.

Schonlau, M., Fricker, R. D., & Elliot, M. N. (2002). Conducting research surveys via e-mail and the web. Santa-Monica: Rand Corporation. Scott, S. D., Albrecht, L., O’Leary, K., Ball, G. D. C., Hartling, L., & Hofmeyer, A. (2012). Systematic review of knowledge translation strategies in the allied health professions. Implementation Science, 7, 70–88. Shapiro, D., Tomasa, L., & Koff, N. A. (2009). Patients as teachers, medical students as filmmakers: The video slam, a pilot study. Academic Medicine, 84(9), 1235–1243. Shih, T. H., & Fan, X. (2008). Comparing response rates from web and mail surveys: A meta-analysis. Field Methods, 20(3), 249–271. Sones, W. W. D. (1944). The comics and instructional method. Journal of Educational Sociology, 18(4), 232–240. Stenne, R., Hurlimann, T., & Godard, B. (2013). Benefits associated with nutrigenomics research and their reporting in the scientific literature: Reserchers’ perspectives. Accountability in Research, 20(3), 167–183. Stenne, R., Hurlimann, T., & Godard, B. (2012). Are research papers reporting results from nutrigenetics clinical research a potential source of biohype? Accountability in Research: Policies and Quality Assurance, 19(5), 285–307. Strauss, S. E., Tetroe, J., & Graham, I. D. (2009). Defining knowledge translation? CMAJ, 181(3–4), 165–168. Tait, A. R., Voepel-Lewis, T., Zikmund-Fisher, B. J., & Fagerlin, A. (2010). The effect of format on parents’ understanding of the risks and benefits of clinical research: A comparison between text, tables, and graphics. Journal of Health Communication, 15(5), 487–501. Teti, M., Murray, C., Johnson, L., & Binson, D. (2012). Photovoice as a community-based participatory research method among women living with HIV/AIDS: Ethical opportunities and challenges. Journal of Empirical Research on Human Research Ethics: An International Journal, 7(4), 34–43. Vaughn, L. M., Rojas-Guyler, L., & Howell, B. (2008). Picturing health. A photovoice pilot of Latina girls’ perceptions of health. Family & Community Health, 31(4), 305–316. Weymann, N., Harter, M., & Dirmaier, J. (2013). A tailored, interactive health communication application for patients with type 2 diabetes: Study protocol of a randomised controlled trial. BMC Medical Informatics and Decision Making, 13(24) http://dx.doi.org/10.1186/1472-6947-13-24 Wyatt, J. (2000). When to use web-based surveys. Journal of the American Medical Informatics Association, 7, 426–429. Yang, G. (2013). Comics in education. http://www.humblecomics.com/comicsedu/ strengths.html Darquise Lafrenie`re is Clinical Assistant Professor in the Department of Preventive and Social Medicine at the University of Montreal and a researcher in the OMICSETHICS research group. Her research interests include knowledge translation in health research, especially the use of arts-based methods; research ethics; and ethics and communication. Thierry Hurlimann is the research coordinator of the OMICS-ETHICS research group at the University of Montreal. He was admitted to the Geneva bar as a lawyer in 1997 and completed an LL.M, specialization in bioethics at McGill University in 2004. He plays an active role as a researcher and critical thinker in the evaluation and development of projects concerning nutrigenomics and personalized medicine as they relate to global justice and equity issues in both developed and developing countries. Vincent Menuz is a postdoctoral fellow in the Research Centre on Ethics at the University of Montreal and a member of the OMICS-ETHICS research group. He completed a PhD in biology at the University of Geneva, Switzerland, where he developed a significant expertise in molecular biology, genetics and genomics. His current work focuses on the ethical and social issues of human enhancement, as well as anti-ageing interventions. Be´atrice Godard is Professor in the Department of Preventive and Social Medicine at the University of Montreal. Her current research work explores emerging ethical responsibilities where research and the clinic intersect. She is also engaged in seeking to better understand prevailing attitudes and behaviours regarding genetic research.