Dealing with conflicting or consistent medical information on the web: When expert information breeds laypersons' doubts about experts

Learning and Instruction 21 (2011) 193e204 www.elsevier.com/locate/learninstruc

Dealing with conflicting or consistent medical information on the web: When expert information breeds laypersons’ doubts about experts Dorothe Kienhues*, Marc Stadtler, Rainer Bromme University of Muenster, Institute for Psychology, Fliednerstr. 21, 48149 Muenster, Germany

Abstract The present study investigated how dealing with conflicting versus consistent medical information on the Web impacts on topic-specific and medicine-related epistemic beliefs as well as aspects of health decision making. One hundred mostly female university students were randomly assigned to three groups. Two intervention groups searched the Web for information on cholesterol to advise a fictitious friend about treatment. Pre-selected websites for these groups provided either conflicting or consistent information. The third group, the control group, did not conduct Web search. Results showed that the intervention groups differed in topic-specific epistemic beliefs. After the Web search, their medicine-related epistemic beliefs were more advanced while remaining unchanged in controls. The intervention groups also differed in some aspects of decision making. Ó 2010 Elsevier Ltd. All rights reserved. Keywords: Epistemic beliefs; Web search; Medical information; Health decision making

1. Introduction The World Wide Web (WWW), because it offers convenient access to an unprecedented amount of information, is frequently used for information seeking. One particularly popular use is to gain health information about a specific topic (Fox, 2006). Morahan-Martin (2004), for example, has called the Internet the world’s largest medical library. People searching for medical information on the Web are predominately laypersons looking for further (or alternative) information on the treatment of a specific disease (Fox & Rainie, 2000) that will answer their health-related questions and help them make informed decisions. However, this goal is not always easy to attain. Eysenbach (2003) has pointed out that when people search the Web for medical information, they are commonly confronted with conflicting evidence. Furthermore, because the publishing of * Corresponding author. Tel.: þ49 251 8331347; fax: þ49 251 8339105. E-mail addresses: [email protected] (D. Kienhues), [email protected] (M. Stadtler), [email protected] (R. Bromme). 0959-4752/$ - see front matter Ó 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.learninstruc.2010.02.004

information on the Internet is unregulated (Britt & Aglinskas, 2002), low-quality health websites are considered to be as common as (if not more common than) those providing highquality and, therefore, credible information (Bates, Romina, Ahmed, & Hopson, 2006). Several studies have pointed out the importance of epistemic beliefs for dealing with the diversity of information on the Web (Bra˚ten & Strømsø, 2006; Hofer, 2004; Mason & Boldrin, 2008; Tu, Shih, & Tsai, 2008). Epistemic beliefs are beliefs about the nature of knowledge and knowing (Hofer & Pintrich, 1997). These studies conceive them as a predictor of how people deal with information from the Web, pointing out that more advanced epistemic beliefs lead to a more adequate handling of Webbased information. However, both theoretical considerations and some initial empirical evidence (Spiro, Feltovich, & Coulson, 1996; Tsai, 2008) indicate that this causal link can also take the opposite direction: Dealing with information on the Web may also lead to more advanced epistemic beliefs. The present study addressed the latter, relatively underresearched question. Because information on the Web not only offers access to new knowledge but also is often conflicting, it

194

D. Kienhues et al. / Learning and Instruction 21 (2011) 193e204

might provide a good environment for fostering more advanced beliefs. Specifically, the present study examined the influence of receiving either conflicting or consistent information on different levels of epistemic beliefs and on decisions based on the different kinds of information found in the Internet. 1.1. Changes in epistemic beliefs During the last two decades, research on epistemic beliefs has gained considerable popularity in developmental and educational research (Bendixen & Feucht, 2010; Hofer & Pintrich, 2002; Khine, 2008). This research indicates that epistemic beliefs range from a less advanced view to more advanced epistemologies and develop through life and educational experiences (Chandler, Boyes, & Ball, 1990; Kuhn & Weinstock, 2002). Whereas a less advanced view includes beliefs such as ‘‘knowledge is certain and stable, either true or false, and can be handed down by an authority’’, a more advanced view is characterized by beliefs that knowledge is rather complex and relativistic, by accepting the uncertainty and changeability of truth, and acknowledging that knowledge is construed individually. Within educational psychology, research has focused especially on the relation between epistemic beliefs and academic performance, cognition, or the learning process (Bartholome´, Stahl, Pieschl, & Bromme, 2006; Mason & Boscolo, 2004). Several studies have emphasized that gaining knowledge influences epistemic beliefs (Kienhues, Bromme, & Stahl, 2008; Schommer, Calvert, Gariglietti, & Bajaj, 1997). However, results on the relation between knowledge and epistemic beliefs are somewhat contradictory (see, for a more comprehensive overview, Bromme, Kienhues, & Stahl, 2008). Most empirical studies indicate a relation between more advanced epistemic beliefs and more knowledge, pointing to the influence of schooling on the development of epistemic beliefs (Maggioni, Riconscente, & Alexander, 2006). Schommer (1990) found that participants who had completed more years of schooling were more likely to hold more advanced beliefs such as knowledge is tentative, and that the development of epistemic beliefs across the secondary school years is related to learning during secondary school (Schommer et al., 1997). Cano (2005) has pointed out that epistemic beliefs become more complex and realistic in the later compared with the earlier years of secondary education. In contrast, other studies have revealed an inverse relation between knowledge and epistemic beliefs. For example, in a study on students’ epistemic beliefs about physics, Ko¨ller, Baumert, and Neubrand (2000) found that the longer and more intensively students learned about physics, the more they showed dualistic views, and the more they thought that absolute ‘‘truth’’ could be attained in the subject. Therefore, one can conclude that gaining further knowledge influences epistemic beliefs. However, the direction of change seems to vary depending on the kind of knowledge acquired as well as on people’s pre-existing epistemic beliefs. This view is fostered by some studies focusing on instructional interventions deliberately designed to elicit changes in epistemic beliefs (Conley, Pintrich, Vekiri, & Harrison, 2004; Gill, Ashton, & Algina, 2004). Most such interventions are based

on the assumption that confronting participants with different and opposing perspectives can serve as a starting point to make epistemic beliefs more advanced (Bendixen & Rule, 2004; Jacobson & Spiro, 1995). For example, Conley et al. (2004) found that 5th-grade students showed reliable changes towards a more advanced view in their beliefs about the source and certainty of scientific knowledge after a 9-week hands-on science course. The authors traced the changes back to the fact that the results of the hands-on experiments varied greatly, and that this might have helped students understand that answers to scientific questions are subject to revision and change and do not derive solely from authorities. Kienhues et al. (2008) compared the effects of reading a refutational instruction (emphasizing the dynamics and uncertainties of knowledge within the domain of DNA fingerprinting) with reading an informatory instruction (outlining facts on DNA fingerprinting) on participants’ epistemic beliefs about genetics. In a group of participants with less advanced beliefs, the refutational instruction elicited changes towards a more advanced view, whereas the informatory instruction did not elicit significant changes. In contrast, in participants holding an advanced epistemic stance, reading the informatory instruction elicited changes towards a less advanced view about knowledge in genetics, whereas the refutational instruction did not elicit any change. The present study exemplifies the specific influence of the kind of information gained. It can be assumed that confronting students with conflicting information on a certain topic will foster more advanced epistemic beliefs. Furthermore, it can be assumed that although gaining consistent information about a certain domain will impact on epistemic beliefs, the size (relative to the impact of being confronted with conflicting information) of that effect will be smaller. 1.2. Specificity of epistemic beliefs When considering changes in or the development of epistemic beliefs, it is crucial to be aware of the different layers or levels of epistemic beliefs. In general, most authors assume that people possess both discipline-general and discipline-specific1 epistemic beliefs concurrently (Bromme et al., 2008; Buehl, Alexander, & Murphy, 2002; Muis, Bendixen, & Haerle, 2006), and several studies have underlined the potential diversity of epistemic beliefs across different disciplines (Hofer, 2000; Stahl & Bromme, 2007). The elaborateness of discipline-specific epistemic beliefs might depend on the degree of exposure to questions regarding the nature of knowledge and knowing in a specific discipline (Kienhues et al., 2008). Therefore, people might not have well-established and widely explored epistemic beliefs about every single scientific discipline (Buehl & Alexander, 2001). In consequence, whereas more general beliefs seem to be relatively stable, discipline-specific beliefs may well be more variable (Gill et al., 2004). 1 We use the term ‘‘discipline’’ instead of ‘‘domain’’ to emphasize that we are referring to academic fields. The problematic use of the complex term ‘‘domain’’ has been discussed comprehensively by Hofer (2006) and Limon (2006).

D. Kienhues et al. / Learning and Instruction 21 (2011) 193e204

Furthermore, beyond the discipline-specific level, epistemic beliefs can also be considered on a more fine-grained level. Some studies have focused on the topic-specificity of epistemic beliefs, assuming that epistemic beliefs may vary with topics within specific disciplines (Gil Pelluch, 2009). Trautwein and Lu¨dtke (2007) pointed out that topic-specificity means taking domain-specificity ‘‘one step further than the more frequent discipline-specific approaches’’ (p. 908). For example, Bra˚ten, Strømsø, and Samuelstuen (2008) examined beliefs about the nature of knowledge and knowing in climate change. Trautwein, Lu¨dtke, and Beyer (2004) examined epistemic beliefs about different theories like the theory of relativity or big bang theory. The controversially discussed relation between knowledge and epistemic beliefs can also be found on the topic-specific level. Specifically, Trautwein et al. (2004) found that having more knowledge about a specific topic also includes less advanced, more positivistic beliefs, whereas Bra˚ten et al. (2008) assumed that this relation takes the opposite direction. In conclusion, epistemic beliefs can be considered on different levels, because they are a graded concept in terms of specificity. Such gradedness requires a diversity of research instruments. On the ‘‘discipline level’’ participants can simply be asked to complete the same items while referring to different disciplines (Buehl et al., 2002; Hofer, 2000). Such studies require participants to reflect on their epistemic beliefs in relation to a specific discipline. Because such an instrument is applicable to different disciplines, it cannot focus on aspects that are specific or unique to a certain discipline. Therefore, we refer to such instruments as discipline-related in the following. In contrast, there are also clearly discipline-specific instruments like the Epistemological Beliefs Assessment for Physical Science (EBAPS; Elby, Frederiksen, Schwarz, & White, 2003). Furthermore, topic-specific questions can be used to measure the most specific level of epistemic beliefs (as the above mentioned authors have done). The present study distinguishes between discipline-related and topic-specific epistemic beliefs. The former refer to broad ideas about epistemic aspects of medical knowledge (medicine-related epistemic beliefs), whereas the latter refer to assumptions about specific medical knowledge on a certain topic of a layperson’s Internet search. 1.3. Health decision making Why should an advanced epistemic stance be helpful for laypersons when it comes to health-related decisions? For shared decision making and for patients’ compliance in general, it is necessary for patients to understand the complexity and, at times, uncertainty of medical knowledge. Healthcare organizations and policymakers are both interested in involving the patient in treatment decisions (Hack, Degner, Parker, & the SCRN Communication Team, 2005). To increase the patients’ knowledge, the physician should provide them with diagnostic information and information about treatment options (Elwyn et al., 2003). Such involvement in the treatment decision-making even goes beyond patients and also includes (though more indirectly) relatives and close

195

friends who often follow the treatment process of a diseased person actively and influence treatment decisions (Ernst, Go¨tze, Weissflog, Schro¨der, & Schwarz, 2006). Because many health-related problems are contentious and lack clear-cut solutions, they can be described as ill-structured problems, that is, ‘‘problems about which reasonable people reasonably disagree’’ (King & Kitchener, 2002, p. 37). Such health-related problems require decisions based on informal reasoning. Sadler (2004) has pointed out that informal reasoning ‘‘involves the generation and evaluation of positions in response to complex issues that lack clear cut solutions’’ (p. 514). It is important that not only physicians’ (Knight & Mattick, 2006), but also laypersons’ (or patients’) medicine-related epistemic beliefs permit an appropriate understanding of such a lack of clear cut solutions. Patients have to cope with the fact that there are alternative treatments, differing interpretations of symptoms, or even several interpretations of the underlying pathophysiological explanations of illnesses and risks. People often turn to the Internet to search for information that might help them to make more informed decisions about their (or their relatives’ and friends’) healthcare (DeLenardo, 2004; Eysenbach, 2003). However, this confronts them all the more with a plenitude of information and a multitude of opinions on their specific health-related problems. 1.4. The present study In the present study it was investigated how conflicting versus consistent scientific information encountered when searching the Web might impact differentially on both topicspecific and discipline-related epistemic beliefs. Hence, we considered epistemic beliefs on different levels. It was decided to consider both topic-specific and discipline-related beliefs to gain insight on the breadth of impact of Web-based information on a specific topic. The focus was in how far dealing with very specific information on a topic (differentially) would influence beliefs on the higher, discipline-related level. From the different dimensions2 of epistemic beliefs, the interest was particularly in aspects of the certainty and simplicity of knowledge. Because these aspects are addressed fairly directly when people deal with (conflicting) information, changes on these dimensions were considered to be most likely. With regard to topic-specific beliefs, it was hypothesized that gaining conflicting information about a specific topic would result in more advanced beliefs regarding the tentative and preliminary nature of topic-related knowledge. Conflicting information might mirror the diversity of views on the topic. Gaining consistent topic information, on the other hand, might 2 Following Hofer and Pintrich (1997), epistemic beliefs can be described by means of four identifiable and more or less interrelated dimensions. The first two dimensions represent the nature of knowledge and refer to the certainty of knowledge (e.g., the perceived stability) and the structure of knowledge in terms of simplicity of knowledge. The remaining two dimensions refer to the nature of knowing and include the justification of knowledge (e.g., how knowledge claims can be warranted) and the source of knowledge (e.g., whether knowledge resides rather internally than externally).

196

D. Kienhues et al. / Learning and Instruction 21 (2011) 193e204

also influence topic-specific epistemic beliefs, taking into account the interplay between gaining knowledge and epistemic beliefs. However, because this influence (often) takes a different direction, we assumed that the group dealing with conflicting information would show more advanced topic-specific beliefs than the group dealing with consistent information (Hypothesis 1). Furthermore, it was assumed that dealing with conflicting medical information about a specific topic might also influence discipline-related beliefs (or medicine-related beliefs in the present study). Due to the generally assumed role of different and opposing information for changes in epistemic beliefs, changes towards a more advanced view were anticipated. Conflicting topic information might challenge laypeople’s initial discipline-related beliefs, because this presents an example of discipline-related knowledge that is not very exact and rather complex. Again, some change in medicine-related beliefs for the group dealing with consistent information was predicted. However, as consistent information lacks controversies, it consequently lacks a powerful impetus to change beliefs. Therefore, it was assumed that if changes were to occur, they might be not as strong as changes in the conflict group. For changes in medicine-related beliefs, it was hypothesized that the two groups dealing with scientific information on the Web would show considerably more changes in medicine-related epistemic beliefs than the no-intervention control group (see below) whose beliefs should remain unchanged (Hypothesis 2). Furthermore, it was anticipated that the group dealing with conflicting information would show more change in medicinerelated epistemic beliefs than the group dealing with consistent information (Hypothesis 3). The present study also examined how the different kinds of information provided impact on participants’ health decision making, their perceived personal knowledge and the certainty in their decision. The aim was to gain insight into how far dealing with information on the Web influences laypeople’s dealing with medical issues. Additionally, the study investigated how far dealing with health-related Internet information influences laypeople’s perceived epistemic authority of medical experts. The guiding research questions were: (a) Do the two intervention groups differ in their post-Web search decisions? If yes, in what way? (b) Are participants in the group dealing with conflicting information less certain in their own decision and (c) in the assumed certainty of experts than participants in the group dealing with consistent information? (d) Do the two intervention groups differ in their estimates of their knowledge about the topic after the Web search? First, it was investigated how far dealing with conflicting versus consistent information influences the decision participants made on the problem they had to deal with during the Web search (a health-related problem). It was anticipated that the two groups would show considerable differences in their decision depending on the different information (conflicting vs. consistent) they had to deal with (Hypothesis 4). Second, the certainty in this decision was investigated; the hypothesis was that participants who dealt with conflicting evidence would be less certain in their decision than those who dealt with consistent information, because they

had been confronted with divergent views on the problem (Hypothesis 5). Third, participants were asked to estimate the certainty of experts making the same decision. It was anticipated that dealing with conflicting evidence would lead to lower estimates of experts’ certainty in the same decision than dealing with consistent information (Hypothesis 6). Fourth, participants were asked to estimate their personal knowledge about the specific topic they had gathered information about on the Web. It was anticipated that participants dealing with conflicting information would feel less knowledgeable than participants dealing with consistent information (Hypothesis 7). 2. Method 2.1. Sample Participants were 100 predominantly female (84%) students attending a German university who were paid 10 Euros for their participation. Their average age was 22.57 years (SD ¼ 4.25). They had been studying various subjects predominantly in the humanities, for an average of 3.48 semesters (SD ¼ 2.78), that is, about one and a half year. Because none of the participants’ studies focused on medical knowledge (e.g., medicine, pharmacy), they were assumed to be laypersons in this domain. 2.2. Design The study had a pretest/posttest experimental design. Participants were randomly assigned to three different experimental conditions: an intervention group provided with websites containing conflicting contents (conflict group, IG1, n ¼ 30), an intervention group provided with websites containing consistent contents (consistency group, IG2, n ¼ 34), and a no-intervention group (control group, CG, n ¼ 36) that did not conduct a Web search. The control group, in which only medicine-related epistemic beliefs were surveyed, was added to ensure that any differences between the two intervention groups and between the measurements of medicinerelated beliefs before and after the Web search would be due to the differences in the information dealt with. Specifically, the intervention groups should not differ from the control group in their pretest measurements of medicine-related beliefs but should significantly differ from it in their respective posttest measurements. 2.3. Measures 2.3.1. Topic-specific epistemic beliefs The two intervention groups completed a specially developed 8-item scale focusing retrospectively on topic-specific epistemic beliefs. When constructing the items, we not only adapted items from various existing instruments (e.g., Conley et al., 2004; Hofer, 2000) but also added new items. These items addressed the clear cut solvability of the task (e.g., whether more than one answer could be correct; whether experts would clearly know; and whether everyone who

D. Kienhues et al. / Learning and Instruction 21 (2011) 193e204

197

searches the Web on the topic would come to the same solution).3 A sample item is ‘‘I think it is justifiable that there are different opinions on this topic’’. The scale was tailored explicitly to fit the experiences and reasoning while dealing with the Web information and was, therefore, administered after the Web search (for the full list of items on topic-specific beliefs, see Appendix A). Although the scale is topic-specific, it could be applied easily to different topics people search the Web for, because all items contain a form of placeholder (e.g., ‘‘this topic’’, ‘‘this problem’’). Participants rated items on a 5-point scale ranging from 1 (I totally disagree) to 5 (I totally agree). They were also allowed to indicate that the question could not be meaningfully answered (which was rated with 0). Higher values indicate that participants did not believe in a single correct and easy answer, whereas lower values indicate a clear cut solvability of the task. A principal components analysis4 was conducted to test whether the eight items could be assigned to one factor. The index of sampling adequacy, reporting the appropriateness of data for a factor analysis, was .72 for the sample. Following Kaiser and Rice (1974) this can be interpreted as moderate. Bartlett’s test of sphericity was statistically significant ( p < .05). The analysis revealed one factor solution that accounted for 38.28% of the total variance. All items showed substantial factor loadings (> .32; cf. Tabachnick & Fidell, 2001), and five variables showed coefficients each greater than .60. The factor yielded an acceptable Cronbach’s alpha of .75.

perceived as being. Variability addresses beliefs on the dynamics and stability of knowledge over time and ranges from beliefs that knowledge is dynamic and flexible to beliefs that it is stable and inflexible. This factor structure has been replicated in different data sets with values for Cronbach’s alpha ranging from .73 to .83 (Stahl & Bromme, 2007). Although the two CAEB factors correspond to the Simplicity and Certainty dimensions found in most popular questionnaires assessing epistemic beliefs (e.g., Schommer, 1990), they differ in some nuances. For example, Texture differs from Simplicity because it focuses on the issue of knowledge complexity by examining how structured, exact, and precise knowledge is perceived as being. Variability, which is related to Certainty, addresses beliefs on the variability or stability of knowledge over time as well as beliefs about exactness. The CAEB is easy to apply to different disciplines, because the discipline to which the instrument refers to is specified in an introductory sentence. In the present study, participants were asked to fill in the CAEB for knowledge in medicine. The CAEB was administered twice to all three groups. Before the pretest/posttest comparisons, the Cronbach’s alphas for the two assumed factors were checked in order to estimate how well Stahl and Bromme’s (2007) original factor solution fitted the present sample. Internal consistency was acceptable for both factors at both measurement times (Texture: Cronbach’s a ¼ .72 in the pretest and ¼ .71 in the posttest; Variability: Cronbach’s a ¼ .70 in the pretest and ¼ .79 in the posttest).

2.3.2. Medicine-related epistemic beliefs We measured medicine-related beliefs with the instrument on Connotative Aspects of Epistemological Beliefs (CAEB; Stahl & Bromme, 2007). This is a semantic differential comprising 24 pairs of oppositional adjectives (antonyms) that describe the nature of knowledge (e.g., dynamic-static, objective-subjective). Participants have to describe their personal beliefs about knowledge in a specific academic discipline for each adjective pair on a 7-point scale, ranging from 1 to 7. When rating the items, participants refer mainly to their evaluative associations about the nature of knowledge in a certain discipline rather than their factual knowledge about the nature of knowledge within that discipline. Therefore, judgments are assumed to be more personal, emotional, and context-dependent than reports on more declarative epistemic beliefs. The original instrument contains 24 items, of which 17 represent two factors, namely Texture of Knowledge (Factor 1) and Variability of Knowledge (Factor 2). Texture focuses on the issue of knowledge complexity and reflects how structured and exact versus unstructured and vague knowledge is

2.3.3. Health decision making and certainty In the two intervention groups, participants’ decision making on the health problem at hand was assessed, as well as whether they thought both they themselves and experts could solve the health problem, and their presumed knowledge about the topic. Therefore, participants were told (Question 1) to advise a fictitious friend who had asked them whether she should take medicine to lower her high cholesterol. They could choose between the answers ‘‘More likely yes’’, ‘‘More likely no’’ and ‘‘I don’t know’’. They were also asked (Question 2) to estimate how certain they were about their answer to Question 1. A third question (Question 3) asked participants to estimate how certainly they assumed experts could answer Question 1. Both Questions 2 and 3 had to be answered on 5-point scales ranging from 1 (very uncertain) to 5 (very certain). Additionally, participants rated (Question 4) their perceived personal knowledge about the topic cholesterol, as compared to an average person, on a 5-point scale ranging from 1 (very little) to 5 (very much). All questions were presented twice, that is, once, after being introduced to their fictitious friend and her request, and then again after the 30-min Web search on cholesterol.

3 The focus of the topic-specific instrument corresponds to the dimension of certainty of knowledgeda core component in most conceptions of personal epistemology. 4 The number of cases is relatively small for a factor analysis. However, Thompson (2004, in reference to Guadagnoli & Velicer, 1988) states that ‘‘replicable factors tend to be estimated if [.] factors are each defined by four or more measured variables with structure coefficients each greater than j0.6j, regardless of sample size’’ (p. 24).

2.4. Materials For the present study, a health-related topic, namely, medication use for the control of cholesterol was chosen. Discussions on this topic are truly controversial in medical literature (Dale, Coleman, Henyan, Kluger, & White, 2006) and it has proven

198

D. Kienhues et al. / Learning and Instruction 21 (2011) 193e204

to be popular enough to ensure participant motivation during Web searches in a previous study (Stadtler & Bromme, 2008). From a Google search of German websites on cholesterol 15 websites for the conflict group and 15 websites for the consistency group were chosen. All websites presented information in a predominantly text-based manner. While some of them also contained pictures, none had animations, films, or audio files. Websites were chosen to be either conflicting or consistent with regard to four main aspects of the discussion on the causes and treatment of high cholesterol: (a) the hypothesis that food high in cholesterol influences the cholesterol level; (b) the hypothesis that high cholesterol can cause arteriosclerosis or heart attacks; (c) the idea that cut off values can be defined for high cholesterol; and (d) the idea that medication, at least after trying alternative ways, is a very useful way to lower high cholesterol. For example, one website chosen for the conflict group stated clearly that increased cholesterol is one main cause of arteriosclerosis, while another explicitly rejected any link between the two. Another website provided for the conflict group illustrated different cut off values for high cholesterol for different at-risk groups, whereas another website emphasized that the main reasons for introducing such cut off values are profit-oriented. In short, websites chosen for the conflict group mirrored the heterogeneity of perspectives on the controversially discussed topic high cholesterol. Such contradictions were not present in the websites chosen for the consistency group. In both intervention conditions, different website providers were included to reflect the diversity of information providers on the Web. For example, we chose websites of pharmaceutical companies, health insurers, online health portals, nutrition companies, physicians, and consumer protection services. The distribution of kinds of provider differed between the two research conditions.5 Websites suitable for the conflict condition were predominantly online health portals (5 out of 15), whereas those suitable for the consistency condition were predominantly from pharmaceutical companies (8 out of 15). Three of the websites chosen for the consistency group were also part of the set chosen for the conflict group (one website of a pharmaceutical company and two websites of informatory online health portals). Links to the chosen websites were presented in alphabetical order.6 5

This uneven distribution of sources was due to the inevitable confounds between the message of a website and the information provider. Because this study did not focus on the trustworthiness of information (in contrast to, e.g., Bra˚ten, Strømsø, & Britt, 2009; Mason, Ariasi, & Boldrin, 2011; Stadtler & Bromme, 2008) or other source-related aspects (e.g., epistemic dimensions like ‘‘source of knowledge,’’ ‘‘omniscient authority,’’ or ‘‘justification of knowledge’’), we believe that this uneven distribution did not exert any strong influence on our results. 6 Although websites were presented in alphabetical order, the conflicting orientations towards treating increased cholesterol had a quite balanced distribution across the websites provided to the conflict group. For example, the first website was a critical one from the German association of statutory health insurance physicians questioning the usefulness of statins (cholesterol lowering medications). The second website was from a diet margarine company that emphasized the value of healthy nutrition in lowering cholesterol, whereas the third was hosted by a pharmaceutical company that markets statins.

2.5. Procedure For all groups, the first part of the study took place online. Participants were administered an online questionnaire lasting about 10 min. This questionnaire contained the CAEB and several demographic variables. The control group received a second online questionnaire containing the CAEB several days after filling in the first one, so that both questionnaires were completed within one week. For the two intervention groups, the main part of the study was held in the university. Participants were given individual appointments within one week of filling out the first online questionnaire. After first completing another online questionnaire lasting about 10 min (this measured several other variables not reported in this article), they were introduced to a fictitious friend with a high level of cholesterol who is asking for support in deciding on medical treatment. They were given more information on this friend (e.g., age, lifestyle habits) on a separate sheet that they could refer to during the following Internet search. They were, then, asked whether they would advise their fictional friend to take medicine to lower her high cholesterol, how certain they are about their decision, how certain experts making the same decision would be, and how much they know about cholesterol (see subchapter 2.3.3). They were then given access to a website containing alphabetically ordered links to the different pre-selected websites on cholesterol. For the next 30 min, participants searched the Web. They were free to click as many of the links as they wanted. Searching time was fixed to avoid time-on-task effects. Log files were recorded using Attention Recorder, an open-source Firefox extension to track clickstream and browsing history, in order to collect information on the websites visited. After the search phase, participants answered the four questions on decision making, certainty, and perceived personal knowledge again, before completing the scale on topic-specific epistemic beliefs and the CAEB. 3. Results 3.1. Equivalence of the groups Before analyzing treatment effects, several variables were checked to ensure that any potential differences between pre and posttest measures were due to differences in the content of the websites. Specifically, the three groups (i.e., conflict group, IG1; consistency group, IG2; control group, CG) should not differ significantly in terms of their initial epistemic beliefs as measured by the CAEB. An ANOVA revealed no significant differences between the three groups for the factors Texture of Knowledge, F(2, 97) ¼ 0.839, ns, or Variability of Knowledge, F(2, 97) ¼ 0.115, ns. Therefore the three groups were considered to be equivalent in terms of CAEB pretest measures. Self-reports of participants in the three groups on their medical knowledge, using the German school grades ranging from 1 (very good) to 5 (inadequate), indicated that participants assumed their medical knowledge to be pretty poor (M ¼ 3.48, SD ¼ 0.82). Therefore, participants were assumed

D. Kienhues et al. / Learning and Instruction 21 (2011) 193e204

to be medical laypersons. However, an ANOVA revealed that the groups significantly differed in terms of their self-reported medical knowledge, F(2, 97) ¼ 5.69, p < .05, partial h2 ¼ .10. Post hoc Scheffe´ tests showed that the control group differed significantly from the conflict group (CG: M ¼ 3.22, SD ¼ 0.80; IG1: M ¼ 3.87, SD ¼ 0.78; p < .01). More importantly in the present context, however, post hoc Scheffe´ tests also showed that the two intervention groups did not differ significantly with regard to their perceived medical knowledge (IG1: M ¼ 3.87, SD ¼ 0.78, IG2: M ¼ 3.41, SD ¼ 0.78). Finally, the three groups also did not differ significantly in terms of the reported time they spent on the Web per week, F(2, 97) ¼ 0.28, ns, or on searching for medical information on the Web per week, F(2, 97) ¼ 0.27, ns. The two intervention groups also did not differ in terms of the number of websites visited within the 30-min Web search (IG1: M ¼ 10.04, SD ¼ 3.80; IG2: M ¼ 11.68, SD ¼ 3.94), t(58) ¼ 1.62, ns. 3.2. Topic-specific epistemic beliefs A t-test for independent groups indicated that the conflict group’s scores on the scale measuring topic-specific epistemic aspects during Web search differed significantly from those of the consistency group, t(52.55) ¼ 2.01, p < .05, Cohen’s d ¼ 0.50. In line with Hypothesis 1, significantly fewer members of the conflict group (M ¼ 3.59, SD ¼ 0.45), compared to members of the consistency group (M ¼ 3.23, SD ¼ 0.81), believed in a clear cut solvability of the task and that all people performing a Web search on the topic would come to the same conclusion. 3.3. Medicine-related epistemic beliefs To test the priori specified hypotheses on changes in medicine-related epistemic beliefs (Hypotheses 2 and 3), planned contrasts were conducted in line with Rosenthal, Rosnow, and Rubin’s (2000) approach for testing focused questions. Specifically, planned contrasts were conducted between the control group and the two intervention groups (CG vs. [IG1 þ IG2]/2) to test Hypothesis 2 and between the two intervention groups (IG1 vs. IG2) to test Hypothesis 3 for both factors of the CAEB. Therefore, the present analyses were designed to compare results only with prior predictions, thereby increasing statistical sensitivity compared with an unfocused test like an ANOVA. To test these planned contrasts, a new dependent measure was computed by subtracting the pretest measures from the posttest measures for each CAEB factor, thereby taking into account any potential influence of the pretest scores. Scale means and standard deviations for the three groups for both measurement times and both CAEB factors are presented in Table 1; mean gain scores and standard deviations are presented in Table 2. For the factor Texture of Knowledge, planned contrasts between the control group and the two intervention groups indicated a significant difference, F(1, 97) ¼ 7.24, p < .01,

199

Table 1 Mean scores and standard deviations for the CAEB factors as a function of experimental group and measurement time. Factor 1: texture of knowledge Pretest

Conflict group Consistency group Control group

Factor 2: variability of knowledge

Posttest

Pretest

M

SD

M

SD

M

SD

Posttest M

SD

3.32 3.51 3.50

0.61 0.62 0.74

3.62 3.96 3.56

0.68 0.66 0.64

4.87 4.84 4.93

0.73 0.83 0.86

5.36 5.41 5.08

0.85 0.77 0.97

partial h2 ¼ .07. In line with Hypothesis 2, the two intervention groups significantly outperformed the control group in terms of changes on the factor Texture of Knowledge. Both groups showed higher values in the posttest, indicating more advanced beliefs after the Web search. Also, for the factor Texture of Knowledge, planned contrasts between the conflict group and the consistency group showed a nonsignificant difference between these groups, F(1, 97) ¼ 0.98, ns; contrary to Hypothesis 3, the two intervention groups did not differ in terms of changes on the factor Texture of Knowledge. For the factor Variability of Knowledge, planned contrasts between the control group and the two intervention groups revealed a significant difference, F(1, 97) ¼ 7.68, p < .01, partial h2 ¼ .07. In line with Hypothesis 2, the two intervention groups showed significantly more change in their beliefs about the variability of medical knowledge than the control group. An inspection of the means revealed that both groups who dealt with information on the Web showed more advanced beliefs about the variability of knowledge in medicine after the Web search. Also, for the factor Variability of Knowledge, planned contrasts between the conflict group and the consistency group revealed a nonsignificant difference between these groups, F(1, 97) ¼ 0.24, ns; contrary to Hypothesis 3, the two intervention groups did not differ in terms of changes on the factor Variability of Knowledge. 3.4. Health decision making and certainty 3.4.1. Question 1: decision making For the two intervention groups, differences between the two measurement times were tested regarding the decision on whether participants’ fictitious friend should take medication Table 2 Mean gain scores and standard deviations for the CAEB factors as a function of experimental group.

Conflict group Consistency group Control group

Factor 1: texture of knowledge

Factor 2: variability of knowledge

M

SD

M

SD

0.30 0.44

0.48 0.72

0.49 0.57

0.65 0.75

0.06

0.42

0.12

0.56

Positive scores indicate higher values and therefore a more advanced view in the post measurement.

D. Kienhues et al. / Learning and Instruction 21 (2011) 193e204

200

Table 3 Frequencies for question 1 in the two intervention groups as a function of measurement time.

Pretest Posttest

Conflict group (n ¼ 30)

Consistency group (n ¼ 34)

Yes

No

I don’t know

Yes

No

I don’t know

6 7

12 19

12 4

11 16

8 11

15 7

to lower her cholesterol. The two intervention groups did not differ significantly in their pretest answers to Question 1, c2(2, N ¼ 64) ¼ 2.36, ns. However, in line with Hypothesis 4, they differed significantly after the Web search, c2(2, N ¼ 64) ¼ 6.25, p < .05, Cohen’s w2 ¼ 0.10. A test of the partial hypothesis (cf. Kimball, 1954, cited in Bortz, Lienert, & Boehnke, 2008) that the two groups would differ with regard to their yes- or no-answers after the Web search was significant, c2(1, N ¼ 64) ¼ 8.86, p < .05, Cohen’s w2 ¼ 0.14. Members of the consistency group advised medication significantly more often, whereas members of the conflict group refused medication significantly more often. Table 3 shows the frequencies for the decision on Question 1 for both measurement times. 3.4.2. Question 2: participants’ own certainty For the question asking participants how certain they were about their decisions, a repeated measures ANOVA was conducted with condition (conflict vs. consistency) as between subjects factor and measurement time (pretest vs. posttest answers) as within subjects factor. Results revealed a significant difference between pretest and posttest answers, F(1, 62) ¼ 25.24, p < .01, partial h2 ¼ .29. However, the interaction between measurement time and condition did not reach significance, F(1, 62) ¼ 0.83, ns. Hence, both intervention groups showed a similar change in the certainty of their decision from the pretest to the posttest contrary to Hypothesis 5. An inspection of the means revealed that both groups indicated a higher certainty about their decision after the Web search (see Table 4). 3.4.3. Question 3: experts’ certainty With regard to the question asking participants how certain experts would be if they had to make a recommendation, a repeated measures ANOVA was conducted with condition (conflict vs. consistency) as between subjects factor and measurement time (pretest vs. posttest answers) as within subjects factor. Results revealed a significant difference between pretest and posttest answers, F(1, 62) ¼ 5.26, p < .05, partial h2 ¼ .08. There was also a significant interaction between condition and measurement time, F(1, 62) ¼ 5.26, p < .05, partial h2 ¼ .08. Inspection of the means indicated that members of the conflict group believed that experts would be less certain after the Web search, whereas members of the consistency group did not show any change between the two measurement times (see Table 4). These findings are in line with Hypothesis 6.

3.4.4. Question 4: perceived personal knowledge For the question asking participants to estimate their personal knowledge about cholesterol, as compared to an average person, a repeated measures ANOVA was conducted with condition (conflict vs. consistency) as between subjects factor and measurement time (pretest vs. posttest answers) as within subjects factor. Results showed a significant difference between pretest and posttest estimates, F(1, 62) ¼ 183.91, p < .01, partial h2 ¼ .75, and a nonsignificant interaction between measurement time and condition, F(1, 62) ¼ 1.00, ns. Contrary to Hypothesis 7, members of both intervention groups thought they knew more about cholesterol after the Internet search (see Table 4). Bivariate correlations between the answers to Questions 4 and 2 showed a positive correlation for both measurement times and both intervention groups (IG1: r ¼ .41, p < .05, IG2: r ¼ .52, p < .01, in the pretest; IG1: r ¼ .42, p < .05, IG2: r ¼ .42, p < .05, in the posttest).

4. Discussion 4.1. Web search effects The aim of the present study was to investigate the potential differential effect of dealing with conflicting versus consistent Web information on different levels of epistemic beliefs, on decision making and certainty. In short, the results revealed that the type of information presented on the websites influenced topic-specific beliefs supporting Hypothesis 1, whereas both intervention groups similarly changed their medicinerelated beliefs towards a more advanced view, contrary to both Hypotheses 2 and 3. For the decision making and certainty, participants dealing with different kinds of information made different final decisions, in line with Hypothesis 4. However, the two intervention groups did not differ in terms of own certainty about their decision or perceived personal knowledge, contrary to both Hypotheses 5 and 7. Yet, the conflict group had less certainty in the ability of experts to come to a certain decision after the Web search, whereas the consistency group did not show this decrement, in line with Hypothesis 6. For topic-specific epistemic beliefs, the results revealed that the two intervention groups differed significantly. The conflict group has less belief in the possibility of finding the one best solution for the task and stated that different opinions on the question at hand may all be (partly) correct, whereas the consistency group held a more positivistic view. It can be concluded that dealing with different types of information evokes qualitatively different beliefs. The conflicting information probably contradicts participants’ prior beliefs, because laypeople’s beliefs about medicine are frequently oversimplified (Feltovich, Spiro, & Coulson, 1989). Therefore, when knowledge is not assumed to be controversial, but the Web search points out that it is, participants are confronted with an opposing view, and this is seen as an impetus for change (Bendixen & Rule, 2004; Kienhues et al., 2008).

D. Kienhues et al. / Learning and Instruction 21 (2011) 193e204

201

Table 4 Mean scores (and SD) for Questions 2, 3, and 4 in the two intervention groups as a function of measurement time. Question 2

Conflict group Consistency group

Question 3

Question 4

Pretest

Posttest

Pretest

Posttest

Pretest

Posttest

2.73 (1.23) 2.47 (1.24)

3.47 (0.90) 3.53 (1.02)

4.10 (0.66) 4.00 (0.92)

3.57 (0.93) 4.00 (0.70)

1.97 (0.81) 2.50 (1.08)

3.43 (0.73) 3.76 (0.65)

Also, as regards the discipline-related epistemic beliefs, changes were found after participants have dealt with information from the Web. But, contrary to what was expected, both intervention groups showed similar changes towards a more advanced view. When asked more generally about medicine and not about the specific topic, both groups found knowledge in medicine to be rather imprecise or unstructured as well as rather open or incomplete after their Web search. The no-intervention control group did not show any major changes between the two times of measurement, and we can therefore exclude a posttest effect. Furthermore, due to the meaningful and significant differences in decision making and topic-specific epistemic beliefs, we assume that participants actually did perceive the different intervention conditions differentially. Because it did not make any difference whether participants dealt with conflicting or consistent websites, the effects obtained cannot be assigned predominantly to the different contents of the pre-selected sites. On the contrary, what was similar for both intervention groups was that they gained additional knowledge through the Web search within the context of a concrete problem (prevention of cholesterol). Both types of pre-selected websites offered new insights into an example of complex knowledge in medicine. Regardless of whether the new information is conflicting or not, it seems to make students aware of the complexity and diversity of medical knowledge. Different studies reveal that laypeople often tend to oversimplify knowledge about medicine (Knight & Mattick, 2006; Spiro et al., 1996). Taking this into account, it would seem that by gaining more knowledge about cholesterol within the context of a concrete problem, the participants realized that they had previously had an oversimplified idea of the texture and variability of medical knowledge. It is worth emphasizing that the changes observed occurred solely through dealing with information on the Web for half an hour. However, the changeability of beliefs might depend on the strength of the pre-existing beliefs: Because all participants were medical laypersons, their initial beliefs may well have been not very elaborated. Therefore, it might be more appropriate to conceive the initially measured beliefs as a kind of epistemic stance rather than as an elaborated and stable system of beliefs. As regards decision making and certainty, answers to the question whether they would advise their fictitious friend to take medication to lower her cholesterol (Question 1) showed that after the Web search, most members of the consistency group recommended medication, whereas most members of the conflict group rejected it. This indicates that the Web information had a crucial contrastive influence on decision making. Whereas members of the consistency group dealt with websites that unanimously endorsed the usefulness of treating cholesterol medically, the members of the conflict group dealt with websites in which taking medicine is partly rebutted and

even declared to be useless or dangerous. It is important to note that most participants opted explicitly for or against medication after the Web search, and did not choose to say ‘‘I don’t know.’’ This is remarkable given the ill-structured nature of the problem at hand. Yet, the two intervention groups did not differ in their certainty about the decision made after the Web search (Question 2). It was pointed out above that patients often strive to make an informed decision by gathering further or alternative information on the treatment of a specific disease. This seems to be an effective strategy with regard to the subjective certainty about the decision to be made, that is, gaining further information increases the certainty about one’s decision, irrespective of the quality of the information dealt with. This is remarkable, because it is clear that the members of the conflict group were aware of the conflicting nature of the information at their disposal. However, they still gained a lot of information about the topic, even if it were not all highly valuable, whereas their first decision was made without being able to refer to any concrete information. The above findings lead one to ask how people, especially in the conflict group, rate the information on the different websites they assessed. Because the group dealing with conflicting evidence did not end up being entirely confused, it is possible that they appraised the information provided by the sources differently, for example, as a function of its trustworthiness. Consequently, individuals may ignore some websites in their decision-making process. Furthermore, the conflict group had less belief in the ability of experts to come to a certain decision after the Web search, whereas the consistency group did not show this decrement (Question 3). This finding means that participants had less belief in the epistemic authority of experts with regard to the question at hand only when they themselves had to deal with conflicting evidence. Due to the fact that different experts on different websites hold different and sometimes conflicting views, participants seem to have realized that there are different views on the topic, and that there is not just one correct solution, but that solutions are partly influenced by the interests people represent. Finally, both intervention groups reported comparable gains of knowledge on the topic (Question 4). It seems that the more knowledgeable people feel about the topic, the more certain they are about their decision and vice versa. This subjective assessment of their own knowledge gain corroborates the interpretation of the results on Question 2. 4.2. Limitations of the study There are some limitations in the study that need to be considered for future research. First, the scope of the CAEB is

202

D. Kienhues et al. / Learning and Instruction 21 (2011) 193e204

limited, because it covers only two dimensions. The CAEB focuses on evaluative associations about the nature of medical knowledge and knowing and therefore on discipline-related beliefs. Future research might consider more explicit aspects of medicine-specific epistemic beliefs, because we still do not know how far the effects found are conferrable to these beliefs. Furthermore, future research should consider further dimensions of epistemic beliefs not covered by the CAEB, such as the justification of knowledge or the role of authority as a source of knowledge (cf. Hofer, 2000). For a deeper understanding of the results of this study, it might also be worth investigating how far the evoked beliefs transfer to related tasks. In addition, future research might consider how results depend on special features of the topic at hand. How to treat increased cholesterol is a rather ill-structured problem. Although knowledge in medicine is never certain, as Fox (1957) has emphasized, there are, of course, areas that are better defined than others. Because participants’ epistemic beliefs are assumed to be rather unelaborated, it would be interesting to replicate the study with a sample possessing more elaborated beliefs. One additional challenge for future research might be to find ways to improve people’s ability to deal with the vast amount of information properly, for example, to provide help on reflecting on the trustworthiness of sources or to support people’s evaluation of their own knowledge gaps. This goal does not just apply to medical information on the Web, but to information on the Web in general. Educational psychology (e.g., research on metacognition) can provide a promising basis for improving people’s Web information literacy. Although this study indicates that people are capable of dealing with conflicting evidence, developmental trends are to be expected in this context. Therefore, the development of competence in handling conflicting evidence deserves further empirical investigation. In addition, school-based instruction addressing the existence of conflicting evidence on one topic might be an important tool for promoting what is often a neglected goal of school-based science instruction: fostering the formation of adequate epistemic beliefs that will form the basis of a critical use of scientific information in everyday situations. Furthermore, it would be a fruitful future research perspective to move beyond focusing on the final outcomes of Web search. Future research would certainly benefit from additionally focusing on a process level of Web search. Such an approach might yield valuable insights into the application of search strategies. Distinct patterns of navigation might, for instance, reflect highly functional search strategies, such as cross-checking information found in one document against the information presented by other authors. Furthermore, it might be the case that people exclude websites from further consideration or do not visit websites at all due to some special characteristics of these websites, for example the assumed trustworthiness of the provider. The selection of websites that are considered for an in-depth elaboration might also be a function of personality variables such as need for cognitive closure.

4.3. Implications of the study Regarding the implications of our results, it is important to note that even though participants shifted towards the view that medical knowledge is rather imprecise or unstructured as well as rather open or incomplete, they were still able to arrive at a decision on the medical problem, and they seemed to handle the knowledge gathered without feeling uncertain or helpless. Therefore, they were able to evaluate different viewpoints, and they did not conceive medical knowledge as completely relativistic. Developmental research has shown that experiencing real and generic epistemic doubt during adolescence is not just a very unpleasant condition. It is also not easy to overcome and can temporarily result in becoming ‘‘relativized’’ (Bendixen, 2002; Chandler et al., 1990). Nonetheless, it is an important condition for the development of beliefs. The findings of the present study imply that a bit of doubt in ‘‘small doses’’ seems to be quite easy to handle. Our participants seem to be equipped to deal with complex information on the Web, and it can be seen that they gain a more realistic discipline-related epistemic viewpoint through searching the Web, because their beliefs in the complexity and variability of knowledge increase. The findings on the perceived personal knowledge, that is, that participants felt they knew much more about the topic after the Web search (cf. Stadtler & Bromme, 2007), have implications for communication between health professionals and their patients. The easy access to medical knowledge on the Web is a challenge to the doctor-patient relation, because laypeople often believe they have acquired considerable knowledge about a specific topic after a Web search (McMullan, 2006). Therefore it is important for health professionals to deal with the new ‘‘informed’’ patient properly. For example, health professionals have to take into account that patients already have formed an opinion before entering the surgery. This opinion has to be addressed, because it might well contain wrong assumptions and misconceptions that will need to be challenged properly. Furthermore, the patients’ wish to be better informed should be acknowledged and supported by guiding them to reliable health websites. In summary, the findings of the present study strengthen the view that new information technologies provide a promising environment in which to exert a desirable impact on laypeople’s epistemic beliefs and decision making. Future studies should take a closer look at the interplay between new information technologies and epistemic beliefs.

Appendix A. Topic-specific epistemic beliefs scale* Introduction Please rate the following statements about your Internet search on the topic cholesterol. Terms like ‘‘this topic’’ or ‘‘this question’’ refer to the impressions you gained while searching the Internet to find out how far increased cholesterol should be treated with statins.

D. Kienhues et al. / Learning and Instruction 21 (2011) 193e204

Statements 1. I think it is justifiable that there are different opinions on this topic. 2. If a group of specialists had to answer this question, they would have known the right decision. (r) 3. If people hold different views on this topic, one opinion is correct and the other is wrong. (r) 4. I don’t think that it is possible for there to be more than one right opinion on this topic. (r) 5. I feel that it is possible to find out the truth on this topic. (r) 6. In my opinion, everybody performing this online search will arrive at the same answer. (r) 7. In my opinion, there is no way to decide which solution is the best one. 8. I do not doubt that there is one right answer to this question, even though I did not find it. (r) *All statements were answered on a 5-point scale ranging from 1 (I totally disagree) to 5 (I totally agree). Statements marked with ‘‘r’’ were re-coded so that lower values indicate a stronger belief in a single clear cut solution, whereas higher values indicate that people do not believe in a clear cut solvability of the task. References Bartholome´, T., Stahl, E., Pieschl, S., & Bromme, R. (2006). What matters in help-seeking? A study of help effectiveness and learner-related factors. Computers in Human Behavior, 22, 113e129. Bates, B. R., Romina, S., Ahmed, R., & Hopson, D. (2006). The effect of source credibility on consumers’ perceptions of the quality of health information on the Internet. Medical Informatics and the Internet in Medicine, 31, 45e52. Bendixen, L. D. (2002). A process model of epistemic belief change. In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 191e208). Mahwah, NJ: Erlbaum. Bendixen, L. D., & Rule, D. C. (2004). An integrative approach to personal epistemology: a guiding model. Educational Psychologist, 39, 69e80. Bendixen, L. D., & Feucht, F. C. (2010). Personal epistemology in the classroom Theory, research, and implications for practice. Cambridge, UK: Cambridge University Press. Bortz, J., Lienert, G. A., & Boehnke, K. (2008). Verteilungsfreie methoden in der biostatistik [Distribution-free methods in biostatistics]. Berlin: Springer. Bra˚ten, I., & Strømsø, H. I. (2006). Epistemological beliefs, interest, and gender as predictors of Internet-based learning activities. Computers in Human Behavior, 22, 1027e1042. Bra˚ten, I., Strømsø, H. I., & Britt, M. A. (2009). Trust matters: examining the role of source evaluation in students’ construction of meaning within and across multiple texts. Reading Research Quarterly, 44, 6e28. Bra˚ten, I., Strømsø, H. I., & Samuelstuen, M. S. (2008). Are sophisticated students always better? The role of topic-specific personal epistemology in the understanding of multiple expository texts. Contemporary Educational Psychology, 333, 814e840. Britt, M. A., & Aglinskas, C. (2002). Improving student’s ability to identify and use source information. Cognition and Instruction, 20, 485e522. Bromme, R., Kienhues, D., & Stahl, E. (2008). Knowledge and epistemological beliefs: an intimate but complicate relationship. In M. S. Khine (Ed.), Knowing, knowledge and beliefs (pp. 423e441). New York: Springer.

203

Buehl, M. M., & Alexander, P. A. (2001). Beliefs about academic knowledge. Educational Psychology Review, 3, 385e418. Buehl, M. M., Alexander, P. A., & Murphy, P. K. (2002). Beliefs about schooled knowledge: domain general or domain specific? Contemporary Educational Psychology, 27, 415e443. Cano, F. (2005). Epistemological beliefs and approaches to learning: their change through secondary school and their influence on academic performance. British Journal of Educational Psychology, 75, 203e221. Chandler, M., Boyes, M., & Ball, L. (1990). Relativism and stations of epistemic doubt. Journal of Experimental Child Psychology, 50, 370e395. Conley, A. M., Pintrich, P. R., Vekiri, I., & Harrison, D. (2004). Changes in epistemological beliefs in elementary science students. Contemporary Educational Psychology, 29, 186e204. Dale, K. M., Coleman, C. I., Henyan, N. N., Kluger, J., & White, C. M. (2006). Statins and cancer risk: a meta-analysis. Journal of the American Medical Association, 295, 74e80. DeLenardo, C. (2004). Web-based tools steer patient-focused care. Nursing Management, 35, 60e64. Elby, A., Frederiksen, J., Schwarz, C., & White, B. (2003). Epistemological beliefs assessment for physical science (EBAPS). Retrieved May 20, 2003, from. http://www2.physics.umd.edu/~elby/EBAPS/home.htm. Elwyn, G., Edwards, A., Wensing, M., Hood, K., Atwell, C., & Groll, R. (2003). Shared decision making: developing the OPTION scale for measuring patient involvement. Quality and Safety in Health Care, 12, 93e99. Ernst, J., Go¨tze, H., Weissflog, G., Schro¨der, C., & Schwarz, R. (2006). Angeho¨rige von krebspatienten: die dritte kraft im medizinischen entscheidungsprozess? Explorative befunde zum Shared decision-making [Relatives of cancer patients: the third party in medical decision-making? Explorative findings on shared decision making]. Familiendynamik, 31(1), 47e69. Eysenbach, G. (2003). The impact of the Internet on cancer outcomes. CA: A Cancer Journal for Clinicians, 53, 356e371. Feltovich, P., Spiro, R., & Coulson, R. (1989). The nature of conceptual understanding in biomedicine: the deep structure of complex ideas and the development of misconceptions. In D. Evans & V. Patel (Eds.), Cognitive science in medicine: Biomedical modeling (pp. 113e172). Cambridge, MA: MIT Press. Fox, R. C. (1957). Training for uncertainty. In R. K. Merton, G. G. Reader, & P. L. Kendall (Eds.), The studentephysician (pp. 207e241). Cambridge, MA: Harvard University Press. Fox, S. (2006). Online health search 2006. Retrieved March 5, 2007, from. Pew Internet and American Life Project. http://www.pewinternet.org/pdfs/ PIP_Online_Health_2006.pdf. Fox, S., & Rainie, L. (2000). The online health care revolution: how the web helps Americans take better care of themselves. Retrieved March 5, 2008, from. www.pewinternet.org. Gil Pelluch, L. (2009). Efectos de las creencias epistemolo´gicas, las condiciones de tarea y el conocimiento previo en la comprensio´n de textos multiples [Effects of personal epistemology beliefs, task conditions, and prior knowledge on understanding of multiple texts]. Unpublished doctoral thesis, Universitat de Valencia, Spain. Gill, M. G., Ashton, P. T., & Algina, J. (2004). Changing preservice teachers’ epistemological beliefs about teaching and learning in mathematics: an intervention study. Contemporary Educational Psychology, 29, 164e185. Guadagnoli, E., & Velicer, W. F. (1988). Relation to sample size to the stability of component patterns. Psychological Bulletin, 103, 265e275. Hack, T. F., Degner, L. F., Parker, P., & the SCRN Communication Team. (2005). The communication goals and needs of cancer patients: a review. Psycho-Oncology, 14, 831e845. Hofer, B. K. (2000). On dimensionality and disciplinary differences in personal epistemology. Contemporary Educational Psychology, 25, 378e405. Hofer, B. K. (2004). Epistemological understanding as a metacognitive process: thinking aloud during online searching. Educational Psychologist, 39, 43e55. Hofer, B. K. (2006). Domain specificity of personal epistemology: resolved questions, persistent issues, new models. International Journal of Educational Research, 45, 85e95.

204

D. Kienhues et al. / Learning and Instruction 21 (2011) 193e204

Hofer, B. K., & Pintrich, P. R. (1997). The development of epistemological theories: beliefs about knowledge and knowing and their relation to learning. Review of Educational Research, 67, 88e140. Hofer, B. K., & Pintrich, P. R. (2002). Personal epistemology: The psychology of beliefs about knowledge and knowing. Mahwah, NJ: Erlbaum. Jacobson, M. J., & Spiro, R. J. (1995). Hypertext learning environments, cognitive flexibility, and the transfer of complex knowledge: an empirical investigation. Journal of Educational Computing Research, 12, 301e333. Kaiser, H. F., & Rice, J. (1974). Little Jiffy, Mark IV. Educational and Psychological Measurement, 34, 111e117. Khine, M. S. (Ed.). (2008). Knowing, knowledge and beliefs: Epistemological studies across diverse cultures. New York: Springer. Kienhues, D., Bromme, R., & Stahl, E. (2008). Changing epistemological beliefs: the unexpected impact of a short-term intervention. British Journal of Educational Psychology, 78, 545e565. Kimball, A. W. (1954). Short-cut formulae for the exact partition of c2 in contingency tables. Biometrics, 10, 42e458. King, P. M., & Kitchener, K. S. (2002). The reflective judgment model: twenty years of research on epistemic cognition. In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 37e61). Mahwah, NJ: Erlbaum. Knight, L. V., & Mattick, K. (2006). ‘‘When I first came here, I thought medicine was black and white’’: making sense of medical students’ ways of knowing. Social Science & Medicine, 63, 1084e1096. Ko¨ller, O., Baumert, J., & Neubrand, J. (2000). Epistemologische u¨berzeugungen und fachversta¨ndnis im mathematik- und physikunterricht [Epistemological beliefs and understanding of science in mathematics and physics lessons]. In J. Baumert, W. Bos, & R. Lehmann (Eds.), TIMSS/III: Dritte internationale mathematik- und naturwissenschaftsstudie e Mathematische und naturwissenschaftliche bildung am ende der schullaufbahn (pp. 229e269). Opladen, Germany: Leske Und Budrich. Kuhn, D., & Weinstock, M. (2002). What matters in epistemological thinking and why does it matter? In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 121e144) Mahwah, NJ: Erlbaum. Limon, M. (2006). The domain generality-specificity of epistemological beliefs: a theoretical, a methodological problem or both? International Journal of Educational Research, 45, 7e27. Maggioni, L., Riconscente, M. M., & Alexander, P. A. (2006). Perceptions of knowledge and beliefs among undergraduate students in Italy and in the United States. Learning and Instruction, 16, 467e491. Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action: spontaneous reflections about knowledge and knowing during online information searching and their influence on learning. Learning and Instruction, 21(1), 137e151. Mason, L., & Boldrin, A. (2008). Epistemic metacognition in the context of information searching on the web. In M. S. Khine (Ed.), Knowing, knowledge and beliefs: Epistemological studies across diverse cultures (pp. 377e404). New York: Springer. Mason, L., & Boscolo, P. (2004). Role of epistemological understanding and interest in interpreting a controversy in topic-specific belief change. Contemporary Educational Psychology, 29, 103e128.

McMullan, M. (2006). Patients using the Internet to obtain health information: how this affects the patient-health professional relationship. Patient Education and Counseling, 63, 24e28. Morahan-Martin, J. M. (2004). How Internet users find, evaluate, and use online health information: a cross-cultural review. Cyberpsychology and Behavior, 7, 497e510. Muis, K. R., Bendixen, L. D., & Haerle, F. C. (2006). Domain-generality and domain-specificity in personal epistemology research: philosophical and empirical reflections in the development of a theoretical framework. Educational Psychology Review, 18, 3e54. Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (2000). Contrasts and effect sizes in behavioral research: A correlational approach. Cambridge, UK: Cambridge University Press. Sadler, T. D. (2004). Informal reasoning regarding socioscientific issues: a critical review of research. Journal of Research in Science Teaching, 41, 513e536. Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82, 498e504. Schommer, M., Calvert, C., Gariglietti, G., & Bajaj, A. (1997). The development of epistemological beliefs among secondary students: a longitudinal study. Journal of Educational Psychology, 89, 37e40. Spiro, R. J., Feltovich, P. J., & Coulson, R. L. (1996). Two epistemic worldviews: prefigurative schemas and learning in complex domains. Applied Cognitive Psychology, 10, 51e61. Stadtler, M., & Bromme, R. (2007). Dealing with multiple documents on the WWW: the role of metacognition in the formation of documents models. International Journal of Computer Supported Collaborative Learning, 2, 191e210. Stadtler, M., & Bromme, R. (2008). Effects of the metacognitive tool met.a.ware on the web search of laypersons. Computers in Human Behavior, 24, 716e737. Stahl, E., & Bromme, R. (2007). The CAEB: an instrument for measuring connotative aspects of epistemological beliefs. Learning and Instruction, 17, 773e785. Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics. Needham Heights, MA: Allyn & Bacon. Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. Washington, DC: American Psychological Association. Trautwein, U., & Lu¨dtke, O. (2007). Epistemological beliefs, school achievement, and college major: a large-scale longitudinal study on the impact of certainty beliefs. Contemporary Educational Psychology, 32, 348e366. Trautwein, U., Lu¨dtke, O., & Beyer, B. (2004). Rauchen ist to¨dlich, computerspiele machen aggressiv?. [Smoking kills and computer games lead to aggressive behavior?] Zeitschrift Fu¨r Pa¨dagogische Psychologie, 18, 187e199. Tsai, C.-C. (2008). The use of Internet-based instruction for the development of epistemological beliefs: a case study in Taiwan. In M. S. Khine (Ed.), Knowing, knowledge, and beliefs: Epistemological studies across diverse cultures (pp. 270e282). New York: Springer. Tu, Y.-W., Shih, M., & Tsai, C.-C. (2008). Eighth graders’ web searching strategies and outcomes: the role of task types, web experiences and epistemological beliefs. Computers & Education, 51, 1142e1153.