Exploring students’ affect and achievement goals in the context of an intervention to improve web searching skills

Computers in Human Behavior 49 (2015) 156–170

Contents lists available at ScienceDirect

Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh

Exploring students’ affect and achievement goals in the context of an intervention to improve web searching skills Dionysia Kroustallaki a, Theano Kokkinaki a, Georgios D. Sideridis b, Panagiotis G. Simos c,⇑ a

Psychology Department, University of Crete, Greece Harvard Medical School, USA c School of Medicine, Department of Psychiatry, University of Crete, Greece b

a r t i c l e

i n f o

Article history:

Keywords: Web search Information problem solving Intervention Affect Emotions Achievement goals

a b s t r a c t The present study assessed the effects of a short-term intervention designed to enhance students’ web searching skills, particularly query formulation, information selection and credibility evaluation. The study also explored students’ affective experiences during web searching and examined the influence of achievement goals on positive and negative affect. Using a longitudinal treatment/control design, 96 fifth and sixth graders searched for information on curriculum-related topics at four sessions. Positive and negative affect was measured before, during and after each search. Multilevel analyses showed that the patterns of change in searching skills differed across conditions, with experimental group showing significant growth throughout intervention in all searching skills, while the control group remained constant or worsened across sessions. Students also experienced high levels of positive and low levels of negative affect. Positive affect remained constant during and across sessions, while negative affect showed a quadratic trend during sessions and decreased slightly across sessions. Main effects of achievement goals on positive and negative affect were found only for mastery-approach goals. A masteryapproach by performance-approach goal interaction was found for negative affect. Ó 2015 Elsevier Ltd. All rights reserved.

1. Introduction In recent years the Internet has provided students easy access to enormous amounts of information. Although children have embraced this opportunity and they are increasingly using web to get information for schoolwork or leisure activities (Livingstone, 2003), they often lack the necessary skills to effectively use online resources, while few educational interventions have attempted to provide instruction and support to young children (Walraven, Brand-Gruwel, & Boshuizen, 2008). Students’ affective states that naturally occur during online information searching appear to influence their overall performance and the cognitive strategies they adopt to complete the search task (Lopatovska & Arapakis, 2011). Although research on affect during online searching is limited (e.g., Bilal, 2000; Flavian-Blanco, Gurrea-Sarasa, & OrusSanclemente, 2011; Kuhlthau, Heinström, & Todd, 2008; Nahl, 2005; Wang, Hawk, & Tenopir, 2000), numerous studies have examined the role of emotion in different learning settings, i.e., in both technology-enhanced (e.g., D’Mello, 2013) and traditional contexts ⇑ Corresponding author at: School of Medicine, University of Crete, Voutes Campus, Herakleion 70013, Greece. Tel.: +30 2811112888. E-mail address: [email protected] (P.G. Simos). http://dx.doi.org/10.1016/j.chb.2015.02.060 0747-5632/Ó 2015 Elsevier Ltd. All rights reserved.

(e.g. Artino & Jones, 2012; Pekrun, Goetz, Frenzel, Barchfeld, & Perry, 2011; Shen, Wang, & Shen, 2009). Studies on naturally-occurring and experimentally induced affective states highlight the critical role of emotions for the regulation of cognitive processes such as attention, decision making, problem solving, and achievement (Blanchette & Richards, 2010; Isen, 2008; Valiente, Swanson, & Eisenberg, 2012). Research has also established the critical role of motivation in traditional achievement settings for determining students’ behavior, cognitive strategies, performance, and affective states (Linnenbrink & Pintrich, 2002; Pekrun, Elliot, & Maier, 2006; Wirthwein, Sparfeldt, Pinquart, Wegerer, & Steinmayr, 2013). One of the most prominent motivational constructs, namely achievement goals, emphasizes the end results that students strive to attain. During learning, students seek to be competent (Elliot & Church, 1997). Thus, some students may be oriented toward learning and understanding new material, while others may be primarily interested in demonstrating competence and performing better than others (Harackiewicz, Barron, Tauer, Carter, & Elliot, 2000). In that respect, the primary reason why students engage in learning activities is either to demonstrate high ability or to develop ability for the task at hand (Brophy, 2010).

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170

The present study addresses the scarcity of research on the relationship between motivation and affect experienced by children who systematically engage in educational tasks involving assistive technologies (web-searching). Specifically, we monitored the trajectories of general (i.e., positive and negative) affect during performance of web-searching tasks in relation to students’ preferred achievement goals. Moreover, a specific educational (learning) context was simulated through a systematic training program on websearching skills and compared to a condition of free web search. The ultimate goal of the study was to contribute to a better comprehension of the emotional and motivational processes that operate during learning. A thorough understanding of how these processes interact holds the potential for building better learning environments (Pekrun, 2011) and helping students recognize and self-regulate their emotional states in adapting to new and complex learning situations. In the next section, we will first review existing research on the cognitive skills involved in web searching, focusing on elementary school students and including intervention programs aimed at skill enhancement. Then, we will present studies examining affective and motivational processes during web searching or, in the absence of relevant studies, in other academic contexts. 2. Research framework 2.1. Elementary school students and web searching Students from a young age search the Internet for schoolwork-related information both at home and at school (Haddon & Livingstone, 2012; Purcell, Heaps, Buchanan, & Friedrich, 2013). During web searching they engage in an inquiry process involving numerous cognitive skills (Leu et al., 2011). Until recently, what we knew about web searching skills was largely based upon traditional research on print sources and texts. Yet, several researchers have argued that simultaneous, online searching and reading involves novel literacy practices (Coiro & Dobler, 2007; Leu, Kinzer, Coiro, & Cammack, 2004; Mills, 2010). Based on the few available studies on this topic, it appears that effective online searching and reading requires two distinct sets of skills, namely navigation and text processing skills (OECD, 2011). The former skills concern the ability to recognize and employ various navigation tools (e.g. scroll bars, hyperlinks and menus) and be familiar with heterogeneous textual structures and features (Rouet, 2006). Text processing, involves locating appropriate information, making relevance and credibility judgments and integrating information across multiple texts (Brand-Gruwel & Stadtler, 2011). Several studies have argued that the majority of elementary and middle school students have not mastered either set of skills adequately (Walraven et al., 2008). Thus, they often encounter difficulty in specifying appropriate keyword terms and use, instead, full sentences or natural language (Bilal, 2000; Kafai & Bates, 1997; Large, Beheshti, & Rahman, 2002; Schacter, Chung, & Dorr, 1998; Wallace, Kupperman, Krajcik, & Soloway, 2000), look for readymade answers in online texts (Hirsh, 1999; Wallace et al., 2000), do not read in depth (Kafai & Bates, 1997; Wallace et al., 2000), and usually accept information without evaluating its accuracy or validity (Hirsh, 1999; Schacter et al., 1998). Previous evidence suggests that students with experience in web searching and other Internet activities are more successful in locating information, complete search tasks faster and are more thoughtful in their selection of credible sources of information (Bilal, 2000; Lazonder, Biemans, & Wopereis, 2000; Metzger et al., 2013; Tu, Shih, & Tsai, 2008). Another variable that has been investigated in the context of online searching is gender (e.g. Kafai & Bates, 1997; Roy, Taylor, & Chi, 2003). Although existing evidence suggests that boys and girls exhibit different search patterns (Large, 2005; Roy & Chi, 2003; Schacter et al., 1998), more recent studies do not support the impact of gender

157

in the search process (e.g. Kingsley, 2011). The role of gender in cognition is complex and understudied in the current context, and thus the issue remains unresolved. Apart from gender and experience, age may influence search performance and the mastering of the constituent searching skills. As one would expect, children, teenagers and adults experience different problems during web searching (Large, 2005). Although older children have in general better searching skills (Livingstone, Bober, & Helsper, 2005), both children and teenagers encounter problems in specifying search terms, judging search results and deciding on source and information relevancy (Walraven et al., 2008). Also, it seems that some of the searching skills are acquired before others (e.g., in contrast to younger students, teenagers can adequately store relevant information). A related issue is whether and how young students benefit from instruction targeting web searching skills (De Vries, van der Meij, & Lazonder, 2008; Kingsley, 2011; Kuiper, Volman, & Terwel, 2008). Among recent attempts in this direction, Zhang and Quintana (2012) created a software scaffolding tool for sixth grade students’ inquiry projects supporting the different stages of web searching (i.e., planning, searching, analyzing, evaluating and synthesizing information) by using relevant prompts, facilitating planning and monitoring processes and providing computer-aided techniques for recording and citing information. Similarly, Kuiper, Volman, and Terwel (2009) focused their instruction on three sets of web literacy skills, namely web searching, web reading and interpreting and web assessment and evaluating of information. This successful training program was implemented in four fifth grade classes during a period of 10 weeks, with students searching online collaboratively and being supported by explicit instruction and guidance during their projects. Nevertheless, they applied skills inconsistently and irregularly during assignments, moving between newly acquired strategies to earlier, less effective ways of searching, reading and evaluating information. Lastly, most published research to date has measured online searching at single time points, while limited studies to date have collected longitudinal data (e.g. Hirsh, 1999; Wallace et al., 2000) and only one has analyzed individual growth (e.g. Gerjets & Hellenthal-Schorr, 2008). As a result, conclusions regarding age differences rely mostly on cross-sectional data, which should make us cautious in drawing inferences about developmental processes (Kraemer, Yesavage, Taylor, & Kupfer, 2000). The present study aimed to extend previous research in two ways. On the one hand, we attempted to capture the developmental change of web searching skills. As Paris (2005) suggested with regard to reading acquisition, the developmental trajectories of skills differ in terms of rate and duration of acquirement. Some skills are learned quickly, in relatively brief developmental spans, while others continue to develop throughout life (Paris & Hamilton, 2009). In a similar way, the component skills of online searching may grow at different rates. Keeping this in mind, we examined whether searching skills displayed distinct developmental trajectories, both within and across individuals, by measuring skills at multiple searching sessions (Hedeker & Gibbons, 2006). We also implemented a short-term training program targeting web searching skills of upper elementary students. Concurrently, we examined the influence of background variables like gender and web experience on the development of searching skills. On the other hand, apart from the cognitive dimensions of web searching, in the current research we examined the affective and motivational aspects of information problem solving. 2.2. Affect during web searching What we know about school students’ affect during web searching is largely based upon descriptive studies with small samples,

158

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170

using interviews, diaries or single-item measures (e.g., Bilal, 2000; Bilal & Bachir, 2007; Kuhlthau et al., 2008). For example, Bilal (2000) and Bilal and Kirby (2002) investigated twenty-two secondary students while using a search engine designed for children and reported that most students enjoyed using the Web, despite occasional confusion and frustration. Similarly, Kuhlthau et al. (2008) reported predominant feelings of uncertainty and confusion at Time 1 of searching either print or electronic resources turning into frustration at Time 2, and confidence, relief and satisfaction at the completion of the search. With the few aforementioned exceptions, research to date has measured affect at a single time point and thus have not been able to describe affective trajectories during web searching. Given that in principle, web searching is a dynamic process that evolves over relatively short time periods (in the order of minutes), it is important to describe students’ affective trajectories during searching. The present study was designed to examine the course of affective states before, during and after each web searching session. Using multiple measures instead of a single snapshot in time, we aimed to study differences in affect both between and within students (Shockley, Ispas, Rossi, & Levine, 2012). In contrast to other approaches that focus on discrete emotions in school settings (Pekrun & Linnenbrink-Garcia, 2012), we assessed general dimensions of affective experience (positive and negative) (Watson, Clark, & Tellegen, 1988) which can be readily related to by school-aged students (Schwartz & Trabasso, 1984). 2.3. Goal orientations and affect The goals adopted by students in school work are referred to as achievement goals (Dweck & Leggett, 1988; Elliot, 2005). According to a popular framework, achievement goals are classified into four distinct categories: mastery approach, mastery avoidance, performance approach and performance avoidance goals (Elliot & McGregor, 2001; Pintrich, 2000a). Two of the four types of goals refer to approach motivation – the tendency toward positive events or objects, i.e., success, while the other two goal orientations refer to avoidance motivation– the tendency away from negative events or objects, i.e. failure (Elliot, 2008). Achievement goal research has demonstrated that mastery approach goals are positively related to positive affect (for a review, see Hulleman, Schrager, Bodmann, & Harackiewicz, 2010; Kaplan & Maehr, 1999; Linnenbrink, 2005; Senko, Hulleman, & Harackiewicz, 2011) and negatively correlated with negative affect (Kaplan & Maehr, 1999; Linnenbrink, 2005). Conversely, mastery avoidance goals may be related to negative affective experiences (Elliot & Church, 1997; Elliot & McGregor, 1999; Sideridis, 2008; but see also Madjar, Kaplan, & Weinstock, 2011). Findings regarding the relationship between performance-approach goals and affect are less consistent. A number of studies have reported positive correlations between positive affect and performance-approach orientation (Linnenbrink, 2005; Pekrun et al., 2006), whereas the opposite trend was reported by others (Kaplan & Maehr, 1999). Previous studies have also shown either positive (Kaplan & Maehr, 1999; Meyer, Turner, & Spencer, 1997) or negligible correlations (Linnenbrink, 2005) between performance-approach goals and negative affect. Conversely, it appears that performance-avoidance goals are positively related to negative affect (Elliot & McGregor, 2001; Pekrun et al., 2006; Sideridis, 2003, 2005). Mastery and performance goals may also operate jointly on cognitive and emotional outcomes (Harackiewicz, Barron, Pintrich, Elliot, & Trash, 2002). Within a multiple-goals perspective, Barron and Harackiewicz (2003) described two alternative models. The additive goal model postulates that mastery approach and performance approach goals have distinct, positive outcomes on

achievement and affective experiences. Thus, mastery approach goals focus student attention on the development of competence, encourage involvement in the task and promote adaptive patterns of affect (Pintrich, 2000b). Respectively, performance approach goals focus attention on the possibility of achieving positive outcomes (i.e., do better than others) and are expected to be beneficial with regard to emotions and task involvement (Pekrun et al., 2006). According to the interactive goal model, students who simultaneously adopt mastery and performance approach goals may develop an adaptive pattern of affect, cognition and achievement (Harackiewicz et al., 2002). In other words, when students focus on developing their competence together with demonstrating their ability, they are advantaged by having multiple goals to motivate them. Previous work has demonstrated that students who adopt both mastery and performance goals have an advantage in achievement (Pintrich, 2000b), persist longer in learning activities (Sideridis, 2006) and experience more enjoyment in learning (Dela Rosa & Bernardo, 2013) compared to students who pursue either mastery or performance goals alone. In view of the purported importance of achievement goals for student affective experiences, and the growing role of web searching in the educational process, it is surprising that the relation between goals and affect in the context of web searching has not been systematically investigated. Given the hypothesized specificity of achievement goals and emotions for distinct academic and learning domains (Bong, 2001; Goetz, Frenzel, Pekrun, Hall, & Lüdtke, 2007), we explored how the motivation construct of achievement goals apply to a novel setting, namely online searching. As outlined in detail below, one of the aims of this study is to examine sources of systematic individual variability in affective trajectories within the students’ personal achievement goals. Another objective of our study is to examine the additive and interactive effects of achievement goals on affective states, i.e., evaluate whether mastery approach and performance approach goals have independent and interactive effects on students’ affect. 2.4. Current study In the present study we developed and implemented a shortterm training program targeting web searching strategies, reading and evaluation skills of upper elementary students. We also explored students’ affective states during web searching and examined the influence of achievement goals on positive and negative affect in this learning setting. We addressed the following research questions: (1) What is the pattern of change in students’ web searching skills over time and across different experimental groups (instructional support versus free search), as measured by their performance on query formulation, information selection and credibility evaluation? (2) What is the impact of individual characteristics, such as gender, grade, web experience and cognitive abilities on students’ developmental trajectories? (3) How does students’ reported level of positive and negative affect evolve during web searching? (4) How do achievement goals separately and jointly, that is additively or interactively, impact affect level and rate of change during a web search session? 3. Method 3.1. Participants and research design The participants were randomly selected from 16 classrooms in seven school units in southern Greece (Crete; 13 urban, 2 two small town and one rural classroom. From each participating classroom we randomly selected six students to be assigned to either (1) the experimental condition (n = 51), in which students were

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170

taught web search skills in four instructional sessions, or (2) the control condition (n = 45), in which students did not receive any training. Students drawn from a given classroom were assigned to the same condition given that cluster randomization minimizes treatment diffusion and makes intervention procedures more feasible (Rhoads, 2011). Thus, eight classrooms were assigned to each experimental group. The majority of students had access to the Internet from a computer at home (71%). Of the latter, near a third (33%) reported going online daily, and 33% going online at least 2–3 times a week. Complete data were available from 96 Grade 5 (n = 47, 24 girls) and Grade 6 students (n = 49, 31 girls), aged 11.4 years (SD = 0.64, range = 10–13 years). The majority of students had access to the Internet from a computer at home (71%). Of the latter, near a third (33%) reported going online daily, and 33% going online at least 2–3 times a week. Chi square analyses revealed no reliable differences between experimental and control group in frequency of Internet use and availability of Internet connection at home. 3.2. Procedures and instructional sessions Research procedures had been approved by the Greek Ministry of Education and all parents provided written informed consent. The experiment was implemented over one school year (2010– 2011) at schools’ computer laboratories during regular school hours. In order to gain experience in implementing the training, we conducted a pilot study with 36 Grade 5 and 6 students. In the main study experimental participants engaged in instructional activities in groups of six students. Following each instructional unit and on a different day of the week, students completed individually a searching task using a popular search engine within a preset amount of time, namely, a period of twenty minutes. Participants in the control group were, again, assigned to groups of six students, and performed the same research tasks individually and at equal time intervals over a period of three weeks, without receiving any training or other search-related instructions. Development and implementation of the training program was based on the Information Problem Solving-Internet Model (ISP-I) (Brand-Gruwel, Wopereis, & Walraven, 2009). According to the ISP-I, searching for specific answers on the web requires students to engage in complex cognitive activities in five stages: (1) define information problem by formulating questions and clarifying task requirements, (2) search information using specific search terms, (3) scan and locate information, (4) process information by reading texts in depth and evaluate the accuracy and utility of information retrieved, and (5) organize and present information creating a final product. Due to time constraints, a short-term training module was designed focusing on keyword formation, information selection and credibility evaluation. Instruction of Internet searching skills followed the three-phase framework proposed by Leu et al. (2008). The training program consisted of three 45-min sessions taking place over three consecutive weeks and all units were delivered by the first author. The first session introduced techniques for keyword selection. The instructional goal of this unit was for students to be able to select appropriate search terms and revise them when necessary. The following learning activities were included: At the beginning, the instructor introduced the target skill using a stimulating question related to students’ personal experiences. The class discussion gave the instructor the opportunity to explore students’ prior knowledge on how we convert questions into a set of specific search terms. Next, the instructor explicitly modeled and explained the target skill. During this stage, she demonstrated her thinking as she performed the task: she avoided using a question, selected nouns instead of proper sentences, looked at the results to find alternative phrasing and used more precise words to narrow results. Then, students were given the opportunity to practice the

159

process of keyword selection and modification themselves, forming a simple search for schoolwork-related questions. The second session introduced techniques for reading and selecting information. The instructional goal of this unit was for students to be able to skim the text, identify various elements of text structure and read selectively to locate relevant information. For the development of instructional activities we applied the TaskBased Relevance Assessment and Content Extraction (TRACE) Model (Rouet, Ros, Goumi, Macedo-Rouet, & Dinet, 2011). Following Rouet and colleagues’ guidelines, we first encouraged students to make sure that they understood the given questions and instructions and could determine the kind of information that was needed for their assignments. The next step concerned the use of textual devices such as headings, illustrations and other text organizers. By paying attention to these text features students would be able to determine what was important and avoid distracting material. Students were also encouraged to quickly browse contents in order to be able to make quick relevance assessments and skip irrelevant parts. When relevant pieces of information were located, students were instructed to process information deeper in order to retrieve suitable content. More specifically, lesson began with a class discussion about the central topic (e.g. How do we read a web text?). During this stage, students exchanged effective reading strategies and shared ideas and problem-solving skills. Next, the instructor explored selected web sites and demonstrated online reading strategies by displaying computer screen and navigation moves on a projector. She used text organizers such as headings, highlighted words, summaries and hyperlinks to locate relevant information while simultaneously avoiding reading word by word. Instead, she scanned text to locate related sections, skipped irrelevant material and when information seemed especially important, she read slowly to locate relevant parts. Subsequently, students worked in pairs to locate appropriate information within pre-selected websites for a curriculum-related question, while instructor provided the necessary support in the form of reminders and hints. The third session introduced criteria for assessing the quality of information. The instructional goal of this unit was to enable students to use different criteria in order to judge trustworthiness of information and sources. Five evaluative criteria that students should apply in their assessments, have been identified in previous work, including accuracy, authority, objectivity, currency and scope (Tate, 2010). Here we focused on two key criteria in order to reduce instruction time and increase compliance (Metzger, 2007): (a) accuracy – whether information can be verified online by checking different Websites or offline by using external sources, such as books, teachers and personal experience, and (b) authority – examining who writes or publishes the information and whether websites belongs to a particular genre (e.g. commercial, educational, informational, personal or forum). At the beginning of this session, the instructor started with a question to gauge students’ knowledge of the subject (i.e., whether they can trust what they read on the Web). Students worked as a group to discuss the characteristics of credible information and created a list of evaluation criteria in the form of a question (e.g. Who wrote the page? Is information consistent with other sources? Does information agree with our prior knowledge?). Students were also introduced to a variety of websites and their corresponding features, i.e., audience, type of information and purpose of publishing information. Students later visited different kinds of pre-selected websites in pairs and decided on their genre and purposes. 3.3. Research tasks Students in both conditions were asked to complete four written assignments on given topics. These information seeking tasks

160

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170

were related to the issue of recycling and pilot-tested to ensure that they were motivating, at an appropriate level of difficulty and could be completed within a school-time period. As all teachers confirmed, students had been introduced to the topic of recycling and were involved in recycling-related projects in different subjects across the curriculum. For this reason, we did not expect students’ prior content knowledge on the topic of recycling to vary considerably. The inquiry tasks were: (a) (b) (c) (d)

Why should we recycle used batteries? Where can we dispose of our broken electronics? How does an old car get recycled? What are the benefits of recycling?

At the beginning of each task students were provided with a worksheet including the research question and three columns with the following headings: (1) Relevant information, (2) The URL of the source, and (3) Can I trust the information? Students were instructed to write their answers in the blank rows under each heading. The assignments were later evaluated on the following scoring criteria: (1) relevance of information, (2) comprehensiveness of answer, (3) critical evaluation of information, and (4) referencing of web sources. Each item included four levels of performance (poor, fair, good and excellent) and students’ assignments were evaluated accordingly. Feedback in the form of written and verbal comments was provided to students individually on each subsequent session. The inquiry tasks were presented in both experimental and control group using a variety of motivational strategies. Specifically, each inquiry task was introduced in a creative manner in order to stimulate curiosity (e.g. short video clips, probing questions, PowerPoint presentations), linking research questions to students’ everyday experiences. These activities are known to foster a mastery-oriented searching environment, emphasizing learning and understanding of online material. We further introduced a scoring system based on students’ search performance. At the beginning of the program, students were informed that top scoring performances would be recognized by asking them to present their project to the rest of their class. After each searching session, written assignments were marked and students were awarded a score for their performance (poor, fair, good, or excellent). Later, points achieved for each inquiry task were added to provide a total score. In this manner the instruction program attempted to simulate an actual classroom environment, which typically features performance standards. 3.4. Measures 3.4.1. Query formulation Web navigation sessions were recorded using a screen capture software. Students’ queries were first transcribed and then analyzed by counting the number of appropriate search terms entered into the search box. Appropriateness was judged according to whether students followed the learned rules of query formation as stated in Google Search Basics instructions, i.e., whether students used simple and descriptive terms, avoided full questions and ignored unnecessary words. Next, the number of appropriate search terms was divided by the total number of each students’ search terms entered during separate sessions. Thus, the resulting dependent variable was a proportion with ranging values between 0 to 1. 3.4.2. Information selection Students’ written answers to the inquiry tasks were rated in relation to information relevance on a four-point rating scale, ranging from 0 (no answer or all information unrelated to the task), 1

(some information related to task), 2 (most information related to task) to 3 points (all information related to task). Information was judged as relevant by the extent to which it appropriately matched to the target assignment and it was useful in the context of the research topic (Hirsh, 1999). Students’ answers were scored by the instructor, blindly to the students’ identity. Following a brief training session, one independent rater scored 15% of the written answers and reached substantial agreement with first observer (Cohen’s Kappa = 0.72). 3.4.3. Credibility evaluation Students’ credibility assessments were rated with respect to their quality on a four-point rating scale, ranging from 0 (does not apply evaluation criteria), 1 (refers to some evaluation criteria without further justification or explanation), 2 (shows sufficient evidence of the application of one evaluation criterion) to 3 points (uses a variety of evaluation criteria). Examples of specific behaviors for the four levels of performance are provided in Table 1 (see Appendix A). Students’ answers were scored by the instructor, who was blind to students’ identity. Following a brief training session, two independent graduate assistants/observer scored 15% of students’ assignments. Cohen’s inter-rater reliability coefficient was k = 0.84, suggesting almost perfect agreement between raters (Landis & Koch, 1977). 3.4.4. Positive and negative affect Positive and negative affect was measured using twelve items chosen from the Positive and Negative Affect Scale for Children (PANAS-C; Laurent et al., 1999), an adapted version of PANAS-X (Watson & Clark, 1994). Positive affect was assessed using the items strong, proud, satisfied, enthusiastic, interested and active. Likewise, negative affect was assessed with the items afraid, distressed, disappointed, irritable, ashamed and confused. Students responded on a scale from 1 (not at all) to 5 (very much), reporting the extent they experienced each emotion at that particular moment; that is, prior to, during and immediately after each of the four search sessions. Responses were recorded on paper before and after the searching session and electronically during the search task in order to minimize interference with the task itself (10 min after the onset of the task labeled smileys appeared on the computer screen directly above the rating scale and students clicked on the appropriate response category). Cronbach’s alphas across time points ranged from .70 to .92 for positive affect items and from .72 to .88 for negative affect items. 3.4.5. Achievement goals Achievement goals were assessed with a modified version of the Achievement Goals Questionnaire (Elliot & McGregor, 2001), adapted in Greek (Sideridis, 2008). Some of the scale items were modified to render them relevant to web searching tasks (as opposed to general classroom achievement). The scale measures four goal orientations and each sub-scale reflects the following types of motivation: (1) improve understanding of the material encountered and strive to reach one’s full potential (mastery approach), (2) avoid insufficient understanding of the material and miss opportunities to master the task at hand (mastery avoidance), (3) strive to do better than others and to appear competent (performance approach), and (4) avoid doing worse than others (performance avoidance). Masteryapproach goals were assessed with six items (Cronbach’s a = .71; e.g., How important is it to you to understand how to solve tasks in this class?); mastery-avoidance goals were assessed with three items (Cronbach’s a = .84; e.g., Do you worry that you will not do as well as you would like to in this class?); six items were used to assess performance-approach goals (Cronbach’s a = .86; e.g., How important is it to you to get a better grade than most of the other students?); the

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170 Table 1 Descriptive statistics for the variables included in web searching skills models. Variables

Session 1 M (SD) %

Session 2 M (SD) %

Session 3 M (SD) %

Session 4 M (SD) %

Query formulationa Information selection No relevant information Few relevant information Most relevant All relevant

.50 (.36)

.62 (.37)

.58 (.37)

.43 (.43)

15.6 14.3 16.9 53.2

29.3 13.0 9.8 47.8

33.0 10.2 8.0 48.9

16.0 11.1 6.2 66.7

Credibility evaluation No criterion One unjustified criterion One criterion justified A variety of criteria

49.4 27.3 22.1 1.3

40.2 22.8 29.3 7.6

39.1 18.4 32.2 10.3

28.8 25.0 37.5 8.8

Searching for schoolwork Yes No

68.8 31.3

Searching for leisure Yes No

50 50

Subtests Vocabulary Similarities Block design Coding

10.9 12.1 10.4 10.5

(2.5) (2.9) (2.4) (3.0)

Note. Means and standard deviations are presented for continuous variables. Percentages are reported for the categorical and ordinal variables. a Proportion data (the number of appropriate search terms divided by the number of total search terms used in each session) bounded at 0 and 1.

remaining five items were used to assess performance-avoidance goals (Cronbach’s a = .70; e.g., Are you worried that you may not get a good grade in this class?). Given the longitudinal design of the study, a prerequisite to evaluating changes in positive and negative affect over time was the presence of configural and metric invariance. That is, factor structures of the brief version of the PANAS should be equivalent across all three time points (configural invariance) when tested independently and the additional constraint of equal factor loadings should also hold (metric invariance). Initially, power for the confirmatory factor analysis (CFA) model in order to obtain a root mean square error of approximation (RMSEA) values between .05 and .08 was excessive (99.9%) (MacCallum, Browne, & Sugawara, 1996). Thus, a level of significance equal to .01 was adopted to balance those power levels. Two models were run: (a) a simultaneous 3-group CFA model, and, (b) the configural model with the additional imposition of equality constraints across all combinations of groups (3  12 = 36 constraints). Results indicated that configural invariance was met as model fit was good (SRMR = .06, RMSEA = .09, IFI = .91) when the factor structure of the three groups was tested simultaneously. Furthermore, when 36 equality constraints were imposed only two exceeded conventional levels of significance. Thus, partial metric invariance was present. These findings support the validity of testing longitudinal models of affective change in that the observed point estimates and variances were comparable across time points. 3.4.6. Web searching experience A questionnaire was designed to measure aspects of students’ web experience, such as computer and Internet access at home, hours spent on the Web, and frequency of various online activities, including browsing the Internet for fun, using e-mail, chatting online and information searching for schoolwork-related projects or leisure-related activities. The questionnaire was administered at the beginning of the program.

161

3.4.7. Cognitive abilities The Vocabulary, Similarities, Block Design, and Coding subtests from the Greek standardization of the Wechsler Intelligence Scale for Children-Third Edition (WISC-III; Georgas, Paraskevopoulos, Bezevegis, & Giannitsas, 1997; Wechsler, 1991) were administered to students during the last week of the program according to standard instructions. The raw scores of the four subtests were included in the analysis. The subscales of WISC-III were used to assess different aspects of intelligence and statistically control for individual differences on preexisting cognitive abilities that might account for searching performance. 4. Results We analyzed two sets of dependent variables related to: (a) web searching skills, including query formulation, information selection and credibility evaluation, and (b) affective experience (positive and negative affect). Firstly, we tested changes in web searching skills over time attributable both to the experimental condition and individual characteristics. Secondly, we tested changes in affect over time and the impact of achievement goals on affect level and rate of change during web searching sessions. Granted that repeated measurements from the same subjects are intercorrelated and have a nested structure (i.e., observations are nested within individual students), we applied multilevel modeling, which models correlation between measurements accordingly and takes the hierarchical nature of the data into account (see Raudenbush & Bryk, 2002). Additionally, given that we used clusters (i.e., classrooms) instead of individuals as the unit of random assignment, we specified three-level models to account for the cluster randomization (Friedman, Furberg, & DeMets, 2010), having repeated observations at the first level, individual students at the second level and different classrooms at the third level. Linear or generalized linear mixed models were constructed on the basis of whether the dependent variables were continuous, ordinal or proportions. There is growing recognition that multilevel modeling is one of the most appropriate tools for assessing within and between subject changes for nested data (i.e., for models that include random effects; Bolker et al., 2009). Multilevel models can also handle data with unequal number of repeated measures, using all available observations for the estimation of parameters and, thus, reducing loss to the sample due to missingness (Hox, 2010). Generalized multilevel models can be considered as extensions of logistic regression models for nested, non-normal data, such as proportions and ordinal outcomes (Agresti, Booth, Hobert, & Caffo, 2000). 4.1. Web searching skills Descriptive statistics for the variables included in the analysis are reported in Table 1. As a proportion Query Formulation has a binomial distribution and was analyzed using quasi-likelihood methods in SPSS v.20 (SPSS, Inc.). Information Selection and Credibility Evaluation were four-level ordinal variables. In order to test the assumption of proportional odds (i.e., whether the effects of independent variables were similar across different levels of the dependent variables), we performed a series of binary regressions on dummy variables (see, Hox, 2010), using as independent measures the variables of time and time by intervention interaction term. Since regression coefficients were inconsistent across categories, we assumed differential effects of the covariates on the thresholds separating the four categories of the outcome measures. In that respect, the effects of time and intervention were not similar across categories of the dependent variables. As a result, the proportional odds assumption was relaxed,

162

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170

creating partial proportional odds models and allowing for the explanatory variables of time and time by intervention group interaction to have heterogeneous effects across thresholds (Hedeker & Mermelstein, 1998). The parameters were estimated using numerical quadrature in Supermix (Hedeker, Gibbons, Du Toit, & Cheng, 2009). All data were first grand-mean centered before model computations. Table 2 presents the results from the generalized linear mixed models for each of the three outcome variables: query formulation, information selection and credibility evaluation. We tested the effects of time and experimental condition on web searching skills and the interaction effects between these variables. We did not test for polynomial trends considering that the logistic growth curve describes a nonlinear developmental curve (Hox, 2010). We included baseline measure as part of the response vector and made no assumptions about group differences at baseline. Although there is extensive literature on how to handle baselines, Fitzmaurice, Laird, and Ware (2004) have recommended that retaining baselines as part of the outcomes is a viable strategy in randomized longitudinal studies (Fitzmaurice et al., 2004). All analysis controlled for cognitive abilities and web searching experience. Having the variance fixed at p2/3, the unconditional models gave Intraclass Correlations Coefficients (ICCs) of .22 for query formulation, .19 for information selection and .28 for credibility evaluation at the classroom level (level 3), indicating significant differences between classrooms that formed the higher level. The corresponding intraclass correlations at the student level (level 2) were .01 for query formulation, .12 for information selection and .19 for credibility evaluation. With respect to Query Formulation, a main effect of time (i.e., session) was significant (b = 0.71, p < .001, OR = 0.49), indicating that the odds of selecting appropriate keywords for a student in the control group (reference category) in each following session decreased by a factor of 0.49. Each additional point on the WISCIII Vocabulary scale resulted in a 13% increase in the odds of selecting appropriate keywords (b = 0.12, p < .01, OR = 1.13). The significant interaction between Session and Condition (b = 0.98, p < .001, OR = 2.68) suggests that the odds of selecting appropriate keywords were 2.68 times higher every following session for the students in the experimental group than for the students in the control group. With respect to Information Selection only the interaction of Condition by Session was significant (b = 0.58, p < .05, OR = 1.79), suggesting that the odds of selecting only relevant information (versus the three lower categories) on each subsequent session were 1.79 times higher for the students in the experimental group than for the students in the control group. Concerning Credibility Evaluation there was, again, a significant interaction of Condition and Session (b = 0.68, p < .001, OR = 1.97), indicating that the odds of using a variety of evaluative criteria on each subsequent session were 1.97 times higher for the students in the experimental group than for the students in the control group. There were no significant effects of gender, grade and cognitive abilities on the rate of change on any of the dependent variables. 4.2. Affect and achievement goals The presence of violations of the assumptions of normality, linearity and homoscedasticity in the data, rendered the use of robust estimation methods (Huber/White or sandwich estimator (Maas & Hox, 2004) in SAS PROC MIXED, v. 9.2) necessary. First, for each dependent variable we calculated intraclass correlations for the intercept-only models. Initially, three-level models were run in order to assess differences between classrooms, suggesting that all classrooms responded similarly and all variability lay instead within classrooms. Final analyses adopted a two-level structure

with repeated measures (level 1) nested within students (level 2). The lowest level included the linear predictor time describing measurements of affect at three time points (i.e., before, during and after each searching session) and the time-varying covariate session describing the four web searching periods. Both variables were centered by coding the first measurement point as zero. At the student level, scores on each of the four goals subscales (mastery approach, mastery avoidance, performance approach and performance avoidance goals) were included as predictor variables centered around their respective grand mean. The intraclass correlations at the repeated measure level (level 1) were .51 and .68 for positive and negative affect, respectively. Corresponding intraclass correlations for the student level (level 2) were .48 and .31, indicating the presence of considerable similarity among observations, both within and between students, and suggested that multilevel modeling was appropriate for dealing with the dependency between measurements and the presence of variability at both levels of measurement. Two interaction terms were added to the final model (mastery approach by performance approach, and mastery avoidance by performance avoidance). Power to detect large effects on point estimates over time (i.e., .80 based on Cohen, 1992) was 80% with 50 participants per group for negative affect and 38 participants per group for positive affect. Those estimates were based on the amounts of within and between-person variability for each of the two constructs (positive and negative affect) from the current dataset, a level of significance equal to 5%, power levels equal to 80%, and a large effect size of 0.80 standard deviations (Cohen, 1992). The present studies’ sample size mostly covers the above estimates. Thus, ample power was present to evaluate the study hypotheses. The actual power curves are shown in Appendix B. All analyses were conducted using the OD software (Spybrook, Raudenbush, Liu, & Congdon, 2006). The means, standard deviations, possible and actual ranges, and correlations among variables of interest are reported in Table 3. Descriptives for positive and negative affect for each measurement occasion are presented in Table 4, together with the correlations among repeated measures. As can be seen, relatively stable and high levels of positive affect were reported across time points and searching sessions. Conversely, stable but low mean negative affect ratings were maintained both within and across sessions. Inspection of the raw trajectories in Figs. 1 and 2 reveals considerable individual variability on both positive and negative affect intercepts as well as slopes across searching sessions. There was also a slight tendency of an inverted U-shaped curve, indicating a small increase in negative affect during each session. Table 5 presents the estimated fixed effects, random effects and corresponding standard errors. The predicted mean value for positive affect was 25.4 (SE = .41, p < .001) at Time 1 during the search, remaining constant at each subsequent time point (b = .72, SE = .78, p = .35). Similarly, positive affect remained stable across successive searching sessions (b = .02, SE = .17, p = .88). The quadratic and cubic terms for Session did not reduce variance and were therefore dropped from the model on the basis of the deviance difference test. There were also no significant differences between experimental and control group on affect level or rate of change. The effect of mastery approach goals as level-2 predictor was significant (b = 1.09, SE = .15, p < .001), indicating that the more mastery-approach oriented students were, the higher levels of positive affect they reported. There were no significant main effects of mastery avoidance, performance approach and performance avoidance goals on positive affect. Finally, there were no significant interactions between Mastery Approach X Performance Approach goals and Mastery Avoidance X Performance Avoidance goals for positive affect. Mastery approach goals explained 41% of the variance between students, an effect size of considerable magnitude.

163

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170 Table 2 Multilevel models for web searching skills. Query formulation B

* ** ***

Fixed effects Session Condition (control as reference) Session X Condition Web searching experience (no searching at home as reference) Vocabulary subtest Similarities subtest Coding subtest Block design subtest Random effects Student level random effect variance Classroom level random effect variance

Information selection

SE

OR

0.71*** 0.02 0.98*** 0.03 0.12** 0.01 0.02 0.02

0.12 0.43 0.18 0.16 0.04 0.05 0.02 0.02

0.49 0.97 2.68 0.97 1.13 0.98 1.02 0.98

0.02 0.31

0.09 0.19

B

Credibility evaluation

SE

OR

0.25 1.30* 0.58* 0.38 0.03 0.01 0.06 0.04

0.17 0.46 0.26 0.32 0.07 0.07 0.05 0.06

0.77 3.68 1.79 1.46 0.96 0.99 1.06 1.04

0.59 0.11

0.37 0.17

B

SE

OR

0.07 1.40* 0.68** 1.04** 0.11 0.03 0.06 0.03

0.18 0.65 0.26 0.40 0.09 0.08 0.06 0.08

1.07 4.07 1.97 2.84 1.11 1.03 1.06 0.96

1.09* 0.73

0.47 0.46

p < .05. p < .01. p < .001.

Table 3 Descriptive statistics and bivariate correlations of affect and achievement goals. Variables

1. 2. 3. 4. 5. 6. ⁄

M

Positive affect Negative affect Mastery approach Mastery avoidance Performance approach Performance avoidance

SD

25.7 7.5 21.7 5.5 17. 2 13.3

Range

5.0 2.9 2.1 2.2 4.7 3.2

1

Potential

Actual

6–30 6–30 6–24 3–12 6–24 5–20

6–30 6–26 15–24 3–12 7–24 5–20

2

3

4

5

6

1 .22** .49**

1 .38**

1

1 .45** .46** .09** .12** .01

1 .12** .16** .03 .12**

1 .04 .31** .13**

p < .05. ** p < .01.

Table 4 Means, standard deviations, and correlations among measurement occasions for positive affect (above the diagonal) and negative affect (below the diagonal). V

SD

1

2

Session 1 1. Time 1 2. Time 2 3. Time 3

3

4

7.9 8.3 7.7

3.0 3.3 3.4

– .33 .43

– .58

.71 .75 –

.65 .44 .37

Session 2 4. Time 1 5. Time 2 6. Time 3

6.6 8.3 7.2

1.6 3.5 2.7

.31 .32 .01

.12 .01 .14

.22 .21 .28

– .13 .43

Session 3 7. Time 1 8. Time 2 9. Time 3

6.7 8.6 7.6

1.9 3.3 3.3

.29 .14 .03

.26 .14 .09

.19 .11 .11

Session 4 10. Time 1 11. Time 2 12. Time 3

7.3 6.7 6.9

2.9 1.5 2.5

.19 .12 .10

.12 .39 .14

.07 .16 .01

.67

5

6

7

8

9

10

11

12

M

SD

.29 .35 .10

.40 .45 .27

.45 .38 .37

.38 .55 .26

.39 .42 .23

.41 .40 .27

.47 .31 .34

.35 .22 .26

25.2 25.2 26.4

4.5 5.5 4.8

.46 .29

.57 .59 –

.67 .50 .63

.48 .65 .65

.61 .63 .71

.62 .54 .58

.53 .67 .69

.59 .33 .55

25.7 24.7 26.9

3.8 6.2 4.5

.54 .12 .32

.53 .56 .34

.43 .31 .43

– .55 .51

.49 – .63

.66 .74 –

.72 .40 .59

.58 .57 .71

.59 .29 .50

25.6 24.5 26.1

4.5 5.6 5.2

.40 .22 .41

.15 .11 .11

.23 .39 .19

.40 .40 .21

.34 .34 .19

.47 .24 .45

– .29 .73

.62 – .35

.58 .60 –

24.6 26.0 26.7

5.5 5.6 4.7

–

Note. For positive affect Min–Max: 6–30; for negative affect Min–Max: 6–26.

The predicted mean value for negative affect was 7.67 (SE = .25, p < .001) at Time 1, increasing by 1.45 points (SE = .40, p < .001) during each web searching session. Since the quadratic term was significant (b = .69, SE = .19, p < .001), the rate of increase was dampened at the end of each session. Also, negative affect declined by 0.26 points on each successive session (SE = .11, p = .02). The quadratic and cubic terms for session did not reduce variance and were therefore dropped from the model on the basis of the deviance difference test. The effect of mastery approach goals as

a level-2 predictor was significant (b = .25, SE = .09, p = .004). Thus, the more mastery-approach oriented students were, the lower levels of negative affect they reported. There were no significant main effects of mastery avoidance, performance approach and performance avoidance goals on negative affect. The interaction between mastery approach and performance approach goal orientations was significant (b = .03, SE = .01, p = .01), suggesting that the effect of mastery approach goals on negative affect decreased by 0.03 point for each scale point increase in

164

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170

of positive affect. Similarly, students with a low level of negative affect at the first session reported a stronger decline of negative affective experiences at subsequent sessions. The within-student variance associated with measurement occasions (i.e., sessions), was further reduced by 14% in the positive affect model and 11% in the negative affect model after including the random effect parameter. We did not find evidence that the rate of change of affect levels varied with student-reported achievement goals. 5. Discussion 5.1. Web searching skills

Fig. 1. Raw trajectories of positive affect at each of the three time points and across the four searching sessions. The bold lines indicate average trajectories of growth within each session.

performance approach goals. Thus, the effect of mastery approach goals on negative affect (b = .25), which was predicted for students with average performance approach goals, was found to be much smaller for students scoring higher than average on the scale of performance approach goals. Finally, there was no significant Mastery Avoidance X Performance Avoidance interaction for negative affect. Session explained approximately 3% of the variance within students, whereas the linear and quadratic terms of time explained another 2%, indicating a small change of negative affect both across and during web searching sessions. On the other hand, mastery approach goals explained 5% of the variance between students, while the interaction between mastery approach and performance approach goals explained another 4% of the variance, effect sizes of small magnitude. Furthermore, we found significant variability across students in the rates of change in reported positive and negative affect levels over sessions, as both the intercept and slope variance estimates were significant and the deviance chi-square test suggested an improved model. This indicates that students not only started at different affective states, but they also had different rates of change. The negative covariance found between intercept and slope indicates that students who started their search task with low positive affect demonstrated a faster decline of positive emotional reactions, than students who initially reported higher levels

The first research question concerned the process of change over time across experimental conditions. Multilevel analysis showed that the patterns of change in web searching skills differed across conditions, with the experimental group showing significant growth throughout the course of intervention on all searching skills measured. Given that students in the control group did not improve on their ability to formulate better keywords, neither selected more relevant information or applied more evaluation criteria, it appears that practice alone did not lead to better searching in the current study. This finding is in agreement with Gerjets and Hellenthal-Schorr’s (2008) report, who measured sixth grade students’ performance on two occasions, and showed that search performance significantly worsened during unguided web exploration. Our results are in contrast with earlier studies in the field of reading, demonstrating mastery of basic reading skills through mere print exposure by not-at-risk students (e.g., Mol & Bus, 2011). In a similar vein, correlational studies on web searching behavior have reported that experience alone can improve students’ ability to search effectively (Bilal, 2000; OECD, 2011; Tu et al., 2008). Our failure to find short-term practice-related gains in the control condition was surprising. One key difference between the searching context of our study and the informal settings of other studies was the available time for unstructured experimentation, which was limited to four 20-min sessions over a one-month period. Importantly, our findings suggest that instructional activities and support provided in this limited time framework had a significant effect on key web searching skills taught during intervention, namely query formulation, information selection and credibility Table 5 Predicting positive and negative affect: results from best fitting multilevel models.

Fixed effects Intercept Session Time Time quadratic Mastery approach Mastery avoidance Performance approach Performance avoidance Mastery approach X Performance approach Mastery avoidance X Performance avoidance Random effects Intercept Slope Covariance Residual Fig. 2. Raw trajectories of negative affect at each of the three time points and across the four searching sessions. The bold lines indicate average trajectories of growth within each session.

* ** ***

p < .05. p < .01. p < .001.

Positive affect

Negative affect

B

SE

B

25.41*** 0.02 0.72 0.63 1.09*** 0.07 0.01 0.09 0.05

0.41 0.17 0.78 0.37 0.15 0.15 0.07 0.09 0.02

7.67*** 0.26* 1.45*** 0.69*** 0.25** 0.14 0.02 0.07 0.03*

0.25 0.11 0.40 0.19 0.09 0.08 0.04 0.04 0.01

0.07

0.03

0.02

0.02

8.49*** 1.50*** 1.81* 10.91***

1.86 0.40 0.72 0.61

2.99*** 0.57*** 0.65* 5.20***

0.72 0.17 0.27 0.28

SE

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170

evaluation. Similar results have been reported in fifth and sixth graders after receiving instructional support by teachers (Gerjets & Hellenthal-Schorr, 2008; Kuiper et al., 2008) or a software tool (Zhang & Quintana, 2012). While it is clear that interventions work better than no intervention at all, it is largely uncertain whether training programs worked for the reasons specified by the theory and instructional design. Nevertheless, it is possible that students benefitted from the structure of training sessions. The above studies, including our own, organized sessions along constituent subskills following a recurrent scheme. Thus, in each session most instructors emphasized a set of target subskills by initially modeling worked-out examples, then offering opportunities for individual or paired practice and finally discussing and giving feedback concerning successful or problematic aspects of students’ performance. Secondly, it is possible that improvement was due to the instructional principles that interventions adapted to meet their needs. In most cases researchers applied a set of widely accepted instructional principles, such as modeling individual steps, providing meaningful inquiry context, activating prior knowledge and giving prompts and feedback through the searching process. In conjunction with the present study, we may speculate that the above instructional practices have been effective in promoting better searching skills in a short amount of time. The second question addressed here concerned the potential effect of individual characteristics, such as gender, age, experience and cognitive abilities, on the development of web searching skills. Firstly, we did not find any gender differences in the level of performance or the rate of change in web searching skills, suggesting that boys and girls had identical growth patterns. Similarly, grade was not a significant predictor of performance: students in the sixth grade did not start searching more advanced than their younger peers in the fifth grade and did not exhibit different growth rates during subsequent searches. Students’ trajectories were examined while controlling for the background variables of searching experience and cognitive abilities. Interestingly, experience in web searching at home for schoolwork had a positive effect on information evaluation, indicating that students searching at home for school assignments used more evaluation criteria to judge the quality of information at baseline. However, these students did not seem to improve the target skills at a faster rate. Concerning gender, our results disagree with previous studies reporting gender effects on navigating webpages, formulating queries and selecting hyperlinks (Large et al., 2002; Roy et al., 2003; Schacter et al., 1998). However, other studies show limited gender differences (OECD, 2011) and still others no differences at all (Kingsley, 2011) in online reading practices. This inconsistency may be due to the fact that Internet access and usage have changed over time. In contrast to earlier research where many online activities were found to differentiate between boys and girls, nowadays children of both genders have equal online access opportunities and take up nearly similar online activities (Livingstone, Haddon, Görzig, & Ólafsson, 2011). Hence, research findings obtained from young students ten years ago most likely cannot be transferred to young students today (Gossen & Nürnberger, 2013). The extent, however, to which any gender differences are disappearing permanently remains to be determined. Concerning age, we did not find differences between fifth and sixth grade students, supporting earlier research that reported similar search performance of students within a narrow age range (Gerjets & Hellenthal-Schorr, 2008). However, as students move through the grades, they accumulate more experience with web searching, acquire better general knowledge and improve their reading ability. In that respect, one might expect that older students would perform search tasks faster, navigate more efficiently and have better performance (Bilal, 2000; Lazonder et al., 2000; OECD, 2011). Accordingly, detecting variation in students’ skills

165

and performance would likely need a sample of a wider range of ages. Lastly, as far as experience is concerned, it is interesting to note that students who searched at home for school-related information applied more evaluation criteria. Similar findings were noted in other studies (Brand-Gruwel, Wopereis, & Vermetten, 2005; Flanagin & Metzger, 2010), suggesting that students’ experience with different kinds of information online -occasionally false or misleading- foster their ability to consider information trustworthiness. Another possible explanation for this result comes from the heuristics research. According to the cognitive heuristics framework, rather than evaluating information in a systematic manner, information seekers try to minimize their cognitive effort and maximize their information gain through the use of quick strategies (Metzger, Flanagin, & Medders, 2010). In other words, users employ cognitive heuristics to save effort and time. Some strategies that adult users have mentioned in Metzger et al.’ study (2010) are including consistency (e.g. checking different websites), reputation (e.g. relying on the respectability of website), and persuasive intent (e.g. identifying commercial or advertising purpose). It may be the case, then, that students in our research had developed evaluation heuristics during their searching at home, which allowed them to evaluate information credibility spontaneously (Hilligoss & Rieh, 2008). Two main insights emerge from the above findings. On the one hand, it can be suggested that the present short-term training program had beneficial effects on the development of students’ searching skills. Thus, this study highlights important instructional principles and elements that educators might apply and adapt to facilitate students’ inquiry processes. On the other hand, this study has gone some way toward enhancing our understanding about the cognitive skills involved in web searching, their development during a short-time period and some influencing factors affecting their change. 5.2. Affective trajectories The third question addressed in this research concerned students’ reported level of positive and negative affect and its evolution during web searching. Results indicated that, generally, students experienced high levels of positive affect and low levels of negative affect in both experimental and control conditions. During each searching session, positive affect remained constant, while negative affect rose at a decelerating rate. Across successive search sessions, positive affect remained relatively stable, while negative affect decreased slightly. In the only other previous study that we could find assessing changes in children’s affect while searching online, Kuhlthau et al. (2008) reported that negative feelings dominated the initial and middle stages of students’ inquiry projects. The present findings, however, are in agreement with those of Bilal (2000) and Bilal and Kirby (2002) who found that most students enjoyed searching the Web. A possible explanation for these results may be found in the task and search environment. Students looked for information in the context of tasks that were relevant to their everyday experiences and were presented in a stimulating context. Moreover, students’ attention and interest were reinforced through discussions and relevant instructional materials, such as videos, photographs, drawings, and slide sequences. Additionally, given that the Internet is part of the daily lives of the majority of students, they could readily associate web searching in school to real world experiences. The current findings add to a growing body of literature examining students’ affect with respect to particular academic activities and subjects. Considering that different school activities or courses produce different emotional responses (Goetz, Frenzel, Pekrun, & Hall, 2006; Lipnevich, MacCann, Bertling, Naemi, & Roberts, 2012), we found a general tendency for positive reported

166

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170

affect during web searching in school. Generally, positive emotions may facilitate task-related effort, motivation to learn, and ability to regulate cognitive activity (Buff, Reusser, Rakoczy, & Pauli, 2011; Pekrun, 2005). Similarly, positive affect seems to benefit problem solving by facilitating flexible and creative thinking (Isen, 2004). For that reason, our findings highlight Internet searching as an attractive learning environment and would predict beneficial effects of positive emotional states on performance outcomes during web searching. This view is also supported by Pekrun (2005, p. 505), who stresses the need to examine ways in which learning enjoyment is fostered and negative emotions are used productively in the classroom. In that respect, web searching on authentic and stimulating tasks should be encouraged by teachers. Surprisingly, we did not find differences between the experimental and control groups on students’ affective trajectories. This finding was unexpected given that the former group received considerable feedback, teacher support, and opportunities to develop competencies which would have fostered positive emotions (Pekrun, Goetz, Titz, & Perry, 2002). On the other hand, both groups of students were searching within a learning environment offering them a variety of resources and choices. Internet, despite its complexities, provides a flexible environment with unlimited choices (Dalton & Proctor, 2008). Thus, during searching for information students in both conditions were actively and continually engaged in selecting links, were interacting with content and were choosing among different types of information. In that respect, the intrinsic value of the activity, the ability to make choices and the self-regulation of the searching processes may have fulfilled students’ need for autonomy (Shen, Liu, & Wang, 2013), and thus positively affected their emotions (Patrick, Skinner, & Connell, 1993). 5.3. Achievement goals and affect The final question addressed in the current study concerned the influence of achievement goals on positive and negative affect. The results of this study indicate that mastery approach goals may have a positive impact on affect. Thus, the more students tried to develop their skills, the more positive and less negative affect they experienced during information searching. By contrast, the study failed to document a significant impact of performance approach on affect. The current results do not lend direct support to the multiple goal perspective by failing to confirm the additive goal model. Thus, results demonstrated independent, positive effects on affective states only for mastery approach goals, without detecting a similar effect for performance approach goals. Although the multiple goals perspective asserts that mastery and performance approach goals can both promote beneficial outcomes, no effect was found for performance approach goals in the current study. The findings related to mastery goals appear consistent with frequent earlier reports linking mastery approach goals to beneficial affective outcomes, including higher positive affect (Harackiewicz et al., 2000; Hulleman et al., 2010; Linnenbrink, 2005). A possible reason for this might be that students with a mastery-approach orientation tend to focus their attention on the value of activity and their own effort to control the outcome (Pekrun, 2009), and thus are more likely to experience positive rather than negative affect. With regard to performance goals, however, findings linking performance goals and affect are not consistent across studies. According to the multiple goals perspective, one would expect positive effects of performance goals on affective experience (Barron & Harackiewicz, 2003). Yet, in earlier studies performance-approach goals have been found to be positively related, negatively related, or even unrelated to negative affect (Bong, 2009; Kaplan & Maehr, 1999; Linnenbrink, 2005). The incongruent findings can be partly explained by the different dimensions of the goal constructs described in different

measurement scales. In a recent meta-analysis, Hulleman et al. (2010) concluded that goal measures often include similarly named constructs (i.e., performance-approach goal) to assess different components of them (i.e., competence demonstration versus outperforming others). This means that the relationship between performance goals and affect may be proved to be beneficial when a normative component of performance approach goals is measured (i.e., doing better than others) and disadvantageous when a self-presentation component is involved (i.e., showing others that someone is good at something; Hulleman et al., 2010). Although in the present study we used Elliot and colleagues’ Achievement Goals Questionnaire emphasizing the normative component, we did not uncover any potential benefit in pursuing performance approach goals. However, we did not uncover any costs either. A possible explanation for the null results is related to the methods we used in the current study to assess performance outcomes and provide feedback to students. In both experimental and control conditions students after each session were assessed for the quality of their written work and were given an overall grade. Based on that score, prizes were eventually awarded to all students. Nevertheless, these grades were not included in their report cards, and did not influence the grading of any other school subject. So, it may be that in this particular context performance goals themselves were not a strong enough influence to produce a measurable affective response. The results of the present study are not directly consistent with the interactive goal model, either, according to which individuals who adopt both goals have the most adaptive patterns of affect and achievement (Barron & Harackiewicz, 2001). Specifically, there was no evidence to support that endorsing both mastery and performance orientations could be beneficial for affective-related outcomes. On the contrary, students who endorsed mastery approach goals and simultaneously were more performance-approach oriented experienced more negative affect. In other words, the interaction between mastery and performance approach goals was found to be negative. This finding is in contrast to previous research that has demonstrated that the simultaneous pursuit of mastery and performance goals encourages adaptive behaviors, enhances enjoyment in learning (Dela Rosa & Bernardo, 2013), increases the level of effort being exerted during learning activities (Sideridis, 2006), and may have positive effects on achievement (Pintrich, 2000b). Yet, other researchers testing the multiple goal perspective have failed to find any significant mastery and performance approach goal interaction for a variety of outcomes, including positive and negative affect (e.g., Linnenbrink, 2005; for a review, see Pintrich, Conley, & Kempler, 2003). Thus, we may argue that prior research has produced inconclusive results on the interactive effects between the two goals (Hulleman & Senko, 2010), possibly due to sample characteristics. Specifically, it has been suggested that the interactive effects may apply to undergraduate student samples, employed in the majority of studies, and not to less competitive educational settings, such as elementary schools (Hulleman et al., 2010). It is possible, therefore, that students who adopt simultaneously high levels of mastery and performance goals may benefit in certain developmental stages and contexts and not others (Pintrich et al., 2003). To conclude, one may argue that the simultaneous adoption of performance approach and mastery approach goals by elementary school students may be not be adaptive in the context of web searching. Finally, the results of this study did not reveal any effects of mastery-avoidance or performance-avoidance goals on students’ affect. Both orientations emphasize negative outcomes (i.e., avoiding misunderstanding/failure to learn or avoiding being the worst in class) and thus are expected to cause negative affect. While performance-avoidance goals have been previously related to negative emotions (Elliot & McGregor, 2001; Sideridis, 2003, 2005), we have been unable to demonstrate a similar effect in our study. On the

167

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170

other hand, existing findings on mastery avoidance goals are based on limited studies and are, therefore, suggestive rather than definitive (Huang, 2011). Nevertheless, further work is required on this issue, before we fully understand the relationship between affect and avoidance achievement motivation. To summarize, the present study provides additional evidence on the influence of mastery approach goals on affective states. One implication of our findings concerns instructional practices and classroom environment. It seems, therefore, that during web searching students should be encouraged to develop their searching skills and also focus on personal improvement and understanding information resources. Even if students do not eventually succeed in finding relevant and credible information, it is important to focus on their steady progress. This study has also found that students who strive to develop their competencies and simultaneously aim to demonstrate their ability are more likely to experience negative affect during searching the Web for information. This finding may imply that in the elementary school context competition and trying to perform better than others can be maladaptive in the presence of mastery goals. Nevertheless, given that the effect size of the interaction was small, further work is required to establish this result.

contexts. Thus, before we can generalize results to normal practice, it is first necessary to evaluate the effects of the intervention in more conventional academic settings. Lastly but very importantly, an issue that was not addressed in this study was the transfer of skills. Although the longitudinal design of this study does allow us to assess intervention’s short-term effects on successive searching sessions (Raudenbush, 2007), still the longer-term effects of the intervention were not examined. Acknowledging this limitation, future research would do best by examining transfer of skills across a variety of tasks, subjects and contexts. Appendix A Scoring criteria for the information evaluation task. Score

Operational definition criteria

Example of student response

0

Does not apply evaluation criteria or gives no justification for selecting information and sources Refers to some evaluation criteria without further explanation or gives incomplete justification for selecting information and sources Shows sufficient evidence of the application of one evaluation criterion Applies two or more evaluation criteria with reasoned justification for selecting information or sources

I can trust the information

1

6. Limitations and future directions The design of this study bears three notable limitations. First, the current research has only examined reading and evaluation skills using the written products of students’ assignments. Yet it has been argued that indirect measures of evaluation and navigation behavior miss the opportunity to capture the details of evolving cognitive processes (Wopereis & van Merriënboer, 2011). In that case, the eye-tracking methodology, combined with cued retrospective reporting, might have been more appropriate to study developmental processes of web searching skills (BrandGruwel & Stadtler, 2011). Second, affective states were assessed along two general dimensions, positive and negative, using the PANAS-C. The scale, however, implies a high activation component of affect, including only high-arousal emotion items (Tellegen, Watson, & Clark, 1999). An important component of affective experience, pertaining to low-activation affective states, was not properly assessed. Similarly, by focusing on global affective dimensions instead of discrete emotions, the study ignored qualitative differences between specific emotional states (Pekrun, 2006; Shockley et al., 2012). A third limitation of the study is related to the measurement of achievement goals, given that the subscale of mastery-avoidance goals included only negatively-biased items (e.g., I am afraid that..). According to Hulleman et al. (2010), mastery-avoidance scales should not depend so heavily on negative affectivity, but instead items should reflect more the theoretical construct of masteryavoidance goals. Correspondingly, Elliot and Murayama (2008) suggest that affective references should be either spread evenly across achievement goal subscales, or omitted altogether. In fact, Elliot has updated the mastery-avoidance subscale removing the affective content from the goal items. In this respect, the revised scale may be preferable for future studies. Lastly, although the present results may generate recommendations for educational practice, one must keep in mind that our study was conducted in a small group setting instead of a traditional whole classroom environment, and the instruction was delivered by a researcher instead of the classroom teachers. For this reason, present findings may not translate readily to other

2

3

The website is reliable

I have read similar information in a book on recycling I checked the information in a different website and I have also seen the described procedure in a recycling factory

Appendix B Power analysis for the multilevel model using the OD software. The parameters of variability were drawn from the present data.

Negative Affect

168

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170

Positive Affect

For negative affect 50 participants per group would suffice to model a large effect with 3 time points and the ratio of level-1 and level-2 variances observed with the current data. The respective estimate for positive affect equaled 38 participants per group. References Agresti, A., Booth, J. G., Hobert, J. P., & Caffo, B. (2000). Random-effects modeling of categorical response data. Sociological Methodology, 30, 27–80. Artino, A. R., & Jones, K. D. (2012). Exploring the complex relations between achievement emotions and self-regulated learning behaviors in online learning. The Internet and Higher Education, 15, 170–175. Barron, K. E., & Harackiewicz, J. M. (2001). Achievement goals and optimal motivation: Testing multiple goal models. Journal of Personality and Social Psychology, 80, 706–722. Barron, K. E., & Harackiewicz, J. M. (2003). Revisiting the benefits of performanceapproach goals in the college classroom: Exploring the role of goals in advanced college courses. International Journal of Educational Research, 39, 357–374. Bilal, D. (2000). Children’s use of the Yahooligans! Web search engine: I. Cognitive, physical, and affective behaviors on fact-based search task. Journal of the American Society for Information Science and Technology, 51, 646–665. Bilal, D., & Bachir, I. (2007). Children’s interaction with cross-cultural and multilingual digital libraries. II. Information seeking, success, and affective experience. Information Processing and Management, 43, 65–80. Bilal, D., & Kirby, J. (2002). Differences and similarities in information seeking: Children and adults as Web users. Information Processing and Management, 38, 649–670. Blanchette, I., & Richards, A. (2010). The influence of affect on higher level cognition: A review of research on interpretation, judgment, decision making and reasoning. In J. De Houwer & D. Hermans (Eds.), Cognition & Emotion (pp. 276–324). New York: Psychology Press. Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., et al. (2009). Generalized linear mixed models: A practical guide for ecology and evolution. Trends in Ecology & Evolution, 24, 127–135. Bong, M. (2001). Between-and within-domain relations of academic motivation among middle and high school students: Self-efficacy, task value, and achievement goals. Journal of Educational Psychology, 93, 23–34. Bong, M. (2009). Age-related differences in achievement goal differentiation. Journal of Educational Psychology, 101, 879–896. Brand-Gruwel, S., & Stadtler, M. (2011). Solving information-based problems: Evaluating sources and information. Learning and Instruction, 21, 175–179. Brand-Gruwel, S., Wopereis, I., & Vermetten, Y. (2005). Information problem solving by experts and novices: Analysis of a complex cognitive skill. Computers in Human Behavior, 21, 487–508. Brand-Gruwel, S., Wopereis, I., & Walraven, A. (2009). A descriptive model of information problem solving while using Internet. Computers & Education, 53, 1207–1217. Brophy, J. (2010). Motivating students to learn (3rd ed.). New York: Routledge. Buff, A., Reusser, K., Rakoczy, K., & Pauli, C. (2011). Activating positive affective experiences in the classroom: ‘‘Nice to have’’ or something more? Learning and Instruction, 21, 452–466. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159. Coiro, J., & Dobler, E. (2007). Exploring the online reading comprehension strategies used by sixth-grade skilled readers to search for and locate information on the Internet. Reading Research Quarterly, 42, 214–257. Dalton, B., & Proctor, C. P. (2008). The changing landscape of text and comprehension in the age of new literacies. In J. Coiro, M. Knobel, C. Lankshear, & D. Leu (Eds.), Handbook of new literacies (pp. 297–324). Mahwah, NJ: Erlbaum.

De Vries, B., van der Meij, H., & Lazonder, A. W. (2008). Supporting reflective web searching in elementary schools. Computers in Human Behavior, 24, 649–665. Dela Rosa, E. D., & Bernardo, A. B. (2013). Are two achievement goals better than one? Filipino students’ achievement goals, deep learning strategies and affect. Learning and Individual Differences, 27, 97–101. D’Mello, S. (2013). A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. Journal of Educational Psychology, 105, 1082–1099. Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95, 256–273. Elliot, A. J., & Church, M. A. (1997). A hierarchical model of approach and avoidance achievement motivation. Journal of Personality and Social Psychology, 72, 218–232. Elliot, A. J. (2008). Approach and avoidance motivation. In A. J. Elliot (Ed.), Handbook of approach and avoidance motivation (pp. 3–14). New York: Psychology Press. Elliot, A. J. (2005). A conceptual history of the achievement goal construct. In A. J. Elliot & C. S. Dweck (Eds.), Handbook of competence and motivation (pp. 52–72). New York: Guilford. Elliot, A. J., & McGregor, H. A. (1999). Test anxiety and the hierarchical model of approach and avoidance achievement motivation. Journal of Personality & Social Psychology, 76, 628–644. Elliot, A. J., & McGregor, H. A. (2001). A 2  2 achievement goal framework. Journal of Personality and Social Psychology, 80, 501–519. Elliot, A. J., & Murayama, K. (2008). On the measurement of achievement goals: Critique, illustration, and application. Journal of Educational Psychology, 100, 613–628. Fitzmaurice, G., Laird, N., & Ware, J. (2004). Applied longitudinal analysis. New Jersey: John Wiley & Sons. Flanagin, A. J., & Metzger, M. J. (2010). Kids and credibility: An empirical examination of youth, digital media use, and information credibility. Cambridge, MA: MIT Press. Flavian-Blanco, C., Gurrea-Sarasa, R., & Orus-Sanclemente, C. (2011). Analyzing the emotional outcomes of the online search behavior with search engines. Computers in Human Behavior, 27, 540–551. Friedman, L. M., Furberg, C., & DeMets, D. L. (2010). Fundamentals of clinical trials (3rd ed.). New York: Springer. Georgas, D. D., Paraskevopoulos, I. N., Bezevegis, I. G., & Giannitsas, N. D. (1997). Greek WISC-III: Wechsler intelligence scale for children. Athens: Ellinika Grammata. Gerjets, P., & Hellenthal-Schorr, T. (2008). Competent information search in the World Wide Web: Development and evaluation of a web training for pupils. Computers in Human Behavior, 24, 693–715. Goetz, T., Frenzel, A. C., Pekrun, R., & Hall, N. C. (2006). The domain specificity of academic emotional experiences. Journal of Experimental Education, 75, 5–29. Goetz, T., Frenzel, A. C., Pekrun, R., Hall, N. C., & Lüdtke, O. (2007). Between-and within-domain relations of students’ academic emotions. Journal of Educational Psychology, 99, 715–733. Gossen, T., & Nürnberger, A. (2013). Specifics of information retrieval for young users: A survey. Information Processing & Management, 49, 739–756. Haddon, L., & Livingstone, S. (2012). EU Kids Online: National perspectives. EU Kids Online. London, UK: The London School of Economics and Political Science. Retrieved from . Harackiewicz, J. M., Barron, K., Pintrich, P., Elliot, A., & Trash, T. (2002). Revision of achievement goal theory: Necessary and illuminating. Journal of Educational Psychology, 94, 638–645. Harackiewicz, J. M., Barron, K. E., Tauer, J. M., Carter, S. M., & Elliot, A. J. (2000). Short-term and long-term consequences of achievement goals: Predicting interest and performance over time. Journal of Educational Psychology, 92, 316–330. Hedeker, D., Gibbons, R. D., Du Toit, M., & Cheng, Y. (2009). SuperMix 1.1. Chicago: Scientific Software International. Hedeker, D., & Gibbons, R. D. (2006). Longitudinal data analysis. New Jersey: John Wiley & Sons. Hedeker, D., & Mermelstein, R. J. (1998). A multilevel thresholds of change model for analysis of stages of change data. Multivariate Behavioral Research, 33, 427–455. Hilligoss, B., & Rieh, S. Y. (2008). Developing a unifying framework of credibility assessment: Construct, heuristics, and interaction in context. Information Processing & Management, 44, 1467–1484. Hirsh, S. G. (1999). Children’s relevance criteria and information seeking on electronic resources. Journal of the American Society for information Science, 50, 1265–1283. Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York: Routledge. Huang, C. (2011). Achievement goals and achievement emotions: A meta-analysis. Educational Psychology Review, 23, 359–388. Hulleman, C. S., Schrager, S. M., Bodmann, S. M., & Harackiewicz, J. M. (2010). A meta-analytic review of achievement goal measures: Different labels for the same constructs or different constructs with similar labels? Psychological Bulletin, 136, 422–449. Hulleman, C. S., & Senko, C. (2010). Up around the bend: Forecasts for achievement goal theory and research in 2020. Advances in Motivation and Achievement, 16, 71–104. Isen, A. M. (2008). Some ways in which positive affect influences decision making and problem solving. In M. Lewis, J. M. Haviland-Jones, & L. F. Barrett (Eds.), Handbook of emotions (3rd ed., pp. 548–573). New York: The Guildford Press.

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170 Isen, A. M. (2004). Some perspectives on positive feelings and emotions: Positive affect facilitates thinking and problem solving. In A. S. R. Manstead, N. Frijda, & A. Fischer (Eds.), Feelings and emotions: The Amsterdam Symposium (pp. 263–281). New York: Cambridge University Press. Kafai, Y., & Bates, M. J. (1997). Internet web-searching instruction in the elementary classroom: Building a foundation for information literacy. School Library Media Quarterly, 25, 103–111. Kaplan, A., & Maehr, M. L. (1999). Achievement goals and student well-being. Contemporary Educational Psychology, 24, 330–358. Kingsley, T. L. (2011). Integrating new literacy instruction to support online reading comprehension: An examination of online literacy performance in 5th grade classrooms. (Doctoral Dissertation). Available from ProQuest Dissertations & Theses. (UMI No. 3466808). Kraemer, H. C., Yesavage, J. A., Taylor, J. L., & Kupfer, D. (2000). How can we learn about developmental processes from cross-sectional studies, or can we? The American Journal of Psychiatry, 157, 163–171. Kuhlthau, C. C., Heinström, J., & Todd, R. J. (2008). The ‘information search process’ revisited: Is the model still useful? Information Research, 13, paper 355. Retrieved from . Kuiper, E., Volman, M., & Terwel, J. (2008). Integrating critical Web skills and content knowledge: Development and evaluation of a 5th grade educational program. Computers in Human Behavior, 24, 666–692. Kuiper, E., Volman, M., & Terwel, J. (2009). Developing Web literacy in collaborative inquiry activities. Computers & Education, 52, 668–680. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. Large, A., Beheshti, J., & Rahman, T. (2002). Gender differences in collaborative Web searching behavior: An elementary school study. Information Processing and Management, 38, 427–443. Large, A. (2005). Children, teenagers, and the Web. In B. Cronin (Ed.), Annual review of information science and technology (pp. 347–392). Medford, NJ: Information Today. Laurent, J., Catanzaro, S. J., Joiner, T. E., Rudolf, K. D., Potter, K. I., Lambert, S., et al. (1999). A measure of positive and negative affect for children: Scale development and preliminary validation. Psychological Assessment, 1, 326–338. Lazonder, A. W., Biemans, H. J., & Wopereis, I. G. (2000). Differences between novice and experienced users in searching information on the World Wide Web. Journal of the American Society for Information Science, 51, 576–581. Leu, D. J., Coiro, J., Castek, J., Hartman, D. K., Henry, L. A., & Reinking, D. (2008). Research on instruction and assessment in the new literacies of online reading comprehension. In C. C. Block, S. Parris, & P. Afflerbach (Eds.), Comprehension instruction: Research-based best practices (2nd ed., pp. 321–346). New York, NY: Guilford. Leu, D. J., Kinzer, C. K., Coiro, J. L., & Cammack, D. W. (2004). Toward a theory of new literacies emerging from the Internet and other information and communication technologies. In R. B. Ruddell & N. Unrau (Eds.), Theoretical models and processes of reading (pp. 1570–1613). Newark, DE: International Reading Association. Leu, D. J., McVerry, J. G., O’Byrne, W. I., Kiili, C., Zawilinski, L., Everett-Cacopardo, H., et al. (2011). The new literacies of online reading comprehension: Expanding the literacy and learning curriculum. Journal of Adolescent & Adult Literacy, 55, 5–14. Linnenbrink, E. A. (2005). The dilemma of performance-approach goals: The use of multiple goal contexts to promote students’ motivation and learning. Journal of Educational Psychology, 97, 197–213. Linnenbrink, E. A., & Pintrich, P. R. (2002). Achievement goal theory and affect: An asymmetrical bidirectional model. Educational Psychologist, 37, 69–78. Lipnevich, A. A., MacCann, C., Bertling, J. P., Naemi, B., & Roberts, R. D. (2012). Emotional reactions toward school situations: Relationships with academic outcomes. Journal of Psychoeducational Assessment, 30, 387–401. Livingstone, S. (2003). Children’s use of the Internet: Reflections on the emerging research agenda. New Media & Society, 5, 147–166. Livingstone, S., Bober, M., & Helsper, E. (2005). Internet literacy among children and young people: Findings from the UK Children Go Online project. London: LSE Research Online. Retrieved from . Livingstone, S., Haddon, L., Görzig, A., & Ólafsson, K. (2011). Risks and safety on the Internet: The perspective of European children. Full Findings. LSE, London: EU Kids Online. Lopatovska, I., & Arapakis, I. (2011). Theories, methods and current research on emotions in library and information science, information retrieval and human– computer interaction. Information Processing and Management, 47, 575–592. Maas, J. M., & Hox, J. J. (2004). Robustness issues in multilevel regression analysis. Statistica Neerlandica, 58, 127–137. MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130–149. Madjar, N., Kaplan, A., & Weinstock, M. (2011). Clarifying mastery-avoidance goals in high school: Distinguishing between intrapersonal and task-based standards of competence. Contemporary Educational Psychology, 36, 268–279. Metzger, M. J. (2007). Making sense of credibility on the Web: Models for evaluating online information and recommendations for future research. Journal of the American Society for Information Science and Technology, 58, 2078–2091. Metzger, M. J., Flanagin, A. J., & Medders, R. B. (2010). Social and heuristic approaches to credibility evaluation online. Journal of Communication, 60, 413–439.

169

Metzger, M. J., Pure, R., Flanagin, A. J., Markov, A., Medders, R., & Hartsell, E. (2013). The special case of youth and digital information credibility. In M. Folk & S. Apostel (Eds.), Online credibility and digital ethos: Evaluating computer-mediated communication: Evaluating computer-mediated communication (pp. 148–168). Hershey: Information Science Reference. Meyer, D. K., Turner, J. C., & Spencer, C. A. (1997). Challenge in a mathematics classroom: Students’ motivation and strategies in project-based learning. The Elementary School Journal, 97, 501–521. Mills, K. A. (2010). A review of the ‘‘digital turn’’ in the new literacy studies. Review of Educational Research, 80, 246–271. Mol, S. E., & Bus, A. G. (2011). To read or not to read: A meta-analysis of print exposure from infancy to early adulthood. Psychological Bulletin, 137, 267. Nahl, D. (2005). Affective and cognitive information behavior: Interaction effects in Internet use. In A. Grove (Ed.), Proceedings 68th annual meeting of the american society for information science and technology (ASIST), 42. Medford, NJ: Information Today. doi: 10.1002/meet.1450420196. OECD (2011). PISA 2009 Results: Students On Line. Retrieved from . Paris, S. G. (2005). Reinterpreting the development of reading skills. Reading Research Quarterly, 40, 184–202. Paris, S. G., & Hamilton, E. E. (2009). The development of children’s reading comprehension. In S. E. Israel & G. G. Duffy (Eds.), Handbook of research on reading comprehension (pp. 32–53). New York: Routledge. Patrick, B. C., Skinner, E. A., & Connell, J. P. (1993). What motivates children’s behavior and emotion? Joint effects of perceived control and autonomy in the academic domain. Journal of Personality and Social Psychology, 65, 781–791. Pekrun, R. (2005). Progress and open problems in educational emotion research. Learning and Instruction, 15, 497–506. Pekrun, R. (2006). The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice. Educational Psychology Review, 18, 315–341. Pekrun, R. (2011). Emotions as drivers of learning and cognitive development. In R. A. Calvo, & S. K. D’ Mello (Eds.), New perspectives on affect and learning technologies (pp. 23–39). New York: Springer. Pekrun, R., Elliot, A. J., & Maier, M. A. (2006). Achievement goals and discrete achievement emotions: A theoretical model and prospective test. Journal of Educational Psychology, 98, 583–597. Pekrun, R., Goetz, T., Frenzel, A. C., Barchfeld, P., & Perry, R. P. (2011). Measuring emotions in students’ learning and performance: The Achievement Emotions Questionnaire (AEQ). Contemporary Educational Psychology, 36, 36–48. Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Positive emotions in education. In E. Frydenberg (Ed.), Beyond coping: Meeting goals, visions, and challenges (pp. 149–174). Oxford, UK: Elsevier. Pekrun, R., & Linnenbrink-Garcia, L. (2012). Academic emotions and students engagement. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 259–282). New York: Springer. Pekrun, R. (2009). Emotions at school. In K. R. Wentzel & A. Wigfield (Eds.), Handbook of motivation at school (pp. 575–604). New York: Routledge. Pintrich, P. R. (2000a). An achievement goal theory perspective on issues in motivation terminology, theory, and research. Contemporary Educational Psychology, 25, 92–104. Pintrich, P. R. (2000b). Multiple goals, multiple pathways: The role of goal orientation in learning and achievement. Journal of Educational Psychology, 92, 544–555. Pintrich, P. R., Conley, A. M., & Kempler, T. M. (2003). Current issues in achievement goal theory and research. International Journal of Educational Research, 39, 319–337. Purcell, K., Heaps, A., Buchanan, J., & Friedrich, L. (2013). How teachers are using technology at home and in their classrooms. Washington, DC: Pew Research Center’s Internet and American Life Project. Retrieved from . Raudenbush, S. W. (2007). Designing field trials of educational innovations. In: B. Schneider, & S. -K. McDonald (Eds.), Scale-up in education: Ideas in principle (Vol. 2, pp. 23–40). Maryland, USA: Rowman and Littlefield Publishers. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage Publications. Rhoads, C. H. (2011). The implications of ‘‘contamination’’ for experimental design in education. Journal of Educational and Behavioral Statistics, 36, 76–104. Rouet, J. F. (2006). The skills of document use: From text comprehension to Web-based learning. New Jersey: Erlbaum Associates Inc.. Rouet, J. F., Ros, C., Goumi, A., Macedo-Rouet, M., & Dinet, J. (2011). The influence of surface and deep cues on primary and secondary school students’ assessment of relevance in Web menus. Learning and Instruction, 21, 205–219. Roy, M., & Chi, M. T. (2003). Gender differences in patterns of searching the web. Journal of Educational Computing Research, 29, 335–348. Roy, M., Taylor, R., & Chi, M. T. (2003). Searching for information on-line and offline: Gender differences among middle school students. Journal of Educational Computing Research, 29, 229–252. Schacter, J., Chung, G. K. W. K., & Dorr, A. (1998). Children’s Internet searching on complex problems: Performance and process analyses. Journal of the American Society for Information Science, 49, 840–849. Schwartz, R. M., & Trabasso, T. (1984). Children’s understanding of emotions. In C. E. Izard, J. Kagan, & R. B. Zajonc (Eds.), Emotions, cognition, and behavior (pp. 409–437). New York: Cambridge University Press.

170

D. Kroustallaki et al. / Computers in Human Behavior 49 (2015) 156–170

Senko, C., Hulleman, C. S., & Harackiewicz, J. M. (2011). Achievement goal theory at the crossroads: Old controversies, current challenges, and new directions. Educational Psychologist, 46, 26–47. Shen, C. X., Liu, R. D., & Wang, D. (2013). Why are children attracted to the Internet? The role of need satisfaction perceived online and perceived in daily real life. Computers in Human Behavior, 29, 185–192. Shen, L., Wang, M., & Shen, R. (2009). Affective e-Learning: Using’’ emotional’’ data to improve learning in pervasive learning environment. Educational Technology & Society, 12(2), 176–189. Shockley, K. M., Ispas, D., Rossi, M. E., & Levine, E. L. (2012). A meta-analytic investigation of the relationship between state affect, discrete emotions, and job performance. Human Performance, 25, 377–411. Sideridis, G. D. (2003). On the origins of helpless behavior of students with learning disabilities: Avoidance motivation? International Journal of Educational Research, 39, 497–517. Sideridis, G. D. (2005). Goal orientation, academic achievement, and depression: Evidence in favor of a revised goal theory framework. Journal of Educational Psychology, 97, 366–375. Sideridis, G. D. (2006). Achievement goal orientations, ‘‘oughts’’, and self-regulation in students with and without learning disabilities. Learning Disability Quarterly, 29, 3–18. Sideridis, G. D. (2008). The regulation of affect, anxiety, and stressful arousal from adopting mastery-avoidance goals orientations. Stress and Health, 24, 55–69. Spybrook, J., Raudenbush, S. W., Liu, X., & Congdon, R. (2006). Optimal design for longitudinal and multilevel research: Documentation for the ‘‘Optimal Design’’ software. Retrieved from . Tate, M. A. (2010). Web wisdom: How to evaluate and create information quality on the Web. Boca Raton, FL: CRC Press. Tellegen, A., Watson, D., & Clark, L. A. (1999). On the dimensional and hierarchical structure of affect. Psychological Science, 10, 297–303.

Tu, Y.-W., Shih, M., & Tsai, C.-C. (2008). Eighth graders’ web searching strategies and outcomes: The role of task types, web experiences and epistemological beliefs. Computers & Education, 51, 1142–1153. Valiente, C., Swanson, J., & Eisenberg, N. (2012). Linking students’ emotions and academic achievement: When and why emotions matter. Child Development Perspectives, 6, 129–135. Wallace, R. M., Kupperman, J., Krajcik, J., & Soloway, E. (2000). Science on the Web: Students online in a sixth-grade classroom. Journal of the Learning Sciences, 9, 75–104. Walraven, A., Brand-Gruwel, S., & Boshuizen, H. (2008). Information-problem solving: A review of problems students encounter and instructional solutions. Computers in Human Behavior, 24, 623–648. Wang, P., Hawk, W. B., & Tenopir, C. (2000). Users’ interaction with world wide web resources: An exploratory study using a holistic approach. Information Processing and Management, 36, 229–251. Watson, D., & Clark, L. A. (1994). The PANAS-X: Manual for the Positive and Negative Affect Schedule-expanded form. Unpublished manuscript, University of Iowa, Iowa City. Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The PANAS Scales. Journal of Personality and Social Psychology, 54, 1063–1070. Wechsler, D. (1991). Manual for the wechsler intelligence scale for children – Third Edition. San Antonio, TX: The Psychological Corporation. Wirthwein, L., Sparfeldt, J. R., Pinquart, M., Wegerer, J., & Steinmayr, R. (2013). Achievement goals and academic achievement: A closer look at moderating factors. Educational Research Review, 10, 66–89. Wopereis, I. G., & van Merriënboer, J. J. (2011). Evaluating text-based information on the World Wide Web. Learning and Instruction, 21, 232–237. Zhang, M., & Quintana, C. (2012). Scaffolding strategies for supporting middle school students’ online inquiry processes. Computers & Education, 58, 181–196.