Test-takers’ eye movements: Effects of integration aids and types of graphical representations

Computers & Education 109 (2017) 85e97

Contents lists available at ScienceDirect

Computers & Education journal homepage: www.elsevier.com/locate/compedu

Test-takers’ eye movements: Effects of integration aids and types of graphical representations Steffani Saß*, Kerstin Schütte, Marlit Annalena Lindner Leibniz Institute for Science and Mathematics Education (IPN), Olshausenstraße 62, 24118 Kiel, Germany

a r t i c l e i n f o

a b s t r a c t

Article history: Received 19 November 2016 Received in revised form 26 January 2017 Accepted 17 February 2017 Available online 21 February 2017

The study focuses on integration aids (i.e., signals) and their effect on how students process different types of graphical representations (representational pictures vs. organizational pictures vs. diagrams) in standardized multiple-choice items assessing science achievement. Based on text-picture integration theories each type of pictorial representation hold different cognitive requirements concerning integration processes of two representations. Further, depending on type of representation not every picture is needed to answer an item correctly. Students from fifth sixth grade (N ¼ 60) work through 12 multiple choice items while their eye movements were recorded. Results showed that students achieved higher test scores when items were presented in an integrated format than in a non-integrated format, however, this was only true for diagrams. Eye movement data revealed that students looked longer on the graphical representations in items presented in the integrated format condition compared to the non-integrated format condition. Furthermore, relations between looking at the diagrams and achievement in the integrated format emerged. © 2017 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Keywords: Text-picture integration Signals Graphical representations Eye-tracking

1. Introduction Even though graphical representations constitute an essential part of large-scale assessments (LSA), such as the Trends in International Mathematics and Science Study (TIMSS; e.g., Martin, Mullis, Foy, & Stanco, 2012) or the Programme for International Student Assessment (PISA; e.g., Organisation for Economic Co-operation and Development (OECD, 2007), so far little is known of how test-takers process graphical representations in test items. Recent studies could show that adding a representational picture (i.e., a picture that reiterates information provided by the text) to the stem of multiple-choice items in a test assessing science achievement decreases item difficulty compared to identical items without a picture (Lindner, €ller, 2016; Sab, Wittwer, Senkbeil, & Ko €ller, 2012). Furthermore, an eye-tracking study showed that adding Ihme, Sab, & Ko a representational picture to the stem of text-only items not only enhances students’ performance but also affects how the €ller, 2017). Because research has so far focused on representational pictures, item is processed (Lindner, Eitel, Strobel, & Ko however, it is unknown whether different types of pictures (i.e., representational pictures, organizational pictures and diagrams) differentially affect how students process items. In the absence of theory explicitly pertaining to cognitive processes during testing we draw on theories in the context of learning and investigate their applicability to testing. Specifically,

* Corresponding author. E-mail address: [email protected] (S. Saß). http://dx.doi.org/10.1016/j.compedu.2017.02.007 0360-1315/© 2017 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/ licenses/by-nc-nd/4.0/).

86

S. Saß et al. / Computers & Education 109 (2017) 85e97

integration aids (i.e., signals) are a potential means to foster integration of multiple external representations (Scheiter & Eitel, 2015) that might lead to a better comprehension of the task at hand and, thus, to higher achievement. Accordingly, we present an experimental eye-tracking study that investigates (a) whether the use of different types of graphical representations in multiple-choice test items (i.e., representational pictures, organizational pictures, and diagrams) affects secondary school students' item solving processes in a science test and (b) whether integration aids support processes of textepicture integration and thus affect students’ achievement. 1.1. Processing test items including text and picture Prominent cognitive theories like the cognitive theory of multimedia learning (CTML; Mayer, 2005) and the integrative model of text and picture comprehension (ITPC; Schnotz & Bannert, 2003) aim to build hypotheses how students process learning material including text and picture. Because explicit theories for item processing in testing situations are missing, we rely on those cognitive theories to explain how students process test items. Based on the dual coding paradigm proposed by Paivio (1986) these theories assume that processing verbal and graphical information takes place in two separate cognitive channels. The ITPC describes a multistage processing of multimedia representations (i.e., text and picture) resulting in a mental representation. In brief, the processing of textual information results in a propositional representation of the text's semantic content whereas the graphical information is used to create a mental model of its subject (for details, see Schnotz, 2002; Schnotz & Bannert, 2003). According to Schnotz (2002), the propositional representation and the mental model interact with each other through processes of model inspection and model construction which in turn leads to a coherent mental representation of the given information (e.g., textual and graphical information in an item stem). Depending on how much information each representation carries, different ways to the construction of a coherent mental representation are possible: Text and picture can both contribute to the mental representation (e.g., complementary information in diagrams), butdto the extent that the representations are information-equivalentdeither one can replace the other (e.g., representational or organizational pictures; cf. Ainsworth, 1999, 2006). Indeed, a number of studies €na €, 1999; have shown that students mostly focus on the text when processing science learning material (Hannus & Hyo Schmidt-Weigand, Kohnert, & Glowalla, 2010). Whether this effect is moderated by the representations' informational relation has not yet been investigated. Nevertheless, learning through text and picture promotes better comprehension than learning through text alone (multimedia effect; Anglin, Vaez, & Cunningham, 2004; Levie & Lentz, 1982; Mayer, 2005, 2009). Not only in learning but also in testing it is important that students comprehend the external information (i.e., textual and graphical information) provided in the item stimulus. In order to solve an item, students need to construct an adequate internal mental representation of the scientific problem (problem representation) stated in the item stimulus (Leighton & Sternberg, 2003; Schnotz & Bannert, 2003) and to apply their scientific knowledge (problem solving). A multimedia effect has also been shown in testing situations: Items were less difficult when they contained text and a representational picture as compared to text-only items (Lindner et al., 2016; Sab et al., 2012). Presentation of both text and picture in test items seems to facilitate the construction of an elaborated mental model. Pictures are easily comprehensible and thus provide a more direct and efficient path to mental model construction (Lindner et al., 2017). Building an elaborated mental model constitutes a prerequisite for a successful problem solving process (Brünken, Steinbacher, Schnotz, & Leutner, 2001; Sab et al., 2012; Schnotz & Kürschner, 2008). In addition, pictures might serve as external representations to help answer the question (Zhao, Schnotz, Wagner, & Gaschler, 2014), for instance as cues for mental model updating during the answering process (Lindner et al., 2017). Yet, how students use graphical representations in the process of mental model building might also depend on their specific function (Ainsworth, 2006; Carney & Levin, 2002; Sab et al., 2012). 1.2. Types of graphical representations in test items Standardized science tests commonly employ graphical representations to present information beside textual information (cf. e.g., Martin et al., 2012; OECD, 2007). These graphical representations in test items fulfil different functions (Yeh & McTigue, 2009). For example, pictures may be concrete illustrations of the textual information (representational pictures; e.g., Carney & Levin, 2002). Such pictures just reiterate a specific part or all information provided by the text. In these items, the text itself provides all information necessary to solve them. Nevertheless, such representational pictures may serve as an external scaffold for mental model construction and thus contribute to a better understanding of the problem and ultimately to solving the item (Lindner et al., 2017). Secondly, pictures can organize or structure the textual information (Carney & Levin, 2002). Due to their computational properties organizational pictures are especially suitable for presenting visuo-spatial information or complex situations (e.g., relationship between concepts; Larkin & Simon, 1987). They can support the understanding of the textual content by presenting textual information more efficiently and thus help with the mental model construction. Lastly, graphical representations can include information that is essential for solving the task (e.g., data in a diagram), which is not also stated in the text. Accordingly, test-takers need to interpret the graphical representation and relate its information to the textual description in order to solve the item. Diagrams are a typical example of such essential graphical representations. Consequently, to what extent test-takers utilize the textual information and the graphical information for the construction of a mental representation might depend on the information content of the external representations (Ainsworth, 2006;

S. Saß et al. / Computers & Education 109 (2017) 85e97

87

Carney & Levin, 2002; Sab et al., 2012). Graphical representations containing information necessary for solving the item (e.g., diagrams or charts) constitute the majority of visualizations that are used in standardized science achievement tests (Yeh & McTigue, 2009). However, it is also important to understand what role representational and organizational pictures might play in the process of mental model construction of problems stated in an item stem. 1.3. Eye-tracking studies examining attention allocation while processing text and picture A lot of eye-tracking studies were conducted in the past to shed light on underlying cognitive processes when students process text and picture. Eye-tracking studies analyze how attention is allocated to different representations. Many studies showed that eye movement patterns varied as a function of (a) the processing of multiple-choice items (e.g., Lindner et al., 2014), (b) the successful integration of text and picture (e.g., Johnson & Mayer, 2012), and (c) the processing of text and picture including integration aids (i.e., signals and cues; e.g., Scheiter & Eitel, 2015). Eye-tracking studies investigating how test-takers process multiple-choice items showed that unsuccessful test-takers are less able to distinguish between relevant and irrelevant information whereas successful test-takers spend more time inspecting relevant information than irrelevant information when solving science items (Tsai, Hou, Lai, Liu, & Yang, 2012). Moreover, test-takers have been shown to use representational pictures in the item stem twofold: First, the picture serves as a mental scaffold during the construction of a mental representation of the text and thus supports the problem representation; second, the picture is used for mental model updating during the process of problem solving (Lindner et al., 2017). The number of transitions between text and picture is argued to indicate students attempt to integrate information from text and picture (Holsanova, Holmberg, & Holmqvist, 2009; Johnson & Mayer, 2012). Indeed, the number of transitions correlates positively with learning outcome (e.g., Mason, Tornatora, & Pluchino, 2013). For example, the more transitions students made between multiple representations that included task-relevant unique information, the better was the learning outcome in a chemistry simulation (O’Keefe, Letourneau, Homer, Schwartz, & Plass, 2014). An effective tool to foster the process of integrating both sources and thus enhance coherence formation (Seufert, 2003) is the use of integration aids (i.e., signals or cues). Specifically, integration aids are used to highlight corresponding elements in the text and the graphical representation which in turn should enhance learning (e.g., Florax & Ploetzner, 2010; Kalyuga, Chandler, & Sweller, 1999; Richter, Scheiter, & Eitel, 2016; Scheiter & Eitel, 2015). For example, the use of visual cues like color coding or bolding guides the attention to corresponding information between text and picture and thus enhances integrative processes by directing students' attention to particular graphical elements (cf. attention guidance hypothesis; Ozcelik, Karakus, Kursun, & Cagiltay, 2009). The use of integration aids in representations changes the allocation of visual attention: Representations with signals (text or picture) were shown to be fixated more frequently and longer than representations without signals, indicating more extensive processing of the material (e.g., Boucheix & Lowe, 2010; Ozcelik et al., 2009; Scheiter & Eitel, 2015). Furthermore, signals support students' attempts to integrate information from two sources (Mason, Pluchino, & Tornatora, 2013). A further possibility to enhance integration processes is to arrange the representations which have to be related to each other in close proximity (spatial contiguity principle; Mayer, 1989, 2005; Mayer & Moreno, 2003). Similarly, the integration format influences viewing behavior. Johnson and Mayer (2012) showed that participants make more transitions between text and diagram when they are presented in an integrated format (i.e., text integrated in the diagram) rather than in a non-integrated format (i.e., text separated from the diagram). Finally, students’ viewing behavior (i.e., number of fixations) seems to be linked to their learning outcome, as shown for example in a study by Scheiter and Eitel (2015). A mediation analysis indicated that the number of fixations mediated the effect of signals on learning outcome. In summary, integration aids effectively foster students’ learning. However, studies transferring principles, like this signaling principle, to testing situations are still lacking. One study already showed that integration aids seem to play an important role in that students solve multiple-choice items including integration aids better than items without integration aids (Sab & Schütte, 2016). 1.4. Aim of the study and hypotheses The study focused on integration aids and their effect on how students process different types of graphical representations (representational pictures, organizational pictures, or diagrams) in standardized multiple-choice items assessing science achievement. More specifically, we examined how students inspect items with different types of graphical representations as a function of an integrated versus a non-integrated item format and students’ corresponding achievement, using eye-tracking methodology. Given the conceptual resemblance of comprehension in learning processes and item solving processes we assume that the use of integration aids affects item processing as well. 1.4.1. Problem representation hypotheses In the first four hypotheses, we argue that the item stem layout is crucial for comprehending the problem presented in the item stem. Presenting the item stem in an integrated format (use of integration aids) should foster the process of the problem representation and, thus, the building of a mental model (problem representation hypotheses). Embedding text and picture into an item as well as using integration aids should facilitate relating information in text and picture to each other, which is reflected in changes of visual attention (Hypothesis 1). In particular, a longer relative fixation time on the picture (Hypothesis 1a) and on the text (Hypothesis 1b) is expected to occur in the integrated format condition indicating a more thorough

88

S. Saß et al. / Computers & Education 109 (2017) 85e97

construction of the mental model, which in turn facilitates comprehending the problem. Regarding integration processes we assume that the facilitative function of integration aids results in test-takers exerting more effort to relate both representations to each other and therefore executing more transitions between text and picture in an integrated format than in a nonintegrated format (Hypothesis 2). Moreover, we assumed that visual attention changed depending on the type of graphical representation (Hypothesis 3) and that the type of graphical representation determines how much integration aids facilitate the integration processes: More integration processes are required and integration aids should therefore be more expedient the higher a graphical representation's contribution to the construction of a mental model (Hypothesis 4). Thus, integration aids should especially help test-takers integrate information from diagrams with text, because this type of graphical representation usually provides additional information that is crucial for constructing a coherent mental model of the problem situation. The more successfully text and picture information are integrated, the higher the probability of solving the respective item should become. The correctness of students' responses can therefore be used as an indicator of the quality of students' mental model construction and their understanding of the item information. Students' test scores are expected to be higher when the test employs an integrated item format compared to a format without integration aids (Hypothesis 5). Furthermore, we assumed that students’ eye movements are related to achievement. To the extent that integration aids prompt test-takers to engage in more thorough model construction the relative fixation time on the text and on the picture should be longer and the test scores obtained should be higher in the integrated format condition. (Hypothesis 6). 1.4.2. Problem solving hypotheses The second set of hypotheses relate to how integration aids affect students' allocation of attention when answering the item (problem solving hypotheses). We investigated in a rather exploratory manner whether students change their attention as a function of integration format (Hypothesis 7). For example, students who work through items in a non-integrated format are assumed to shift their fixation back from the answer options to the item stem (text and picture) more frequently than students who work through items in an integrated format that includes integration aids. This effect is expected because students who fail to build up an adequate mental model in the problem representation phase need to reevaluate their mental model when answering the item. However, the usage of the graphical representation in this answering process probably also depends on its type and content. Accordingly, Hypothesis 8 assumed that graphical representations that are necessary for solving the item (i.e., diagrams) are revisited more frequently than redundant ones (i.e., representational and organizational pictures) in the non-integrated format whereas in the integrated condition no such effect is expected. Concerning the relations between the eye movement data and achievement, we expected that transitions between answer options and item stem (text and picture) are related to students’ correctness of response (Hypothesis 9): The more transitions students make between answer options and item stem (text and picture) the worse their achievement. 2. Method 2.1. Participants and design Participants were 60 students from fifth (n ¼ 35) and sixth grade (n ¼ 25). The mean age of the 39 boys and 21 girls was 10.83 (SD ¼ 0.76). We used a 3  2 mixed design. The within-subjects factor referred to the type of graphical representation in the item stem (representational vs. organizational vs. diagram). The between-subjects factor referred to the integration format in the item stimulus (integrated vs. non-integrated). Students were randomly assigned to one of the two experimental conditions (integrated format vs. non-integrated format). We unobtrusively recorded their eye movements while students took the science test. 2.2. Material 2.2.1. Science achievement test We adapted a multiple-choice test assessing science achievement from the National Educational Panel Study (NEPS; Hahn et al., 2013). The test consisted of 12 items. Every item included an item stem consisting of a short text (approximately five sentences) and a graphical representation. The question and four answer options were displayed below the item stem. All items covered important science concepts which are part of the science curriculum in Grade 5; these concepts were embedded in everyday phenomena. The type of graphical representation in the item stimulus differed according to the within-subjects factor: Four items contained representational pictures, four items contained organizational pictures, and four items displayed diagrams in the item stem. Representational pictures just reiterated the textual information. Organizational pictures were schematic illustrations that structured and summarized the textual information. Diagrams included information that was not mentioned in the text but was necessary to solve the item; solving the items was thus impossible when the diagrams were disregarded. The same 12 items were presented in the same order on both levels of the between-subjects factor, but with a different item layout. In the integrated format, we used three forms of integration aids in the different items: (a) referential connections (arrows) related words in the textual representation to the corresponding element in the graphical representation; (b) the textual information was presented in closer proximity to the picture than in the non-integrated format condition; (c)

S. Saß et al. / Computers & Education 109 (2017) 85e97

Fig. 1. Areas of interest and eye movements from one test-taker processing an item presented in an integrated format (left) and one test-taker processing an item presented in a non-integrated format (right).

89

90

S. Saß et al. / Computers & Education 109 (2017) 85e97

corresponding information in the text and the picture was highlighted using bold print. In some items, more than one integration aid was used. In the non-integrated format, the item stimuli consisted of a textual representation and a corresponding graphical representation located above the text; no referential connections related the two representations to each other. With regards to the within-subjects factor (i.e., graphical representation), items were arranged in a mixed order. As an indicator of students' achievement we assessed the accuracy of their response to the science test items: Specifically, we computed the proportion of items students answered correctly. The reliability of the test as measured by Cronbach's alpha was sufficient, a ¼ 0.73. 2.2.2. Control variables A questionnaire was administered collecting students' gender, age, and most recent grade in science. Reading comprehension was assessed using an established standardized test (ELFE 1e6; Lenhard & Schneider, 2006). This computer-based test consists of four subtests (word comprehension, sentence comprehension, text comprehension, reading speed). For example, in the sentence comprehension subtest students are presented with sentences in which single words are intentionally omitted. Students are required to choose the option that reasonably completes the sentence from five options. General cognitive abilities were assessed using a non-verbal subtest of the KFT 4e12 þ R (Heller & Perleth, 2000), the German version of the Cognitive Abilities Tests (CAT; Thorndike & Hagen, 1971) assessing students' reasoning ability. The subscale consists of 25 items, each of which presents students with a pair of meaningfully related figures and a single figure, to which the appropriate counterpart has to be selected among five answer options. How the two figures that constitute a pair relate to each other has to be inferred from the complete pair for each item. The reliability of the test as measured by Cronbach's alpha was satisfactory, a ¼ 0.81. 2.3. Procedure and apparatus The first session took place in our laboratory, where students were tested in a single-session. The student was seated at an approximate distance of 27.5 inches in front of a 22-inch screen with a 1680  1050 pixel resolution that presented each item on a separate screen. It was not possible for the students to return to previous items. We conducted an animated 8-point calibration image and subsequent validation before students started working on the test. Eye movements were recorded using a video-based remote eye-tracking system (SMI iView X™ RED; 120 Hz sampling rate). In a subsequent group session, students completed the background questionnaire and the reading ability test. Every student received 15 Euro for their participation. 2.4. Eye movement data analysis We defined three areas of interest (AOI) for each item: Text, picture, and answer options (Fig. 1). For the AOIs text and picture we calculated the proportion of fixation by relating the time students spent fixating the text or the picture, respectively, to the total fixation time on the corresponding item. We calculated such relative fixation times for text and for picture for each type of graphical representation. Subsequently, the number of transitions between the text and the picture, the text and the answer options, and the picture and the answer options were calculated for each item by determining the number of saccades that occurred between the AOIs. We considered transitions in both directions and calculated the average number of transitions for each type of graphical representation. 3. Results 3.1. Control variables We computed independent t tests to check whether students’ background variables differed between the two experimental groups. Because the time allowed for taking the test was not limited, we considered the overall time for students to complete the test as an additional control variable. The tests revealed no statistically significant differences between the groups (see Table 1).

Table 1 Control variables. Background Variables

Condition: integrated M (SD)

Condition: non-integrated M (SD)

Statistics

Age Science grade Reading ability General cognitive ability Overall time (in min)

10.67 (0.76) 2.78 (0.80) 55.99 (22.86) 17.77 (5.88) 13.16 (3.79)

11.00 (0.74) 2.47 (0.78) 58.40 (21.94) 17.13 (5.56) 12.74 (4.80)

t(58) t(57) t(57) t(58) t(58)

¼ ¼ ¼ ¼ ¼

1.72, p ¼ 0.09 0.93, p ¼ 0.36 0.41, p ¼ 0.68 0.43, p ¼ 0.67 0.37, p ¼ 0.71

Note: Grades in elementary schools in Germany range between 1 and 6 with 1 being the highest score; general cognitive ability sumscore, reading ablity percentile rank.

S. Saß et al. / Computers & Education 109 (2017) 85e97

91

Table 2 Means and standard deviations on the dependent variables.

Integrated Representational Organizational Diagrammatic Non-integrated Representational Organizational Diagrammatic

Achievement

Relative fixation time: text in %

Relative fixation time: picture in %

Number of transitions between text and picture

Number of transitions between answer options and text

Number of transitions between answer options and picture

M

SD

M

SD

M

SD

M

SD

M

SD

M

SD

0.48 0.41 0.53

0.26 0.26 0.22

33.35 34.51 38.68

10.37 12.20 11.94

6.94 13.95 23.66

2.54 5.25 7.27

3.82 4.26 3.60

1.80 1.85 1.66

0.45 0.58 0.83

0.37 0.55 0.52

2.20 2.51 6.38

1.75 1.22 2.31

0.48 0.43 0.33

0.25 0.30 0.25

35.72 36.60 41.32

8.02 7.63 11.69

4.99 9.07 20.47

2.92 3.72 7.31

3.42 3.48 3.56

1.58 1.39 1.17

0.71 0.95 1.80

0.56 0.67 1.01

1.23 1.98 5.44

1.12 1.10 2.30

3.2. Problem representation hypothesis 3.2.1. Eye movement parameters We investigated the effects of the experimental conditions on three aspects of students’ eye movements: (a) The number of transitions between the text and the picture, (b) the relative fixation time on the picture, and (c) the relative fixation time on the text. With regard to the number of transitions between the text and the picture, a 3  2 mixed-model analysis of variance (ANOVA) revealed neither a significant effect of the integration format, F(1, 58) ¼ 1.46, p ¼ 0.23, h2p ¼ 0.02, nor an effect of the type of graphical representation, F(2, 116) ¼ 1.24, p ¼ 0.30, h2p ¼ 0.02. The interaction effect was not statistically significant either, F(2, 116) ¼ 1.65, p ¼ 0.20, h2p ¼ 0.03. Contrary to our expectations, the two item formats did not prompt students to relate the information in the text and the picture in a different way. Concerning the relative fixation time on the picture the data revealed a significant main effect of integration format, F(1, 58) ¼ 10.98, p ¼ 0.002, h2p ¼ 0.16. Students who worked on items in the non-integrated format spent less time fixating the picture than students who worked on the same tasks presented in the integrated format with integration aids (see Table 2). In addition, the type of graphical representation had a statistically significant main effect on relative fixation time on the picture, F(2, 116) ¼ 225.01, p < 0.001, h2p ¼ 0.80, indicating that the time students spent looking at the picture in relation to their total fixation time while working on the respective item differed depending on the representation type used in the item stem. Bonferroni corrected post hoc tests showed that the relative time students spent looking at the graphical representation was significantly higher for diagrams than for representational pictures or organizational pictures (Table 2; ps < 0.001). Moreover, the relative fixation time on the graphical representation was higher for organizational pictures compared to representational pictures (p < 0.001). The interaction effect of type of graphical representation and integration format on the relative fixation time on the picture was not statistically significant, F(2, 116) ¼ 1.83, p ¼ 0.17, h2p ¼ 0.03. Thus, the integration format had no different effects on the relative fixation time on the graphical representation depending on which representation type was used. In contrast to the results obtained for the relative fixation time on the picture, the integration format had no statistically significant effect on the relative fixation time on the text, F(1, 58) ¼ 1.11, p ¼ 0.30, h2p ¼ 0.02. Students in the integrated format condition did not spent a higher proportion of their fixation time on the text than students in the non-integrated format condition. A significant main effect of the type of graphical representation was observed, F(2, 116) ¼ 9.99, p < 0.001, h2p ¼ 0.15, suggesting that students’ fixation time on the text differed depending on which type of graphical representation was used in the item stem. Bonferroni corrected post hoc tests showed that the relative fixation time on the text in items with diagrams was significantly higher as compared with items containing a representational or organizational picture (ps < 0.001). The relative fixation time on the text in items with representational pictures and organizational pictures did not significantly differ (p ¼ 0.42). The interaction effect between the integration format and the type of graphical representation in regards to the time students spent looking at the text was also not statistically significant, F(2, 116) ¼ 0.02, p ¼ 0.98, h2p ¼ 0.001. 3.2.2. Achievement We investigated students' achievement as an indicator of the quality of students' mental model construction. A 3  2 mixed-model ANOVA showed neither a significant main effect for the type of graphical representation, F(2, 116) ¼ 1.78, p ¼ 0.17, h2p ¼ 0.03, nor a significant main effect of the integration format on students' achievement in the test, F(1, 58) ¼ 1.51, p ¼ 0.23, h2p ¼ 0.03. The interaction effect was, however, statistically significant, F(2, 116) ¼ 4.91, p ¼ 0.009, h2p ¼ 0.08. This indicates that the integration format had different effects on students' achievement depending on the type of graphical representation. In particular, post hoc tests revealed that integration format only influenced students’ achievement on items

92

S. Saß et al. / Computers & Education 109 (2017) 85e97

including diagrams, t(58) ¼ 3.30, p ¼ 0.002, h2p ¼ 0.16. More precisely, the proportion of solved items including diagrams was greater when items were presented in an integrated rather than a non-integrated format (Table 2). No statistically significant differences were observed between the integrated and the non-integrated format for representational and organizational pictures (all t(58) < 0.23, ps > 0.82). 3.2.3. Eye movement parameters and achievement A regression analysis for the relationship between eye movements and achievement is displayed in Table 3. Because a statistically significant effect of integration format on achievement emerged only for items including diagrams, we solely report results for this type of items. The relative fixation time, both for picture and text positively predicted students' achievement. The longer students looked at the text and at the diagram in the item stem the more items they solved. The relative fixation time students spent on the diagram was by far the strongest predictor of students’ achievement. 3.3. Problem solving hypotheses 3.3.1. Eye movement parameters The results of a 3  2 mixed-model ANOVA revealed a significant main effect for the integration format on transitions between answer options and the picture, F(1, 58) ¼ 6.31, p ¼ 0.015, h2p ¼ 0.10. Students made more transitions between answer options and the picture in the integrated format (M ¼ 3.70, SD ¼ 1.29) than in the non-integrated format (M ¼ 2.89, SD ¼ 1.20). There was also a significant main effect of type of graphical representation for transitions between answer options and the picture, F(2, 116) ¼ 151.33, p < 0.001, h2p ¼ 0.72. Bonferroni corrected post hoc tests showed that significantly more transitions were made between answer options and the picture in items that included diagrams (M ¼ 5.90, SD ¼ 2.33) compared to items with representational pictures (M ¼ 1.72, SD ¼ 1.54) or organizational pictures (M ¼ 2.25, SD ¼ 1.18; ps < 0.05). The interaction effect of type of graphical representation and integration format was not statistically significant, F(2, 116) ¼ 0.45, p ¼ 0.64, h2p ¼ 0.01. Results regarding the transitions between answer options and the text also revealed a significant main effect of integration format, F(1, 58) ¼ 19.81, p < 0.001, h2p ¼ 0.26; here, students in the non-integrated format condition made more transitions (M ¼ 1.15, SD ¼ 0.58) than students in the integrated format condition (M ¼ 0.62, SD ¼ 0.31). The main effect for the type of graphical representation was also significant, F(2, 116) ¼ 28.99, p ¼ 0.001, h2p ¼ 0.33. Bonferroni corrected post hoc tests showed that the number of transitions between answer options and the text was significantly larger in items including diagrams (M ¼ 1.32, SD ¼ 0.94) compared to items with representational pictures (M ¼ 0.58, SD ¼ 0.49) or organizational pictures (M ¼ 0.76, SD ¼ 0.64; ps < 0.001). There was no significant difference in the transitions between answer options and text between items with representational and organizational pictures (p ¼ 0.16). The interaction effect between the integration format and type of graphical representation was statistically significant, F(2, 116) ¼ 7.09, p < 0.001, h2p ¼ 0.11. This indicates that the integration format had different effects on transitions between answer options and text depending on which type of graphical representation was used. Follow up analyses showed that for all three types of graphical representations students made more transitions between answer options and text in the non-integrated condition compared with the integrated condition (all t(58) > 2.09, ps < 0.02). However, the effect of integration format showed the largest effect size for diagrams (h2p ¼ 0.27) and a medium effect size for organizational pictures (h2p ¼ 0.08) and representational pictures (h2p ¼ 0.07). 3.3.2. Eye movement parameters and achievement Because the integration format did not affect achievement on items including representational or organizational pictures, we again solely report results for items including diagrams. Results from a regression analysis that regressed achievement on students’ transitions between answer options on the one hand and the text or the diagram on the other hand revealed that students answered more items correctly the more they switched between looking at the diagram and the answer options (Table 4).

Table 3 Regression analysis of achievement for students’ relative fixation time on diagrams.

Step 1 Constant Integration format Step 2 Constant Integration format Relative fixation time on text Relative fixation time on diagram

B

SE B

b

0.43 0.10

0.03 0.03

0.40**

0.43 0.08 0.07 0.11

0.03 0.03 0.03 0.03

0.33** 0.26* 0.45**

Note. R2 ¼ 0.16 for Step 1, adj. R2 ¼ 0.31 for Step 2 (p < 0.001). *p < 0.05. **p < 0.01. Integration format is coded 1 ¼ integrated format, 1 ¼ non-integrated format.

S. Saß et al. / Computers & Education 109 (2017) 85e97

93

Table 4 Regression analysis of achievement in items containing diagrams.

Step 1 Constant Integration Step 2 Constant Integration Transitions Transitions

B

SE B

b

format

0.42 0.10

0.03 0.03

0.41*

format answer options to diagrams answer options to text

0.23 0.07 0.03 -0.04

0.08 0.04 0.01 0.04

2

2

0.27y 0.30* -0.15

y

Note. R ¼ 0.17 for Step 1, adj. R ¼ 0.25 for Step 2 (p ¼ 0.01). p < 0.10. *p < 0.05. Integration format is coded 1 ¼ integrated format, 1 ¼ non-integrated format.

4. Discussion Graphical representations are commonly used in standardized science tests. This study investigated how fifth- and sixthgrade students inspect items with different types of graphical representations as a function of an integrated versus a nonintegrated item format using eye-tracking methodology and students corresponding test achievement. We thus applied a well-known principle from multimedia learning research, namely the use of integration aids, to a testing situation. Beyond the insights provided by processes underlying test-taking behavior and its practical implications for the generation of achievement tests, the study thus contributes to the question whether multimedia layout principles can be translated to testing situations.

4.1. Differential effects during the problem representation phase Based on the attention guidance hypothesis (Ozcelik et al., 2009; Ozcelik, Arslan-Ari, & Cagiltay, 2010) more attention was expected to be directed to elements in the text and corresponding elements in the graphical representation in items presented in an integrated format than in a non-integrated format (Hypothesis 1). There was only evidence that students in the integrated format condition spent more time on the graphical representation than students in the non-integrated format condition (Hypothesis 1a); no corresponding effect was observed with regard to the time students spent fixating the text (Hypothesis 1b). The proposed function of integration aids to facilitate the processing and understanding of the text may not have taken effect in our study, because the text information in the science test items was rather short and the associated cognitive demands therefore comparatively low. However, integration aids apparently guided test-takers’ attention to the picture. Despite test-takers spending more time fixating the graphical representation the overall time spent taking the test was not statistically different between integration format conditions. With regards to transitions between text and picture, results indicate that test-takers in the integrated format condition exerted more effort to relate both representations to each other; they made more transitions between text and picture than test-takers in the non-integrated format condition (Hypothesis 2). Yet, we only found a small number of transitions between texts and pictures in general. The absence of an interaction effect of integration format and type of graphical representation on the amount of transitions between text and picture might thus be caused by a floor effect (Hypothesis 4). The number of transitions may mask different processes: On the one hand, an integrated format might foster transitions between text and picture within the scope of targeted item processing; on the other hand, a large number of transitions between text and picture might have indicated effortful but unsuccessful search behavior. Yet another possible explanation is that textepicture integration in the current test items differed from processing text and picture in the context of learning. The test items included only short text passages. Therefore, the problem representation might have required only few transitions between text and picture. Apparently, the mapping requirements in our test items were less demanding compared to other studies with undergraduate students (Ozcelik et al., 2009; Scheiter & Eitel, 2015). Comparing the fixations between the different types of graphical representation (independent from integration format) results revealed that students spent relatively more time on an item stem including a diagram, compared with item stems including organizational or representational pictures (Hypothesis 3). This is probably because items including diagrams involve different cognitive requirements than those with an organizational picture or a representational picture. Information given in a diagram has to be used in order to build a coherent and full mental model that includes all the necessary information, whereas representational or organizational pictures might be most of all used as mental scaffolds and a second source of information for building a text-based mental model. Consequently, we expected that integration aids would especially lead to more attention to diagrams compared to the other graphical representations (Hypothesis 4). But that was not the case. In line with research demonstrating students’ general preference for text (Hannus & €na €, 1999), integration aids lead to more attention to the graphical representation regardless of the particular represenHyo tation type. The relatively longer fixation time of the diagrams as compared with representational or organizational pictures was observed across integration formats. Even without respective aids, students seemed to realize that the different graphical

94

S. Saß et al. / Computers & Education 109 (2017) 85e97

representations required differential attention. As stated above, a floor effect might explain why no interaction effect of integration format and type of graphical representation was observed on the number of transitions between text and picture. Furthermore, we assumed that students' test scores would be higher in an integrated format than in a non-integrated format (Hypothesis 5). However, the results revealed that this was only true for items with diagrams in the item stem. In our study, the diagrams included information that was not presented in the text but was essential for solving the item. It seemed that signals nudged students to inspect the picture which in turn lead to better achievement. Further evidence was given by the results showing that an increase of the relative fixation time on the diagram respectively the text was positively related to students' test scores (Hypothesis 6). These results are in line with studies showing that the longer students look at a picture, the higher their learning outcome was (Hung, 2014). Additionally, studies that analyzed students' strategies when processing text and diagrams could demonstrate that students who ignored the diagram had worse learning outcomes (Jian & Wu, 2015; Mason, Tornatora, et al., 2013). It is important to notice, that this effect only occurred for items with diagrams, which are essential graphical representations, not for items including representational or organizational pictures. One explanation might be again that items involving diagrams are more cognitively demanding. Selecting and identifying relevant information (verbal and pictorial), maintaining the information in working memory, and integrating the verbal and pictorial information (as well as prior knowledge) require cognitive resources (see cognitive load theory; Sweller, 2005). Item features like integration aids, might reduce cognitive load. A study by Gillmor, Poggio, and Embretson (2015) demonstrated that the use of signals as a tool to reduce extraneous load (Clark, Nguyen, & Sweller, 2011) in mathematics items positively effects students' performance (see also Sab & Schütte, 2016). Hence, test-takers’ performance is not only affected by their ability (cognitive skills, knowledge) but also by information-processing demands determined by item features (amount of information to be processed). 4.2. Allocation of attention during the problem solving phase The second set of hypotheses examined students’ attentional behavior while answering the items. We expected that students in the integrated format condition already built up an adequate mental model and would not need to look back to the item stem (neither to the text nor to the graphical representation) while answering the item (Hypothesis 7). Supporting this hypothesis, our results revealed that students switched more often between answer options and the text when items were presented in a non-integrated format than in an integrated format; possibly students presented with items in an integrated format had built up a fairly adequate mental model, whereas students needed to update their mental model more often in the non-integrated format condition. This effect, however, varied as a function of the type of graphical representation and was strongest for items with diagrams (Hypothesis 8). Yet, the effect size was quite small and so was the overall number of transitions students made between answer options and text. Moreover, a higher rate of revisiting the text during the problem solving phase did not contribute to their test achievement (Hypothesis 9). A reverse effect of integration format occurred for pictures: Students switched more often between answer options and the picture in the integrated format than in the non-integrated format condition. The overall number of transitions between answer options and picturedparticularly as compared with the number of transitions between answer options and textdsuggests that students realized the value of graphical representation for building a mental model; despite the inadequate mental model building in the problem representation phase, integration aids seem to have fostered its updating during the problem solving phase. Our assumption that reevaluation would not be necessary in the integrated format condition was, however, not supported; whether or not integration aids were present, students needed to update their mental model while answering the items. The finding that pictures are used as an external representation when answering questions (Lindner et al., 2017; Schnotz et al., 2014; Zhao et al., 2014) thus also holds when items are presented in an integrated format. All three types of graphical representations were used as external scaffolds, but diagrams more often than representational or organizational pictures; the interaction effect of integration format and type of graphical representation was, however, not statistically significant. Students in both conditions seem to have realized that the diagram was indispensable for solving the item, but only as they entered the problem solving phase. Moreover, only in the case of diagrams did students achieve better test scores the more transitions they made between answer options and the graphical representation (Hypothesis 9). 4.3. Limitations and further directions Limitations of our study might first concern the generalizability to different age groups. We focused on how fifth- and sixth-grade students process multiple-choice science items comprising graphical representations in addition to text. The ability to integrate text and picture in this age group largely depends on how textepicture integration was taught in school (Nitz, Ainsworth, Nerdel, & Prechtl, 2014). We had no information about how teachers of our sample so far introduced the handling of graphical representations in their classroom. Looking at the transitions (as an indicator for the attempts to integrate information from text and picture) the results showed that students made very few transitions between text and picture while processing the item stem (problem representation). It seems that students at this age struggle integrating text and picture (Mason, Pluchino, et al., 2013; Schnotz et al., 2014). It is conceivable that the ability to process multiple representations develops as students mature and are increasingly exposed to such material in science classes. Whether or not older students react differently to variations in the item format is a question for future research.

S. Saß et al. / Computers & Education 109 (2017) 85e97

95

Second, although we analyzed three different types of graphical representations we only analyzed items assessing science achievement and focused on the cognitive process of applying scientific knowledge. Hence, it is conceivable that graphical representations indeed have another function in items, for example, where students just need to apply factual knowledge or perform even more complex problem solving processes. There the type of graphical representation may interact with the cognitive processes needed to apply. Thus, research is needed to not only examine the effects of different types of representations in test items, but also other representation formats commonly used in large-scale assessments. In order to find out how far our results can be applied in other testing situations (i.e., other grades, other domains, other task types), it would be necessary to conduct comparable studies. Third, only four items in each type of graphical representation set were administered. In principle, a wider set of items would constitute a more adequate measure of performance in each stimuli set. Employing type of graphical representation in the item stem as a within-subjects factor, however, increased our study's ecological validity by preventing students adapting to the items' cognitive demands. Students' age did not allow for a more comprehensive test; it resulted in severe time constraints to avoid overworking them (and resulting confounding effects on item processing). Moreover, we presented the items to all students in a fixed order. Therefore, we cannot be sure whether or not our results are affected by order effects. That fixed order, however, was counterbalanced with regard to the different types of graphical representations rendering differential effects less plausible. The test design also prohibits direct comparisons of the withinsubject factor levels: Each problem was presented with a particular type of graphical representation, not the respective other two. Therefore, it is possible that the obtained effects also depended more on other item features than the different graphical representations (e.g., the everyday phenomena they were embedded in). Future research might also explicitly focus on moderating variables. One likely moderator is test-takers’ prior knowledge. Prior knowledgedconceptual knowledge as well as knowledge about textepicture integrationdlikely influence the viewing behavior when processing different types of graphical representations and signals. For instance, students with prior knowledge might (more quickly) recognize that representational or organizational pictures do not include unique information. Signals are also assumed to be more beneficial for students with low prior knowledge (Richter et al., 2016). They might be less helpful for students with high prior knowledge because when students are able to integrate information from two representations without aids they ignore signals (Scheiter & Eitel, 2015). Nevertheless, these students also have to process the presented information. In the case of representational pictures, signals might provoke unnecessary memory load that is detrimental for item processing (Kalyuga, Ayres, Chandler, & Sweller, 2003). For students with high prior knowledge, the graphical representation itself might constitute unnecessary load unless it contributes unique information, whereas students with lower prior knowledge likely profit from additional pictures, even if those pictures just reiterate the text: They can help the test-taker build a mental model about the problem situation (Lindner et al., 2017). However, studies investigating the role of pictures in multiple-choice items are scarce and are needed to close the research gap. 5. Conclusion The study contributes to the theoretical understanding of how test-takers process multiple-choice items with different types of graphical representations and how these processes are influenced by different item formats (integrated format vs. non-integrated format). Our study illustrates that applying multimedia learning principles can come to fruition in research on cognitive processes during testing situations; eye movement analysis is a valuable tool to provide information on processes underlying test-taking behavior. An eye-tracking approach provides information how test-takers process standardized test items and, more importantly, the processing data can be related to achievement data, which in turn leads to more valid statements about causal effects in experimental manipulation of item formats (i.e., how item characteristics affect achievement on the test). Ultimately, such research will inform item generation and thus contribute to the quality of achievement tests. Funding This research did not receive any specific grant from public or commercial funding agencies or the non -profit sector. References Ainsworth, S. (1999). The function of multiple representations. Computer & Education, 33, 131e152. http://dx.doi.org/10.1016/S0360-1315(99)00029-9. Ainsworth, S. (2006). DeFT: A conceptual framework for considering learning with multiple representations. Learning and Instruction, 16, 183e198. http://dx. doi.org/10.1016/j.learninstruc.2006.03.001. Anglin, G. J., Vaez, H., & Cunningham, K. L. (2004). Visual representations and learning: The role of static and animated graphics. Handbook of research on educational communications and technology, 2, 865e916. Boucheix, J.-M., & Lowe, R. K. (2010). An eye tracking comparison of external pointing cues and internal continuous cues in learning with complex animations. Learning and Instruction, 20, 123e135. http://dx.doi.org/10.1016/j.learninstruc.2009.02.015. €sentations- und Abrufkodalit€ Brünken, R., Steinbacher, S., Schnotz, W., & Leutner, D. (2001). Mentale Modelle und Effekte der Pra at beim Lernen mit €dagogische Psychologie, 15, Multimedia [Mental models and effects of presentation and retrieval cue codality in multimedia learning]. Zeitschrift für Pa 16e27. http://dx.doi.org/10.1024//1010-0652.15.1.16. Carney, R. N., & Levin, J. R. (2002). Pictorial illustrations still improve students' learning from text. Educational Psychology Review, 14, 5e26. http://dx.doi.org/ 10.1023/A:1013176309260. Clark, R. C., Nguyen, F., & Sweller, J. (2011). Efficiency in learning: Evidence-based guidelines to manage cognitive load. San Francisco, CA: John Wiley & Sons.

96

S. Saß et al. / Computers & Education 109 (2017) 85e97

Florax, M., & Ploetzner, R. (2010). What contributes to the split-attention effect? The role of text segmentation, picture labelling, and spatial proximity. Learning and Instruction, 20, 216e224. http://dx.doi.org/10.1016/j.learninstruc.2009.02.021. Gillmor, S. C., Poggio, J., & Embretson, S. (2015). Effects of reducing the cognitive load of mathematics test items on student performance. Numeracy, 8. http://dx.doi.org/10.5038/1936-4660.8.1.4. €ps, K., Ro €nnebeck, S., Martensen, M., Hansen, S., Saß, S., et al. (2013). Assessing scientific literacy over the lifespan e a description of the NEPS Hahn, I., Scho science framework and the test development. Journal for Educational Research Online, 5(2), 110e138. € na €, J. (1999). Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemporary Hannus, M., & Hyo Educational Psychology, 24, 95e123. http://dx.doi.org/10.1006/ceps.1998.0987. €higkeitstest für 4. bis 12. Klassen, Revision. Go €ttingen: Beltz-Testgesellschaft. Heller, K. A., & Perleth, C. (2000). KFT 4-12þ R kognitiver Fa Holsanova, J., Holmberg, N., & Holmqvist, K. (2009). Reading information graphics: The role of spatial contiguity and dual attentional guidance. Applied Cognitive Psychology, 23, 1215e1226. http://dx.doi.org/10.1002/acp.1525. Hung, Y. N. (2014). What are you looking at? An eye movement exploration in science text reading. International Journal of Science and Mathematics Education, 12, 241e260. http://dx.doi.org/10.1007/s10763-013-9406-z. Jian, Y.-C., & Wu, C.-J. (2015). Using eye tracking to investigate semantic and spatial representations of scientific diagrams during text-diagram integration. Journal of Science Education and Technology, 24, 43e55. http://dx.doi.org/10.1007/s10956-014-9519-3. Johnson, C. I., & Mayer, R. E. (2012). An eye movement analysis of the spatial contiguity effect in multimedia learning. Journal of Experimental Psychology: Applied, 18, 178e191. http://dx.doi.org/10.1037/a0026923. Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychlogist, 38, 23e31. http://dx.doi.org/10.1207/ s15326985ep3801_4. Kalyuga, S., Chandler, P., & Sweller, J. (1999). Managing split-attention and redundancy in multimedia instruction. Applied Cognitive Psychology, 13, 351e371. http://dx.doi.org/10.1002/(SICI)1099-0720(199908)13:4. <351::AID-ACP589>3.0.CO;2e6. Larkin, J., & Simon, H. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11, 65e100. http://dx.doi.org/10.1111/j.1551-6708. 1987.tb00863.x. Leighton, J. P., & Sternberg, R. J. (2003). Reasoning and problem solving. In I. B. Weiner, A. F. Healy, & R. W. Proctor (Eds.), Handbook of psychology: Experimental psychology (pp. 623e648). New York, NY: John Wiley & Sons. €ndnistest für Erst-und Sechstkla €ssler: ELFE 1-6. Go €ttingen: Hogrefe. Lenhard, W., & Schneider, W. (2006). Ein Leseversta Levie, W. H., & Lentz, R. (1982). Effects of text illustrations: A review of research. Educational Communication and Technology, 30, 195e232. http://dx.doi.org/ 10.1007/BF02765184. €ller, O. (2017). Identifying processes underlying the multimedia effect in testing: An eye-movement analysis. Lindner, M. A., Eitel, A., Strobel, B., & Ko Learning and Instruction, 47, 91e102. http://dx.doi.org/10.1016/j.learninstruc.2016.10.007. €ller, O. (2014). Tracking the decision making process in multiple-choice assessment: Lindner, M. A., Eitel, A., Thoma, G.-B., Dalehefte, I. M., Ihme, J. M., & Ko Evidence from eye movements. Applied Cognitive Psychology, 28, 738e752. http://dx.doi.org/10.1002/acp.3060. €ller, O. (2016). How representational pictures enhance students' performance and test-taking pleasure in low-stakes Lindner, M. A., Ihme, J. M., Saß, S., & Ko assessment. European Journal of Psychological Assessment. http://dx.doi.org/10.1027/1015-5759/a000351. Advance online publication. Martin, M. O., Mullis, I. V. S., Foy, P., & Stanco, G. M. (2012). TIMSS 2011 international results in science. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. Mason, L., Pluchino, P., & Tornatora, M. C. (2013). Effects of picture labeling on science text processing and learning: Evidence from eye movements. Reading Research Quarterly, 48, 199e214. http://dx.doi.org/10.1002/rrq.41. Mason, L., Tornatora, M. C., & Pluchino, P. (2013). Do fourth graders integrate text and picture in processing and learning from an illustrated science text? Evidence from eye-movement patterns. Computers & Education, 60, 95e109. http://dx.doi.org/10.1016/j.compedu.2012.07.011. Mayer, R. E. (1989). Systematic thinking fostered by illustrations in scientific text. Journal of Educational Psychology, 81, 240e246. http://dx.doi.org/10.1037/ 0022-0663.81.2.240. Mayer, R. E. (2005). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 31e48). New York, NY: Cambridge University Press. Mayer, R. E. (2009). Multimedia learning (2nd ed.). University of California, Santa Barbara: Cambridge University Press. Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38, 43e52. http://dx.doi.org/10.1207/ S15326985EP3801_6. Nitz, S., Ainsworth, S. E., Nerdel, C., & Prechtl, H. (2014). Do students perception of teaching predict the development of representational competence and biological knowledge? Learning and Instruction, 31, 13e22. http://dx.doi.org/10.1016/jlearninstruc2013.12.003ainsw. Organisation for Economic Cooperation and Development. (2007). PISA 2006: Science competencies for tomorrow's world. Paris: OECD. Ozcelik, E., Arslan-Ari, I., & Cagiltay, K. (2010). Why does signaling enhance multimedia learning? Evidence from eye movements. Computers in Human Behavior, 26, 110e117. http://dx.doi.org/10.1016/j.chb.2009.09.001. Ozcelik, E., Karakus, T., Kursun, E., & Cagiltay, K. (2009). An eye-tracking study of how color coding affects mulitmedia learning. Computers & Education, 53, 445e453. http://dx.doi.org/10.1016/j.compedu.2009.03.002. O'Keefe, P. A., Letourneau, S. M., Homer, B. D., Schwartz, R. N., & Plass, J. L. (2014). Learning from multiple representations: An examination of fixation patterns in a science simulation. Computers in Human Behavior, 35, 234e242. http://dx.doi.org/10.1016/j.chb.2014.02.040. Paivio, A. (1986). Mental representations: A dual coding approach. Oxford, England: Oxford University Press. Richter, J., Scheiter, K., & Eitel, A. (2016). Signaling text-picture relations in multimedia learning: A comprehensive meta-analysis. Educational Research Review, 17, 19e36. http://dx.doi.org/10.1016/j.edurev.2015.12.003. Saß, S., & Schütte, K. (2016). Helping poor readers demonstrate their science competence: Item characteristics supporting textepicture integration. Journal of Psychoeducational Assessment, 34(1), 91e96. http://dx.doi.org/10.1177/0734282915588389. € ller, O. (2012). Pictures in test items: Effects on response time and response correctness. Applied Cognitive Psychology, Saß, S., Wittwer, J., Senkbeil, M., & Ko 26, 70e81. http://dx.doi.org/10.1002/acp.1798. Scheiter, K., & Eitel, A. (2015). Signals foster multimedia learning by supporting integration of highlighted text and diagram elements. Learning and Instruction, 36, 11e26. http://dx.doi.org/10.1016/j.learninstruc.2014.11.002. Schmidt-Weigand, F., Kohnert, A., & Glowalla, U. (2010). Explaining the modality and contiguity effects: New insights from investigating students' viewing behaviour. Applied Cognitive Psychology, 24, 226e237. http://dx.doi.org/10.1002/acp.155. Schnotz, W. (2002). Towards an integrated view of learning from text and visual displays. Educational Psychology Review, 14, 101e120. http://dx.doi.org/10. 1023/A:1013136727916. Schnotz, W., & Bannert, M. (2003). Construction and interference in learning from multiple representations. Learning and Instruction, 13, 141e156. http://dx. doi.org/10.1016/S0959-4752(02)00017-8. Schnotz, W., & Kürschner, C. (2008). External and internal representations in the acquisition and use of knowledge: Visualization effects on mental model construction. Instructional Science, 36, 175e190. http://dx.doi.org/10.1007/s11251-007-9029-2. Schnotz, W., Ludewig, U., Ullrich, M., Horz, H., McElvany, N., & Baumert, J. (2014). Strategy shifts during learning from texts and pictures. Journal of Educational Psychology, 106(4), 974e989. http://dx.doi.org/10.1037/a0037054. Seufert, T. (2003). Supporting coherence formation in learning from multiple representations. Learning and Instruction, 13, 227e237. http://dx.doi.org/10. 1016/S0959-4752(02)00022-1. Sweller, J. (2005). Implications of cognitive load theory for multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 19e30). New York, NY: Cambridge.

S. Saß et al. / Computers & Education 109 (2017) 85e97

97

Thorndike, R. L., & Hagen, E. (1971). Cognitive abilities test. Boston, MA: Houghton-Mifflin. Tsai, M.-J., Hou, H.-T., Lai, M.-L., Liu, W.-Y., & Yang, F.-Y. (2012). Visual attention for solving multiple-choice science problem: An eye-tracking analysis. Computers & Education, 58, 375e385. http://dx.doi.org/10.1016/j.compedu.2011.07.012. Yeh, Y.-F. Y., & McTigue, E. M. (2009). The frequency, variation, and function of graphical representations within standardized state science tests. School Science and Mathematics, 109, 435e449. http://dx.doi.org/10.1111/j.1949-8594.2009.tb18291.x. Zhao, F., Schnotz, W., Wagner, I., & Gaschler, R. (2014). Eye tracking indicators of reading approaches in text-picture comprehension. Frontline Learning Research, 2(5), 46e66. http://dx.doi.org/10.14786/flr.v2i4.98.