- Email: [email protected]
Visual signaling in virtual world-based assessments: The SAVE Science project
Information Sciences 264 (2014) 32–40
Contents lists available at ScienceDirect
Information Sciences journal homepage: www.elsevier.com/locate/ins
Visual signaling in virtual world-based assessments: The SAVE Science project Brian C. Nelson ⇑, Younsu Kim, Cecile Foshee, Kent Slack Arizona State University, Brickyard Engineering, 699 S. Mill Ave., Tempe, AZ 85281, USA
a r t i c l e
i n f o
Article history: Available online 8 September 2013 Keywords: Virtual world Instructional design Game-based learning Assessment Science education
a b s t r a c t In this paper we describe a study into the impact of visual signaling techniques used in a virtual world-based assessment of science inquiry and content on (1) student cognitive load and (2) assessment efficiency. The study, run with 7th grade students in the United States, found that use of visual signaling was significantly associated with lower levels of student self-reported cognitive load versus students in a no-signaling version of the assessment. Further, the efficiency of the virtual world-based assessment was significantly higher, as measured by in-world object interaction rates, for students in the visual signaling version of the assessment than for those in the no-signaling treatment. In the paper, we discuss the results and their meaning for the design of virtual world and game-based assessments. Ó 2013 Elsevier Inc. All rights reserved.
1. Introduction The burden of assessment in the United States centers on standardized tests. Unfortunately, these tests frequently do not give a full picture of what students know about the complexity of science [39]. Research indicates that students tend to pass science tests, but often are not able to understand larger concepts, which typically are not assessed on multiple-choice tests [20]. Indeed, students are often assessed on whether they understand isolated vocabulary words such as ‘‘hypothesis’’, while in-depth assessment of their abilities to formulate hypotheses and design experiments is neglected [24]. Further, a Carnegie report [3] suggests that the current testing system in the United States focuses heavily on assessing knowledge and interpretation to the detriment of scientific inquiry topics. In an attempt to address these challenges, a growing number of researchers are turning to immersive virtual environments (IVEs). These game-like virtual worlds are able to situate science inquiry and content problems in authentic contexts for students to solve (e.g. [2,29,26]. Research is emerging on the question of how well IVEs can be used as a viable platform for science assessments. By following a systematic, theory-based approach to designing curricula and the activities of learning within those curricula, IVEs can produce data from students that more fully and validly demonstrate their evolving levels of competency around science inquiry and concepts [28]. Well-designed IVE-based assessments produce a steady stream of data to students and teachers, giving both groups new insights into student understanding and application of inquiry and content over time [35,27]. For example, Shute et al. [35] explore the idea of conjoining game-based IVEs with embedded assessments to create what they label ‘‘stealth’’ formative assessments. Shute and her colleagues argue that player interactions in an IVE can be assessed in real-time using probability analysis techniques. Students can be continuously and invisibly assessed as they work through series of challenging tasks situated seamlessly into game play and narrative [5]. The sum of
⇑ Corresponding author. Address: Brickyard Engineering (BYENG) M1-04, 699 S. Mill Ave., Tempe, AZ 85281, USA. Tel.: +1 480 965 0383. E-mail address: [email protected] (B.C. Nelson). 0020-0255/$ - see front matter Ó 2013 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.ins.2013.09.011
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
33
these interactions over the course of an assessment adds up to meaningful evidentiary records of understanding of the content and processes taught in the IVE. 1.1. SAVE Science SAVE Science is a 5-year study funded by the National Science Foundation centered on creating and evaluating an innovative IVE-based system for assessment of learning in science. SAVE Science is designing and implementing a series of virtual environment modules for assessing both science content and inquiry in 7th and 8th grades. The modules make use of a novel assessment rubric based on student interactions within an authentic context-based science curriculum, embedded in a virtual environment. We hypothesize that doing so will enable us to capture and analyze different patterns of scientific understanding amongst the students in a classroom. During the first 4 years of the SAVE Science project, we have designed four assessment modules, two introductory modules, and a dashboard for teachers and researchers that puts teachers in charge of student sign-up and gives them access to their students’ data. Results from implementations with approximately 2000 students, 15 teachers at 12 schools indicate that SAVE Science assessments: Provide valuable information about misconceptions students have about specific science ideas, even after having studied them in class [11]. Offer useful information for teachers that they cannot see through traditional testing methods [25]. Offer contextual clues that aid students in applying their content learning in solving problems [12]. Provide opportunities for students to ‘‘feel present’’ in the IVE, raising engagement and opportunities to show evidence of learning [34]. A major sub-focus of SAVE Science centers on conducting controlled and exploratory studies to investigate design approaches aimed at helping students manage the high cognitive load experienced while conducting assessment activities embedded in virtual environments. By reducing the perceived complexity of IVE-based assessments, we hypothesize that students will be better able to attend to the processes and tasks associated with the assessments, leading to more accurate assessment evidentiary data. To explore this hypothesis, we are investigating the use of design principles drawn from cognitive processing literature shown to support learning in presentational multimedia learning environments [25]. In this paper, we present the results from a quasi-experimental study into the use of the signaling principle in designing IVE-based assessments. This principle states that people learn better when the design of multimedia materials employs visual or auditory cues that highlight the organization of essential material to be learned [18,31,32]. This is accomplished in part by reducing the extraneous cognitive load learners are thought to experience when such signals are used and by directing the learner to important material. For example, Rey [33] found that college students (N = 113) who received instruction enhanced by visual signals outperformed those who received instruction without signaling, with higher retention scores (Cohen’s d = .04) and statistically significant higher transfer scores (F(1, 109) = 4.61, p < .05; Cohen’s d = .41). Another application of the signaling principle explored by Moreno et al. (2001) [21] was the use of an animated arrow or the deictic movements of a pedagogical agent compared to a control group with no signaling prompts. They found that both signaling conditions outperformed the control condition, with the pedagogical agent signaling group experiencing higher posttest scores (p = .002). To explore whether the signaling principle can have a similar effect when applied to the design of an immersive virtual world, we conducted a study with middle school students in which we focused on the use of visual signaling techniques to reduce perceived student cognitive load in the SAVE Science virtual world while simultaneously increasing the number of times students click on or collide with assessment-relevant objects in the virtual world (assessment efficiency). 2. Theory 2.1. Cognitive load theory and signaling Cognitive load theory [41] is based on the assumptions that working memory has a limited capacity, that learning is the process of storing information in long-term memory in the form of schemas, that these schemas automate the process of retrieving and responding to information, and that learning requires active cognitive processing. Cognitive load refers to the cognitive demands or mental effort a given task imposes on the learner. Sweller et al. [42] break down cognitive load into three types, or what they term the consequences, of cognitive load: intrinsic load, extrinsic load, and germane load. Intrinsic load is the cognitive demand inherent in the task itself, while extrinsic load are the demands imposed by the learning environment itself, e.g.: extraneous or irrelevant information. Germane load [42] is cognitive load that is associated with processing information, constructing schema, and developing automation of skills. Germane cognitive load facilitates the achievement of an instructional goal by enhancing the processing of information or aiding in schema construction. One of the goals of effective instructional design is to increase germane load while reducing extrinsic load. According to Sweller [40] cognitive load is present when there is a high level of element interactivity, intrinsically and extrinsically. The intrinsic element interactivity refers to the amount of information that a student must understand in order
34
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
to learn subsequent concepts; this complexity is inherent to content that cannot be learned in isolation. Extrinsic element interactivity refers to the complexity imposed by instructional procedures or the learning environment. Immersive virtual environments are inherently complex by design, and this complexity can lead to additional extrinsic cognitive load for the student. The active processing assumption of cognitive load theory states that for meaningful learning to occur, learners need to engage in the active cognitive processing of information. This active processing occurs when learners select, organize, and integrate information [16]. Mayer [17] explained that when cognitively processing images, learners pay attention to elements of the images they are seeing and they select aspects of these images to be processed in working memory. Once in working memory, the elements of the image will need to be organized and arranged by the learner to form coherent mental representations. The mental representations that have been constructed in the working memory of the learner are then integrated with their current schemata that include long-term memories and prior knowledge. By helping learners select, organize, and integrate information, designers of instruction are able to influence the amount of cognitive load that is experienced by learners. One way that germane load is increased while extrinsic load is reduced is by directing the learner’s efforts to important aspects of the instruction. Visual signaling is one approach used to help learners to select important information and to direct their attention to relevant objects and locations in a learning or assessment environment (Sweller, 2005). Mayer [19] examined six empirical studies using eye-tracking tools to understand how students process learning tasks, and found a consistent link between learning gains and the use of the signaling principle. In a study by Morozov [23] where the impact on cognitive load of visual signaling in the form of visual maps were investigated, he found that while the maps did not increase the learning, it did not hinder it and it enhanced navigation. Morozov points to the idea that disorientation can be a symptom of cognitive overload and can be reduced by effective use of signaling via navigation maps. In another study by Chen and Fauzy [4], the authors found that the use of visual signaling techniques such as traces and directional arrows increased germane load, and translated to significant positive learning effects. de Koning et al. [7] studied the use of visual signals and cues in animations using eye tracking and self report mental effort scales. It was found that learners looked longer and more often at contents that was signaled. This indicates that the signaling helped guide learners’ attention to the general area of the contents, and it can be used to determine when elements are processed and in what order. Finally, Wouters et al. [44] put forth a set of guidelines to optimize learning and minimize extraneous cognitive load, among these guidelines is the suggestion to use signals to direct attention to important parts of instruction through the use of signaling. The term visual search indicates the level of difficulty that is involved in learners cognitively processing visual materials. This is commonly impacted by the number of distinct objects in the visual, how compact or scattered are distinct objects, and the complexity of the visual. Visuals that require high visual search often require learners to view and process many different visual objects, thus requiring more cognitive resources to select, organize, integrate, and process the information contained in the visual. Visuals that require a low amount of visual search are easier for learners to cognitively process, require less mental resources and thus impose less extraneous cognitive load. Visual signals are visual elements that can be added to materials and environments that are designed to help direct users attention to specific elements in the visual field. Mautone and Mayer [16] stated that these signals can be used to help individuals select, organize, integrate, and process information. For example, arrows can be added to visual materials to help users determine the most important information to attend to in the visual field. Flashing text can also be added to direct user’s attention to specific aspects of a text. Jeung et al. [10] found that adding visual signals or indicators to images with high visual search was beneficial to learning; while visual signals added to low search environments had little to no effect on learning. In their 2007 study, de Koning et al. discussed the need for additional research regarding visual search in order to understand the effects of signaling or cueing on attention allocation. Their 2010 paper [7] hypothesizes that the addition of visual cues to an animation would reduce the amount of visual search and extraneous cognitive load. Both of these hypotheses could not be confirmed, and may have been a result of the spotlight type visual cueing that were used in the study. The SAVE Science virtual environments aim to address extrinsic cognitive load issues by using a minimalist approach, presenting student with only the most essential of elements needed to perform the assessments. Reducing the extraneous load imposed by the learning environment is an important aspect of managing cognitive load, but studies suggest that it is equally important to create learning experiences purposely designed to increase germane load. In a multidisciplinary review Ayres and Paas [1] looked at the biological underpinnings for the cognitive load theory and concluded that task complexity and realistic learning environments are essential components for learning. The use of immersive virtual worlds to assess knowledge is a way to support this concept, and the addition of visual signals to some treatments is another method used to foster germane cognitive load. 2.2. Beyond static images and animations Many of the studies regarding cognitive load were conducted using static images and text [10,16,43]. The principles and concepts learned from these studies were then extended to the study of animations [6,7,15,22], with generally similar findings (lower perceived cognitive load and improved learning outcomes). Few researchers, however, have investigated the viability of multimedia and cognitive load principles to reduce perceived cognitive load in virtual worlds and games [14,27,36]. It is theorized that the higher amounts of interactivity present in games and virtual worlds may impact the ways that
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
35
learners select, organize, and integrate information in addition to how they deal with cognitive load. The complexity of the concepts and the learning environment may be a deciding factor when including visual signals in a learning environment. Goldman [9] held that novice learners may benefit from cues that help them identify and interpret important information when dealing with complex concepts or dynamic processes. That is, the affordances of the learning environment should match the demands of the tasks students are expected to master or demonstrate. In two related studies, visual signals were added to an immersive physics education computer game to determine differences in perceived mental effort between students who received the visual signals and those who did not. No significant differences in perceived mental effort were found between those who received the visual signaling and those who did not. One reason for this finding may have been the low amount of visual search required by the game [36–38]. 2.3. Hypotheses and research questions Our specific hypotheses and associated research questions in the current study were: H1. Visual signaling applied to the design of a virtual world-based assessment module reduces a student’s perceived extraneous cognitive load.
RQ1. In terms of student perceived cognitive load, how does the use of visual signaling incorporated into a SAVE Science assessment module compare to a version of the assessment module not designed to incorporate this design principle? H2. Visual signaling applied to the design of a virtual world-based assessment module increases the efficiency of the assessment. RQ2. In terms of assessment efficiency, how does the use of visual signaling incorporated into a SAVE Science assessment module compare to a version of the assessment module not designed to incorporate this design principle?
3. Materials and methods In the study described here, we focused on the use of visual signaling techniques to reduce perceived student cognitive load in the SAVE Science virtual world while simultaneously increasing assessment efficiency – the number of interactions a student has with objects embedded in the virtual world. For the study, we designed interactive objects in the ‘‘Sheep Trouble’’ assessment IVE containing information related to the assessment tasks with symbols used to indicate that students could interact with them (Figs. 1 and 2). In this particular assessment module, students are able to interact with a limited set of objects, including two ‘‘Non Player Characters’’ (NPCs) and a large number of sheep scattered around a farmyard. Visual signaling was used to foreground these objects to students, increasing the likelihood (we hypothesized) that students would interact with them and reducing perceived extraneous
Fig. 1. Sheep signaling.
36
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
Fig. 2. NPC signaling.
cognitive load by focusing attention away from visually appealing but extraneous details in the situated context of the virtual world. 3.1. Conditions For the purposes of the study, two different conditions of the virtual world-based assessment were developed: Visual signaling and non-visual signaling. The visual signaling condition showed 3D symbols hovering above assessment-related objects that indicated the interactive status of the objects. Prior to a student interacting with a signaled object, it displayed a red question mark (Fig. 1). Once the students interacted with a signaled object for collecting or observing information related to the object, the symbol changed to green exclamation mark to indicate that that object had been viewed. For the non-signaling condition, all of the symbols were removed, but the assessment stayed same as in the visual signaling condition. In both conditions, students were expected to solve the problem presented to them and provide enough empirical data to justify their conclusions arrived at through the exploration of the virtual environment and through interactions with characters, sheep, and other in-world objects. 3.2. Participants A total of 221 middle schools participants (7th grade) were recruited from four schools, from two districts in a mid-Atlantic state. Roughly equal numbers of boys and girls took part in the study. Of the original participants, 193 completed the assessment and post-survey resulting in a sample of 193 students for the study. From the participating students, 98 students were randomly assigned to the virtual world with visual signaling and the other 95 students were assigned to the virtual world without visual signaling. 3.3. Environment Sheep Trouble assesses student understanding of concepts of adaptation and structure/function that underlie beginning speciation. The module is designed so that by interacting with farmers and sheep on a virtual farm, students gain the contextual understanding they need to apply their classroom learning to more meaningful assessment tasks. In Sheep Trouble, students enter a medieval-like world (Figs. 1 and 2) where they are met by a farmer who asks them to help him find a scientific explanation for why his recently imported herd of sheep is in poor health. Students discover that many of the townsfolk think that the new sheep are sick due to ‘‘bad magic’’ and that the new sheep must be destroyed quickly before the bad magic spreads to the local sheep. The farmer asks the student-scientists to apply their skills to suggest possible science-based contributors to the new sheep’s poor condition. Students are given virtual tools for data collection and can also observe visual clues about the environment itself. For example, a poster on the farm explains that the new sheep come from a very different geographic locale (a flat, snowy island) from the current farm (hilly, rocky, and dry). Students use a question and answer system to communicate with a farmer and his brother on the farm. They can also interact with a large number of the new and local sheep scattered around the farmyard. The local ‘‘original’’ sheep (sheep from stock that have been raised locally for centuries) are all healthy. Students can use virtual rulers to measure the sheep’s legs, body length, and
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
37
ears, and can access information on sheep weight loss or gain, age and gender. By applying their knowledge of scientific inquiry and of adaptation through exploration of the virtual world, students can discover that to get to the best grass, the new sheep need to climb a hill. Students gather evidence and can reach the conclusion that the newly imported sheep are unable to adapt to climb steep hills for food, due to their structural differences. As a result, they are failing to thrive. Students collect data, view and interpret the data, and report their findings to the Farmer to complete their assessment ‘quest’.
3.4. Procedure Participants completed a one class period virtual world-based assessment using the ‘‘Sheep Trouble’’ module, designed to measure their understanding of the concept of an organism adapting to a given environment over time. This topic had been taught as part of the regular in-class curriculum just prior to the participants taking the virtual world-based assessment. After completing the assessment, participants completed a survey containing a number of items related to science interest, game play experience, immersion, and perceived cognitive load. For the current sub-study under the larger SAVE Science umbrella of studies, we are examining only the cognitive load survey questions.
3.5. Instruments 3.5.1. Cognitive load survey After completing the IVE-based assessment module, students answered seven questions on a web-based survey related to perceived cognitive load (adopted from [30]). Paas and Van Merrienboer [30] reported an internal consistency reliability for the measure of .82. The perceived cognitive load measure, or subjective rating scale [42], is based on the assumption of people’s ability for accurate introspection on their cognitive processes and reporting on the amount of mental effort experienced while completing instructional tasks. In our study, we used the self-report cognitive load measure rather than measuring cognitive load through eye-tracking software and hardware primarily for convenience of implementation in a classroom setting. The self-report survey used in this study consists of 10 point scale Likert-style questions contained items such as ‘‘How hard did you have to work to find the things in Scientopolis you wanted to see or interact with?’’, ‘‘How much effort did you have to invest in order to navigate in Scientopolis?’’, or ‘‘How hard did you have to work to communicate with the people you met in Scientopolis?’’
3.5.2. Assessment efficiency As we have described, for purposes of this study we defined efficiency as the number of interactions with assessmentrelated objects completed during the assessment. We measured: total number of ‘‘collisions’’: the number of times a student walks up to an in-world object to interact with it (triggering interaction windows), total number of collisions with sheep: collisions with sheep only, other collisions: collisions with NPCs or poster, total number of measurements: total number of measurements taken while interacting with sheep, measure per sheep: average number of measurements per sheep, total number of records in clipboard: number of records recorded in an electronic clipboard, and clipboard per sheep: average number of records recorded per sheep.
4. Results A one-way analysis of variance (ANOVA) was conducted to assess the effect of visual signaling on perceived cognitive load and on efficiency of the IVE-based assessment. The factor of signaling treatment included two levels: visual signaling group and non-visual signaling group. For perceived cognitive load, we found a statistically significant difference for the combined survey means with an alpha level of .05, F(1, 175.97) = 4.27, p = .04, d = .29, with students in the visual signaling group reporting lower average cognitive load than those in non-visual signaling group (see Table 1 and Fig. 3). For individual aspects of cognitive load, the effect of visual signaling was significant on the question ‘‘How hard did you have to work to communicate with people you met in Scientopolis?’’, F(1, 151.81) = 5.97, p = .02, d = .31. Weakly significant differences were seen on two questions ‘‘How hard did you have to work to find things in Scientopolis you wanted to interact with?’’ and ‘‘How much effort did you have to invest in order to navigate in Scientopolis?’’, F(1, 176.94) = 2.92, p = .09 and F(1, 191) = 3.37, p = .07 respectively. In all cases, students in the visual signaling group reported lower mean levels of perceived cognitive load. The means and standard deviations of these items for both groups are reported in Table 1. On the efficiency question, results showed that the effect of visual signaling was significant for four types of interactions with assessment-related objects at an alpha level of .05: Total collisions with all possible in-world objects, F(1, 191) = 5.57, p = .02, Cohen’s d = .34, Total collisions with sheep, F(1, 191) = 7.09, p < .01, d = .38, Total number of measurements of sheep taken, F(1, 183.96) = 13.63, p < .001, d = .51, and Total number of records entered into an electronic clipboard, F(1, 187.56) = 11.68, p = .001, d = .48. Average values of these interaction measures for visual signaling group were higher than those for non-visual signaling group (Table 2 and Fig. 4).
38
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
Table 1 Means and standard deviations of cognitive load survey items. Conditions
Cognitive load survey items 50
Visual signaling Non-visual signaling
a
52
b
54
c
Combined
M
SD
M
SD
M
SD
M
SD
3.56 4.20
2.24 2.90
3.24 3.98
2.57 2.96
2.40 3.25
1.75 2.95
24.81 28.35
10.20 13.34
Note: Subjects were asked to indicate the degree of perceived effort with each of survey items. Ratings were made using 1–10 Likert-type scale, in which 1 = ‘‘not much effort’’ to 10 = ‘‘a lot of effort’’. a How hard did you have to work to find the things in Scientopolis you wanted to see or interact with? b How much effort did you have to invest in order to navigate in Scientopolis (e.g. for deciding between different buttons or keys, finding your way around)? c How are did you have to work to communicate with the people you met in Scientopolis?
Fig. 3. Perceived cognitive load by treatment.
Table 2 Means and standard deviations of interactions with assessment-related objects. Conditions
Visual signaling Non-visual signaling
Interactions with assessment-related objects Total number of collisions
Total number of collisions with sheep
Total number of measurements
Total number of records in clipboard
M
SD
M
SD
M
SD
M
SD
21.04 18.31
8.24 7.86
14.21 12.15
4.88 5.88
44.43 33.01
23.82 18.93
40.54 30.76
21.50 18.18
5. Discussion and conclusion Our findings from this study support the hypotheses that use of visual signaling can reduce a student’s overall perceived cognitive load while completing a virtual world-based assessment, and that visual signaling can increase the efficiency of IVE-based assessments as measured by numbers of interactions with relevant materials. The findings for perceived cognitive load are not particularly strong, however, with only marginally significant differences between groups for perceived cognitive load related to finding interactive objects and navigating the virtual world. The weak mean differences between groups for finding objects and navigating the world may reflect the fact that the world itself was designed to contain very few interactive objects and contained a limited amount of virtual space to explore. The low number of objects in the world limited the amount of visual search that was required of learners in the environment. As we have described, low amounts of visual search have been found to reduce the amount of cognitive load experienced by learners. In future studies, we will explore the use of visual (and auditory) signaling in more visually complex, ‘‘high search’’ virtual environments. It is interesting to note that there were relatively strong differences between reported effort in communicating with characters in the virtual world between signaling and non-signaling groups, particularly given that there are only two characters in the virtual world. It is not clear from the data collected why use of signaling should impact interaction rates with in-world
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
39
Fig. 4. Object interactions by treatment.
characters despite the low complexity of the world and few characters present. It may be that the presence of visual signaling is acting as a kind of ‘umbrella signal’ that results in an overall increased likelihood of students interacting with all objects with signals. The differences between groups in terms of assessment efficiency as shown by total interactions (aka ‘collisions’ and mouse clicks) with interactive objects were quite strong. Use of visual signaling in a virtual world-based assessment was clearly demonstrated to be associated with increased efficiency of completing assessment tasks. Strongly significant mean differences were seen between groups in total number of measurements taken while completing the assessment, and total number of records saved into the electronic clipboard. From a virtual world-based assessment design perspective, these findings provide powerful incentive to include visual signaling as a way to help ensure that students are able to find and use the tools that enable them to provide evidence of their level of understanding of the concepts being assessed. If assessments are embedded into virtual worlds, it is critical that the design of the worlds supports them in completing the tasks of assessment, so that valid and reliable inferences can be made from the tasks performed. In our own work, the strong findings from this study related to object interaction rates led our team to include visual signaling in all subsequently created assessment modules. Despite the demonstrated benefits of visual signaling in virtual world-based assessments, questions remain. One of the most challenging is whether the use of visual signaling in a virtual world-based curriculum reduces the learning power of, and participant engagement with, the world. If, as researchers have purported (e.g. [5,8,13,2,29,26]), the power for learning in virtual worlds is centered on open-ended inquiry situated in realistic contexts, does the use of visual signaling reduce that power? And if unguided discovery of, and interaction with, embedded objects in a virtual world is central to the learning power of virtual worlds, does use of signaling rob a student of that experience? We suspect that signaling does have some deleterious impact on the overall learner immersion and embodiment in a virtual world, but only when used ‘too much’. The challenge is to discover the appropriate level of signaling that will reduce extraneous cognitive load and increase efficiency of learning, while simultaneously maintaining the sense of immersion and open-ended exploration that are hallmarks of virtual worlds. Achieving two possibly contradictory goals is a tricky balancing act, but one that may be key to the successful and sustained use of virtual worlds as educational tools. Acknowledgments The study discussed in this article was conducted under the umbrella of the SAVE Science research project. Our sincere thanks to the SAVE Science teams at University of Maryland and Temple University, and its leadership team: PI Diane Jass Ketelhut and co-PI Catherine Schifter. References [1] P. Ayres, F. Paas, Interdisciplinary perspectives inspiring a new generation of cognitive load research, Educational Psychology Review 21 (2008) 1–9. [2] S. Barab, A. Arici, C. Jackson, Eat your vegetables and do your homework: a design based investigation of enjoyment and meaning in learning, Educational Technology 45 (1) (2005) 15–20.
40
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
[3] Carnegie Corporation, The Opportunity Equation: Transforming Mathematics and Science Education for Citizenship and the Global Economy, Carnegie Corporation of New York, New York, 2009. [4] C.J. Chen, W.M.W.I. Fauzy, Guiding exploration through three-dimensional virtual environments: a cognitive load reduction approach, Journal of Interactive Learning Research 19 (4) (2008) 579–596. [5] D. Clark, B. Nelson, P. Sengupta, C. D’Angelo, Rethinking Science Learning through Digital Games and Simulations: Genres, Examples, and Evidence, An NAS Commissioned Paper, 2009.. [6] B.B. de Koning, H.K. Tabbers, R.M.J.P. Rikers, F. Paas, Attention cueing as a means to enhance learning from an animation, Applied Cognitive Psychology 21 (6) (2007) 731–746, http://dx.doi.org/10.1002/acp.1346. [7] B.B. de Koning, H.K. Tabbers, R.M.J.P. Rikers, F. Paas, Attention guidance in learning from a complex animation: seeing is understanding?, Learning and Instruction 20 (2) (2010) 111–122 [8] J.P. Gee, D.W. Shaffer, Looking Where the Light is Bad: Video Games and the Future of Assessment, (Epistemic Games Group Working Paper No. 2010– 02), University of Wisconsin-Madison, Madison, 2010.
Contents lists available at ScienceDirect
Information Sciences journal homepage: www.elsevier.com/locate/ins
Visual signaling in virtual world-based assessments: The SAVE Science project Brian C. Nelson ⇑, Younsu Kim, Cecile Foshee, Kent Slack Arizona State University, Brickyard Engineering, 699 S. Mill Ave., Tempe, AZ 85281, USA
a r t i c l e
i n f o
Article history: Available online 8 September 2013 Keywords: Virtual world Instructional design Game-based learning Assessment Science education
a b s t r a c t In this paper we describe a study into the impact of visual signaling techniques used in a virtual world-based assessment of science inquiry and content on (1) student cognitive load and (2) assessment efficiency. The study, run with 7th grade students in the United States, found that use of visual signaling was significantly associated with lower levels of student self-reported cognitive load versus students in a no-signaling version of the assessment. Further, the efficiency of the virtual world-based assessment was significantly higher, as measured by in-world object interaction rates, for students in the visual signaling version of the assessment than for those in the no-signaling treatment. In the paper, we discuss the results and their meaning for the design of virtual world and game-based assessments. Ó 2013 Elsevier Inc. All rights reserved.
1. Introduction The burden of assessment in the United States centers on standardized tests. Unfortunately, these tests frequently do not give a full picture of what students know about the complexity of science [39]. Research indicates that students tend to pass science tests, but often are not able to understand larger concepts, which typically are not assessed on multiple-choice tests [20]. Indeed, students are often assessed on whether they understand isolated vocabulary words such as ‘‘hypothesis’’, while in-depth assessment of their abilities to formulate hypotheses and design experiments is neglected [24]. Further, a Carnegie report [3] suggests that the current testing system in the United States focuses heavily on assessing knowledge and interpretation to the detriment of scientific inquiry topics. In an attempt to address these challenges, a growing number of researchers are turning to immersive virtual environments (IVEs). These game-like virtual worlds are able to situate science inquiry and content problems in authentic contexts for students to solve (e.g. [2,29,26]. Research is emerging on the question of how well IVEs can be used as a viable platform for science assessments. By following a systematic, theory-based approach to designing curricula and the activities of learning within those curricula, IVEs can produce data from students that more fully and validly demonstrate their evolving levels of competency around science inquiry and concepts [28]. Well-designed IVE-based assessments produce a steady stream of data to students and teachers, giving both groups new insights into student understanding and application of inquiry and content over time [35,27]. For example, Shute et al. [35] explore the idea of conjoining game-based IVEs with embedded assessments to create what they label ‘‘stealth’’ formative assessments. Shute and her colleagues argue that player interactions in an IVE can be assessed in real-time using probability analysis techniques. Students can be continuously and invisibly assessed as they work through series of challenging tasks situated seamlessly into game play and narrative [5]. The sum of
⇑ Corresponding author. Address: Brickyard Engineering (BYENG) M1-04, 699 S. Mill Ave., Tempe, AZ 85281, USA. Tel.: +1 480 965 0383. E-mail address: [email protected] (B.C. Nelson). 0020-0255/$ - see front matter Ó 2013 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.ins.2013.09.011
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
33
these interactions over the course of an assessment adds up to meaningful evidentiary records of understanding of the content and processes taught in the IVE. 1.1. SAVE Science SAVE Science is a 5-year study funded by the National Science Foundation centered on creating and evaluating an innovative IVE-based system for assessment of learning in science. SAVE Science is designing and implementing a series of virtual environment modules for assessing both science content and inquiry in 7th and 8th grades. The modules make use of a novel assessment rubric based on student interactions within an authentic context-based science curriculum, embedded in a virtual environment. We hypothesize that doing so will enable us to capture and analyze different patterns of scientific understanding amongst the students in a classroom. During the first 4 years of the SAVE Science project, we have designed four assessment modules, two introductory modules, and a dashboard for teachers and researchers that puts teachers in charge of student sign-up and gives them access to their students’ data. Results from implementations with approximately 2000 students, 15 teachers at 12 schools indicate that SAVE Science assessments: Provide valuable information about misconceptions students have about specific science ideas, even after having studied them in class [11]. Offer useful information for teachers that they cannot see through traditional testing methods [25]. Offer contextual clues that aid students in applying their content learning in solving problems [12]. Provide opportunities for students to ‘‘feel present’’ in the IVE, raising engagement and opportunities to show evidence of learning [34]. A major sub-focus of SAVE Science centers on conducting controlled and exploratory studies to investigate design approaches aimed at helping students manage the high cognitive load experienced while conducting assessment activities embedded in virtual environments. By reducing the perceived complexity of IVE-based assessments, we hypothesize that students will be better able to attend to the processes and tasks associated with the assessments, leading to more accurate assessment evidentiary data. To explore this hypothesis, we are investigating the use of design principles drawn from cognitive processing literature shown to support learning in presentational multimedia learning environments [25]. In this paper, we present the results from a quasi-experimental study into the use of the signaling principle in designing IVE-based assessments. This principle states that people learn better when the design of multimedia materials employs visual or auditory cues that highlight the organization of essential material to be learned [18,31,32]. This is accomplished in part by reducing the extraneous cognitive load learners are thought to experience when such signals are used and by directing the learner to important material. For example, Rey [33] found that college students (N = 113) who received instruction enhanced by visual signals outperformed those who received instruction without signaling, with higher retention scores (Cohen’s d = .04) and statistically significant higher transfer scores (F(1, 109) = 4.61, p < .05; Cohen’s d = .41). Another application of the signaling principle explored by Moreno et al. (2001) [21] was the use of an animated arrow or the deictic movements of a pedagogical agent compared to a control group with no signaling prompts. They found that both signaling conditions outperformed the control condition, with the pedagogical agent signaling group experiencing higher posttest scores (p = .002). To explore whether the signaling principle can have a similar effect when applied to the design of an immersive virtual world, we conducted a study with middle school students in which we focused on the use of visual signaling techniques to reduce perceived student cognitive load in the SAVE Science virtual world while simultaneously increasing the number of times students click on or collide with assessment-relevant objects in the virtual world (assessment efficiency). 2. Theory 2.1. Cognitive load theory and signaling Cognitive load theory [41] is based on the assumptions that working memory has a limited capacity, that learning is the process of storing information in long-term memory in the form of schemas, that these schemas automate the process of retrieving and responding to information, and that learning requires active cognitive processing. Cognitive load refers to the cognitive demands or mental effort a given task imposes on the learner. Sweller et al. [42] break down cognitive load into three types, or what they term the consequences, of cognitive load: intrinsic load, extrinsic load, and germane load. Intrinsic load is the cognitive demand inherent in the task itself, while extrinsic load are the demands imposed by the learning environment itself, e.g.: extraneous or irrelevant information. Germane load [42] is cognitive load that is associated with processing information, constructing schema, and developing automation of skills. Germane cognitive load facilitates the achievement of an instructional goal by enhancing the processing of information or aiding in schema construction. One of the goals of effective instructional design is to increase germane load while reducing extrinsic load. According to Sweller [40] cognitive load is present when there is a high level of element interactivity, intrinsically and extrinsically. The intrinsic element interactivity refers to the amount of information that a student must understand in order
34
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
to learn subsequent concepts; this complexity is inherent to content that cannot be learned in isolation. Extrinsic element interactivity refers to the complexity imposed by instructional procedures or the learning environment. Immersive virtual environments are inherently complex by design, and this complexity can lead to additional extrinsic cognitive load for the student. The active processing assumption of cognitive load theory states that for meaningful learning to occur, learners need to engage in the active cognitive processing of information. This active processing occurs when learners select, organize, and integrate information [16]. Mayer [17] explained that when cognitively processing images, learners pay attention to elements of the images they are seeing and they select aspects of these images to be processed in working memory. Once in working memory, the elements of the image will need to be organized and arranged by the learner to form coherent mental representations. The mental representations that have been constructed in the working memory of the learner are then integrated with their current schemata that include long-term memories and prior knowledge. By helping learners select, organize, and integrate information, designers of instruction are able to influence the amount of cognitive load that is experienced by learners. One way that germane load is increased while extrinsic load is reduced is by directing the learner’s efforts to important aspects of the instruction. Visual signaling is one approach used to help learners to select important information and to direct their attention to relevant objects and locations in a learning or assessment environment (Sweller, 2005). Mayer [19] examined six empirical studies using eye-tracking tools to understand how students process learning tasks, and found a consistent link between learning gains and the use of the signaling principle. In a study by Morozov [23] where the impact on cognitive load of visual signaling in the form of visual maps were investigated, he found that while the maps did not increase the learning, it did not hinder it and it enhanced navigation. Morozov points to the idea that disorientation can be a symptom of cognitive overload and can be reduced by effective use of signaling via navigation maps. In another study by Chen and Fauzy [4], the authors found that the use of visual signaling techniques such as traces and directional arrows increased germane load, and translated to significant positive learning effects. de Koning et al. [7] studied the use of visual signals and cues in animations using eye tracking and self report mental effort scales. It was found that learners looked longer and more often at contents that was signaled. This indicates that the signaling helped guide learners’ attention to the general area of the contents, and it can be used to determine when elements are processed and in what order. Finally, Wouters et al. [44] put forth a set of guidelines to optimize learning and minimize extraneous cognitive load, among these guidelines is the suggestion to use signals to direct attention to important parts of instruction through the use of signaling. The term visual search indicates the level of difficulty that is involved in learners cognitively processing visual materials. This is commonly impacted by the number of distinct objects in the visual, how compact or scattered are distinct objects, and the complexity of the visual. Visuals that require high visual search often require learners to view and process many different visual objects, thus requiring more cognitive resources to select, organize, integrate, and process the information contained in the visual. Visuals that require a low amount of visual search are easier for learners to cognitively process, require less mental resources and thus impose less extraneous cognitive load. Visual signals are visual elements that can be added to materials and environments that are designed to help direct users attention to specific elements in the visual field. Mautone and Mayer [16] stated that these signals can be used to help individuals select, organize, integrate, and process information. For example, arrows can be added to visual materials to help users determine the most important information to attend to in the visual field. Flashing text can also be added to direct user’s attention to specific aspects of a text. Jeung et al. [10] found that adding visual signals or indicators to images with high visual search was beneficial to learning; while visual signals added to low search environments had little to no effect on learning. In their 2007 study, de Koning et al. discussed the need for additional research regarding visual search in order to understand the effects of signaling or cueing on attention allocation. Their 2010 paper [7] hypothesizes that the addition of visual cues to an animation would reduce the amount of visual search and extraneous cognitive load. Both of these hypotheses could not be confirmed, and may have been a result of the spotlight type visual cueing that were used in the study. The SAVE Science virtual environments aim to address extrinsic cognitive load issues by using a minimalist approach, presenting student with only the most essential of elements needed to perform the assessments. Reducing the extraneous load imposed by the learning environment is an important aspect of managing cognitive load, but studies suggest that it is equally important to create learning experiences purposely designed to increase germane load. In a multidisciplinary review Ayres and Paas [1] looked at the biological underpinnings for the cognitive load theory and concluded that task complexity and realistic learning environments are essential components for learning. The use of immersive virtual worlds to assess knowledge is a way to support this concept, and the addition of visual signals to some treatments is another method used to foster germane cognitive load. 2.2. Beyond static images and animations Many of the studies regarding cognitive load were conducted using static images and text [10,16,43]. The principles and concepts learned from these studies were then extended to the study of animations [6,7,15,22], with generally similar findings (lower perceived cognitive load and improved learning outcomes). Few researchers, however, have investigated the viability of multimedia and cognitive load principles to reduce perceived cognitive load in virtual worlds and games [14,27,36]. It is theorized that the higher amounts of interactivity present in games and virtual worlds may impact the ways that
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
35
learners select, organize, and integrate information in addition to how they deal with cognitive load. The complexity of the concepts and the learning environment may be a deciding factor when including visual signals in a learning environment. Goldman [9] held that novice learners may benefit from cues that help them identify and interpret important information when dealing with complex concepts or dynamic processes. That is, the affordances of the learning environment should match the demands of the tasks students are expected to master or demonstrate. In two related studies, visual signals were added to an immersive physics education computer game to determine differences in perceived mental effort between students who received the visual signals and those who did not. No significant differences in perceived mental effort were found between those who received the visual signaling and those who did not. One reason for this finding may have been the low amount of visual search required by the game [36–38]. 2.3. Hypotheses and research questions Our specific hypotheses and associated research questions in the current study were: H1. Visual signaling applied to the design of a virtual world-based assessment module reduces a student’s perceived extraneous cognitive load.
RQ1. In terms of student perceived cognitive load, how does the use of visual signaling incorporated into a SAVE Science assessment module compare to a version of the assessment module not designed to incorporate this design principle? H2. Visual signaling applied to the design of a virtual world-based assessment module increases the efficiency of the assessment. RQ2. In terms of assessment efficiency, how does the use of visual signaling incorporated into a SAVE Science assessment module compare to a version of the assessment module not designed to incorporate this design principle?
3. Materials and methods In the study described here, we focused on the use of visual signaling techniques to reduce perceived student cognitive load in the SAVE Science virtual world while simultaneously increasing assessment efficiency – the number of interactions a student has with objects embedded in the virtual world. For the study, we designed interactive objects in the ‘‘Sheep Trouble’’ assessment IVE containing information related to the assessment tasks with symbols used to indicate that students could interact with them (Figs. 1 and 2). In this particular assessment module, students are able to interact with a limited set of objects, including two ‘‘Non Player Characters’’ (NPCs) and a large number of sheep scattered around a farmyard. Visual signaling was used to foreground these objects to students, increasing the likelihood (we hypothesized) that students would interact with them and reducing perceived extraneous
Fig. 1. Sheep signaling.
36
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
Fig. 2. NPC signaling.
cognitive load by focusing attention away from visually appealing but extraneous details in the situated context of the virtual world. 3.1. Conditions For the purposes of the study, two different conditions of the virtual world-based assessment were developed: Visual signaling and non-visual signaling. The visual signaling condition showed 3D symbols hovering above assessment-related objects that indicated the interactive status of the objects. Prior to a student interacting with a signaled object, it displayed a red question mark (Fig. 1). Once the students interacted with a signaled object for collecting or observing information related to the object, the symbol changed to green exclamation mark to indicate that that object had been viewed. For the non-signaling condition, all of the symbols were removed, but the assessment stayed same as in the visual signaling condition. In both conditions, students were expected to solve the problem presented to them and provide enough empirical data to justify their conclusions arrived at through the exploration of the virtual environment and through interactions with characters, sheep, and other in-world objects. 3.2. Participants A total of 221 middle schools participants (7th grade) were recruited from four schools, from two districts in a mid-Atlantic state. Roughly equal numbers of boys and girls took part in the study. Of the original participants, 193 completed the assessment and post-survey resulting in a sample of 193 students for the study. From the participating students, 98 students were randomly assigned to the virtual world with visual signaling and the other 95 students were assigned to the virtual world without visual signaling. 3.3. Environment Sheep Trouble assesses student understanding of concepts of adaptation and structure/function that underlie beginning speciation. The module is designed so that by interacting with farmers and sheep on a virtual farm, students gain the contextual understanding they need to apply their classroom learning to more meaningful assessment tasks. In Sheep Trouble, students enter a medieval-like world (Figs. 1 and 2) where they are met by a farmer who asks them to help him find a scientific explanation for why his recently imported herd of sheep is in poor health. Students discover that many of the townsfolk think that the new sheep are sick due to ‘‘bad magic’’ and that the new sheep must be destroyed quickly before the bad magic spreads to the local sheep. The farmer asks the student-scientists to apply their skills to suggest possible science-based contributors to the new sheep’s poor condition. Students are given virtual tools for data collection and can also observe visual clues about the environment itself. For example, a poster on the farm explains that the new sheep come from a very different geographic locale (a flat, snowy island) from the current farm (hilly, rocky, and dry). Students use a question and answer system to communicate with a farmer and his brother on the farm. They can also interact with a large number of the new and local sheep scattered around the farmyard. The local ‘‘original’’ sheep (sheep from stock that have been raised locally for centuries) are all healthy. Students can use virtual rulers to measure the sheep’s legs, body length, and
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
37
ears, and can access information on sheep weight loss or gain, age and gender. By applying their knowledge of scientific inquiry and of adaptation through exploration of the virtual world, students can discover that to get to the best grass, the new sheep need to climb a hill. Students gather evidence and can reach the conclusion that the newly imported sheep are unable to adapt to climb steep hills for food, due to their structural differences. As a result, they are failing to thrive. Students collect data, view and interpret the data, and report their findings to the Farmer to complete their assessment ‘quest’.
3.4. Procedure Participants completed a one class period virtual world-based assessment using the ‘‘Sheep Trouble’’ module, designed to measure their understanding of the concept of an organism adapting to a given environment over time. This topic had been taught as part of the regular in-class curriculum just prior to the participants taking the virtual world-based assessment. After completing the assessment, participants completed a survey containing a number of items related to science interest, game play experience, immersion, and perceived cognitive load. For the current sub-study under the larger SAVE Science umbrella of studies, we are examining only the cognitive load survey questions.
3.5. Instruments 3.5.1. Cognitive load survey After completing the IVE-based assessment module, students answered seven questions on a web-based survey related to perceived cognitive load (adopted from [30]). Paas and Van Merrienboer [30] reported an internal consistency reliability for the measure of .82. The perceived cognitive load measure, or subjective rating scale [42], is based on the assumption of people’s ability for accurate introspection on their cognitive processes and reporting on the amount of mental effort experienced while completing instructional tasks. In our study, we used the self-report cognitive load measure rather than measuring cognitive load through eye-tracking software and hardware primarily for convenience of implementation in a classroom setting. The self-report survey used in this study consists of 10 point scale Likert-style questions contained items such as ‘‘How hard did you have to work to find the things in Scientopolis you wanted to see or interact with?’’, ‘‘How much effort did you have to invest in order to navigate in Scientopolis?’’, or ‘‘How hard did you have to work to communicate with the people you met in Scientopolis?’’
3.5.2. Assessment efficiency As we have described, for purposes of this study we defined efficiency as the number of interactions with assessmentrelated objects completed during the assessment. We measured: total number of ‘‘collisions’’: the number of times a student walks up to an in-world object to interact with it (triggering interaction windows), total number of collisions with sheep: collisions with sheep only, other collisions: collisions with NPCs or poster, total number of measurements: total number of measurements taken while interacting with sheep, measure per sheep: average number of measurements per sheep, total number of records in clipboard: number of records recorded in an electronic clipboard, and clipboard per sheep: average number of records recorded per sheep.
4. Results A one-way analysis of variance (ANOVA) was conducted to assess the effect of visual signaling on perceived cognitive load and on efficiency of the IVE-based assessment. The factor of signaling treatment included two levels: visual signaling group and non-visual signaling group. For perceived cognitive load, we found a statistically significant difference for the combined survey means with an alpha level of .05, F(1, 175.97) = 4.27, p = .04, d = .29, with students in the visual signaling group reporting lower average cognitive load than those in non-visual signaling group (see Table 1 and Fig. 3). For individual aspects of cognitive load, the effect of visual signaling was significant on the question ‘‘How hard did you have to work to communicate with people you met in Scientopolis?’’, F(1, 151.81) = 5.97, p = .02, d = .31. Weakly significant differences were seen on two questions ‘‘How hard did you have to work to find things in Scientopolis you wanted to interact with?’’ and ‘‘How much effort did you have to invest in order to navigate in Scientopolis?’’, F(1, 176.94) = 2.92, p = .09 and F(1, 191) = 3.37, p = .07 respectively. In all cases, students in the visual signaling group reported lower mean levels of perceived cognitive load. The means and standard deviations of these items for both groups are reported in Table 1. On the efficiency question, results showed that the effect of visual signaling was significant for four types of interactions with assessment-related objects at an alpha level of .05: Total collisions with all possible in-world objects, F(1, 191) = 5.57, p = .02, Cohen’s d = .34, Total collisions with sheep, F(1, 191) = 7.09, p < .01, d = .38, Total number of measurements of sheep taken, F(1, 183.96) = 13.63, p < .001, d = .51, and Total number of records entered into an electronic clipboard, F(1, 187.56) = 11.68, p = .001, d = .48. Average values of these interaction measures for visual signaling group were higher than those for non-visual signaling group (Table 2 and Fig. 4).
38
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
Table 1 Means and standard deviations of cognitive load survey items. Conditions
Cognitive load survey items 50
Visual signaling Non-visual signaling
a
52
b
54
c
Combined
M
SD
M
SD
M
SD
M
SD
3.56 4.20
2.24 2.90
3.24 3.98
2.57 2.96
2.40 3.25
1.75 2.95
24.81 28.35
10.20 13.34
Note: Subjects were asked to indicate the degree of perceived effort with each of survey items. Ratings were made using 1–10 Likert-type scale, in which 1 = ‘‘not much effort’’ to 10 = ‘‘a lot of effort’’. a How hard did you have to work to find the things in Scientopolis you wanted to see or interact with? b How much effort did you have to invest in order to navigate in Scientopolis (e.g. for deciding between different buttons or keys, finding your way around)? c How are did you have to work to communicate with the people you met in Scientopolis?
Fig. 3. Perceived cognitive load by treatment.
Table 2 Means and standard deviations of interactions with assessment-related objects. Conditions
Visual signaling Non-visual signaling
Interactions with assessment-related objects Total number of collisions
Total number of collisions with sheep
Total number of measurements
Total number of records in clipboard
M
SD
M
SD
M
SD
M
SD
21.04 18.31
8.24 7.86
14.21 12.15
4.88 5.88
44.43 33.01
23.82 18.93
40.54 30.76
21.50 18.18
5. Discussion and conclusion Our findings from this study support the hypotheses that use of visual signaling can reduce a student’s overall perceived cognitive load while completing a virtual world-based assessment, and that visual signaling can increase the efficiency of IVE-based assessments as measured by numbers of interactions with relevant materials. The findings for perceived cognitive load are not particularly strong, however, with only marginally significant differences between groups for perceived cognitive load related to finding interactive objects and navigating the virtual world. The weak mean differences between groups for finding objects and navigating the world may reflect the fact that the world itself was designed to contain very few interactive objects and contained a limited amount of virtual space to explore. The low number of objects in the world limited the amount of visual search that was required of learners in the environment. As we have described, low amounts of visual search have been found to reduce the amount of cognitive load experienced by learners. In future studies, we will explore the use of visual (and auditory) signaling in more visually complex, ‘‘high search’’ virtual environments. It is interesting to note that there were relatively strong differences between reported effort in communicating with characters in the virtual world between signaling and non-signaling groups, particularly given that there are only two characters in the virtual world. It is not clear from the data collected why use of signaling should impact interaction rates with in-world
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
39
Fig. 4. Object interactions by treatment.
characters despite the low complexity of the world and few characters present. It may be that the presence of visual signaling is acting as a kind of ‘umbrella signal’ that results in an overall increased likelihood of students interacting with all objects with signals. The differences between groups in terms of assessment efficiency as shown by total interactions (aka ‘collisions’ and mouse clicks) with interactive objects were quite strong. Use of visual signaling in a virtual world-based assessment was clearly demonstrated to be associated with increased efficiency of completing assessment tasks. Strongly significant mean differences were seen between groups in total number of measurements taken while completing the assessment, and total number of records saved into the electronic clipboard. From a virtual world-based assessment design perspective, these findings provide powerful incentive to include visual signaling as a way to help ensure that students are able to find and use the tools that enable them to provide evidence of their level of understanding of the concepts being assessed. If assessments are embedded into virtual worlds, it is critical that the design of the worlds supports them in completing the tasks of assessment, so that valid and reliable inferences can be made from the tasks performed. In our own work, the strong findings from this study related to object interaction rates led our team to include visual signaling in all subsequently created assessment modules. Despite the demonstrated benefits of visual signaling in virtual world-based assessments, questions remain. One of the most challenging is whether the use of visual signaling in a virtual world-based curriculum reduces the learning power of, and participant engagement with, the world. If, as researchers have purported (e.g. [5,8,13,2,29,26]), the power for learning in virtual worlds is centered on open-ended inquiry situated in realistic contexts, does the use of visual signaling reduce that power? And if unguided discovery of, and interaction with, embedded objects in a virtual world is central to the learning power of virtual worlds, does use of signaling rob a student of that experience? We suspect that signaling does have some deleterious impact on the overall learner immersion and embodiment in a virtual world, but only when used ‘too much’. The challenge is to discover the appropriate level of signaling that will reduce extraneous cognitive load and increase efficiency of learning, while simultaneously maintaining the sense of immersion and open-ended exploration that are hallmarks of virtual worlds. Achieving two possibly contradictory goals is a tricky balancing act, but one that may be key to the successful and sustained use of virtual worlds as educational tools. Acknowledgments The study discussed in this article was conducted under the umbrella of the SAVE Science research project. Our sincere thanks to the SAVE Science teams at University of Maryland and Temple University, and its leadership team: PI Diane Jass Ketelhut and co-PI Catherine Schifter. References [1] P. Ayres, F. Paas, Interdisciplinary perspectives inspiring a new generation of cognitive load research, Educational Psychology Review 21 (2008) 1–9. [2] S. Barab, A. Arici, C. Jackson, Eat your vegetables and do your homework: a design based investigation of enjoyment and meaning in learning, Educational Technology 45 (1) (2005) 15–20.
40
B.C. Nelson et al. / Information Sciences 264 (2014) 32–40
[3] Carnegie Corporation, The Opportunity Equation: Transforming Mathematics and Science Education for Citizenship and the Global Economy, Carnegie Corporation of New York, New York, 2009. [4] C.J. Chen, W.M.W.I. Fauzy, Guiding exploration through three-dimensional virtual environments: a cognitive load reduction approach, Journal of Interactive Learning Research 19 (4) (2008) 579–596. [5] D. Clark, B. Nelson, P. Sengupta, C. D’Angelo, Rethinking Science Learning through Digital Games and Simulations: Genres, Examples, and Evidence, An NAS Commissioned Paper, 2009.