Investigating learning outcomes and subjective experiences in 360-degree videos

Accepted Manuscript Investigating learning outcomes and subjective experiences in 360-degree videos Michael A. Rupp, Katy L. Odette, James Kozachuk, Jessica R. Michaelis, Janan A. Smither, Daniel S. McConnell PII:

S0360-1315(18)30251-3

DOI:

10.1016/j.compedu.2018.09.015

Reference:

CAE 3458

To appear in:

Computers & Education

Received Date: 17 February 2018 Revised Date:

2 August 2018

Accepted Date: 23 September 2018

Please cite this article as: Rupp M.A., Odette K.L., Kozachuk J., Michaelis J.R., Smither J.A. & McConnell D.S., Investigating learning outcomes and subjective experiences in 360-degree videos, Computers & Education (2018), doi: https://doi.org/10.1016/j.compedu.2018.09.015. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

ACCEPTED MANUSCRIPT

Investigating Learning Outcomes and Subjective Experiences in 360-Degree Videos

and Daniel S. McConnell1

RI PT

Michael A. Rupp1, Katy L. Odette2, James Kozachuk1, Jessica R. Michaelis1, Janan A. Smither1,

SC

1: Technology and Aging Laboratory, Department of Psychology University of Central Florida

TE D

M AN U

2: Institute for Simulation and Training, University of Central Florida

Author Note:

EP

Correspondence concerning this article should be addressed to Michael A. Rupp, Ph.D., who is now at the Department of Psychology, University of California, Riverside CA 92507. E-mail:

AC C

[email protected].

ACCEPTED MANUSCRIPT

SC

RI PT

1

AC C

EP

TE D

M AN U

Investigating Learning Outcomes and Subjective Experiences in 360-Degree Videos

ACCEPTED MANUSCRIPT 2 ABSTRACT Virtual Reality experiences, particularly the 360-degree video, have become popular in recent years for creating immersive educational experiences. However, much is still unknown regarding

RI PT

the educational effectiveness of this medium. Here we examined pre-to-post changes in wellbeing, simulator sickness, and learning outcomes across four devices of varying levels of immersion: a smartphone, Google Cardboard, Oculus Rift DK2, and Oculus CV1 using a space-

SC

themed 360° educational video. More immersive devices induced greater induction of place illusion, greater positive affect, and better learning outcomes while demonstrating low

M AN U

prevalence of simulator sickness. Greater immersion was also associated with an increased interest in learning more about the video’s subject-matter. On the other hand, less immersive technology led to increased simulation sickness which may have led to suboptimal educational experiences. Overall, we found support for the hypothesis that highly immersive experiences

TE D

using 360° videos provide positive educational experiences.

Keywords: Virtual reality learning environments; simulator sickness; immersion; presence;

AC C

EP

place illusion

ACCEPTED MANUSCRIPT 3 1. Introduction Virtual reality (VR) learning experiences are engaging and allow students to immerse themselves in content beyond what is possible in the real world (Snelling, 2016). For example, a

RI PT

class could use VR to go on a virtual field trip (e.g., Çalişkan, 2011) and experience history by exploring the Old North Church guided by a simulated Paul Revere. Previously, a barrier to the adoption of this technology has been its cost, availability, and lack of quality content. However,

SC

2016 was dubbed the year of VR (Cellan-Jones, 2016), as consumer-level affordable technology became available. This advance, along with a greater focus on the creation of educational

M AN U

experiences, supports the use of VR for augmenting instruction (Ciobanu, 2016; Yoder, 2016). Of concern, then, is ensuring this medium is delivering content effectively so that learning outcomes are reliably achieved.

1.2 Immersion and place illusion in virtual reality

TE D

There are several factors relevant to the effectiveness of VR learning experiences. One is the level of immersion allowed by the technology. A highly immersive VR experience is one that provides sensory immersion (Dede, Jacobson, & Richards, 2017). This means the outside world

EP

is shut out, reducing distractions away from the content, owing to the use of a head mounted display and headphones. Such an immersive virtual environment may lead learners to spend

AC C

more time engaged with the content and decrease instances of mind-wandering (Bulu, 2012; Dede, 2009; Hollis & Was, 2016). Tuzun & Ozdinc (2016) reported better conceptual learning in a virtual environment that was correlated with reduced distractions, making this facet of immersion important to the effectiveness of the VR experience. Next, research into maximizing positive learning outcomes has suggested videos with vivid graphics and increased interactivity are key to effectively engaging learners with the content

ACCEPTED MANUSCRIPT 4 (Zhang, Zhou, Briggs, & Nunamaker, 2006). One way highly immersive VR experiences do this is by aligning action and perception in the virtual space so that engaging in an action provides the same perceptual information as reality. These experiences are known as sensorimotor

RI PT

contingencies of the virtual environment (Slater, 2009). Thus, VR experiences with greater

sensorimotor contingencies are also ones that have higher quality display resolution, wider field of views, and stimulate multiple sensory modalities. VR experiences that do not provide

SC

sensorimotor contingencies may limit users’ agency or the simulated movement may cause

mismatches in the body’s vestibular systems leading to dramatic decreases in well-being (i.e.,

M AN U

simulation sickness; Lawson, 2014). People who experience simulator sickness may display overt signs of nausea or vomiting which may impair their cognitive abilities and prevent a positive learning experience (Lackner, 2014).

On the other hand, highly immersive VR experiences may lead users to feel as if their

TE D

consciousness has been relocated to the virtual space, called place illusion or presence (Slater, 2009). In other words, when viewing a virtual environment, one may view it as just images displayed on a screen, or it may be experienced as an actual place. When the user feels they are

EP

“in” the virtual environment, they are said to feel present within it. The suggested benefit of place illusion is that it enables improved performance within VR by providing accurate

AC C

perceptual cues to the user and allowing them to form accurate cognitive maps (Slater & Sanchez-Vives, 2016).

Dale’s (1946, 1954, 1969) cone of experience instructional theory states that direct

participation helps provide a foundation for novice learners. Immersive VR then provides a way for students to gain more direct experiences. This in turn increases the chance for situated learning and transfer to real world skills and knowledge by ensuring the learning environment is

ACCEPTED MANUSCRIPT 5 similar to situations in which it will be applied (Dede, Jacobson, & Richards, 2017). Consistent with this latter point is the relationship between place illusion and plausibility illusion – the illusion that what is happening is really occurring (regardless of an awareness that the situation is

RI PT

not real) leading users to act as if the simulation is real, and obeying physical laws (Slater, 2009). For example, place and plausibility illusion has been linked to increased physiological stress reactions to fearful stimuli (Petkova, 2008) and a tendency to avoid walking through objects and

SC

walls (Slater & Sanchez-Vives, 2016). Thus, learners who experience place illusion may have better learning outcomes than learners who do not. Specifically, increased plausibility illusion

M AN U

may facilitate spatial learning, because users would travel similar routes as they would in the real world, constrained by obstacles real or virtual. Several studies have demonstrated that place illusion was associated with improved spatial performance, e.g. route learning in a navigation task (Bailey & Witmer, 1994; Tüzün & Özdinç, 2016; Witmer, Bailey, Knerr, & Parsons, 1996).

TE D

Other studies that did not directly evaluate the relationship between place illusion and spatial learning nevertheless have reported relationships between manipulations of various dimensions of immersion and spatial knowledge and memory (Dinh, Walker, Song, Kobayashi, & Hodges,

EP

1999; Mania & Chalmers, 2001; Papadakis, Mania, Coxon, & Koutroulis, 2011; Ruddle, Volkova, & Bülthoff, 2011).

AC C

Another reason why place illusion may impact the learning is by creating emotional

experiences (e.g., Diemer, Alpers, Peperkorn, Shian, & Mühlberger, 2015). Experiencing an immersive VR simulation of cutting down a tree has been shown to be a more persuasive motivator of pro-environmental attitudes behaviors than reading printed material about deforestation (Ahn, Bailenson, & Park, 2014). Additionally, Sheridan (2016) posited providing VR simulations that evoke the emotional response of a vehicle crash or near miss would be an

ACCEPTED MANUSCRIPT 6 impactful training tool to improve driving education. Furthermore, because, emotional experiences have been shown to lead to the formation of more detailed memories (e.g., Adelman & Estes, 2013; Yonelinas, & Ritchey, 2015), more immersive VR experiences that engage place

RI PT

illusion may also be associated with more positive learning outcomes.

However, it is unclear whether these benefits translate to users of lower-fidelity consumer VR; thus, more research is needed to support the efficacy of VR learning experiences using

SC

currently available technologies and learning content typical of currently available experiences. While fully immersive virtual reality learning environments exist, more common is the use of

M AN U

360° videos due to their lower cost and wide availability (Bessa, Melo, Narciso, Barbosa, & Vasconcelos-Raposo, 2016; Zhou, Li, & Liu, 2017). 360° videos are omnidirectional panoramic videos that allow the viewers to pan and tilt in an uninterrupted circle rather than the fixed viewpoint of a traditional video. These videos can be viewed on phones by panning and tilting

TE D

the phone, or on other devices, like low cost phone-enabled VR headsets (e.g., Google Cardboard), or dedicated VR displays (e.g., Oculus Rift), by turning one’s head similar to exploring the real world. Because viewers of 360° videos have the agency to look around and

EP

explore different parts of the scene, these videos are more immersive than traditional 2D videos, but less than a truly immersive VR learning experience.

AC C

Because watching a 360° video using virtual reality technology is not the same as

engaging with a virtual reality experience specifically designed to teach a specific topic, many questions remain regarding their effectiveness. First, does increased immersion affect the effectiveness of a 360° video as an educational tool? According to an updated version of Dale’s cone of experience theory (Baukal, Ausburn, & Ausburn, 2013), while immersive VR experiences may provide direct experiences, 360° video is more abstract and thus may not

ACCEPTED MANUSCRIPT 7 provide the same quality of experience for novice learners. Additionally, videos relative to VR experiences may limit situated learning because of the lack of student-user agency (Dede, Jacobson, & Richards, 2017). Increasing immersion may overcome these limitations.

RI PT

Second, will people watching 360° videos using less immersive devices experience

increased rates of simulator sickness? Viewing 360° videos on technology that does not provide sensorimotor contingencies has been hypothesized to increase simulator sickness relative to more

SC

immersive technology (Smith, 2015). Next, do people who report experiencing place illusion have better learning outcomes and greater enjoyment of the experience than people who do not?

M AN U

Place illusion is aided by factors that increase immersion, but also by individual differences (Kober & Neuper, 2013; Ling, Nefs, Brinkman, Qu, & Heynderickx, 2013), indicating that the induction of place illusion depends on both the VR technology to generate sensory contingencies, but also the individual’s ability to perceive them.

TE D

2. Current Study’s Goal and Working Hypotheses

The goal of the current study was to test the effectiveness of a 360° video learning experience to facilitate learning of declarative knowledge. We hypothesize that participants who

EP

watch the educational video using more immersive technology will exhibit greater subsequent recall of the video content, report greater feelings of place illusion, positive affect, and a greater

AC C

enjoyment of the subject matter covered in the video. Moreover, any increase in signs of simulator sickness will be associated with decreased recall, presence, and well-being. In the current study, we tested four devices of varying degrees of immersion capable of displaying 360° videos to determine how well each would support the immersive learning or negative experience hypothesis. Combining these hypotheses, we expect immersion may moderate this relationship

ACCEPTED MANUSCRIPT 8 so that learners will have a negative experience with less immersive devices, but a positive experience with more immersive devices. 3. Method

RI PT

3.1 Participants

We recruited 136 students (70 men; 66 women) from a large American university

between the ages of 18 – 31 years old (Mage = 19.82; SDage = 2.44) who had not previous used a

SC

VR device. All participants passed a screening for visual acuity, color deficiency, and depth perception (stereopsis) prior to participating in the study. Additionally, participants were

M AN U

required to rate no more than a score of “2” on any question used to calculate any of the SSQ subscales to ensure participants were in a normal physical state before beginning the experiment. All participants were recruited through the university’s participant pool and were provided course credit for their participation. All reported little knowledge of the subject-matter used in

3.2 Study design

TE D

our study.

The study used four different devices to manipulate the degree of immersion across

EP

conditions. Each participant was assigned randomly to one of the four conditions. Participants in the least immersive condition viewed the 360° video on a smartphone. The phone condition,

AC C

unlike the other three conditions, did not provide a stereoscopic display to the user and instead displayed the video simply in a 3D format upon the phone screen itself. Users could pan and tilt the phone to look around the environment. Next, Google Cardboard was used in conjunction with the phone’s screen to display the 360° video in a stereoscopic fashion while partially blocking out the external environment. The Oculus Rift Development Kit 2 (DK2) further blocks out the surrounding environment, provides an improved graphical resolution, field of view, and

ACCEPTED MANUSCRIPT 9 reduced head tracking latency as compared to the Google Cardboard. Finally, the Oculus Rift Consumer version 1 (CV1) further improves the graphical quality, motion latency, and field of view over the DK2. Further the CV1 is lighter and was designed to be more comfortable as well

RI PT

as includes additional motion tracking sensors to better achieve a 1:1 mapping between the user’s head movements and the camera’s viewpoint in the virtual world. Figure 1 shows examples of

A)

B)

C)

M AN U

SC

each experimental device used in this study.

D)

TE D

Figure 1. Examples of devices used for 360-degree video playback. A) HTC M8 phone, B) Google Cardboard, C) Oculus DK2, and D) Oculus CV 1.

3.3 Materials and apparatus

4.3.1 Video hardware

EP

Participants in the smartphone condition watched the video using an HTC M8 phone running Android Marshmallow. Its dimensions were: 146.36mm × 70.6mm × 9.35mm (~220 PPI

AC C

per eye). Participants in the Google Cardboard condition used a standard cardboard viewer v.1 using the same HTC phone. Participants in the Oculus Rift DK2 condition wore a DK2 headset. The DK2 provided a resolution of 960 × 1080px per eye (~220 PPI per eye) at a refresh rate of 75Hz and a 100° field of view. Participants in the Oculus Rift CV1 condition wore a CV1 headset which provided a resolution of 2160 × 1200px per eye at a refresh rate of 90Hz and 110° field of view. We used the software Virtual Desktop on a computer running Windows 10. The

ACCEPTED MANUSCRIPT 10 computer had an i5 Skylake processor with 32GB of RAM and a Nvidia GTX 980 graphics card which exceeded the recommended technical specifications for the Oculus Rift CV1. 3.3.2 360° video

RI PT

We chose a publicly available 360° stereoscopic 3D video tour, 6.25 minutes in length, of the International Space Station (ISS). We downloaded the video from YouTube in the “1080s HD” resolution (see Manley, 2016 for the full video). This video was chosen for its high

SC

production quality and information density. Further, we felt the video’s approach of mimicking floating through space would limit viewer motion sickness. This virtual experience provided a

M AN U

guided tour from the perspective of an astronaut floating along the space station and an audio track that provided descriptions and information about the different modules and history of the international space station. 3.3.3 Dependent Variables

TE D

Feelings of place illusion were measured using the Slater, Usoh, and Steed (SUS) Presence Questionnaire (Usoh, Catena, Arman, & Slater, 2000) which is a 7-item Likert-type scale (1 = not at all; 7 = very much). We also used the spatial presence (HSP) and potential

EP

action scales (HPA; Hofer et al., 2012) consisting of three questions each on a 7-point Likerttype scale (1 = not at all; 7 = very much). Emotional reactions were measured using the Positive

AC C

and Negative Affect Scale (Watson, Clark, & Tellegen, 1988), which consisted of several positive and negative emotions and participants rated the extent they felt these emotions on a 5point Likert-type scale (1 = not at all; 5 = extremely). Simulator sickness was measured using the Simulator Sickness Questionnaire (SSQ; Kennedy, Lane, Berbaum, & Lilienthal, 1993). This scale consisted of several potential symptoms of simulator sickness across three facets of simulation sickness (disorientation, nausea, & oculomotor discomfort). We calculated

ACCEPTED MANUSCRIPT 11 individuals’ scores individually on each subscale by summing the items on each subscale and multiplying by each subscales factor weight to equate the three facets of simulation sickness. We also calculated scores using the total score method. This method involved calculating each

RI PT

unweighted subscale and multiplying the total by 3.74 to yield the total score with a maximum possible value of 300. The weighting of 3.74 is designed to equate the scores calculated by each method (see Kennedy et al., 1993 for more information).

SC

We measured the amount of information viewers recalled from the video by creating a knowledge check containing 38 auditory and visual questions derived from the video. Auditory

M AN U

questions asked about information presented in the video narration while visual questions presented a screenshot of the video and asked participants to identify a specific part of the ISS which was specifically identified in the video narration. Each question was presented as a multiple-choice question with four possible answers. Each question was designed to have a

TE D

single factual answer that could be learned from the video. We conducted a pilot study to determine both whether the survey should be given pre-post as well as which questions should be used on the questionnaire. Five people were pretested using the questionnaire then watched the

EP

video on a phone (same as phone condition in current study). Participants then took the same questionnaire as a post-test. An additional five people completed the same procedure except were

AC C

not given the pre-test. Results showed no significant differences between participants given the pretest first and participants without the pretest. Additionally, we removed questions with the highest incorrect responses that did not significantly correlate with the overall score. The remaining 28 questions (18 auditory; 10 visual) were used. Participants received 1 point for each correct answer; their score adjusted by reducing it by 0.25 for every incorrect answer and received no points if they reported they did not know the answer. This was used to discourage

ACCEPTED MANUSCRIPT 12 guessing. Finally, we assessed participants’ expectations of VR and of the subject matter by administering a 12-question survey. Each question asked participants to rate how excited they were about VR technology (e.g. “How positive are your expectations for VR in the future?”) on a

RI PT

7-point Likert-type scale (1 = strongly disagree; 7 = strongly agree) or how excited they would be to learn more about NASA or the International Space Station in the future. All reverse-coded items were positively recoded prior to analysis. Table 1 showed a list of the study’s dependent

SC

variables Table 1. List of study dependent variables and time of administration.

Dependent Variables

1

Place Illusion Composite Measure 1a) Slater, Usoh, and Steed Presence Questionnaire (SUS) 1b) Spatial Presence Scale (HSP)

M AN U

Administered PreSimulation

#

TE D

1c) Potential Action Scale (HPA)

Administered PostSimulation

-

Yes

-

Yes

-

Yes

-

Yes

-

Yes

Yes

Yes

Post Simulation Auditory and Visual Knowledge Check

3

Positive and Negative Affect Scale (PANAS)

4

Simulator Sickness Questionnaire (SSQ)

Yes

Yes

5

Post Simulation Auditory and Visual Knowledge Check

Yes

Yes

6

Subject-Matter Interest

Yes

Yes

7

Virtual Reality Expectations

Yes

Yes

AC C

EP

2

Note: HSP and HPA were adapted from Hofer et al., (2012)

ACCEPTED MANUSCRIPT 13 3.4 Procedure After participants arrived at the lab and gave their informed consent they were given a vision pre-screening. They were tested on visual acuity, color deficiency, and depth perception.

RI PT

Next, participants completed the VR expectation, subject interest, Pre-SSQ, and PANAS scales on a computer, in that order. Next, participants watched the 360° video tour using one of the four possible conditions assigned at random. Participants were instructed that their goal was to learn

SC

as much as they could about the ISS while they watched the video. All participants sat in a

typical swivel office chair in an open area so they could turn and move around during the video.

M AN U

Participants in the phone, cardboard and DK2 conditions wore separate noise canceling headphones while participants in the CV1 condition wore the headphone attached to the headset. Following the video participants filled out the information recall test, place illusion scales, VR expectation, subject matter interest questionnaires, PANAS, Post-SSQ, and demographics in that

TE D

order. Participants voluntarily stayed 15 minutes after the study to check for signs of simulator sickness before they left the lab. 4. Results

EP

Prior to data analysis, variables were tested for assumptions of normality. SSQ and PANAS scores were found to be slightly skewed, but otherwise no other significant violations of

AC C

assumptions were noted. We calculated standardized change z-scores for all pre-post measures (SSQ, positive and negative affect, VR expectations, and subject-interest) by subtracting the post – pre-simulation ratings and dividing by the σpre to control for pre-study differences between participants. Therefore, a score of 1 indicates that the rating has increased by 1 standard deviation after the simulation while a score of -1 would indicate a decrease of a standard deviation post simulation.

ACCEPTED MANUSCRIPT 14 To analyze our data, we conducted both an OLS multiple regression analysis as well as its Bayesian equivalent. Comparisons of Bayes factor was used as post-hoc tests to probe differences between conditions. JASP v. 0.8.5, a freely available graphical statistical program

RI PT

designed to run R statistical packages and conduct Bayesian analyses (JASP 2017) was used to analyze all of our results. For the multiple regression analysis, we examined the main effect of our parametrically manipulated immersion conditions, subjective place illusion felt following the

SC

simulation using our calculated composite score, and the interaction between immersion

condition and place illusion. All variables were entered into the model at the same time in the

M AN U

same block. In order to determine significant pre-to-post changes within each condition, we examined the 95% confidence intervals of the change scores and only those that did not overlap zero were determined to be statistically significant. While interaction effects were tested continuously using regression, for ease of display or significant findings, low and high groups on

TE D

the variable of interest will be created using a median split in order to create equal groups. The Bayesian analysis followed the procedure in Rouder, Morey, Verhagen, Swagman, and Wagenmakers (2016). The main test statistic of interest in this analysis was the Bayes factor.

EP

The Bayes factor (BF) indicates the number of times that the observed data is more likely either under the alternative (BF10) or null (BF01) hypotheses. Guidance on the interpretation of BFs

AC C

indicate that values for BF10 above 3 show substantial evidence for the alternative hypothesis (Wetzels & Wagenmakers, 2012). We chose a prior Cauchy width of r = .5 to include realistic effect sizes within the range of our hypothesized medium effect size. Unlike traditional NHST, multiple comparisons can be made without an inflation of Type I error rates (Howard, Maxwell, & Fleming, 2000; Masson, 2011). Using a combination of these techniques we examined the data in two separate ways. First, we investigated the changes between conditions and second, we

ACCEPTED MANUSCRIPT 15 examined the variables that were administered pre-to-post, to determine for a given condition if that change indicates a significant deviation from zero. In the latter case, we deemed a change score significant only if the 95% confidence interval of the SD change did not include zero.

RI PT

5.1 Presence scores

We found large positive correlations between all place illusion subscales (all rs .61 - .88, ps < .001). Due to these strong correlations, we created a composite score by converting all three

SC

scales into z-scores and then added the three z-scores together and conducted the analysis using the composite scale. Next, before analyzing the effects of our conditions, we examined the

M AN U

correlation between our independent variables due to potential issues with multicollinearity that may be present in the data since immersion was hypothesized to be associated with increases in subjective place illusion. Although we did find a significant positive correlation (r = .28, p = .001) between immersion and place illusion indicating that greater immersion led to greater

TE D

ratings of place illusion. This effect was mostly driven by the lack of place illusion experienced in the phone condition due to its lack of immersion (Figure 2). The strength of this relationship

AC C

EP

was small enough to not cause an issue in our regression models.

ACCEPTED MANUSCRIPT 16 5.00

Phone

Cardboard

DK2

CV1

4.00 3.50 3.00 2.50

RI PT

Mean Place Illusion Score

4.50

2.00 1.50 1.00

0.00 SUS

HSP

SC

0.50

HPA

5.2 Overall differences pre-to-post

M AN U

Figure 2. Mean post-simulation raw place illusion scores for all three measures across conditions. Error bars are 95% CI. SUS is the Slater Usoh and Steed presence questionnaire, HSP is the spatial presence scale from Hofer et al. (2012), and HPA is the potential for action scale from Hofer et al. (2012).

To ensure there were no significant differences at the pre-simulation time-point due to

TE D

failures of randomization which may affect the later calculation of change scores we used a Bayesian ANOVA to compare each condition on each of the 5 pre-post measures (Simulator Sickness, Positive Affect, Negative Affect, VR Expectations, and Subject-Matter Interest). The

EP

Bayes factors for each of these ANOVA models showed a BF10 statistic less than 1.0 indicating

AC C

no support for the alternative hypothesis that differences existed between conditions presimulation using the Wetzels and Wagenmakers (2012) criteria. Next, we compared the raw difference scores between conditions in order to provide

information on the absolute magnitude of the overall effects. For simulator sickness, we found both the DK2 and CV1 showed small pre-to-post changes on all three subscales yielding a total score increase of 10.02 for the CV1 and 22.10 for the DK2. However, the phone condition yielded increases in both oculomotor and disorientation which led to an average increase of

ACCEPTED MANUSCRIPT 17 48.42. Finally, the Google Cardboard condition showed the greatest increase in motion sickness

Table 2. Simulator Sickness subscales and total scores on the SSQ by condition.

Variable

Phone

Cardboard

Simulator Sickness – Nausea Pre-simulation

5.33 ± 0.24

6.73 ± 0.24

Post-simulation

7.86 ± 0.26

11.5 ± 0.24

Simulator Sickness – Oculomotor Pre-simulation Post-simulation Difference

Post-simulation Difference

Difference

7.58 ± 0.3

6.17 ± 0.33

6.17 ± 0.31

1.68

-1.41

8.47 ± 0.31

10.0 ± 0.37

12.71 ± 0.28

14.05 ± 0.38

10.26 ± 0.34

12.48 ± 0.37

8.25

1.79

2.48

4.5 ± 0.29

4.91 ± 0.19

6.96 ± 0.25

5.73 ± 0.45

10.24 ± 0.51

16.38 ± 0.57

9.42 ± 0.67

7.37 ± 0.6

66.80 ± 2.95

TE D

Pre-simulation

4.49 ± 0.15

5.8 ± 0.13

5.74

Simulator Sickness – Total Score

CV1

8.03 ± 0.26 4.68

Simulator Sickness – Disorientation Pre-simulation

Post-simulation

4.77

DK2

SC

2.53

M AN U

Difference

RI PT

with a total score of 91.59, almost double the score of the phone condition (Table 2).

115.22 ± 3.92

11.47

2.46

1.64

65.23 ± 2.09

74.5 ± 2.65

87.29 ± 4.18

156.82 ± 4.45

96.6 ± 5.01

97.31 ± 4.78

48.42

91.59

22.1

10.02

EP

In terms of positive and negative affect, we found small increases in positive affect and decreases in negative affect for all conditions except the phone condition following the

AC C

simulation. Both the cardboard and CV1 showed the largest increases in positive affect, while the DK2 showed the greatest decrease in negative affect. For VR expectations, the phone condition was the only group to not show an increase pre-to-post simulation and the CV1 condition showed the greatest increase after the video (Table 3).

ACCEPTED MANUSCRIPT 18 Table 3. Positive and Negative affect, virtual environment Expectations, and Subject-Matter Interest scores by condition.

Variable

Phone

Cardboard

DK2

CV1

2.94 ± 0.16

3.23 ± 0.13

3.15 ± 0.12

2.75 ± 0.15

Post-simulation

2.87 ± 0.14

3.45 ± 0.1

3.26 ± 0.13

3.15 ± 0.16

-0.07

PANAS – Negative Affect Pre-simulation Post-simulation

1.23 ± 0.09

1.27 ± 0.08

1.24 ± 0.07

1.24 ± 0.05

0.01

VR Expectations Pre-simulation Post-simulation Difference

0.11

0.40

1.29 ± 0.10

1.29 ± 0.09

1.16 ± 0.04

1.21 ± 0.05

-0.13

-0.08

5.25 ± 0.25

5.47 ± 0.21

5.41 ± 0.23

5.11 ± 0.29

5.29 ± 0.25

5.69 ± 0.20

5.71 ± 0.20

5.51 ± 0.28

0.04

Subject-Matter Interest

-0.03

M AN U

Difference

0.22

SC

Difference

RI PT

PANAS – Positive Affect Pre-simulation

0.22

0.3

0.40

Pre-simulation

3.58 ± 0.22

3.61 ± 0.14

3.58 ± 0.21

3.60 ± 0.14

Post-simulation

3.84 ± 0.10

4.33 ± 0.09

4.10 ± 0.10

4.26 ± 0.14

Difference

0.26

0.52

0.66

TE D

5.3 Multiple regression analysis

0.72

Our multiple regression analysis, found a significant pre-to-post SD change for positive affect, VR expectations, and subject-matter interest as well as a significant effect for auditory

EP

recall scores. No significant results were found for pre-to-post SD change on simulator sickness, negative affect, or visual recall scores. Table 4 shows regression and Bayes factor results for

AC C

each variable while Figure 2 shows pre-to-post change scores for each dependent variable.

ACCEPTED MANUSCRIPT 19 1.50 Phone

1.00

Cardboard

DK2

CV1

0.00 -0.50

RI PT

Mean SD Change

0.50

-1.00 -1.50 -2.00 -2.50 -3.00 Postitive Affect (PANAS)

Negative Affect VR Expectations (PANAS)

M AN U

Dependent Variable

Subject-Matter Interest

SC

Simulator Sickness (SSQ)

Figure 2. Mean SD change scores across conditions. Error bars indicate 95% CI.

We found a main effect of both immersion and place illusion on changes in positive affect. For the main effect of immersion, greater immersion led to greater increases in positive

TE D

affect. The change in positive affect for the CV1 (B10 = 105.51), DK2 (B10 = 18.67), and Google Cardboard (B10 = 73.46) conditions was significantly greater than the phone condition. The beta weight for the effect of condition indicated a small effect size, which may be due to only findings

EP

a significant standardized change in the Cardboard and CV1 conditions. Greater subjective feelings of place illusion also led to greater changes in positive affect. The beta weight and the

AC C

correlation between the place illusion composition variable and both post-simulation positive affect (r = .32, p < .001) and standardized change scores (r = .29, p < .001) indicate a moderate effect. Pre-simulation positive affect scores did not correlate with increased feelings of place illusion in the simulation (r = .32, p < .001). This may indicate that the direction of the effect was that feeling present led to increases in positive affect instead of differences in positive affect leading to increases in feelings of place illusion.

ACCEPTED MANUSCRIPT 20 A main effect of immersion and an interaction between immersion and place illusion were found for changes in VR expectations. As immersion increased so did participants expectations for VR. This finding indicated that providing a more immersive experience was

RI PT

more compelling for participants leading them to a greater likelihood that they would want to try the technology again. Specifically, the largest differences were found between the CV1 and phone conditions (B10 = 1.53). Only the DK2 and CV1 conditions showed significant increases

SC

in VR expectations from baseline. In terms of place illusion, the interaction indicated that for the phone condition greater place illusion led to decreases in wanting to try a VR device while no

M AN U

differences were found for Google Cardboard. However, for the more immersive conditions a sense of place illusion led to increases in VR expectations while no changes were found if participants did not have a sense of place illusion while watching the video (Figure 3). 1.20

0.40 0.20 0.00

TE D

0.60

High Pres

EP

0.80

AC C

Mean SD Change in Virtual Reality Expectations

Low Pres 1.00

-0.20 -0.40 -0.60

Phone

Cardboard Condition

DK2

CV1

Figure 3. The interaction effect of immersion and place illusion on changes in VR Expectations using mean SD change. For ease of display a median split was used to split individuals into low and high place illusion groups. Error bars represent 95% CI.

ACCEPTED MANUSCRIPT 21 While no main effect of immersion was found for subject-matter interest, we discovered a significant main effect of place illusion. Apart from the Google Cardboard, participants who experienced greater place illusion had greater post simulation increases in interest for learning

RI PT

more about space the and ISS. Participants in the phone condition who did not report feeling present did not have a significant post-simulation change in subject-matter interest. All other conditions, regardless of place illusion score, showed a positive increase in subject-matter

SC

interest after watching the 360° video, but this increase was on average greater if the experience activated a sense of place illusion (Figure 4). The beta weight for this effect along with the

M AN U

correlations between the place illusion composite variable and post-simulation subject-matter interest (r = .15, p = .08) and the standardized change in subject-matter interest (r = .35, p < .001) indicate a moderate effect of place illusion. Pre-simulation subject-matter interest scores did not correlate with increases in presence (r = -.07, p = .42). Finding only a significant

TE D

correlation between place illusion and the pre-to-post change score may indicate that feeling present in the simulation lead to the increase in subject-matter interest and not the converse that

AC C

EP

participants who were more interested in the subject felt more present following the video.

ACCEPTED MANUSCRIPT 22 0.80 High Pres

0.60

RI PT

0.50 0.40 0.30 0.20

SC

0.10 0.00 -0.10 Phone

M AN U

Mean SD Change in Subject-Matter Interest

Low Pres 0.70

Cardboard

DK2

CV1

Figure 4. Main effect of place illusion on increases in subject-matter interest using mean SD change. For ease of display a median split was used to split individuals into low and high place illusion groups. Error bars represent 95% CI.

TE D

Finally, main effects were found for both immersion and place illusion for auditory, but not visual recall scores. While the amount of auditory information recalled increased with immersion, significant differences were only found in the CV1 condition. Auditory recall scores

EP

in the CV1 condition were greater than in the phone (B10 = 72.03), Cardboard (B10 = 532.61),

AC C

and the DK2 (B10 = 3.25) conditions (Figure 5).

ACCEPTED MANUSCRIPT 23 10 Auditory

9

Visual

7 6

RI PT

Mean Recall Score

8

5 4 3

SC

2 1 0 Cardboard

DK2

CV1

M AN U

Phone

Condition

Figure 5. Mean auditory and visual recall scores across immersion conditions. Error bars are 95% CI.

In terms of the null effects, using the Wetzels and Wagenmakers (2012) criteria for Bayes

TE D

factors, anecdotal evidence was found in support of the null hypothesis for the effect of immersion on simulator sickness, and visual recall scores. Substantial evidence for the null

EP

hypothesis was found for the effect of immersion on changes in negative affect and subjectmatter interest. Further, substantial evidence for the null hypothesis was also found for the effect

AC C

of place illusion on changes in simulator sickness as well as the interaction between immersion and place illusion for changes in simulator sickness, and visual recall scores. From the data observed no evidence was found to suggest our conditions impacted these variables. However, we note that for changes in negative affect the phone condition was the only condition not to show pre-to-post decreases in negative affect scores which led to a lack of non-significant findings amongst the condition. Further for simulator sickness scores, the Google Cardboard was the only device to show significant increases post-simulation, which may indicate a precursor to

ACCEPTED MANUSCRIPT 24 a negative experience (Figure 2). To probe this finding a little deeper we examined the correlations between SSQ scores and our dependent measures in the Cardboard condition using a one-tailed test with the hypothesis that greater simulator sickness will lead to a negative

RI PT

experience. Greater SSQ scores were significantly associated with a decreased change in positive

AC C

EP

TE D

M AN U

SC

affect (r = -.37, p = .017) and decreased visual recall scores (r = -.36, p = .017).

ACCEPTED MANUSCRIPT

25

Table 4. Regression results for the main effect of levels of immersion, place illusion, and the place illusion x level of immersion interaction.

R

R2

F

df

p

p

Simulator Sickness (SSQ)

.14

.02

0.94

3,132

.42

Positive Affect (PANAS)

.36

.13

6.65

3,132

<.001

Negative Affect (PANAS)

.16

.03

1.15

3,132

.33

Auditory Recall Score

.41

.17

9.11

3,132

<.001

Visual Recall Score

.19

.03

1.55

3,132

.20

VR Expectations

.34

.12

5.81

3,132

<.001

2.01

.023

.18

Subject-Matter Interest

.35

.12

6.16

3,132

<.001

2.22

.014

.47

2.24

<.001

.19

.41

AC C

EP

4.81

.014

β

BF10

BF01

0.43

2.31

21.9

0.05

0.18

5.7

t

P

1.89

.03

-1.67

.049

β

.40

195.53

0.005

0.61

1.63

1.15

0.87

-1.3

.10

-.28

0.29

3.5

2.22

.014

.47

TE D

t

SC

Variable

Main Effect of Place Illusion

RI PT

Main Effect of Immersion

M AN U

Overall Model

-.34

BF10

place illusion x Immersion Interaction BF01

t

p

0.14

7.1

40.21

0.025

0.64

1.56

0.6

1.69

0.48

2.08

7.11

0.14

2.52

.007

494.07

0.002

-0.65

.26

-0.85

.20

β

-.17

BF10

BF01

0.14

6.96

3.08 0.09

0.43

.34

.09

0.74

0.33 2.56 1.36

0.32

3.18

.52

44.19

0.023

-.13

43.27

0.023

ACCEPTED MANUSCRIPT 26 We found anecdotal evidence for the null hypothesis that there were no differences between conditions for simulator sickness. However, while we found both the DK2 (MSD-change = 0.11, SDSD-change = 0.67) and CV1 (MSD-change = 0.09, SDSD-change = 0.49) conditions had relatively

RI PT

little change from baseline while the Google Cardboard (MSD-change = 0.66, SDSD-change = 1.58) and Phone (MSD-change = 0.35, SDSD-change = 1.15) conditions induced much greater increases in

simulator sickness (Figure 3). Further, only the Cardboard condition had a significant increase in

SC

simulator sickness pre-to-post simulation. Substantial evidence in favor of the null hypothesis was found for negative affect, visual recall score, VR expectations, and subject-matter interest

M AN U

indicating there were no differences between conditions for these variables. In spite of this lack of difference between conditions, we found significant decreases in negative affect in the Cardboard, DK2, and CV1 conditions. We also found a small but significant change in VR expectations for the DK2 and CV1 conditions. Finally, in all conditions, watching the virtual tour

interest). 5. Discussion

TE D

of the ISS increased their interest in learning more about the ISS and NASA (i.e. subject-matter

EP

VR learning experiences promise to transport users to far off worlds where they can have novel and exciting experiences, making people part of the action in a way that has not been

AC C

possible before. This promise makes VR technology exciting to educators and students alike to create memorable learning experiences. However, less is known about the effectiveness of the more accessible 360° video-based learning experiences presented using VR technologies. To alleviate this knowledge gap, the objective of the current study was to test the effect of immersion and place illusion on increasing student interest, arousal, and content knowledge using this media. Thus, we tested not only if learners performed better using more immersive

ACCEPTED MANUSCRIPT 27 technologies but were more engaged in the experience. Overall our findings were consistent with previous instructional theory (i.e., Cone of Experience; Situated Learning) and the notion that 360° videos displayed with highly immersive technology support positive learning experiences

RI PT

by increasing student’s interest in and knowledge of the subject-matter.

5.1 Content knowledge

SC

We found a significant effect of level of immersion on auditory questions but not visual questions. Participants in the most immersive condition, the CV1, had a mean score of 2.29

M AN U

points higher than the DK2 condition indicating they remembered more verbally presented information from the narrator in the video. This finding furthers the literature on content learning in VRLEs and supports previous findings that immersive virtual reality supports knowledge gains in VR (Cobb et al., 2009; Lee et al., 2010; Webster, 2016). Further, our study varied in two

TE D

important ways from these previous studies. First, they did not systematically vary the level of immersion. For example, in Webster (2016) and Cobb et al. (2009) VR groups were compared with standard lecture groups. Further, our study used an unstructured learning environment that

EP

was not specifically designed to teach knowledge and skill. Thus, our findings indicate a learning advantage for unstructured 360° videos, but only highly immersive ones.

AC C

However, it is also important to mention a few potential limitations of our findings in this

area. First, one reason for the dissociation between audio and visual questions may be that the visual questions were overall more difficult leading to a floor effect. One reason this may be the case is that the mean visual recall score remained consistent across conditions. Thus, instead our finding may represent an overall increase in recall score as immersion increased instead of a dissociation between audio and visual questions. Another potential limitation may be the lack of

ACCEPTED MANUSCRIPT 28 knowledge pretest. Because we do not know individuals’ prior knowledge before watching the video it may be that more knowledgeable people were assigned to the CV1 condition. However, we feel that possibility is unlikely because participants were randomized to conditions and were

RI PT

selected based on self-reports that they were unfamiliar with the subject-matter. Further, all participants reported similar amounts of subject-matter interest before watching the video.

SC

5.2 Positive and negative affect

We found significant main effects of both the level of immersion and place illusion on

M AN U

pre-to-post standardized change in positive affect. For immersion, the Cardboard and CV1 conditions showed significant increases in positive affect. Because the phone condition was our non-immersive active control we expected to find the least change in this condition, which we observed. However, contrary to our hypothesis, we did not observe a linear increase in positive

TE D

affect as level of immersion increased, which may indicate other non-measured effects impacted positive affect in this condition. In terms of place illusion, we found a positive association between place illusion and positive affect following the simulation, but not pre-simulation

EP

scores. This may indicate feeling present in the virtual environment led to increases in positive affect rather than the alternative that having a greater pre-study positive affect led to changes in

AC C

the ability to achieve place illusion.

Contrary to Riva et al. (2007) we did not find an association between negative affect

change scores and place illusion. However, we note that all conditions except the phone displayed reduced negative affect which is an important finding because unlike other studies (e.g., Baños et al., 2006) our videos were not specifically designed to include an emotional

ACCEPTED MANUSCRIPT 29 reaction. It is unclear if this represented a general effect of VR technology or a specific quality of

5.3 VR expectations and subject-matter interest

RI PT

the video we used in this study.

Our analysis revealed a significant small effect of level of immersion but not place

illusion on changes in VR expectations pre-to-post simulation. Increases in VR expectations

SC

were observed in all VR conditions, but not in the phone control group. This group difference was driven by the CV1 group which showed the greatest increase. Overall only the DK2 and

M AN U

CV1 conditions resulted in a significant positive increase following the video which may indicate that higher levels of immersion are required for individuals to engage people in VR experiences. While no significant differences were found for place illusion, the place illusion composite variable was more strongly correlated with post and pre-to-post standardized change

TE D

scores which led to a significant interaction between both of our conditions. When watching a 360° video on non-immersive technology, feeling present may distract the user or lead to an overall negative experience. On the other hand, for highly immersive VR technology the novelty

EP

of feeling place illusion accounted for the effect of condition on increasing individuals’ expectations of the technology. This finding may also suggest that users may only accept

AC C

watching 360° videos on immersive VR technologies if they activate a sense of place illusion. Muetterlein and Hess (2017) supported this idea by arguing for the inclusion of immersion and place illusion to a version of the technology and acceptance model for virtual reality. One reason for this could be the fact that although immersive VR is now widely available it still requires significant monetary investment. If the phone condition is viewed as “just as good” it may deter interest in the more expensive options.

ACCEPTED MANUSCRIPT 30 In terms of subject-matter interest, significant effects of both level of immersion and place illusion were found using the regression model and post-hoc t-tests. Our analysis of the Bayes factors showed support for the effect of place illusion, but not of immersion. This

RI PT

conflicting evidence may be due to the null effect of place illusion in the cardboard condition while all others followed the hypothesized relationship. While activating place illusion in the phone condition reduced expectations of VR, it significantly increased participants’ interest of

SC

the subject-matter. The next greatest difference was found in the CV1 condition, while the error bars in the Cardboard and DK2 conditions overlapped between place illusion groups. Taken

M AN U

together, it suggests that place illusion may benefit non-immersive technology by increasing learners’ engagement and reducing outside distractions not otherwise possible. For highly immersive technology, place illusion may enhance the benefits of what immersion offers the user, similar to a flow state. Similar findings have been echoed by a meta-analysis investigating

TE D

the benefits of VR on pain management. In this review, place illusion was found to be successful in distracting users from the real world and shifting their focus of attention to the content in the virtual environment (Gupta, Scott, & Dukewich, 2018). All of these data support previous

EP

research that immersive experiences would increase learner engagement and improve learning outcomes (O’Brien & Toms, 2008; Snelling, 2016) especially for the CV1 condition.

AC C

5.4 Simulator sickness

Finally, we investigated the effect of simulator sickness on reducing the positive effects

of the VR devices we tested. Overall, our regression model and Bayesian analysis were in agreement that simulation sickness was not a concern in the current study. One key reason for the lack of significance of this effect was that the phone, DK2, and CV1 conditions failed to show a significant increase pre-to-post standardized change in simulator sickness following

ACCEPTED MANUSCRIPT 31 exposure to the 360° video. The Google Cardboard on the other hand, did show a significant increase in simulator sickness following the educational video. This supports a concern found in the literature for widely available and cheap VR devices with limited immersion (Smith, 2015).

RI PT

After examining the raw change in simulator sickness on the SSQ subscales, we found the

Cardboard had an average increase in simulator sickness of 91.59 with the greatest increases found on the disorientation and oculomotor subscales. Previous findings for commercial flight

SC

simulators have found that an average total score increase greater than 20 points indicates a severe risk of motion sickness (Kennedy et al., 2003). This indicates that participants developed

M AN U

moderate to severe symptoms in both the phone and Cardboard conditions and were at risk of becoming sick in the DK2 condition. These findings present a concern to positive learning outcomes when watching 360° videos with these devices as previous research has indicated that becoming sick will increase feelings of misery and decrease performance on tasks that require

TE D

sustained attention (Valk, Munnoch, & Bos, 2008) and motion sickness will impair short term memory performance (Dahlman, Jjörs, Lindström, Ledin, & Falkmer, 2009). However, after correcting for the variance in pre-simulation scores using the standardized pre-to-post change the

EP

cardboard was the only condition found to be at risk for getting our participants sick. However, because the Cardboard condition as a whole did not support previous research on the negative

AC C

effects of simulator sickness we delved more deeply into this condition. In this condition, we found that greater standardized pre-to-post-simulation increases in simulator sickness were associated with significant moderate decreases in positive affect and lower visual question recall scores. However, these associations were driven by a small percentage of users who became sick in this condition. Only 7 users or 20.59% of users of the Google Cardboard reported a post simulator total SSQ score of greater than 20 points. Thus, our results only partially supported

ACCEPTED MANUSCRIPT 32 Smith (2015) and indicate that lower immersion devices may pose a greater risk of simulator sickness, but not everyone who uses such a device is at risk. To combat this risk, we recommend that learners using similar devices be screened for their susceptibility for motion sickness prior to

RI PT

exposure (Golding, 2006). However, to minimize the risk to all learners, it is recommended that

sickness. 6. Conclusions and Future Research

SC

devices for educational experiences meet minimum specifications to reduce the risk of simulator

In the current study, we found that feeling present in the virtual environment was

M AN U

associated with generating increased interest in the subject-matter and greater positive affect. Thus, VRLEs that activate place illusion may be more likely to retain learners for a longer period. Further, increases in subject-matter not only occurred for immersive VR technology, but even for the non-immersive phone condition. Thus, individual differences in the ability to feel

TE D

present may be an important predictor of success using online and VR learning environments. However, feelings of place illusion in VR has been argued to be a novel and strange experience for users and repeated exposure may be needed for users to calibrate themselves to what they are

EP

experiencing in the virtual environment. Therefore, one limitation in the current study is that we only tested novice users of VR and due to the difficulty of measuring subjective feelings of place

AC C

illusion we used post-hoc indices which may have induced a false memory affecting their responses (Slater, Lotto, Arnold, & Sanchez-Vives, 2009). While this is one potential limitation of the current study, our observation that increased place illusion was reported as we parametrically increased immersion adds support that our measure of place illusion was valid. Additionally, one participant in the Cardboard condition stated, “that felt strange, like I was there” when they finished the video before completing the place illusion questionnaire. Also, our

ACCEPTED MANUSCRIPT 33 finding that the mean place illusion rating was 0.6 standard deviations below the study mean for the phone condition is consistent with theories of place illusion and indicating an association with immersion (e.g., Slater, 2009).

RI PT

Based on the results of the current study, we conclude that minimizing simulator sickness while finding ways to create immersive experiences for learners is key to developing effective educational VR experiences. Additionally, eliciting the subjective feelings of place illusion

M AN U

References

SC

should also be a goal of these VR experiences designed for novice users of VR.

Adelman, J. S., & Estes, Z. (2013). Emotion and memory: A recognition advantage for positive and negative words independent of arousal. Cognition, 129(3), 530-535. Ahn, S. J. G., Bailenson, J. N., & Park, D. (2014). Short-and long-term effects of embodied

TE D

experiences in immersive virtual environments on environmental locus of control and behavior. Computers in Human Behavior, 39, 235-245. Bailey, J. H., & Witmer, B. G. (1994, October). Learning and transfer of spatial knowledge in a

EP

virtual environment. In Proceedings of the Human Factors and Ergonomics Society

Annual Meeting 38, 1158-1162, Los Angeles, CA.

AC C

Baukal, C. E., Ausburn, F. B., & Ausburn, L. J. (2013). A Proposed Multimedia Cone of Abstraction: Updating a Classic Instructional Design Theory. Journal of Educational

Technology, 9(4), 15-24.

Baños, R., Liaño, V., Botella, C., Alcañiz, M., Guerrero, B., & Rey, B. (2006). Changing induced moods via virtual reality. Persuasive Technology, 7-15.

ACCEPTED MANUSCRIPT 34 Bessa, M., Melo, M., Narciso, D., Barbosa, L., & Vasconcelos-Raposo, J. (2016, September). Does 3D 360 video enhance user's VR experience? An Evaluation Study. In Proceedings

Spain.

RI PT

of the XVII International Conference on Human Computer Interaction, Salamanca,

Bulu, S. T. (2012). Place presence, social presence, co-presence, and satisfaction in virtual worlds. Computers & Education, 58(1), 154-161.

Bilimleri Fakultesi Dergisi, 44(1), 91.

SC

Çaliskan, O. (2011). Virtual field trips in education of earth and environmental sciences. Egitim

M AN U

Cellan-Jones, R. (2016) 2016: the year when VR goes from virtual to reality. BBC Technology

News. Retrieved from: http://www.bbc.com/news/technology-35205783. Ciobanu, M. (2016). The New York Times aims to make 360-degree video part of the ‘daily news report’. Retrieved from: https://www.journalism.co.uk/news/the-new-york-times-

TE D

aims-to-make-360-degree-video-part-of-the-daily-news-report-/s2/a688796/. Cobb, S., Heaney, R., Corcoran, O., & Henderson-Begg, S. (2009). The learning gains and student perceptions of a Second Life virtual lab. Bioscience Education, 13(1), 1-9.

EP

Dahlman, J., Sjörs, A., Lindström, J., Ledin, T., & Falkmer, T. (2009). Performance and autonomic responses during motion sickness. Human factors. 51, 56-66.

AC C

Dale, E. (1946). The cone of experience. Audio-visual methods in teaching, New York: Dryden Press 37-52.

Dale, E. (1954). Audiovisual methods in teaching (Rev. ed.). New York: Holt, Rinehart & Winston.

Dale, E. (1969). Audiovisual methods in teaching. New York: Dryden Press.

ACCEPTED MANUSCRIPT 35 Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66-69. Dede, C. J., Jacobson, J., & Richards, J. (2017). Introduction: virtual, augmented, and mixed

16). Springer, Singapore.

RI PT

realities in education. In Virtual, Augmented, and Mixed Realities in Education (pp. 1-

Diemer, J., Alpers, G. W., Peperkorn, H. M., Shiban, Y., & Mühlberger, A. (2015). The impact

reality. Frontiers in psychology, 6(26), 1-9.

SC

of perception and presence on emotional reactions: a review of research in virtual

Dinh, H. Q., Walker, N., Song, C., Kobayashi, A., & Hodges, L. F. (1999). Evaluating the

M AN U

importance of multi-sensory input on memory and the sense of presence in virtual environments. In Proceedings of IEEE virtual reality, 222–228. Golding, J. F. (2006). Predicting individual differences in motion sickness susceptibility by questionnaire. Personality and Individual Differences, 41(2), 237-248.

TE D

Gupta, A., Scott, K., & Dukewich, M. (2018). Innovative Technology Using Virtual Reality in the Treatment of Pain: Does It Reduce Pain via Distraction, or Is There More to It?. Pain

Medicine, 19(1), 151-159.

EP

Hollis, R. B., & Was, C. A. (2016). Mind wandering, control failures, and social media distractions in online learning. Learning and Instruction, 42, 104-112.

AC C

Howard, G. S., Maxwell, S. E., & Fleming, K. J. (2000). The proof of the pudding: an illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis. Psychological Methods, 5(3), 315-332.

Lackner, J. R. (2014). Motion sickness: more than nausea and vomiting. Experimental brain

research, 232(8), 2493-2510.

ACCEPTED MANUSCRIPT 36 Lawson, B. D. (2014). Motion sickness symptomatology and origins. In Handbook of Virtual Environment: Design, implementation, and applications, 2nd ed. (pp. 532-587). Boca Raton, FL: CRC Press.

RI PT

Lee, E. A. L., Wong, K. W., & Fung, C. C. (2010). How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach. Computers &

Education, 55(4), 1424-1442.

SC

Just Another Statistics program (JASP; Version 0.8.5) (2017) [computer software]. Amsterdam, The Netherlands: Wagenmakers, E.J. Retrieved from https://jasp-stats.org/.

M AN U

Kennedy, R. S., Drexler, J. M., Compton, D. E., Stanney, K. M., Lanham, D. S., & Harm, D. L. (2003). Configural Scoring of Simulator Sickness, Cybersickness and Space Adaptation Syndrome: Similarities and Differences. In Hettinger, L. J., & Haas, M. W. (Eds.).

(2003). Virtual and adaptive environments: Applications, implications, and human

TE D

performance issues. CRC Press., 247-277.

Kennedy, R. S., Lane, N. E., Berbaum, K. S., & Lilienthal, M. G. (1993). Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. The international

EP

journal of aviation psychology, 3(3), 203-220. Kober, S. E., & Neuper, C. (2013). Personality and presence in virtual reality: does their

AC C

relationship depend on the used presence measure?. International Journal of Human-

Computer Interaction, 29(1), 13-25.

Mania, K., & Chalmers, A. (2001). The effects of levels of immersion on memory and presence in virtual environments: A reality centered approach. CyberPsychology & Behavior, 4(2),

247-264.

ACCEPTED MANUSCRIPT 37 Masson, M. E. (2011). A tutorial on a practical Bayesian alternative to null-hypothesis significance testing. Behavior research methods, 43(3), 679-690.

https://www.youtube.com/watch?v=678XGgNv-h4.

RI PT

Manley, S. (2016). VR Guide to the International Space Station. Retrieved from:

Mütterlein, J., & Hess, T. (2017, August). Immersion, Presence, Interactivity: Towards a Joint Understanding of Factors Influencing Virtual Reality Acceptance and Use. Paper

SC

presented at the meeting of the Association for Information Systems, Boston, MA.

O'Brien, H. L., & Toms, E. G. (2008). What is user engagement? A conceptual framework for

M AN U

defining user engagement with technology. Journal of the Association for Information

Science and Technology, 59(6), 938-955.

Papadakis, G., Mania, K., Coxon, M., & Koutroulis, E. (2011, December). The effect of tracking delay on awareness states in immersive virtual environments: an initial exploration.

TE D

In Proceedings of the 10th International Conference on Virtual Reality Continuum and

Its Applications in Industry, 475-482.

Petkova, V. I., & Ehrsson, H. H. (2008). If I were you: perceptual illusion of body

EP

swapping. PloS one, 3(12), e3832-e3841. Riva, G., Mantovani, F., Capideville, C. S., Preziosa, A., Morganti, F., Villani, D., ... & Alcañiz,

AC C

M. (2007). Affective interactions using virtual reality: the link between presence and emotions. CyberPsychology & Behavior, 10(1), 45-56.

Rouder, J. N., Morey, R. D., Verhagen, J., Swagman, A. R., & Wagenmakers, E. J. (2016). Bayesian analysis of factorial designs. Psychological Methods, 22(2), 304-321.

ACCEPTED MANUSCRIPT 38 Seymour, N. E., Gallagher, A. G., Roman, S. A., O’brien, M. K., Bansal, V. K., Andersen, D. K., & Satava, R. M. (2002). Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Annals of surgery, 236(4), 458-464.

RI PT

Sheridan, T. B. (2016). Recollections on Presence beginnings, and some challenges for

augmented and virtual reality. Presence: Teleoperators and Virtual Environments, 25(1), 75-77.

SC

Slater, M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society of London B:

M AN U

Biological Sciences, 364(1535), 3549-3557.

Slater, M., Lotto, B., Arnold, M. M., & Sanchez-Vives, M. V. (2009). How we experience immersive virtual environments: the concept of presence and its measurement. Anuario

de psicología, 40(2).

TE D

Slater, M., Usoh, M., & Steed, A. (1994). Depth of presence in virtual environments. Presence:

Teleoperators & Virtual Environments, 3(2), 130-144. Smith, W. (2015). Stop Calling Google Cardboard’s 360-degree Video ‘VR’ Wired. Retrieved

EP

from: http://www.wired.com/2015/11/360-video-isntvirtual-reality/ Snelling, J. (2016). Virtual Reality in K-12 Education: How Helpful Is it? Converge. Retrieved

AC C

from: http://www.centerdigitaled.com/k-12/Virtual-Reality-in-K-12-Education-Is-ItReally-Helpful.html.

Stevens, J. A., & Kincaid, J. P. (2015). The Relationship between Presence and Performance in Virtual Simulation Training. Open Journal of Modelling and Simulation, 3(02), 41.

Usoh, M., Catena, E., Arman, S., & Slater, M. (2000). Using presence questionnaires in reality. Presence: Teleoperators and Virtual Environments, 9(5), 497-503.

ACCEPTED MANUSCRIPT 39 Tüzün, H., & Özdinç, F. (2016). The effects of 3D multi-user virtual environments on freshmen university students' conceptual and spatial learning and presence in departmental orientation. Computers & Education, 94, 228-240.

RI PT

Valk P. J. L., Munnoch K., Bos J. E. (2008, January). Human performance at sea: Effects on vigilance and complex task performance. Paper presented at the Pacific 2008 International Maritime Conference, Sydney, Australia.

SC

Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures

psychology, 54(6), 1063.

M AN U

of positive and negative affect: the PANAS scales. Journal of personality and social

Webster, R. (2016). Declarative knowledge acquisition in immersive virtual learning environments. Interactive Learning Environments, 24(6), 1319-1333. Wetzels, R., & Wagenmakers, E. J. (2012). A default Bayesian hypothesis test for correlations

TE D

and partial correlations. Psychonomic bulletin & review, 19(6), 1057-1064. Witmer, B. G., Bailey, J. H., Knerr, B. W., & Parsons, K. C. (1996). Virtual spaces and real world places: transfer of route knowledge. International Journal of Human-Computer

EP

Studies, 45(4), 413-428.

Yoder, M. (2016). Reality revisited: Getting started with virtual reality in your classroom.

AC C

International Society for Teaching and Education (ISTE). Retrieved from:

https://www.iste.org/explore/articleDetail?articleid=715&category=In-the-classroom

Zhang, D., Zhou, L., Briggs, R. O., & Nunamaker, J. F. (2006). Instructional video in e-learning: Assessing the impact of interactive video on learning effectiveness. Information &

Management, 43(1), 15-27.

ACCEPTED MANUSCRIPT

AC C

EP

TE D

M AN U

SC

RI PT

40

ACCEPTED MANUSCRIPT

AC C

EP

TE D

M AN U

SC

RI PT

Highlights • Videos were more effective when watched on the most immersive technology • Feeling present led to a greater desire to learn about the video’s subject-matter • Greater simulator sickness was associated with decreased learning outcomes