Seeing possibilities for action: Orienting and exploratory behaviors in VR

Accepted Manuscript Seeing Possibilities for Action: Orienting and Exploratory Behaviors in VR

Joomi Lee, Allison Eden, David Ewoldsen, David Beyea, Sanguk Lee PII:

S0747-5632(19)30136-0

DOI:

10.1016/j.chb.2019.03.040

Reference:

CHB 5977

To appear in:

Computers in Human Behavior

Received Date:

25 August 2018

Accepted Date:

31 March 2019

Please cite this article as: Joomi Lee, Allison Eden, David Ewoldsen, David Beyea, Sanguk Lee, Seeing Possibilities for Action: Orienting and Exploratory Behaviors in VR, Computers in Human Behavior (2019), doi: 10.1016/j.chb.2019.03.040

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

ACCEPTED MANUSCRIPT Running Head: ORIENTING IN VIRTUAL ENVIRONMENTS

Seeing Possibilities for Action: Orienting and Exploratory Behaviors in VR

Joomi Lee a *, Allison Eden a, David Ewoldsen b, David Beyea a, and Sanguk Lee a a Department

of Communication, Michigan State University, Department of Communication,

Michigan State University, 404 Wilson Road, East Lansing, MI 48824, United States b Department

of Media and Information, Michigan State University, 404 Wilson Road, East

Lansing, MI 48824, United States

Paper submitted to Computers in Human Behavior August 2018

Declarations of interest: none * Corresponding author email address: [email protected] (J. Lee)

1

ACCEPTED MANUSCRIPT Running head: ORIENTING IN VIRTUAL ENVIRONMENTS

1

Seeing Possibilities for Action: Orienting and Exploratory Behaviors in VR

Abstract This study aims to apply the concept of affordances (J. Gibson, 1979) to the context of immersive virtual reality (VR). As a first step, this study investigates users’ visual orienting and exploratory behaviors while they navigate a virtual house indexed by their gaze on virtual objects, viewing time, and time spent in different rooms in the environment. A content analysis of participants’ (N = 22) video recordings of VR exploration was conducted based on second-bysecond time frame. Results indicate that participants tend to orient to the virtual objects when they display novel or unusual information. Also, users’ exploratory behaviors were focused on the virtual objects that are related to available actions of the given virtual environment. Keywords: affordances; attention; virtual reality

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

2

Seeing Possibilities for Action: Orienting and Exploratory Behaviors in VR The current study aims to understand virtual environment (VE) users’ general and basic behaviors within an immersive virtual reality (VR). VR is gaining wide scholarly interest given its potential for providing sensory-rich, immersive media for exploring novel worlds and environments. An immersive VR possesses unique sensory features distinguished from previous types of screen media such as television. Specifically, an immersive VR replaces the whole visual field while previous screen media takes portions of the visual field (Slater & SanchesVivis, 2016). It is important to understand the extent users perceptually and behaviorally engage in the content of virtual environments—how do users interact with and move within a virtual environment? What elements of the VE are critical for eliciting presence (i.e. sense of “being there”, Lee, 2004; Wirth et al., 2007) or for encouraging specific behaviors by users? There have been multiple studies on effects of virtual environments focusing on VR effects, for example, prosocial outcomes (Ahn, Bailenson, & Park, 2014), health (Fox & Bailenson, 2009), or educational (Freina & Ott, 2015) interventions for users. Yet, studies on foundational user behaviors in VR are relatively lacking. As an exploratory step to understand how users initially interact with the virtual environment, this study intends to incorporate J. Gibson’s (1966; 1967; 1977; 1979) ecological perception theory and the concept of affordances. In general, the ecological perspective explains perception of the environment in terms of the observer’s overtime interaction with the meaningful properties of the environment through actions. The meaningful information to be perceived is called an affordance, defined as a certain action capability that can be done with, or on, an object within the environment. This study assumes that the development of behaviors in VR is characterized by perceiving, learning, and using affordances in the virtual environment. Given that perception of

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

3

affordances is tied to allocating attention to parts of the external world, this study investigates VR users’ orienting of attention and attention allocation to environmental features and exploratory locomotion while they navigate a virtual environment. The next part of this paper introduces literature outlining J. Gibson’s fundamental assumptions about ecological perception, and connects them to the context of VE exploration. Based on this literature, this paper proposes a set of hypotheses related to orienting and attention allocation in the VE and presents an analysis of video recordings of participants’ VE exploration. Ecological Perception and Affordances According to J. Gibson (1979), perception is based on adaptive sensory and perceptual systems mediated by one’s actions. Perception and action work by continuously making “orienting and exploratory adjustments of the perceptual organs to resonate in a particular way when a distant kind of information is picked up” (J. Gibson, 1967, p. 163). In the ecological perspective, the environment consists of surfaces, substances, objects (attached and detached), animals, and medium (i.e. air, water) and the information from these components of the environment can be thought of as invariant (stable properties) and variant (unstable, changing properties over time). These invariant or variant properties of the environment and some combination of them are directly picked up by our perceptual system in regard to certain action capabilities, which are called affordances (J. Gibson, 1979). However, the concept of affordances is not just defined by properties of environment; rather, perception of affordances emerges from the relationship between the observer’s motives and needs on one hand, and the properties of the environment on the other hand. In other words, people do not perceive what things are separately from what they can do with the things in the environment at a given time and space. For example, when we see layouts and textures of

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

4

surfaces, we perceive affordances for support and modes of locomotion, and when we see other animals and humans, we see affordances of social interaction or comfort. The concept of affordances and ecological perception provides a different way to explain how we make sense of our environment or situation by performing orienting and exploratory behaviors. During early stages of development, animals explore the environment to discover optimal ways of adjusting perceptual systems and to learn basic behaviors such as locomotion (e.g. Adolph, Eppler, & E. Gibson, 1993). Developmental studies of perceptual learning indicate that exploration by looking at objects becomes gradually elaborated as infants develop the ability to control eye and head orientations. At a later stage in development when infants develop the ability to stretch an arm toward an object and grab it, they gain a set of new action capabilities to learn object properties and affordances (E. Gibson, 1988; E. Gibson, & Pick, 2000). Even skilled animals explore the environment by using information based on affordances to suggest possibilities for higher-order or more meaningful actions (E. Gibson, 1988). Critically, visual orienting and exploratory behaviors are necessary for obtaining information about environmental objects and their affordances (J. Gibson, 1966). Orienting and Exploratory Behaviors Orienting is mostly about “where” or “on what” to focus attention in an environment (P. Lang, Simons, & Balaban, 1997; Posner, 1980). Orienting responses to external stimuli are made by adjusting of perceptual or sensory organs to a source of information. Given that there is a limited capacity in cognitive resources, orienting allows selection of information from a complex environment, as well as facilitation of processing selected information via attention allocation. Physiological studies have observed that involuntary orienting responses occurs with the presence of novel, motivationally relevant, and signaling (i.e. significance/relevance of the

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

5

stimulus to the perceiver) stimulus as indexed by cardiac deceleration (Graham, 1979; P. Lang et al., 1997; Bradley, 2009), demonstrating the biological grounding of automatic orienting responses. Attention allocation can happen in either an involuntary or controlled manner: involuntary attention allocation is initiated by automatic orienting responses to aspects of the environment, thus occurs quickly and unconsciously (P. Lang et al., 1997). Controlled attention allocation is made by users’ conscious intention (Öhman, 1979). Despite the different functional and neurological processes involved in orienting responses, attention allocation during exploration involves a combination of both automatic and controlled attentional activities. That is, distinctive kinds of information are automatically picked up while moving and acting on the environment, which provides clues for attention allocation. In case of visual attention, orienting and information selection can be indexed by duration and shifts of eye fixation (Posner, 1980). J. Gibson’s (1979) approach further suggests that visual perception involves the eye, head, brain, and body that are all working together so that meaningful information (e.g. affordances) is picked up from continuously changing optic flow corresponding to the movements. In other words, rotation of the eyes will provide meaningful information for actions only when coupled with head and body movements in a 3D space in order to identify which affordance an observer perceives at a moment (Albert et al., 2005). Learning Affordances and Implications to the VE Inhabiting a new virtual environment can be compared to learning affordances during the early development of infants in the physical world. Virtual environments and videogames are operated by the mechanics of the system that define and constrain users’ actions (Sherry, 2014). Because these mechanics usually offer only some portion of possible behaviors in the real world,

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

6

users’ actions and behaviors in VEs are delimited compared to those in the real world. However, there may be action capabilities that are not common or impossible given the rules in the physical world, such as having different modes of locomotion (teleport, flying) and use of superpower. Accordingly, newcomers to VEs need to make sense of the environment first by actively interacting with and perceiving constraints of the VE. Researchers note that learning affordances of the VE to achieve individual goals and overcome challenges can constitute enjoyment when playing videogames (Sherry, Lucas, Greenberg, & Lachlan, 2006; Boyan & Sherry, 2011). Also, because users already have developed their knowledge of affordances in the physical world for most basic actions (e.g. locomoting, reaching, grabbing, etc.), the process involves mapping their accumulated landscape of affordances in the real world to perception of the mediated environment (Skalski, Tamborini, Shelton, Buncher, & Lindmark, 2011). Applying the ecological perspective and the concept of affordances to VEs, A. Lang and colleagues (A. Lang, 2014; A. Lang et al., 2018) conceptualize behavioral outcomes within a VE as a function of individuals’ initial condition, VE affordances, and their interaction with the environment. Using the videogame Grand Theft Auto, in which available objects and in-game goals promote various criminal and violent behaviors, A. Lang et al. (2018), investigated development and stabilization of players’ in-game behaviors. Their analysis revealed that although behavioral trajectories of all participants were different, some participants showed more and more behaviors they would not normally do (e.g., reckless driving, dying, killing), while other participants stabilized on performing non-violent and everyday life behaviors. Results from the study imply that while players’ initial actions were guided by their initial conditions (e.g., familiarity with the game), actions performed during the later part of gameplay were shaped by the affordances and actions they learned during the initial stage.

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

7

Thus, by focusing on development of behaviors within a VE and the potential elicitors of those behaviors, we may find a clue for understanding how and why people show diverse behavioral patterns interacting with a fully immersive VE, and the affordances they perceive from the visual information present in the VE. The current study is concerned with the initial interaction with the VE in terms of users’ orienting behaviors and development of their behavioral patterns. In particular, what kinds of environmental features do people orient to? And what aspects of orienting-inducing environmental features will lead to further exploratory behaviors, such as close investigation and approach behaviors? Orienting Inducing Features in the VE Orienting-inducing attributes considered in this study are novelty and functionality of virtual objects. Orienting to novel information can be thought as investigatory reaction to figure out “what it is” (Bradley, 2009). In VE where presentation of various features may not be congruent with the real-world-expectation, it is likely that novel or unusual parts of visual information will demand automatic investigatory attention. To some extent, on the other hand, introduction to any object or feature of a new environment (i.e. a VE that requires to use the head-mount display and controllers) can be novel to the users, demanding perceptual learning of its affordances (Albert et al., 2005). Therefore, this study asks if functionality (affordances) of virtual objects attracts users’ orienting response and attention allocation. Previous study has demonstrated that people preferentially devote early stage visual attention to affording (graspable) object in reachable distance compared to nonaffording but reachable objects (Garrido-Vásquez & Schubö, 2014). This initial attention to objects with functionality is likely to guide users’ successive exploration, such as prolonged investigation and approach behaviors. Furthermore, A. Lang and colleagues (2018) theorize that

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

8

VE users who are not accustomed to the new world would first attempt to perform actions they normally do in the real world (e.g., walking, using everyday objects), and then adapt their behaviors based on actual affordances in the VE as they explore and learn (e.g., using weapons in Grand Theft Auto). That said, users will develop their attention and behaviors focused on what the VE actually offers overtime, and this should be evident in gaze length or time spent in areas attracting orienting behavior. Our conceptualization of novelty and functionality of virtual objects was further inspired by A. Lang and Bailey’s (2015) typology of environmental attributes that are likely to be selected for attention allocation; stability, imminence, motivational relevance, and task relevance. Stability indicates the extent to which an environment feature remains relatively invariant or changing over time. In general, changing entities such as animated objects are more likely to be attended because invariant information, once perceived automatically, does not require much cognitive efforts to be processed. Imminence is defined in terms of how close or far the observer is to the mediated stimuli. The basic notion is that events or objects close to the observer is more consequential for survival, therefore elicits attention allocation. When thinking about affordances in the context of VE, imminence will be a continuously changing property in accordance with the individual’s self-movement. Motivational relevance is conceptualized as the extent to which the environmental stimuli signal either an opportunity or threat for survival (A. Lang, 2006; A. Lang, 2014; A. Lang & Bailey, 2015), and Task relevance refers to the extent to which information helps or interrupts completion of intended tasks (A. Lang & Bailey, 2015). Each of those dimensions described above are conceptualized based on how different components of human motivational and attentional systems might interact with some combination of these properties in the mediated environments. A. Lang and Bailey (2015)

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

9

investigated users’ encoding and information selection after they viewed video messages that contain varying information in regard to stability, imminence, motivational relevance, and task relevance. Although imminent, unstable (fleeting), and motivationally relevant or task relevant pieces of information are expected to be selected in general, information selection from different combinations of those properties would yield more diverse results. For example, a signal detection analysis revealed that for fleeting information, changes were detected better for imminent (central) than non-imminent (peripheral) information regardless of motivational or task relevance. For stable information, changes in imminent information were detected better when they were not relevant (motivationally and in terms of task) or had only task-relevance. A. Lang and Bailey’s (2015) notion of stability of object properties coupled with imminence can inform media researchers who study media features that attract attention. Perception of invariants such as flatness of the surface or height of the desk does not require excessive allocation of attention unless the objects provide additional information that needs to be processed (e.g., novelty). The notion of imminence also becomes more important in VEs and videogames where users have more autonomy in navigating, exploring, and manipulating the virtual environment, than for traditional mass media settings. In regard to task relevance, users’ primary task in VEs can be either intrinsic or manifest. In videogames for example, intrinsic tasks involve learning of affordances, constraints, and rules of the VE, while manifest tasks are tied to a set of explicit actions as well as skill levels required for a user to achieve a goal, such as finding an object or defeating an enemy (Boyan & Sherry, 2011). When users are free to make their own behavioral choices, their perceived task relevance may have more degrees of freedom, guiding their attention to various offerings of the environment compared to having specific tasks to perform. That said, if users’ task is to simply

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

10

spend some time in a virtual environment, their internal task will likely involve exploration and finding out the affordances of the virtual objects and environment. For example, if the VE is predominantly equipped with capacity to move and collect the objects, users’ attention and behaviors may be focused on moveable objects or those with appropriate size and shape of reaching and grabbing. The Current Study The current study is concerned with identifying the features in the virtual environment toward which users orient and allocate their attention either voluntarily or involuntarily, and in what occasions users show different attention allocation patterns with what they do in the real environment. We assume that entering and spending time in a new VE will posit an inherent task of exploration and learning affordances. Therefore, it is expected that users will orient to virtual objects that signal functions or usages they would normally expect in the real world. However, as an exploratory step, this study uses the VE stimulus that has limited offerings of available behaviors—walking and looking around using a head-mount virtual reality display (HMD)—in a simulated virtual house featuring objects which should be common to most adult humans. Given the limited available actions, we predict that users’ will orient to virtual objects that are more relevant to available behaviors (e.g., objects affording looking behaviors, such as windows and wall paintings) in the virtual environment. The first research question is concerned with characteristics of orienting-inducing stimuli in the VE: RQ1: When users explore a VE, to which environmental and surface features do they orient? The current study tests hypotheses associated with the first question for both room-level and object-level surfaces in order to investigate the global and local orienting patterns. Given

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

11

that the terms affordance and functionality are often used interchangeably in the field of human computer interaction (e.g., Yao, Ma, & Fei-Fei, 2013), we use the term functionality to indicate a broader and looser concept of affordance that does not account for users’ specific skills or capacity. Hypotheses are formulated in terms of both functionality and novelty as potential orienting-inducing features. The functionality hypotheses are in line with previous findings showing that objects offering certain functionality attracts preferential attention allocation (Garrido-Vásquez & Schubö, 2014). H1a: Users will spend more time in the virtual rooms that contain functionality than those that do not. H1b: Users will spend more time viewing virtual objects that contain functionality than those that do not. In terms of stimulus novelty, a basic prediction is that users will orient their attention when the VE contains unstable, new, and variant information that they would not normally expect in the real world. In a VE, there are unlimited possible ways a designer can alter the fabric of perceived reality. However, using A. Lang and Bailey’s (2015) list of attributes, we may consider that items which are unstable, or change in imminence, growing larger or smaller may present new opportunities for threat or promise to an individual. Therefore, in some of the rooms, we included items which are from a normal house (toilet, bed, hanging picture) and effected a change on these items so they are not naturally presented but instead presented in an unrealistic fashion drawing on one of these affordances. For example, the bathroom, which in the “real” version of the house contains a toilet, bathtub, and sink mounted on the floor, in the unreal condition features the tub on the ceiling and the toilet sliding up and down on the wall. We

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

12

predict that shifting the imminence and stability of objects in this environment will require more allocation of resources to process, resulting in increased looking time. Specifically: H2a: Users will spend more time in virtual rooms that have unreal representation of virtual objects than rooms with real representation of virtual objects. H2b: Users will spend more time viewing virtual objects that have unreal representations than those with real representations. As mentioned above, an orienting response involves the initial allocation of attentional resources that are often automatically directed by external inputs. Individuals may or may not allocate attentional resources for further processing the environmental stimuli. Therefore, the following question is asked: RQ2: What aspects of orienting-inducing environmental features will generate persisting attention allocation? Method In order to address the hypotheses and research questions articulated above, this study uses a combined method of laboratory experiment and content analysis. Specifically, this study first recorded individuals’ VE exploration and then a content analysis of users’ behavioral acts was conducted based on second-by-second time frame analysis of eye gaze and user movement in the VE. A complete version of coding scheme for content analysis including coding instruction to determine eye-gaze is available online via Open Source Framework (https://osf.io/jvcsm/?view_only=ff459f83604e4a88a1e53bbd1c65078b). Samples Twenty-two participants were recruited from a university in the Netherlands (male N= 10, female N = 10). The average age of participants was 21.8 (SD = 1.69). Two participants did

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

13

not report gender and age. Participants were mostly from western-culture countries including the Netherlands (N = 5), the United States (N = 5), Canada (N = 1), and other European countries (N = 9). Although IRB approval is not required in Netherlands, participants were provided with and signed a consent form. The anonymity and confidentiality policy were also secured by masking identifiable data and by saving the data under a passworded computer. VE Stimulus and VE Exploration Procedure A two-story virtual house was created using the Unity engine and presented via the Oculus Rift virtual reality system (Oculus Rift, 2016). The virtual house was equipped with normal household items and rooms for a typical private residence. Participants (N = 22) were randomly assigned to a realistic or unrealistic representation condition, and explored the virtual environment for 4-6 minutes. Participants spent about 1-2 minutes on the first floor, and then were transported to the second floor, which varied in terms of objects with real versus unreal displays. Ten participants in the second floor saw realistic room and object conditions, and twelve participants saw unrealistic condition, which included objects that moved or changed size, as well as rooms that were novel or unusual in some way (e.g. filled with snow or water). The number and types of objects and layout of rooms were the same across the two between-subjects conditions. Example screenshots of virtual rooms used in the study are shown in Figure 1 (real representation) and Figure 2 (unreal representation). Video recordings of participants’ VE exploration sessions were coded. Three trained coders content analyzed central features of each second of the video recording for both environmental features and user movements. [Figure 1 here] [Figure 2 here] Reliability Analysis

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

14

Three trained coders double-coded three randomly selected videos (13.6% of the total) in order to assure inter-coder reliability. Coding training took approximately five hours across two weeks. The reliability test was conducted using the ReCal intercoder reliability calculation software (Freelon, 2010). The average reliability coefficient between the coders for rooms was sufficient after the first round of coding (Scott’s Pi = .95). For the variable of viewed object, the coding team enhanced reliability after resolving disagreements among the coders. Eventually, viewed objects reached acceptable average reliability between coders (Scott’s Pi = .81). Then each of the three coders was randomly assigned to code a third of the VE exploration video recordings. Virtual Feature Coding Room/object functionality. Functionality of rooms were categorized as living room, kitchen, dining, hallway, bedroom, bathroom, activity room, and nursery. Virtual objects were categorized referring to J. Gibson’s (1979) categorization of the ecological environment— detached objects (including tools), attached objects, and surfaces. Items were coded based on primary functionality, for example, chairs for sitting, bed for sleeping, musical instruments for playing, and so on. These were further categorized for the final analysis so they can represent characteristics of objects that were actually viewed by participants. The list of objects per each room and object categories are detailed in the coding scheme. Room/object realism. Each of the rooms and objects in the VE were coded for real and unreal representation. As not all the objects in a given room were real or unreal, specific object real/unreal representation for each object was coded as well. Visual Attention Coding

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

15

Room/object coding. In each second of the video, an object was coded as a viewed object when the object was located at the center of participant’s visual field for at least a second. Some additional judgment criteria in reference to the participants’ movement were further clarified during coder training sessions. A viewed object was either big enough in the screen (i.e. majority of visual field), only centered object in the participant’s visual field at a smaller size, or a target of approach movement. Indeed, coders determined viewing orientation based on head movement and head orientation from the video recordings. For example, when the participant’s visual field was moving and a centered object was small at the beginning but increased in size as participant moved toward it, seconds involved in the moving process were considered approaching while orienting to the object. When an object was viewed for only a short amount of time (e.g., 1-2 seconds) and then the visual field switches to another object, it was regarded as orienting response without further attentional processing. When an object was viewed for more than two seconds, its total viewing duration were coded to index the amount of allocated attention. Participants’ time spent in each room or hallway was also coded to index time spent investigating each room. Results Orienting Allocation to Virtual Objects: General Functionality The first set of hypotheses predicted that participants would be more likely to spend a longer time staying in rooms and viewing objects with clear and motivationally relevant functionality. On average, participants spent the majority of their given time in the living room (M = 57.56%), followed by kitchen/dining (M = 27.03%) and entrance (M = 19.41%) on the first floor. On the second floor, participants spent more time in the hallway (M = 33.9%) than rooms

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

16

that have specific/typical use, such as the bathroom (M = 11.65%), bedroom (M = 19.78), activity room (M = 17.81%), and nursery (M = 16.86%). Participants spent 33.3% of their time viewing specific objects (2522 seconds out of 6075 aggregated seconds across all participants). The 10 most frequently viewed objects are shown in Table 1 for the first floor, and Table 2 for the second floor. In general, participants’ visual orienting was primarily focused on windows and larger objects such as desk and bed. [Table 1 here] [Table 2 here] Among the total of 176 possible objects in the VE stimulus, only 118 objects were coded as viewed by at least one participant for at least one second. In order to extract notable patterns of object viewing, we categorized those 118 objects by their common functionality (e.g., food items were categorized as “eating,” media screens and wall paintings as “looking at” and windows as “looking through”), based on J. Gibson’s (1979) categorization of surfaces and objects (e.g., attached and detached objects, hollow objects). Table 3 shows the labeling of 10 extracted object categories from actually viewed objects. In order to control for the number of objects in different categories presented in the VE stimulus, the ratio of each participant’s object viewing time per category to the total number of objects in the category was used for a statistical analysis. A repeated measure analysis of variance (ANOVA) revealed that the ratio of object viewing time depending on categories was significantly different, F(9, 180) = 29.575, p < .001, 𝜂𝑝2= .06. Post hoc analysis revealed that 13 out of 45 paired comparisons were significant at the level of p < .001 with a Bonferroni correction applied. As shown in Table 3, the ratio of object viewing time was greatest for transparent surfaces that afford looking through (i.e. windows) followed by objects that afforded

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

17

placing and objects for sitting. Based on post-hoc comparisons, viewing time ratio from the other object categories did not significantly differ from each other. [Table 3 here] Impact of Novelty The second set of hypotheses predicted that participants would be more likely to spend longer time in the rooms with novel objects, based on imminence and stability manipulations in the environment. As expected from the small sample size (N = 22) and given that participants were instructed to spend time in the virtual environment without a manifest task, participants showed highly individualized visiting patterns. Because real versus unreal object representation varied only on the second floor in the VE, a repeated measure ANOVA was conducted to compare time spent per room in the second floor. The average time spent per room was not significantly different across conditions, F(4, 80) = .16, p = .96. Orienting Allocation to Virtual Objects: Unreal Objects. The hypothesis H2a predicted that participants would spend more time viewing objects that were displayed with an unrealistic representation than those with real/normal representations. Although participants did spend more time viewing objects in the unreal representation condition (M = 94.92, SD = 31.3) than the real representation condition (M = 68.4, SD = 30.24), the results only approached significance, t(20) = -2.06, p = .058, when comparing viewing time of all objects on the second floor. However, the object viewing time was considerably different across the two conditions when looking at particular objects with dramatically unusual representation in the unreal condition. Figure 3 shows aggregated viewing time of featured objects that differed in terms of real/unreal displays across conditions. These objects were selected because they were particularly viewed by participants for longer time when

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

18

they had unreal representation. Types of unreal displays of the objects are specified below the graph. For example, the toilet, which was almost never central to participants’ attention in the real representation condition (viewed by one participant for one second), attracted notable attention from participants in the unreal representation condition, in which the toilet was bouncing up and down along the wall (viewed by seven participants for 49 seconds, M = 7). [Figure 3 here] Discussion This study investigated how much time participants spent exploring virtual spaces and allocating visual attention to virtual objects in order to understand VE users’ basic exploratory behaviors. In doing so, the study incorporated the ecological perspective and theory of affordances (J. Gibson, 1979) to predict participants’ exploration of the novel virtual environment. The theoretical framework and methodological approach introduced in this study offer a unique way to investigate users’ basic interaction with various virtual environment features and objects, and to generalize to other aspects of VR/VE work. As predicted, users tend to choose rooms to explore based on the functionality of rooms. In general, on the first floor, participants spent the majority of their given time in the living room, followed by kitchen/dining and entrance. The living room and kitchen/dining area had more objects with varying functionality, therefore it makes sense to say participants had many things to interact with compared to the entrance. However, this could be also due to the size of spaces each room occupies (e.g., living room occupied the largest portion of space) in addition to the functionality of the living room. On the second floor, participants spent more time in the hallway than rooms that had a specific/typical use. This may be because the functionality of the hallway that connects different rooms and afford transportability was more salient when

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

19

participants’ major task was exploration, which requires moving from one place to another, so they spent time in the hallway as part of their functional exploration of the house. Analysis of object viewing time indicated that participants spent considerable time viewing surfaces that afford seeing through the window despite the small number of windows in the VE stimulus when controlling for the total number of objects per category. This is probably because they learned the available behaviors in the VE stimulus were mostly looking, and focused their attention on windows instead of other objects that had functionality in the real world (e.g., manipulability, portability) but not in the virtual world. Participant also allocated attention to view larger attached objects that afford placement of other detached objects or sitting. These objects for placing (e.g., desk) and sitting (e.g., bed, chair) tended to contain smaller objects on them, which could provide additional information to look. Furthermore, although the viewing time of objects for ‘looking at’ was not significant when investigating its ratio to the total number of objects in the category, few objects in the category (e.g., wall pictures) were found to be viewed frequently as shown in Table 1 and 2. Taken together, participants’ attention during the VE exploration were oriented toward objects that had functionality for looking behaviors. Participants’ attention on windows or display surfaces brings about the interesting notion of multiple embeddedness during virtual environment interaction. When an individual uses a virtual environment, the experience is considered as one embedded in the physical environment, yet still with a sense of being in the virtual environment (Steuer, 1992). Results from the current study indicate that when VE users allocate visual attention on window surfaces that display additional information of the outside, they may be figuring out or creating a mental model of the location of the virtual house as well as themselves in the virtual space (Wirth et al., 2007).

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

20

Furthermore, although difference in viewing time of real versus unreal objects was only marginally significant, descriptive findings indicate that participants spent more time looking at unreal objects, especially those displaying particularly unusual representation. Objects such as the toilet and bed may not require excessive attentional resources to investigate, but the bouncing toilet and bed hanging would likely violate participants’ anticipation and attract attention. Although our results are promising in terms of conceptualizing VE user behaviors from the affordances perspective, one question our study raises is how to move from perceptual and behavioral data reflecting affordances to higher order cognitions and goals. One such higher order cognition of great interest to VE researchers is in understanding the feeling of ‘presence,’ a psychological state in which users experience objects and environments in the virtual world as actual ones, which leads to the sense of ‘being there’. (Lee, 2007; Wirth et al., 2007). The notion of presence has gained scholarly attention because it is theorized to be associated with psychological and behavioral effects of using virtual environments such as enjoyment and physiological arousal (Heeter, 1995). When it comes to establishing presence, scholars often emphasize the role of constructing spatial mental model from allocated attention to VE stimuli as well as users’ perception of their self-location within the VE. Navigation based on spatial cues and formation of spatial memory are crucial to establish spatial presence in this perspective (Wirth et al., 2007). In this study, users’ presence in the VE and their orienting of attention to virtual objects are already assumed to be established. When a VR technology occupies individuals’ sensory organs and blocks inputs from the real world (e.g., when using HMD with headphones, users’ visual and audio inputs only come from the VE), the resource allocation required in order to feel present in the mediated environment will be minimized (Slater & Sanches-Vivis, 2016). Instead,

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

21

moving within the VE would enable interaction between users’ perceptual systems and features of the VE, allowing users to have a sense of their spatial location within the environment. That said, users do not need to put much effort to be present in the virtual environment struggling between sensory inputs from two worlds—the real and mediated world. Instead, users may adaptively interact with the VE, actively picking up information from the environment. Users may switch their attention allocation when the stimulus is no longer novel or surprising, or when there is other orienting-inducing information. In this regard, it is of great importance to focus on how specific aspects of the VE are further processed for investigatory attention and approach. We expect that fundamental orienting and exploring behaviors will become a basis for further meaningful experiences from the virtual interaction and development of specific behavioral patterns. The results from this study indicate that users have developed preferential attention to objects that are most relevant to looking behaviors (e.g., windows), which were primarily supported in the given VE. This implies that user behaviors particularly associated with training and persuasive intervention can be developed based on offerings and limitations in the VE. In the ecological perspective, learning is process of developing precise action capacity for specific tasks as well as perceiving the environment in a specific way (Linderoth, 2012). For example, connecting affordances of the VE with specific prosocial behaviors, such as cooperative actions both salient in the VE system and manifest tasks, may facilitate helping behaviors among the users (Velez & Ewoldsen, 2013). Furthermore, perceptual learning occurred during virtual interaction has potential to be transferred to the real world. The notion of Game Transfer Phenomena (GTP; Ortiz de Gortari, & Griffiths, 2012) delineates how gamers’ real world view can be altered by their sensory and behavioral experiences with the virtual world.

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

22

Limitations Despite the novel approach and findings, it is important to note limitations of this study and implications for future research. First, the small sample size has made it difficult to account for the results comparing the two experimental conditions. Second, this study did not utilize an eye-tracker device and automated head-body movement tracking systems. Although trained coders could reliably identify which object was viewed at a given second, use of the automated eye-tracking system would allow tracking of saccadic eye movements between objects within a second as well as precise fixation point on a small portion of the visual field. Third, incorporating other individual and socially relevant variables, such as previous experience of using VR, gender, and/or cultural background, could have impacted participants’ viewing and movement patterns. Future studies involving larger sample size and variations in available behaviors or objects will benefit from considering individual and social variables relevant to specific behaviors of interest. Future studies need to integrate more sophisticated analysis technique to more effectively identify notable patterns in VE user behavior. Of great importance is to identify from which context different behavioral outcomes emerge by looking at the behavioral trajectory of each individual as it unfolds over time. By looking at the dynamic data, we can also examine development of preferential attention and actions toward certain features and objects over time. Finally, future work should extend the current paradigm by using the VE with multiple affordances in order to examine learning trajectories of different behaviors as well as eye gaze. Conclusion Taken together, this study understands behaviors of VR users as adaptive behaviors guided by the relationship between features in the virtual environments and users’ actions. By

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

23

focusing on looking, locomotion, and orienting, this study addresses fundamental behaviors that are often unconscious but could serve as a basis for more targeted attention and complex behaviors such as using and manipulating objects. The results suggest that when the available actions are limited to basic exploratory and looking behaviors, users’ orientations were focused towards objects related to looking, developing their primary task in line with the affordances in the VE. Furthermore, this study found that users tended to allocate more attention to objects that are used in everyday real life when those objects displayed strange or novel representation, implying how users’ attention might be guided during their initial interaction with a new virtual environment. References Adolph, K. E., Eppler, M. A., & Gibson, E. J. (1993). Crawling versus walking infants' perception of affordances for locomotion over sloping surfaces. Child Development, 64, 1158-1174. doi: 10.1111/j.1467-8624.1993.tb04193.x Ahn, S. J. G., Bailenson, J. N., & Park, D. (2014). Short-and long-term effects of embodied experiences in immersive virtual environments on environmental locus of control and behavior. Computers in Human Behavior, 39, 235-245. https://doi.org/10.1016/j.chb.2014.07.025 Albert, G., Renaud, P., Chartier, S., Renaud, L., Sauvé, L., & Bouchard, S. (2005). Scene perception, gaze behavior, and perceptual learning in virtual environments. CyberPsychology & Behavior, 8, 592-600. doi:10.1089/cpb.2005.8.592 Boyan, A., & Sherry, J. L. (2011). The challenge in creating games for education: Aligning mental models with game models. Child Development Perspectives, 5, 82-87. doi: 10.1111/j.1750-8606.2011.00160.x

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

24

Bradley, M. M. (2009). Natural selective attention: Orienting and emotion. Psychophysiology, 46, 1-11. doi:10.1111/j.1469-8986.2008.00702.x Ortiz de Gortari, A., & Griffiths, M. D. (2012). An introduction to Game Transfer Phenomena in video game playing. In J. I. Gackenbach (Ed.), Video game play and consciousness (pp. 217-244). Hauppauge NY: Nova Science Publisher. Freina, L., & Ott, M. (2015). A literature review on immersive virtual reality in education: State of the art and perspectives. Proceedings of the 11th eLearning and Software for Education, Bucharest, Romania, 133-141. doi:10.1.1.725.5493 Fox, J., & Bailenson, J. N. (2009). Virtual self-modeling: The effects of vicarious reinforcement and identification on exercise behaviors. Media Psychology, 12, 1-25. http://dx.doi.org/10.1080/15213260802669474 Freelon, D. (2010). ReCal: Intercoder reliability calculation as a web service. International Journal of Internet Science, 5, 20-33. Garrido-Vásquez, P., & Schubö, A. (2014). Modulation of visual attention by object affordance. Frontiers in Psychology, 5. doi:10.3389/fpsyg.2014.00059 Gibson, E. J. (1988). Exploratory behavior in the development of perceiving, acting, and the acquiring of knowledge. Annual Review of Psychology, 39, 1-42. https://doi.org/10.1146/annurev.ps.39.020188.000245 Gibson, E. J., & Pick, A. (2000). An ecological approach to perceptual learning and development. New York, NY: Oxford University Press. Gibson, J. J. (1966). The senses considered as perceptual systems. Boston, MA: Houghton Mifflin.

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

25

Gibson, J. J. (1967). New reasons for realism. Synthese, 17, 162-172. https://doi.org/10.1007/BF00485025 Gibson, J. J. (1977). The theory of affordances. In R. Shaw & J. Brnasford (Eds.), Perceiving, acting, and knowing: Toward an ecological psychology (pp. 67-82). Hillsdale, NJ: Lawrence Erlbaum Associates. Gibson, J. J. (1979). The ecological approach to visual perception. Boston, MA: Houghton Mifflin. Heeter, C. (1995). Communication research on consumer VR. In F. Biocca & M. R. Levy (Eds.), Communication in the age of virtual reality (pp. 191-218). Hillsdale, NJ: Lawrence Erlbaum. Graham, F. K. (1979). Distinguishing among orienting, defense, andstartle reflexes. In H. D. Kimmel, E. H. van Olst, & J. F. Orlebeke (Eds.), The orienting reflex in humans (pp. 137-167). Hillsdale, NJ: Erlbaum. Lang, A. (2006). Motivated cognition (LC4MP): The influence of appetitive and aversive activation on the processing of video games. In P. Messarsis & L. Humphries (Eds.), Digital media: transformation in human communication (pp. 237-256). NY: Peter Lang Publishing. Lang, A. (2014). Dynamic human-centered communication systems theory. The Information Society, 30, 60-70. doi:10.1080/01972243.2013.856364 Lang, A., & Bailey, R. L. (2015). Understanding information selection and encoding from a dynamic, energy saving, evolved, embodied, embedded perspective. Human Communication Research, 41, 1-20. doi:10.1111/hcre.12040

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

26

Lang, A., Matthews, N. L., Lynch, T., Almond, A., Han, J., & Zheng, X. (2018) Driving, dating, and dying: The destabilization of real world behaviors in Grand Theft Auto. In J. Breuer, D. Pietschmann, B. Liebold, & B. P. Lange (Eds.), Evolutionary psychology and digital games: Digital hunter-gatherers. Abingdon, UK: Routledge. Lang, P. J., Simons, R. F., & Balaban, M. T. (1997). Attention and orienting: Sensory and motivational processes. Hillsdale, NJ: Lawrence Erlbaum. Lee, K. M. (2004). Presence, explicated. Communication Theory, 14, 27-50. doi:10.1111/j.14682885.2004.tb00302.x Linderoth, J. (2012). Why gamers don't learn more: An ecological approach to games as learning environments. Journal of Gaming & Virtual Worlds, 4, 45-62. doi:10.1386/jgvw.4.1.45_1 Oculus Rift [Virtual reality hardware]. (2016). Irvine, CA: Oculus VR, LLC. Öhman, A. (1979). The orienting response, attention, and learning: An information processing perspective. In H. D. Kimmel, E. H. van Olst, & J. F. Orlebeke (Eds.), The orienting reflex in humans (pp. 443-471). Hillsdale, NJ: Lawrence Erlbaum Associates. Posner, M. I. (1980). Orienting of attention. Quarterly Journal of Experimental Psychology, 32, 3-25. http://dx.doi.org/10.1080/00335558008248231 Reeves, B., & Nass, C. (1996). The media equation: How people treat computers, television, and new media like real people and places. New York, NY: Cambridge University Press. Sherry, J. L. (2014). Media effects, communication, and complexity science insights on games for learning. In F. C. Blumberg (Ed.), Learning by playing: Video gaming in education (pp. 104-120). Cambridge, MA: MIT Press. Sherry, J. L., Lucas, K., Greenberg, B. S., & Lachlan, K. (2006) Video game uses and gratifications as predictors of use and game preferences. In P. Vorderer & J. Bryant

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

27

(Eds.), Playing video games: Motives, responses, consequences (pp. 213-224). Mahwah, NJ: Erlbaum. Skalski, P., Tamborini, R., Shelton, A., Buncher, M., & Lindmark, P. (2011). Mapping the road to fun: Natural video game controllers, presence, and game enjoyment. New Media & Society, 13, 224-242. https://doi.org/10.1177/1461444810370949 Slater, M., & Sanchez-Vives, M. V. (2016). Enhancing our lives with immersive virtual reality. Frontiers in Robotics and AI, 3(74), 1-47. doi:10.3389/frobt.2016.00074 Steuer, J. (1992). Defining virtual reality: Dimensions determining telepresence. Journal of Communication, 42, 73-93. https://doi.org/10.1111/j.1460-2466.1992.tb00812.x Velez, J. A., & Ewoldsen, D. R. (2013). Helping behaviors during video game play. Journal of Media Psychology, 25, 190-200. doi:10.1027/1864-1105/a000102 Wirth, W., Hartmann, T., Böcking, S., Vorderer, P., Klimmt, C., Schramm, H., ... & Biocca, F. (2007). A process model of the formation of spatial presence experiences. Media Psychology, 9, 493-525. http://dx.doi.org/10.1080/15213260701283079 Yao, B., Ma, J., & Fei-Fei, L. (2013, December). Discovering object functionality. Paper presented at 2013 IEEE International Conference on Computer Vision, Sydney. doi: 10.1109/ICCV.2013.312

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

1

Figures Figure 1. Selected screenshots of virtual rooms with real representation

Figure 2. Selected screenshots of virtual rooms with unreal representation

Figure 3. Aggregated viewing frequency of featured objects that change in unreal condition

100

89

80 60

49

47 36

40

22

20 20

4

41

32 20

6

1

18

1

0

Cabinet

Toilet

Bed

Bathtub

Weird Movement in unreal condition Weird Placement in unreal condition

Viewed by real condition

Cycle

Bird toy mobile

Owl picture pair

Change in shape/size

Viewed by unreal condition

ACCEPTED MANUSCRIPT

Highlights 

Users spend time in virtual rooms differently based on functionality of rooms



Attention allocation is guided by functionality and novelty of virtual objects



Object functionality of “looking through” drew more attention than others



People spend more time viewing virtual objects with unreal displays

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

1

Tables Table 1. Top 10 most frequently viewed objects across all participants (N = 22), 1st floor Room Object Viewed frequency Viewed by (sec) (# of participants) Living Window (patio 109 17 side) Living

Bookshelf large

67

11

Kitchen/Dining

Dining table

56

14

Kitchen/Dining

Window (road side)

37

10

Entrance

Wall picture small

34

7

Living

Computer

27

7

Entrance

Entrance door/window

23

4

Kitchen/Dining

Kitchen counter

23

6

Kitchen/Dining

Refrigerator

23

12

Kitchen/Dining

Microwave

20

5

Table 2. Top 10 most frequently viewed objects across all participants (N = 22), 2nd floor Room Object Viewed Viewed by frequency(sec) (# of participants) Bedroom Window 145 20 Bedroom Bed 125 18 Bedroom Vanity 120 20 Activity Room Window 120 14 Hallway Wall picture 119 18 Activity Room Desk 112 20 Nursery 90 14 Wall painting Bathroom Window 89 18 Nursery Window 83 16 Activity Room Cycle 81 18

ACCEPTED MANUSCRIPT ORIENTING IN VIRTUAL ENVIRONMENTS

Table 3. Room Sitting (e.g., chair, bed) (a) Placing (e.g., table, shelf) (b) Looking through (e.g., window) (c) Looking at (d) Storage (e) Moveable objects-tool (f) Moveable objects-clutter (g) Wearing (h) House/Appliance (i) Eating (j)

2

Mean (SD) 1.53 (.98) f, g, h, i 2.03 (1.28) g, h 3.94 (2.19) d, e, f, g, h, i .92 (.63) c .81 (.42) c, h .34 (.25) a, c .31 (.31) a, b, c .26 (.3) a, b, c, e .49 (.36) a, c .68 (1.2)

Note. For each row, superscripted text indicates statistically significant pairwise comparisons after a Bonferroni correction at the p < .001 level.