Combining virtual reality and mobile eye tracking to provide a naturalistic experimental environment for shopper research

Journal of Business Research xxx (xxxx) xxx–xxx

Contents lists available at ScienceDirect

Journal of Business Research journal homepage: www.elsevier.com/locate/jbusres

Combining virtual reality and mobile eye tracking to provide a naturalistic experimental environment for shopper research☆ Martin Meißnera,b,⁎, Jella Pfeifferc, Thies Pfeifferd, Harmen Oppewalb a

Department of Sociology, Environmental and Business Economics, University of Southern Denmark, Niels Bohrs Vej 9, 6700 Esbjerg, Denmark Department of Marketing, Monash Business School, Monash University, Building S7, 26 Sir John Monash Drive, Caulfield East, VIC 3145, Australia c Karlsruhe Institute of Technology, Fritz-Erler-Straße 23, 76133 Karlsruhe, Germany d Cognitive Interaction Technology Center of Excellence, Bielefeld University, Inspiration 1, 33619 Bielefeld, Germany b

A R T I C L E I N F O

A B S T R A C T

Keywords: Eye tracking Visual attention Virtual reality Augmented reality Assistance system Shopper behavior

Technological advances in eye tracking methodology have made it possible to unobtrusively measure consumer visual attention during the shopping process. Mobile eye tracking in field settings however has several limitations, including a highly cumbersome data coding process. In addition, field settings allow only limited control of important interfering variables. The present paper argues that virtual reality can provide an alternative setting that combines the benefits of mobile eye tracking with the flexibility and control provided by lab experiments. The paper first reviews key advantages of different eye tracking technologies as available for desktop, natural and virtual environments. It then explains how combining virtual reality settings with eye tracking provides a unique opportunity for shopper research in particular regarding the use of augmented reality to provide shopper assistance.

1. Introduction Eye tracking is becoming increasingly popular in retailing as a way to better understand consumers' visual attention and so gain insights as to how to stimulate sales of particular products or help consumers make better decisions. Retail researchers have for example investigated how a product's position on a shelf affects attention and sales and public policy researchers have used eye tracking to evaluate the effectiveness of mandatory health warning and nutrition labels (Wedel & Pieters, 2008). Mobile eye tracking in particular is seen as a promising tool due to its capability to unobtrusively gather data and to provide “insights into naturalistic shopping behavior” (Harwood & Jones, 2014, p. 183). Shankar, Inman, Mantrala, Kelley, & Rizley (2011), for example, mention mobile eye tracking besides electronic tracking (i.e., clickstream analysis) and shopping path recording as a new instrument to collect data on shopper navigational behavior, aisle placements and shelf position. Grewal et al. (2011) see mobile eye tracking as an opportunity for analyzing the effectiveness of promotional design elements. A main argument for using mobile eye tracking in real world retail settings is that attentional processes could differ considerably between the lab and the real world (Foulsham, Walker, & Kingstone, 2011; Hayhoe & Ballard, 2005). Supermarket shoppers often make choices within seconds and only consider a very limited set of options which

suggests that retailing research should investigate attentional processes “in situ” (Wästlund, Otterbring, Gustafsson, & Shams, 2015), that is, in field settings at the point of sale. Most of the current research into consumers' attentional processes, however, is undertaken in laboratory settings using desktop eye tracking and thus we agree with Kahn (2017, p. 40) that “effects that work in the laboratory may work differently in the field”. A good example of a retail-related study that uses a lab setting is Chandon, Hutchinson, Bradlow, & Young (2009). They use 2D–pictures (planograms) and desktop eye tracking to study how instore factors such as shopping goals and brand awareness influence brand evaluations. While an impressive study, many of the interesting effects observed by the authors have not yet been replicated in real settings. Replicating these studies with mobile eye tracking, however, is very cumbersome. In line with earlier suggestions by Bigné, Llinares, & Torrecilla (2016), this paper proposes that virtual reality mobile eye tracking offers a unique research opportunity. It allows levels of control that can typically only be achieved in lab environments while at the same time being able to provide the realistic 3D experience and freedom of movement that is typical for real store environments. As such it can provide a controlled shopping experience that “feels like reality”. In fully immersive 3D settings, shelf positions and body movements interact with visual angle and can be expected, among many other

☆ ⁎

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Corresponding author at: Department of Sociology, Environmental and Business Economics, University of Southern Denmark, Niels Bohrs Vej 9, 6700 Esbjerg, Denmark. E-mail addresses: [email protected] (M. Meißner), jella.pfeiff[email protected] (J. Pfeiffer), tpfeiff[email protected] (T. Pfeiffer), [email protected] (H. Oppewal).

http://dx.doi.org/10.1016/j.jbusres.2017.09.028 Received 29 March 2017; Received in revised form 15 September 2017; Accepted 17 September 2017 0148-2963/ © 2017 Elsevier Inc. All rights reserved.

Please cite this article as: Meissner, M., Journal of Business Research (2017), http://dx.doi.org/10.1016/j.jbusres.2017.09.028

Journal of Business Research xxx (xxxx) xxx–xxx

M. Meißner et al.

to identify new research opportunities arising from the availability of virtual reality environments. We review previous research and discuss three areas of research that can benefit from virtual reality settings: instore decision processes, retail environment and in-store design, and augmented reality assistance systems that support consumers. As such, our paper is relevant not only for researchers in marketing and retailing but also for researchers from other disciplines, like information systems and psychology, who are interested in applying eye tracking in virtual reality settings.

factors, to influence attentional processes to the extent that the effects observed in 2D may not generalize to virtual or real 3D-environments (Foulsham et al., 2011). Furthermore, as 2D lab studies typically are designed to utilize pre-defined tasks, they tend to ignore how the shopping context affects the flow of attention in a naturalistically conducted shopping task. The study of attentional processes in fully immersive 3D environments requires the use of mobile eye-trackers instead of desktop eye trackers. In contrast to desktop eye trackers, which are eye-tracking devices installed in desktop computers to allow the monitoring of eye movements on the screen, mobile eye trackers look similar to glasses and allow the user to move around freely and consider objects in her or his (3D) environment. Only a small number of studies so far has used mobile eye tracking in real-world retailing contexts (including Burke & Leykin, 2014; Clement, Kristensen, & Grønhaug, 2013; Gidlöf, Wallin, Dewhurst, & Holmqvist, 2013; Hendrickson & Ailawadi, 2014; Otterbring, Wästlund, Gustafsson, & Shams, 2014; Wästlund et al., 2015). Researchers emphasize that there is still a lack of research in this area, especially when it comes to using mobile eye tracking to understand decision making for fast moving consumer goods (Clement et al., 2013; Wästlund et al., 2015). A practical but major reason for the lack of research in this area is that conducting and analyzing mobile eye tracking studies is very cumbersome for the researchers. Annotating fixations of the respondents' eyes to an area of interest (AOI) defined for a real and dynamic environment is still a huge challenge for the analyst (Brône, Oben, & Goedemé, 2011; Kurzhals, Hlawatsch, Seeger, & Weiskopf, 2017). The head movements of the respondents continuously change the position of the objects in the head-mounted scene camera recordings of the environment. This necessitates analyzing every recorded video frame separately to determine on which objects the respondent's gaze rested. Clement et al. (2013, p. 237) for example explain in their paper that their mobile eye tracking videos had to be “analyzed frame by frame, coding gaze time and number of fixations”. Our experience is that such manual annotations are extremely time-consuming. As a consequence, researchers often avoid analyzing the eye tracking data quantitatively. Harwood & Jones (2014), for example, used content analysis of consumers' gaze fixations to analyze their mobile eye tracking data, meaning that four coders qualitatively analyzed respondents' gaze behavior. Clement et al. (2013) and Gidlöf et al. (2013) both recorded complete shopping trips, but then selected only one product category (jam in Clement et al. (2013) and pasta in Gidlöf et al. (2013)) for further analysis. The fact that these researchers only analyzed a small fraction of their datasets just indicates how much work it is to prepare the respective datasets. We argue that the high workload for preparing the data is a severe hindrance to research using mobile eye tracking and at least partly explains why only a few papers have been published that actually used mobile eye tracking in retail settings. Although algorithms that make the preparation of mobile eye tracking data easier are in development (see, e.g., Toyama, Kieninger, Shafait, & Dengel, 2012; Harmening & Pfeiffer, 2013, as further discussed below), there are as yet no reliable algorithms for automatic annotations (Kurzhals et al., 2017; Brône et al., 2011).We argue that, instead, virtual reality as an experimental environment for mobile eye tracking studies can reduce the burden of data coding and allow greater flexibility in creating realistic research settings, while maintaining high levels of experimental control. The purpose of this paper is twofold. First, we aim to show how mobile eye tracking can be used in the virtual reality as well as to discuss advantages and disadvantages of conducting attentional research in desktop, virtual and natural environments. Our contribution is that we develop a set of criteria for researchers to make informed decisions which eye tracking setting to use in empirical studies. By discussing these issues we aim to help marketing, retailing and decisionmaking researchers make smarter decisions about which equipment to use and how to design their eye tracking experiments. Second, we aim

2. Immersive virtual reality and mobile eye tracking Virtual reality can be defined as a simulated environment in which the perceiver experiences telepresence, which is the extent to which a person feels present in a virtual environment (Steuer, 1992). An immersive virtual environment is one in which the user is perceptually surrounded by the virtual environment (Loomis, Blascovich, & Beall, 1999, p. 577). Blascovich et al. (2002) emphasize that making virtual environments more immersive is one of the most important directions for its further technological development as it makes the perception of the virtual environments more similar to perception of reality. The aforementioned authors have been using virtual environments successfully in their experimental research on social behaviors in psychology for over 15 years. The benefits of using virtual reality as an alternative to real-life situations has been recognized as early as 1995 (Durlach & Mavor, 1995) by the US military for applications such as simulation and training. Since then, it has been applied to several other areas, such as education, the training of pilots and firefighters, and the training of medical personnel. Two main implementations of virtual environments can be distinguished (Loomis et al., 1999), so-called CAVEs (Cave Automatic

Fig. 1. Three sides of a CAVE projected from the rear. The user's perspective is tracked using optical tracking systems.

Fig. 2. Stereo glasses using a passive filter technology called Infitec.

2

Journal of Business Research xxx (xxxx) xxx–xxx

M. Meißner et al.

Fig. 3. Left: An HMD called HTC Vive as visual stereo display and a leap motion controller for tracking the hands; right: An HMD called Oculus Rift DK2 and a Razer Hydra controller for interaction.

orientation of the user's orientation. An important part of virtual reality is the interaction with the user. The action and reaction in the virtual reality simulation is what makes it different from computer graphics and 3D movies. The most important interaction is the dynamic change of the perspective, i.e. the position and orientation of the user. Details about the necessary technological setup can be found in the Appendix. Besides body tracking, a virtual reality system should also track the user's hand to provide appropriate means for manual interactions.1 The application of eye tracking to virtual reality environments combines the strengths of mobile and desktop-based eye tracking. It provides the respondent with full flexibility regarding natural movements in a fully immersive 3D environment. At the same time, the analysis of the data is not more complicated than with 2D desktop eye tracking applications. Once the 3D models of the target stimuli have been created, subparts of the objects can be annotated by additional geometries to define areas of interest (or better: volumes of interest) (Fig. 4). These geometries are later used to identify the gaze-rays cast from the respondent's eyes, using information about the head position and orientation as well as about the orientation of the eye ball provided by the eye tracking system, into the 3D environment. Whenever a gazeray hits an area (or volume) of interest (Fig. 5), an appropriate event is generated and logged for later analysis. As the areas of interest are modelled in a local coordinate system of the objects, they follow every movement. If, for example, the respondent picks up an object, that object's position and orientation is updated to match the movements of the respondent's hand. The areas of interest will then also automatically update without any delay and gaze will continue to be correctly annotated with the appropriate area of interest (Fig. 6).

Fig. 4. Annotated 3D model of a product package. The AOIs are 3D geometries invisible to the respondent.

Virtual Environments) and HMDs (head-mounted displays): CAVEs consist of stationary display surfaces, typically fed by projectors (CruzNeira, Sandin, & DeFanti, 1993), with multiple projection screens and loudspeakers surrounding the user. The computer-generated visual imagery is back-projected onto the translucent walls, floor, and ceiling of a medium-size cubical room (Fig. 1). In the room the user is free to move around. Glasses with shutters or other filters (Fig. 2) provide stereoscopic stimulation presenting two images for the same scene with slightly different perspectives matching the eye positions of the observer. The respondent perceives the projections on the room surfaces, which may show resolutions of 2048 × 1536 pixels or higher, depending on the projectors being used. An important measure is the pixel size on the surface. When the pixel size is smaller than 1 mm, humans will no longer be able to identify individual pixels when standing at more than an arm's length away from the screens. Smaller set-ups include single projection screens (Power Walls) or the combination of projections on the floor and one screen (L-Shape). The projection on the floor is relevant to simulate interactions proximal to the body, including manual actions such as picking up and handling of objects. Besides CAVEs, the second form of implementation is to use a HMD (Fig. 3) together with a computer and a head tracker. Inside an HMD are two screens (except if a split-screen presentation mode is used), one for each eye, to provide stereo images. Consumer HMDs of 2017 typically provide a field of view of about 100° diagonally and small but perceivable pixels. This technology is currently in transition and new devices with higher resolutions (4 k) are expected to be introduced to the market in the near future. In both approaches, CAVE and HMD, changes in the position and orientation of the user's head are tracked within the physical environment and reproduced in a simulated environment on a computer. A fast processor generates the visual and auditory imagery from a perspective that is based on the position and

3. Advantages and disadvantages of using eye tracking technologies in desktop, natural and virtual environments In line with our first research goal, the aim of this section is to discuss the advantages and disadvantages of applying mobile eye tracking in the lab and in the field. As a benchmark we also compare these two mobile settings with the use of desktop eye tracking in the lab. We assemble criteria that should be evaluated by researchers to make an informed decision which eye tracking setting to use in their empirical studies. These criteria are presented in Table 1. Our focus here is specifically on mobile eye tracking (see Holmqvist et al. (2015) for an introduction), not on the general discussion about the differences between lab and field experiments; we refer the reader to McGrath (1981) for a broader discussion on the latter issue. It is also worth 1 For this, either specific controllers can be used, such as the aforementioned HTC controller, the Oculus Touch controller, the Razer Hydra (see http://sixense.com/ razerhydra) or the ART Flystick (http://www.ar-tracking.com/products/interaction/ flystick2/), just to provide some examples. Alternatively, the hand can be tracked using specific data gloves, or, more recently, using optical tracking solutions, such as the Leap Motion (see https://www.leapmotion.com/) or the Intel Real Sense camera (http://www. intel.de/content/www/de/de/architecture-and-technology/realsense-overview.html).

3

Journal of Business Research xxx (xxxx) xxx–xxx

M. Meißner et al.

create stimuli when measuring eye movements in the natural environment (e.g., in a store). The eye movements are simply recorded in the real world (as in Gidlöf et al. (2013) and Clement et al. (2013)). In a desktop eye tracking study, however, the product packages have either to be photographed, scanned in or respective pictures have to be found online so that it is afterwards possible to display them on a computer screen. Creating the stimuli takes even more effort for virtual environments. For example, to study product packages, a 3D-model has to be created for every single product package. This includes scanning or photographing the product from all sides and creating a 3D-model with the exact measures of the real product. In the study presented in Section 5, creating such a 3D-model took lab assistants about 30 min for each product package on average. That example shows that creating existing stimuli in a virtual environment is more effortful than in the other two setups. When considering non-existing products or stimuli, the desktop and VR applications have the advantage that these do not have to be manufactured or prototyped physically, but only need to be created as digital prototypes. Clement, Smith, Zlatev, Gidlöf, & van de Weijer (2017), for example, investigate potentially misleading effects of graphic elements on food packaging. Testing the effect of these graphics requires redesigning the respective product packages by adding design elements to the package which can be easily achieved when the stimuli are digital models (2D–images or 3D–models). Testing these non-existing packages in the field would require printing or producing the respective product packages. For other stimuli, such as automobile prototypes, it is even more difficult and expensive to create or produce physical prototypes. For this reason virtual reality has been used in the new product development process for quite a while (Purschke et al., 1998).

Fig. 5. During the study, gaze-rays are cast into the environment. When they hit a predefined AOI, an appropriate log is written.

3.1.2. Realism of stimulus display Social psychologists describe two basic experimental scenarios, one with experimental control over independent variables, but less realism, another one with less control, but more “mundane realism”, which is the extent to which an experiment is similar to everyday life situations. Blascovich et al. (2002) emphasize that “mundane realism” increases respondents' engagement within experiments, but often has the high price of losing experimental control. Loomis et al. (1999) argue that visual perception researchers have experienced the trade-off between experimental control and ecological validity (i.e., “mundane realism”) and that immersive virtual environments are promising because they allow a more realistic presentation of stimuli while at the same time giving researchers more experimental control. For example, respondents can take the product packages from the shelves, see the side and rear of the product package and stand further away or closer to the product shelf while still being in a strictly controlled virtual environment. The trade-off between experimental control and ecological validity is visualized in Fig. 7. A good example of this trade-off is provided by early studies using tachistoscopes to investigate the perception of advertisements. In these experiments, 2D pictures were shown to respondents for very short time spans which allowed researchers to manipulate experimental factors. However, this also reduced the ecological validity of the experiments (Loomis et al., 1999). Investigating real world purchases with mobile eye tracking in the supermarket, the natural environment, represents the other end of the continuum, as this setting is ecologically valid, but there is less experimental control over factors which are supposed to have an additional effect on the purchase. The decision which eye tracking technology to use therefore is often based on researchers assessing how much experimental control is needed or could be given up to make the experiment more ecologically valid. If more experimental control is needed, researchers will often use desktop eye tracking in laboratory settings instead of mobile eye tracking in natural environments. The ability of the virtual environment to solve the trade-off conflict

Fig. 6. AOI definitions are local to the object and will thus follow every movement with no delay.

noting that eye tracking can be combined with other physiological data, such as skin conductance measurement or facial recognition, in all three experimental environments. For an example study that combines eye tracking with EEG measurement, we refer the reader to Léger et al. (2014). 3.1. Experimental control and ecological validity 3.1.1. Ease of creating/using existing and non-existing stimuli The choice of the environment (desktop, virtual reality, field) largely influences how much effort researchers have to put into designing the respective stimuli for their studies. In general, in real world settings, we can use stimuli that exist in reality. In desktop and virtual reality studies, we rely on presentations of the real world. In the virtual reality, these models must necessarily be in 3D. If one seeks to measure attention to product packages, if the product packages already exist and if they are offered in a supermarket, then very little effort is needed to 4

Journal of Business Research xxx (xxxx) xxx–xxx

M. Meißner et al.

Table 1 Criteria for deciding which environment to use (desktop, virtual reality and field; + indicates a relative advantage of the environment, o indicates the medium level, and - indicates a relative disadvantage of the environment) – Eye tracking specific criteria are highlighted in grey.

Experimental criteria

Desktop eye tracking

Experimental control and ecological validity Ease of creating/using o existing stimuli (Medium) Ease of creating/using non+ existing stimuli (Easy) Realism of stimulus display (Low) + Ease of controling and (Easy) randomizing treatment and extraneous factors Realism of interaction (Low) Naturalness of the eye + tracking task (High) Data gathering Accuracy of eye tracking o (Medium) Ease of analyzing and + reacting to respondent’s (Easy) attention and behavior in real time Ease of generating large + sample sizes (Easy) Ease of obtaining retailer + permission to record (Not necessary) Ability to monitor multiple o modalities of respondents (Medium) Data analysis Ease of data preparation + (Easy) Reliability of AOI coding + (High) Additional factors Costs of creating test + environment (Low) Required expertise to o acquire and analyze data (Medium) Reproducibility of + experimental setting (High)

Mobile eye tracking in virtual reality

Mobile eye tracking in the field

(Hard) + (Easy) o (Medium) + (Easy)

+ (Easy) (Hard) + (High) (Hard)

o (Medium) o (Medium)

+ (High) o (Medium)

+ (High) + (Easy)

o (Medium) (Hard)

(Hard) + (Not necessary) + (High)

(Hard) (Necessary) o (Medium)

+ (Easy) + (High)

(Hard) 1(Low)

(High) (High) + (High)

o (Medium) o (Medium) (Low)

case the “user will not have the feeling of being in the mediated environment and the benefits of the virtual setting compared to laboratory or field studies are nullified.” (Brade et al., 2017, p. 76).

3.1.3. Ease of controling and randomizing treatment and extraneous factors As depicted in Fig. 7, virtual environments are promising for retailing research because they allow greater ecological validity while maintaining high levels of experimental control. Purchase decisions made in a virtual supermarket may serve as a good example. Factors like the position of the product on the shelf (Atalay, Bodur, & Rasolofoarison, 2012), the salience of product packages (Milosavljevic, Navalpakkam, Koch, & Rangel, 2012), design elements of the food package (Clement et al., 2017), and the prices shown next to the products (Yao & Oppewal, 2016) can be more easily varied or controlled for in a lab experiment. The horizontal centrality of the products on the shelf, for example, might increase visual attention and choice (Atalay et al., 2012). Husić-Mehmedović, Omeragić, Batagelj, & Kolar (2017) argue that package studies in the field typically fail to control for confounding variables, such as shelf position, which have been shown to affect attention. Thus, researchers may want to control for the potential effect of centrality and randomize the position of the product packages on the shelf. Implementing such a randomization in the field would mean reshuffling the products on the shelf after each trial for every respondent which is labor intensive and not very practical.

Fig. 7. Trade-off between experimental control and ecological validity (based on Loomis et al., 1999).

between experimental control and ecological validity also depends on the quality of the VR environment. Brade et al. (2017) emphasize that VR systems need to meet high requirements concerning realistic visualization and convincing presentation of the setting. If that is not the 5

Journal of Business Research xxx (xxxx) xxx–xxx

M. Meißner et al.

3.2.2. Ease of analyzing and reacting to respondent's attention and behavior in real time Desktop eye tracking and mobile eye tracking in virtual environments allows analyzing attentional information while the respondent processes the information and permits reacting to a respondent's attention in real time. From a system analysis point of view, using real time eye tracking information means a substantial shift from using eye tracking as a diagnostic system to ascertain the user's attentional patterns over the given stimuli to using eye tracking as an interactive, gaze-contingent system that “exploits knowledge of the user's gaze to facilitate the rapid rendering of complex displays (e.g. graphical environments)” (Duchowski, 2002, p. 1). An example for such a gazebased interaction is the study by Losing, Rottkamp, Zeunert, & Pfeiffer (2014). They successfully used real time information about the current gaze position to support visual search by providing acoustic feedback modulated by the distance to the search target. The experiment we describe in Section 5 provides another example. Respondents receive additional product information depending on their current gaze position. Real-time identification of gaze patterns and the objects of interest is a prerequisite for building an attention-based assistance system because the attentional processes have to be analyzed before recommendations can be given. Because reliable algorithms for automatic annotations are still lacking (Kurzhals et al., 2017; Brône et al., 2011, see below), it is currently not possible to analyze information search in real-time in these settings.

3.1.4. Realism of interaction For naturalistic interaction with virtual objects, the user needs a means of manual control. There are multiple ways that this can be realized, for example by analyzing hand gestures or by tracking the user's position and orientation in the room. Research in virtual reality currently focuses on the challenge of providing versatile haptic cues for manual interactions (Blake & Gurocak, 2009; Meli, Salvietti, Gioioso, Malvezzi, & Prattichizzo, 2016), for example in the EU project WEARable HAPtics (WEARHAP 2013–2017). To create a more immersive experience, modalities such as haptics or olfactorics (Mihelj, Novak, & Beguš, 2014, p.12), can be served with experimental specific hardware installations but no off-the-shelf technologies exist today that can easily be used and integrated. While the realism of interaction has improved a lot with recent technological developments (improved computer graphics and faster processor speed), it is obvious that the virtual reality still is not indistinguishable from physical reality and not fully equivalent to the one within a natural environment. Today, the limited realism of interaction can still be seen as a disadvantage of the VR environment. As the technology improves and the resolution of the 3D environments gets better the experience in the virtual environment for the respondents is meant to become more or less “indistinguishable from ‘normal reality’” (Loomis et al., 1999, p. 577).

3.1.5. Naturalness of the eye tracking task It is not a problem to track the eye movements of respondents who wear glasses. Tobii (and other eye tracking companies) have developed prescription lenses that can be used in mobile eye tracking systems. However, the intrusivness and cumbersomeness of the eye tracking device for the particpant is a concern. While modern desktop eye tracking technology is unobtrusive, with the participant no noticing the tracking after calibration (unless a mounted device is used), mobile eyetracking requires wearing special glassess and VR with HDM requires the use of elaborate headsets, limiting the naturalness of the task.

3.2.3. Ease of generating large sample sizes Conducting studies that include larger numbers of respondents is a problem when using mobile eye tracking for two main reasons: The expensiveness of the mobile equipment and the effort for preparing the respective datasets. First, because of the higher costs of the mobile equipment, most labs will have only few mobile eye tracking devices. Professional desktop eye tracking tends to be cheaper and so it will be easier to conduct several eye tracking interviews in parallel. Furthermore, in the past decade, researchers have continuously investigated the use of low-cost equipment, in form of standard RGBcameras, such as WebCams, for eye tracking (recent examples are WebGazer and SearchGazer; see Papoutsaki et al. (2016) and Papoutsaki, Laskey, & Huang (2017)). If successful, these efforts would allow for large scale elicitation of eye movement data. In fact, recent evaluations have shown that this is indeed possible (Semmelmann & Weigelt, 2017), however, only if the required accuracy is not below 4° (both in lab and online conditions). While for desktopbased studies, RGB-based eye tracking is a feasible option, it is not well suited for mobile eye tracking and in particular eye tracking in virtual reality.2 Second, as will be discussed below, manual annotation of fixations is needed when analyzing mobile eye tracking data from the field. The effort for preparing the datasets will strongly depend on the sample size in this case. Preparing the mobile eye tracking data thus is an additional hurdle for generating larger samples.

3.2. Data gathering 3.2.1. Eye tracking accuracy The accuracy of the eye tracking data depends on the hardware used but also on the quality of the mapping between gazes and the objects fixated in the environment (either on the desktop, the 3D model in the VR or the physical reality in the field). The eye tracker needs to be calibrated in order to learn this mapping. If the calibration is weak, the quality of the recorded data will suffer and thus the interpretation of the fixated objects might be wrong. During calibration, the respondent is asked to look at certain points in the respective environment. These points are arranged in a way that lets the camera of the eye tracker capture the eyeball in different postures. Therefore it is important that the users hold their head still during calibration and do not turn towards the target. If they would, the captured eyeball postures would not cover a broad range of the postures the system will be faced with later during the study and thus the estimation of the gaze position will suffer. This is in particular relevant for desktop and mobile eye tracking in natural environments, as the calibration targets are typically statically placed in the environment and thus a head rotation will result in a different eyeball rotation during fixation on the target. In virtual reality this is different, as the position of the calibration target can be linked to the head movement. The target can thus easily be kept in a position that will elicit the intended eyeball rotation. In desktop systems, the use of a chin rest can help to eliminate head movements during calibration, or at all during the experiment. In principle, eye tracking in an HMD achieves the highest accuracy, as both external lighting and head position of the user can be fully controlled.

2 The reason is that either the highly varying amount of light or the dynamically changing projection of the environment (e.g. due to movements of the head) challenge the computer vision processes. In virtual reality, the eyes of the user are typically kept in a dark environment, where no external sun or other light sources should influence the perception of the computer generated visual virtual world. RGB-cameras do not work properly in the dark, exposure time has to be increased, sensor noise increases and sampling rate will drop. Also, in mobile and VR settings the accuracy of the eye tracking system has to be even higher than in desktop systems. This is due to the much broader range of depth in which the content is distributed. In desktop scenarios, the content is typically on a fixed plane within 90 cm of distance to the user. In mobile and VR scenarios, even in smaller indoor scenarios, relevant content may lay between 30 cm and 6 m. A measurement error of only 1° of visual angle will already result in errors of more than 10 cm at 6 m distance. Larger measurement errors will render the tracking of gaze on typical objects rather difficult.

6

Journal of Business Research xxx (xxxx) xxx–xxx

M. Meißner et al.

3.2.4. Ease of obtaining retailer permission to record Obtaining permission from external partners to record eye movements in the field can sometimes be challenging. In a retail context, for example, that would require to get an agreement with a supermarket that consumers can be tracked. Also, there may be ethical concerns regarding the extent to which other consumers are filmed unintentionally by the scene camera of a mobile eye tracking system. The presence and the activity of other consumers might also influence the behavior of the respondents and should thus be controlled for. These problems become potentially more challenging with larger sample sizes, but do not occur when conducting the experiment in a laboratory.

“automatic” annotation of fixations to the AOIs has several advantages: Annotators are no longer needed, which makes the data preparation more reliable and less effortful. The annotators are also a potential source of error which is avoided when automatically assigning fixations. It is important to highlight that the determination of fixations and saccades depends on the algorithm used for determining fixations, but not on the experiment environment used. Therefore, if researchers use the same algorithm for determining fixations, the fixation patterns of different environments can be directly compared.

3.2.5. Ability to monitor multiple modalities of respondents Virtual environments provide modality richness which means that various forms of communication and bi-directional interaction are possible, for example, between sales-representatives and consumer avatars. Jin (2009) investigates how using multiple modalities (text, audio and video animation) changes consumers' product evaluations, buying intentions and enjoyment of online shopping when presenting marketing information through 3D recommendations. Jin found that modality richness is a significant factor that positively influences attitudes towards the product and increases purchase intentions. Similarly, multiple modalities describing the behavior of the respondent can be recorded. Besides the tracking of head position and orientation, as well as eye gaze, typical virtual reality systems allow for the recording of hand and body movements together with voice recordings. Mobile eye tracking in virtual environments will enhance the interactivity with consumers because recommendations can be based on consumers' attentional patterns. The adaptation of recommendation agents to consumers' attentional processes can be expected to significantly reduce consumers' burden to verbalize their needs and real time analysis of mobile eye tracking allows us to re-evaluate and transfer results from desktop-based studies to real world shopping (González, López, Angulo, & de la Rosa, 2005; Felfernig, Jeran, Ninaus, Reinfrank, & Reiterer, 2013).

3.3.2. Reliability of AOI coding In order to increase the reliability of the coding, normally at least two independent annotators assign the fixations for all respondents. Afterwards software is used to check for inconsistencies between the independent annotators. The two annotators then discuss every difference in their annotations until they agree. In some studies (see, e.g., Harwood & Jones, 2014; Pfeiffer, Meißner, Prosiegel, & Pfeiffer, 2014) an inter-rater agreement score is reported (e.g., Krippendorff's alpha) which quantifies the degree of agreement when first annotating the data. Besides the huge amount of work which goes into manually coding the data, it is a problem that after the annotators have finished the coding of the data, changing the areas of interest would require repeating the manual annotation process. The following example may help to illustrate the problem: Let us assume that eye tracking data have been recorded in a supermarket with respondents making their daily purchase decisions. After recording the data, the researchers decide that they would like to distinguish between fixations on prices, brand logos, detailed product information as well as remaining fixations on the product package. The annotators receive a list of the areas of interest and assign all of the fixations to all areas of interest for all respondents. After finishing the coding, the researchers realize that some of the products have an organic label on the package and therefore get interested in how much attention goes to the organic labels. As a consequence, the annotators have to add the respective annotations and go through all video material again.

3.3. Data analysis 3.3.1. Ease of data preparation One of the key advantages of eye tracking in virtual environments compared to eye tracking in natural environments is the lower effort required for preparing the dataset. After the data has been recorded using mobile eye tracking in the field, annotators have to review all recorded videos. These annotators have to look at every video frame and manually assign every fixation to one of the objects of interest. Given that for every second, 24 or more frames of the scene camera and up to 250 frames of the eye cameras have been recorded, the amount of work which goes into preparing a dataset is very time-consuming. Our own experience shows that annotation time for 24 Hz videos can be up to 15 times the original recording time. Recently, attempts have been made to develop algorithms which make the manual coding of fixations faster (e.g., SemantiCode by Pontillo, Kinsman, & Pelz (2010) or similar implementations by eye tracking system developers) or avoid manual coding altogether: object recognition algorithms can be used to recognize objects at the image region around the gaze coordinate (Harmening & Pfeiffer, 2013; Toyama et al., 2012). These approaches, however, often rely on static environments and have problems differentiating between many similar looking objects (e.g., different packages of the same brand). When using mobile eye tracking in the virtual environment, researchers can easily change the definition of the areas of interest or add some new areas (as explained in Section 2). Because the position of the gaze in the 3D space is known, either absolute or relative to the individual objects, the allocation of fixations to AOIs can be easily calculated by rerunning the respective algorithms using geometric intersection tests (Pfeiffer, Renner, & Pfeiffer-Leßmann, 2016). This

3.4. Additional factors 3.4.1. Costs of creating test environment Mobile eye tracking equipment is still more costly than simple desktop eye tracking equipment. In the past, researchers who liked to record eye movements in virtual environments had to build the respective laboratory, which used to be costly and effortful. Things have changed, however, due to the advent of virtual reality technology on the consumer market with systems such as Oculus Rift and HTC Vive. Eye tracking vendors, such as PupilLabs or Tobii, today also provide their mobile eye tracking hardware for the integration with such consumer-market head-mounted displays. Together, all hardware required for mobile eye tracking in virtual reality has a cost similar to standard mobile eye tracking equipment. Still, the creation of the virtual simulation of a certain scenario will require effort in terms of 3D modeling and programming of 3D simulations. The created objects and code, however, are highly re-usable and, as has been explained below, support the reproducibility of the experimental setting.

3.4.2. Required expertise to acquire and analyze data The implementation of a virtual environment like the one described in Section 5 requires overcoming many hardware and software challenges (which researchers do not face when using mobile eye tracking in the real world). In most cases a computer scientist with focus on VR technology will be needed. In many cases the challenge will be to build the interaction process so that it feels natural to the user. 7

Journal of Business Research xxx (xxxx) xxx–xxx

M. Meißner et al.

of visual differentiation of a brand from its competitors and test the effects in a desktop eye tracking study. Given that brand salience has a significant influence on how consumers direct their attention in realworld retail settings (Chandon et al., 2009; Van der Lans et al., 2008), it is important to replicate these empirical findings in a setting which is more similar to real world purchase situations, but at the same time allows manipulating key factors, like the position in the shelf, in line with an experimental design. As outlined in Section 3, the virtual reality setting is the ideal environment for this kind of empirical test. Marketers might also want to understand how the influence of visual salience changes when consumers are in different search modes, e.g. when they are in an explorative (browsing) instead of a goal-directed search mode (Janiszewski, 1998). Consumers in a browsing mode are likely to gather information about the available products or scan the shelves for new and interesting items. Thus, exploratory search is more likely to lead to impulse purchases than goal-oriented search (Bhatnagar, De, Sen, & Sinha, 2016). It can be hypothesized that the salience of product packages is more important for bringing the attention to the respective product when consumers are in an exploratory search mode. At the same time, when consumers are familiar with the product, salient product packages may make information gathering more efficient and less time consuming (Janiszewski, 1998). The influence of visual salience can more easily be tested in a virtual environment in which the degree of the salience of the products can be manipulated and other factors, such as the position on the shelf, can be controlled for.

3.4.3. Reproducibility of the experimental setting The reproducibility of experimental settings is a problem for studies using mobile eye tracking in the real world. Due to real environments changing continuously, the study situation most often will have changed at a later point in time. Desktop eye tracking and also eye tracking in virtual environments allow presenting the exact same experimental scenarios at a later point in time. It would also allow the researcher to reanalyze an experimental situation by changing single factors of the experimental setup. For example, researchers could use the same product shelves, but change the salience of some products in the shelf (see next section). 3.5. Importance of the criteria The criteria discussed in this section may differ considerably with respect to their importance when researchers make the decision which research environment to use. For example, Husić-Mehmedović et al. (2017, p. 4) emphasize that they conducted their experiment in a laboratory “that allowed control over factors like shelf position and number of facing, both of which influence consumers' attention.” Control and randomization of factors is likely to be an important criterion for many researchers, whereas the required expertise might be of lower importance for most researchers as possible collaboration partners could bring in the respective expertise. 4. Implications for shopper research The goal of this section is to outline how mobile eye tracking in virtual reality can contribute to answering unresolved questions in retailing research. First, a basic goal is to better understand and describe consumers' in-store decision processes. Second, retailing research is interested in optimizing store designs by using virtual environments to save time and costs. Third, based on an improved understanding of information acquisition processes researchers can start building assistance systems that help consumers to make better shopping decisions, for example, by using augmented reality apps. All three subfields of research are highly interdisciplinary and are especially relevant for marketing and retailing, decision making, human-computer interaction as well as information systems research. Empirical studies in the three subfields contribute to an improved understanding of customer experience and the customer journey which is a leading management objective for many companies (Lemon & Verhoef, 2016).

4.1.2. Horizontal centrality Chandon et al. (2009) were the first to show that the horizontal position of the brand on the shelf influences the amount of attention to the brand and the choice probability. The results were then replicated and extended by Atalay et al. (2012) who showed that respondents looked at the centrally located products more often, especially just prior to making the decision, and were more likely to choose these centrally located products. While Greenacre, Martin, Patrick, & Jaeger (2016) replicated the results shown by Atalay et al. (2012) under two conditions, Meißner, Musalem, & Huber (2016) in their conjoint-analytic studies found only minimal effects on choice. All of the above-mentioned authors used 2D computer displays to test the effect of centrality on attention and choice. In addition, Atalay et al. (2012) used a simple product array, but not a shelf-like arrangement of the products. We argue that the question to which degree centrality drives attention and subsequent choice is still not answered sufficiently because previous empirical studies all used 2D computer displays (Bigné et al., 2016; Husić-Mehmedović et al., 2017). A valid test of the centrality effect requires a more realistic 3D environment. Important factors for the perception of objects in a shelf, such as the personal viewing perspective and the real efforts required for the information retrieval as well as efforts for reaching for objects have been neglected in previous empirical studies. Moreover, having increased experimental control in a more realistic 3D environment is important because the potential effect of horizontal centrality might interact with other bottom-up effects of visual attention, such as brand salience. For example, horizontal centrality might only increase choice probabilities if the centrally positioned products are also visually salient. However, research might also find that salient products are fixated independent of their central position. To our knowledge the interaction of brand salience and horizontal centrality has not been tested yet, especially not in real-world retail settings. We consider the outcome of such an experiment very important for practitioners as quantifying the size of the effect would allow to let producers pay for getting their products into a central shelf position.

4.1. Consumers' brand-related in-store decision processes Research studying visual attention distinguishes bottom-up and topdown attentional processes (Orquin and Mueller-Loose, 2013). Bottomup processes are also known as stimulus-driven whereas top-down processes are known as goal-driven processes. Shopper attention is a function of both high-level goals and expectations and low-level visual features of the store environment (Burke & Leykin, 2014). Having some purchase goals in mind, it will be one of the major tasks for many consumers to filter out most of what is seen, i.e. to discard irrelevant products and focus on and engage with relevant purchase items (Suher & Sorensen, 2010). 4.1.1. Visual salience Visual salience is one of the bottom-up processes that has been studied intensively. Salience comprises different aspects of visual conspicuity, such as color, movement or contrast. Understanding the importance of bottom-up processes of visual attention is especially important for brand managers who seek to direct the consumer's attention to their products in a competitive store environment. Van der Lans, Pieters, & Wedel (2008) defined brand salience as the extent to which a brand visually stands out from its competitors. The authors propose a methodology for competitive salience analysis by optimizing the degree

4.2. The retail environment and optimization of the store design Many consumers today are looking for multisensory experiences 8

Journal of Business Research xxx (xxxx) xxx–xxx

M. Meißner et al.

which are environmental friendly. A consumer with such “special” requirements coming to a typical store today can ask the supermarket staff for help and, increasingly, can receive support through dedicated mobile apps that for example provide health and dietary or price comparison information, however such apps are typically generic and not yet guiding a person to particular location or shelf in the store, and certainly, as yet, cannot tailor the shelf display to the individual consumer's needs. Providing individualized recommendations based on consumer preferences is a strategy to reduce consumer effort and help the consumer to achieve her shopping goal (Aljukhadar, Senecal, & Daoust, 2012). Identifying consumer preferences in retail settings, however, is difficult because oftentimes consumers do not provide preference information to retailers voluntarily or it takes too much time for the consumer to provide the respective information (Lu, Xiao, & Ding, 2016). An assistance system therefore ideally should provide the required information within milliseconds without increasing the effort on the consumer's side. So far a number of different technologies have been suggested to assist consumers. These technologies include the extended use of smartphone apps, augmented shopping trolleys, video tracking and smart glasses (Kalnikaite, Bird, & Rogers, 2013; Lee & Benbasat, 2010). Smartphone apps often include scanning a barcode using the mobile phone which requires some additional effort which may make the use of the technology unattractive for the consumer. Future shopping technology should therefore focus on information frugality and simplicity, which is a major challenge (Kalnikaite et al., 2013). While the information provided has to be relevant enough for consumers to help them achieve their behavioral goals, they should not become overwhelmed by too much information, which should be sufficiently simple and streamlined. The acquisition of additional information should be cost efficient. Currently, there is a lack of understanding in the retailing discipline as to how consumers will make use of additional information, if made available, and how such information will change their in-store attentional and purchase processes. In the not so far future, smart glasses (such as Google glasses) that include eye tracking technology will no doubt be able to provide personally-relevant information in many daily-life situations. With smart glasses the actual acquisition of additional information only requires movement of the eyes and so will take very little effort. The processing of all this information will however be an extra burden and require additional attention as well potentially result in a prolongation of the search process. From a commercial perspective providing personalized recommendations in natural purchase situations seems to be the next logical step, as it is now pretty much the standard to give personalized recommendations in online purchase settings based on what consumers purchased and searched before. In order to build smart glasses that analyze consumers' attentional processes in the real world, we propose to first of all investigate consumers attentional processes using mobile eye tracking in a virtual reality setting. In the next section, we outline the experimental setup for such an empirical study and present some basic descriptive findings.

seeking for hedonic and utilitarian value when shopping (Yaoyuneyong, Foster, Johnson, & Johnson, 2016). Retail stores and flag ship stores are therefore concerned with redesigning the interiors of their stores, for example, by making them much more entertaining (Kozinets et al., 2002). Retailers know that the atmosphere of a store is very important in many purchase contexts because it can largely influence the mood or arousal level of consumers, which may unconsciously influence how they scan the shelves and process the decision relevant information. Tesco, a large multinational grocer from the UK released a video showing how they test future retail environments using virtual reality (Corriea, 2014). Thus, virtual reality offers an opportunity to test which design makes a store more pleasant and consumer-specific. Research testing in-store design factors started using simpler approaches. Baker, Parasuraman, Grewal, & Voss (2002) investigated how different environmental cues together influence consumers' perceptions of a store environment. The authors used videotapes to simulate the store environment experience. Respondents simply viewed five-minute videotapes that “visually walked them through the store environment, simulating a shopping or browsing experience” (Baker et al., 2002, p. 129) and answered questions afterwards regarding the perception of the “simulated” environments. While this approach might be well suited to test simple in-store factors, such as color, shelf layout and general organization of the merchandise watching a videotape is still very different from experiencing the store-environment in a virtual reality setting. Respondents, for example, were not immersed and could not freely direct their attention. Virtual environments allow presenting the stimuli, i.e. alternative store designs, much more realistic and also permit to interact with the respective environment. We suggest that, in combination with mobile eye tracking, virtual environments are even more suited to evaluate several store design factors, such as the in-store communication and signage, the shelf configuration and the design of the product packages. That is because immersive environments allow consumers grading the design perceptions as well as the store music perceptions based on a holistic experience. Kahn (2017) emphasizes that retailers are especially interested in investigating which features of the assortment direct consumers' attention and which assortment variables retailers can use to increase the ease of processing (processing flow). As another example, Deng, Kahn, Unnava, & Lee (2016) test the effect of horizontal versus vertical display on assortment processing, perceived variety and choice in a desktop eye tracking study. Testing these effects in a properly designed 3D virtual environment would be helpful to test and demonstrate the external validity of these findings and thereby increase their relevance for retailing practice. Bottom-up factors that should be investigated in future studies are, for example, visual symbols that consumers can use to classify different brands, to interpret the meaning of product labels and features and to form price perceptions. Future research will also focus more on better understanding the interaction between humans and virtual agents, e.g. virtual sales agents giving recommendations. Jin (2009) and Qiu & Benbasat (2009), for example, suggest using humanlike avatars as interactive and animated recommendation agents, shopping assistants, or salespeople.

5. Example virtual reality study

4.3. Help consumers achieve their purchase goals

The demonstration is in the context of a project that aimed to design and test an assistance system that provides additional information in real time based on consumers' gaze behavior. In addition, we wanted to empirically test whether recommending products based on previous purchases influences consumers' choice behavior.We designed a virtual supermarket including several shelves with the goal to create a realistic shopping experience. Since our focus was not on testing store design factors (see Section 4.2) and the respondents did not have the task to walk around in the virtual store, we hold factors like lightning and color schemes constant (Turley & Milliman, 2000). Particularly, we asked respondents to focus on making decisions in two product categories in front of a single shelf. We put effort into making the lightning and

How do managers of regular supermarkets help consumers to achieve their purchase goals? The common practice in most stores seems to be to use in-store material and signage to highlight new products or products being on sale, improve the shelf layout or point to special offers. These conventional means for supporting and guiding customers are however not very specific or targeted to the consumer's needs. A special customer need, for example, could be that she is allergic against certain food ingredients or is diabetic and therefore wants to exclude respective products from further consideration. Another goal might be that she wants to find products which are low in calories or 9

Journal of Business Research xxx (xxxx) xxx–xxx

M. Meißner et al.

in scene 1 was fixated again 55 times (s.d. 50) in stage 3 in comparison to an average number of fixations per recommended product of 30 times (s.d. 30). Across respondents 43% of the attention to recommended products was directed to the star ratings. This result shows that respondents did not only look at prices, brand labels, details on the product packages or tried to locate the products on the shelf, but also directed a substantial amount of their attention to the augmented reality information. 69% of respondents decided to switch at least once in their four purchase decisions and in 29% of the decisions we observed a switch, meaning that they decided to change their initial selection from stage 1 to choosing one of the recommended products. We consider the percentage of respondents who switched as being substantial because switching in fact means that the respondents implicitly admit that they have not chosen the optimal product in stage 1. At the end of the experiment we asked respondents to what extent they would like to receive information in real supermarkets similar to the additional information in the augmented reality (“How desirable is it to get additional product information like the one you have just seen when making regular purchases in supermarkets on a 7-item scale from 1= not at all desirable to 7= very desirable?”). 69% of respondents answered that it would be rather desirable, desirable or very desirable to get additional product information. 23% of the respondents answered that it was neither undesirable nor desirable and only 8% answered that it would be rather undesirable to get this kind of information. Although obtained only for a small student sample, these results suggest that consumers may be interested in using the technology once it is available. Furthermore, we also asked how realistic the purchase experience in the virtual supermarket was. Only 20% of the respondents answered that the experimental setup was rather or very unrealistic (scale: 1 = very unrealistic, 2 = rather unrealistic, 3 = neutral, 4 = rather realistic, 5 = very realistic) while the remaining 80% evaluated the experience with the medium scale level or better. We conclude that this relatively low percentage of respondents who stated that our supermarket was unrealistic is a promising result given that we did not put much effort into improving the ambience of our virtual supermarket, as discussed above. However, at this point we cannot draw valid conclusions whether the VR setup was perceived as being more realistic than a standard 2D environment. To investigate that question, we would have to test the same experimental setup in a 2D environment and ask the respondents to rate the realism of the desktop display. Our study demonstrates that respondents made use of the additional information that was provided in the augmented reality and that it influenced the respondents' decisions substantially. Moreover, the additional questions asked at the end of the experiment show that the technology is evaluated as being helpful. A majority of respondents believed that it would be desirable to have additional information readily available in an augmented reality like way when making regular purchases in supermarkets. In sum and referring back to the criteria included in Table 1, the study first shows that the purchase decision has high ecological validity because it displays products in a very naturalistic manner (realism of stimulus display). Second, the consumer can interact with the shelf and therefore experience the products similar to the real world (realism of interaction). Third, in the study we randomize the product categories, the variants of additional information, the positions of products in the shelf and the selection of products (control and randomization of treatment and extraneous factors). Fourth, in the study we automatically monitor not only eye gazes but also the grasping of objects, the turning of objects and the movements of the respondents (possibility to monitor multiple modalities of respondents). This extra data can be analyzed to investigate how respondents interact with the shelves and thus can contribute to a largely unexplored question in retailing. Fifth, eye tracking data is annotated automatically by the system and therefore data analysis can start immediately without having to assign gazes to objects manually (effort and reliability of coding). Also gazes to areas of

products as realistic as possible and also inserted a 3D model of a shopping cart in the virtual reality environment in which respondents could put the chosen products. We chose granola and baking mixture as product categories and created assortments that represent the product range of a typical supermarket: 49 different sorts of granola and 44 different baking mixtures. We represented real products by scanning all six sides of original product packages using ordinary office document scanners and next putting these pictures as textures on 3D models of the packages; this took no more than about 30 min per product. The virtual supermarket was presented in a virtual reality lab at a large European university. The front projection screen of the CAVE environment was 3 m wide and 2.25 m high and had a resolution of 2100 × 1600. It displayed the respective product shelf using an Infitec passiv-stereo system and 2 NVIDIA Quadro K5000 graphic cards for rendering. The floor showed dark tiles and marked a central starting point as orientation for the respondents. Respondents entered the CAVE equipped with an input device called “flystick” which allows to grasp, flip, rotate and put back products from the shelf. They wore 3D–glasses that included the eye tracker device (SMI eye tracking glasses). The latter was connected to the eye tracking server with a long cable in order to allow freedom of movement in the CAVE. Experimental sessions started with a practice task to make respondents familiar with the virtual environment and to explain how to use the flystick. In the main part of the experiment, every respondent had to make four purchase decisions that consisted of three stages each, as depicted in Fig. 8.3 The respondents could spend as much time as they wanted in each stage and had to make two purchases in each of the product categories. The presentation order of the product categories was randomized. Each decision started with the calibration of the eye tracker using a nine-point calibration grid. In stage 1, respondents chose their preferred product out of 20 products from the shelf. We decided to show 20 products on each shelf so that respondents could choose multiple times from different randomly selected sets of products. There was no opt out option and respondents confirmed their choices by putting the selected product in their virtual shopping cart. Then, automatically, the next stage started. In stage 2, the same set of 20 products that had been shown in stage 1 reappeared and a red frame highlighted the initially chosen product (Fig. 8, stage 2). We instructed respondents to look at the chosen product again and access the available additional information. This information appeared in a bubble (see Fig. 8, stage 2) that popped up automatically next to the product at the moment the eye-tracker registered a gaze longer than 200 milliseconds on the product. This information comprised nutrition facts and a consumer rating. In stage 3, six product recommendations were highlighted with a blue frame (Fig. 8, stage 3) and respondents decided whether they wanted to stay with the initially chosen product from stage 1 or whether they wanted to switch to one of the six recommended ones. Additional information could be accessed exactly as in stage 2 for all highlighted products by looking at a product. For all recommended products we generated the additional information for customer ratings and nutrition ratings so that the recommended products were in trade-off conflict with the product that had been chosen in stage 1. We expected this trade-off conflict to result in an increased relevance of the additional information. We however leave the analysis of this effect for future reference and regard it as beyond the scope of the present paper; instead the current purpose is to illustrate how observations for different conditions can be obtained. The study included 33 respondents (average age: 23.8 years, 62% females, all students). Basic descriptive analyses show that it took respondents on average 48 s (s.d. 20 s) to make a decision in the last stage which included the recommendations. Each product originally chosen

3 A video showing the respective sequence of stages is available here: http://dx.doi. org/10.1016/j.jbusres.2017.09.028

10

Journal of Business Research xxx (xxxx) xxx–xxx

M. Meißner et al.

Fig. 8. Setup of our mobile eye tracking study in virtual reality with augmentations of additional information using colored outlines and information bubbles.

now able to easily conduct follow ups or repetitions of the study (reproducibility of the experimental setting). The complete experimental virtual reality setup can be shared with interested researchers (and is available from the authors upon request), making reproduction and validation much simpler than in mobile eye tracking studies in the field.

interest on packages while they are in the respondents' hands and turned around can be analyzed automatically. Sixth, the automatic annotation allows to build assistance systems that provide additional information in real time based on consumers' gaze behavior (analyze and react to respondent's attention and behavior in real time). Seventh, based on the experimental setup we have created for this study, we are 11

Journal of Business Research xxx (xxxx) xxx–xxx

M. Meißner et al.

6. Conclusions

of new technologies and the consumers' intended behavioral reaction in the retailing context and should be applied to more rigorously test the acceptance of the technology. It can be expected that in the near future eye tracking technology will be integrated into many electronic devices, such as mobile phones, tablets, or laptops that will generate large quantities of eye tracking data and can be used to get insights into the underlying search and choice processes of consumers (Van der Lans and Wedel, 2017). Moreover, new types of smart glasses, as discribed above, will allow building context-aware assistance systems. Such gaze-based assistance systems have the potential to greatly change retailing research and practice. We believe that helping consumers to fulfill their personal needs should positively influence consumers' satisfaction with the shopping experience and retention. Thus, we agree with Grewal, Roggeveen, & Nordfält (2017) who emphasized recently that VR is one of the new technologies that is revolutionizing the consumer shopping experience and will change the expectations of what shopping should be in the future. The development of assistance systems is one of the important developments in retailing research and has many implications for retailers, consumers and for society. These include many ethical issues emerging from such use of technology (Stanton, SinnottArmstrong, & Huettel, 2016) but also a potentially huge positive impact on society and consumers as they are able to be better informed and to receive more relevant information as they make their product selections. One of the most urgent tasks for future research is to investigate how similarly information is processed when using mobile eye tracking in the field and in the virtual reality (as well as compared to desktopbased eye tracking). Van Herpen, van den Broek, van Trijp, & Yu (2016) focused on the shopping behavior and found that VR elicits behavior that is more similar to behavior in a physical store than to behavior when respondents see 2D pictorial stimuli. It is however unclear whether respondents perhaps process information differently in the VR. That might be the case because respondents are not yet used to the VR environment or because of important structural differences remaining between VR and real store environments. We are not aware of studies that have systematically tested attentional differences using eye tracking and suggest to investigate this in future empirical work. Supplementary data to this article can be found online at http://dx. doi.org/10.1016/j.jbusres.2017.09.028.

This paper discussed advantages and disadvantages of using eye tracking technologies in desktop, natural and virtual environments. We consider this discussion to be a first key contribution because challenges and efforts put into mobile eye tracking studies are seldom made explicit in academic literature. First, the need for manual annotation and preparation of mobile eye tracking data results in extensive work demands, which hinders the adoption and use of this type of eye tracking as a method. Second, as is typical for field settings, while they enhance external validity, they have a limited reproducibility. This limitation therefore exacerbates the earlier mentioned issue, as every respondent requires unique and laborious coding of the unique and changing field setting. Third, respondents in such field settings cannot be analyzed in real time and therefore marketers cannot give recommendations based on real-time gaze information. By using eye tracking in virtual environments all three of these problems can be resolved. We therefore suggest that researchers consider using these environments as an option when thinking about how to conduct their future eye tracking studies. The second contribution of our paper is that we outline how research in retailing and decision making can benefit from the use of mobile eye tracking in virtual reality. Virtual reality retains most of the perceptual modalities including especially 3D perception and movement sensation and thus presents an experience that is closer to the real life target setting. This means it can provide a greater external validity of findings when the aim is to generalize findings to real store settings. We illustrated the setup of an eye tracking study in virtual reality using augmented reality information displays. Furthermore, the study exemplifies how by showing additional product information as reactions to the users' pure eye movements the environment interacts with the user in real time. The automatic displaying of additional product information once the user looks at a product is a feature that is not available in real stores but no doubt will become available as time proceeds, for example in smart glasses with integrated eye tracking. Our virtual reality setting allows early and efficient testing of consumer responses to such a new shopping feature. Thus, an important task for future research is to test the acceptance of augmented reality information displays among consumers. Inman & Nikolova (2017) recently developed a theoretical framework that considers the perception Appendix A. Appendix

A VR system needs to update the presented computer graphical renderings to match the rendered perspective to the current perspective of the user. These updates have to be made with 30 Hz or faster in a CAVE and with 90 Hz or faster when using an HMD. Also, the updates need to have a low latency, at best below 5 ms. Typical latencies in a CAVE system would be around 30 to 50 ms due to the projection technology being used (see e.g. Waltemate, Hülsmann, Pfeiffer, Kopp, & Botsch (2015)). Current consumer HMD systems, such as the HTC Vive or the Oculus Rift, are tuned to provide motion to photon latencies, i.e. from the movement of the user to the updated resulting photon that hits the eye of the user, of about 15 ms. If the virtual reality simulation exceeds these latencies (5–15 ms for HMDs, 50 ms for CAVEs), the chance for evoking simulator sickness in the users of the virtual reality environment increases. People are more sensitive to latencies in HMDs, as they cover the whole field of view. If the head is moved and the display of the HMD is not updated in time, no optical flow is perceived during the movement, which is not the natural sensation of humans. Projection-based systems, such as CAVEs, remain physically stable, so if the head moves, the eyes will always perceive an optical flow, even when the perspective of the image is not yet properly updated. This apparently evokes less dissonances in our perceptual system. There are different tracking technologies available for tracking the user's head pose. The HMDs produced for the end-consumer market often come with their own tracking system: HTC Vive comes with Lighthouse for room scale tracking, the Oculus Rift with Oculus Sensor for smaller interaction spaces. Both systems also come with a tracked controller for manual interactions (Vive controllers vs. Oculus Touch controllers). Other tracking devices for body movements are Microsoft Kinect (V1 or V2, https://developer.microsoft.com/de-de/windows/kinect/develop) from the consumer market or the more expensive marker-based optical motion capturing systems from Advanced Real-time Tracking (http://www.artracking.com), Vicon (https://www.vicon.com/products/software/tracker) or OptiTrack (http://optitrack.com/motion-capture-virtual-reality/) from NaturalPoint. Both CAVE and HMD have a high degree of immersion. However, there is an important difference: in the CAVE-based systems, the user's body is between her eyes and the projected content. In the HMD-based system, the screen is between the user and her body. Thus, when using an HMD, the user will not be able to perceive her real body, while the body is easily visible in the CAVE. Special efforts have to be taken for HMD setups to recreate a 3D model of the user and adapt its appearance to the user's motion in real time. As the body is an important external reference for our interpretation of the environment, CAVE environments are providing more realistic cues to the user.

12

Journal of Business Research xxx (xxxx) xxx–xxx

M. Meißner et al.

References

Holmqvist, K., Nystrom, M., Andersson, R., Dewhurst, R., Jarodzka, H., & Van de Weijer, J. (2015). Eye tracking: A comprehensive guide to methods and measures. Oxford University Press. Husić-Mehmedović, M., Omeragić, I., Batagelj, Z., & Kolar, T. (2017). Seeing is not necessarily liking: Advancing research on package design with eye-tracking. Journal of Business Research, 80, 145–154. Inman, J. J., & Nikolova, H. (2017). Shopper-facing retail technology: A retailer adoption decision framework incorporating shopper attitudes and privacy concerns. Journal of Retailing, 93(1), 7–28. Janiszewski, C. (1998). The influence of display characteristics on visual exploratory search behavior. Journal of Consumer Research, 25(3), 290–301. Jin, S. A. A. (2009). The roles of modality richness and involvement in shopping behavior in 3D virtual stores. Journal of Interactive Marketing, 23(3), 234–246. Kahn, B. E. (2017). Using visual design to improve customer perceptions of online assortments. Journal of Retailing, 93(1), 29–42. Kalnikaite, V., Bird, J., & Rogers, Y. (2013). Decision-making in the aisles: Informing, overwhelming or nudging supermarket shoppers? Personal and Ubiquitous Computing, 17(6), 1247–1259. Kozinets, R. V., Sherry, J. F., DeBerry-Spence, B., Duhachek, A., Nuttavuthisit, K., & Storm, D. (2002). Themed flagship brand stores in the new millennium: Theory, practice, prospects. Journal of Retailing, 78(1), 17–29. Kurzhals, K., Hlawatsch, M., Seeger, C., & Weiskopf, D. (2017). Visual analytics for mobile eye tracking. IEEE Transactions on Visualization and Computer Graphics, 23(1), 301–310. Lee, Y. E., & Benbasat, I. (2010). Interaction design for mobile product recommendation agents: Supporting users' decisions in retail stores. ACM Transactions on Computer Human Interaction TOCHI, 17(4), 9–38. Léger, P. M., Sénecal, S., Courtemanche, F., de Guinea, A. O., Titah, R., Fredette, M., & Labonte-LeMoyne, É. (2014). Precision is in the eye of the beholder: Application of eye fixation-related potentials to information systems research. Journal of the Association for Information Systems, 15(10), 651–678. Lemon, K. N., & Verhoef, P. C. (2016). Understanding customer experience throughout the customer journey. Journal of Marketing, 80(6), 69–96. Loomis, J. M., Blascovich, J. J., & Beall, A. C. (1999). Immersive virtual environment technology as a basic research tool in psychology. Behavior Research Methods, Instruments, & Computers, 31(4), 557–564. Losing, V., Rottkamp, L., Zeunert, M., & Pfeiffer, T. (2014). Guiding visual search tasks using gaze-contingent auditory feedback. Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing (pp. 1093–1102). Adjunct Publication. Lu, S., Xiao, L., & Ding, M. (2016). A video-based automated recommender (VAR) system for garments. Marketing Science, 35(3), 484–510. McGrath, J. E. (1981). Dilemmatics: The study of research choices and dilemmas. The American Behavioral Scientist, 25(2), 179–210. Meißner, M., Musalem, A., & Huber, J. (2016). Eye tracking reveals processes that enable conjoint choices to become increasingly efficient with practice. Journal of Marketing Research, 53(1), 1–17. Meli, L., Salvietti, G., Gioioso, G., Malvezzi, M., & Prattichizzo, D. (2016). Multi-contact bilateral telemanipulation using wearable haptics. Proceedings of the 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 1431–1436). . Mihelj, M., Novak, D., & Beguš, S. (2014). Virtual reality technology and applications. Dordrecht: Springer. Milosavljevic, M., Navalpakkam, V., Koch, C., & Rangel, A. (2012). Relative visual salience differences induce sizable bias in consumer choice. Journal of Consumer Psychology, 22(1), 67–74. Orquin, J. L., & Mueller-Loose, S. (2013). Attention and choice: A review on eye movements in decision making. Acta Psychologica, 144(1), 190–206. Otterbring, T., Wästlund, E., Gustafsson, A., & Shams, P. (2014). Vision (im) possible? The effects of in-store signage on customers' visual attention. Journal of Retailing and Consumer Services, 21(5), 676–684. Papoutsaki, A., Laskey, J., & Huang, J. (2017). SearchGazer: Webcam eye tracking for remote studies of web search. Proceedings of the 2017 conference on conference human information interaction and retrieval (pp. 17–26). ACM. Papoutsaki, A., Sangkloy, P., Laskey, J., Daskalova, N., Huang, J., & Hays, J. (2016). Webgazer: Scalable webcam eye tracking using user interactions. Proceedings of the twenty-fifth international joint conference on artificial intelligence - IJCAI. 2016. Proceedings of the twenty-fifth international joint conference on artificial intelligence IJCAI (pp. 3839–3845). Pfeiffer, J., Meißner, M., Prosiegel, J., & Pfeiffer, T. (2014). Classification of goal-directed search and exploratory search using mobile eye tracking. Proceedings of the international conference on information systems. ICIS. Pfeiffer, T., Renner, P., & Pfeiffer-Leßmann, N. (2016). EyeSee3D 2.0: Model-based realtime analysis of mobile eye tracking in static and dynamic three-dimensional scenes. Proceedings of the ninth biennial ACM symposium on eye tracking research & applications (pp. 189–196). . Pontillo, D. F., Kinsman, T. B., & Pelz, J. B. (2010). SemantiCode: Using content similarity and database-driven matching to code wearable eyetracker gaze data. Proceedings of the 2010 symposium on eye tracking research & applications (pp. 267–270). . Purschke, F., Rabätje, R., Schulze, M., Starke, A., Symietz, M., & Zimmermann, P. (1998). Virtual reality (VR) - New methods for improving and accelerating vehicle development. In F. Dai (Ed.). Virtual reality for industrial applications (pp. 105–122). Berlin, Heidelberg: Springer. Qiu, L., & Benbasat, I. (2009). Evaluating anthropomorphic product recommendation agents: A social relationship perspective to designing information systems. Journal of Management Information Systems, 25(4), 145–182. Semmelmann, K., & Weigelt, S. (2017). Online webcam-based eye tracking in cognitive

Aljukhadar, M., Senecal, S., & Daoust, C. E. (2012). Using recommendation agents to cope with information overload. International Journal of Electronic Commerce, 17(2), 41–70. Atalay, A. S., Bodur, H. O., & Rasolofoarison, D. (2012). Shining in the center: Central gaze cascade effect on product choice. Journal of Consumer Research, 39(4), 848–866. Baker, J., Parasuraman, A., Grewal, D., & Voss, G. B. (2002). The influence of multiple store environment cues on perceived merchandise value and patronage intentions. Journal of Marketing, 66(2), 120–141. Bhatnagar, A., De, P., Sen, A., & Sinha, A. P. (2016). Goal-directed and exploratory search under competing demands for attention: The influence of promotional tools on online search behavior, working paper. 1–48. (Retrieved September, 2017 from) https:// papers.ssrn.com/sol3/papers.cfm?abstract_id=2817978. Bigné, E., Llinares, C., & Torrecilla, C. (2016). Elapsed time on first buying triggers brand choices within a category: A virtual reality-based study. Journal of Business Research, 69(4), 1423–1427. Blake, J., & Gurocak, H. B. (2009). Haptic glove with MR brakes for virtual reality. IEEE/ ASME Transactions on Mechatronics, 14(5), 606–615. Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L., & Bailenson, J. N. (2002). Immersive virtual environment technology as a methodological tool for social psychology. Psychological Inquiry, 13(2), 103–124. Brade, J., Lorenz, M., Busch, M., Hammer, N., Tscheligi, M., & Klimant, P. (2017). Being there again–presence in real and virtual environments and its relation to usability and user experience using a mobile navigation task. International Journal of HumanComputer Studies, 101, 76–87. Brône, G., Oben, B., & Goedemé, T. (2011). Towards a more effective method for analyzing mobile eye tracking data: Integrating gaze data with object recognition algorithms. Proceedings of the 1st international workshop on pervasive eye tracking & mobile eye-based interaction (pp. 53–56). . Burke, R. R., & Leykin, A. (2014). Identifying the drivers of shopper attention, engagement, and purchase. Shopper marketing and the role of in-store marketing. Review of Marketing Research, 11, 147–187. Chandon, P., Hutchinson, J. W., Bradlow, E. T., & Young, S. H. (2009). Does in-store marketing work? Effects of the number and position of shelf facings on brand attention and evaluation at the point of purchase. Journal of Marketing, 73(6), 1–17. Clement, J., Kristensen, T., & Grønhaug, K. (2013). Understanding consumers' in-store visual perception: The influence of package design features on visual attention. Journal of Retailing and Consumer Services, 20(2), 234–239. Clement, J., Smith, V. J. M. C. D., Zlatev, J., Gidlöf, K., & van de Weijer, J. (2017). Assessing information on food packages. European Journal of Marketing, 51(1), 219–237. Corriea, A. (2014). Supermarket giant Tesco looking into Oculus Rift to change consumer behavior (Retrieved September, 2017 from) http://www.polygon.com/2014/3/26/ 5549890/supermarketgiant-tesco-looking-into-oculus-rift-to-change-consumer. Cruz-Neira, C., Sandin, D. J., & DeFanti, T. A. (1993). Surround-screen projection-based virtual reality: the design and implementation of the CAVE. Proceedings of the 20th annual conference on computer graphics and interactive techniques (pp. 135–142). . Deng, X., Kahn, B. E., Unnava, H. R., & Lee, H. (2016). A “wide” variety: Effects of horizontal versus vertical display on assortment processing, perceived variety, and choice. Journal of Marketing Research, 53(5), 682–698. Duchowski, A. T. (2002). A breadth-first survey of eye-tracking applications. Behavior Research Methods, Instruments, & Computers, 34(4), 455–470. Durlach, N., & Mavor, A. S. (1995). Virtual reality: Scientific and technological challenges. Washington: National Academy Press. Felfernig, A., Jeran, M., Ninaus, G., Reinfrank, F., & Reiterer, S. (2013). Toward the next generation of recommender systems: Applications and research challenges. In G. A. Tsihrintzis, M. Virvou, & L. C. Jain (Eds.). Multimedia services in intelligent environments: Recommendation services (pp. 81–98). Heidelberg: Springer (et al.). Foulsham, T., Walker, E., & Kingstone, A. (2011). The where, what and when of gaze allocation in the lab and the natural environment. Vision Research, 51(17), 1920–1931. Gidlöf, K., Wallin, A., Dewhurst, R., & Holmqvist, K. (2013). Using eye tracking to trace a cognitive process: Gaze behaviour during decision making in a natural environment. Journal of Eye Movement Research, 6(1), 1–14. González, G., López, B., Angulo, C., & de la Rosa, J. L. (2005). Acquiring unobtrusive relevance feedback through eye-tracking in ambient recommender systems. In B. López, J. Meléndez, P. Radeva, & J. Vitrià (Eds.). Artificial intelligence research and development (pp. 181–188). Amsterdam: IOSD Press (et al.). Greenacre, L., Martin, J., Patrick, S., & Jaeger, V. (2016). Boundaries of the centrality effect during product choice. Journal of Retailing and Consumer Services, 32, 32–38. Grewal, D., Ailawadi, K. L., Gauri, D., Hall, K., Kopalle, P., & Robertson, J. R. (2011). Innovations in retail pricing and promotions. Journal of Retailing, 87(S1), S43–S52. Grewal, D., Roggeveen, A. L., & Nordfält, J. (2017). The future of retailing. Journal of Retailing, 93(1), 1–6. Harmening, K., & Pfeiffer, T. (2013). Location-based online identification of objects in the centre of visual attention using eye tracking. Proceedings of the first international workshop on solutions for automatic gaze-data analysis 2013 (pp. 38–40). (SAGA 2013). Harwood, T., & Jones, M. (2014). Mobile eye tracking in retail research. In M. Horsley, M. Eliot, B. A. Knight, & R. Reilly (Eds.). Current trends in eye tracking research (pp. 183– 199). Cham: Springer International Publishing (et al.). Hayhoe, M., & Ballard, D. (2005). Eye movements in natural behavior. Trends in Cognitive Sciences, 9(4), 188–194. Hendrickson, K., & Ailawadi, K. L. (2014). Six lessons for in-store marketing from six years of mobile eye tracking research. Shopper marketing and the role of in-store marketing. Review of Marketing Research, 11, 57–74.

13

Journal of Business Research xxx (xxxx) xxx–xxx

M. Meißner et al. science: A first look. Behavior Research Methods, 1–15. Shankar, V., Inman, J. J., Mantrala, M., Kelley, E., & Rizley, R. (2011). Innovations in shopper marketing: Current insights and future research issues. Journal of Retailing, 87(1), S29–S42. Stanton, S. J., Sinnott-Armstrong, W., & Huettel, S. A. (2016). Neuromarketing: Ethical implications of its use and potential misuse. Journal of Business Ethics, 1–13. Steuer, J. (1992). Defining virtual reality: Dimensions determining telepresence. The Journal of Communication, 42(4), 73–93. Suher, J., & Sorensen, H. (2010). The power of atlas: Why in-store shopping behavior matters. Journal of Advertising Research, 50(1), 21–29. Toyama, T., Kieninger, T., Shafait, F., & Dengel, A. (2012). Gaze guided object recognition using a head-mounted eye tracker. Proceedings of the symposium on eye tracking research and applications (pp. 91–98). . Turley, L. W., & Milliman, R. E. (2000). Atmospheric effects on shopping behavior: A review of the experimental evidence. Journal of Business Research, 49(2), 193–211. Van der Lans, R., Pieters, R., & Wedel, M. (2008). Eye-movement analysis of search effectiveness. Journal of the American Statistical Association, 103(482), 452–461. Van der Lans, R., & Wedel, M. (2017). Eye movements during search and choice. In B. Wierenga, & R. Van der Lans (Eds.). Handbook of marketing decision models (pp. 331– 359). Cham: Springer. Van Herpen, E., van den Broek, E., van Trijp, H. C., & Yu, T. (2016). Can a virtual supermarket bring realism into the lab? Comparing shopping behavior using virtual and pictorial store representations to behavior in a physical store. Appetite, 107, 196–207. Waltemate, T., Hülsmann, F., Pfeiffer, T., Kopp, S., & Botsch, M. (2015). Realizing a lowlatency virtual reality environment for motor learning. Proceedings of the 21st ACM symposium on virtual reality software and technology (pp. 139–147). . Wästlund, E., Otterbring, T., Gustafsson, A., & Shams, P. (2015). Heuristics and resource depletion: Eye tracking customers' in situ gaze behavior in the field. Journal of Business Research, 68(1), 95–101. Wedel, M., & Pieters, R. (2008). A review of eye tracking research in marketing. In N. K. Malhotra (Ed.). Review of marketing research (pp. 123–147). Armonk, NY: M.E. Sharpe. Yao, J., & Oppewal, H. (2016). Unit pricing increases price sensitivity even when products are of identical size. Journal of Retailing, 92(1), 109–121. Yaoyuneyong, G., Foster, J., Johnson, E., & Johnson, D. (2016). Augmented reality marketing: Consumer preferences and attitudes towards hypermedia print ads. Journal of Interactive Advertising, 16(1), 16–30.

holds a PhD in Marketing from Bielefeld University. His research focuses on the analysis and modeling of eye-tracking data, the analysis of brand images, as well as on the development of preference measurement methods. His work has been published, among others, in the Journal of Marketing Research, the International Journal of Innovation Management, and the Proceedings of the International Conference on Information Systems. Jella Pfeiffer is post-doc and “Privatdozent” (PD) at the Institute of Information Systems and Marketing at the Karlsruhe Institute of Technology (KIT). At KIT she also is manager of the Karlsruhe Decision & Design Laboratory (KD2Lab). She received her PhD in Information Systems and her Venia Legendi (“Habilitation”) in Business Administration from the University of Mainz. Her research interests include decision support systems in ecommerce and m-commerce, consumer decision making, Neuro Information Systems (in particular eye-tracking), human-computer-interaction and experimental research in the laboratory, the field and the virtual reality. Her work has been published, among others, in the Journal of the Association for Information Systems, the European Journal of Operational Research and the Journal of Behavioral Decision Making. Thies Pfeiffer is senior researcher at the Center of Excellence Cognitive Interaction Technology at Bielefeld University, were he is technical director of the virtual reality lab. He holds a doctoral degree in informatics (Dr. rer. nat.) with a specialization in humanmachine interaction. His research interests include human-machine interaction with a strong focus on gaze and gesture, augmented and virtual reality, as well as immersive simulations for prototyping. He has organized several scientific events related to the topic of this paper, such as the GI Workshop on Virtual and Augmented Reality 2016, the Workshops on Solutions for Automatic Gaze Analysis (SAGA) in 2013 and 2015, and several others (PETMEI 2014, ISACS 2014). Currently, he is principle investigator in research projects on training in virtual reality (ICSPACE, DFG), augmented reality-based assistance systems (ADAMAAS, BMBF) and prototyping for augmented reality (ProFI, BMBF). Harmen Oppewal is Professor and Head of the Department of Marketing at Monash University, Australia. He holds a PhD from the Technical University of Eindhoven in the Netherlands. His research centers on modeling and understanding consumer decisionmaking behavior in retail and services contexts, in particular assortment perception, channel and destination choice, preference formation, and visualization effects. He has over fifty publications in leading journals including the Journal of Business Research, Journal of Consumer Research, Journal of Marketing Research, and the Journal of Retailing, among others.

Martin Meißner is Associate Professor at the Department of Sociology, Environmental and Business Economics at the University of Southern Denmark (Denmark) and an adjunct senior lecturer at the Department of Marketing at Monash University (Australia). He

14