- Email: [email protected]
Using eye-tracking to for analyzing case study materials
The International Journal of Management Education 17 (2019) 304–315
Contents lists available at ScienceDirect
The International Journal of Management Education journal homepage: www.elsevier.com/locate/ijme
Using eye-tracking to for analyzing case study materials Thomas Berger
T
DHBW Stuttgart, Kronenstr. 40, 70174, Stuttgart, Germany
ARTICLE INFO
ABSTRACT
Keywords: Case method Eye tracking Cognitive load theory Management education
Case studies are an important part of management education, but especially when working with diverse classes, differences in understanding the case materials or in cognitive processing are often unclear. Existing studies use interviews, questionnaires or tests, but cannot gain deeper insights into information processing. Information-processing data would be valuable in designing case materials, e.g., to be able to adjust the cognitive load to different groups of students by assessing difficulty or reading depth. Eye tracking can provide information about where participants focused for how long, what level of cognitive processing took place, and in which areas of the case materials provided. The objective of this paper is to show how eye tracking can be used to gain additional insights into how participants work with the cases. This is illustrated with the help of a study with Indian and German management students and a small case on Expected Utility Theory. Different variables for analyzing case materials are introduced, e.g., fixation duration, dwell time, or reading depth. It is argued that eye tracking can be used more often to analyze case studies to ensure that the cognitive processing matches the intended level and inform instructors about what does and does not attract attention.
1. Introduction Management education around the world uses case studies as one of its prime forms of introduction in addition to classical lectures. This is especially true for MBA programs but is also used for other degrees (Aithal, 2016; Blankley, Kerr, & Wiggins, 2017; Jennings, 1996). The case method is seen as a more active tool of learning than lectures in a controlled setting as well as being more realistic than lectures (Alfieri, Brooks, Aldrich, & Tenenbaum, 2011; Farashahi & Tajeddin, 2018). Cases may be as long as 20 written pages, sometimes with ample additional information like separate tables, graphs, or online databases, but smaller cases, sometimes called caselets, can be as short as one page (Selvam, Babu, & Raja, 2006). Besides their flexibility in length, cases are also very flexible in content and can easily be adjusted to different fields, specific goals of learning content, or different objectives. Lundberg, Rainsford, Shay, and Young (2001, p. 461) differentiate among three objectives in choosing the focus a case: 1) “Acquiring, Differentiating and Using Ideas”; 2) “Issue Identification and Differentiation”; and 3) “Action Formulation and Implementation.” There are, of course, other categorizations, as the one by Jennings (1996), who identifies 13 learning objectives, and Shrivakumar (2012) who identifies six types of cases. Independent from the concrete objective and the type of categorization, the case description (if texts are used) is central to the case method, forming the basis for the casework. When reading through the materials, students must process the given information to create meaning, imposing “heavy loads on working memory detrimental to learning” (Alfieri et al., 2011, p. 3). Working memory capacities must be used efficiently during case-based learning in order to avoid so-called “overload” (Sweller, 1988) that would hinder learning, according to the Cognitive Load Theory (CLT) proposed by Sweller (1988). Instructors also should make sure that the design of the case does not draw disproportionate
E-mail address: [email protected]. https://doi.org/10.1016/j.ijme.2019.05.002 Received 25 November 2018; Received in revised form 29 March 2019; Accepted 12 May 2019 Available online 28 May 2019 1472-8117/ © 2019 Elsevier Ltd. All rights reserved.
The International Journal of Management Education 17 (2019) 304–315
T. Berger
attention to certain aspects, words, or visual displays. Sometimes instructors may want to display more information than is needed so that students must filter the relevant aspects, while at other times, the case is constructed so that no distractors stand out (as is often true when working with caselets). It is therefore the instructor's responsibility not only to focus on the content, style, and flow of a case when preparing it, but also to emphasize the students' possible cognitive processing that takes place when working with a case. Having some indicative results on this would be helpful in designing a case. Cognitive processing—often focusing on cognitive load—is currently assessed primarily via self-reports of difficulty or test scales, such as the NASA task-load index. This is mostly done after the task is fulfilled, which can bias the results (Debue & van de Leemput, 2014). For instructors, it would be advantageous to gain some insights into students’ level of cognitive processing during rather than after the casework. That way, instructors could link that processing to certain aspects of the case description and adjust the materials according to the desired level of cognitive processing. Such an analysis can use eye trackers that record eye movements during a variety of tasks. This study proposes using eye tracking for the analysis of cognitive processing when working with cases, based on the so called “eye-mind hypothesis” (Just & Carpenter, 1980) assuming that eye-tracking provides information “about the user's cognitive activities” (Zagermann, Pfeil, & Reiterer, 2016, p. 79), e.g., indicating attention or cognitive load as set out by the CLT. Currently, eye tracking is applied in a variety of fields related to education, such as exploring learning in general (Lai et al., 2013), testing (Lindner et al., 2014; Tsai, Hou, Lai, Liu, & Yang, 2012), and global text processing (Altmann & Kamide, 2007; Hyönä, Lorch, & Rinck, 2003), but not yet for analyzing case materials. The key contribution of this article, therefore, is to show how this technology can be applied to evaluate case materials with respect to cognitive processing. Eye tracking could be applied to enrich the current methodologies used to analyze the effectiveness of cases. This study, therefore, might be of use for future researchers by showing what kind of metrics can be used to analyze data on case analysis, what to keep in mind when recording data, and how such data could be statistically analyzed and the results presented. As a result of such studies, case materials for students (or teaching notes for instructors) could be adjusted to hint at possible characteristics, e.g., where most attention might be paid to and how to assist in working with the case based on cognitive processing. In this paper, an overview of existing studies on the case method is first provided before introducing the eye-tracking method, variables and study design. In addition to general visualizations of attention, different variables for analyzing cognitive processing in more detail are explained, based on measures of fixations, i.e., times when the eyes are focusing. Efficiency in reading, difficulty in understanding, and how to measure the level of cognitive load are discussed. In addition, reading depth is introduced along with how reading patterns could be analyzed, making use of transitions and areas of interest. Finally, it is shown how to conduct a search for patterns while reviewing the materials to learn the order in which participants go through the materials. A study with German and Indian management students where these variables are applied is then included as an example of such an analysis. In the last section, the use of eye tracking for studies is discussed, along with an outlook for potential fields of research. 2. Research on case method teaching As a pedagogical tool, the case method is itself an object of empirical research. Research focusing on the effectiveness of the case study method hints at its higher effectiveness in teaching in general compared to lectures but is not as clear about its value compared to that of simulations. Druckman and Ebner (2018) saw a higher effectiveness of case-based teaching by comparing the design of cases where “the learner is asked to actively create an artifact” (Druckman & Ebner, 2018, p. 353) to case analysis where learners work with an existing case. Both approaches of guided discovery learning were used to teach cognitive-bias concepts. They found that case analysis compared to the design of cases was more effective in the short term, but less ineffective so in the long term, but that both were always more effective than lectures. Farashahi and Tajeddin (2018) found that simulations and case studies were more effective than lectures regarding problem-solving skills (but not so for interpersonal skills or self-awareness), which is partially contrary to the findings of Jennings (1996), whose 1996 study —which was conducted by asking instructors—found that instructors mostly saw communication and interpersonal skills as the prime advantage of case studies. Other findings suggest that students were mostly more satisfied when working with cases, as they are more actively involved than in lectures (Bayona & Castañeda, 2017; Chu & Libby, 2010; Hoag, Lillie, & Hoppe, 2005; Hughes, 2017; Thistlethwaite et al., 2012). The above findings show “evidence for the technique's effectiveness,” as Easton and Ormerod (2001, p. 2) state in their meta-study. Case-based teaching has also been criticized. One strand of criticism focuses on the over-simplification of reality in the case description (Jennings, 1996; Mintzberg, 2005), together with the focus on (rational) decision-making, which means that students read some pages of materials and must then make a decision without having a proper understanding of the situation (Argyris, 1980; Lund Dean & Fornaciari, 2016; Lundberg et al., 2001). In addition, some also criticize the over-reliance on this method in business schools, leaving aside other important methods of instruction (Mintzberg, 2005). While these may be valid points, in this paper we focus on the cognitive processing during such case analysis and leave aside the discussion on how many, when, and what kind of cases should be used. The use of eye tracking to inform us about the cognitive processing of case materials is independent of most of these points of criticism of the case method. On the contrary, eye tracking may even help improve case materials. When we look at existing research on case teaching, we can see that those studies typically focus on effectiveness or satisfaction and involve tests or questionnaires—distributed to the students afterward—or interviews conducted to reflect on the students'
305
The International Journal of Management Education 17 (2019) 304–315
T. Berger
experiences with the case (Burgoyne & Mumford, 2001; Chu & Libby, 2010; Hughes, 2017; Jennings, 1996; Nkhoma, Sriratanaviriyakul, & Le Quang, 2017). A few studies, like the ones by Bayona and Castañeda (2017) or Easton and Ormerod (2001), also include observations recorded on video to gain insights into students' or teachers' interactions. The study by Bayona and Castañeda (2017) combined this method with self-testing by the instructors to analyze the role of personality when teaching with cases. These methods are of use for the above-stated objectives like student or instructor satisfaction, social interaction or the instructor's personality, but not if one wants to know how attention is distributed over the given materials or the kind of cognitive processing of different aspects of the case. The following chapters make the case to complement such methods with eye tracking to gain a more complete picture with regard to cognitive processing. How this can be analyzed is discussed in the next section, introducing eye tracking and the theoretical assumptions behind it. 3. Analyzing cognitive processing 3.1. Methods for analyzing cognitive processing The more classical methods for analyzing difficulty, as one aspect of cognitive processing, are questionnaires, self-reports or interviews, e.g., asking learners to rate the case's difficulty after having read a section of it. There are also tests on cognitive load imposed on the working memory, such as the NASA task-load index or dual-task set-ups in which participants perform a secondary task “in parallel to assess the cognitive load devoted to the main task” (Debue & van de Leemput, 2014, p. 3). These may be valid approaches, but one of the issues with such techniques is their intrusive character as seen by participants e.g., forcing participants to perform a second task while already conducting another one. In addition, if questions or tests, such as the NASA task-load index, are performed afterward, one cannot directly link the results to all areas of interest to collect information on the dispersion of attention. Relatedly, we can expect self-reporting biases, as participants must recall their own thought processes (also called “free-recall”), which often “leads to forgetting the content” and produces responses that “are very likely to be made up” (Holmqvist et al., 2015, p. 105). Techniques that would allow researchers to obtain objective data while working with a case and that would do so without interfering with the participant would be advantageous. This is where eye tracking might be of help. The application of eye tracking involves recording eye movements (and other eye behavior like pupillary response) with a camera while a participant is exposed to a stimulus, such as a text, picture or video. The apparatus can be attached to a computer screen or a laptop so that participants can move their heads with some freedom while they go through the materials without disturbance. In contrast to, e.g., self-reports, the data obtained is objective, as the eye-tracking system itself is calibrated as well as adjusted to the participant's eye characteristics before recording, independent of the character of the subject (e.g., facts vs. opinions). The interpretation of the collected data, of course, is still a subjective matter, and it should also be clear that different devices may record and transform raw data with different levels of precision and accuracy (Wass, Forssman, & Leppänen, 2014), but after setting these parameters, data will be concisely and objectively collected. 3.2. Cognitive processes behind eye movements To understand how eye movements help us obtain an indication of underlying processes, we must keep in mind that the eyes have a small circular area in the middle of the retina called the fovea. This area “provides high acuity visual information, and beyond which, in the parafovea and the periphery, vision is of much reduced visual acuity” (Liversedge et al., 2016, p. 2). Humans, therefore, must move this area toward the source of the information they want to grasp. The move to the focus area is called a saccade, and the time an eye stays focused on a certain region is called a fixation. Fixations show when the eyes remain on a position for a certain time after placing the fovea on the center of attention to clearly see what is displayed. During this fixation, it is assumed that humans are able to take in information, while during saccades this is generally not the case (Holmqvist et al., 2015). The assumption that humans only take in what is fixated upon is called the eye-mind hypotheses (Just & Carpenter, 1980). This is the central theoretical basis for most eye-tracking studies, assuming that humans “process the visual information we are currently looking at” (Strobel, Lindner, Saß, & Köller, 2018, p. 140). Eye tracking then presents a way of gaining "insights into the cognitive processes underlying a wide variety of human behaviors" (Ashby, Johnson, Krajbich, & Wedel, 2016, p. 97) The eye-mind hypothesis is not without criticism, as it assumes that someone processes information only when looking at a certain point. Hyönä (2010, p. 173) states that the eye-mind-hypothesis is only true “as long as the available visual environment in front of our eyes is pertinent to the task we would like to study.” Thus, when recording eye movements, special emphasis should be placed on the stimuli and/or the environment to avoid unwanted disturbances or saliency. If these are taken into account, the dominant view, according to Ludwig and Evens (2017, p. 246) is that “the duration of an individual fixation may be influenced by the current cognitive processing.” The question, then, is why one moves the eyes toward a certain region. This is explained with two different cognitive processes: top-down-processes and bottom-up processes. Top-down processes are task- or goal-oriented (Meiβner & Oll, 2017), e.g., if participants must find the right answer to a question as part of a multiple-choice question, they may move the eyes to the possible answers
306
The International Journal of Management Education 17 (2019) 304–315
T. Berger
several times in order to take in the written information and then come up with a solution (Ludwig & Evens, 2017). Bottom-up processes, on the other hand, are driven by the saliency of the options available: e.g., if one of the answers is written in bold capital letters, it may catch the attention even before the question is read. Eye movements are therefore seen as outcomes of such cognitive processes and can be measured, indicating where the eyes are focused, for how long, and how often. This may cognitively reflect constructs like attention or the level of processing. This is displayed in the below model for the analysis of case materials, based on Meiβner and Oll (2017).
Fig. 1. Model for analyzing case materials.
It is assumed here that top-down, goal-oriented processes deliberately guide the eye to certain areas of the case material, e.g., based on the instructions given. At the same time, bottom-up processes may occur, which means that certain characteristics of the stimulus (e.g., size or colour) attract the eye to this specific area. These two processes seem to be interconnected (Meiβner & Oll, 2017), but depending on the stimuli design and the concrete task (e.g., free examination vs. search tasks), one process may be dominant (Scholz, Helversen, & Rieskamp, 2015). The next section discusses possible variables that can be used to analyze eye movements with regard to case reading. 3.3. Variables for analyzing case materials As stated above, fixations are of special importance here, as only during fixations can people take in information. This means that the first set of variables is based on these fixations as detected by the eye tracker. Terminology is not very strict in eye-movement research, which is why what here is called a dwell—the time of “one visit in an AOI from entry to exit” (Holmqvist et al., 2015)—which may also be called a visit. Total dwell time, then, would be defined as all dwells within an AOI (or stimuli) summed up, also called total visit or gaze duration and often used in analyzing reading (Hyönä et al., 2003). The case description itself should be broken into several logical areas of interest (AOI): e.g., an area around the task, background information, text boxes, tables, or the like. This then allows the analysis to distinguish among different aspects of a case description. The following section provides a brief overview on variables and visualizations currently used in eye-tracking research, which might be of help when analyzing case materials. These are later applied as part of an exemplary study.
• Attention can be visualized with the help of heat maps based on fixations. They are a very popular way of visualizing eye
•
•
movements (Meiβner & Oll, 2017). “Typically, regions with many fixations or data samples are highlighted with warm colours (red) and regions where few or no people looked at are marked with a colder (blue) colour” (Holmqvist et al., 2015, p. 231). Often such coding can also be adjusted, so that, e.g., transparency indicates attention: the more transparent a region, the stronger the attention has been (measured by the number of fixations, fixation durations, or the like). These visualizations are easy to grasp, but further analysis like statistical testing is not easily possible. Cognitive processing and cognitive load could be indicated with the help of fixation durations (Rayner, 2009). Longer fixation durations would be indicative of deeper cognitive processing and higher effort (Holmqvist et al., 2015; Zagermann et al., 2016), also indicating higher cognitive load. As an additional measure, we can use fixation-based dwell times, which are the sum of all fixations across a certain area, including re-visits but excluding saccades (as during these, no information will be taken in). There is a lively debate on how to best measure cognitive load, e.g. with pupil diameter changes, which are often used as indicators of cognitive load. In recent years, several studies have also proposed using fixations, although their evidence is somewhat mixed (Debue & van de Leemput, 2014; Zagermann et al., 2016). Efficiency in searching for the necessary information can first be measured by the number of transitions (movements from one AOI to another). The fewer the numbers, the more efficient the student was in reading without re-reading. If the person is not sure and moves the eyes back for re-reading, this would result in additional transitions: hence, the efficiency measure. Second, the share of fixations to dwell time within an AOI can also be used to indicate efficiency. Here, all fixations during a dwell are summed and divided by the dwell time, leaving aside saccades. This assumes that when the share of fixations is closer to 100% of the dwell times, less time was spent on searching for information and more time on taking in information during fixations. As a third metric, the number of fixations can also be used as an indicator of efficiency (Holmqvist et al., 2015; Liu, Lai, & Chuang, 2011): the higher
307
The International Journal of Management Education 17 (2019) 304–315
T. Berger
•
•
the number of fixations in a certain area (e.g., AOI), the less efficient a person was in searching for the relevant information, as they had to fixate additionally (e.g., because it was difficult for the person to derive meaning from the text, which is why it is also referred to as a measure of difficulty). Reading patterns can be analyzed by making use of so-called transitions. By analyzing where a transition (essentially a saccade) starts and ends, one can detect search patterns while going through the materials and then compare these to those of other participants or clustering various types of reading patterns. In addition to the use of data on transitions, one can also use the order of first entries based on the time it took for a first fixation in a certain area (also called time to first fixation) (Holmqvist et al., 2015). The focus of a participant's first fixation, then, is shown. The order of these first entries then indicates the order in which information was taken in (in the beginning). It is also possible to analyze reading patterns based on this order of first entries with measures of concordances or to run sequence analyses or sequence searches based on the order of AOI visited (e.g., the AOI “A,” “B,” “C,” “D,” and “E”), expressed as a string (Holmqvist et al., 2015), e.g., “ACBEDADEA.” For this study, the software Blickshift was used, which allows for an automated search for a specific string in all records parallely. To analyze reading depth, we can add the time spent on an AOI, including fixations and saccades (called dwell time or visit duration) and then dividing it by the AOI size in thousands of pixels (Holmqvist et al., 2015; Holsanova, Rahm, & Holmqvist, 2016). This indicator includes saccades, as browsing here is also seen as an aspect of the depth of reading, which is not without criticism, as it is assumed that reading in the sense of taking in information can occur only during fixations (Holmqvist et al., 2015). An alternative measure, therefore, could be to use fixation-based dwell time per thousand pixels, based on fixations only.
In addition to the above measures of efficiency, the total trial duration (the time elapsed from the start to the end of a trial) could also be of interest, but it is not a clear-cut measure, as longer durations could indicate a deeper cognitive processing, a higher difficulty in reading, or simply that a participant has spent time outside AOI or even outside the stimuli. This is why it is often used only as the denominator for relative measures like the share of fixations on total trial duration or the share of fixations outside the stimuli or target areas in percentages of total trial duration. Having laid the foundations, these variables are then applied below as part of a study of case materials. We, therefore, first introduce the short case before going into more detail on the sample and eye-tracking settings. 4. Study design 4.1. The exemplary case When preparing an eye-tracking study, one must first consider which case to analyze (field of study, length, focus, etc.). For this study, a short case about an oil spill1 was created. The caselet was created to apply expected utility theory (EUT) in real life and afterward to discuss possible limitations with the class. The case of an oil spill was provided as a typical incident with information on probability and potential charges, together with some context like “no one was actually harmed” and “the area […] is now clean,” which is not relevant in a pure sense but may be helpful in understanding what actually happened. The English version of the case is included in the annex. Such a short case allows students to discover by themselves what kind of information they may want to consider when applying EUT and coming up with a number or decision. The case was distributed several times in classes in India and Germany, and group discussions were conducted that led to some minor changes in the design. Students were then asked to state what information they took into account, what their settlement sum would be, and what kind of information they missed. During these classes, typical discussions arose regarding uncertainty about whether the oil spill was an accident or done on purpose. In addition, discussions also emerged as some students stated tactical offerings and discussed the possible process of negotiating a settlement sum, often with additional discussions on the ethical aspects of striking a deal versus reaching a “proper” court decision. It was, therefore, clear that even this small case functioned as a catalyst for a wider discussion, as intended as part of the case-method teaching (Ellet, 2007). It therefore proved ideal as an exemplary case for testing the use of eye tracking for analyzing case materials, as it was short enough not to consume too much time for solving and was not too complex in nature, but still had some aspects that would fuel a discussion. When applying eye tracking in one's own study, such pre-testing with some classes could prove helpful in, e.g., designing AOI or adjusting the case based on this feedback before running the eye-tracking study. In light of the exemplary case, it is assumed that target-oriented processes are dominant, as the case was short and without many salient features, like words in bold (only the task itself was in bold) to attract attention. The instructions for the participants also made it clear that there would be questions afterward so that their behavior should be task-driven (“top-down process”), not like in the case of a free examination that would most probably be stimuli-driven (“bottom-up process”). Of course, this depends on the specific nature of a case. In some circumstances, it might even be the case that some salient features like important facts are incorporated purposely.
1
Both text versions—English and German—were tested with a readability test and had standard and average readability complexity. 308
The International Journal of Management Education 17 (2019) 304–315
T. Berger
4.2. Sample description and procedure As an additional step, the sample for the analysis must be determined. Here, different possibilities exist, as one could compare different groups or focus on just one target group, e.g., based on work experience, study program, cultural background, or the like. For the exemplary study, only management students from India and Germany currently enrolled in postgraduate study programs (e.g., master's, postgraduate diplomas, or similar2) with more than one year of work experience after their first degree were recruited. The two groups were chosen because the author works with such groups in different management programs and felt that the lessons learned from working in such very different environments (and cultures) for recording might provide some interesting insights for future lectures.3 All students were familiar with case studies and EUT, and all had either normal or corrected-to-normal vision. Participants were recruited after classes at different universities and briefed at the beginning of the study as well as asked for consent for the study. Participants then underwent a five-point calibration process measuring their eye characteristics and adjusting the 3D eye model of the eye tracker. An experienced instructor then assessed the quality of calibration, and recording began. Afterward, a short questionnaire on paper was handed out, with some principal questions on difficulty and demographics and a small amount of feedback obtained. Finally, participants were de-briefed and consented to use the data obtained. Participants received no monetary compensation but did receive some sweets. For the eye-tracking portion, the above case was shown on a laptop screen with a screen size of 17″, and the eye movements were recorded with an eye tracker attached below the screen. This allowed the researchers to record eye movements without disturbing the participants. The eye-tracker (Tobii X3-120, with a sampling rate of 120 register per second) uses near-infrared light to create reflections, which are then captured by the device and visualized. For data analysis and visualization, Blickshift Analytics version 1.09, Tobii Studio Pro 3.4.8, and SPSS 21 were used. For quality assurance, all records were deleted if they had recording gaps of longer than 3s4 or if the overall useable gaze data (recorded eye movements) were below 70% for the whole recording (including the instruction pages). For the analysis of the fixation durations, a threshold of 50 ms was chosen so as to discard short fixations, as they are deemed too short for taking in information. This means that the results for fixation durations are skewed. For some variables, an even higher threshold of 200 ms was chosen to account for the fact that participants needed some time before being able to process the information fully. This higher threshold leads to a fewer number of fixations, as only the ones longer than 200 ms are counted. This is in line with the values Rayner (2009) reports as averages for reading, and it is also what Debue and van de Leemput (2014) used for their studies. In total, 70 records from the Indian students and 60 records from the German students could be used for this analysis (two Indian and ten German records had to be excluded from analysis due to quality issues). The mean age of the German sample was 28.7 (SD = 8.24) and 70.0% of participants were male, while the mean age of the Indian sample was 29.8 (SD = 4.91) and 65.3% of participants were male. Of the Indian students, 62.9% held a bachelor's degree in engineering compared to 34% of the German students, while in the German sample 42% held a business degree, compared to only 29% from India (24% of German students and 8.3% of Indian students held degrees in other fields). The following chapter presents the results of the exemplary study as an orientation for other studies. 5. Results The German students spent less time on the exercise as a whole than did the Indian students (70.9s and SD = 23.73 versus 80.7s and SD = 29.10; t = −2.091; p = 0.038), as measured from the first mouse click after displaying the case on screen until the last mouse click that ended the display of the case (total trial duration). To analyze data in more detail, five areas of interest (AOI) were created, allowing for a comparison of different regions of the stimulus. These AOI are visualized in the Annex. 5.1. Attention and reading depth As one of the next steps of analysis, a heat map was created using the Blickshift Analytics software, visualizing the distribution of fixation lengths, where the intensity and density of fixations are coded. As shown in Fig. 2, regions that received more attention are transparent.
2 Postgraduate programs in Germany and India are not fully comparable, which is why I focused on university-based management seminars and master's degree management programs that require a bachelor's degree and lasted for a minimum of one semester. 3 The motivation to participate was strikingly different: While in India, often people queued outside the recording room, while German students were a bit more reserved. 4 This could be because of technical issues or because participants looked up or closed their eyes during the recording.
309
The International Journal of Management Education 17 (2019) 304–315
T. Berger
Fig. 2. Heat maps for both groups (left: Indian students; right: German students).
We can see that both groups focused heavily on the “10,000 €/$” value (labeled AOI Sum) and to a lesser extent on the “20%” value. The German students focused more intensely on the “80%” value, while the Indian students seemed to have focused more intensely on the context information. It is also interesting that the German students spent more fixations on regions outside the five AOI (42.8%) compared with the Indian students (35.3%). This means that German students focused more of their attention outside the deemed-relevant areas. This was further investigated by looking at reading depth as a metric. The stark differences are displayed below in Table 1 for all AOI. Table 1
Reading depth in ms per thousand pixels of AOI size
AOI Context AOI Task AOI 20% AOI 80% AOI Sum Sum of all AOI
GER
IND
t
p
Hedges g
37.31 20.35 212.82 178.57 311.26 760.64
89.82 64.82 216.01 79.68 447.25 897.83
−12.51 - 8.55 - .070 3.76 - 2.47 - 1.35
.000 < .001 .944 < .001 .015 .184
2.04 1.37 0.01 - 0.69 0.35 0.24
Cognitive processing with respect to reading depth is highest in both samples for the AOI Sum, which is understandable, as this AOI is central for answering. The difference between Indian and German students here is significant with a medium effect size. There are even larger, significant differences for the Context and Task AOIs. In general, we can see that the reading depth differs significantly between German and Indian students for most of the AOI, except for the 20% AOI. Effect sizes are large for the AOI Context and Task and small to medium for the AOI Sum and 80% (opposite direction). This means that, generally speaking, Indian students had a higher reading depth than German students, with the exception of the AOI 80%, which was read more deeply by German students. 5.2. Cognitive processing To measure cognitive processing, the total duration of the fixations (i.e., the sum of all fixations above a threshold of 50 ms) was taken. The average duration of all fixations was slightly different (non-significant) for both groups but varied considerably across the different AOIs, as shown in Table 2. Table 2
Average fixation durations in ms per AOI.
GER IND
AOI Context
AOI Sum
AOI 20%
AOI 80%
AOI Task
Mean
139 151
190 206
166 163
179 157
131 123
161 160
In our example, we can see that the AOI Sum was processed more deeply (190 and 206 ms for the Indian and German students, respectively) than the other AOI in both groups. This seems logical, as the sum stated was central to answering the question. As an additional indicator of cognitive processing, the fixation-based dwell times across all AOIs were taken, indicating cognitive load. German students had a fixation-based dwell time of 32.83s (SD = 11.59), while Indian students fixated for 48.97s (SD = 16.12):
310
The International Journal of Management Education 17 (2019) 304–315
T. Berger
a significant difference with a large effect size (t = −6.483, p < 0.001, g = 1.133), not only in absolute terms but also relative to the trial duration. This indicates that Indian students had a much higher cognitive load when reading the case. 5.3. Efficiency in reading To further assess these differences, two indicators of efficiency are recorded. First, the number of fixations is calculated, as displayed in the Table 3 below. Table 3
Average number of fixations on AOI absolute and relative to size in thousand pixels (t px). GER
AOI 20% AOI 80% AOI Context AOI Sum AOI Task All AOI
IND
count
count per t px
count
count per t px
1.93 1.77 32.72 5.43 17.01 58.85
0.387 0.354 0.054 0.597 0.028 0.048
3.08 1.35 48.13 8.49 27.01 88.04
0.617 0.270 0.141 0.933 0.103 0.142
T (t px)
P (t px)
Hedge's g (t px)
−2.31 1.27 −9.05 −3.10 −8.28 - 11.42
.023 .216 < .001 .002 < .001 < .001
0.40 −0.22 1.58 0.54 1.34 1.86
In total, Indian management students had a much higher number of fixations per thousand pixels than did the German students. To verify these findings further, we made use of the number of transitions, also indicating efficiency. In total, the German participants had 10.5 transitions per recording, while the Indian students had 13.8 transitions, meaning that the German management students were in general more efficient at solving the case as measured by the number of transitions. Together with the fixation-based dwell times and the number of fixations, this may indicate that German students processed the information with less difficulty. 5.4. Analysis of patterns To determine whether the patterns for going through the case materials are different, we can first look at the transitions for the whole group and then at the scan paths of all participants to compare them and find common sequences. For the latter, the Blickshift Analytics software was used to allow the software to scan for more complex patterns.5 The software searched for patterns that would indicate “re-assuring” or “double-checking” between the actual task and the central information needed to fulfill the task (i.e., transitions between the AOI Sum and AOI Task and back to AOI Sum). As Scholz et al. (2015, p. 243) have identified “an evergrowing number of papers showing that when retrieving information from memory, people gaze back at spatial locations that have been associated with the to-be-retrieved information during encoding,” it was assumed that such a pattern indicates that participants gaze back to re-assure themselves of what they have memorized. The software found the above pattern once in 56% of the Indian records and in 45% of the German records. This is further verified by looking at the backwards transitions from the AOI Task to the AOI Sum, which were found at least twice in 50% of the Indian records and in only 38% of the German records. German students, therefore, do not seem to double-check as often as the Indian students, which is also in line with their smaller number of transitions. To further analyze patterns, the transitions based on first fixations in an AOI were identified (Holmqvist et al., 2015). AOI are ordered according to the time it took the participants to fixate their eyes on each AOI, not including fixations outside the AOIs. The whole five-step sequence through all AOIs (meaning that everything was read at least once) was visible in only 50% of the Indian recordings and in only 56.7% of the German recordings. Following an ideal reading pattern, one should transition from the first AOI at the top down to the last, just as they would read normally. This theoretically ideal pattern was only found in 11.7% (seven participants) of the German records and in 5.6% (four participants) of the Indian records. While 79.2% of the Indian students started reading from the top, virtually all German students (96.7%) started reading from the top. 16.7% of Indian students read the task first (only one German student did that) and only one third then moved on to read the rest of the case description from the top moving down. Indian students, therefore, seemed to be more task-oriented than German students.
5 Using the Blickshift software, of course, is not necessary for similar projects. The software's main advantage over manual similarity searches is its use of a clustering algorithm using Levenshtein distance. See Raschke et al. (2014) for more information.
311
The International Journal of Management Education 17 (2019) 304–315
T. Berger
The order of reading through the AOIs for both groups is displayed in Table 4, based on average order ranks for the five AOIs. Table 4
Order of first entries for the five AOIs.
IND GER
AOI Context
AOI Sum
AOI 20%
AOI 80%
AOI Task
Kendall's W
χ2
1.31 1.06
2.91 2.65
3.93 3.29
4.00 4.12
2.85 3.88
0.477 0.601
64.86* 81.69*
*p < 0.001.
The pattern of the German sample is closer to the “ideal” reading pattern than is the pattern of the Indian sample (only the order of the AOIs “80%” and “Task” were different). If we compare the two mean ranking values of German and Indian students further, Kendall's Coefficient of Concordance W stands at 0.850 (χ2 = 6.8, p = 0.147), showing a high commonality between the two samples but one that is not statistically significant. 6. Discussion 6.1. Some guidelines for using eye tracking This paper showed how eye tracking can be applied for case-study analysis. A short case on an oil spill incident was shown on a laptop screen, and participants had to answer a question to the case presented based on the EUT. This case was designed to facilitate discussion on possible limitations of the EUT. Based on the experiences during this study, some general points for the use of eye tracking in analyzing cases are discussed below that may help develop the potential of this method. For this study, we made use of an eye tracker attached to a laptop, but in other circumstances it might be more helpful to use head-mounted eye trackers. They have the advantage of offering more freedom in recording (e.g., to include interactions with other participants or material in addition to what is on the screen). On the other hand, such eye trackers are not as precise as mounted ones, and the analysis, therefore, would be more limited (Holmqvist et al., 2015). The exemplary study had two culturally different participant groups for analysis, and it sometimes proved hard to create a comparable environment in both countries, as variations in settings were vast: e.g., sunlight exposure, brightness, room size, and noise level. In particular, light intensity should always be measured and set to the levels as set out in the specifications of the eye tracker, as this will influence the recordings. Changes in light intensity over the course of the recording should also be avoided. A solution could be to conduct a study in the same room every time (preferably a special laboratory with no windows or shutters), but that would also limit the scope of possible studies. As shown in the above results section, the definition of variables is critical for every study. In the case of eye tracking, even in the standard books—and more so in the different studies—there are various recommendations for the definition of variables, so they are often hard to compare across studies. Careful design of the variables to capture the underlying cognitive processing, therefore, is crucial and should be supported by a methodological framework like the Cognitive Load Theory mentioned above. The underlying theory or hypothesis will then influence the choice of variables and underlying metrics. This study, grounded in the eye-mind hypothesis, mainly made use of fixations, e.g., to assess efficiency and reading depth. It therefore left aside pupillometric data that also could prove helpful when, e.g., assessing cognitive load. Data quality is also of major importance. In this study, we used the indicators on data quality provided by the manufacturer Tobii and supplemented this with own analysis of the data gaps, which were assessed individually, record-by-record. This is time-consuming, and there are also other variables available to assess data quality (Holmqvist, Nyström, & Mulvey, 2012). Data output, and consequently data quality, is also influenced by the definition of the metrics, like fixations or saccades. If one defines the minimum length of a fixation as 20 ms or 100 ms, this will have a major effect on the output, as the device will transfer the raw data based on these settings. Here, results from meta-studies, like those of Rayner (2009) can be used as a guideline. The final definition will still depend on the kind of stimuli presented—text vs. figures—and what the researchers have in mind. As there are almost no studies on case materials, it will not be easy to find existing studies as references. The solution in this study was to make use of typical definitions from reading studies, as the case analyzed was purely text-based. As most studies rely on the use of AOIs, creating them is also critical. For designing AOIs, several methods are available. This study defined AOIs based on semantic groupings like text on the task, the sums involved, or the like. It would also be possible to create a grid and then analyze data based on this (Holmqvist et al., 2015) or on other criteria. The minimum size of the AOI then is determined by the specifications of the eye tracker and a minimum margin of error should be taken into account (which will be larger for mobile eye-trackers than for mounted ones). For analyzing casework more broadly, data on eye tracking should be combined with additional methods like focus group meetings, think-aloud techniques, or in-depth interviews. This will not only enable the verification of data from eye tracking, but could also provide valuable insights into the social interaction taking place or the understanding of contents. Despite the advantages eye tracking may offer, it will by itself “not tell the researcher anything about the success or failure of comprehending the relevant piece of information” (Hyönä, 2010, p. 173). The exemplary study above was done using a questionnaire for eye-tracking participants and group discussions in the pre-test before applying eye tracking. One aspect, then, was the level of difficulty. To analyze difficulty 312
The International Journal of Management Education 17 (2019) 304–315
T. Berger
later on, fixation durations and fixation counts were taken, and the findings may indicate a higher difficulty in extracting information for the Indian students. With additional data, such characteristics can be further verified, as was the case in the exemplary study: while German students rated difficulty at 1.33 in the pre-test, Indian students rated difficulty at 2.59 on a scale from 1 (easy to understand) to five (difficult to understand), which is in line with the eye-tracking data. 6.2. Future research possibilities One possible strand of research could use the existing studies on efficacy to combine eye tracking with questionnaires, interviews, tests, or essays, e.g., based on the findings of Druckman and Ebner (2018) or Farashahi and Tajeddin (2018). It would also be possible to test this with respect to different kinds of learning outcomes. Future studies could also apply controlled experiments where different kinds of cases or guidance strategies are tested and where the effect on different kinds of cognitive load is measured, just as Kirschner, Sweller, and Clark (2006) have proposed. It could, therefore, go as far as to design specific multimedia courses that are flexible and adjustable to the needs of different groups based on culture or other criteria. When working with such cases, the cognitive load could be tracked and the contents adjusted to changes in this cognitive load. With the arrival of cheap eye trackers and systems with built-in eye trackers, this is not too far away (Liu et al., 2011). In addition, one could also study whether the content of the case influences the kind of processing by using cases from different disciplines, such as marketing, accounting, or HR, together with graphical displays and different types of cases. These studies could also be conducted with often-used cases of variable lengths from large providers. Here, one question could be what the typical patterns of going through the materials are, as was done here with the help of the Blickshift software.6 Other avenues related to this may be to compare different cultural groups, like in the exemplary study; even this study, which uses a short case, has shown that there may be large differences regarding reading depth, difficulty, attention, and cognitive processing in general that should be kept in mind when designing or teaching with a case. Again, this could be combined with additional methods to capture the social interactions and other possible variables influencing the results. The differences in fixations witnessed in the exemplary could, e.g., be based on the fact that the language used in the case provided was not identical for both groups, although as relates to reading there seems to be no “strong evidence in favor of the idea that culture systematically modulates eye movements” (Rayner, Li, Williams, Cave, & Well, 2007). Such areas need additional research, as from this study it cannot be determined whether the significant differences in the modulation of eye movements are caused by cultural, educational, or other differences among participants. Funding This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Declaration of interest: none. Acknowledgements I wish to thank Walburga Sarcher of the University of Augsburg, Pankaj Trivedi from KJ Somaiya Institute, Mumbai, and Jagdish and Ramesh Shah for helping me organizing eye-tracking sessions at several organizations, Ignace Hooge who gave mee feedback on an earlier version as well as all others who helped me with this research, including the participants in Germany and India. I also want to thank the two anonymous reviewers who have given me very detailed and helpful feedback.
6
This could also be done with software like MatLab or the like. Many researchers use MatLab for eye-tracking analysis. 313
The International Journal of Management Education 17 (2019) 304–315
T. Berger
Annex
Case description and areas of interest (below), English version.
References Aithal, S. (2016). Innovations in student centric learning: A study of top business schools in India. Munich: MPRA Paper. Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011). Does discovery-based instruction enhance learning? Journal of Educational Psychology, 103(1), 1–18. Altmann, G. T. M., & Kamide, Y. (2007). The real-time mediation of visual attention by language and world knowledge. Linking anticipatory (and other) eye movements to linguistic processing. Journal of Memory and Language, 57(4), 502–518. Argyris, C. (1980). Some limitations of the case method. Experiences in a management development program. Academy of Management Review, 5(2), 291–298. Ashby, N. J. S., Johnson, J. G., Krajbich, I., & Wedel, M. (2016). Applications and innovations of eye-movement research in judgment and decision making. Journal of Behavioral Decision Making, 29 2–3, 96–102.
314
The International Journal of Management Education 17 (2019) 304–315
T. Berger
Bayona, J. A., & Castañeda, D. I. (2017). Influence of personality and motivation on case method teaching. International Journal of Management in Education, 15(3), 409–428. Blankley, A. I., Kerr, D., & Wiggins, C. E. (2017). The state of accounting education in business schools: An examination and analysis of active learning techniques. Advances in Accounting Education: Teaching and Curriculum Innovations, 21, 101–124. Burgoyne, J., & Mumford, A. (2001). Learning from the case method: A report to the European case clearing House, RP0301Lancaster: European Case Clearing House. Chu, L., & Libby, T. (2010). Writing mini-cases. An Active Learning Assignment”, Issues in Accounting Education, 25(2), 245–265. Debue, N., & van de Leemput, C. (2014). What does germane load mean? An empirical contribution to the cognitive load theory. Frontiers in Psychology, 5, 1099. Druckman, D., & Ebner, N. (2018). Discovery learning in management education. Design and case analysis. Journal of Management Education, 42(3), 347–374. Easton, G., & Ormerod, T. C. (2001). Expert/Novices differences in case analysis,Lancaster (UK): Lancaster University. Ellet, W. (2007). The case study handbook: How to read, discuss, and write persuasively about cases. Boston, Mass: Harvard Business School Press. Farashahi, M., & Tajeddin, M. (2018). Effectiveness of teaching methods in business education. A comparison study on the learning outcomes of lectures, case studies and simulations. International Journal of Management in Education, 16(1), 131–142. Hoag, K., Lillie, J., & Hoppe, R. (2005). Piloting case-based instruction. Didactic clinical immunology course. Clincial Laboratory Science, 18(4), 213–220. Holmqvist, K., Nystrom, M., Andersson, R., Dewhurst, R., Jarodzka, H., & van de Weijer, J. (2015). Eye tracking: A comprehensive guide to methods and measures. Oxford: Oxford University Press First published in paperback 2015. Holmqvist, K., Nyström, M., & Mulvey, F. (2012). Eye tracker data quality. In C. H. Morimoto, H. Istance, J. B. Mulligan, P. Qvarfordt, & S. N. Spencer (Eds.). Proceedings of the symposium on eye tracking research and applications - ETRA '12, Santa Barbara, California, (pp. 45). New York, New York: ACM Press. Holsanova, J., Rahm, H., & Holmqvist, K. (2016). Entry points and reading paths on newspaper spreads. Comparing a semiotic analysis with eye-tracking measurements. Visual Communication, 5(1), 65–93. Hughes, S. B. (2017). Student-authored IFRS teaching cases based on European Securities and Markets Authority reports. Experiences from case writing and subsequent classroom use. Journal of Accounting Education, 41(12), 58–74. Hyönä, J. (2010). The use of eye movements in the study of multimedia learning. Learning and Instruction, 20(2), 172–176. Hyönä, J., Lorch, R. F., & Rinck, M. (2003). Eye movement measures to study global text processing. In J. Hyönä, R. Radach, & H. Deubel (Eds.). The mind's eye: Cognitive and applied aspecets of eye movement research (pp. 313–334). Oxford: Elsevier. Jennings, D. (1996). Strategic management and the case method. The Journal of Management Development, 15(9), 4–12. Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87(4), 329–354. Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work. An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86. Lai, M.-L., Tsai, M.-J., Yang, F.-Y., Hsu, C.-Y., Liu, T.-C., Lee, S. W.-Y., et al. (2013). A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educational Research Review, 10(12), 90–115. Lindner, M. A., Eitel, A., Thoma, G.-B., Dalehefte, I. M., Ihme, J. M., & Köller, O. (2014). Tracking the decision-making process in multiple-choice assessment. Evidence from eye movements. Applied Cognitive Psychology, 28(5), 738–752. Liu, H.-C., Lai, M.-L., & Chuang, H.-H. (2011). “Using eye-tracking technology to investigate the redundant effect of multimedia web pages on viewers' cognitive processes”. Computers in Human Behavior, 27(6), 2410–2417. Liversedge, S. P., Drieghe, D., Li, X., Yan, G., Bai, X., & Hyönä, J. (2016). Universality in eye movements and reading. A trilingual investigation. Cognition, 147, 1–20. Ludwig, C. J. H., & Evens, D. R. (2017). Information foraging for perceptual decisions. Journal of Experimental Psychology. Human Perception and Performance, 43(2), 245–264. Lund Dean, K., & Fornaciari, C. J. (2016). How to create and use experiential case-based exercises in a management classroom. Journal of Management Education, 26(5), 586–603. Lundberg, C. C., Rainsford, P., Shay, J. P., & Young, C. A. (2001). Case writing reconsidered. Journal of Management Education, 25(4), 450–463. Meißner, M., & Oll, J. (2017). The promise of eye-tracking methodology in organizational research. Organizational Research Methods, 8(10), 1–28. Mintzberg, H. (2005). Managers not MBAs: A hard look at the soft practice of managing and management development (paperback ed.). San Francisco, Calif: BerrettKoehler. Nkhoma, M., Sriratanaviriyakul, N., & Le Quang, H. (2017). “Using case method to enrich students' learning outcomes”. Active Learning in Higher Education, 18(1), 37–50. Raschke, M., Herr, D., Blascheck, T., Ertl, T., Burch, M., Willmann, S., et al. (2014). A visual approach for scan path comparison. In Association for Computing Machinery (Ed.). Proceedings of the symposium on eye tracking research and applications (pp. 135–142). New York, New York: ETRA ’14. Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology, 62(8), 1457–1506. Rayner, K., Li, X., Williams, C. C., Cave, K. R., & Well, A. D. (2007). Eye movements during information processing tasks. Individual differences and cultural effects. Vision Research, 47(21), 2714–2726. Scholz, A., Helversen, B. von, & Rieskamp, J. (2015). Eye movements reveal memory processes during similarity- and rule-based decision making. Cognition, 136, 228–246. Selvam, M., Babu, M., & Raja, M. (2006). Caslets teaching in business education. SMART Journal of Business Management Studies, 2(2), 70–72. Shrivakumar, K. (2012). The case study method in training and management education. The IUP Journal of Soft Skills, 6(2), 55–64. Strobel, B., Lindner, M. A., Saß, S., & Köller, O. (2018). Task-irrelevant data impair processing of graph reading tasks. An eye tracking study. Learning and Instruction, 55(6), 139–147. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257–285. Thistlethwaite, J. E., Davies, D., Ekeocha, S., Kidd, J. M., MacDougall, C., Matthews, P., et al. (2012). The effectiveness of case-based learning in health professional education. A BEME systematic review. BEME Guide No. 23. Medical Teacher, 34(6), 421–444. Tsai, M.-J., Hou, H.-T., Lai, M.-L., Liu, W.-Y., & Yang, F.-Y. (2012). Visual attention for solving multiple-choice science problem. An eye-tracking analysis. Computers & Education, 58(1), 375–385. Wass, S. V., Forssman, L., & Leppänen, J. (2014). Robustness and precision. How data quality may influence key dependent variables in infant eye-tracker analyses. Infancy, 19(5), 427–460. Zagermann, J., Pfeil, U., & Reiterer, H. (2016). Measuring cognitive load using eye tracking technology in visual computing. In M. Sedlmair, P. Isenberg, T. Isenberg, N. Mahyar, & H. Lam (Eds.). Proceedings of the beyond time and errors on Novel evaluation methods for visualization - BELIV '16, Baltimore, MD, USA (pp. 78–85). New York, New York: ACM Press.
315
Contents lists available at ScienceDirect
The International Journal of Management Education journal homepage: www.elsevier.com/locate/ijme
Using eye-tracking to for analyzing case study materials Thomas Berger
T
DHBW Stuttgart, Kronenstr. 40, 70174, Stuttgart, Germany
ARTICLE INFO
ABSTRACT
Keywords: Case method Eye tracking Cognitive load theory Management education
Case studies are an important part of management education, but especially when working with diverse classes, differences in understanding the case materials or in cognitive processing are often unclear. Existing studies use interviews, questionnaires or tests, but cannot gain deeper insights into information processing. Information-processing data would be valuable in designing case materials, e.g., to be able to adjust the cognitive load to different groups of students by assessing difficulty or reading depth. Eye tracking can provide information about where participants focused for how long, what level of cognitive processing took place, and in which areas of the case materials provided. The objective of this paper is to show how eye tracking can be used to gain additional insights into how participants work with the cases. This is illustrated with the help of a study with Indian and German management students and a small case on Expected Utility Theory. Different variables for analyzing case materials are introduced, e.g., fixation duration, dwell time, or reading depth. It is argued that eye tracking can be used more often to analyze case studies to ensure that the cognitive processing matches the intended level and inform instructors about what does and does not attract attention.
1. Introduction Management education around the world uses case studies as one of its prime forms of introduction in addition to classical lectures. This is especially true for MBA programs but is also used for other degrees (Aithal, 2016; Blankley, Kerr, & Wiggins, 2017; Jennings, 1996). The case method is seen as a more active tool of learning than lectures in a controlled setting as well as being more realistic than lectures (Alfieri, Brooks, Aldrich, & Tenenbaum, 2011; Farashahi & Tajeddin, 2018). Cases may be as long as 20 written pages, sometimes with ample additional information like separate tables, graphs, or online databases, but smaller cases, sometimes called caselets, can be as short as one page (Selvam, Babu, & Raja, 2006). Besides their flexibility in length, cases are also very flexible in content and can easily be adjusted to different fields, specific goals of learning content, or different objectives. Lundberg, Rainsford, Shay, and Young (2001, p. 461) differentiate among three objectives in choosing the focus a case: 1) “Acquiring, Differentiating and Using Ideas”; 2) “Issue Identification and Differentiation”; and 3) “Action Formulation and Implementation.” There are, of course, other categorizations, as the one by Jennings (1996), who identifies 13 learning objectives, and Shrivakumar (2012) who identifies six types of cases. Independent from the concrete objective and the type of categorization, the case description (if texts are used) is central to the case method, forming the basis for the casework. When reading through the materials, students must process the given information to create meaning, imposing “heavy loads on working memory detrimental to learning” (Alfieri et al., 2011, p. 3). Working memory capacities must be used efficiently during case-based learning in order to avoid so-called “overload” (Sweller, 1988) that would hinder learning, according to the Cognitive Load Theory (CLT) proposed by Sweller (1988). Instructors also should make sure that the design of the case does not draw disproportionate
E-mail address: [email protected]. https://doi.org/10.1016/j.ijme.2019.05.002 Received 25 November 2018; Received in revised form 29 March 2019; Accepted 12 May 2019 Available online 28 May 2019 1472-8117/ © 2019 Elsevier Ltd. All rights reserved.
The International Journal of Management Education 17 (2019) 304–315
T. Berger
attention to certain aspects, words, or visual displays. Sometimes instructors may want to display more information than is needed so that students must filter the relevant aspects, while at other times, the case is constructed so that no distractors stand out (as is often true when working with caselets). It is therefore the instructor's responsibility not only to focus on the content, style, and flow of a case when preparing it, but also to emphasize the students' possible cognitive processing that takes place when working with a case. Having some indicative results on this would be helpful in designing a case. Cognitive processing—often focusing on cognitive load—is currently assessed primarily via self-reports of difficulty or test scales, such as the NASA task-load index. This is mostly done after the task is fulfilled, which can bias the results (Debue & van de Leemput, 2014). For instructors, it would be advantageous to gain some insights into students’ level of cognitive processing during rather than after the casework. That way, instructors could link that processing to certain aspects of the case description and adjust the materials according to the desired level of cognitive processing. Such an analysis can use eye trackers that record eye movements during a variety of tasks. This study proposes using eye tracking for the analysis of cognitive processing when working with cases, based on the so called “eye-mind hypothesis” (Just & Carpenter, 1980) assuming that eye-tracking provides information “about the user's cognitive activities” (Zagermann, Pfeil, & Reiterer, 2016, p. 79), e.g., indicating attention or cognitive load as set out by the CLT. Currently, eye tracking is applied in a variety of fields related to education, such as exploring learning in general (Lai et al., 2013), testing (Lindner et al., 2014; Tsai, Hou, Lai, Liu, & Yang, 2012), and global text processing (Altmann & Kamide, 2007; Hyönä, Lorch, & Rinck, 2003), but not yet for analyzing case materials. The key contribution of this article, therefore, is to show how this technology can be applied to evaluate case materials with respect to cognitive processing. Eye tracking could be applied to enrich the current methodologies used to analyze the effectiveness of cases. This study, therefore, might be of use for future researchers by showing what kind of metrics can be used to analyze data on case analysis, what to keep in mind when recording data, and how such data could be statistically analyzed and the results presented. As a result of such studies, case materials for students (or teaching notes for instructors) could be adjusted to hint at possible characteristics, e.g., where most attention might be paid to and how to assist in working with the case based on cognitive processing. In this paper, an overview of existing studies on the case method is first provided before introducing the eye-tracking method, variables and study design. In addition to general visualizations of attention, different variables for analyzing cognitive processing in more detail are explained, based on measures of fixations, i.e., times when the eyes are focusing. Efficiency in reading, difficulty in understanding, and how to measure the level of cognitive load are discussed. In addition, reading depth is introduced along with how reading patterns could be analyzed, making use of transitions and areas of interest. Finally, it is shown how to conduct a search for patterns while reviewing the materials to learn the order in which participants go through the materials. A study with German and Indian management students where these variables are applied is then included as an example of such an analysis. In the last section, the use of eye tracking for studies is discussed, along with an outlook for potential fields of research. 2. Research on case method teaching As a pedagogical tool, the case method is itself an object of empirical research. Research focusing on the effectiveness of the case study method hints at its higher effectiveness in teaching in general compared to lectures but is not as clear about its value compared to that of simulations. Druckman and Ebner (2018) saw a higher effectiveness of case-based teaching by comparing the design of cases where “the learner is asked to actively create an artifact” (Druckman & Ebner, 2018, p. 353) to case analysis where learners work with an existing case. Both approaches of guided discovery learning were used to teach cognitive-bias concepts. They found that case analysis compared to the design of cases was more effective in the short term, but less ineffective so in the long term, but that both were always more effective than lectures. Farashahi and Tajeddin (2018) found that simulations and case studies were more effective than lectures regarding problem-solving skills (but not so for interpersonal skills or self-awareness), which is partially contrary to the findings of Jennings (1996), whose 1996 study —which was conducted by asking instructors—found that instructors mostly saw communication and interpersonal skills as the prime advantage of case studies. Other findings suggest that students were mostly more satisfied when working with cases, as they are more actively involved than in lectures (Bayona & Castañeda, 2017; Chu & Libby, 2010; Hoag, Lillie, & Hoppe, 2005; Hughes, 2017; Thistlethwaite et al., 2012). The above findings show “evidence for the technique's effectiveness,” as Easton and Ormerod (2001, p. 2) state in their meta-study. Case-based teaching has also been criticized. One strand of criticism focuses on the over-simplification of reality in the case description (Jennings, 1996; Mintzberg, 2005), together with the focus on (rational) decision-making, which means that students read some pages of materials and must then make a decision without having a proper understanding of the situation (Argyris, 1980; Lund Dean & Fornaciari, 2016; Lundberg et al., 2001). In addition, some also criticize the over-reliance on this method in business schools, leaving aside other important methods of instruction (Mintzberg, 2005). While these may be valid points, in this paper we focus on the cognitive processing during such case analysis and leave aside the discussion on how many, when, and what kind of cases should be used. The use of eye tracking to inform us about the cognitive processing of case materials is independent of most of these points of criticism of the case method. On the contrary, eye tracking may even help improve case materials. When we look at existing research on case teaching, we can see that those studies typically focus on effectiveness or satisfaction and involve tests or questionnaires—distributed to the students afterward—or interviews conducted to reflect on the students'
305
The International Journal of Management Education 17 (2019) 304–315
T. Berger
experiences with the case (Burgoyne & Mumford, 2001; Chu & Libby, 2010; Hughes, 2017; Jennings, 1996; Nkhoma, Sriratanaviriyakul, & Le Quang, 2017). A few studies, like the ones by Bayona and Castañeda (2017) or Easton and Ormerod (2001), also include observations recorded on video to gain insights into students' or teachers' interactions. The study by Bayona and Castañeda (2017) combined this method with self-testing by the instructors to analyze the role of personality when teaching with cases. These methods are of use for the above-stated objectives like student or instructor satisfaction, social interaction or the instructor's personality, but not if one wants to know how attention is distributed over the given materials or the kind of cognitive processing of different aspects of the case. The following chapters make the case to complement such methods with eye tracking to gain a more complete picture with regard to cognitive processing. How this can be analyzed is discussed in the next section, introducing eye tracking and the theoretical assumptions behind it. 3. Analyzing cognitive processing 3.1. Methods for analyzing cognitive processing The more classical methods for analyzing difficulty, as one aspect of cognitive processing, are questionnaires, self-reports or interviews, e.g., asking learners to rate the case's difficulty after having read a section of it. There are also tests on cognitive load imposed on the working memory, such as the NASA task-load index or dual-task set-ups in which participants perform a secondary task “in parallel to assess the cognitive load devoted to the main task” (Debue & van de Leemput, 2014, p. 3). These may be valid approaches, but one of the issues with such techniques is their intrusive character as seen by participants e.g., forcing participants to perform a second task while already conducting another one. In addition, if questions or tests, such as the NASA task-load index, are performed afterward, one cannot directly link the results to all areas of interest to collect information on the dispersion of attention. Relatedly, we can expect self-reporting biases, as participants must recall their own thought processes (also called “free-recall”), which often “leads to forgetting the content” and produces responses that “are very likely to be made up” (Holmqvist et al., 2015, p. 105). Techniques that would allow researchers to obtain objective data while working with a case and that would do so without interfering with the participant would be advantageous. This is where eye tracking might be of help. The application of eye tracking involves recording eye movements (and other eye behavior like pupillary response) with a camera while a participant is exposed to a stimulus, such as a text, picture or video. The apparatus can be attached to a computer screen or a laptop so that participants can move their heads with some freedom while they go through the materials without disturbance. In contrast to, e.g., self-reports, the data obtained is objective, as the eye-tracking system itself is calibrated as well as adjusted to the participant's eye characteristics before recording, independent of the character of the subject (e.g., facts vs. opinions). The interpretation of the collected data, of course, is still a subjective matter, and it should also be clear that different devices may record and transform raw data with different levels of precision and accuracy (Wass, Forssman, & Leppänen, 2014), but after setting these parameters, data will be concisely and objectively collected. 3.2. Cognitive processes behind eye movements To understand how eye movements help us obtain an indication of underlying processes, we must keep in mind that the eyes have a small circular area in the middle of the retina called the fovea. This area “provides high acuity visual information, and beyond which, in the parafovea and the periphery, vision is of much reduced visual acuity” (Liversedge et al., 2016, p. 2). Humans, therefore, must move this area toward the source of the information they want to grasp. The move to the focus area is called a saccade, and the time an eye stays focused on a certain region is called a fixation. Fixations show when the eyes remain on a position for a certain time after placing the fovea on the center of attention to clearly see what is displayed. During this fixation, it is assumed that humans are able to take in information, while during saccades this is generally not the case (Holmqvist et al., 2015). The assumption that humans only take in what is fixated upon is called the eye-mind hypotheses (Just & Carpenter, 1980). This is the central theoretical basis for most eye-tracking studies, assuming that humans “process the visual information we are currently looking at” (Strobel, Lindner, Saß, & Köller, 2018, p. 140). Eye tracking then presents a way of gaining "insights into the cognitive processes underlying a wide variety of human behaviors" (Ashby, Johnson, Krajbich, & Wedel, 2016, p. 97) The eye-mind hypothesis is not without criticism, as it assumes that someone processes information only when looking at a certain point. Hyönä (2010, p. 173) states that the eye-mind-hypothesis is only true “as long as the available visual environment in front of our eyes is pertinent to the task we would like to study.” Thus, when recording eye movements, special emphasis should be placed on the stimuli and/or the environment to avoid unwanted disturbances or saliency. If these are taken into account, the dominant view, according to Ludwig and Evens (2017, p. 246) is that “the duration of an individual fixation may be influenced by the current cognitive processing.” The question, then, is why one moves the eyes toward a certain region. This is explained with two different cognitive processes: top-down-processes and bottom-up processes. Top-down processes are task- or goal-oriented (Meiβner & Oll, 2017), e.g., if participants must find the right answer to a question as part of a multiple-choice question, they may move the eyes to the possible answers
306
The International Journal of Management Education 17 (2019) 304–315
T. Berger
several times in order to take in the written information and then come up with a solution (Ludwig & Evens, 2017). Bottom-up processes, on the other hand, are driven by the saliency of the options available: e.g., if one of the answers is written in bold capital letters, it may catch the attention even before the question is read. Eye movements are therefore seen as outcomes of such cognitive processes and can be measured, indicating where the eyes are focused, for how long, and how often. This may cognitively reflect constructs like attention or the level of processing. This is displayed in the below model for the analysis of case materials, based on Meiβner and Oll (2017).
Fig. 1. Model for analyzing case materials.
It is assumed here that top-down, goal-oriented processes deliberately guide the eye to certain areas of the case material, e.g., based on the instructions given. At the same time, bottom-up processes may occur, which means that certain characteristics of the stimulus (e.g., size or colour) attract the eye to this specific area. These two processes seem to be interconnected (Meiβner & Oll, 2017), but depending on the stimuli design and the concrete task (e.g., free examination vs. search tasks), one process may be dominant (Scholz, Helversen, & Rieskamp, 2015). The next section discusses possible variables that can be used to analyze eye movements with regard to case reading. 3.3. Variables for analyzing case materials As stated above, fixations are of special importance here, as only during fixations can people take in information. This means that the first set of variables is based on these fixations as detected by the eye tracker. Terminology is not very strict in eye-movement research, which is why what here is called a dwell—the time of “one visit in an AOI from entry to exit” (Holmqvist et al., 2015)—which may also be called a visit. Total dwell time, then, would be defined as all dwells within an AOI (or stimuli) summed up, also called total visit or gaze duration and often used in analyzing reading (Hyönä et al., 2003). The case description itself should be broken into several logical areas of interest (AOI): e.g., an area around the task, background information, text boxes, tables, or the like. This then allows the analysis to distinguish among different aspects of a case description. The following section provides a brief overview on variables and visualizations currently used in eye-tracking research, which might be of help when analyzing case materials. These are later applied as part of an exemplary study.
• Attention can be visualized with the help of heat maps based on fixations. They are a very popular way of visualizing eye
•
•
movements (Meiβner & Oll, 2017). “Typically, regions with many fixations or data samples are highlighted with warm colours (red) and regions where few or no people looked at are marked with a colder (blue) colour” (Holmqvist et al., 2015, p. 231). Often such coding can also be adjusted, so that, e.g., transparency indicates attention: the more transparent a region, the stronger the attention has been (measured by the number of fixations, fixation durations, or the like). These visualizations are easy to grasp, but further analysis like statistical testing is not easily possible. Cognitive processing and cognitive load could be indicated with the help of fixation durations (Rayner, 2009). Longer fixation durations would be indicative of deeper cognitive processing and higher effort (Holmqvist et al., 2015; Zagermann et al., 2016), also indicating higher cognitive load. As an additional measure, we can use fixation-based dwell times, which are the sum of all fixations across a certain area, including re-visits but excluding saccades (as during these, no information will be taken in). There is a lively debate on how to best measure cognitive load, e.g. with pupil diameter changes, which are often used as indicators of cognitive load. In recent years, several studies have also proposed using fixations, although their evidence is somewhat mixed (Debue & van de Leemput, 2014; Zagermann et al., 2016). Efficiency in searching for the necessary information can first be measured by the number of transitions (movements from one AOI to another). The fewer the numbers, the more efficient the student was in reading without re-reading. If the person is not sure and moves the eyes back for re-reading, this would result in additional transitions: hence, the efficiency measure. Second, the share of fixations to dwell time within an AOI can also be used to indicate efficiency. Here, all fixations during a dwell are summed and divided by the dwell time, leaving aside saccades. This assumes that when the share of fixations is closer to 100% of the dwell times, less time was spent on searching for information and more time on taking in information during fixations. As a third metric, the number of fixations can also be used as an indicator of efficiency (Holmqvist et al., 2015; Liu, Lai, & Chuang, 2011): the higher
307
The International Journal of Management Education 17 (2019) 304–315
T. Berger
•
•
the number of fixations in a certain area (e.g., AOI), the less efficient a person was in searching for the relevant information, as they had to fixate additionally (e.g., because it was difficult for the person to derive meaning from the text, which is why it is also referred to as a measure of difficulty). Reading patterns can be analyzed by making use of so-called transitions. By analyzing where a transition (essentially a saccade) starts and ends, one can detect search patterns while going through the materials and then compare these to those of other participants or clustering various types of reading patterns. In addition to the use of data on transitions, one can also use the order of first entries based on the time it took for a first fixation in a certain area (also called time to first fixation) (Holmqvist et al., 2015). The focus of a participant's first fixation, then, is shown. The order of these first entries then indicates the order in which information was taken in (in the beginning). It is also possible to analyze reading patterns based on this order of first entries with measures of concordances or to run sequence analyses or sequence searches based on the order of AOI visited (e.g., the AOI “A,” “B,” “C,” “D,” and “E”), expressed as a string (Holmqvist et al., 2015), e.g., “ACBEDADEA.” For this study, the software Blickshift was used, which allows for an automated search for a specific string in all records parallely. To analyze reading depth, we can add the time spent on an AOI, including fixations and saccades (called dwell time or visit duration) and then dividing it by the AOI size in thousands of pixels (Holmqvist et al., 2015; Holsanova, Rahm, & Holmqvist, 2016). This indicator includes saccades, as browsing here is also seen as an aspect of the depth of reading, which is not without criticism, as it is assumed that reading in the sense of taking in information can occur only during fixations (Holmqvist et al., 2015). An alternative measure, therefore, could be to use fixation-based dwell time per thousand pixels, based on fixations only.
In addition to the above measures of efficiency, the total trial duration (the time elapsed from the start to the end of a trial) could also be of interest, but it is not a clear-cut measure, as longer durations could indicate a deeper cognitive processing, a higher difficulty in reading, or simply that a participant has spent time outside AOI or even outside the stimuli. This is why it is often used only as the denominator for relative measures like the share of fixations on total trial duration or the share of fixations outside the stimuli or target areas in percentages of total trial duration. Having laid the foundations, these variables are then applied below as part of a study of case materials. We, therefore, first introduce the short case before going into more detail on the sample and eye-tracking settings. 4. Study design 4.1. The exemplary case When preparing an eye-tracking study, one must first consider which case to analyze (field of study, length, focus, etc.). For this study, a short case about an oil spill1 was created. The caselet was created to apply expected utility theory (EUT) in real life and afterward to discuss possible limitations with the class. The case of an oil spill was provided as a typical incident with information on probability and potential charges, together with some context like “no one was actually harmed” and “the area […] is now clean,” which is not relevant in a pure sense but may be helpful in understanding what actually happened. The English version of the case is included in the annex. Such a short case allows students to discover by themselves what kind of information they may want to consider when applying EUT and coming up with a number or decision. The case was distributed several times in classes in India and Germany, and group discussions were conducted that led to some minor changes in the design. Students were then asked to state what information they took into account, what their settlement sum would be, and what kind of information they missed. During these classes, typical discussions arose regarding uncertainty about whether the oil spill was an accident or done on purpose. In addition, discussions also emerged as some students stated tactical offerings and discussed the possible process of negotiating a settlement sum, often with additional discussions on the ethical aspects of striking a deal versus reaching a “proper” court decision. It was, therefore, clear that even this small case functioned as a catalyst for a wider discussion, as intended as part of the case-method teaching (Ellet, 2007). It therefore proved ideal as an exemplary case for testing the use of eye tracking for analyzing case materials, as it was short enough not to consume too much time for solving and was not too complex in nature, but still had some aspects that would fuel a discussion. When applying eye tracking in one's own study, such pre-testing with some classes could prove helpful in, e.g., designing AOI or adjusting the case based on this feedback before running the eye-tracking study. In light of the exemplary case, it is assumed that target-oriented processes are dominant, as the case was short and without many salient features, like words in bold (only the task itself was in bold) to attract attention. The instructions for the participants also made it clear that there would be questions afterward so that their behavior should be task-driven (“top-down process”), not like in the case of a free examination that would most probably be stimuli-driven (“bottom-up process”). Of course, this depends on the specific nature of a case. In some circumstances, it might even be the case that some salient features like important facts are incorporated purposely.
1
Both text versions—English and German—were tested with a readability test and had standard and average readability complexity. 308
The International Journal of Management Education 17 (2019) 304–315
T. Berger
4.2. Sample description and procedure As an additional step, the sample for the analysis must be determined. Here, different possibilities exist, as one could compare different groups or focus on just one target group, e.g., based on work experience, study program, cultural background, or the like. For the exemplary study, only management students from India and Germany currently enrolled in postgraduate study programs (e.g., master's, postgraduate diplomas, or similar2) with more than one year of work experience after their first degree were recruited. The two groups were chosen because the author works with such groups in different management programs and felt that the lessons learned from working in such very different environments (and cultures) for recording might provide some interesting insights for future lectures.3 All students were familiar with case studies and EUT, and all had either normal or corrected-to-normal vision. Participants were recruited after classes at different universities and briefed at the beginning of the study as well as asked for consent for the study. Participants then underwent a five-point calibration process measuring their eye characteristics and adjusting the 3D eye model of the eye tracker. An experienced instructor then assessed the quality of calibration, and recording began. Afterward, a short questionnaire on paper was handed out, with some principal questions on difficulty and demographics and a small amount of feedback obtained. Finally, participants were de-briefed and consented to use the data obtained. Participants received no monetary compensation but did receive some sweets. For the eye-tracking portion, the above case was shown on a laptop screen with a screen size of 17″, and the eye movements were recorded with an eye tracker attached below the screen. This allowed the researchers to record eye movements without disturbing the participants. The eye-tracker (Tobii X3-120, with a sampling rate of 120 register per second) uses near-infrared light to create reflections, which are then captured by the device and visualized. For data analysis and visualization, Blickshift Analytics version 1.09, Tobii Studio Pro 3.4.8, and SPSS 21 were used. For quality assurance, all records were deleted if they had recording gaps of longer than 3s4 or if the overall useable gaze data (recorded eye movements) were below 70% for the whole recording (including the instruction pages). For the analysis of the fixation durations, a threshold of 50 ms was chosen so as to discard short fixations, as they are deemed too short for taking in information. This means that the results for fixation durations are skewed. For some variables, an even higher threshold of 200 ms was chosen to account for the fact that participants needed some time before being able to process the information fully. This higher threshold leads to a fewer number of fixations, as only the ones longer than 200 ms are counted. This is in line with the values Rayner (2009) reports as averages for reading, and it is also what Debue and van de Leemput (2014) used for their studies. In total, 70 records from the Indian students and 60 records from the German students could be used for this analysis (two Indian and ten German records had to be excluded from analysis due to quality issues). The mean age of the German sample was 28.7 (SD = 8.24) and 70.0% of participants were male, while the mean age of the Indian sample was 29.8 (SD = 4.91) and 65.3% of participants were male. Of the Indian students, 62.9% held a bachelor's degree in engineering compared to 34% of the German students, while in the German sample 42% held a business degree, compared to only 29% from India (24% of German students and 8.3% of Indian students held degrees in other fields). The following chapter presents the results of the exemplary study as an orientation for other studies. 5. Results The German students spent less time on the exercise as a whole than did the Indian students (70.9s and SD = 23.73 versus 80.7s and SD = 29.10; t = −2.091; p = 0.038), as measured from the first mouse click after displaying the case on screen until the last mouse click that ended the display of the case (total trial duration). To analyze data in more detail, five areas of interest (AOI) were created, allowing for a comparison of different regions of the stimulus. These AOI are visualized in the Annex. 5.1. Attention and reading depth As one of the next steps of analysis, a heat map was created using the Blickshift Analytics software, visualizing the distribution of fixation lengths, where the intensity and density of fixations are coded. As shown in Fig. 2, regions that received more attention are transparent.
2 Postgraduate programs in Germany and India are not fully comparable, which is why I focused on university-based management seminars and master's degree management programs that require a bachelor's degree and lasted for a minimum of one semester. 3 The motivation to participate was strikingly different: While in India, often people queued outside the recording room, while German students were a bit more reserved. 4 This could be because of technical issues or because participants looked up or closed their eyes during the recording.
309
The International Journal of Management Education 17 (2019) 304–315
T. Berger
Fig. 2. Heat maps for both groups (left: Indian students; right: German students).
We can see that both groups focused heavily on the “10,000 €/$” value (labeled AOI Sum) and to a lesser extent on the “20%” value. The German students focused more intensely on the “80%” value, while the Indian students seemed to have focused more intensely on the context information. It is also interesting that the German students spent more fixations on regions outside the five AOI (42.8%) compared with the Indian students (35.3%). This means that German students focused more of their attention outside the deemed-relevant areas. This was further investigated by looking at reading depth as a metric. The stark differences are displayed below in Table 1 for all AOI. Table 1
Reading depth in ms per thousand pixels of AOI size
AOI Context AOI Task AOI 20% AOI 80% AOI Sum Sum of all AOI
GER
IND
t
p
Hedges g
37.31 20.35 212.82 178.57 311.26 760.64
89.82 64.82 216.01 79.68 447.25 897.83
−12.51 - 8.55 - .070 3.76 - 2.47 - 1.35
.000 < .001 .944 < .001 .015 .184
2.04 1.37 0.01 - 0.69 0.35 0.24
Cognitive processing with respect to reading depth is highest in both samples for the AOI Sum, which is understandable, as this AOI is central for answering. The difference between Indian and German students here is significant with a medium effect size. There are even larger, significant differences for the Context and Task AOIs. In general, we can see that the reading depth differs significantly between German and Indian students for most of the AOI, except for the 20% AOI. Effect sizes are large for the AOI Context and Task and small to medium for the AOI Sum and 80% (opposite direction). This means that, generally speaking, Indian students had a higher reading depth than German students, with the exception of the AOI 80%, which was read more deeply by German students. 5.2. Cognitive processing To measure cognitive processing, the total duration of the fixations (i.e., the sum of all fixations above a threshold of 50 ms) was taken. The average duration of all fixations was slightly different (non-significant) for both groups but varied considerably across the different AOIs, as shown in Table 2. Table 2
Average fixation durations in ms per AOI.
GER IND
AOI Context
AOI Sum
AOI 20%
AOI 80%
AOI Task
Mean
139 151
190 206
166 163
179 157
131 123
161 160
In our example, we can see that the AOI Sum was processed more deeply (190 and 206 ms for the Indian and German students, respectively) than the other AOI in both groups. This seems logical, as the sum stated was central to answering the question. As an additional indicator of cognitive processing, the fixation-based dwell times across all AOIs were taken, indicating cognitive load. German students had a fixation-based dwell time of 32.83s (SD = 11.59), while Indian students fixated for 48.97s (SD = 16.12):
310
The International Journal of Management Education 17 (2019) 304–315
T. Berger
a significant difference with a large effect size (t = −6.483, p < 0.001, g = 1.133), not only in absolute terms but also relative to the trial duration. This indicates that Indian students had a much higher cognitive load when reading the case. 5.3. Efficiency in reading To further assess these differences, two indicators of efficiency are recorded. First, the number of fixations is calculated, as displayed in the Table 3 below. Table 3
Average number of fixations on AOI absolute and relative to size in thousand pixels (t px). GER
AOI 20% AOI 80% AOI Context AOI Sum AOI Task All AOI
IND
count
count per t px
count
count per t px
1.93 1.77 32.72 5.43 17.01 58.85
0.387 0.354 0.054 0.597 0.028 0.048
3.08 1.35 48.13 8.49 27.01 88.04
0.617 0.270 0.141 0.933 0.103 0.142
T (t px)
P (t px)
Hedge's g (t px)
−2.31 1.27 −9.05 −3.10 −8.28 - 11.42
.023 .216 < .001 .002 < .001 < .001
0.40 −0.22 1.58 0.54 1.34 1.86
In total, Indian management students had a much higher number of fixations per thousand pixels than did the German students. To verify these findings further, we made use of the number of transitions, also indicating efficiency. In total, the German participants had 10.5 transitions per recording, while the Indian students had 13.8 transitions, meaning that the German management students were in general more efficient at solving the case as measured by the number of transitions. Together with the fixation-based dwell times and the number of fixations, this may indicate that German students processed the information with less difficulty. 5.4. Analysis of patterns To determine whether the patterns for going through the case materials are different, we can first look at the transitions for the whole group and then at the scan paths of all participants to compare them and find common sequences. For the latter, the Blickshift Analytics software was used to allow the software to scan for more complex patterns.5 The software searched for patterns that would indicate “re-assuring” or “double-checking” between the actual task and the central information needed to fulfill the task (i.e., transitions between the AOI Sum and AOI Task and back to AOI Sum). As Scholz et al. (2015, p. 243) have identified “an evergrowing number of papers showing that when retrieving information from memory, people gaze back at spatial locations that have been associated with the to-be-retrieved information during encoding,” it was assumed that such a pattern indicates that participants gaze back to re-assure themselves of what they have memorized. The software found the above pattern once in 56% of the Indian records and in 45% of the German records. This is further verified by looking at the backwards transitions from the AOI Task to the AOI Sum, which were found at least twice in 50% of the Indian records and in only 38% of the German records. German students, therefore, do not seem to double-check as often as the Indian students, which is also in line with their smaller number of transitions. To further analyze patterns, the transitions based on first fixations in an AOI were identified (Holmqvist et al., 2015). AOI are ordered according to the time it took the participants to fixate their eyes on each AOI, not including fixations outside the AOIs. The whole five-step sequence through all AOIs (meaning that everything was read at least once) was visible in only 50% of the Indian recordings and in only 56.7% of the German recordings. Following an ideal reading pattern, one should transition from the first AOI at the top down to the last, just as they would read normally. This theoretically ideal pattern was only found in 11.7% (seven participants) of the German records and in 5.6% (four participants) of the Indian records. While 79.2% of the Indian students started reading from the top, virtually all German students (96.7%) started reading from the top. 16.7% of Indian students read the task first (only one German student did that) and only one third then moved on to read the rest of the case description from the top moving down. Indian students, therefore, seemed to be more task-oriented than German students.
5 Using the Blickshift software, of course, is not necessary for similar projects. The software's main advantage over manual similarity searches is its use of a clustering algorithm using Levenshtein distance. See Raschke et al. (2014) for more information.
311
The International Journal of Management Education 17 (2019) 304–315
T. Berger
The order of reading through the AOIs for both groups is displayed in Table 4, based on average order ranks for the five AOIs. Table 4
Order of first entries for the five AOIs.
IND GER
AOI Context
AOI Sum
AOI 20%
AOI 80%
AOI Task
Kendall's W
χ2
1.31 1.06
2.91 2.65
3.93 3.29
4.00 4.12
2.85 3.88
0.477 0.601
64.86* 81.69*
*p < 0.001.
The pattern of the German sample is closer to the “ideal” reading pattern than is the pattern of the Indian sample (only the order of the AOIs “80%” and “Task” were different). If we compare the two mean ranking values of German and Indian students further, Kendall's Coefficient of Concordance W stands at 0.850 (χ2 = 6.8, p = 0.147), showing a high commonality between the two samples but one that is not statistically significant. 6. Discussion 6.1. Some guidelines for using eye tracking This paper showed how eye tracking can be applied for case-study analysis. A short case on an oil spill incident was shown on a laptop screen, and participants had to answer a question to the case presented based on the EUT. This case was designed to facilitate discussion on possible limitations of the EUT. Based on the experiences during this study, some general points for the use of eye tracking in analyzing cases are discussed below that may help develop the potential of this method. For this study, we made use of an eye tracker attached to a laptop, but in other circumstances it might be more helpful to use head-mounted eye trackers. They have the advantage of offering more freedom in recording (e.g., to include interactions with other participants or material in addition to what is on the screen). On the other hand, such eye trackers are not as precise as mounted ones, and the analysis, therefore, would be more limited (Holmqvist et al., 2015). The exemplary study had two culturally different participant groups for analysis, and it sometimes proved hard to create a comparable environment in both countries, as variations in settings were vast: e.g., sunlight exposure, brightness, room size, and noise level. In particular, light intensity should always be measured and set to the levels as set out in the specifications of the eye tracker, as this will influence the recordings. Changes in light intensity over the course of the recording should also be avoided. A solution could be to conduct a study in the same room every time (preferably a special laboratory with no windows or shutters), but that would also limit the scope of possible studies. As shown in the above results section, the definition of variables is critical for every study. In the case of eye tracking, even in the standard books—and more so in the different studies—there are various recommendations for the definition of variables, so they are often hard to compare across studies. Careful design of the variables to capture the underlying cognitive processing, therefore, is crucial and should be supported by a methodological framework like the Cognitive Load Theory mentioned above. The underlying theory or hypothesis will then influence the choice of variables and underlying metrics. This study, grounded in the eye-mind hypothesis, mainly made use of fixations, e.g., to assess efficiency and reading depth. It therefore left aside pupillometric data that also could prove helpful when, e.g., assessing cognitive load. Data quality is also of major importance. In this study, we used the indicators on data quality provided by the manufacturer Tobii and supplemented this with own analysis of the data gaps, which were assessed individually, record-by-record. This is time-consuming, and there are also other variables available to assess data quality (Holmqvist, Nyström, & Mulvey, 2012). Data output, and consequently data quality, is also influenced by the definition of the metrics, like fixations or saccades. If one defines the minimum length of a fixation as 20 ms or 100 ms, this will have a major effect on the output, as the device will transfer the raw data based on these settings. Here, results from meta-studies, like those of Rayner (2009) can be used as a guideline. The final definition will still depend on the kind of stimuli presented—text vs. figures—and what the researchers have in mind. As there are almost no studies on case materials, it will not be easy to find existing studies as references. The solution in this study was to make use of typical definitions from reading studies, as the case analyzed was purely text-based. As most studies rely on the use of AOIs, creating them is also critical. For designing AOIs, several methods are available. This study defined AOIs based on semantic groupings like text on the task, the sums involved, or the like. It would also be possible to create a grid and then analyze data based on this (Holmqvist et al., 2015) or on other criteria. The minimum size of the AOI then is determined by the specifications of the eye tracker and a minimum margin of error should be taken into account (which will be larger for mobile eye-trackers than for mounted ones). For analyzing casework more broadly, data on eye tracking should be combined with additional methods like focus group meetings, think-aloud techniques, or in-depth interviews. This will not only enable the verification of data from eye tracking, but could also provide valuable insights into the social interaction taking place or the understanding of contents. Despite the advantages eye tracking may offer, it will by itself “not tell the researcher anything about the success or failure of comprehending the relevant piece of information” (Hyönä, 2010, p. 173). The exemplary study above was done using a questionnaire for eye-tracking participants and group discussions in the pre-test before applying eye tracking. One aspect, then, was the level of difficulty. To analyze difficulty 312
The International Journal of Management Education 17 (2019) 304–315
T. Berger
later on, fixation durations and fixation counts were taken, and the findings may indicate a higher difficulty in extracting information for the Indian students. With additional data, such characteristics can be further verified, as was the case in the exemplary study: while German students rated difficulty at 1.33 in the pre-test, Indian students rated difficulty at 2.59 on a scale from 1 (easy to understand) to five (difficult to understand), which is in line with the eye-tracking data. 6.2. Future research possibilities One possible strand of research could use the existing studies on efficacy to combine eye tracking with questionnaires, interviews, tests, or essays, e.g., based on the findings of Druckman and Ebner (2018) or Farashahi and Tajeddin (2018). It would also be possible to test this with respect to different kinds of learning outcomes. Future studies could also apply controlled experiments where different kinds of cases or guidance strategies are tested and where the effect on different kinds of cognitive load is measured, just as Kirschner, Sweller, and Clark (2006) have proposed. It could, therefore, go as far as to design specific multimedia courses that are flexible and adjustable to the needs of different groups based on culture or other criteria. When working with such cases, the cognitive load could be tracked and the contents adjusted to changes in this cognitive load. With the arrival of cheap eye trackers and systems with built-in eye trackers, this is not too far away (Liu et al., 2011). In addition, one could also study whether the content of the case influences the kind of processing by using cases from different disciplines, such as marketing, accounting, or HR, together with graphical displays and different types of cases. These studies could also be conducted with often-used cases of variable lengths from large providers. Here, one question could be what the typical patterns of going through the materials are, as was done here with the help of the Blickshift software.6 Other avenues related to this may be to compare different cultural groups, like in the exemplary study; even this study, which uses a short case, has shown that there may be large differences regarding reading depth, difficulty, attention, and cognitive processing in general that should be kept in mind when designing or teaching with a case. Again, this could be combined with additional methods to capture the social interactions and other possible variables influencing the results. The differences in fixations witnessed in the exemplary could, e.g., be based on the fact that the language used in the case provided was not identical for both groups, although as relates to reading there seems to be no “strong evidence in favor of the idea that culture systematically modulates eye movements” (Rayner, Li, Williams, Cave, & Well, 2007). Such areas need additional research, as from this study it cannot be determined whether the significant differences in the modulation of eye movements are caused by cultural, educational, or other differences among participants. Funding This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Declaration of interest: none. Acknowledgements I wish to thank Walburga Sarcher of the University of Augsburg, Pankaj Trivedi from KJ Somaiya Institute, Mumbai, and Jagdish and Ramesh Shah for helping me organizing eye-tracking sessions at several organizations, Ignace Hooge who gave mee feedback on an earlier version as well as all others who helped me with this research, including the participants in Germany and India. I also want to thank the two anonymous reviewers who have given me very detailed and helpful feedback.
6
This could also be done with software like MatLab or the like. Many researchers use MatLab for eye-tracking analysis. 313
The International Journal of Management Education 17 (2019) 304–315
T. Berger
Annex
Case description and areas of interest (below), English version.
References Aithal, S. (2016). Innovations in student centric learning: A study of top business schools in India. Munich: MPRA Paper. Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011). Does discovery-based instruction enhance learning? Journal of Educational Psychology, 103(1), 1–18. Altmann, G. T. M., & Kamide, Y. (2007). The real-time mediation of visual attention by language and world knowledge. Linking anticipatory (and other) eye movements to linguistic processing. Journal of Memory and Language, 57(4), 502–518. Argyris, C. (1980). Some limitations of the case method. Experiences in a management development program. Academy of Management Review, 5(2), 291–298. Ashby, N. J. S., Johnson, J. G., Krajbich, I., & Wedel, M. (2016). Applications and innovations of eye-movement research in judgment and decision making. Journal of Behavioral Decision Making, 29 2–3, 96–102.
314
The International Journal of Management Education 17 (2019) 304–315
T. Berger
Bayona, J. A., & Castañeda, D. I. (2017). Influence of personality and motivation on case method teaching. International Journal of Management in Education, 15(3), 409–428. Blankley, A. I., Kerr, D., & Wiggins, C. E. (2017). The state of accounting education in business schools: An examination and analysis of active learning techniques. Advances in Accounting Education: Teaching and Curriculum Innovations, 21, 101–124. Burgoyne, J., & Mumford, A. (2001). Learning from the case method: A report to the European case clearing House, RP0301Lancaster: European Case Clearing House. Chu, L., & Libby, T. (2010). Writing mini-cases. An Active Learning Assignment”, Issues in Accounting Education, 25(2), 245–265. Debue, N., & van de Leemput, C. (2014). What does germane load mean? An empirical contribution to the cognitive load theory. Frontiers in Psychology, 5, 1099. Druckman, D., & Ebner, N. (2018). Discovery learning in management education. Design and case analysis. Journal of Management Education, 42(3), 347–374. Easton, G., & Ormerod, T. C. (2001). Expert/Novices differences in case analysis,Lancaster (UK): Lancaster University. Ellet, W. (2007). The case study handbook: How to read, discuss, and write persuasively about cases. Boston, Mass: Harvard Business School Press. Farashahi, M., & Tajeddin, M. (2018). Effectiveness of teaching methods in business education. A comparison study on the learning outcomes of lectures, case studies and simulations. International Journal of Management in Education, 16(1), 131–142. Hoag, K., Lillie, J., & Hoppe, R. (2005). Piloting case-based instruction. Didactic clinical immunology course. Clincial Laboratory Science, 18(4), 213–220. Holmqvist, K., Nystrom, M., Andersson, R., Dewhurst, R., Jarodzka, H., & van de Weijer, J. (2015). Eye tracking: A comprehensive guide to methods and measures. Oxford: Oxford University Press First published in paperback 2015. Holmqvist, K., Nyström, M., & Mulvey, F. (2012). Eye tracker data quality. In C. H. Morimoto, H. Istance, J. B. Mulligan, P. Qvarfordt, & S. N. Spencer (Eds.). Proceedings of the symposium on eye tracking research and applications - ETRA '12, Santa Barbara, California, (pp. 45). New York, New York: ACM Press. Holsanova, J., Rahm, H., & Holmqvist, K. (2016). Entry points and reading paths on newspaper spreads. Comparing a semiotic analysis with eye-tracking measurements. Visual Communication, 5(1), 65–93. Hughes, S. B. (2017). Student-authored IFRS teaching cases based on European Securities and Markets Authority reports. Experiences from case writing and subsequent classroom use. Journal of Accounting Education, 41(12), 58–74. Hyönä, J. (2010). The use of eye movements in the study of multimedia learning. Learning and Instruction, 20(2), 172–176. Hyönä, J., Lorch, R. F., & Rinck, M. (2003). Eye movement measures to study global text processing. In J. Hyönä, R. Radach, & H. Deubel (Eds.). The mind's eye: Cognitive and applied aspecets of eye movement research (pp. 313–334). Oxford: Elsevier. Jennings, D. (1996). Strategic management and the case method. The Journal of Management Development, 15(9), 4–12. Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87(4), 329–354. Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work. An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86. Lai, M.-L., Tsai, M.-J., Yang, F.-Y., Hsu, C.-Y., Liu, T.-C., Lee, S. W.-Y., et al. (2013). A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educational Research Review, 10(12), 90–115. Lindner, M. A., Eitel, A., Thoma, G.-B., Dalehefte, I. M., Ihme, J. M., & Köller, O. (2014). Tracking the decision-making process in multiple-choice assessment. Evidence from eye movements. Applied Cognitive Psychology, 28(5), 738–752. Liu, H.-C., Lai, M.-L., & Chuang, H.-H. (2011). “Using eye-tracking technology to investigate the redundant effect of multimedia web pages on viewers' cognitive processes”. Computers in Human Behavior, 27(6), 2410–2417. Liversedge, S. P., Drieghe, D., Li, X., Yan, G., Bai, X., & Hyönä, J. (2016). Universality in eye movements and reading. A trilingual investigation. Cognition, 147, 1–20. Ludwig, C. J. H., & Evens, D. R. (2017). Information foraging for perceptual decisions. Journal of Experimental Psychology. Human Perception and Performance, 43(2), 245–264. Lund Dean, K., & Fornaciari, C. J. (2016). How to create and use experiential case-based exercises in a management classroom. Journal of Management Education, 26(5), 586–603. Lundberg, C. C., Rainsford, P., Shay, J. P., & Young, C. A. (2001). Case writing reconsidered. Journal of Management Education, 25(4), 450–463. Meißner, M., & Oll, J. (2017). The promise of eye-tracking methodology in organizational research. Organizational Research Methods, 8(10), 1–28. Mintzberg, H. (2005). Managers not MBAs: A hard look at the soft practice of managing and management development (paperback ed.). San Francisco, Calif: BerrettKoehler. Nkhoma, M., Sriratanaviriyakul, N., & Le Quang, H. (2017). “Using case method to enrich students' learning outcomes”. Active Learning in Higher Education, 18(1), 37–50. Raschke, M., Herr, D., Blascheck, T., Ertl, T., Burch, M., Willmann, S., et al. (2014). A visual approach for scan path comparison. In Association for Computing Machinery (Ed.). Proceedings of the symposium on eye tracking research and applications (pp. 135–142). New York, New York: ETRA ’14. Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology, 62(8), 1457–1506. Rayner, K., Li, X., Williams, C. C., Cave, K. R., & Well, A. D. (2007). Eye movements during information processing tasks. Individual differences and cultural effects. Vision Research, 47(21), 2714–2726. Scholz, A., Helversen, B. von, & Rieskamp, J. (2015). Eye movements reveal memory processes during similarity- and rule-based decision making. Cognition, 136, 228–246. Selvam, M., Babu, M., & Raja, M. (2006). Caslets teaching in business education. SMART Journal of Business Management Studies, 2(2), 70–72. Shrivakumar, K. (2012). The case study method in training and management education. The IUP Journal of Soft Skills, 6(2), 55–64. Strobel, B., Lindner, M. A., Saß, S., & Köller, O. (2018). Task-irrelevant data impair processing of graph reading tasks. An eye tracking study. Learning and Instruction, 55(6), 139–147. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257–285. Thistlethwaite, J. E., Davies, D., Ekeocha, S., Kidd, J. M., MacDougall, C., Matthews, P., et al. (2012). The effectiveness of case-based learning in health professional education. A BEME systematic review. BEME Guide No. 23. Medical Teacher, 34(6), 421–444. Tsai, M.-J., Hou, H.-T., Lai, M.-L., Liu, W.-Y., & Yang, F.-Y. (2012). Visual attention for solving multiple-choice science problem. An eye-tracking analysis. Computers & Education, 58(1), 375–385. Wass, S. V., Forssman, L., & Leppänen, J. (2014). Robustness and precision. How data quality may influence key dependent variables in infant eye-tracker analyses. Infancy, 19(5), 427–460. Zagermann, J., Pfeil, U., & Reiterer, H. (2016). Measuring cognitive load using eye tracking technology in visual computing. In M. Sedlmair, P. Isenberg, T. Isenberg, N. Mahyar, & H. Lam (Eds.). Proceedings of the beyond time and errors on Novel evaluation methods for visualization - BELIV '16, Baltimore, MD, USA (pp. 78–85). New York, New York: ACM Press.
315