The effectiveness of adaptive difficulty adjustments on students' motivation and learning in an educational computer game

Computers & Education 69 (2013) 452–462

Contents lists available at SciVerse ScienceDirect

Computers & Education journal homepage: www.elsevier.com/locate/compedu

The effectiveness of adaptive difficulty adjustments on students’ motivation and learning in an educational computer game Sandra Sampayo-Vargas a, *, Chris J. Cope b, Zhen He a, Graeme J. Byrne c a

Department of Computer Science and Computer Engineering, La Trobe University, Plenty Rd, Bundoora, Melbourne, VIC 3086, Australia Department of Computer Science and Computer Engineering, La Trobe University, Edwards Rd, Bendigo, Melbourne, VIC 3552, Australia c Department of Mathematics and Statistics, La Trobe University, Edwards Rd, Bendigo, Melbourne, VIC 3552, Australia b

a r t i c l e i n f o

a b s t r a c t

Article history: Received 25 March 2013 Received in revised form 28 June 2013 Accepted 1 July 2013

Computer games that adaptively adjust difficulty are used to continuously challenge players according to their abilities. The adjustment of difficulty occurs automatically in response to a game’s ongoing assessment of a player’s performance. This approach to difficulty adjustment is likely to be of value in educational computer games as a means of scaffolding learning for students. However, there is limited research evaluating the effectiveness of educational computer games with adaptive difficulty adjustment when compared to non-adaptive difficulty adjustment. To expand on this research a quasi-experimental study was designed to isolate the impact of the difficulty adjustment game element on motivation and learning. A total of 234 secondary school students were allocated to one of three activities involving learning about Spanish cognates: an adaptive difficulty adjustment game, an incremental difficulty adjustment game that was non-adaptive, and a written activity. The three learning activities were designed following the same learning and motivation theories. The two games were identical apart from the difficulty adjustment mechanism. The results for motivation indicated that all students experienced high levels and there was no significant difference between the three learning activities. The pre- and post-tests results for learning indicated that significantly higher learning outcomes were achieved by students who played the adaptive game. Analysis of a game log recording the correctness of students’ responses indicated that the adaptive difficulty adjustment game, in contrast to the non-adaptive incremental difficulty adjustment game, provided a scaffolding structure to enhance student learning. Ó 2013 Elsevier Ltd. All rights reserved.

Keywords: Educational computer games Adaptive difficulty adjustments Scaffolding Secondary education Teaching/learning strategies

1. Introduction Educational computer games have become a popular teaching tool because of their ability to increase students’ motivation (Connolly, Boyle, MacArthur, Hainey, & Boyle, 2012; Egenfeldt-Nielsen, 2007; Hays, 2005; Kirschner, Sweller, & Clark, 2006; Squire, 2005). This increase in motivation has been identified when educational computer games are not only educational but also fun to play (or intrinsically motivational) (Berns, Gonzalez-Pardo, & Camacho, 2013; Brom, Preuss, & Klement, 2011; Reinders, 2012). In education, students that are intrinsically motivated and whose levels of perceived competence are high often engage in academic behaviours that lead them towards higher learning outcomes (Cordova, 1983). Conversely, a lack of motivation is more likely to hinder learning. One way to make educational computer games intrinsically motivational is to provide an optimal level of challenge. An optimal level of challenge is one where players are able to solve tasks that are neither too easy nor too difficult (Aponte, Levieux, & Natkin, 2011; Johnson, Vilhjalmsson, & Marsella, 2005). Learning activities that optimally challenge students and also recognise when they struggle and provide support are said to incorporate a scaffolding strategy (Van Der Stuyf, 2012). Learning activities implementing scaffolding have resulted in better learning outcomes than using the same activities without scaffolding (Chang, Sung, & Chen, 2001; Chang, Wernhuar, & Shin, 2009; Murphy & Messer, 2000). Of the multiple game elements that comprise an educational computer game, difficulty adjustment seems most likely to provide students with an optimal level of challenge and a scaffolding strategy. In particular,

* Corresponding author. Tel.: þ61 3 94793036. E-mail addresses: [email protected] (S. Sampayo-Vargas), [email protected] (C.J. Cope), [email protected] (Z. He), [email protected] (G.J. Byrne). 0360-1315/$ – see front matter Ó 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.compedu.2013.07.004

S. Sampayo-Vargas et al. / Computers & Education 69 (2013) 452–462

453

adaptive difficulty adjustment can be used to monitor the correctness of players’ responses and continuously alter the level of challenge up and down as necessary. Whilst educational computer games with adaptive difficulty adjustment would seem to have the necessary capabilities to promote student learning there is little research evaluating their effectiveness (Koidl, Mehm, Hampson, Conlan, & Gobel, 2010; Rodriguez, Cheng, & Basu, 2007). In particular, there are few studies comparing an educational computer game with adaptive difficulty adjustment with an equivalent game with non-adaptive difficulty adjustment (Bauer, Brusso, & Orvis, 2012; Orvis, Horn, & Belanich, 2008). Therefore, the aim of the study reported in this paper was to evaluate the effectiveness of an educational computer game with adaptive difficulty adjustment on students’ motivation and learning outcomes when compared to both an educational computer game with non-adaptive difficulty adjustment and a traditional written activity. 2. Background 2.1. Motivation and learning theories A number of motivation and learning theories were used in the design of the learning activities reported in this paper, including the educational computer games. The adaptive difficulty adjustment game used in the study was intended to scaffold students’ learning. Scaffolding is a learning theory focused on tailoring teaching within the students’ zone of proximal development (ZPD). The ZPD refers to what a learner can achieve alone and what can be accomplished with assistance (Rodgers & Rodgers, 2004; Vygotsky, 1978). Scaffolding student learning involves supporting students’ ability to build on prior knowledge and internalise new information by providing them with learning activities that are just beyond the current level of what they can achieve by themselves (Olson & Platt, 2000; Van Der Stuyf, 2012; Vygotsky, 1978). A learning activity can be designed to scaffold instruction by recognising the point at which difficulty occurs and structuring supports at that time (Rodgers & Rodgers, 2004). The design of educational computer games can be enhanced through the incorporation of motivation and learning theories (KickmeierRust et al., 2007; Koidl et al., 2010). For the game presented in this study, Malone’s motivational theory and Csikszentmihalyi’s flow theory were used to enhance the motivational aspect (Csikszentmihalyi, 1990; Malone, 1981), and the cognitive load and perceptual learning theories were used to enhance the educational aspect (Colvin, Nguyen, & Sweller, 2006; Goldstone, 1998). The principles underpinning these learning and motivation theories were incorporated in a set of guidelines used for the game design (see Section 4.1). Malone’s motivational theory emphasises how to make educational computer games motivational and fun to play through the use of elements such as: challenge, goals, feedback, uncertain outcomes, self-esteem, player skills, curiosity, control and fantasy (Malone & Lepper, 1987). Similarly, the Flow theory relates to elements that make an activity enjoyable such as: completion, challenge, player skills, goals, feedback, concentration and control (Sweetser & Wyeth, 2005). These elements were included in the learning activities of this study. For example, the element of challenge was included in the game by increasing the difficulty of both the educational content and the gameplay. In addition, the level of challenge was tailored according to the students’ skills using adaptive difficulty adjustments. Due to space constraints, it is not possible to present how each one of the motivational elements was incorporated into the game and the written activity. The details are instead available in Sampayo-Vargas (2012). The cognitive load theory specialises on methods to design and deliver instructional environments in ways that best utilise human cognition. The cognitive load theory follows a set of principles such as: redundancy – to minimise redundant teaching material; modality – to facilitate learning by providing audio explanations of visuals rather than text explanations; multimedia – to present the content as words and graphics rather than words alone, etc. Moreover, the learning theory of perceptual learning involves repetition to stimuli and long lasting changes to improve the ability to respond to those stimuli. Perceptual learning is based on mechanisms to better respond to stimuli such as: attention-weighting, categorical perception, stimulus-imprinting, differentiation and unitisation. These principles and mechanisms were used to design the learning activities of this study. For example, the principle of modality was included in the game using short, relevant and clear audio of the correct Spanish pronunciation when the English word was translated into Spanish. Again, due to space constraints, it is not possible to present the details of how each one of the learning principles and mechanisms were incorporated into the game and written activity. The details are instead available in Sampayo-Vargas (2012). 2.2. Difficulty adjustment The definition of difficulty refers to something laborious, not easy to do or understand, which requires an effort to be accomplished (Nicholls & Miller, 1983). Following that definition, in this paper difficulty refers to the effort required to overcome the challenges presented in the learning activities of the empirical study. There are two broad approaches to adjust difficulty in educational computer games: non-adaptive and adaptive difficulty adjustment. Non-adaptive difficulty refers to either the use of the same difficulty settings throughout a game or to adjustment based on game settings that are unrelated to the players’ performance. An example of the latter is a game that incrementally increases difficulty after pre-defined intervals of time. In contrast, an adaptive difficulty mechanism adjusts the game difficulty according to the players’ performance during gameplay (Andrade & Corruble, 2005). In educational computer games the challenges are related not only to the difficulty of the gameplay but also to the difficulty of the educational content. The difficulty of the gameplay refers to the complexity of the game mechanics and settings. For example the number of objects to manoeuvre, speed, time limit to respond, etc. The difficulty of the educational content can be categorised depending on its complexity. Difficulty adjustments can be applied to the gameplay, the difficulty of the content or both. Thus adaptive difficulty adjustment can be used to provide an optimal level of challenge not only to the gameplay but also to the educational content. One possibility to support scaffolding is for an adaptive difficulty mechanism to reduce the difficulty of the gameplay when the students’ responses indicate that they are having difficulty with the level of challenge of the content. The reduction in the difficulty of the gameplay would allow students more time to learn the content.

454

S. Sampayo-Vargas et al. / Computers & Education 69 (2013) 452–462

The main drawback of using adaptive difficulty adjustment in computer games is that it is more difficult and costly to implement than non-adaptive difficulty. Therefore, there is a need to ensure that adaptive difficulty can indeed increase the learning outcomes of educational computer games compared to non-adaptive difficulty to justify the higher implementation costs. This improvement in learning outcomes has yet to be established. 3. Related work Research evaluating the effectiveness of educational computer games through comparison with other learning activities, and without regard for specific game elements, has returned mixed results. Although educational computer games have been shown to produce high levels of motivation, there is no clear evidence of improved learning outcomes in all situations (Connolly et al., 2012; Hays, 2005). Most of the research has suffered from problems with study design as pointed out by the surveys of Hays (2005) and Egenfeldt-Nielsen (2007). In some instances the games were not compared with a traditional learning activity. In other studies learning activities that differed in multiple aspects were compared. For example, some games were designed to follow a learning theory and were compared to activities that did not. Where traditional learning activities were used for comparison they did not necessarily follow the same educational objectives and equivalent design as the games. All of these reasons have made it difficult to understand why some educational computer games produced better results than others. One way to better understand this problem is to isolate a game element, to design the game following sound learning theories and to follow the same or equivalent design for all of the learning activities compared. The limited research that has evaluated educational computer games with adaptive difficulty has shown positive results for learning and motivation (Klinkenberg, Straatemeier, & Van Der Maas, 2011; Moser, 2000; Wilson, Revkin, Cohen, Cohen, & Dehaene, 2006). A comparison between the games or equivalent traditional learning activities was not generally a feature of this research. In a study where the same educational computer game was compared varying only the method used to adjust difficulty the results did not show statistically significant differences for motivation and performance (Orvis et al., 2008). In this study the games were used to help military trainees to improve their shooting skills. The games did not include educational content in the form of knowledge and they were not compared with a traditional learning activity. It is beneficial to compare the effectiveness of educational computer games over traditional activities to justify the costs incurred in the development of the games. Research evaluating educational computer games with other adaptive mechanisms, such as adaptive storytelling, adaptive narrative and adaptive feedback, have reported positive results for learning and motivation (Conati & Manske, 2009; Gobel, Wendel, Ritter, & Steinmetz, 2010; Hodhod, 2010; Kickmeier-Rust, Marte, Linek, Lalonde, & Albert, 2008; Law, Kickmeier-Rust, Albert, & Holzinger, 2008; Peirce, Conlan, & Wade, 2008). The positive results indicated that the use of adaptive educational computer games could improve learning and motivation. However, this research did not focus on adaptive difficulty adjustment and lacked reliability due to the shortage of research, lack of comparisons with equivalent learning activities, and the influence of differences in multiple game elements. Research in the domain of entertainment computer games has indicated that adaptive difficulty provided a more suitable level of challenge than non-adaptive difficulty (Andrade, Ramalho, Gomes, & Corruble, 2006; Hsieh & Wang, 2008; Spronck, Ponsen, SprinkhuizenKuyper, & Postma, 2006). As a result, players are more likely to experience high levels of motivation, immersion and to engage in longer hours of gameplay (Hua, Pei-Luen, & Gavriel, 2010). Nevertheless, these results are only indicative because most of the games evaluated used only virtual players (computer generated). The focus of entertainment computer games is to increase the player’s motivation, hence learning outcomes are not measured. The design of adaptive difficulty in entertainment computer games has commonly used complex artificial intelligent (A.I.) techniques which often require more time and high costs to be well-implemented. These complex A.I. techniques are often unsuitable for simple educational computer games. The game used in this study is characterised as a simple educational computer game, thus, a more simplistic method was chosen to adjust difficulty adaptively. In summary, there has been limited research investigating educational computer games with adaptive difficulty adjustment and their impact on students’ motivation and learning. The research has given indications of the benefits of this difficulty adjustment mechanism but a more definitive study designed to isolate the impact of adaptive difficulty adjustment is warranted. This was the approach taken in this study which compared two versions of a game that only differ in the difficulty adjustment mechanism used and a traditional learning activity that followed an equivalent design as the games.

Table 1 Guidelines used to design the game. Guidelines used for the game design 1 2 3 4 5 6 7 8 9 10 11 12

Educational content Simplify the educational content by removing any irrelevant information. Categorise and order the educational content in varying degrees of difficulty. Limit the number of curriculum categories to be given at once. Gameplay mechanism Define a quick movement and quick reward gameplay mechanism. Embed the educational content within the gameplay mechanism. Create levels of difficulty by including curriculum categories and game difficulty settings. The speed of advancement in the gameplay mechanism allows faster progression according to each student’s skills. Use an active tutorial to teach how to use the gameplay mechanism and play the game. Adaptive difficulty Define two thresholds to increase and decrease difficulty. Adapt the difficulty of the game to students’ different abilities. Feedback Provide meaningful immediate feedback for students’ actions. Provide a summary of progress and positive feedback.

S. Sampayo-Vargas et al. / Computers & Education 69 (2013) 452–462

455

4. Learning activities Three equivalent learning activities were developed to evaluate the effectiveness of adaptive difficulty adjustment in an educational computer game. The first two activities consisted of two versions of a Spanish cognates bubble game, varying only the method used to adjust difficulty. The first version used adaptive difficulty adjustment and the second version used incremental difficulty adjustment which was non-adaptive. The educational objectives expected from the games were categorised, using the Bloom’s revised taxonomy, as remembering, understanding and applying conceptual and factual knowledge (Anderson et al., 2001). The third learning activity consisted of a written activity designed following the same educational objectives, educational content and theories used to design the games. 4.1. Game design The design of the Spanish cognates bubble game was based on a set of guidelines created from preliminary observations of a pilot study, knowledge gained from learning and motivation theories and common features from existing simple educational computer games with adaptive difficulty. The guidelines were created to ensure that the game design contained both motivational and learning aspects and to isolate the game element of adaptive difficulty adjustment. To do so, other game elements such as graphics, feedback and sounds were designed in their most basic form. A summary of the guidelines used to design the Spanish cognates bubble game is presented in Table 1, and a detailed description of each guideline can be found in Sampayo-Vargas (2012). The method chosen to adjust difficulty adaptively was based on the Computer Adaptive Test of ‘On Demand Testing’ by the Victorian Curriculum and Assessment Authority (VCAA)1,2. From this knowledge an adaptive difficulty algorithm was created to maximise learning by allowing a student to progress to more difficult educational content once the student had mastered the content and to decrease the difficulty of the gameplay if the student was struggling (see Appendix A). 4.2. Overview of the Spanish cognates bubble game The game starts with a login screen where students are required to type their name, age and email address. This facilitates the identification and recording of the students’ responses during gameplay. Once the students’ details are recorded, the game begins with an interactive tutorial to show students what to do and how to play the game. The objective of playing the Spanish cognates bubble game is for students to translate English words into Spanish words by using different Spanish endings, hence its name. A screenshot of the Spanish cognates bubble game is presented in Fig. 1. The underlying structure of the game consists of boxes containing English words and bubbles containing Spanish endings. The boxes are located at the top of the game window and the bubbles move from the bottom to the top of the game window. Students can move the bubbles in any direction by clicking around them, and the main objective is to match the bubbles with the boxes. A correct response provides students with the correct Spanish translation in both text and voice formats. The box containing the English word is replaced with the Spanish word and the box is highlighted in green colour. The score increases, and the Spanish word is replaced with another English word belonging to a random curriculum category. An incorrect response is highlighted by turning the box to a red colour and the English word remains in the box for students to try again. The game was programmed to finish after approximately 13 min, including transition between levels. This time was limited for experimental purposes and to fit within one class session of secondary schools. The educational content included in the game is divided into curriculum categories, determined by the type and difficulty of the Spanish ending. The game includes a total of 16 curriculum categories and each one is composed of an average of 103 words. The endings ‘or, al, ble, sión’ represent easy educational content. Easy endings refer to English words which are written and mean the same in the Spanish language, such as director, animal, sociable and tension. Endings with medium difficulty (such as ‘cion, ente, ante, ista, ismo, ico, mento, ivo’) require at least one change to be translated properly. An example is the English word direction ¼ ‘dirección in the Spanish language’. Endings that require more than one change (such as ‘oso, ar, dad, ificar’) are categorised as a difficult content. An example is curious ¼ ‘curioso’. False cognates and words that require other changes besides the endings were not part of the educational content. The difficulty settings used in the Spanish cognates bubble game are presented in this section, and the adjustment of difficulty for each version of the game are presented in Sections 4.2.1 and 4.2.2. The game has 11 levels of difficulty, each defined by the number of boxes, number of bubbles, bubble speed and the difficulty of the educational content. For example level of difficulty 1 is composed of the easy difficulty endings ‘or, al’ and the medium difficulty ending ‘ción’. Level 1 also contains the game settings of two boxes, one bubble, and the speed of the bubble is 80. The bubble speed refers to units per second at which the bubble moves in the screen. A speed of 80, 70, and 60 indicate that it takes around 9, 10 and 11 s respectively for the bubbles to reach the top of the screen, without any interaction. For the difficulty levels, the bubble speed decreased as the difficulty of the other settings increased. These settings allow students to concentrate more on the educational content, rather than be challenged by the difficulty of the gameplay. Table 2 presents the composition of each level of difficulty used in the Spanish cognates bubble game. A preliminary validation of the game was conducted by interviewing four Year 7 students who played the game (two for each version of the game). The students indicated that the game was fun to play, educational and feasible to be completed within a time frame of 20 min. 4.2.1. Adaptive difficulty adjustment version Both versions of the game contain the same educational content and difficulty settings. The only difference between the games was the method used to adjust difficulty. The adaptive version of the game considers a curriculum category learnt when the student provides three consecutive correct responses for the same curriculum category. The game displays the consecutive correct responses with a blue progress bar. The game increases by one level of difficulty when all the curriculum categories per level of difficulty are learnt. Three incorrect

1 2

http://www.vcaa.vic.edu.au. http://www.aimonline.vic.edu.au.

456

S. Sampayo-Vargas et al. / Computers & Education 69 (2013) 452–462

Fig. 1. Spanish cognates bubble game.

responses to the same curriculum category result in the decreasing of the gameplay difficulty by reducing the number of boxes by one, stopping at the minimum of two boxes. The other settings that compose the gameplay difficulty remain fixed as they were set up in each level of difficulty. The game records the curriculum categories that are learnt and unlearnt and gives priority to unlearnt curriculum categories. This allows students a faster progression to new and more difficult educational content. The adaptive difficulty game implements scaffolding because it recognises when the student struggles to learn the content and provides support by decreasing the difficulty of the gameplay, hence decreasing the challenge. Once the student has demonstrated the mastering of the content, the scaffold is removed and the level of challenge is increased by allowing the student to progress to the next level of difficulty. The adaptive version of the Spanish cognates bubble game can be played at the URL: http://adaptivegame.cognatebubbles.com. The State Transition Diagram (STD) representing the adaptive difficulty algorithm is shown in the Appendix A. 4.2.2. Incremental difficulty adjustment version The incremental version is non-adaptive and increases the level of difficulty based on pre-defined intervals of time and does not decrease difficulty. The first 10 levels of difficulty each increase after one minute, and the last level finishes after two minutes. Time was chosen to adjust difficulty for two main reasons. Both games were programmed to finish after 13 min. Students were able to adjust how long they spent answering each question to some extent by making the bubbles move up faster. Thus, one way to ensure that students progressed through all the levels of difficulty in the incremental game, was to control the aspect of time rather than number of questions completed. Finally, a common feature of simple and short educational computer games is to increase difficulty based on time. The incremental version does not display a blue progress bar because it does not take into account the student’s responses to adjust difficulty, hence the game does not implement scaffolding because it does not recognise when students are struggling nor provide the necessary support. The incremental version of the Spanish cognates bubble game can be played at the URL: http://incrementalgame. cognatebubbles.com. 4.3. Written activity To justify the costs of the development and use of educational computer games, they should guarantee improved learning compared to the less expensive and easy-to-use traditional activities. Thus, the written activity was used to serve as a control group to study the effectiveness of educational computer games. The written activity consists of four pages, each with text instructions describing four rules with examples and 16 English words to translate to Spanish, for a total of 64 words. The Spanish endings are ordered from easy to more difficult, in the same way as the games. The written activity does not adjust difficulty and students could choose any order to solve it. A solution page was given to the students upon completion to provide feedback. The written activity was approved by the Languages Other Than English (LOTE) teachers from participating schools. A preliminary validation was conducted by interviewing two students who indicated that the written activity was clear, easy to understand, and feasible to be completed within a time frame of 20 min. Table 3 presents an example of the structure used for the written activity. Table 2 Levels of difficulty used in the game. Level of difficulty Curriculum categories and degree of difficulty 1 2 3 4 5 6 7 8 9 10 11

or, ción, al ble, sión, ente ante, ista, ismo, ico mento, ivo, oso, ar dad, ificar, or, al, ble sión, ción, ente, ante, ista ismo, ico, mento, ivo, oso, ar dad, ificar, or, al, ble, sion ción, ente, ante, ista, ismo, ico mento, ivo, oso, ar, dad, ificar or, ción, al, ble, sión, ente, ante, ista, ismo, ico, mento, ivo, oso, ar, dad, ificar

Number of boxes, bubbles Bubble speed easy and medium easy and medium medium medium and difficult difficult and easy easy and medium medium and difficult difficult and easy medium medium and difficult easy, medium and difficult

2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 6,

1 1 1 1 1 2 2 2 2 2 3

80 80 80 80 80 70 70 70 70 70 60

S. Sampayo-Vargas et al. / Computers & Education 69 (2013) 452–462

457

Table 3 Example of the written activity structure.

5. Empirical study The empirical study consisted of a quasi-experimental non-equivalent design. A control group was used to compare the outcomes of the game with an equivalent written activity. The hypotheses used to guide the study were the following: H1_motivation. The adaptive difficulty adjustment game will lead to significantly higher levels of motivation than the incremental difficulty adjustment game and the written activity. H2_learning. The adaptive difficulty adjustment game will lead to significantly higher learning outcomes than the incremental difficulty adjustment game and the written activity.

5.1. Participants A convenience sample was recruited based on voluntary participation following the Ethics procedure from La Trobe University and the Department of Education and Early Childhood Development. A total of 234 year 7 students were allocated to one of three groups. The nonequivalent allocation was based on the teacher’s class availability to use a computer laboratory or a classroom. The students allocated to play the games were randomly sub-divided to either play the adaptive or the incremental version of the Spanish cognates bubble game. A total of 83 students were allocated to the adaptive game group, 78 to the incremental game group and 73 to the written activity group. The mean age of the participants was 12 years old (from 11 to 16). There were a total of 133 males and 101 females distributed uniformly in the three groups. Regarding language proficiency, the students had a very limited knowledge of Spanish with 43% of the students knowing only a few words and 51% not knowing any Spanish at all. This data was obtained from the participant survey (see Section 5.2). The learning activities designed for the experiment were not part of the students’ coursework and they were not graded. In addition, the students did not receive any classes of Spanish inside or outside school hours. However, they were studying their first year of Italian classes as part of their LOTE subject. Italian and Spanish words can be similar but when the experiment was undertaken the students were only able to repeat a few Italian words with the help of their LOTE teacher. Thus, categorizing the students’ Italian proficiency according to the Common European Framework of Reference (CEFR)3 resulted not even in level A1 or Basic user. The level A1 requires students to understand, speak and write familiar everyday expressions, very basic phrases, introduce themselves and answer personal details. 5.2. Data collection Quantitative data was collected from a participant survey used to record the students’ background; pre- and post-tests and game log data files to measure learning outcomes; and an engagement survey to provide insights into motivation in relation to engagement, perceived competence and preferred level of difficulty. Quantitative data collection and the conclusion of the experiment occurred in three 45-min consecutive sessions, which took place in 2011 at two secondary government schools, during school hours. Depending on the teachers’ and classes’ schedules each session was consecutive or where necessary no more than 24 h apart. The first session was used to attend a lesson, complete a participant survey and respond to a pre-test. The lesson consisted of introducing the topic of Spanish cognates and was delivered using a power point presentation

3

http://www.cambridgeenglishteacher.org/what_is_this.

458

S. Sampayo-Vargas et al. / Computers & Education 69 (2013) 452–462

by the same instructor to all the participants. The second session involved allocating students to one learning activity, completing an engagement survey and responding to a post-test. The final session concluded the experiment by thanking and awarding all the participants with small prizes. Participant survey. The participant survey was used to find out if the students allocated to each group differed in background characteristics. The results of the participant survey showed no evidence of any statistically significant effect (p-value > 0.05) for the students’ background characteristics of age, gender, level of Spanish language, interest in learning Spanish, computer game playing skills, computer skills, reading and writing skills, ability to remember information, and perceived competence at school. This result of no significant effect indicated that students in the three groups shared similar characteristics such as: being proficient regular computer game players, proficient regular computer users, having pretty good reading and writing skills, having neutral skills to remember information and being pretty good at school. The participant survey and the descriptive results obtained by the three groups are shown in the Appendix B and Table B.1. The participants belonged to different socio-economic backgrounds and the only difference between them was that the incremental difficulty game group spoke more languages other than English than the adaptive difficulty game group. However, when language ability was included in the results for learning and motivation, this factor had little influence on the outcomes. The most common languages spoken by students as a second language were Macedonian, Arabic, Vietnamese and Lebanese. Pre- and post-tests. The pre- and post-tests were alternate forms of each other and were used to measure the learning of Spanish cognates. The tests were similar to those used in Madrigal (2001) and Cognates (2008) consisting of fill-in-the-blank items with no word bank provided. Each test contained 16 different English words to be translated into Spanish by applying the correct Spanish ending. Students needed to remember different categories of Spanish endings and understand how to apply them for a correct translation. LOTE teachers reviewed and approved the tests, which had a high internal reliability with a Cronbach’s alpha coefficient of 0.764 for the pre-test and 0.864 for the post-test. Engagement survey. The engagement survey was used to provide insights into motivation in relation to the factors of engagement, perceived competence and preferred level of difficulty. Engagement is often related to experiencing high levels of continuous intrinsic motivation because the activity has all the qualities to hold people’s interest, involvement and participation. Engagement was measured with an adapted version of the survey used in Whitton (2007, 2010), consisting of 18 questions using a 5-point Likert scale. Six of those questions were changed following the recommendations of LOTE teachers and an expert in survey design to better suit the experiment and to facilitate students’ understanding of the questions. Two extra questions and two statements included in the engagement survey were used to gather insights into perceived competence and preferred level of difficulty. Perceived competence relates to intrinsic motivation because high levels of motivation can be achieved if students perceive that they are competent and capable of accomplishing a task (Cordova, 1983; Malone, 1981). A preferred level of difficulty refers to the students’ perceptions of the level of difficulty that they want and is often related to their abilities. This indicates that providing students with their preferred level of difficulty could result in them experiencing an optimal level of challenge, with a high level of motivation (Andrade & Corruble, 2005). The engagement survey had a high internal consistency with a Cronbach’s alpha coefficient of 0.897, 0.750 and 0.860 reported on three different occasions by Whitton (2007). For this study, the modified engagement survey reported a high internal reliability with a Cronbach’s alpha coefficient of 0.892. The engagement survey and the descriptive results obtained by the three groups are shown in the Appendix C, Tables C.1 and C.2. 5.3. Method for data analysis H1_motivation was tested using the non-parametric statistical test of Kruskal–Wallis One Way Analysis of Variance, followed by Mann– Whitney U tests with a Bonferroni correction of 0.0167. These tests were used to compare the means of the score obtained in the engagement survey for the three groups: adaptive difficulty adjustment game, incremental difficulty adjustment game and written activity. The score was calculated by adding the 5-point Likert scale responses. H2_learning was tested using a one-factor analysis of variance (ANOVA) and linear contrasts. This test was used to compare the mean of the score obtained in the pre-test against that achieved in the post-test for the three groups. A correct response in the tests was allocated 1 point and an incorrect response 0. A total score per test and per student was calculated by adding the points of each response. H2_learning was also investigated using the game log data which consisted of recording each response given by each student during gameplay. A correct response in the game was recorded with 1 point and an incorrect response with 0. The data was tested using a binary logistic regression with game type and curriculum category difficulty as factors, and level of difficulty as a covariate. The software SPSS was used to analyse and interpret the data. The hypotheses were tested at a significance level of p-value ¼ 0.05, and with a Bonferroni correction of p-value ¼ 0.0167 when specified. The results tested at a significance level of 0.0167 (¼0.05/3) were used to minimise the Type 1 error rate from the execution of various non-parametric tests to compare the three learning activities (Field, 2009). The evidence used to support the hypotheses was classified as follows:  Weak evidence, if p-value is between 0.05 and 0.10  Strong evidence, if p-value is between 0.01 and 0.05  Very strong evidence, if p-value is less than 0.01

5.4. Research rigour This research attempted to isolate the game element of difficulty adjustment. To do so, the same game was developed twice with the only difference being the method used to adjust difficulty. The game design followed the same set of guidelines with the purpose of isolating difficulty adjustment. The guidelines were characterised by the design of other game elements in their most simplistic form. The scope of the game chosen was characterised by being a simple, gender-neutral and short educational computer game of the puzzle genre type. This

S. Sampayo-Vargas et al. / Computers & Education 69 (2013) 452–462

459

scope also facilitated the study of one game element. The experiment was blind, consisting of the evaluation of three equivalent learning activities where students were only aware of the learning activity allocated to them. To achieve this, different classes were allocated to either play a game or complete a written activity (instead of allocating individual students from the same class to different activities), and the two versions of the game looked and behaved similarly. 6. Results This section presents the results of the statistical tests used to investigate the hypotheses. The discussion of the interpretation of the results is presented in Section 7. 6.1. Motivation H1_motivation failed to be supported because no statistically significant difference was found favouring the adaptive difficulty adjustment game over the other two learning activities with regard to motivation. The three learning activities compared scored equal levels of engagement, both games scored equal levels of perceived competence and students who played the adaptive difficulty adjustment game preferred more difficulty. The factor of engagement indicated no statistically significant differences between the three groups (p-value > 0.05). In the engagement survey, a score of 54 points indicates an above average level of engagement. The results from the comparisons were 62.94 points for the adaptive difficulty adjustment game, 62.62 points for the incremental difficulty adjustment game and 62.93 points for the written activity. These results indicated that students were equally highly motivated in relation to engagement after using any of the three learning activities. The factor of perceived competence indicated that both games resulted in statistically significantly higher levels of perceived competence than the written activity (p-value < 0.0167). However, there was no statistically significant difference between the games. These results indicated that students were equally highly motivated in relation to perceived competence after playing either version of the game. The factor of preferred level of difficulty indicated that both the incremental difficulty adjustment game and the written activity provided a more preferred level of difficulty than the adaptive difficulty adjustment game (p-value < 0.0167). There was no statistically significant difference between the incremental game and the written activity groups. The results for the adaptive game group indicated that students preferred to have more difficulty which could be translated to students perceiving the adaptive game as easier than the other two learning activities. However, the results of preferred level of difficulty are inconclusive as the questions did not specify whether students’ responses referred to the whole game, different stages of the game, difficulty of the gameplay or difficulty of the educational content. Thus, future research is necessary to expand on these findings. 6.2. Learning outcomes H2_learning was supported because the adaptive difficulty adjustment game led to statistically significantly higher learning outcomes than the other two learning activities, with a p-value ¼ 0.000. The results of the pre- and post-tests indicated that students who played the adaptive difficulty adjustment game were able to learn a quantitatively higher number of Spanish endings than students who played the incremental difficulty adjustment game or completed the written activity (p-value ¼ 0.000). The improvement in score was calculated from the mean differences between the pre- and post-tests scores. The results of the mean differences between the pre- and post-tests are shown in Fig. 2, indicating that students who played the adaptive difficulty adjustment game learnt on average 7 Spanish endings after playing the game. H2_learning was also supported from the results of the game log data. This data indicated that the adaptive difficulty game group achieved a statistically significantly higher number of correct responses than the incremental difficulty game group (p-value ¼ 0.000). The analysis of the game log data considered level of difficulty as a covariate. This accounted for any differences in the data regarding the last level of difficulty reached by each game.

Fig. 2. Mean differences between the pre and post-test.

460

S. Sampayo-Vargas et al. / Computers & Education 69 (2013) 452–462

The game log data results for correct responses per level of difficulty are shown in Fig. 3, indicating that the adaptive game group maintained a steady progress for the achievement of correct responses through the levels of difficulty 1 – 9. In contrast, in the incremental difficulty adjustment game as the level of difficulty increased the number of correct responses decreased and this effect was more evident with difficult educational content. Due to the time constraint given to play the games, students playing the adaptive difficulty adjustment game were not able to reach the most difficult levels 10 and 11. The levels of difficulty in the game are composed of both the difficulty of the gameplay and the difficulty of the educational content. In general as the level of difficulty increases, both the difficulty of the gameplay and educational content increase. To ensure mastering of the educational content the difficulty of the educational content increases at a slow rate and sometimes easier content is revisited at higher levels (see Fig. 3). 7. Discussion and conclusions 7.1. Motivation The finding of no statistically significant differences for motivation is consistent with research that has compared equivalent game-based learning activities (Orvis et al., 2008; Whitton, 2007). All of the learning activities compared in this study had an equivalent design. Both games were designed following the same set of guidelines and the written activity was designed following equivalent learning theories, educational objectives and content. Both games provided immediate feedback for responses, whereas the written activity provided feedback at completion with a solution page. This may have influenced the results for perceived competence favouring both games over the written activity. It seems that the design of the games in this study, which was equivalent for both versions of the game, had a greater impact on motivation than the method used to adjust difficulty. In the domain of entertainment computer games, adaptive difficulty adjustment has been favoured over non-adaptive difficulty for player’s satisfaction (Andrade et al., 2006). This is true, when the game is capable of providing an optimal level of challenge. The adaptive difficulty game of this study was not rated by students as providing a preferred level of difficulty. Thus, the challenges provided in the adaptive difficulty game were not enough for students to experience an optimal level of challenge which could have impacted on the results for motivation. Nevertheless, the three learning activities were rated by students with a high score for motivation in relation to engagement. 7.2. Learning outcomes Based on the test results the three activities helped students to learn. However, the adaptive game group achieved statistically significantly higher learning outcomes than the incremental game or written activity groups. The difference between both games was the method used to adjust difficulty. This indicates that the adaptive difficulty adjustment had an impact on the achievement of higher learning outcomes. An analysis of the game log data presented in Fig. 3 indicated that the adaptive difficulty adjustment game ensured the learning of basic content before allowing progression to more difficult material. The educational content was ordered from easy to more difficult which was

Fig. 3. Game log data results for correct responses.

S. Sampayo-Vargas et al. / Computers & Education 69 (2013) 452–462

461

incorporated into each level of difficulty. As the levels of difficulty increased, the difficulty and quantity of the educational content also increased. This made the game difficult to play and harder for students to concentrate on content that had not been learnt. To ensure learning, the adaptive difficulty game decreased the gameplay difficulty when students were struggling and did not allow students to progress to the next level of difficulty until all the content in the current level was mastered. The game provided a scaffold for student learning by decreasing the gameplay difficulty if students had problems with the content, and the scaffold was removed when the game progressed to the next level. The game log data results indicated that students who played the adaptive difficulty adjustment game experienced a more suitable level of challenge, with an appropriate time to solve each challenge, according to their abilities. Combining the results of motivation in relation to preferred level of difficulty, it appears that there were two types of perceptions of difficulty. One was the preferred difficulty based on what the student would like to have, and the other was the actual difficulty based on their performance (Alexander, Sear, & Oikonomou, 2013). This study indicates that the difficulty of what students would like to have does not ensure the achievement of higher learning outcomes. Students perceived that the adaptive difficulty game was not difficult enough, and yet their learning was significantly better. However, the adaptive difficulty game allowed students to comfortably learn the content in their own time, at the expense of not being able to finish the game in the time allocated for experimental purposes. To expand on these results equivalent research should be conducted by allocating more time and using other educational topics. 7.3. Conclusions This study contributes towards an understanding of the effectiveness of adaptive difficulty adjustment in educational computer games. The educational computer games used in this study were motivational and fun to play, however this was not sufficient to ensure high learning outcomes. The game incorporating adaptive difficulty adjustment produced significantly higher learning outcomes than the equivalent game with incremental difficulty adjustment and the written activity. The game log data measuring the correctness of students’ responses over time clearly indicated that the students playing the adaptive difficulty adjustment game were scaffolded in their learning by timely reductions in the gameplay difficulty. The game log data also indicated that students playing the adaptive difficulty adjustment game required extra time to reach the highest levels of difficulty. The study provides positive encouragement for paying the extra cost and development time required for educational computer games with adaptive difficulty adjustment. On top of the motivational aspect of the game higher learning outcomes were evident. The game also has the capability to provide a suitable level of challenge and scaffold the learning of students at different ability levels and working at different paces. 8. Limitations of the study and future research The limitations of this study can be seen as directions for future research. Firstly, it is important to evaluate other topics using equivalent experimental settings used in this study. The game was characterised as a simple puzzle-genre type, thus other game genres could be evaluated to expand the results across different types of games. Another area of future research is to use different experimental settings such as a randomised sample, different participants, different geographical regions, providing more time to play the games and using a larger sample. An interesting area of future research would be to ask students to play the games and to solve the written activity several times. This can be followed by students responding to the post-test each time to compare their results after each session. This study measured the students’ short-term memory, thus, it will be interesting to analyse the impact of each learning activity on students’ long-term memory. This can be done by students responding to the post-test, at least twice. For example, the first time immediately after completing the activity and the second time four weeks after. However, it will be necessary to carefully factor in the possibility that the students may practice the topic with other activities during the time apart. It will also be beneficial to study the effectiveness of other adaptive difficulty algorithms which could be feasible, are low-cost and capable of ensuring not only higher learning outcomes but also higher motivation than an equivalent game with non-adaptive difficulty adjustment. Finally, the quantitative results can be followed up with a strong emphasis on qualitative research. Acknowledgements This research was supported by the National Council for Science and Technology and the Mexican government. Thanks to all of the school principals, students, teachers and staff who participated in the study. Appendix A. Supplementary data Supplementary data related to this article can be found at http://dx.doi.org/10.1016/j.compedu.2013.07.004. References Alexander, J. T., Sear, J., & Oikonomou, A. (2013). An investigation of the effects of game difficulty on player enjoyment. Entertainment Computing, 4(1), 53–62. Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., et al. (2001). A taxonomy for learning teaching and assessing. A revision of Bloom’s taxonomy of educational objectives (Complete ed.). Addison Wesley Longman, Inc. Andrade, G., & Corruble, V. (2005). Challenge-sensitive action selection: an application to game balancing. In IEEE/WIC/ACM international conference on intelligent agent technology (IAT) (pp. 194–200). Andrade, G., Ramalho, G., Gomes, A. S., & Corruble, V. (2006). Dynamic game balancing: an evaluation of user satisfaction. In AAAI conference on artificial intelligence and interactive digital entertainment (pp. 3–8). Aponte, M. V., Levieux, G., & Natkin, S. (2011). Measuring the level of difficulty in single player video games. Entertainment Computing, 2(4), 205–213. Bauer, K. N., Brusso, R. C., & Orvis, K. A. (2012). Using adaptive difficulty to optimize videogame-based training performance: the moderating role of personality. Military Psychology, 24(2), 148–165. Berns, A., Gonzalez-Pardo, A., & Camacho, D. (2013). Game-like language learning in 3-D virtual environments. Computers and Education, 60, 210–220.

462

S. Sampayo-Vargas et al. / Computers & Education 69 (2013) 452–462

Brom, C., Preuss, M., & Klement, D. (2011). Are educational computer micro-games engaging and effective for knowledge acquisition at high-schools? A quasi-experimental study. Computers and Education, 57, 1971–1988. Chang, K. E., Sung, Y. T., & Chen, S. F. (2001). Learning through computer-based concept mapping with scaffolding aid. Journal of Computer Assisted Learning, 17, 21–33. Chang, M.-Y., Wernhuar, T., & Shin, F.-Y. (2009). The effectiveness of scaffolding in a web-based adaptive learning system. International Journal of Web-Based Learning and Teaching Technologies, 4(1), 1–15. Cognates, S. (2008). Spanish English cognates. 13235 Corte Villanueva 101, San Diego CA 92129: ESDICT.com. Colvin, R. C., Nguyen, F., & Sweller, J. (2006). Efficiency in learning: Evidence-based guidelines. 989 Market Street, San Francisco, CA: Pfeiffer. Conati, C., & Manske, M. (2009). Evaluating adaptive feedback in an educational computer game. In Intelligent virtual agents 9th international conference, September, pp. 14–16. The Netherlands, Amsterdam. Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers and Education, 59(2), 661–686. Cordova, D. I. (1983). The effects of personalization and choice on students’ intrinsic motivation and learning. PhD thesis. Department of Psychology. Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. Harper Perennial. Modern Classics. 10 East 53rd Street, New York, NY 10022. Egenfeldt-Nielsen, S. (2007). Third generation educational use of computer games. Journal of Educational Multimedia and Hypermedia, 16(3), 263–281. Field, A. (2009). Discovering statistics using SPSS (3rd ed.). 1 Oliver’s Yard, 55 City Road, London EC1Y 1SP: SAGE. Gobel, S., Wendel, V., Ritter, C., & Steinmetz, R. (2010). Personalized, adaptive digital educational games using narrative game-based learning objects. In X. Zhang, S. Zhong, Z. Pan, K. Wong, & R. Yun (Eds.), Entertainment for education. Digital techniques and systems, Vol. 6249 (pp. 438–445). Berlin/Heidelberg: Springer. Goldstone, R. L. (1998). Perceptual learning. Annual Review of Psychology, 49, 585–612. Hays, R. T. (2005). The effectiveness of instructional games: A literature review and discussion. Orlando, Florida: Naval Air Warfare Center Training Systems Division. Hodhod, R. (2010). Interactive narrative for adaptive educational games: Architecture and an application to character education. PhD thesis. The University Of York. Hsieh, H.-M., & Wang, L.-L. (2008). A fuzzy approach to generating adaptive opponents in the dead end game. Asian Journal of Health and Information Sciences, 3(1–4), 19–37. Hua, Q., Pei-Luen, P. R., & Gavriel, S. (2010). Effects of different scenarios of game difficulty on player immersion. Interacting with Computers, 22(3), 230–239. Johnson, W. L., Vilhjalmsson, H., & Marsella, S. (2005). Serious games for language learning: how much game, how much A.I.?. In 12th International conference on artificial intelligence in education (AIED), July. Amsterdam. Kickmeier-Rust, M. D., Marte, B., Linek, S., Lalonde, T., & Albert, D. (2008). The effects of individualized feedback in digital educational games. In Proceedings of the 2nd European conference on games-based learning (pp. 227–236). Academic Publishing Limited. Kickmeier-Rust, M. D., Peirce, N., Conlan, O., Schwarz, D., Verpoorten, D., & Albert, D. (2007). Immersive digital games: the interfaces for next-generation e-learning?. In C. Stephanidis (Ed.), Universal access in human–computer interaction. Applications and services, Vol. 4556 (pp. 647–656) Berlin/Heidelberg: Springer. Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: an analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86. Klinkenberg, S., Straatemeier, M., & Van Der Maas, H. L. (2011). Computer adaptive practice of maths ability using a new item response model for on the fly ability and difficulty estimation. Computers and Education, 57(2), 1813–1824. Koidl, K., Mehm, F., Hampson, C., Conlan, O., & Gobel, S. (2010). Dynamically adjusting digital educational games towards learning objectives. In Conference on games-based learning. Copenhagen, Denmark: Danish School of Education Aarhus University. Law, E., Kickmeier-Rust, M. D., Albert, D., & Holzinger, A. (2008). Challenges in the development and evaluation of immersive digital educational games. In A. Holzinger (Ed.), HCI and usability for education and work, Vol. 5298 (pp. 19–30). Berlin/Heidelberg: Springer. Madrigal, M. (2001). Madrigal’s magic key to Spanish. A creative and proven approach. 1540 Broadway, New York, NY 10036: Broadway Books/New York. Malone, T. W. (1981). Towards a theory of intrinsically motivating instruction. Cognitive Science, 5(4), 333–369. Malone, T. W., & Lepper, M. R. (1987). Making learning fun: a taxonomy of intrinsic motivations for learning. Aptitude Learning and Instruction, 3(3), 223–253. Moser, R. B. (2000). A methodology for the design of educational computer adventure games. PhD thesis. University of New South Wales. Murphy, N., & Messer, D. (2000). Differential benefits from scaffolding and children working alone. Educational Psychology, 20(1). Nicholls, J. G., & Miller, A. T. (1983). The differentiation of the concepts of difficulty and ability. Child Development, 54(4), 951–959. Olson, J., & Platt, J. (2000). The instructional cycle. Teaching children and adolescents with special needs. Upper Saddle River, NJ: Prentice-Hall, Inc. Orvis, K. A., Horn, D. B., & Belanich, J. (2008). The roles of task difficulty and prior videogame experience on performance and motivation in instructional videogames. Computers in Human Behavior, 24(5), 2415–2433. Peirce, N., Conlan, O., & Wade, V. (2008). Adaptive educational games: providing non-invasive personalised learning experiences. In Second IEEE international conference on digital games and intelligent toys based education, November (pp. 28–35). Reinders, H. (2012). Digital games in language learning and teaching. Houndmills, Basingstoke, Hampshire RG21 6XS: Palgrave Macmillan. Rodgers, A., & Rodgers, E. M. (2004). Scaffolding literacy instruction: Strategies for K-4 classrooms. 361 Hanover Street, Portsmouth, NH: Heinemann. Rodriguez, S. D., Cheng, I., & Basu, A. (2007). Multimedia games for learning and testing physics. In International conference on multimedia & expo (ICME) (pp. 1838–1841). Sampayo-Vargas, S. (2012). The effectiveness of adaptive difficulty adjustments in educational computer games. PhD thesis. La Trobe University. Spronck, P., Ponsen, M., Sprinkhuizen-Kuyper, I., & Postma, E. (2006). Adaptive game AI with dynamic scripting. Springer Science, 63, 217–248. Squire, K. D. (2005). Changing the game: what happens when video games enter the classroom? Innovate. Journal of Online Education, 1(6). Sweetser, P., & Wyeth, P. (2005). GameFlow: a model for evaluating player enjoyment in games. Computers in Entertainment, 3(3), 3A. Van Der Stuyf, R. R. (2012). Scaffolding as a teaching strategy. Adolescent learning and development. Section 0500A. Vygotsky, L. (1978). Mind in society: The development of higher psychological processes. Harvard Univ. Press. Whitton, N. J. (2007). An investigation into the potential of collaborative computer game-based learning in higher education. PhD thesis. Napier University. Whitton, N. J. (2010). Learning with digital games: A practical guide to engaging students in higher education. 270 Madison Ave, New York, NY 10016: Routledge. Wilson, A. J., Revkin, S. K., Cohen, D., Cohen, L., & Dehaene, S. (2006). An open trial assessment of ‘the number race’, an adaptive computer game for remediation of dyscalculia. Behavioral and Brain Functions, 20(2).