- Email: [email protected]
Enhancing the Attractiveness of Learning through Augmented Reality
Available online at www.sciencedirect.com Available online at www.sciencedirect.com
ScienceDirect ScienceDirect
Available online at www.sciencedirect.com Available online at www.sciencedirect.com Procedia Computer Science 00 (2018) 000–000 Procedia Computer Science 00 (2018) 000–000
ScienceDirect ScienceDirect
Procedia Computer Science 126 (2018) 166–175 Procedia Computer Science 00 (2018) 000–000
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
International International Conference Conference on on Knowledge Knowledge Based Based and and Intelligent Intelligent Information Information and and Engineering Engineering Systems, KES2018, 3-5 September 2018, Belgrade, Serbia Systems, KES2018, 3-5 September 2018, Belgrade, Serbia International Conference on Knowledge Based and Intelligent Information and Engineering Enhancing the of through Augmented KES2018, 3-5 September 2018, Belgrade, Serbia EnhancingSystems, the Attractiveness Attractiveness of Learning Learning through Augmented
Reality Reality Enhancing the Attractiveness of Learning through Augmented Adrian Diana Adrian Iftene, Iftene, Diana Trandabăț Trandabăț Reality Faculty of Computer Science, Alexandru Ioan Cuza University, General Berthelot, 16, Iasi 700483, Romania Faculty of Computer Science, Alexandru Ioan Cuza University, General Berthelot, 16, Iasi 700483, Romania
Adrian Iftene, Diana Trandabăț
Abstract Abstract
Faculty of Computer Science, Alexandru Ioan Cuza University, General Berthelot, 16, Iasi 700483, Romania
Over the last years, augmented reality was used in various domains, from medical, industrial design, modeling and production, Over the last years, augmented reality was used in various domains, from medical, industrial design, modeling and production, robot teleoperation, military, entertainment, leisure activities to translation, facial recognition, assistance while driving, interior Abstract robot teleoperation, military, entertainment, leisure activities to translation, facial recognition, assistance while driving, interior and exterior design, virtual friends, internet of things and eLearning. In eLearning, the combination between classical and and exterior design, virtual friends, internet of things and eLearning. In eLearning, the combination between classical and augmented content later coming with models, images, sounds,from animations, Internet browsing, can help teacher to Over the last years,(the augmented reality was3D in various domains, medical, industrial design,etc.) modeling andthe augmented content (the later coming with 3Dused models, images, sounds, animations, Internet browsing, etc.) can help theproduction, teacher to better teleoperation, explain the content of the courses. In leisure this paper, we present four augmented reality applications, with theinterior aim to robot military, entertainment, activities to translation, facial recognition, assistancecreated while driving, better explain the content of the courses. In this paper, we present four augmented reality applications, created with the aim to improve communication and collaboration skills (two of them) and to ease the learning of biology and geography (the other two). and exterior design, virtual friends, internet of(two things and eLearning. combination between classical and improve communication and collaboration skills of them) and to easeIntheeLearning, learning ofthe biology and geography (the other two). The motivation behind applications enhanceimages, the attractiveness of the classes, allow studentsetc.) to retrain newthe information augmented content (thethese later coming with is 3Dto sounds, animations, Internet browsing, can help teacher to The motivation behind these applications is tomodels, enhance the attractiveness of the classes, allow students to retrain new information more andthe reduce theof stress behind tests when presented as games. bettereasily explain content the courses. In this paper, we present four augmented reality applications, created with the aim to more easily and reduce the stress behind tests when presented as games. improve communication and collaboration skills (two of them) and to ease the learning of biology and geography (the other two). © 2018 The Authors. Published by Elsevier Ltd. Keywords: Augmented Reality; eLearning; Autism; Usability Testing The behind these eLearning; applications to Usability enhance Testing the attractiveness of the classes, allow students to retrain new information Keywords: Reality; Thismotivation is an Augmented open access article under theAutism; CCisBY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) more easily and reduce theunder stressresponsibility behind tests when presented as games. Selection and peer-review of KES International.
1. Introduction Keywords: Augmented Reality; eLearning; Autism; Usability Testing 1. Introduction Augmented Reality (AR) improves the perception of the user, helps him to better understand the surrounding Augmented Reality (AR) improves the perception of the user, helps him to better understand the surrounding 1. Introduction reality and interact with the real world. Virtual objects display information that user cannot detect directly with their reality and interact with the real world. Virtual objects display information that user cannot detect directly with their own senses. The information transmitted by virtual objects can help the user in performing his tasks. AR is a specific own senses. The information transmitted by virtual objects can help the user in performing his tasks. AR is a specific Augmented (AR) improves the perception of the (IA): user, using helpsthe himcomputer to betterasunderstand the surrounding example of whatReality Fred Brooks calls intelligence amplification a tool to make it easier for example of what Fred Brooks calls intelligence amplification (IA): using the computer as a tool to make it easier for and interact areality human being [1]. with the real world. Virtual objects display information that user cannot detect directly with their a human being [1]. own information transmitted virtual objects can help the usercontinuum, in performing his tasks. AR is by a specific Insenses. 2010, The in [2], the authors presentedbythe notion of reality-virtuality originally defined [3], in In 2010, in [2], the authors presented the notion of reality-virtuality continuum, originally defined by [3], in example ofiswhat Fred calls realm intelligence amplification theenvironments computer as (i.e. a toolvirtual to make it easier for which AR a part of Brooks the general of combined reality.(IA): Bothusing virtual realities - VR) which AR is a part of the general realm of combined reality. Both virtual environments (i.e. virtual realities - VR) a human being [1]. and augmented virtualization, in which real objects are added to virtual ones, replace the environment with a virtual and augmented virtualization, in which real objects are added to virtual ones, replace the environment with a virtual In 2010, inIn[2], the authors presented the notion reality-virtuality continuum, originally defined [3],user in environment. contrast, AR provides a local virtual of reality. When considering not only artificiality butby also environment. In contrast, AR provides a local virtual reality. When considering not only artificiality but also user which ARitisclassifies a part ofAR theasgeneral realm of combined reality. Both virtual[4]. environments (i.e. virtual realities - VR) transport, being separated by both VR and telepresence transport, it classifies AR as being separated by both VR and telepresence [4]. andThe augmented virtualization, in intensive which realuse objects added to virtual replace environment a virtual most important areas of have are been described for ones, the first time the in [5], and later with in [6]. Thus, The most important areas of intensive use have been described for the first time in [5], and later in [6]. Thus, environment. In contrast, provides a local virtual reality. in When considering not only but also user doctors can use AR as a AR visual aid and training instrument performing surgeries [5].artificiality Canon’s Mixed Reality doctors can use AR as a visual aid and training instrument in performing surgeries [5]. Canon’s Mixed Reality transport, it classifies AR asthe being separated byby both VR and [4]. System (MREAL) supports design process allowing 3Dtelepresence computer generated models to be combined with realSystem (MREAL) supports the design process by allowing 3D computer generated models to be combined with realTheobjects most important areascategory of intensive use have reality been described for is thethe first time in [5], and later and in [6]. Thus, world [7]. Another of augmented applications assembly, maintenance repair of world objects [7]. Another category of augmented reality applications is the assembly, maintenance and repair of doctors use AR[8]. as AR a visual and training instrument surgeries [5]. Canon’s Mixed Reality complexcan machines couldaid be used to annotate objects in andperforming environments with public or private information complex machines [8]. AR could be used to annotate objects and environments with public or private information System (MREAL) supports the design process by allowing 3D computer generated models to be combined with real1877-0509 2018[7]. The Authors. Published by Elsevier world objects category of augmented reality applications is the assembly, maintenance and repair of 1877-0509 2018 The Another Authors. Published by Elsevier B.V. Peer-review under responsibility of KES International B.V. Peer-review under responsibility of KES complex machines [8]. AR could be International used to annotate objects and environments with public or private information 1877-0509 Published by Elsevier 1877-0509 © 2018 2018 The TheAuthors. Authors. Published by Elsevier Ltd. B.V. under responsibility KES This Peer-review is an open access article underofthe CCInternational BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Selection and peer-review under responsibility of KES International. 10.1016/j.procs.2018.07.220
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
167
[9]. For many years, military airplanes and helicopters have used HUDs (Head-Up Displays) and HMS (HelmetMounted Sights) to overlay vector graphics over the images that the pilot had of the real world [10]. Recon Jet [11] is an AR system already available for recreational activities. One of the most promising areas of application in the AR is the translation field [12]. An example of easy access to information from the Internet in real-life situations is the combination of face detection with AR [13]. Nowadays, many car manufacturers have included on-board information displayed on the windscreen [14], such as speed, direction of movement based on real-world object recognition, parking assistance, sensor information, and warnings related to traffic, warnings related to possible collisions, etc. One of the most suggestive examples of using AR in interior design is the most recent Ikea catalog using AR [15]. A hacker from Japan used an available 3D model and motion sensors to have an “AR meeting” with a famous Japanese cartoon star (Hatsune Miku [16]). An example of a gesture interface system is SixthSense [17], developed by MIT. 2. Related Work - Augmented Reality in eLearning As the authors of 2010 Horizon Report [18] assert, the augmented reality has a huge potential to provide a useful context for education, allowing learning and discovery experiences connected to real world information. Mixed realities have been used in education for a long time [19]: the augmentation of wall paintings in caves is an approach to the transfer of knowledge about hunting and survival; Heilig’s patent for Sensorama describes the need for a solution for teaching, training and educating people in armed forces, industries and schools [20]; Ivan Sutherland saw the augmented image as a solution to give the user a “chance to get acquainted with concepts that cannot be achieved in the physical world” [21]; and the pioneering activity of Caudell and Mizel in Boeing’s augmented reality were designed to teach workers how to assemble complex components into aircraft [22]. Large-scale applications for AR in education were impossible before recent releases of cheap and affordable smartphones and tablets and before the emergence of software that allows the development and experimentation of real world augmentation. Opportunities to exploit AR technologies in education are now growing, while mobile hardware is proliferating, and access to the internet becomes universal. Among the first usage of augmented reality in education are applications in industrial training; surgical assistance; maintenance of equipment’s; service and car repair; behavioral changes; architecture, urban and environmental education [23]. Applying AR in an educational context equals to using “technology to add virtual objects to real scenes, by adding missing information to real life” [24]. As one can see, the emphasis is on providing additional information, such as a type that may be missing or inaccessible to students in the real world. Teaching anatomy, which requires a considerable amount of effort, expertise and temporal resources, is an example where AR can be used effectively to provide additional information [25]. In artificial intelligence augmented reality can be used to show what are the steps executed by an algorithm based on a Greedy paradigm [26]. Our ARBio and GeoAR game are another example of AR applications. One of the AR’s most significant features in terms of pedagogy is that it offers a student-oriented and flexible space to provide learning opportunities. Learning is taken away from traditional spaces, such as classrooms, amphitheaters and labs, and follows the students wherever they are. Learning opportunities can be present, for example, at home, at the workplace, in public transportation – and can be taken everywhere or can be transferred to anyone. As the AR develops more about interactive applications, students can become critics and co-creators, leaving behind a record of their learning the specific artifact or the place they have met [27]. AR does not come as a revolution: AR will not replace the existing pedagogical paradigm with a new world based on high technology. Instead, the pedagogical past is a rich resource for the future. The work of Mishra and Koehler [28] highlighted the importance of understanding the nuances between content, pedagogy and technology in designing learning environments. The model they propose, TPACK, highlights the dynamic interaction between these three areas. In this paper, we will present four AR applications developed for different types of children: (1) children with autism, (2) kindergarten children, (3) middle-aged children and (4) high school children. Our main goal is to see how AR applications are accepted by children, and what impact they have on children as well as on teachers.
168
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
3. Proposed solutions 3.1 Developing Communication Skills and Teamwork More and more emphasis is now being put on improving and developing soft skills, such as communication skills and teamwork, in children and adults of all ages. A method often used by teachers in schools is to involve pupils in different kinds of projects, where each member needs to make their own contribution to achieving the final goals. Typically, such projects aim to improve verbal and visual communication, but there are situations, as in the case of autistic children, where indirect communication through games can represent a breakthrough in the behavior of these children with special needs. Within our group, we implemented two games with network game components: (1) the first for autistic children, in order to develop attention, guidance and communication, in which they gain bonuses when they collaborate during the game to achieve a common goal, each being encouraged to follow the actions of others and to actively intervene in the game, and (2) the second one is to synchronize players’ activities, by verbal or visual instruction. Supporting Ships in the Air The purpose of this first game is to fuel more ships from the flight to allow them to carry out their missions. Once the missions at a specific level of the game are completed, the user goes to next level, to discover other missions, and so on until the final level is reached. In multiplayer mode, fueling is made easier and faster when two or more players fuel up the same ship, allowing the accumulation of a higher number of points and thus moving faster to next levels. Non-collaborating players stay more at a level and arrive harder at higher levels, while players who collaborate perform their tasks more quickly and manage to get faster to the end of the game. Figure 1 shows a sequence of the game level having only one ship to fuel.
Fig. 1. One of the Three Ships to be fueled
The application architecture is based on the client-server model. Mobile users are customers who communicate with the server through the UDP protocol to have the highest and most efficient speed. The client, implemented in Android, presents all the graphic elements, contains the augmented reality elements, made using the Metaio framework [29] and ensures communication with the user. The server, deployed in Java, generates the necessary data for a game level, ensures communication between users, and manages the information needed to unfold the current level, before storing in a database the score for the current user and global statistics.
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
169
With the help of four volunteers, usability tests have been made and the game was constantly improved. They proposed that switching from one level to another to involve an increasing in the number of ships, in the distance between ships and the user, rotating them, moving faster so that their fueling becomes more challenging. Also, a time counter has been introduced, which provides bonuses when the level is completed faster. In order to observe the effect of the application on autistic children, we contacted an expert, the observations made by him being positive. Children who suffer from this disease prefer to communicate with images. The ability to understand and think is enhanced when they receive visual stimulation. Multiplayer mode fits best for them. The emphasis in their case is on focusing on the environment and the detection of color-changing visuals. The radar provides the concrete position of the fueled ship, which develops the attention. Game partners will realize that they have to work together to fuel the ship efficiently. In conclusion, this direction of development shows promising first results. Hunting Treasure In the second game, the Sphero 2.0 [30] robotic ball is used, and two players can control it through the same robot controls (a first version of this application is presented in [31]). Depending on the play session and the task it has to fulfill, a player can influence a specific direction (i.e. up/down or right/left) and the speed of the robot. In addition, the surface on which the robot moves is augmented on a tablet or phone, so that each player looks at the scene of the game through the camera of the device he is using. The main objective here is that players work together to accomplish different tasks: collecting objects, solving puzzles, driving on augmented tracks, going through walls or gates, escaping from labyrinths, and so on. To achieve a goal, the game requires collaboration: augmented objects are distributed between players, so they will often have to communicate to drive Sphero in the right direction. Figure 2 shows the multiplayer option and the way the two players crawl throughout the game.
Fig. 2. Hunting treasure – Multiplayer option
To control Sphero, we have built a specific Android activity that simulates a TCP server. When this activity is started, the Bluetooth connection is checked, to turn on and discover Sphero devices. Once Sphero is discovered, we connect to it and start a separate thread running with the TCP server in standby mode. The multi-player option has been implemented with a Client-Server architecture, where the server is started by one of the players (the host). Then, for each player who connects to the server (including the host), a client is created. For the augmented reality component we used the Vuforia platform [32]. From this platform, we used two components, Smart Terrain and Image trackers, which allowed adding augmented content over Sphero’s workspace.
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
170
In order to have a stable location for game objects, Smart Terrain offers the ability to set a central point. For this we have chosen as a marker a feature-rich image, so that it can be easily recognized by the Vuforia engine. To improve the quality of the app, we have tested usability with four volunteers. We analyzed and summarized the qualitative and quantitative data, as well as the redesign recommendations that were made. In conclusions, the volunteers signaled in initial versions of the application, both (1) negative elements: the ambiguity of the interface, the multi-player component had problems of performance and stability, (2) and positive elements: Sphero was easier to control than expected, and interaction elements between players, 3D models, and game dynamics were most appreciated by the volunteers. 3.2 Lessons for the Primary and Secondary Cycles Apps for primary and secondary classes help students acquire information from biology, geography, history, and astronomy. Here are two of our applications, for biology and geography, two others for history and astronomy being under construction. ARBio Application ARBio is an application that uses augmented reality to respond to the curiosities of children in primary school, but not only. With the help of the application, the user lives the experience of augmented reality by enriching the reality of the images of some animals with their three-dimensional representation and with onomatopoeias produced by animals. The user is also able to browse the list of animals and read information about each one. ARBio is a modern learning method and an interesting experience for a user of any age (see Figure 3).
Fig. 3. (left) ARBio - the view is based on the marker (right) the application allows simultaneous viewing of several animals
The main modules of the application are: the augmented reality module and the module for viewing and reading animal information, including local marker downloading. The augmented reality module is the most important and complex part of the application because it augments the reality of the environment by adding virtual elements. This module was made using the Artoolkit framework [33]. From the implementation point of view, the logic of the augmented reality module is divided into two main components. The first component uses C++ programming language. This module contains methods from the
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
171
ARWrapper library to initialize and manipulate the augmented reality section of the application. This module is basically the link between the Artoolkit framework and the ARBio application. The second component containing the augmented reality logic uses the Java programming language and manages objects (patterns, markers, sound files) in data structures that it sends to the other component by JNI (Java Native Interface [34]) method calls. Therefore, the communication between the two components is made by native calls of predefined methods. The first step in achieving the augmentation of reality was to create markers. For this application, we only used markers consisting of square templates. Each marker, pre-trained in the application, corresponds to a virtual model so that whenever a marker appears in the camera view, the reality will be augmented by adding the appropriate animal. The markers are made up of a black border and the inside of the marker is a suggestive image of the animal with which the marker is correlated (see Figure 3 on the left). A marker can be identified and tracked by the app only if trained, the training being performed through the Artoolkit framework. A model is a three-dimensional object that is linked to the application with a marker so that each time the marker appears within the camera, the model will be displayed above the marker. In the ARBio application, the models are represented by animals. Models are the ones that augment reality by being displayed above the marker. To model, create, view, and add texture to some models, we used the Autodesk Maya utility [35], allowing us to model and animate 3D objects. Maya incorporates the laws of physics to control the behavior of virtual objects. The functionalities they offer are numerous and can produce animations that seem to be detached from reality. The Android Application Module is the module for creating tasks, managing resources, designing and running the application. The main functionality of this module is to interact with the user, to respond to the user’s commands. The novelty of this application refers primarily to the way we augment reality: associating the image from the marker with the virtual representation of the animals, while also playing the corresponding sound. Another element of novelty is finding a way to combine the facilities offered by the Android platform to those offered by the augmented reality framework. All these elements are meant to increase the quality of the augmented reality experience. Following usability tests conducted with 7 volunteers, we drew the following conclusions about the ARBio application: • Positive aspects: the association that the app makes between the marker, three-dimensional model, associated sounds, and additional Wikipedia details helps students to keep track of the details of the animals they select. • Negative aspects: they blocked the app when multiple markers are being used at the same time, and multiple audio files are being played simultaneously. • Future work: There were users who asked for application development on other platforms (for example, iOS). We will also need to increase the collection of animals, but this will involve the need for clustering them together, plus the introduction of a filtering option that will allow to find an animal faster. In addition, teachers have asked for pupil activity to be evaluated through the existence of a test game to check the pupil to see if the animals have been taught. GeoAR Application The GeoAR application helps pupils in secondary school where they have to learn the geography of Europe (countries, capitals, flags and neighbors). GeoAR is an application that lets you play and learn at the same time. GeoAR offers the opportunity to test and deepen your knowledge in the memory with a full game of various questions. In order to have different levels of difficulties, several questions have included in they answer candidates few of the most common confusions made. For example, the flag of Andorra resembles the flag of Romania and the one of Republic of Moldova. Thus, in the variants for picking the answer, we introduced the two confusing variants in addition to the correct answer. Many questions have response variants that are meant to mislead if one is not very familiar with all the details, making the game even more exciting. Also, if the user needs more information about a particular country, the application guides you to the Wikipedia page containing the country’s information (such as: hymn, population, history, sights etc.). The printed maps accompanying the mobile software do not contain written information (such as neighbors and country names), but only the contours for each country (these are actually the markers used by our application).
172
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
When we look at the maps through its phone and video camera, we see them filled in with the name of the country, along with the flag, capital and neighbors (Figure 4).
Fig. 4. GeoAR – the Information View is based on the Marker
If one of the country’s flag and country maps is to be saved, this is possible with the extra feature of the application that can produce a printed map photo and the country name flag positioned above, depending on the user’s preference (he can enlarge/shrink and re-position overlapped objects). The application has three important modules: • The Learn Europe module - was mostly made in the Unity interface that allowed the use of the Vuforia development kit. The project exported to Android Studio has packaged the functionality provided by Unity [36] and Vuforia in several libraries. With the help of JNI Android technology, it’s easy to make links between libraries and Java code. • The Test Your Knowledge Module - As each piece of information you want to keep is tested, this module deals with these tests. The created game aims to test the user about the information presented in the learning part. Questions and variations of responses are read from a configuration file and mixed every new run to raise the difficulty level. • The Find out more module - has the functionality to redirect the user to the Wikipedia page of the country they select. If the user wants to know more about a country in Europe, there is no need to search on Google, but it can be done through the app. An internet connection is required and a single push. We also introduced a filter to find the country easier. During the development of the application, following the discussion with the teachers, we came to the conclusion that the idea of teaching pupils new things through a game is good enough because they will memorize some things without being aware of the effort they made. They will consider playing a game and they have to retain as much detail as possible to be the best player. Also, the test and knowledge assessment component is not a stress and emotional component as a written test or as an oral evaluation. 4. Evaluation We performed usability tests and collected end-users opinions about our four applications (both from professors and from students), to see what can be improved or changed in the future in these applications.
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
173
Methodology: The conducted usability test consisted of an introduction, tasks for every application, a short interview and a post-test questionnaire. We instructed the participants to think out loud and express their thoughts during the test. After the task series that we communicated verbally to the participants, we gathered their assessment of the overall experience using the QUIS (The Questionnaire for User Interaction Satisfaction) scale. The tasks that users performed covered the main options from the applications and each session took around 4-5 minutes. In some cases, users have executed the steps without having seen them executed in advance by someone, just as there were cases where they performed the tasks after someone else went previously through the steps, presenting them. 4.1 Professors opinion During all experiments, before showing our applications to students, we asked professors to use our applications. We have recorded their experience with the AR applications while they were performing usual tasks. Participants: We collaborated for evaluation with six participants from the professors involved in the student’s classes. Their selection was random, the group being formed of 4 women and 2 men. 5 of them didn’t have previous experience with AR applications. They received all 4 applications and their interactions with AR applications were during two or three consecutive days. Results: From our observation during the test sessions, the professors had the best experience while performing the tasks after someone else. They found quickly the application options and they were able to use them appropriately. The execution of tasks wasn’t very fast, thus devices have not been blocked very often. The participants were asked to rate their experience with a note from 1 to 9, where 1 stands for confusing/frustrating experience, and 9 for clear/pleasant experience. The user responses to the post-test questionnaire show that the “overall look and feel experience” is most appreciated by participants (8.5 for Ships, 9 for Hunting Treasure, 8.16 for ARBio and 8.33 for GeoAR), followed by “selection controls”, “menus”, “game options” (all with scores around 8) and by “Status bar”, “Tests” and “Collaborative options” (all with scores over 7). 4.2 Students opinion After our initial interaction with professors we presented our applications to their students. Again, we have recorded their experience with the AR applications while they were performing the same tasks as professors. Participants: We collaborated for evaluation with twelve students from the classes involved in the experiments. Their selection was random, the group being formed of 6 girls and 6 boys (from 9 years old to 16 years old, from primary classes to high school classes). All have previous experience with games from tablets or smart phones, and only 4 of them didn’t have previous experience with AR applications. They received all 4 applications and their interactions with AR applications were during one day. Results: From our observation during the test sessions, the students had the best experience while performing the tasks after previously observing someone else doing the tasks. If at the beginning they follow very strictly the steps of the tasks, after the second or third application they performed the tasks with the goal to obtain the fastest results (in order to receive more points). They found quickly the application options and they were able to use them properly. In many cases they press too fast on the screen of the devices and the majority of them managed to block one or more of the applications during the experiments. Similar to professor experiments, the students were asked to rate their experience with a score from 1 to 9, where 1 stands for confusing/frustrating experience, and 9 for clear/pleasant experience. Again, the user responses to the post-test questionnaire show that the “overall look and feel experience” is most appreciated by participants (around 8.5) followed by rest of the options which are between 7.5 and 8.
174
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
4.3 Collaborative vs. non-collaborative settings Two of our developed games involve collaborative behavior: Supporting Ships in the Air and Hunting Treasure. For the first game, playing in the collaborative mode enables children to finish the current level and pass to the next one in an average of 2/3 of the time used to play in the single player mode. For the second game, the collaborative playing requires children to communicate, either verbal or non-verbal, i.e. through gestures. If two players do not communicate during the game, it is almost impossible to finish the current level, since they have different controls (up/down, respectively left/right). When the children interact, they are able to finish the level in a time similar to the one a single player, with all controllers, would have. 4.4 Remarks When comparing to professors, students generally didn’t follow very strictly the steps of the tasks, and they tried to find shortcuts to finish sooner the assigned tasks. Also, they blocked more often the applications, mainly because they tried to obtain better results in games or in evaluation components. If, at the beginning, students from primary school were impressed by applications created for students from middle and high school (GeoAR and Treasure AR), after explanations they were very happy to use these applications even they didn’t obtain very good results. Also, in the AR application that involved communication, they were very verbose and very happy to solve together tasks. Recordings analysis clearly showed that: • The user interfaces based on AR look very attractive and were appreciated positive both by professors and by students; • Even in the few cases where UI was a bit confusing, after explanations and after given examples, things became more clear; • The devices present minor performance and stability issues, especially when moved fast or when more than one time the touch screen is fast clicked; • Social interaction between participants, visual effects, and game dynamics are well received by everyone; • Both professors and students agree that the lessons can be more attractive with AR applications and it can help professors better present the new content of the lessons. • Also, evaluations based on games reduce the stress of the children and can provide a fast way to professors to see which the level of an entire class is. 5. Conclusions The augmented reality domain has a consistent history since the 1950s, evolving from year to year and being increasingly used. The areas of applicability of augmented reality are increasingly diversified, from classical fields, such as medicine, engineering, military, collaborative 3D modeling or robotics, to newer areas adapted to our present days: entertainment, social networks, interior and exterior design, translation, security, etc. In the next period, we expect greater engagement of augmented reality applications in commands and actions via gestures, entertainment applications, smarter house management applications, games, but also eLearning. This article has presented four applications developed through augmented reality for eLearning, two focused on collaborative work of students and two on biology and geography. Following discussions with teachers and students, we could deduce that one of the contributions of the use of images, 3D models, sounds and animations is that it attracts more students than classical teaching methods. Additionally, these augmented elements also allow students to retain new information more easily, and tests designed as games contribute to reduce their stress. The main novelty of this paper is the use of augmented reality in order to improve the communication and collaboration skills between children, especially autistic children, and the game-based evaluation of pupils in various teaching areas, allowing for a stress free testing environment. Future work is related to introduction in all of our four applications of a component based on Alexa from Amazon to evaluate students that uses our augmented reality applications. This component will use voice recognition to take answers from students and will automatically evaluate them. We already performed some steps in this direction, with promising results.
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
175
Acknowledgements We would like to thank the students involved in developing the applications presented in the paper: Ciprian Florescu (Supporting Ships in the Air), Ana-Maria Daradici (ARBio), Mirela Chitaniuc (GeoAR), Marian Pînzariu (Hunting Treasure) and volunteers involved in evaluating these applications. This work is partially supported by POC-A1-A1.2.3-G-2015 program, as part of the PrivateSky project (P_40_371/13/01.09.2016). References 1. Brooks P F. The Computer Scientist as Toolsmith II. CACM, 1996: 39(3), p. 61-68. 2. Krevelen R V, Poelman R. A Survey of Augmented Reality Technologies, Applications and Limitations. The International Journal of Virtual Reality; 2010: 9(2), p. 1-20. 3. Milgram P, Kishino F. A taxonomy of mixed reality visual displays. IEICE Trans. Information and Systems; 1994: E77-D (12), p. 1321– 1329. 4. Azuma T R, Baillot Y, Behringer R, Feiner K S, Julier S, MacIntyre B. Recent advances in augmented reality. IEEE Computer Graphics and Applications; 2001:21(6), p. 34–47. 5. Azuma TR. A survey of Augmented Reality. In Presence: Teleoperators and Virtual Environments; 1997: 6(4), p. 355-385. 6. Glockner H, Jannek K, Mahn J, Theis B. Augmented reality in logistics. Changing the way we see logistics – a DHL perspective. Publisher: DHL Customer Solutions & Innovation; 2014. 7. Canon MREAL System: http://www.canon.com/technology/future/mixedreality.html accessed last time on March 10, 2018. 8. Volkswagen Marta System: https://www.volkswagenag.com/en/group/research/virtual-technologies.html accessed last time on March 10, 2018. 9. Fitzmaurice WG. Situated Information Spaces: Spatially Aware Palmtop Computers. CACM 1993; 36(7), p. 38-49. 10. Wanstall B. HUD on the Head for Combat Pilots. Interavia; 1989: 44, p. 334-338. 11. Recon Jet: https://www.reconinstruments.com/products/jet/ accessed last time on March 10, 2018. 12. Word Lens: http://bigthink.com/design-for-good/word-lens-real-time-translation-via-augmented-reality accessed last time on March 10, 2018. 13. Infinity AR: http://www.infinityar.com/ accessed last time on March 10, 2018. 14. BMW AR: http://www.motoringfile.com/2011/10/12/bmw-group-developing-augmented-reality-windshield-displays/ accessed last time on March 10, 2018. 15. Ikea AR: https://www.youtube.com/watch?v=vDNzTasuYEw accessed last time on March 10, 2018. 16. Hatsune Miku: https://www.youtube.com/watch?v=9jpWiTVR0GA accessed last time on March 10, 2018. 17. MIT SixthSense: http://www.pranavmistry.com/projects/sixthsense/ accessed last time on March 10, 2018. 18. Johnson L, Levine A, Smith R, Stone S. The 2010 Horizon Report. Austin, Texas: The New Media Consortium; 2000. 19. Munnerley D, Bacon M, Wilson A, Steele J, Hedberg J, Fitzgerald R. Confronting an augmented reality. Research in Learning Technology; 2012; 20: p. 39–48. 20. Heilig M. Sensorama Simulator. 1962 http://www.mortonheilig.com/SensoramaPatent.pdf 21. Sutherland EI. The Ultimate Display. In Proceedings of the IFIP Congress; 1965. p. 506–508. 22. Caudell PT, Mizell WD. Augmented reality: an application of heads-up display technology to manual manufacturing processes. In Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences; 1992:II. p. 659–669. 23. Forsyth E. Ar U Feeling Appy? Augmented Reality, Apps and Mobile Access to Local Studies Information. APLIS, 2011:24(3); p. 125–132. 24. El Sayed NAM, Zayed HH, Sharawy MI. (2011) ARSC: Augmented reality student card. An augmented reality solution for the education field. Computers and Education; 2011:56(4). p. 1045–1061. 25. Blum T, Kleeberger V, Bichlmeier C, Navab N. mirracle: An Augmented Reality Magic Mirror System for Anatomy Education. In Virtual Reality (VR) 2012 IEEE, Costa Mesa, California: IEEE, 2012. pp. 115–116. 26. Păduraru BM, Iftene A. Tower Defense with Augmented Reality. In Proceedings of the 14th Conference on Human Computer Interaction RoCHI 2017, 11-12 September 2017, Craiova, Romania; 2017. p. 113-118. 27. Billinghurst M, Dünser A. Augmented Reality in the Classroom. Computer; 2012:45(7). p. 56–63. 28. Mishra P, Koehler MJ. Technological Pedagogical Content Knowledge: A Framework for Teacher Knowledge. Teachers College Record; 2006:108(6). p. 1017–1054. 29. Metaio SDK: http://www.softpedia.com/get/Programming/SDK-DDK/metaio-Mobile-SDK.shtml accessed last time on March 10, 2018. 30. Sphero: http://www.sphero.com/sphero accessed last time on March 10, 2018. 31. Pinzariu MN, Iftene A. Sphero - Multiplayer Augmented Game (SMAUG). In Proceedings of International Conference on Human-Computer Interaction, 8-9 September 2016, Iasi, Romania; 2016. p. 46-49. 32. Vuforia: https://www.vuforia.com/ accessed last time on March 10, 2018. 33. Artoolkit: https://artoolkit.org/ accessed last time on March 10, 2018. 34. Java Native Interface: https://www3.ntu.edu.sg/home/ehchua/programming/java/JavaNativeInterface.html accessed last time on March 10, 2018. 35. Autodesk Maya: https://www.autodesk.com/products/maya/overview accessed last time on March 10, 2018. 36. Unity: https://unity3d.com/ accessed last time on March 10, 2018.
ScienceDirect ScienceDirect
Available online at www.sciencedirect.com Available online at www.sciencedirect.com Procedia Computer Science 00 (2018) 000–000 Procedia Computer Science 00 (2018) 000–000
ScienceDirect ScienceDirect
Procedia Computer Science 126 (2018) 166–175 Procedia Computer Science 00 (2018) 000–000
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
International International Conference Conference on on Knowledge Knowledge Based Based and and Intelligent Intelligent Information Information and and Engineering Engineering Systems, KES2018, 3-5 September 2018, Belgrade, Serbia Systems, KES2018, 3-5 September 2018, Belgrade, Serbia International Conference on Knowledge Based and Intelligent Information and Engineering Enhancing the of through Augmented KES2018, 3-5 September 2018, Belgrade, Serbia EnhancingSystems, the Attractiveness Attractiveness of Learning Learning through Augmented
Reality Reality Enhancing the Attractiveness of Learning through Augmented Adrian Diana Adrian Iftene, Iftene, Diana Trandabăț Trandabăț Reality Faculty of Computer Science, Alexandru Ioan Cuza University, General Berthelot, 16, Iasi 700483, Romania Faculty of Computer Science, Alexandru Ioan Cuza University, General Berthelot, 16, Iasi 700483, Romania
Adrian Iftene, Diana Trandabăț
Abstract Abstract
Faculty of Computer Science, Alexandru Ioan Cuza University, General Berthelot, 16, Iasi 700483, Romania
Over the last years, augmented reality was used in various domains, from medical, industrial design, modeling and production, Over the last years, augmented reality was used in various domains, from medical, industrial design, modeling and production, robot teleoperation, military, entertainment, leisure activities to translation, facial recognition, assistance while driving, interior Abstract robot teleoperation, military, entertainment, leisure activities to translation, facial recognition, assistance while driving, interior and exterior design, virtual friends, internet of things and eLearning. In eLearning, the combination between classical and and exterior design, virtual friends, internet of things and eLearning. In eLearning, the combination between classical and augmented content later coming with models, images, sounds,from animations, Internet browsing, can help teacher to Over the last years,(the augmented reality was3D in various domains, medical, industrial design,etc.) modeling andthe augmented content (the later coming with 3Dused models, images, sounds, animations, Internet browsing, etc.) can help theproduction, teacher to better teleoperation, explain the content of the courses. In leisure this paper, we present four augmented reality applications, with theinterior aim to robot military, entertainment, activities to translation, facial recognition, assistancecreated while driving, better explain the content of the courses. In this paper, we present four augmented reality applications, created with the aim to improve communication and collaboration skills (two of them) and to ease the learning of biology and geography (the other two). and exterior design, virtual friends, internet of(two things and eLearning. combination between classical and improve communication and collaboration skills of them) and to easeIntheeLearning, learning ofthe biology and geography (the other two). The motivation behind applications enhanceimages, the attractiveness of the classes, allow studentsetc.) to retrain newthe information augmented content (thethese later coming with is 3Dto sounds, animations, Internet browsing, can help teacher to The motivation behind these applications is tomodels, enhance the attractiveness of the classes, allow students to retrain new information more andthe reduce theof stress behind tests when presented as games. bettereasily explain content the courses. In this paper, we present four augmented reality applications, created with the aim to more easily and reduce the stress behind tests when presented as games. improve communication and collaboration skills (two of them) and to ease the learning of biology and geography (the other two). © 2018 The Authors. Published by Elsevier Ltd. Keywords: Augmented Reality; eLearning; Autism; Usability Testing The behind these eLearning; applications to Usability enhance Testing the attractiveness of the classes, allow students to retrain new information Keywords: Reality; Thismotivation is an Augmented open access article under theAutism; CCisBY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) more easily and reduce theunder stressresponsibility behind tests when presented as games. Selection and peer-review of KES International.
1. Introduction Keywords: Augmented Reality; eLearning; Autism; Usability Testing 1. Introduction Augmented Reality (AR) improves the perception of the user, helps him to better understand the surrounding Augmented Reality (AR) improves the perception of the user, helps him to better understand the surrounding 1. Introduction reality and interact with the real world. Virtual objects display information that user cannot detect directly with their reality and interact with the real world. Virtual objects display information that user cannot detect directly with their own senses. The information transmitted by virtual objects can help the user in performing his tasks. AR is a specific own senses. The information transmitted by virtual objects can help the user in performing his tasks. AR is a specific Augmented (AR) improves the perception of the (IA): user, using helpsthe himcomputer to betterasunderstand the surrounding example of whatReality Fred Brooks calls intelligence amplification a tool to make it easier for example of what Fred Brooks calls intelligence amplification (IA): using the computer as a tool to make it easier for and interact areality human being [1]. with the real world. Virtual objects display information that user cannot detect directly with their a human being [1]. own information transmitted virtual objects can help the usercontinuum, in performing his tasks. AR is by a specific Insenses. 2010, The in [2], the authors presentedbythe notion of reality-virtuality originally defined [3], in In 2010, in [2], the authors presented the notion of reality-virtuality continuum, originally defined by [3], in example ofiswhat Fred calls realm intelligence amplification theenvironments computer as (i.e. a toolvirtual to make it easier for which AR a part of Brooks the general of combined reality.(IA): Bothusing virtual realities - VR) which AR is a part of the general realm of combined reality. Both virtual environments (i.e. virtual realities - VR) a human being [1]. and augmented virtualization, in which real objects are added to virtual ones, replace the environment with a virtual and augmented virtualization, in which real objects are added to virtual ones, replace the environment with a virtual In 2010, inIn[2], the authors presented the notion reality-virtuality continuum, originally defined [3],user in environment. contrast, AR provides a local virtual of reality. When considering not only artificiality butby also environment. In contrast, AR provides a local virtual reality. When considering not only artificiality but also user which ARitisclassifies a part ofAR theasgeneral realm of combined reality. Both virtual[4]. environments (i.e. virtual realities - VR) transport, being separated by both VR and telepresence transport, it classifies AR as being separated by both VR and telepresence [4]. andThe augmented virtualization, in intensive which realuse objects added to virtual replace environment a virtual most important areas of have are been described for ones, the first time the in [5], and later with in [6]. Thus, The most important areas of intensive use have been described for the first time in [5], and later in [6]. Thus, environment. In contrast, provides a local virtual reality. in When considering not only but also user doctors can use AR as a AR visual aid and training instrument performing surgeries [5].artificiality Canon’s Mixed Reality doctors can use AR as a visual aid and training instrument in performing surgeries [5]. Canon’s Mixed Reality transport, it classifies AR asthe being separated byby both VR and [4]. System (MREAL) supports design process allowing 3Dtelepresence computer generated models to be combined with realSystem (MREAL) supports the design process by allowing 3D computer generated models to be combined with realTheobjects most important areascategory of intensive use have reality been described for is thethe first time in [5], and later and in [6]. Thus, world [7]. Another of augmented applications assembly, maintenance repair of world objects [7]. Another category of augmented reality applications is the assembly, maintenance and repair of doctors use AR[8]. as AR a visual and training instrument surgeries [5]. Canon’s Mixed Reality complexcan machines couldaid be used to annotate objects in andperforming environments with public or private information complex machines [8]. AR could be used to annotate objects and environments with public or private information System (MREAL) supports the design process by allowing 3D computer generated models to be combined with real1877-0509 2018[7]. The Authors. Published by Elsevier world objects category of augmented reality applications is the assembly, maintenance and repair of 1877-0509 2018 The Another Authors. Published by Elsevier B.V. Peer-review under responsibility of KES International B.V. Peer-review under responsibility of KES complex machines [8]. AR could be International used to annotate objects and environments with public or private information 1877-0509 Published by Elsevier 1877-0509 © 2018 2018 The TheAuthors. Authors. Published by Elsevier Ltd. B.V. under responsibility KES This Peer-review is an open access article underofthe CCInternational BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Selection and peer-review under responsibility of KES International. 10.1016/j.procs.2018.07.220
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
167
[9]. For many years, military airplanes and helicopters have used HUDs (Head-Up Displays) and HMS (HelmetMounted Sights) to overlay vector graphics over the images that the pilot had of the real world [10]. Recon Jet [11] is an AR system already available for recreational activities. One of the most promising areas of application in the AR is the translation field [12]. An example of easy access to information from the Internet in real-life situations is the combination of face detection with AR [13]. Nowadays, many car manufacturers have included on-board information displayed on the windscreen [14], such as speed, direction of movement based on real-world object recognition, parking assistance, sensor information, and warnings related to traffic, warnings related to possible collisions, etc. One of the most suggestive examples of using AR in interior design is the most recent Ikea catalog using AR [15]. A hacker from Japan used an available 3D model and motion sensors to have an “AR meeting” with a famous Japanese cartoon star (Hatsune Miku [16]). An example of a gesture interface system is SixthSense [17], developed by MIT. 2. Related Work - Augmented Reality in eLearning As the authors of 2010 Horizon Report [18] assert, the augmented reality has a huge potential to provide a useful context for education, allowing learning and discovery experiences connected to real world information. Mixed realities have been used in education for a long time [19]: the augmentation of wall paintings in caves is an approach to the transfer of knowledge about hunting and survival; Heilig’s patent for Sensorama describes the need for a solution for teaching, training and educating people in armed forces, industries and schools [20]; Ivan Sutherland saw the augmented image as a solution to give the user a “chance to get acquainted with concepts that cannot be achieved in the physical world” [21]; and the pioneering activity of Caudell and Mizel in Boeing’s augmented reality were designed to teach workers how to assemble complex components into aircraft [22]. Large-scale applications for AR in education were impossible before recent releases of cheap and affordable smartphones and tablets and before the emergence of software that allows the development and experimentation of real world augmentation. Opportunities to exploit AR technologies in education are now growing, while mobile hardware is proliferating, and access to the internet becomes universal. Among the first usage of augmented reality in education are applications in industrial training; surgical assistance; maintenance of equipment’s; service and car repair; behavioral changes; architecture, urban and environmental education [23]. Applying AR in an educational context equals to using “technology to add virtual objects to real scenes, by adding missing information to real life” [24]. As one can see, the emphasis is on providing additional information, such as a type that may be missing or inaccessible to students in the real world. Teaching anatomy, which requires a considerable amount of effort, expertise and temporal resources, is an example where AR can be used effectively to provide additional information [25]. In artificial intelligence augmented reality can be used to show what are the steps executed by an algorithm based on a Greedy paradigm [26]. Our ARBio and GeoAR game are another example of AR applications. One of the AR’s most significant features in terms of pedagogy is that it offers a student-oriented and flexible space to provide learning opportunities. Learning is taken away from traditional spaces, such as classrooms, amphitheaters and labs, and follows the students wherever they are. Learning opportunities can be present, for example, at home, at the workplace, in public transportation – and can be taken everywhere or can be transferred to anyone. As the AR develops more about interactive applications, students can become critics and co-creators, leaving behind a record of their learning the specific artifact or the place they have met [27]. AR does not come as a revolution: AR will not replace the existing pedagogical paradigm with a new world based on high technology. Instead, the pedagogical past is a rich resource for the future. The work of Mishra and Koehler [28] highlighted the importance of understanding the nuances between content, pedagogy and technology in designing learning environments. The model they propose, TPACK, highlights the dynamic interaction between these three areas. In this paper, we will present four AR applications developed for different types of children: (1) children with autism, (2) kindergarten children, (3) middle-aged children and (4) high school children. Our main goal is to see how AR applications are accepted by children, and what impact they have on children as well as on teachers.
168
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
3. Proposed solutions 3.1 Developing Communication Skills and Teamwork More and more emphasis is now being put on improving and developing soft skills, such as communication skills and teamwork, in children and adults of all ages. A method often used by teachers in schools is to involve pupils in different kinds of projects, where each member needs to make their own contribution to achieving the final goals. Typically, such projects aim to improve verbal and visual communication, but there are situations, as in the case of autistic children, where indirect communication through games can represent a breakthrough in the behavior of these children with special needs. Within our group, we implemented two games with network game components: (1) the first for autistic children, in order to develop attention, guidance and communication, in which they gain bonuses when they collaborate during the game to achieve a common goal, each being encouraged to follow the actions of others and to actively intervene in the game, and (2) the second one is to synchronize players’ activities, by verbal or visual instruction. Supporting Ships in the Air The purpose of this first game is to fuel more ships from the flight to allow them to carry out their missions. Once the missions at a specific level of the game are completed, the user goes to next level, to discover other missions, and so on until the final level is reached. In multiplayer mode, fueling is made easier and faster when two or more players fuel up the same ship, allowing the accumulation of a higher number of points and thus moving faster to next levels. Non-collaborating players stay more at a level and arrive harder at higher levels, while players who collaborate perform their tasks more quickly and manage to get faster to the end of the game. Figure 1 shows a sequence of the game level having only one ship to fuel.
Fig. 1. One of the Three Ships to be fueled
The application architecture is based on the client-server model. Mobile users are customers who communicate with the server through the UDP protocol to have the highest and most efficient speed. The client, implemented in Android, presents all the graphic elements, contains the augmented reality elements, made using the Metaio framework [29] and ensures communication with the user. The server, deployed in Java, generates the necessary data for a game level, ensures communication between users, and manages the information needed to unfold the current level, before storing in a database the score for the current user and global statistics.
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
169
With the help of four volunteers, usability tests have been made and the game was constantly improved. They proposed that switching from one level to another to involve an increasing in the number of ships, in the distance between ships and the user, rotating them, moving faster so that their fueling becomes more challenging. Also, a time counter has been introduced, which provides bonuses when the level is completed faster. In order to observe the effect of the application on autistic children, we contacted an expert, the observations made by him being positive. Children who suffer from this disease prefer to communicate with images. The ability to understand and think is enhanced when they receive visual stimulation. Multiplayer mode fits best for them. The emphasis in their case is on focusing on the environment and the detection of color-changing visuals. The radar provides the concrete position of the fueled ship, which develops the attention. Game partners will realize that they have to work together to fuel the ship efficiently. In conclusion, this direction of development shows promising first results. Hunting Treasure In the second game, the Sphero 2.0 [30] robotic ball is used, and two players can control it through the same robot controls (a first version of this application is presented in [31]). Depending on the play session and the task it has to fulfill, a player can influence a specific direction (i.e. up/down or right/left) and the speed of the robot. In addition, the surface on which the robot moves is augmented on a tablet or phone, so that each player looks at the scene of the game through the camera of the device he is using. The main objective here is that players work together to accomplish different tasks: collecting objects, solving puzzles, driving on augmented tracks, going through walls or gates, escaping from labyrinths, and so on. To achieve a goal, the game requires collaboration: augmented objects are distributed between players, so they will often have to communicate to drive Sphero in the right direction. Figure 2 shows the multiplayer option and the way the two players crawl throughout the game.
Fig. 2. Hunting treasure – Multiplayer option
To control Sphero, we have built a specific Android activity that simulates a TCP server. When this activity is started, the Bluetooth connection is checked, to turn on and discover Sphero devices. Once Sphero is discovered, we connect to it and start a separate thread running with the TCP server in standby mode. The multi-player option has been implemented with a Client-Server architecture, where the server is started by one of the players (the host). Then, for each player who connects to the server (including the host), a client is created. For the augmented reality component we used the Vuforia platform [32]. From this platform, we used two components, Smart Terrain and Image trackers, which allowed adding augmented content over Sphero’s workspace.
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
170
In order to have a stable location for game objects, Smart Terrain offers the ability to set a central point. For this we have chosen as a marker a feature-rich image, so that it can be easily recognized by the Vuforia engine. To improve the quality of the app, we have tested usability with four volunteers. We analyzed and summarized the qualitative and quantitative data, as well as the redesign recommendations that were made. In conclusions, the volunteers signaled in initial versions of the application, both (1) negative elements: the ambiguity of the interface, the multi-player component had problems of performance and stability, (2) and positive elements: Sphero was easier to control than expected, and interaction elements between players, 3D models, and game dynamics were most appreciated by the volunteers. 3.2 Lessons for the Primary and Secondary Cycles Apps for primary and secondary classes help students acquire information from biology, geography, history, and astronomy. Here are two of our applications, for biology and geography, two others for history and astronomy being under construction. ARBio Application ARBio is an application that uses augmented reality to respond to the curiosities of children in primary school, but not only. With the help of the application, the user lives the experience of augmented reality by enriching the reality of the images of some animals with their three-dimensional representation and with onomatopoeias produced by animals. The user is also able to browse the list of animals and read information about each one. ARBio is a modern learning method and an interesting experience for a user of any age (see Figure 3).
Fig. 3. (left) ARBio - the view is based on the marker (right) the application allows simultaneous viewing of several animals
The main modules of the application are: the augmented reality module and the module for viewing and reading animal information, including local marker downloading. The augmented reality module is the most important and complex part of the application because it augments the reality of the environment by adding virtual elements. This module was made using the Artoolkit framework [33]. From the implementation point of view, the logic of the augmented reality module is divided into two main components. The first component uses C++ programming language. This module contains methods from the
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
171
ARWrapper library to initialize and manipulate the augmented reality section of the application. This module is basically the link between the Artoolkit framework and the ARBio application. The second component containing the augmented reality logic uses the Java programming language and manages objects (patterns, markers, sound files) in data structures that it sends to the other component by JNI (Java Native Interface [34]) method calls. Therefore, the communication between the two components is made by native calls of predefined methods. The first step in achieving the augmentation of reality was to create markers. For this application, we only used markers consisting of square templates. Each marker, pre-trained in the application, corresponds to a virtual model so that whenever a marker appears in the camera view, the reality will be augmented by adding the appropriate animal. The markers are made up of a black border and the inside of the marker is a suggestive image of the animal with which the marker is correlated (see Figure 3 on the left). A marker can be identified and tracked by the app only if trained, the training being performed through the Artoolkit framework. A model is a three-dimensional object that is linked to the application with a marker so that each time the marker appears within the camera, the model will be displayed above the marker. In the ARBio application, the models are represented by animals. Models are the ones that augment reality by being displayed above the marker. To model, create, view, and add texture to some models, we used the Autodesk Maya utility [35], allowing us to model and animate 3D objects. Maya incorporates the laws of physics to control the behavior of virtual objects. The functionalities they offer are numerous and can produce animations that seem to be detached from reality. The Android Application Module is the module for creating tasks, managing resources, designing and running the application. The main functionality of this module is to interact with the user, to respond to the user’s commands. The novelty of this application refers primarily to the way we augment reality: associating the image from the marker with the virtual representation of the animals, while also playing the corresponding sound. Another element of novelty is finding a way to combine the facilities offered by the Android platform to those offered by the augmented reality framework. All these elements are meant to increase the quality of the augmented reality experience. Following usability tests conducted with 7 volunteers, we drew the following conclusions about the ARBio application: • Positive aspects: the association that the app makes between the marker, three-dimensional model, associated sounds, and additional Wikipedia details helps students to keep track of the details of the animals they select. • Negative aspects: they blocked the app when multiple markers are being used at the same time, and multiple audio files are being played simultaneously. • Future work: There were users who asked for application development on other platforms (for example, iOS). We will also need to increase the collection of animals, but this will involve the need for clustering them together, plus the introduction of a filtering option that will allow to find an animal faster. In addition, teachers have asked for pupil activity to be evaluated through the existence of a test game to check the pupil to see if the animals have been taught. GeoAR Application The GeoAR application helps pupils in secondary school where they have to learn the geography of Europe (countries, capitals, flags and neighbors). GeoAR is an application that lets you play and learn at the same time. GeoAR offers the opportunity to test and deepen your knowledge in the memory with a full game of various questions. In order to have different levels of difficulties, several questions have included in they answer candidates few of the most common confusions made. For example, the flag of Andorra resembles the flag of Romania and the one of Republic of Moldova. Thus, in the variants for picking the answer, we introduced the two confusing variants in addition to the correct answer. Many questions have response variants that are meant to mislead if one is not very familiar with all the details, making the game even more exciting. Also, if the user needs more information about a particular country, the application guides you to the Wikipedia page containing the country’s information (such as: hymn, population, history, sights etc.). The printed maps accompanying the mobile software do not contain written information (such as neighbors and country names), but only the contours for each country (these are actually the markers used by our application).
172
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
When we look at the maps through its phone and video camera, we see them filled in with the name of the country, along with the flag, capital and neighbors (Figure 4).
Fig. 4. GeoAR – the Information View is based on the Marker
If one of the country’s flag and country maps is to be saved, this is possible with the extra feature of the application that can produce a printed map photo and the country name flag positioned above, depending on the user’s preference (he can enlarge/shrink and re-position overlapped objects). The application has three important modules: • The Learn Europe module - was mostly made in the Unity interface that allowed the use of the Vuforia development kit. The project exported to Android Studio has packaged the functionality provided by Unity [36] and Vuforia in several libraries. With the help of JNI Android technology, it’s easy to make links between libraries and Java code. • The Test Your Knowledge Module - As each piece of information you want to keep is tested, this module deals with these tests. The created game aims to test the user about the information presented in the learning part. Questions and variations of responses are read from a configuration file and mixed every new run to raise the difficulty level. • The Find out more module - has the functionality to redirect the user to the Wikipedia page of the country they select. If the user wants to know more about a country in Europe, there is no need to search on Google, but it can be done through the app. An internet connection is required and a single push. We also introduced a filter to find the country easier. During the development of the application, following the discussion with the teachers, we came to the conclusion that the idea of teaching pupils new things through a game is good enough because they will memorize some things without being aware of the effort they made. They will consider playing a game and they have to retain as much detail as possible to be the best player. Also, the test and knowledge assessment component is not a stress and emotional component as a written test or as an oral evaluation. 4. Evaluation We performed usability tests and collected end-users opinions about our four applications (both from professors and from students), to see what can be improved or changed in the future in these applications.
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
173
Methodology: The conducted usability test consisted of an introduction, tasks for every application, a short interview and a post-test questionnaire. We instructed the participants to think out loud and express their thoughts during the test. After the task series that we communicated verbally to the participants, we gathered their assessment of the overall experience using the QUIS (The Questionnaire for User Interaction Satisfaction) scale. The tasks that users performed covered the main options from the applications and each session took around 4-5 minutes. In some cases, users have executed the steps without having seen them executed in advance by someone, just as there were cases where they performed the tasks after someone else went previously through the steps, presenting them. 4.1 Professors opinion During all experiments, before showing our applications to students, we asked professors to use our applications. We have recorded their experience with the AR applications while they were performing usual tasks. Participants: We collaborated for evaluation with six participants from the professors involved in the student’s classes. Their selection was random, the group being formed of 4 women and 2 men. 5 of them didn’t have previous experience with AR applications. They received all 4 applications and their interactions with AR applications were during two or three consecutive days. Results: From our observation during the test sessions, the professors had the best experience while performing the tasks after someone else. They found quickly the application options and they were able to use them appropriately. The execution of tasks wasn’t very fast, thus devices have not been blocked very often. The participants were asked to rate their experience with a note from 1 to 9, where 1 stands for confusing/frustrating experience, and 9 for clear/pleasant experience. The user responses to the post-test questionnaire show that the “overall look and feel experience” is most appreciated by participants (8.5 for Ships, 9 for Hunting Treasure, 8.16 for ARBio and 8.33 for GeoAR), followed by “selection controls”, “menus”, “game options” (all with scores around 8) and by “Status bar”, “Tests” and “Collaborative options” (all with scores over 7). 4.2 Students opinion After our initial interaction with professors we presented our applications to their students. Again, we have recorded their experience with the AR applications while they were performing the same tasks as professors. Participants: We collaborated for evaluation with twelve students from the classes involved in the experiments. Their selection was random, the group being formed of 6 girls and 6 boys (from 9 years old to 16 years old, from primary classes to high school classes). All have previous experience with games from tablets or smart phones, and only 4 of them didn’t have previous experience with AR applications. They received all 4 applications and their interactions with AR applications were during one day. Results: From our observation during the test sessions, the students had the best experience while performing the tasks after previously observing someone else doing the tasks. If at the beginning they follow very strictly the steps of the tasks, after the second or third application they performed the tasks with the goal to obtain the fastest results (in order to receive more points). They found quickly the application options and they were able to use them properly. In many cases they press too fast on the screen of the devices and the majority of them managed to block one or more of the applications during the experiments. Similar to professor experiments, the students were asked to rate their experience with a score from 1 to 9, where 1 stands for confusing/frustrating experience, and 9 for clear/pleasant experience. Again, the user responses to the post-test questionnaire show that the “overall look and feel experience” is most appreciated by participants (around 8.5) followed by rest of the options which are between 7.5 and 8.
174
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
4.3 Collaborative vs. non-collaborative settings Two of our developed games involve collaborative behavior: Supporting Ships in the Air and Hunting Treasure. For the first game, playing in the collaborative mode enables children to finish the current level and pass to the next one in an average of 2/3 of the time used to play in the single player mode. For the second game, the collaborative playing requires children to communicate, either verbal or non-verbal, i.e. through gestures. If two players do not communicate during the game, it is almost impossible to finish the current level, since they have different controls (up/down, respectively left/right). When the children interact, they are able to finish the level in a time similar to the one a single player, with all controllers, would have. 4.4 Remarks When comparing to professors, students generally didn’t follow very strictly the steps of the tasks, and they tried to find shortcuts to finish sooner the assigned tasks. Also, they blocked more often the applications, mainly because they tried to obtain better results in games or in evaluation components. If, at the beginning, students from primary school were impressed by applications created for students from middle and high school (GeoAR and Treasure AR), after explanations they were very happy to use these applications even they didn’t obtain very good results. Also, in the AR application that involved communication, they were very verbose and very happy to solve together tasks. Recordings analysis clearly showed that: • The user interfaces based on AR look very attractive and were appreciated positive both by professors and by students; • Even in the few cases where UI was a bit confusing, after explanations and after given examples, things became more clear; • The devices present minor performance and stability issues, especially when moved fast or when more than one time the touch screen is fast clicked; • Social interaction between participants, visual effects, and game dynamics are well received by everyone; • Both professors and students agree that the lessons can be more attractive with AR applications and it can help professors better present the new content of the lessons. • Also, evaluations based on games reduce the stress of the children and can provide a fast way to professors to see which the level of an entire class is. 5. Conclusions The augmented reality domain has a consistent history since the 1950s, evolving from year to year and being increasingly used. The areas of applicability of augmented reality are increasingly diversified, from classical fields, such as medicine, engineering, military, collaborative 3D modeling or robotics, to newer areas adapted to our present days: entertainment, social networks, interior and exterior design, translation, security, etc. In the next period, we expect greater engagement of augmented reality applications in commands and actions via gestures, entertainment applications, smarter house management applications, games, but also eLearning. This article has presented four applications developed through augmented reality for eLearning, two focused on collaborative work of students and two on biology and geography. Following discussions with teachers and students, we could deduce that one of the contributions of the use of images, 3D models, sounds and animations is that it attracts more students than classical teaching methods. Additionally, these augmented elements also allow students to retain new information more easily, and tests designed as games contribute to reduce their stress. The main novelty of this paper is the use of augmented reality in order to improve the communication and collaboration skills between children, especially autistic children, and the game-based evaluation of pupils in various teaching areas, allowing for a stress free testing environment. Future work is related to introduction in all of our four applications of a component based on Alexa from Amazon to evaluate students that uses our augmented reality applications. This component will use voice recognition to take answers from students and will automatically evaluate them. We already performed some steps in this direction, with promising results.
Adrian Iftene et al. / Procedia Computer Science 126 (2018) 166–175 Adrian Iftene, Diana Trandabăț/ Procedia Computer Science 00 (2018) 000–000
175
Acknowledgements We would like to thank the students involved in developing the applications presented in the paper: Ciprian Florescu (Supporting Ships in the Air), Ana-Maria Daradici (ARBio), Mirela Chitaniuc (GeoAR), Marian Pînzariu (Hunting Treasure) and volunteers involved in evaluating these applications. This work is partially supported by POC-A1-A1.2.3-G-2015 program, as part of the PrivateSky project (P_40_371/13/01.09.2016). References 1. Brooks P F. The Computer Scientist as Toolsmith II. CACM, 1996: 39(3), p. 61-68. 2. Krevelen R V, Poelman R. A Survey of Augmented Reality Technologies, Applications and Limitations. The International Journal of Virtual Reality; 2010: 9(2), p. 1-20. 3. Milgram P, Kishino F. A taxonomy of mixed reality visual displays. IEICE Trans. Information and Systems; 1994: E77-D (12), p. 1321– 1329. 4. Azuma T R, Baillot Y, Behringer R, Feiner K S, Julier S, MacIntyre B. Recent advances in augmented reality. IEEE Computer Graphics and Applications; 2001:21(6), p. 34–47. 5. Azuma TR. A survey of Augmented Reality. In Presence: Teleoperators and Virtual Environments; 1997: 6(4), p. 355-385. 6. Glockner H, Jannek K, Mahn J, Theis B. Augmented reality in logistics. Changing the way we see logistics – a DHL perspective. Publisher: DHL Customer Solutions & Innovation; 2014. 7. Canon MREAL System: http://www.canon.com/technology/future/mixedreality.html accessed last time on March 10, 2018. 8. Volkswagen Marta System: https://www.volkswagenag.com/en/group/research/virtual-technologies.html accessed last time on March 10, 2018. 9. Fitzmaurice WG. Situated Information Spaces: Spatially Aware Palmtop Computers. CACM 1993; 36(7), p. 38-49. 10. Wanstall B. HUD on the Head for Combat Pilots. Interavia; 1989: 44, p. 334-338. 11. Recon Jet: https://www.reconinstruments.com/products/jet/ accessed last time on March 10, 2018. 12. Word Lens: http://bigthink.com/design-for-good/word-lens-real-time-translation-via-augmented-reality accessed last time on March 10, 2018. 13. Infinity AR: http://www.infinityar.com/ accessed last time on March 10, 2018. 14. BMW AR: http://www.motoringfile.com/2011/10/12/bmw-group-developing-augmented-reality-windshield-displays/ accessed last time on March 10, 2018. 15. Ikea AR: https://www.youtube.com/watch?v=vDNzTasuYEw accessed last time on March 10, 2018. 16. Hatsune Miku: https://www.youtube.com/watch?v=9jpWiTVR0GA accessed last time on March 10, 2018. 17. MIT SixthSense: http://www.pranavmistry.com/projects/sixthsense/ accessed last time on March 10, 2018. 18. Johnson L, Levine A, Smith R, Stone S. The 2010 Horizon Report. Austin, Texas: The New Media Consortium; 2000. 19. Munnerley D, Bacon M, Wilson A, Steele J, Hedberg J, Fitzgerald R. Confronting an augmented reality. Research in Learning Technology; 2012; 20: p. 39–48. 20. Heilig M. Sensorama Simulator. 1962 http://www.mortonheilig.com/SensoramaPatent.pdf 21. Sutherland EI. The Ultimate Display. In Proceedings of the IFIP Congress; 1965. p. 506–508. 22. Caudell PT, Mizell WD. Augmented reality: an application of heads-up display technology to manual manufacturing processes. In Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences; 1992:II. p. 659–669. 23. Forsyth E. Ar U Feeling Appy? Augmented Reality, Apps and Mobile Access to Local Studies Information. APLIS, 2011:24(3); p. 125–132. 24. El Sayed NAM, Zayed HH, Sharawy MI. (2011) ARSC: Augmented reality student card. An augmented reality solution for the education field. Computers and Education; 2011:56(4). p. 1045–1061. 25. Blum T, Kleeberger V, Bichlmeier C, Navab N. mirracle: An Augmented Reality Magic Mirror System for Anatomy Education. In Virtual Reality (VR) 2012 IEEE, Costa Mesa, California: IEEE, 2012. pp. 115–116. 26. Păduraru BM, Iftene A. Tower Defense with Augmented Reality. In Proceedings of the 14th Conference on Human Computer Interaction RoCHI 2017, 11-12 September 2017, Craiova, Romania; 2017. p. 113-118. 27. Billinghurst M, Dünser A. Augmented Reality in the Classroom. Computer; 2012:45(7). p. 56–63. 28. Mishra P, Koehler MJ. Technological Pedagogical Content Knowledge: A Framework for Teacher Knowledge. Teachers College Record; 2006:108(6). p. 1017–1054. 29. Metaio SDK: http://www.softpedia.com/get/Programming/SDK-DDK/metaio-Mobile-SDK.shtml accessed last time on March 10, 2018. 30. Sphero: http://www.sphero.com/sphero accessed last time on March 10, 2018. 31. Pinzariu MN, Iftene A. Sphero - Multiplayer Augmented Game (SMAUG). In Proceedings of International Conference on Human-Computer Interaction, 8-9 September 2016, Iasi, Romania; 2016. p. 46-49. 32. Vuforia: https://www.vuforia.com/ accessed last time on March 10, 2018. 33. Artoolkit: https://artoolkit.org/ accessed last time on March 10, 2018. 34. Java Native Interface: https://www3.ntu.edu.sg/home/ehchua/programming/java/JavaNativeInterface.html accessed last time on March 10, 2018. 35. Autodesk Maya: https://www.autodesk.com/products/maya/overview accessed last time on March 10, 2018. 36. Unity: https://unity3d.com/ accessed last time on March 10, 2018.