- Email: [email protected]
Personalized training through Kinect-based games for physical education
J. Vis. Commun. Image R. 62 (2019) 394–401
Contents lists available at ScienceDirect
J. Vis. Commun. Image R. journal homepage: www.elsevier.com/locate/jvci
Personalized training through Kinect-based games for physical education Mingliang Xu a, Yafang Zhai a, Yibo Guo a,⇑, Pei Lv a, Yafei Li a, Meng Wang b, Bing Zhou a a b
Center for Interdisciplinary Infromation Sciences, Zhengzhou University, Zhengzhou, China School of Computer and Information, Hefei University of Technology, Hefei, China
a r t i c l e
i n f o
Article history: Received 18 December 2017 Revised 18 March 2019 Accepted 17 May 2019 Available online 20 June 2019 Keywords: Kinect Educational games
a b s t r a c t In recent years, the Kinect-based systems that enable users to be trained without the participation of teachers have been widely used in the field of physical education. In this paper, we propose a novel technique that helps the Kinect-based training system to select the subsequential training material for the users according to their realtime performance. An algorithm based on the Hidden Markov Model is demonstrated to generate the customized training pathes(training curriculums) for each individual. We present an edutainment gaming system for children in order to illustrate the feasibility of the training method. A user study of 10 children participants is conducted and the results show that the proposed technique enhances the effect of physical training significantly. Ó 2019 Elsevier Inc. All rights reserved.
1. Introduction The traditional sports training or other physical education requires trainees to participate in the training courses under the guidance of the teaching staff [1,5,3,7]. The trainers need to evaluate the trainees’ current performance and customize subsequential courses according to the learning ability of the trainees. Since the number of the teaching staff is much smaller than the trainees, the opportunity of communicating with the trainer is not always available for everyone [2,3,7,5]. In recent years, the educational videos for physical training have thrived in the market, which completely removes human trainers during the teaching process [6,8–10,30]. Although it solved the problem of insufficient trainers, the quality of the teaching process cannot be guaranteed due to the lack of interactions. The misunderstanding of video contents and the incorrect imitated movements are not fed back to the trainees in real time. Therefore, how to improve the quality of physical education without the participation of the teaching staff still needs to be addressed in more detail by both scientific and education community [11–15]. In the last few years, the gaming technologies have been improved enormously with the development of enhanced human-machine interface. The novel interaction methods are comparably cheaper since the invention of low-costed sensors, such as Microsoft Kinect, are popularized in the field of computer vision [16–18]. Many works that utilizes Kinect sensors on interactive ⇑ Corresponding author. E-mail address: [email protected] (Y. Guo). https://doi.org/10.1016/j.jvcir.2019.05.007 1047-3203/Ó 2019 Elsevier Inc. All rights reserved.
physical educations have been proposed [4,13–16]. Most of the current methods identify the trainee’s physical movements, generate feedbacks for trainees according to the degree of conformity with the actual actions, and expect the users to correct their mistakes by themselves [19,21,20]. Compared with the video training method, this kinect-based training systems strengthen the effect of self-teaching effectively. Since each trainee may not be at the same level during the training process, it is necessary to redesign customized learning content for each individual [22–24]. In this article, we present a novel technique for physical education systems based on Kinect, which generate customized learning patches (a bunch of selected learning material) for each individual according to their current learning status [25,26]. In physical training courses, the trainee’s performance is evaluated by his reaction to the teaching material. Therefore, in many physical edutainment games, Kinect is used to collect trainee’s physical activities [27–30]. To estimate human actions and intentions, the common method is to rebuild the pose of the human skeleton based on the image captured by the camera. There are many ways of detecting and tracking human joint points in motion [31–34]. In order to recognize the precise human gesture, three approaches are commonly used depending on its application: template mapping, semantic approach, and statistical approach. The template mapping method compares the input image with stored image pixel-wisely, and therefore it is not suitable for realistic figures with distortion or noises [33,35]. The statistical approaches requires a huge set of images collected in advance, which might be too costly for some simple matching occurrences [28,36,37,24,27]. The semantic approaches, such as Hidden Markov
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401
Models (HMM), recognize the relations between object components, have been proved to be an efficient real-time method for the context-aware situations [38,39,16,18]. As for the physical training courses, the target movements and postures are predefined by the trainers. The similarity of a sequence of user postures compared with the predefined procedure can be precisely evaluated by the HMM models [17,40]. In this article, we need to acquire the best training material for the trainee depending on his completeness of the current course. Therefore, a 2-layer hierarchical HMM model is designed in order to evaluate the current learning status, and choose the most suitable material (vedio or text) for the trainee. The advantage of 2-layer hierarchical HMM model is that we firstly evaluate a sequence of the users’ activities, and then customize the learning patch for different trainees based their unique performance in real time. The contributions of this paper include: We design an algorithm with 2-layer hierarchical Hidden Markov Model that generates customized subsequential learning contents for each individual according to their current performance. We develop an educational gaming system for children utilized the proposed algorithm. A user study is conducted and the results showed that the proposed algorithm has improved the learning effect promisingly. The rest of this paper is organized as follows: Background and related works are provided in Section 2. The 2-layer hieratical HMM algorithm is introduced in Section 3. The algorithm of generating learning patch for Kinect-based systems is proposed in Section 4. The educational gaming system for children utilized the proposed algorithm is demonstrated in Section 5. The user study is presented in Section 6. The paper is concluded in Section 7.
2. Related work 2.1. Serious games The serious gaming technology has been widely adopted in multiple educational fields. The creation of a serious game is aimed to teach professional skills ranging from military to education. The word ‘‘edutainment” is defined as a continuous and innovative brain-training, which stimulates, in an interactive way, the capacity to combine attention and motivation to explore and learn [5–8]. Kinect sensors have been proved to be a cheap and efficient way of human machine interaction [9,10]. The Kinect sensors allow the machine to detect the depth information of the players and the environment as well as the speech instructions [8,9]. The trajectory of the player is interpreted in real time, and is translated to a format to rebuild for new experiences [11,21]. With the 3-D human motion capturing and speech recognition algorithm of the Kinect system, the players are enabled to interact with the game console without touching the controller. The recent development of Kinect technologies have been applied to both situational education and personalized education [17–21]. Among them, Kinect-based games have been noticed as the main field of the application in education. and its development provides more possibilities for edutainment learning. According to the previous works [13,19–21], educational games can foster users’ knowledge, skills, intelligence, emotions, attitudes and values [1,2,5–7]. In [5], the authors discuss the feasibility of edutainment techniques, and believe that edutainment games may increase learners’ intrinsic learning motivation. Because of the requirement of utilizing all the senses of the learner, it showed a great advantage in strengthening and stimulating the focus during the process
395
of learning. It also contributes to the reinforce the learners’ persistence and the sense of immersion [20,21]. 2.2. Pose and activity estimation for Kinect Recognizing semantic human poses and activities given the skeletal joints by Kinect in the context of applications have been studied for many years. The Hidden Markov Model for RGB cameras are widely utilized in many publications to encode the sequential changes of the features. In [41], the authors designed a new feature to capture the relations between human body and the objects in the environment for interaction. In [42], an online action recognition system is developed to process video streams. They represent the concept of ”action points” which indicates the natural temporal anchors of simple human actions. In [43], a human activity recognition benchmark database is developed, which includes 12 different human domestic daily activities recorded by Kinect. Other works aim to apply skeletal tracking to different scenarios. In these works, heuristic rules are defined for a given application depending on the context. In [44], an algorithm is represented to generate musical notes based on the motion of body parts. In [45], Kinect is used to help the examination of dancing gestures. Also, in [46] an action recognition algorithm is presented for an automated performance assessment system in cluttered and dynamic indoor construction environments. Compared with their work, our novelty is that our algorithm is able to customize a sequence of training material based on the user’s current learning status in real time, and this algorithm is suitable for any physical training edutainment games. 3. Model definition and parameters The first challenge of integrating HMMs as algorithms for multimodal interaction scenario is to map the features and states to the actual human-machine interaction model. In our case, the bottom layer of the HMM will evaluate the similarity of a sequence of human poses collected by the Kinect with the standard sequence provided by the course material. A complete sequence of postures can be separated into several fragments. Every time a sequence of input motions is processed by the bottom layer, it is sent to the top layer HMM. The top level HMM consider potential combinations of the user’s learning status, and generate sequence of a combination of learning materials, which is also known as a learning patch. There are several advantages of this hierarchical HMM algorithm. First, the HMM models do not require a huge pre-stored dataset since the motions of the training course is predefined. We can use hierarchical HMM to generate quantitative analysis of the user’s current performance and generate a specific sequence of the combination from the selected learning material. Secondly, the processing time can satisfy the requirement of real-time for our case. Since the combination of learning material grows exponentially, the 2-layer hierarchical HMM is able to choose partially from all combinations step by step with the progress of the procedure. This may remove a huge amount of unnecessary combinations before the customized path generation step. Thirdly, the scalability of the 2-layer hierarchical HMM algorithm is commendable. The proposed edutainment game can be easily updated by adding new material. Furthermore, by replacing the action set and the material set, the algorithm can be adapted to any physical training edutainment games based on Kinect. In this section, a 2-layer hierarchical HMM is defined as follows. A single-layer Markov chain can be defined as a 5-tuple: k ¼ ðN; M; A; B; pÞ. N is the total number of states in p. M is the number of different observation in every state of k. A is the transi-
396
tion
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401
matrix
of
n o N N; A ¼ aij ; aij ¼ P qtnþ1 ¼j jqt¼j ,
where
1 6 i; j 6 N; aij P 0. The N M matrix B is defined as B ¼ bj ðkÞ, where bj ðkÞ ¼ P ½ot ¼ v t jqt ¼ jð1 6 i 6 M Þ. bj is the possibility of observation when ot ¼ v k at state j. The 1 N matrix p is defined as p ¼ fpi g, where pi ¼ P ½q1 ¼ ið1 6 i 6 N Þ. This means in a random sequence, the possibility of first state starts at state i. The architecture of the proposed 2-layer HMM model is shown in Fig. 1. The low level of the HMM model consists of numbers of motion recognition modules, each of which recognizes a single movement of the user. In every time step, the lower level recognizes a sequence of user motion through the Kinect motion recognition modules. The top level of the HMM model generate subsequential learning patches based on the evaluation of the recognition results of bottom level. The observation sequence of the lower level is defined as:
oðtÞ ¼ fo1 ðtÞ; o2 ðt Þ; . . . ; oN ðtÞg
ð1Þ
For each oi ðtÞ, it indicates the number of recognized motion i. The equations for the forward variable at and the backward variable bt are revised as follows:
atþ1 ¼
k N X Y at ðiÞaij bj of ðt þ 1Þ i1
btþ1 ¼
ð2Þ
f 1
k N X Y bt ðiÞaij bj of ðt þ 1Þ i1
ð3Þ
f 1
The new forward variable at ðiÞ is the output possibility at time step t of the observation sequence for the forward variable algorithm; bt ðiÞ is the output possibility at time step t of the observation sequence for the backward variable algorithm; of ðt Þ is the variable in the observation sequence. f The confusion matrix B is revised as follows, where correct k jj indicates the total number of corrected motions for the motion recognition sequence sj , and correctð jÞ is the expected value of the correct motion for the motion recognition sequence. f bj ðkÞ
¼
f correct k jj correctð jÞ
ð4Þ
4. The dynamic learning patch generation An example of the training process generated by the dynamic training path generation is shown as below. In Fig. 2, the lower level of HMM chain recognizes each motion from 1 to 4. According to the recognition result, motion 1 and 3 are consistent with the correct motions, while motion 2 and motion 4 are not correct. Therefore, the training path needs to add new contents into original training process in order to correct the wrong motions. Fig. 3 shows the newly generated training path from the original path. The original training process in our example only has one training path. The algorithm generates new training pathes at each node of the path according to the current learning status of the trainee. The nodes in the training path are predefined before the training process. At each new path, the supplementary learning contents like video demonstrations, audio explanations, or texts can be added to the course. For physical training process, some strengthened training course for certain movement can also be added during the process. After the training content is finished, the training path may go back to the node of the original training path, or may skip some contents and start a new path. For this example, video demonstrations and text explanations are added into the original path in order to strengthens the training process. 5. Application in a gaming system
Fig. 1. The architecture of proposed 2-layer HMM model.
The proposed Kinect-based training method is utilized in an edutainment game for children we have developed. The game is mainly focused on the children in elementary schools. The content of the game is teaching children of Chinese culture by acting as an ancient Chinese heroine. Children will go through several scenes that requiring them to learn skills including operating looms, shooting arrow, riding horses, controlling a panel and etc. The performance of the student is evaluated automatically by the gaming
Fig. 2. An example of motion recognition.
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401
397
Fig. 3. New training path generated from the original path.
system. If the student does not perform well, the game will provide additional teaching material such as video demonstrations, or zoomed-in details of the actions in order to help them understanding their errors. Therefore, the quality of learning is not only depends on the performance of a individual gesture or body movement, but also with the consistency and accuracy of the whole progression. For example, for the learning of operating looms, the evaluation without proposed HMM algorithm only consider the final posture of the user’s body, like stretching arms. However, if the user’s posture is distorted during the middle, it is very hard for the teaching system to detect his movement flaws. In our edutainment game, the system may realizes that the user does not study this action will by detecting the hesitation between continuous movements. And when this happens, it will provide a video with slowed movement
and specific arrows showing in the screen in order to make the user understand the whole sequence of movement better. The demonstration of the gaming system is shown in Fig. 4. We show an example here to illustration the idea of the utilization of our proposed algorithm in the last section. In the scene of ‘‘How to use a loom?”, the student is asked to virtually operating the loom in the correct way. If the performance is agreed with the gaming system, the student will move onto the next gaming scene. If the system notices some obvious mistake, it will add additional learning material to the students according to the kind of error. In Fig. 5, the student was turning his head aside when he was weaving. The system recognized that motion and notified the student to watch the loom when he was weaving by providing an video to him.
Fig. 4. Demonstration of the edutainment game.
398
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401
Fig. 5. The path-choosing process.
6. User study and evaluation 6.1. Configuration of the user study There are totally 10 Participants in the user study (5 males and 5 females), aged between 6 and 15 (average age = 10.7 years), with no prior knowledge or experience of the system. Based on the basis of the participants, an expert assessor is added to evaluate the learning of each participant. This experiments include four movements, which are divided into two groups. Each group composed by a simple action and a complex action among them. Simple actions only need moderate exercise, while complex actions are composed of a series of poses. The two condition for user study are the Kinect-based approach and video-based approach. Instructional video is the recording of the relevant actions, instruction, and video. The experimental environment, experimental equipment and other conditions are consistent [47]. The evaluation was designed as follows: the test was conducted by a double factor repetitive test. Each participant needed to learn a set of actions using the Kinect-based edutainment game. The training procedure is to learn a set of simple actions and a set of complicated actions. Considering the learning order of different type of actions may have some side effects, the number and order of the simple actions and complicated actions in the action set is balanced. The action sets consist of: (1) learning to use mechanics; (2) shooting arrow; (3) special consistent movements; (4) controlling panels. One action is composed of a sequence of continuous
gestures and body movements, and the evaluation of the quality of learning certain action is to compare the whole sequence of user’s movements with the standard movement sequence. Participants took part in an evaluation test after completing the training. The test includes watching a demo video to review what you learned, and then do five exercises by the instructions. After that, participants were given a short questionnaire to collect feedback from participants on the system. The questionnaire is in Appendix A.
6.2. Evaluation results Each time the action is completed on the Kinect based action platform, the system records its recognition results. Upon completion of each action, the expert evaluation will be based on the number of participants to determine the degree of completion of the participants’ movements. According to the professional reviewers, the results of each participant’s performance demonstration were analysis of variance. And according to the correct number of recognition reflected by the system, 10 participants were trained in each way to determine the percentage of the correct number under each condition. There are two independent variants in the analysis of variance: the learning condition and the type of movements. According to the evaluation of professional reviewers and the completion of the full set of movements, we compute the root mean square error (RMSE) of the system. The results are shown in Fig. 6.
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401
399
Fig. 6. RMSE curve of four tests.
In Fig. 6, the video-based method’s score improved 0.55, which is 20% higher after the training. The improved score of proposed Kinect-based method is 1.10, which is 44% higher. The results of different types of movements are almost the same, which indicates the effectiveness of the proposed Kinect-based training method does not rely on the difficulty of the movements. To be mentioned that for some cases in Fig. 6, the result of Kinect-based training showed lower performance compared with video-base methods. In these cases, there are one obvious factor in common: f the kinect-based method provide only text supplementary material. Since the age of the participants are comparable young, they may have some misunderstanding with the text. This can be improved by changing the supplementary material with more detailed videos and figures. In Fig. 7, we compare the accuracy after learning in 1 min (first column of each type) and the deteriorated accuracy after learning
in 4 h (second column of each type) for three different training method: video, game without HMM customized patches, and game with HMM customized patches. It is shown in the figure that both of the two edutainment games have a better teaching performance compared with the traditional video teaching method. However, when the game is applied with HMM customized patches, the accuracy improves much better (average 10%), and the accuracy deteriorates much slower than the game without HMMs. In conclusion, the edutainment game has a better teaching performance compared with the traditional video-based method.
7. Conclusion In this paper, we propose a Kinect-based training technique for physical education. An HMM-based algorithm for generating cus-
Fig. 7. Recognition rate of 10 participants during the test.
400
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401
tomized learning patch for individuals is proposed. We present an edutainment gaming system utilizing the proposed traing method to illustrate the idea. The user study and evaluation shows that the effect of Kinect-based training method is much better than the traditional video-base method. Acknowledgements This work was supported by the National Natural Science Foundation of China [Grant No. 61602421] and the China Postdoctoral Science Foundation [Grant No. 2016M600584]. Appendix A A.1. Questionnaire of the user study See Tables 1–3.
Table 1 Questionnaire for the User Study: Table 1. Questionnaire for the User Study: Table 1 Hello! Thank you for your support of this questionnaire survey We will cherish each of your suggestions. Please fill in them carefully according to the actual situation Please fill in the brackets at the front of the question according to the real experience of the teaching and entertainment system by choosing the right answers from the following options 1. Which teaching method do you think has a better classroom atmosphere? A Traditional Method B Educational Game C Same 2. Which teaching method do you think is more interesting? A. Traditional Method B. Educational Game C. Same 3. Which teaching method do you think is more participatory? A Traditional Method B Educational Game C Same 4. Which teaching method would you like to try when you are learning? A Traditional Method B Educational Game C does not matter 5. Which teaching method are you more accustomed to when you are learning? A Traditional Method B Educational Game C Same 6. Which teaching method makes you feel more comfortable when you are learning? A Traditional Method B Educational Game C Same 7. Which teaching method can stimulate your interest in learning better? A Traditional Method B Educational Game C Same 8. Which teaching method do you think is more interactive? A Traditional Method B Educational Game C does not matter
Table 2 Questionnaire for the User Study: Table 2. Questionnaire for the User Study: Table 2 Are you satisfied with the efficiency of the teaching system? Are you satisfied with the simplicity of the operation of this teaching system? Are you satisfied with the interestingness of the teaching system? Are you satisfied with the effect of this teaching system on learning? Are you satisfied with the fun of the ”Edutainment Game” section? How much do you think the part of ”Edutainment Games” has an impact on learning knowledge? Are you satisfied with the action design of ”Edutainment interactive game”? Are you satisfied with the interface color of the teaching system? Are you satisfied with the interface layout of the teaching system? Are you satisfied with the systematic error correction ability of teaching?
Table 3 Questionnaire for the User Study: Table 3. Questionnaire for the User Study: Table 3 How important do you think the following factors are when learning knowledge in the Edutainment game system? Please give the importance score of each factor according to your understanding [Very important 5, relatively important 4, generally important 3, not very important 2, not important at all 1] This system will be useful to me This system is more interesting This system can achieve learning To be able to make it easier for me to learn how to use the system The operation mode of the system should conform to my operation habits When using this system, I think it’s relatively simple When using this system, I want to feel more comfortable When operating this system, it is more efficient When using this system, I need to be able to easily find the information I need The information provided by the system should be easy for me to understand The system should be able to respond to my instructions in a timely manner The system should be able to respond to my instructions accurately The system should be able to prompt me in time for operational errors The system should be able to tell me how to correct errors in time The interface layout of the system should be clear The interface color of the system should make me feel comfortable Finally, do you have any suggestions for this interactive system? Thank you again for your participation!
References [1] Kunwar Aditya, Recent trends in hci: a survey on data glove, leap motion and microsoft kinect, in: 2018 IEEE International Conference on System, Computation, Automation and Networking (ICSCA), IEEE, 2018, pp. 1–5. [2] Fraser Anderson, Tovi Grossman, Justin Matejka, George Fitzmaurice, Youmove: enhancing movement training with an augmented reality mirror, in: ACM Symposium on User Interface Software and Technology, ACM, 2013, pp. 311–320. [3] Urban Burnik, Janez Zaletelj, Andrej Košir, Kinect based system for student engagement monitoring, in: 2017 IEEE First Ukraine Conference on Electrical and Computer Engineering (UKRCON), IEEE, 2017, pp. 1229–1232. [4] Ziyun Cai, Jungong Han, Li Liu, Ling Shao, Rgb-d datasets using microsoft kinect or similar sensors: a survey, Multimedia Tools Appl. 76 (3) (2017) 4313–4355. [5] Dennis Charsky, From edutainment to serious games: a change in the use of game characteristics, Games Culture 5 (2) (2010) 177–198. [6] Koen de Greef, Erik D. Van der Spek, Tilde Bekker, Designing kinect games to train motor skills for mixed ability players, in: Games for Health, Springer, 2013, pp. 197–205. [7] Sara Reisi Dehkordi, Marina Ismail, Norizan Mat Diah, A review of kinect computing research in education and rehabilitation, Int. J. Eng. Technol. (UAE) 7 (3) (2018) 19–23. [8] José Rodrigues Dias, Rui Penha, Leonel Morgado, Pedro Alves da Veiga, Elizabeth Simão Carvalho, Adérito Fernandes-Marcos, Tele-media-art: Feasibility tests of web-based dance education for the blind using kinect and sound synthesis of motion, Int. J. Technol. Human Interact. (IJTHI) 15 (2) (2019) 11–28. [9] Emily DiGiovanna, Michael Lamba, Peter Sandwall, Survey of kinect v2 applied to radiotherapy patient positioning, Cancer Therapy Oncol. Int. J. 8 (1) (2017). [10] Gabriele Fanelli, Thibaut Weise, Juergen Gall, Luc Van Gool, Real time head pose estimation from consumer depth cameras, in: Joint Pattern Recognition Symposium, Springer, 2011, pp. 101–110. [11] Luigi Gallo, Alessio Pierluigi Placitelli, Mario Ciampi, Controller-free exploration of medical image data: experiencing the kinect, in: International Symposium on Computer-Based Medical Systems, IEEE, 2011, pp. 1–6. [12] Zengguo Ge, Li Fan, Social development for children with autism using kinect gesture games: a case study in Suzhou Industrial Park Renai School, in: Simulation and Serious Games for Education, Springer, 2017, pp. 113–123. [13] J. Hollan, E. Hutchins, D. Kirsh, Distributed cognition: towards a new foundation for hci research, ACM Trans. Comput.-Hum. Interact. 7 (2000) 174–196. [14] J.D. Huang, Kinerehab: a kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities, in: The Proceedings of the International ACM Sigaccess Conference on Computers and Accessibility, 2011, pp. 319–320. [15] Edwin Hutchins, How a cockpit remembers its speeds, Cogn. Sci. 19 (3) (1995) 265–288.
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401 [16] Aamrah Ikram, Yue Liu, A survey on dynamic hand gesture recognition using kinect device, in: Chinese Conference on Image and Graphics Technologies, Springer, 2018, pp. 635–646. [17] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al., Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera, in: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, ACM, 2011, pp. 559–568. [18] Mun Ho Jeong, Yoshinori Kuno, Nobutaka Shimada, Yoshiaki Shirai, Complex gesture recognition using coupled switching linear model, in: Proc. 5th Asian Conference on Computer Vision. Citeseer, 2002, pp. 132–137. [19] Stefan Koehn, Effects of confidence and anxiety on flow state in competition, Eur. J. Sport Sci. 13 (5) (2013) 543–550. [20] Belinda Lange, Chien-Yen Chang, Evan Suma, Bradley Newman, Albert Skip Rizzo, Mark Bolas, Development and evaluation of low cost game-based balance rehabilitation tool using the microsoft kinect sensor, in: Engineering in Medicine and Biology Society. Embc, 2011, pp. 1831–1834. [21] Greg C. Lee, Fu-Hao Yeh, Yi-Han Hsiao, Kinect-based taiwanese sign-language recognition system, Multimedia Tools Appl. 75 (1) (2016) 261–279. [22] Billy Y.L. Li, Mingliang Xue, Ajmal S. Mian, Wanquan Liu, Aneesh Krishna, Robust RGB-D face recognition using kinect sensor, Neurocomputing 214 (2016) 93–108. [23] Lu Xuequan, Xu. Mingliang, Wenzhi Chen, Zonghui Wang, Abdennour El Rhalibi, Adaptive-ar model with drivers’ prediction for traffic simulation, Int. J. Comput. Games Technol. 1–8 (2013) 2013. [24] Roanna Lun, Wenbing Zhao, A survey of using microsoft kinect in healthcare, in: Consumer-Driven Technologies in Healthcare: Breakthroughs in Research and Practice, IGI Global, 2019, pp. 445–456. [25] Tianlu Mao, Hu.a. Wang, Zhigang Deng, Zhaoqi Wang, An efficient lane model for complex traffic simulation, Comput. Animat. Virtual Worlds 26 (3–4) (2015) 397–403. [26] Richard E Mayer, Roxana Moreno, Animation as an aid to multimedia learning, Educ. Psychol. Rev. 14 (1) (2002) 87–99. [27] Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, Andrew Blake, Real-time human pose recognition in parts from single depth images, in: Computer Vision and Pattern Recognition, 2011. [28] Jan Smisek, Michal Jancosek, Tomas Pajdla, 3d with kinect, in: Consumer Depth Cameras for Computer Vision, Springer, 2013, pp. 3–25. [29] Milka Trajkova, Mexhid Ferati, Usability evaluation of kinect-based system for ballet movements, in: Springer International Publishing, Springer, 2015. [30] Christina Vasiliou, Andri Ioannou, Panayiotis Zaphiris, Understanding collaborative learning activities in an information ecology: a distributed cognition account, Comput. Hum. Behav. 41 (2014) 544–553. [31] César Villacís, Walter Fuertes, Andrés Bustamante, Daniel Almachi, Carlos Procel, Susana Fuertes, Theofilos Toulkeridis, Multi-player educational video game over cloud to stimulate logical reasoning of children, in: Proceedings of the 2014 IEEE/ACM 18th International Symposium on Distributed Simulation and Real Time Applications, IEEE Computer Society, 2014, pp. 129–137.
401
[32] H. Wang, X. Kang, Tianlu Mao, Z. Wang, A semantic model of road networks for traffic animations, Jisuanji Fuzhu Sheji Yu Tuxingxue Xuebao/J. Comput.-Aided Des. Comput. Graph. 26 (2014) 1818–1826. [33] H. Wang, M. Xu, Tianlu Mao, X. Jin, Z. Wang, Survey of three-dimension traffic animation, Jisuanji Fuzhu Sheji Yu Tuxingxue Xuebao/J. Comput.-Aided Des. Comput. Graph. 29 (2017) 211–220. [34] Hu.a. Wang, Tianlu Mao, Xingchen Kang, Zhaoqi Wang, An all-in-one efficient lane-changing model for virtual traffic, Comput. Animat. Virtual Worlds 25 (3– 4) (2014) 383–391. [35] Hu.a. Wang, Xu. Mingliang, Fubao Zhu, Zhigang Deng, Yafei Li, Bing Zhou, Shadow traffic: a unified model for abnormal traffic behavior simulation, Comput. Graph. 70 (2017). [36] Lu Xia, Chia-Chih Chen, Jake K Aggarwal, Human detection using depth information by kinect, in: CVPR 2011 Workshops, IEEE, 2011, pp. 15–22. [37] Xu. Mingliang, Hao Jiang, Xiaogang Jin, Zhigang Deng, Crowd simulation and its applications: recent advances, J. Comput. Sci. Technol. 29 (5) (2014) 799– 811. [38] Xu. Mingliang, Zhigeng Pan, Mingmin Zhang, Pei Lv, Pengyu Zhu, Yangdong Ye, Wei Song, Character behavior planning and visual simulation in virtual 3d space, IEEE MultiMedia 20 (1) (2013) 49–59. [39] Xu. Mingliang, Hu.a. Wang, Shili Chu, Yong Gan, Xiaoheng Jiang, Yafei Li, Bing Zhou, Traffic simulation and visual verification in smog, TIST 10 (2019) 1–17. [40] Erman Yukselturk, Serhat Altıok, Zeynep Basßer, Using game-based learning with kinect technology in foreign language education course, J. Educ. Technol. Soc. 21 (3) (2018) 159–173. [41] Tamara Berg, Debaleena Chattopadhyay, Margaret Schedel, Timothy Vallier, Interactive music: human motion initiated music generation using skeletal tracking by kinect, in: Proc. Conf. Soc. Electro-Acoustic Music United States, 2012. [42] Zhongwei Cheng, Lei Qin, Yituo Ye, Qingming Huang, Qi Tian, Human daily action analysis with multi-view and color-depth data, in: European Conference on Computer Vision, Springer, 2012, pp. 52–61. [43] Simon Fothergill, Helena Mentis, Pushmeet Kohli, Sebastian Nowozin, Instructing people for training gestural interactive systems, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2012, pp. 1737–1746. [44] Bingbing Ni, Gang Wang, Pierre Moulin, Rgbd-hudaact: a color-depth video database for human daily activity recognition, in: 2011 IEEE international conference on computer vision workshops (ICCV workshops), IEEE, 2011, pp. 1147–1153. [45] Sebastian Nowozin, Jamie Shotton, Action points: A representation for lowlatency online human action recognition. Microsoft Research Cambridge, Tech. Rep. MSR-TR-2012-68, 2012. [46] Michalis Raptis, Darko Kirovski, Hugues Hoppe, Real-time classification of dance gestures from skeleton animation, in: 2011 ACM SIGGRAPH/ Eurographics Symposium on Computer Animation, ACM, 2011, pp. 147–156. [47] Jiawei Han, Zhiguo Xiao, Lingeng Han, Ying Xu, Nianfeng Li, Sato Reika, A special edutainment system based on somatosensory game, in: IEEE International Conference on Software Engineering & Service Science, 2015.
Contents lists available at ScienceDirect
J. Vis. Commun. Image R. journal homepage: www.elsevier.com/locate/jvci
Personalized training through Kinect-based games for physical education Mingliang Xu a, Yafang Zhai a, Yibo Guo a,⇑, Pei Lv a, Yafei Li a, Meng Wang b, Bing Zhou a a b
Center for Interdisciplinary Infromation Sciences, Zhengzhou University, Zhengzhou, China School of Computer and Information, Hefei University of Technology, Hefei, China
a r t i c l e
i n f o
Article history: Received 18 December 2017 Revised 18 March 2019 Accepted 17 May 2019 Available online 20 June 2019 Keywords: Kinect Educational games
a b s t r a c t In recent years, the Kinect-based systems that enable users to be trained without the participation of teachers have been widely used in the field of physical education. In this paper, we propose a novel technique that helps the Kinect-based training system to select the subsequential training material for the users according to their realtime performance. An algorithm based on the Hidden Markov Model is demonstrated to generate the customized training pathes(training curriculums) for each individual. We present an edutainment gaming system for children in order to illustrate the feasibility of the training method. A user study of 10 children participants is conducted and the results show that the proposed technique enhances the effect of physical training significantly. Ó 2019 Elsevier Inc. All rights reserved.
1. Introduction The traditional sports training or other physical education requires trainees to participate in the training courses under the guidance of the teaching staff [1,5,3,7]. The trainers need to evaluate the trainees’ current performance and customize subsequential courses according to the learning ability of the trainees. Since the number of the teaching staff is much smaller than the trainees, the opportunity of communicating with the trainer is not always available for everyone [2,3,7,5]. In recent years, the educational videos for physical training have thrived in the market, which completely removes human trainers during the teaching process [6,8–10,30]. Although it solved the problem of insufficient trainers, the quality of the teaching process cannot be guaranteed due to the lack of interactions. The misunderstanding of video contents and the incorrect imitated movements are not fed back to the trainees in real time. Therefore, how to improve the quality of physical education without the participation of the teaching staff still needs to be addressed in more detail by both scientific and education community [11–15]. In the last few years, the gaming technologies have been improved enormously with the development of enhanced human-machine interface. The novel interaction methods are comparably cheaper since the invention of low-costed sensors, such as Microsoft Kinect, are popularized in the field of computer vision [16–18]. Many works that utilizes Kinect sensors on interactive ⇑ Corresponding author. E-mail address: [email protected] (Y. Guo). https://doi.org/10.1016/j.jvcir.2019.05.007 1047-3203/Ó 2019 Elsevier Inc. All rights reserved.
physical educations have been proposed [4,13–16]. Most of the current methods identify the trainee’s physical movements, generate feedbacks for trainees according to the degree of conformity with the actual actions, and expect the users to correct their mistakes by themselves [19,21,20]. Compared with the video training method, this kinect-based training systems strengthen the effect of self-teaching effectively. Since each trainee may not be at the same level during the training process, it is necessary to redesign customized learning content for each individual [22–24]. In this article, we present a novel technique for physical education systems based on Kinect, which generate customized learning patches (a bunch of selected learning material) for each individual according to their current learning status [25,26]. In physical training courses, the trainee’s performance is evaluated by his reaction to the teaching material. Therefore, in many physical edutainment games, Kinect is used to collect trainee’s physical activities [27–30]. To estimate human actions and intentions, the common method is to rebuild the pose of the human skeleton based on the image captured by the camera. There are many ways of detecting and tracking human joint points in motion [31–34]. In order to recognize the precise human gesture, three approaches are commonly used depending on its application: template mapping, semantic approach, and statistical approach. The template mapping method compares the input image with stored image pixel-wisely, and therefore it is not suitable for realistic figures with distortion or noises [33,35]. The statistical approaches requires a huge set of images collected in advance, which might be too costly for some simple matching occurrences [28,36,37,24,27]. The semantic approaches, such as Hidden Markov
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401
Models (HMM), recognize the relations between object components, have been proved to be an efficient real-time method for the context-aware situations [38,39,16,18]. As for the physical training courses, the target movements and postures are predefined by the trainers. The similarity of a sequence of user postures compared with the predefined procedure can be precisely evaluated by the HMM models [17,40]. In this article, we need to acquire the best training material for the trainee depending on his completeness of the current course. Therefore, a 2-layer hierarchical HMM model is designed in order to evaluate the current learning status, and choose the most suitable material (vedio or text) for the trainee. The advantage of 2-layer hierarchical HMM model is that we firstly evaluate a sequence of the users’ activities, and then customize the learning patch for different trainees based their unique performance in real time. The contributions of this paper include: We design an algorithm with 2-layer hierarchical Hidden Markov Model that generates customized subsequential learning contents for each individual according to their current performance. We develop an educational gaming system for children utilized the proposed algorithm. A user study is conducted and the results showed that the proposed algorithm has improved the learning effect promisingly. The rest of this paper is organized as follows: Background and related works are provided in Section 2. The 2-layer hieratical HMM algorithm is introduced in Section 3. The algorithm of generating learning patch for Kinect-based systems is proposed in Section 4. The educational gaming system for children utilized the proposed algorithm is demonstrated in Section 5. The user study is presented in Section 6. The paper is concluded in Section 7.
2. Related work 2.1. Serious games The serious gaming technology has been widely adopted in multiple educational fields. The creation of a serious game is aimed to teach professional skills ranging from military to education. The word ‘‘edutainment” is defined as a continuous and innovative brain-training, which stimulates, in an interactive way, the capacity to combine attention and motivation to explore and learn [5–8]. Kinect sensors have been proved to be a cheap and efficient way of human machine interaction [9,10]. The Kinect sensors allow the machine to detect the depth information of the players and the environment as well as the speech instructions [8,9]. The trajectory of the player is interpreted in real time, and is translated to a format to rebuild for new experiences [11,21]. With the 3-D human motion capturing and speech recognition algorithm of the Kinect system, the players are enabled to interact with the game console without touching the controller. The recent development of Kinect technologies have been applied to both situational education and personalized education [17–21]. Among them, Kinect-based games have been noticed as the main field of the application in education. and its development provides more possibilities for edutainment learning. According to the previous works [13,19–21], educational games can foster users’ knowledge, skills, intelligence, emotions, attitudes and values [1,2,5–7]. In [5], the authors discuss the feasibility of edutainment techniques, and believe that edutainment games may increase learners’ intrinsic learning motivation. Because of the requirement of utilizing all the senses of the learner, it showed a great advantage in strengthening and stimulating the focus during the process
395
of learning. It also contributes to the reinforce the learners’ persistence and the sense of immersion [20,21]. 2.2. Pose and activity estimation for Kinect Recognizing semantic human poses and activities given the skeletal joints by Kinect in the context of applications have been studied for many years. The Hidden Markov Model for RGB cameras are widely utilized in many publications to encode the sequential changes of the features. In [41], the authors designed a new feature to capture the relations between human body and the objects in the environment for interaction. In [42], an online action recognition system is developed to process video streams. They represent the concept of ”action points” which indicates the natural temporal anchors of simple human actions. In [43], a human activity recognition benchmark database is developed, which includes 12 different human domestic daily activities recorded by Kinect. Other works aim to apply skeletal tracking to different scenarios. In these works, heuristic rules are defined for a given application depending on the context. In [44], an algorithm is represented to generate musical notes based on the motion of body parts. In [45], Kinect is used to help the examination of dancing gestures. Also, in [46] an action recognition algorithm is presented for an automated performance assessment system in cluttered and dynamic indoor construction environments. Compared with their work, our novelty is that our algorithm is able to customize a sequence of training material based on the user’s current learning status in real time, and this algorithm is suitable for any physical training edutainment games. 3. Model definition and parameters The first challenge of integrating HMMs as algorithms for multimodal interaction scenario is to map the features and states to the actual human-machine interaction model. In our case, the bottom layer of the HMM will evaluate the similarity of a sequence of human poses collected by the Kinect with the standard sequence provided by the course material. A complete sequence of postures can be separated into several fragments. Every time a sequence of input motions is processed by the bottom layer, it is sent to the top layer HMM. The top level HMM consider potential combinations of the user’s learning status, and generate sequence of a combination of learning materials, which is also known as a learning patch. There are several advantages of this hierarchical HMM algorithm. First, the HMM models do not require a huge pre-stored dataset since the motions of the training course is predefined. We can use hierarchical HMM to generate quantitative analysis of the user’s current performance and generate a specific sequence of the combination from the selected learning material. Secondly, the processing time can satisfy the requirement of real-time for our case. Since the combination of learning material grows exponentially, the 2-layer hierarchical HMM is able to choose partially from all combinations step by step with the progress of the procedure. This may remove a huge amount of unnecessary combinations before the customized path generation step. Thirdly, the scalability of the 2-layer hierarchical HMM algorithm is commendable. The proposed edutainment game can be easily updated by adding new material. Furthermore, by replacing the action set and the material set, the algorithm can be adapted to any physical training edutainment games based on Kinect. In this section, a 2-layer hierarchical HMM is defined as follows. A single-layer Markov chain can be defined as a 5-tuple: k ¼ ðN; M; A; B; pÞ. N is the total number of states in p. M is the number of different observation in every state of k. A is the transi-
396
tion
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401
matrix
of
n o N N; A ¼ aij ; aij ¼ P qtnþ1 ¼j jqt¼j ,
where
1 6 i; j 6 N; aij P 0. The N M matrix B is defined as B ¼ bj ðkÞ, where bj ðkÞ ¼ P ½ot ¼ v t jqt ¼ jð1 6 i 6 M Þ. bj is the possibility of observation when ot ¼ v k at state j. The 1 N matrix p is defined as p ¼ fpi g, where pi ¼ P ½q1 ¼ ið1 6 i 6 N Þ. This means in a random sequence, the possibility of first state starts at state i. The architecture of the proposed 2-layer HMM model is shown in Fig. 1. The low level of the HMM model consists of numbers of motion recognition modules, each of which recognizes a single movement of the user. In every time step, the lower level recognizes a sequence of user motion through the Kinect motion recognition modules. The top level of the HMM model generate subsequential learning patches based on the evaluation of the recognition results of bottom level. The observation sequence of the lower level is defined as:
oðtÞ ¼ fo1 ðtÞ; o2 ðt Þ; . . . ; oN ðtÞg
ð1Þ
For each oi ðtÞ, it indicates the number of recognized motion i. The equations for the forward variable at and the backward variable bt are revised as follows:
atþ1 ¼
k N X Y at ðiÞaij bj of ðt þ 1Þ i1
btþ1 ¼
ð2Þ
f 1
k N X Y bt ðiÞaij bj of ðt þ 1Þ i1
ð3Þ
f 1
The new forward variable at ðiÞ is the output possibility at time step t of the observation sequence for the forward variable algorithm; bt ðiÞ is the output possibility at time step t of the observation sequence for the backward variable algorithm; of ðt Þ is the variable in the observation sequence. f The confusion matrix B is revised as follows, where correct k jj indicates the total number of corrected motions for the motion recognition sequence sj , and correctð jÞ is the expected value of the correct motion for the motion recognition sequence. f bj ðkÞ
¼
f correct k jj correctð jÞ
ð4Þ
4. The dynamic learning patch generation An example of the training process generated by the dynamic training path generation is shown as below. In Fig. 2, the lower level of HMM chain recognizes each motion from 1 to 4. According to the recognition result, motion 1 and 3 are consistent with the correct motions, while motion 2 and motion 4 are not correct. Therefore, the training path needs to add new contents into original training process in order to correct the wrong motions. Fig. 3 shows the newly generated training path from the original path. The original training process in our example only has one training path. The algorithm generates new training pathes at each node of the path according to the current learning status of the trainee. The nodes in the training path are predefined before the training process. At each new path, the supplementary learning contents like video demonstrations, audio explanations, or texts can be added to the course. For physical training process, some strengthened training course for certain movement can also be added during the process. After the training content is finished, the training path may go back to the node of the original training path, or may skip some contents and start a new path. For this example, video demonstrations and text explanations are added into the original path in order to strengthens the training process. 5. Application in a gaming system
Fig. 1. The architecture of proposed 2-layer HMM model.
The proposed Kinect-based training method is utilized in an edutainment game for children we have developed. The game is mainly focused on the children in elementary schools. The content of the game is teaching children of Chinese culture by acting as an ancient Chinese heroine. Children will go through several scenes that requiring them to learn skills including operating looms, shooting arrow, riding horses, controlling a panel and etc. The performance of the student is evaluated automatically by the gaming
Fig. 2. An example of motion recognition.
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401
397
Fig. 3. New training path generated from the original path.
system. If the student does not perform well, the game will provide additional teaching material such as video demonstrations, or zoomed-in details of the actions in order to help them understanding their errors. Therefore, the quality of learning is not only depends on the performance of a individual gesture or body movement, but also with the consistency and accuracy of the whole progression. For example, for the learning of operating looms, the evaluation without proposed HMM algorithm only consider the final posture of the user’s body, like stretching arms. However, if the user’s posture is distorted during the middle, it is very hard for the teaching system to detect his movement flaws. In our edutainment game, the system may realizes that the user does not study this action will by detecting the hesitation between continuous movements. And when this happens, it will provide a video with slowed movement
and specific arrows showing in the screen in order to make the user understand the whole sequence of movement better. The demonstration of the gaming system is shown in Fig. 4. We show an example here to illustration the idea of the utilization of our proposed algorithm in the last section. In the scene of ‘‘How to use a loom?”, the student is asked to virtually operating the loom in the correct way. If the performance is agreed with the gaming system, the student will move onto the next gaming scene. If the system notices some obvious mistake, it will add additional learning material to the students according to the kind of error. In Fig. 5, the student was turning his head aside when he was weaving. The system recognized that motion and notified the student to watch the loom when he was weaving by providing an video to him.
Fig. 4. Demonstration of the edutainment game.
398
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401
Fig. 5. The path-choosing process.
6. User study and evaluation 6.1. Configuration of the user study There are totally 10 Participants in the user study (5 males and 5 females), aged between 6 and 15 (average age = 10.7 years), with no prior knowledge or experience of the system. Based on the basis of the participants, an expert assessor is added to evaluate the learning of each participant. This experiments include four movements, which are divided into two groups. Each group composed by a simple action and a complex action among them. Simple actions only need moderate exercise, while complex actions are composed of a series of poses. The two condition for user study are the Kinect-based approach and video-based approach. Instructional video is the recording of the relevant actions, instruction, and video. The experimental environment, experimental equipment and other conditions are consistent [47]. The evaluation was designed as follows: the test was conducted by a double factor repetitive test. Each participant needed to learn a set of actions using the Kinect-based edutainment game. The training procedure is to learn a set of simple actions and a set of complicated actions. Considering the learning order of different type of actions may have some side effects, the number and order of the simple actions and complicated actions in the action set is balanced. The action sets consist of: (1) learning to use mechanics; (2) shooting arrow; (3) special consistent movements; (4) controlling panels. One action is composed of a sequence of continuous
gestures and body movements, and the evaluation of the quality of learning certain action is to compare the whole sequence of user’s movements with the standard movement sequence. Participants took part in an evaluation test after completing the training. The test includes watching a demo video to review what you learned, and then do five exercises by the instructions. After that, participants were given a short questionnaire to collect feedback from participants on the system. The questionnaire is in Appendix A.
6.2. Evaluation results Each time the action is completed on the Kinect based action platform, the system records its recognition results. Upon completion of each action, the expert evaluation will be based on the number of participants to determine the degree of completion of the participants’ movements. According to the professional reviewers, the results of each participant’s performance demonstration were analysis of variance. And according to the correct number of recognition reflected by the system, 10 participants were trained in each way to determine the percentage of the correct number under each condition. There are two independent variants in the analysis of variance: the learning condition and the type of movements. According to the evaluation of professional reviewers and the completion of the full set of movements, we compute the root mean square error (RMSE) of the system. The results are shown in Fig. 6.
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401
399
Fig. 6. RMSE curve of four tests.
In Fig. 6, the video-based method’s score improved 0.55, which is 20% higher after the training. The improved score of proposed Kinect-based method is 1.10, which is 44% higher. The results of different types of movements are almost the same, which indicates the effectiveness of the proposed Kinect-based training method does not rely on the difficulty of the movements. To be mentioned that for some cases in Fig. 6, the result of Kinect-based training showed lower performance compared with video-base methods. In these cases, there are one obvious factor in common: f the kinect-based method provide only text supplementary material. Since the age of the participants are comparable young, they may have some misunderstanding with the text. This can be improved by changing the supplementary material with more detailed videos and figures. In Fig. 7, we compare the accuracy after learning in 1 min (first column of each type) and the deteriorated accuracy after learning
in 4 h (second column of each type) for three different training method: video, game without HMM customized patches, and game with HMM customized patches. It is shown in the figure that both of the two edutainment games have a better teaching performance compared with the traditional video teaching method. However, when the game is applied with HMM customized patches, the accuracy improves much better (average 10%), and the accuracy deteriorates much slower than the game without HMMs. In conclusion, the edutainment game has a better teaching performance compared with the traditional video-based method.
7. Conclusion In this paper, we propose a Kinect-based training technique for physical education. An HMM-based algorithm for generating cus-
Fig. 7. Recognition rate of 10 participants during the test.
400
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401
tomized learning patch for individuals is proposed. We present an edutainment gaming system utilizing the proposed traing method to illustrate the idea. The user study and evaluation shows that the effect of Kinect-based training method is much better than the traditional video-base method. Acknowledgements This work was supported by the National Natural Science Foundation of China [Grant No. 61602421] and the China Postdoctoral Science Foundation [Grant No. 2016M600584]. Appendix A A.1. Questionnaire of the user study See Tables 1–3.
Table 1 Questionnaire for the User Study: Table 1. Questionnaire for the User Study: Table 1 Hello! Thank you for your support of this questionnaire survey We will cherish each of your suggestions. Please fill in them carefully according to the actual situation Please fill in the brackets at the front of the question according to the real experience of the teaching and entertainment system by choosing the right answers from the following options 1. Which teaching method do you think has a better classroom atmosphere? A Traditional Method B Educational Game C Same 2. Which teaching method do you think is more interesting? A. Traditional Method B. Educational Game C. Same 3. Which teaching method do you think is more participatory? A Traditional Method B Educational Game C Same 4. Which teaching method would you like to try when you are learning? A Traditional Method B Educational Game C does not matter 5. Which teaching method are you more accustomed to when you are learning? A Traditional Method B Educational Game C Same 6. Which teaching method makes you feel more comfortable when you are learning? A Traditional Method B Educational Game C Same 7. Which teaching method can stimulate your interest in learning better? A Traditional Method B Educational Game C Same 8. Which teaching method do you think is more interactive? A Traditional Method B Educational Game C does not matter
Table 2 Questionnaire for the User Study: Table 2. Questionnaire for the User Study: Table 2 Are you satisfied with the efficiency of the teaching system? Are you satisfied with the simplicity of the operation of this teaching system? Are you satisfied with the interestingness of the teaching system? Are you satisfied with the effect of this teaching system on learning? Are you satisfied with the fun of the ”Edutainment Game” section? How much do you think the part of ”Edutainment Games” has an impact on learning knowledge? Are you satisfied with the action design of ”Edutainment interactive game”? Are you satisfied with the interface color of the teaching system? Are you satisfied with the interface layout of the teaching system? Are you satisfied with the systematic error correction ability of teaching?
Table 3 Questionnaire for the User Study: Table 3. Questionnaire for the User Study: Table 3 How important do you think the following factors are when learning knowledge in the Edutainment game system? Please give the importance score of each factor according to your understanding [Very important 5, relatively important 4, generally important 3, not very important 2, not important at all 1] This system will be useful to me This system is more interesting This system can achieve learning To be able to make it easier for me to learn how to use the system The operation mode of the system should conform to my operation habits When using this system, I think it’s relatively simple When using this system, I want to feel more comfortable When operating this system, it is more efficient When using this system, I need to be able to easily find the information I need The information provided by the system should be easy for me to understand The system should be able to respond to my instructions in a timely manner The system should be able to respond to my instructions accurately The system should be able to prompt me in time for operational errors The system should be able to tell me how to correct errors in time The interface layout of the system should be clear The interface color of the system should make me feel comfortable Finally, do you have any suggestions for this interactive system? Thank you again for your participation!
References [1] Kunwar Aditya, Recent trends in hci: a survey on data glove, leap motion and microsoft kinect, in: 2018 IEEE International Conference on System, Computation, Automation and Networking (ICSCA), IEEE, 2018, pp. 1–5. [2] Fraser Anderson, Tovi Grossman, Justin Matejka, George Fitzmaurice, Youmove: enhancing movement training with an augmented reality mirror, in: ACM Symposium on User Interface Software and Technology, ACM, 2013, pp. 311–320. [3] Urban Burnik, Janez Zaletelj, Andrej Košir, Kinect based system for student engagement monitoring, in: 2017 IEEE First Ukraine Conference on Electrical and Computer Engineering (UKRCON), IEEE, 2017, pp. 1229–1232. [4] Ziyun Cai, Jungong Han, Li Liu, Ling Shao, Rgb-d datasets using microsoft kinect or similar sensors: a survey, Multimedia Tools Appl. 76 (3) (2017) 4313–4355. [5] Dennis Charsky, From edutainment to serious games: a change in the use of game characteristics, Games Culture 5 (2) (2010) 177–198. [6] Koen de Greef, Erik D. Van der Spek, Tilde Bekker, Designing kinect games to train motor skills for mixed ability players, in: Games for Health, Springer, 2013, pp. 197–205. [7] Sara Reisi Dehkordi, Marina Ismail, Norizan Mat Diah, A review of kinect computing research in education and rehabilitation, Int. J. Eng. Technol. (UAE) 7 (3) (2018) 19–23. [8] José Rodrigues Dias, Rui Penha, Leonel Morgado, Pedro Alves da Veiga, Elizabeth Simão Carvalho, Adérito Fernandes-Marcos, Tele-media-art: Feasibility tests of web-based dance education for the blind using kinect and sound synthesis of motion, Int. J. Technol. Human Interact. (IJTHI) 15 (2) (2019) 11–28. [9] Emily DiGiovanna, Michael Lamba, Peter Sandwall, Survey of kinect v2 applied to radiotherapy patient positioning, Cancer Therapy Oncol. Int. J. 8 (1) (2017). [10] Gabriele Fanelli, Thibaut Weise, Juergen Gall, Luc Van Gool, Real time head pose estimation from consumer depth cameras, in: Joint Pattern Recognition Symposium, Springer, 2011, pp. 101–110. [11] Luigi Gallo, Alessio Pierluigi Placitelli, Mario Ciampi, Controller-free exploration of medical image data: experiencing the kinect, in: International Symposium on Computer-Based Medical Systems, IEEE, 2011, pp. 1–6. [12] Zengguo Ge, Li Fan, Social development for children with autism using kinect gesture games: a case study in Suzhou Industrial Park Renai School, in: Simulation and Serious Games for Education, Springer, 2017, pp. 113–123. [13] J. Hollan, E. Hutchins, D. Kirsh, Distributed cognition: towards a new foundation for hci research, ACM Trans. Comput.-Hum. Interact. 7 (2000) 174–196. [14] J.D. Huang, Kinerehab: a kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities, in: The Proceedings of the International ACM Sigaccess Conference on Computers and Accessibility, 2011, pp. 319–320. [15] Edwin Hutchins, How a cockpit remembers its speeds, Cogn. Sci. 19 (3) (1995) 265–288.
M. Xu et al. / J. Vis. Commun. Image R. 62 (2019) 394–401 [16] Aamrah Ikram, Yue Liu, A survey on dynamic hand gesture recognition using kinect device, in: Chinese Conference on Image and Graphics Technologies, Springer, 2018, pp. 635–646. [17] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al., Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera, in: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, ACM, 2011, pp. 559–568. [18] Mun Ho Jeong, Yoshinori Kuno, Nobutaka Shimada, Yoshiaki Shirai, Complex gesture recognition using coupled switching linear model, in: Proc. 5th Asian Conference on Computer Vision. Citeseer, 2002, pp. 132–137. [19] Stefan Koehn, Effects of confidence and anxiety on flow state in competition, Eur. J. Sport Sci. 13 (5) (2013) 543–550. [20] Belinda Lange, Chien-Yen Chang, Evan Suma, Bradley Newman, Albert Skip Rizzo, Mark Bolas, Development and evaluation of low cost game-based balance rehabilitation tool using the microsoft kinect sensor, in: Engineering in Medicine and Biology Society. Embc, 2011, pp. 1831–1834. [21] Greg C. Lee, Fu-Hao Yeh, Yi-Han Hsiao, Kinect-based taiwanese sign-language recognition system, Multimedia Tools Appl. 75 (1) (2016) 261–279. [22] Billy Y.L. Li, Mingliang Xue, Ajmal S. Mian, Wanquan Liu, Aneesh Krishna, Robust RGB-D face recognition using kinect sensor, Neurocomputing 214 (2016) 93–108. [23] Lu Xuequan, Xu. Mingliang, Wenzhi Chen, Zonghui Wang, Abdennour El Rhalibi, Adaptive-ar model with drivers’ prediction for traffic simulation, Int. J. Comput. Games Technol. 1–8 (2013) 2013. [24] Roanna Lun, Wenbing Zhao, A survey of using microsoft kinect in healthcare, in: Consumer-Driven Technologies in Healthcare: Breakthroughs in Research and Practice, IGI Global, 2019, pp. 445–456. [25] Tianlu Mao, Hu.a. Wang, Zhigang Deng, Zhaoqi Wang, An efficient lane model for complex traffic simulation, Comput. Animat. Virtual Worlds 26 (3–4) (2015) 397–403. [26] Richard E Mayer, Roxana Moreno, Animation as an aid to multimedia learning, Educ. Psychol. Rev. 14 (1) (2002) 87–99. [27] Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, Andrew Blake, Real-time human pose recognition in parts from single depth images, in: Computer Vision and Pattern Recognition, 2011. [28] Jan Smisek, Michal Jancosek, Tomas Pajdla, 3d with kinect, in: Consumer Depth Cameras for Computer Vision, Springer, 2013, pp. 3–25. [29] Milka Trajkova, Mexhid Ferati, Usability evaluation of kinect-based system for ballet movements, in: Springer International Publishing, Springer, 2015. [30] Christina Vasiliou, Andri Ioannou, Panayiotis Zaphiris, Understanding collaborative learning activities in an information ecology: a distributed cognition account, Comput. Hum. Behav. 41 (2014) 544–553. [31] César Villacís, Walter Fuertes, Andrés Bustamante, Daniel Almachi, Carlos Procel, Susana Fuertes, Theofilos Toulkeridis, Multi-player educational video game over cloud to stimulate logical reasoning of children, in: Proceedings of the 2014 IEEE/ACM 18th International Symposium on Distributed Simulation and Real Time Applications, IEEE Computer Society, 2014, pp. 129–137.
401
[32] H. Wang, X. Kang, Tianlu Mao, Z. Wang, A semantic model of road networks for traffic animations, Jisuanji Fuzhu Sheji Yu Tuxingxue Xuebao/J. Comput.-Aided Des. Comput. Graph. 26 (2014) 1818–1826. [33] H. Wang, M. Xu, Tianlu Mao, X. Jin, Z. Wang, Survey of three-dimension traffic animation, Jisuanji Fuzhu Sheji Yu Tuxingxue Xuebao/J. Comput.-Aided Des. Comput. Graph. 29 (2017) 211–220. [34] Hu.a. Wang, Tianlu Mao, Xingchen Kang, Zhaoqi Wang, An all-in-one efficient lane-changing model for virtual traffic, Comput. Animat. Virtual Worlds 25 (3– 4) (2014) 383–391. [35] Hu.a. Wang, Xu. Mingliang, Fubao Zhu, Zhigang Deng, Yafei Li, Bing Zhou, Shadow traffic: a unified model for abnormal traffic behavior simulation, Comput. Graph. 70 (2017). [36] Lu Xia, Chia-Chih Chen, Jake K Aggarwal, Human detection using depth information by kinect, in: CVPR 2011 Workshops, IEEE, 2011, pp. 15–22. [37] Xu. Mingliang, Hao Jiang, Xiaogang Jin, Zhigang Deng, Crowd simulation and its applications: recent advances, J. Comput. Sci. Technol. 29 (5) (2014) 799– 811. [38] Xu. Mingliang, Zhigeng Pan, Mingmin Zhang, Pei Lv, Pengyu Zhu, Yangdong Ye, Wei Song, Character behavior planning and visual simulation in virtual 3d space, IEEE MultiMedia 20 (1) (2013) 49–59. [39] Xu. Mingliang, Hu.a. Wang, Shili Chu, Yong Gan, Xiaoheng Jiang, Yafei Li, Bing Zhou, Traffic simulation and visual verification in smog, TIST 10 (2019) 1–17. [40] Erman Yukselturk, Serhat Altıok, Zeynep Basßer, Using game-based learning with kinect technology in foreign language education course, J. Educ. Technol. Soc. 21 (3) (2018) 159–173. [41] Tamara Berg, Debaleena Chattopadhyay, Margaret Schedel, Timothy Vallier, Interactive music: human motion initiated music generation using skeletal tracking by kinect, in: Proc. Conf. Soc. Electro-Acoustic Music United States, 2012. [42] Zhongwei Cheng, Lei Qin, Yituo Ye, Qingming Huang, Qi Tian, Human daily action analysis with multi-view and color-depth data, in: European Conference on Computer Vision, Springer, 2012, pp. 52–61. [43] Simon Fothergill, Helena Mentis, Pushmeet Kohli, Sebastian Nowozin, Instructing people for training gestural interactive systems, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2012, pp. 1737–1746. [44] Bingbing Ni, Gang Wang, Pierre Moulin, Rgbd-hudaact: a color-depth video database for human daily activity recognition, in: 2011 IEEE international conference on computer vision workshops (ICCV workshops), IEEE, 2011, pp. 1147–1153. [45] Sebastian Nowozin, Jamie Shotton, Action points: A representation for lowlatency online human action recognition. Microsoft Research Cambridge, Tech. Rep. MSR-TR-2012-68, 2012. [46] Michalis Raptis, Darko Kirovski, Hugues Hoppe, Real-time classification of dance gestures from skeleton animation, in: 2011 ACM SIGGRAPH/ Eurographics Symposium on Computer Animation, ACM, 2011, pp. 147–156. [47] Jiawei Han, Zhiguo Xiao, Lingeng Han, Ying Xu, Nianfeng Li, Sato Reika, A special edutainment system based on somatosensory game, in: IEEE International Conference on Software Engineering & Service Science, 2015.