- Email: [email protected]
Improved 1D-CNNs for behavior recognition using wearable sensor network
Computer Communications 151 (2020) 165–171
Contents lists available at ScienceDirect
Computer Communications journal homepage: www.elsevier.com/locate/comcom
Improved 1D-CNNs for behavior recognition using wearable sensor network Zhiou Xu a,b ,∗, Juan Zhao c , Yi Yu d , Haijun Zeng d a
School of Computer Sciences and Technology, China University of Mining & Technology, Xuzhou, China Engineering Research Center of Digital Mine, Ministry of Education, Xuzhou, China Department of Mechanical and Electrical Engineering, Henan Industrial and Trade Vocational College, Zhengzhou, China d Operating Branch, Ningbo Rail Transit Group Co., LTD., Ningbo, China b c
ARTICLE
INFO
Keywords: Human behavior monitoring Wearable device 1D-CNNs Sample autonomous learning
ABSTRACT Wireless Body Sensor Network (BSNs) are wearable sensors with varying sensing, storage, computation, and transmission capabilities. When data is obtained from multiple devices, multi-sensor fusion is desirable to transform potentially erroneous sensor data into high quality fused data. By analyzing and processing the perceived physical activity data of users, they can be provided with services that may be needed. Wearable sensors transmit human acceleration information to the server through 4G network. In this way, online analysis and recognition of human behavior is realized. In this paper, we study on efficiently real-time behavior recognition algorithm using acceleration sensor. We propose a human behavior recognition method based on improved One-Dimensional Convolutional Neural Networks(1D-CNNs). According to the motion characteristics, the eigenvalues are extract which can distinguish the types of activities. At the same time, we propose a sample autonomous learning method, which aims to find the optimal sample training set and avoid overfitting problems in traditional CNNs. In the recognition of 11 human activities, our method can reach the average accuracy of 98.7%. Compared with other behavior recognition methods in the same dataset, better classification is achieved by this method.
1. Introduction Sensors can convert human activity information into data without being affected by external environment. By analyzing and processing the perceived physical activity data of users, they can be provided with services that may be needed. As an important component of human–computer interaction, human behavior recognition has become a new research hotspot. Based on the analysis of acceleration signal and human activity characteristics, a human behavior recognition method based on improved one-dimensional convolution neural network is proposed, and it is verified that the recognition accuracy can be effectively improved by our model. Human activity data can be used as a basis for measuring energy consumption and diagnosing potential physical diseases. There are many kinds of human activities, so in order to identify activities, different parts of the body need to wear different types of inertial sensors to obtain activity data. Research shows that accelerometer sensors can collect many kinds of motion data, so it has been widely used in motion monitoring, gait parameter extraction, motion classification, attitude balance detection. The frequency of human activity in daily activities is usually less than 20 Hz. In [1], the human body running and jumping movements were studied. It was found that the acceleration of ankle joint ranged
from [3.0 g to 12.0 g], head acceleration ranged from [0.8 g to 4.0 g], and back acceleration ranged from [0.9 g to 5.0 g]. Literature [2] studied human walking. It was found that the range of acceleration in the vertical direction is [−0.3 g ∼ 0.8 g], the range of the frontrear direction is [−0.3 g ∼ 0.4 g], and the range of the left–right direction is [−0.2 g ∼ 0.2 g]. Literature [3] found that in daily activities, the body’s acceleration range is [−12 g ∼ 12 g], and the frequency range is [0 ∼ 20 Hz]. Literature [4] found that the vertical direction of the acceleration component of the human body is higher than the horizontal direction. Therefore, according to the different types of human motion, the position of the acceleration sensor placed on the body by the relevant researchers is also different. The rest of this paper is divided as follows: In Section 2, we present some relative studies that work on human activity recognition based on wearable devices. In Section 3, we describe the three-axis accelerometer signal analysis. In Section 4, we will give a detailed description of our model of improved 1D-CNNs. In Section 5, we do some experiments and analyze the results. We conclude this paper in Section 6. 2. Related works At present, human body behavior monitoring based on wearable devices mainly includes the following contents: collecting effective
∗ Corresponding author at: School of Computer Sciences and Technology, China University of Mining & Technology, Xuzhou, China. E-mail address: [email protected] (Z. Xu).
https://doi.org/10.1016/j.comcom.2020.01.012 Received 22 November 2019; Received in revised form 17 December 2019; Accepted 3 January 2020 Available online 8 January 2020 0140-3664/© 2020 Elsevier B.V. All rights reserved.
Z. Xu, J. Zhao, Y. Yu et al.
Computer Communications 151 (2020) 165–171
human activity data, extracting effective feature values [5], studying human behavior recognition or classification methods [6], and establishing a human body behavior monitoring system based on wearable devices [7]. Most human body behavior monitoring methods based on wearable devices contain one or more of the work mentioned above. Classification techniques have evolved into many methods, and they have different recognition performance for behavioral detection. Currently, the classification methods of behavior monitoring based on wearable devices have Decision Trees (DT) [8], Threshold-based classification (TB), Support Vector Machine (SVM) [9] [10], K-Nearest Neighbor (K-NN) [11], Sparse Representation (SR), Artificial Neural Network (ANN) [12], Relevant Vector Machine (RVM) [13], etc. Anguita et al. [14] used the SVM classifier to identify behavior and use a fixed-point algorithm to reduce computational costs. Decision trees are often used for multi-level classification and recognition of human activities. Martin et al. [15] constructed a two-layer classifier, in which the first layer is used to determine the position of the device in the human body, and the second layer is used to identify human behavior. Diep et al. [10] used a three-axis accelerometer to collect data, and used sliding window technology to process the data. The values of each category in the three axes were used as eigenvalues, and were trained with the SVM model to identify the fall event. Literature [16] studies the classification of sports activities, and compares the methods of least squares, Bayes and K-nearest neighbors from the perspective of computational cost and classification accuracy. The experimental results show that the Bayes’ method has the lowest computational complexity and the classification accuracy is the highest among these methods. Literature [17] combines sparse representation and compressed sensing, and uses 1300 body motion sequences as training samples for experiments. Literature [18] combines the advantages of multi-layer perceptron and logistic regression classifiers, and the results show that the recognition effect is better than that of a single classifier. Research on human behavior monitoring based on wearable devices has achieved considerable success, but there are still many challenges in this area, such as: (1) Whether it is a wearable device or a handheld device like a smart phone, the data collected by devices in different body parts is different under the same physical activity. The sensor characteristics and wear position of the device determine the content of the collected data, which is crucial for human behavior monitoring, so how to obtain these device information needs to be solved. (2) The extraction of eigenvalues determines the quality of human behavior monitoring. However, the current effective eigenvalue extraction is subjectively selected by researchers through prior experience, which has great limitations. (3) Human body behavior monitoring system based on wearable devices will have great energy consumption when collecting and processing data. The multi-sensor collaborative work improves the recognition accuracy, but shortens the battery life and the user experience is also poor. Therefore, it is necessary to study a behavior monitoring method based on a single sensor.
Fig. 1. Wearing of accelerometer and direction of coordinate axis.
of the sensor is shown in Fig. 1. Wearable sensors transmit human acceleration information to the server through 4G network. The ADXL345 accelerometer described in this chapter works as the inertia principle, as shown in Eq. (1): 𝐹 (1) 𝑀 In the formula, 𝐴 is acceleration, 𝐹 is force, and 𝑀 is mass. The acceleration sensor measures the following values: acceleration of human motion, acceleration of gravity, acceleration caused by external force. Wherein, the gravity acceleration provides the orientation of the sensor, and the human body acceleration provides the body motion information. The factors influencing the measurement of human acceleration are: the type of motion, the placement of the accelerometer, and the direction of the accelerometer coordinate system. 𝐴=
3.2. Signal feature extraction This section only studies the change in acceleration of a certain action over a period of time, regardless of the details of the complex motion. First, this chapter uses the amplitude of the acceleration signal to describe human motion. The formula for the body’s synthetic acceleration value is shown in Eq. (2): √ 𝑆𝑎𝑐𝑐 = 𝑎𝑐𝑐𝑥2 + 𝑎𝑐𝑐𝑦2 + 𝑎𝑐𝑐𝑧2 (2) In the equation, 𝑆𝑎𝑐𝑐 is the combined acceleration, 𝑎𝑐𝑐𝑥 is the 𝑥axis acceleration, 𝑎𝑐𝑐𝑦 is the 𝑦-axis acceleration, and 𝑎𝑐𝑐𝑧 is the 𝑧-axis acceleration. Slow walking, fast walking, slow running, and fast running are regular sports. The difference between them is the intensity of the exercise, the size of the stride, and the frequency of the step. Like the up and down stairs, sitting down, standing up, jumping up are irregular movements, the data presented by the graphics are different. As shown in Fig. 2, the movement of the human body when going up and down the stairs is an approximate regular movement. But only looking at the 𝑥-axis and 𝑧-axis directions, the waveforms presented by the data are not obvious enough. The waveforms exhibited by the 𝑦-axis direction and the resultant acceleration 𝑆𝑎𝑐𝑐 value are extremely obvious, and the data exhibits regular fluctuations. When the human body performs the ‘‘sit down’’ and ‘‘sit up’’ actions, the acceleration value changes rapidly. When the human body jumps, the acceleration value also changes rapidly. (1) Standard Deviation The discrete level of the data set can be represented by the standard deviation. In the recognition of human motion, we can use the standard
3. Three-axis accelerometer signal analysis 3.1. Triaxial acceleration sensor According to the complexity and duration of human activities, it can be divided into three states: Short Events, Basic Activities and Complex Activities [19]. Short action refers to the type of exercise with short action time, which is the transformation of body posture, like standing up, sitting down, jumping. Basic activities are common simple actions in everyday life, such as walking, running, and standing. Complex activities are a combination of short-acting and basic activities, like eating, drinking tea or even listening to classes. The work in this chapter is to analyze short-action and basic activities by analyzing, identifying, and monitoring a person’s activities. The wearing mode 166
Z. Xu, J. Zhao, Y. Yu et al.
Computer Communications 151 (2020) 165–171
Fig. 2. Acceleration data of D05.
deviation to reflect the intensity of the human body’s exercise. When the person is stationary, the change in acceleration is extremely small, so the standard deviation of the data is small. When a person is exercising, the acceleration will become larger and the standard deviation of the data will vary depending on the type of exercise. Therefore, the standard deviation can be used to distinguish whether the human body is in a stationary state or a moving state. The standard deviation formula is shown in Eq. (3): √ √ 𝑁 √1 ∑ (𝑥 − 𝜇)2 (3) 𝑆𝐷 = √ 𝑁 𝑖=1 𝑖
𝐷𝐴𝑃 𝑇 = 𝑆𝑎𝑐𝑐𝑡 − 𝑆𝑎𝑐𝑐𝑡−𝑘
In the formula, 𝑆𝑎𝑐𝑐𝑡 is the peak (valley) value, 𝑆𝑎𝑐𝑐𝑡−𝑘 is the adjacent trough (peak) value, and 𝑘 is the number of sampling points between the two values. 4. Human behavior recognition based on improved 1D-CNNs 4.1. Analysis of 1D-CNNs Convolutional Neural Networks (CNNs) can be used for sensorbased human behavior recognition [20]. This chapter uses 1D-CNNs to classify and recognize human behavior. The structure of the traditional convolutional neural network [21] is shown in Fig. 3. As can be seen from the figure, CNNs mainly include input layer, hidden layer and output layer. The input layer of CNNs can handle one-dimensional or multidimensional data. The hidden layer is composed of convolutional layers, pooling layers, and fully connected layers. The convolutional layer extracts feature from input data; the pool layer receives the eigenvalues from the convolutional layer and carries out feature selection and information filtering; the fully connected layer is equivalent to the hidden layer in the feedforward neural network, which expands the multidimensional data into vectors and transfers them to the next layer by using the excitation function. Finally, the output layer of CNNs uses the softmax function or the logistic function to output the classification results. The calculation formula for the CNNs convolutional layer is as follows:
In the formula, 𝑁 is the number of items in the data set, and 𝜇 is the mean of the data set. (2) Interval of Peaks The walking frequency of the human body is different when walking and running, so the fluctuation frequency of the data drawing is different. Therefore, we choose the interval between two adjacent peaks of the data to distinguish the type of motion. The formula is shown in (4):
Inv𝑙 = 𝑇𝑡 − 𝑇𝑡−1
(5)
(4)
In the equation, 𝑇𝑡−1 is the time at which a peak appears, and 𝑇𝑡 is the time at which the next peak appears. 𝑙 is the number of points sampled in the adjacent wave peaks. (3) Difference between Adjacent Peaks and Troughs Different types of exercise have different ranges. The recognition of human motion state requires high real-time performance, so how to extract real-time information becomes very important when extracting feature points. Therefore, this chapter will combine the peaks and troughs value of the acceleration 𝑆𝑎𝑐𝑐 as the research object. In the data processing process, we compare each newly received sample value with the previous sample value. If there is no turning point in the waveform of the sampling point, continue to observe the new sampling value. If the waveform has a turning point, the point is considered to be the extreme point. In this chapter, we take the difference between adjacent peaks and troughs of 𝑆𝑎𝑐𝑐 as the eigenvalue, as shown in Eq. (5):
⎛∑ ⎞ 𝑋𝑗𝑙 = 𝑓 ⎜ 𝑋𝑖𝑙−1 ∗ 𝐾𝑖𝑗𝑙 + 𝑏𝑙𝑗 ⎟ ⎟ ⎜𝑖∈𝑀 𝑗 ⎝ ⎠
(6)
In the formula, 𝑋𝑗𝑙 represents the 𝑗th feature map of the 𝑙th layer, 𝑀𝑗 represents the set of selected input feature maps, 𝐾𝑖𝑗𝑙 represents the convolution kernel function, 𝑓 (⋅) represents the activation function, and 𝑏𝑙𝑗 represents the offset parameter. 167
Z. Xu, J. Zhao, Y. Yu et al.
Computer Communications 151 (2020) 165–171
Fig. 3. Structure of CNNs.
𝛿𝑗𝑙
Error signal of the 𝑗th feature map of layer 𝑙: ( ) = 𝜔𝑙+1 𝑓 ′ (𝑖𝑝𝑙𝑗 ) ⋅ 𝑢(𝛿𝑗𝑙+1 ) 𝑗
training sample set is artificially set based on past experience, because a good sample learning method can greatly improve the performance of the network. Therefore, this paper proposes a method called Sample Autonomous Learning (SAL) to construct an effective sample training set. The specific method is as follows: In order to improve the efficiency of network classification and reduce the classification training set in limited resources and time, the method gradually adjusts the number of sample sets with incorrect classification to find the optimal classification sample set. The algorithm process is as follows: First, the initial sample 𝐷𝑖 is selected and the number of training samples of 𝐷𝑖 is set, and the initial sample data 𝐷𝑖 is input into the 1D-CNNs network for training. Then, the error classification rate 𝜀𝑖 of each category at the output is calculated separately and compared with the set threshold 𝜃. If 𝜀𝑖 > 𝜃, the number of data of the training sample is expanded until 𝜀𝑖 ≤ 𝜃 or 𝜀𝑖 reaches convergence on the new sample training set. By verifying the 1D-CNNs network, the optimal training sample set can be obtained to avoid the occurrence of over-fitting problem.
(7)
In the formula, 𝜔𝑙+1 represents the weight of the 𝑗th feature map 𝑗 of the 𝑙 + 1th layer, 𝑖𝑝𝑙𝑗 represents the input value of the 𝑗th neuron of the 𝑙th layer, 𝑓 ′ (⋅) represents the partial derivative of the activation function, and 𝑢 (⋅) represents the upsampling function. The weight gradient function of the convolution kernel is as shown in Eq. (8): ∑( ) ( ) 𝜕𝐺 = 𝛿𝑗𝑙 𝑑𝑖𝑙−1 𝑢𝑣 (8) 𝑙 𝑢𝑣 𝜕𝐾𝑖𝑗 𝑢,𝑣 In the equation, 𝑑𝑖𝑙−1 is the block in the 𝑗th feature map of the 𝑙−1th layer, multiplied by 𝐾𝑖𝑗𝑙 in the convolution process, and (𝑢, 𝑣) represents the position of the convolutional feature map. The calculation formula for the pooling layer is as shown in Eq. (9): ( ) ( ) 𝑋𝑗𝑙 = 𝑓 𝜔𝑙𝑗 𝑑 𝑋𝑖𝑙−1 + 𝑏𝑙𝑗 𝑡 (9) In the formula, 𝑑 (⋅) denotes the downsampling function. The gradient function of the pooled layer is similar to Eq. (8). In CNNs, the weight update from time 𝑡 to time 𝑡 + 1 can be expressed as: 𝑊 (𝑡 + 1) = 𝑊 (𝑡) + 𝜎𝛿(𝑡)𝑥(𝑡)
(10)
𝜎 represents the learning rate, 𝛿(𝑡) indicates the error term, and 𝑥(𝑡) represents the input of the neuron. CNNs are a good way to identify simple patterns in data and then generate more complex patterns in more advanced layers. CNNs contain one-dimensional convolution, two-dimensional convolution, and threedimensional convolution. Their processing methods and characteristics are the same, the difference is the dimension of the input data and the way the filter slides between the data. The one-dimensional convolutional neural networks (1D-CNNs) can effectively obtain effective features from the shorter segments of the overall data set, which can be used for the analysis and processing of sensor data. 4.2. Improvementof 1D-CNNs Traditional convolutional neural networks have been able to handle different classification problems better. However, as a neural network for globally optimized supervised learning, 1D-CNNs still have some problems. For example, the sample learning method of 1D-CNNs cannot handle the over-fitting problem very well. In domestic and foreign research, some researchers tried to alleviate the problem by increasing the number of samples. The over-fitting problem means that the training samples are too strict in the training process of the network, so that the trained network has a poor classification effect on the sample data that is slightly different from the training set. The specific performance is that as the number of 1D-CNNs training increases, the error rate of the sample test first decreases and then rises. In the traditional 1D-CNNs algorithm, the
The formula for calculating the error classification rate 𝜀𝑖 is as follows: 𝑒 𝜀𝑖 = 𝑖 (11) 𝐷𝑖 In the formula, 𝑒𝑖 is the number of samples misclassified, and 𝐷𝑖 is the total number of samples classified. The following describes the process of human behavior state recognition based on improved 1D-CNNs: 168
Z. Xu, J. Zhao, Y. Yu et al.
Computer Communications 151 (2020) 165–171
Table 1 Types of activities. Num
Category
Number of experiments
Duration (s)
D01 D02 D03 D04 D05 D06 D07 D08 D09 D010 D011
Slow walking Fast walking Slow running Fast running Slow up and down stairs Fast up and down stairs Stand up slowly from a half-height chair Stand up quickly from a half-height chair Stand up slowly from a low chair Stand up quickly from a low chair Jumping up
1 1 1 1 5 5 55 5 5 5 5
100 100 100 100 25 25 12 12 12 12 12
positive, and 𝐹 𝑁 is the number of activities incorrectly predicted to be negative. (2) Precision: The ratio of the number of activities of a certain type to that of all activities of that type. It is an index for predicting results. 𝑇𝑃 (13) 𝑇𝑃 + 𝐹𝑃 (3) Recall: A certain type of activity is correctly predicted as the ratio of the number of such activities to the original total number of such activities. It is an indicator for the original sample instance. 𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =
𝑇𝑃 (14) 𝑇𝑃 + 𝐹𝑁 Table 2 shows the confusion matrix for identifying 11 activities based on the improved 1D-CNNs experiment. The confusion matrix can directly show the experimental results and help us evaluate the performance of the algorithm. Each row represents the actual behavioral activity data of the human body, and each column represents data in which the actual behavioral activity is identified as the current behavioral activity. For example, the second row of data in the table: the number of people actually performing ‘‘slow walking’’ activities is 38, of which 36 are correctly identified as ‘‘slow walking’’ activities, and 2 are incorrectly identified as ‘‘fast walking’’ activities. Experiments based on improved 1D-CNNs for human behavior recognition have effectively improved the recognition accuracy of 11 human behaviors compared to other traditional classification methods, and achieved better classification and recognition effects. Table 3 shows the final results of human behavior recognition experiments based on improved 1D-CNNs. It can be seen from the table that the average accuracy of the identification of each activity is 92.8%, the average accuracy rate is 98.7%, and the average recall rate is 92.8%. In this method, the possible causes of misidentification are as follows: First, when the human body walks, the speed is too fast and close to the running speed; correspondingly, the human body is slow in running, and the speed is close to the speed of walking fast. Secondly, when the human body is standing up, the range of motion is too large. Thirdly, when jumping, the human body only slightly leaves the 𝑅𝑒𝑐𝑎𝑙𝑙 =
5. Experiments and analysis In order to build a unified evaluation standard, this paper uses the global open SisFall data set [22] for simulation experiments. SisFall is a complete data set containing fall and daily behavioral activities. As shown in Table 1, these data sets are composed of human activity data collected by self-developed embedded devices, and the data can be easily copied for data analysis and simulation experiments. The data set contains 19 types of daily behavioral activities and 15 types of falls. The data collected were from the daily activities of 38 subjects, including 23 adults aged 19–30 and 15 elderly aged 60–75. In this chapter, the improved 1D-CNNs method is analyzed from the accuracy, precision and recall. (1) Accuracy: Ratio of the number of samples correctly classified to the total number of samples. 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =
𝑇𝑃 + 𝑇𝑁 𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁
(12)
In the formula, 𝑇 𝑃 is the number of activities correctly predicted to be positive, 𝑇 𝑁 is the number of activities correctly predicted to be negative, 𝐹 𝑃 is the number of activities incorrectly predicted to be 169
Z. Xu, J. Zhao, Y. Yu et al.
Computer Communications 151 (2020) 165–171 Table 2 Confusion matrix of human behavior recognition.
Table 3 Results of human behavior recognition experiment.
6. Conclusion
Action
Accuracy
Precision
Recall
D01 D02 D03 D04 D05 D06 D07 D08 D09 D010 D011 Average
97.3% 89.7% 89.7% 97.3% 97.3% 94.9% 90.4% 89.3% 92.1% 91.6% 91.3% 92.8%
99.8% 99.5% 99.5% 99.8% 98.9% 99.6% 97.3% 97.8% 96.8% 98.8% 98.0% 98.7%
94.7% 92.1% 92.1% 94.7% 94.7% 97.3% 89.5% 87.9% 92.1% 91.6% 94.2% 92.8%
Human body behavior monitoring based on wearable devices mainly includes: collecting effective human activity data, extracting effective feature values, studying human behavior recognition or classification methods, and establishing a human body behavior monitoring system based on wearable devices. Most human body behavior monitoring methods based on wearable devices contain one or more of the work mentioned above. For behavior recognition, traditional methods can only recognize human activities offline. But in practical application, real-time monitoring results are required. Considering of the above two points, in this paper, we study on efficiently real-time behavior recognition algorithm using a single sensor. Then, we propose a human behavior recognition method based on improved One-Dimensional Convolutional Neural Networks (1D-CNNs). According to the motion characteristics, we extract the eigenvalues which can distinguish the types of activities. At the same time, we propose a sample autonomous learning method, which aims to find the optimal sample training set and avoid over-fitting problems in traditional CNNs. In the recognition of 11 human activities, our method can reach the average accuracy of 98.7%. Compared with other behavior recognition methods in the same dataset, better classification is achieved by this method. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. CRediT authorship contribution statement
Fig. 4. Results comparison of methods in behavior recognition.
Zhiou Xu: Conceptualization, Methodology, Software. Juan Zhao: Data curation, Writing - original draft, Visualization. Yi Yu: Visualization, Investigation. Haijun Zeng: Supervision.
ground, and the movement is more relaxed. Fourth, the sensors are not fixed properly. For human behavior recognition, this chapter selects the methods of [20] and [21] as reference experiments. The comparison method and the method proposed in this paper are run on SisFall, and the recall result is shown in Fig. 4. It can be seen from the figure that the method of [21] is slightly better than the method of this paper in the identification of D07. The reason may be that the literature [21] selects the acceleration dip as the eigenvalue, and D07 is the ‘‘sitting→starting’’ movement. When the human body performs this activity, the body inclination changes greatly, so the method of [21] is sensitive to the recognition of the activity. However, it can be seen from the figure that the recognition effect of this method is better than the methods of [20] and [21,22] in the recognition of other behaviors.
Acknowledgment Foundation item: Special Topic of Independent Innovation for Double-class Construction of Computer College (2018ZZCX14). References [1] S. Bhattacharya, A.M. Krishna, D. Lombardi, et al., Economic MEMS based 3-axis water proof accelerometer for dynamic geo-engineering applications, Soil Dyn. Earthq. Eng. 36 (4) (2012) 111–118. [2] K. Taraldsen, S.F. Chastin, I.I. Riphagen, et al., Physical activity monitoring by use of accelerometer-based body-worn sensors in older adults: a systematic literature review of current knowledge and applications, Maturitas 71 (1) (2012) 13–19. 170
Z. Xu, J. Zhao, Y. Yu et al.
Computer Communications 151 (2020) 165–171 [22] Huijuan Hao, Maoli Wang, Yongwei Tang, Qingdang Li, Research on data fusion of multi-sensors based on fuzzy preference relations, Neural Comput. Appl. 31 (S-1) (2019) 337–346.
[3] J.J. Kavanagh, H.B. Menz, Accelerometry: A technique for quantifying movement patterns 63 during walking, Gait Posture 28 (1) (2008) 1–15. [4] J. Majid, R.F. Atena, R. Katarzyna, et al., A comprehensive analysis on wearable acceleration sensors in human activity recognition, Sensors 17 (3) (2017) 529–554. [5] Y. Cao, Y.Z. Zhang, T. Wen, P. Li, Research on dynamic nonlinear input prediction of fault diagnosis based on fractional differential operator equation in high-speed train control system, Chaos, https://doi.org/10.1063/1.5085397. [6] T. Plötz, Y. Guan, Deep learning for human activity recognition in mobile computing, Computer 51 (5) (2018) 50–59. [7] J. Cheng, O. Amft, G. Bahle, et al., Designing sensitive wearable capacitive sensors for activity recognition, IEEE Sens. J. 13 (10) (2013) 3935–3947. [8] I.D. Dinov, Decision tree divide and conquer classification, Data Sci. Predict. Anal. (2018) 307–343. [9] Yi Ai, Zongping Li, Mi Gan, Yunpeng Zhang, Daben Yu, Wei Chen, Yanni Ju, A deep learning approach on short-term spatiotemporal distribution forecasting of dockless bike-sharing system, Neural Comput. Appl. 31 (5) (2019) 1665–1677. [10] J. Wang, Y. Chen, S. Hao, et al., Deep learning for sensor-based activity recognition: A survey, Pattern Recognit. Lett. 119 (2017) 3–11. [11] H. Jian, H. Chen, A portable falls detection and alerting system based on k-NN algorithm and remote medicine, China Commun. 12 (4) (2015) 23–31. [12] M.H.M. Noor, Z. Salcic, I.K. Wang, Adaptive sliding window segmentation for physical activity recognition using a single tri-axial accelerometer, Pervasive Mob. Comput. 38 (1) (2017) 41–59. [13] G.M. Foody, RVM-based multi-class classification of remotely sensed data, Int. J. Remote Sens. 29 (6) (2008) 1817–1823. [14] D. Anguita, A. Ghio, L. Oneto, et al., Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine, Ambient Assist. Living Home Care (2012) 216–223. [15] H. Martín, A.M. Bernardos, J. Iglesias, et al., Activity logging using lightweight classification techniques in mobile devices, Pers. Ubiquitous Comput. 17 (4) (2013) 675–695. [16] K. Altun, B. Barshan, Human activity recognition using inertial/magnetic sensor units, in: International Conference on Human Behavior Understanding, 2010, pp. 38–51. [17] Y. Cao, H.K. Lu, T. Wen, A safety computer system based on multi-sensor data processing, Sensors 19 (4) (2019) 818. [18] M.N.S. Zainudin, M.N. Sulaiman, N. Mustapha, et al., Activity recognition based on accelerometer sensor using combinational classifiers, in: Open Systems, IEEE, 2015, pp. 68–73. [19] J.L. Reyes Ortiz, L. Oneto, Samà Monsonís, Albert, et al., Transition-aware human activity recognition using smartphones, Neurocomputing 171 (2016) 754–767. [20] M. Zeng, T.N. Le, B. Yu, et al., Convolutional neural networks for human activity recognition using mobile sensors, in: International Conference on Mobile Computing, Applications and Services, IEEE, 2015, pp. 197–205. [21] Y.L. Lecun, L. Bottou, Y. Bengio, et al., Gradient-based learning applied to document recognition, Proc. IEEE 86 (11) (1998) 2278–2324.
Zhiou Xu was born in 1976. He received Ph.D degree in power electronic and electric drive from China University of Mining and Technology in 2011. He works in China University of Mining and Technology since 2011. His research interests include the power electronic technology, smart grid and computer control technology.
Juan Zhao was born in 1981. In 2019, she obtained a master’s degree in electronics and communication engineering from the school of information engineering. Since 2009, she have been working in Henan industrial and trade vocational college. Her research interests include communication networks and information security.
Yi Yu received B.E. degree from Zhejiang University in 1999. He works in Operating Branch of Ningbo Rail Transit Group Co., LTD. His research interests include power supply technology and measure and control systems.
Haijun Zeng received B.E. degree from Lanzhou Jiaotong University in 2002. He works in Operating Branch of Ningbo Rail Transit Group Co., LTD. His research interests include power supply technology and measure and control systems.
171
Contents lists available at ScienceDirect
Computer Communications journal homepage: www.elsevier.com/locate/comcom
Improved 1D-CNNs for behavior recognition using wearable sensor network Zhiou Xu a,b ,∗, Juan Zhao c , Yi Yu d , Haijun Zeng d a
School of Computer Sciences and Technology, China University of Mining & Technology, Xuzhou, China Engineering Research Center of Digital Mine, Ministry of Education, Xuzhou, China Department of Mechanical and Electrical Engineering, Henan Industrial and Trade Vocational College, Zhengzhou, China d Operating Branch, Ningbo Rail Transit Group Co., LTD., Ningbo, China b c
ARTICLE
INFO
Keywords: Human behavior monitoring Wearable device 1D-CNNs Sample autonomous learning
ABSTRACT Wireless Body Sensor Network (BSNs) are wearable sensors with varying sensing, storage, computation, and transmission capabilities. When data is obtained from multiple devices, multi-sensor fusion is desirable to transform potentially erroneous sensor data into high quality fused data. By analyzing and processing the perceived physical activity data of users, they can be provided with services that may be needed. Wearable sensors transmit human acceleration information to the server through 4G network. In this way, online analysis and recognition of human behavior is realized. In this paper, we study on efficiently real-time behavior recognition algorithm using acceleration sensor. We propose a human behavior recognition method based on improved One-Dimensional Convolutional Neural Networks(1D-CNNs). According to the motion characteristics, the eigenvalues are extract which can distinguish the types of activities. At the same time, we propose a sample autonomous learning method, which aims to find the optimal sample training set and avoid overfitting problems in traditional CNNs. In the recognition of 11 human activities, our method can reach the average accuracy of 98.7%. Compared with other behavior recognition methods in the same dataset, better classification is achieved by this method.
1. Introduction Sensors can convert human activity information into data without being affected by external environment. By analyzing and processing the perceived physical activity data of users, they can be provided with services that may be needed. As an important component of human–computer interaction, human behavior recognition has become a new research hotspot. Based on the analysis of acceleration signal and human activity characteristics, a human behavior recognition method based on improved one-dimensional convolution neural network is proposed, and it is verified that the recognition accuracy can be effectively improved by our model. Human activity data can be used as a basis for measuring energy consumption and diagnosing potential physical diseases. There are many kinds of human activities, so in order to identify activities, different parts of the body need to wear different types of inertial sensors to obtain activity data. Research shows that accelerometer sensors can collect many kinds of motion data, so it has been widely used in motion monitoring, gait parameter extraction, motion classification, attitude balance detection. The frequency of human activity in daily activities is usually less than 20 Hz. In [1], the human body running and jumping movements were studied. It was found that the acceleration of ankle joint ranged
from [3.0 g to 12.0 g], head acceleration ranged from [0.8 g to 4.0 g], and back acceleration ranged from [0.9 g to 5.0 g]. Literature [2] studied human walking. It was found that the range of acceleration in the vertical direction is [−0.3 g ∼ 0.8 g], the range of the frontrear direction is [−0.3 g ∼ 0.4 g], and the range of the left–right direction is [−0.2 g ∼ 0.2 g]. Literature [3] found that in daily activities, the body’s acceleration range is [−12 g ∼ 12 g], and the frequency range is [0 ∼ 20 Hz]. Literature [4] found that the vertical direction of the acceleration component of the human body is higher than the horizontal direction. Therefore, according to the different types of human motion, the position of the acceleration sensor placed on the body by the relevant researchers is also different. The rest of this paper is divided as follows: In Section 2, we present some relative studies that work on human activity recognition based on wearable devices. In Section 3, we describe the three-axis accelerometer signal analysis. In Section 4, we will give a detailed description of our model of improved 1D-CNNs. In Section 5, we do some experiments and analyze the results. We conclude this paper in Section 6. 2. Related works At present, human body behavior monitoring based on wearable devices mainly includes the following contents: collecting effective
∗ Corresponding author at: School of Computer Sciences and Technology, China University of Mining & Technology, Xuzhou, China. E-mail address: [email protected] (Z. Xu).
https://doi.org/10.1016/j.comcom.2020.01.012 Received 22 November 2019; Received in revised form 17 December 2019; Accepted 3 January 2020 Available online 8 January 2020 0140-3664/© 2020 Elsevier B.V. All rights reserved.
Z. Xu, J. Zhao, Y. Yu et al.
Computer Communications 151 (2020) 165–171
human activity data, extracting effective feature values [5], studying human behavior recognition or classification methods [6], and establishing a human body behavior monitoring system based on wearable devices [7]. Most human body behavior monitoring methods based on wearable devices contain one or more of the work mentioned above. Classification techniques have evolved into many methods, and they have different recognition performance for behavioral detection. Currently, the classification methods of behavior monitoring based on wearable devices have Decision Trees (DT) [8], Threshold-based classification (TB), Support Vector Machine (SVM) [9] [10], K-Nearest Neighbor (K-NN) [11], Sparse Representation (SR), Artificial Neural Network (ANN) [12], Relevant Vector Machine (RVM) [13], etc. Anguita et al. [14] used the SVM classifier to identify behavior and use a fixed-point algorithm to reduce computational costs. Decision trees are often used for multi-level classification and recognition of human activities. Martin et al. [15] constructed a two-layer classifier, in which the first layer is used to determine the position of the device in the human body, and the second layer is used to identify human behavior. Diep et al. [10] used a three-axis accelerometer to collect data, and used sliding window technology to process the data. The values of each category in the three axes were used as eigenvalues, and were trained with the SVM model to identify the fall event. Literature [16] studies the classification of sports activities, and compares the methods of least squares, Bayes and K-nearest neighbors from the perspective of computational cost and classification accuracy. The experimental results show that the Bayes’ method has the lowest computational complexity and the classification accuracy is the highest among these methods. Literature [17] combines sparse representation and compressed sensing, and uses 1300 body motion sequences as training samples for experiments. Literature [18] combines the advantages of multi-layer perceptron and logistic regression classifiers, and the results show that the recognition effect is better than that of a single classifier. Research on human behavior monitoring based on wearable devices has achieved considerable success, but there are still many challenges in this area, such as: (1) Whether it is a wearable device or a handheld device like a smart phone, the data collected by devices in different body parts is different under the same physical activity. The sensor characteristics and wear position of the device determine the content of the collected data, which is crucial for human behavior monitoring, so how to obtain these device information needs to be solved. (2) The extraction of eigenvalues determines the quality of human behavior monitoring. However, the current effective eigenvalue extraction is subjectively selected by researchers through prior experience, which has great limitations. (3) Human body behavior monitoring system based on wearable devices will have great energy consumption when collecting and processing data. The multi-sensor collaborative work improves the recognition accuracy, but shortens the battery life and the user experience is also poor. Therefore, it is necessary to study a behavior monitoring method based on a single sensor.
Fig. 1. Wearing of accelerometer and direction of coordinate axis.
of the sensor is shown in Fig. 1. Wearable sensors transmit human acceleration information to the server through 4G network. The ADXL345 accelerometer described in this chapter works as the inertia principle, as shown in Eq. (1): 𝐹 (1) 𝑀 In the formula, 𝐴 is acceleration, 𝐹 is force, and 𝑀 is mass. The acceleration sensor measures the following values: acceleration of human motion, acceleration of gravity, acceleration caused by external force. Wherein, the gravity acceleration provides the orientation of the sensor, and the human body acceleration provides the body motion information. The factors influencing the measurement of human acceleration are: the type of motion, the placement of the accelerometer, and the direction of the accelerometer coordinate system. 𝐴=
3.2. Signal feature extraction This section only studies the change in acceleration of a certain action over a period of time, regardless of the details of the complex motion. First, this chapter uses the amplitude of the acceleration signal to describe human motion. The formula for the body’s synthetic acceleration value is shown in Eq. (2): √ 𝑆𝑎𝑐𝑐 = 𝑎𝑐𝑐𝑥2 + 𝑎𝑐𝑐𝑦2 + 𝑎𝑐𝑐𝑧2 (2) In the equation, 𝑆𝑎𝑐𝑐 is the combined acceleration, 𝑎𝑐𝑐𝑥 is the 𝑥axis acceleration, 𝑎𝑐𝑐𝑦 is the 𝑦-axis acceleration, and 𝑎𝑐𝑐𝑧 is the 𝑧-axis acceleration. Slow walking, fast walking, slow running, and fast running are regular sports. The difference between them is the intensity of the exercise, the size of the stride, and the frequency of the step. Like the up and down stairs, sitting down, standing up, jumping up are irregular movements, the data presented by the graphics are different. As shown in Fig. 2, the movement of the human body when going up and down the stairs is an approximate regular movement. But only looking at the 𝑥-axis and 𝑧-axis directions, the waveforms presented by the data are not obvious enough. The waveforms exhibited by the 𝑦-axis direction and the resultant acceleration 𝑆𝑎𝑐𝑐 value are extremely obvious, and the data exhibits regular fluctuations. When the human body performs the ‘‘sit down’’ and ‘‘sit up’’ actions, the acceleration value changes rapidly. When the human body jumps, the acceleration value also changes rapidly. (1) Standard Deviation The discrete level of the data set can be represented by the standard deviation. In the recognition of human motion, we can use the standard
3. Three-axis accelerometer signal analysis 3.1. Triaxial acceleration sensor According to the complexity and duration of human activities, it can be divided into three states: Short Events, Basic Activities and Complex Activities [19]. Short action refers to the type of exercise with short action time, which is the transformation of body posture, like standing up, sitting down, jumping. Basic activities are common simple actions in everyday life, such as walking, running, and standing. Complex activities are a combination of short-acting and basic activities, like eating, drinking tea or even listening to classes. The work in this chapter is to analyze short-action and basic activities by analyzing, identifying, and monitoring a person’s activities. The wearing mode 166
Z. Xu, J. Zhao, Y. Yu et al.
Computer Communications 151 (2020) 165–171
Fig. 2. Acceleration data of D05.
deviation to reflect the intensity of the human body’s exercise. When the person is stationary, the change in acceleration is extremely small, so the standard deviation of the data is small. When a person is exercising, the acceleration will become larger and the standard deviation of the data will vary depending on the type of exercise. Therefore, the standard deviation can be used to distinguish whether the human body is in a stationary state or a moving state. The standard deviation formula is shown in Eq. (3): √ √ 𝑁 √1 ∑ (𝑥 − 𝜇)2 (3) 𝑆𝐷 = √ 𝑁 𝑖=1 𝑖
𝐷𝐴𝑃 𝑇 = 𝑆𝑎𝑐𝑐𝑡 − 𝑆𝑎𝑐𝑐𝑡−𝑘
In the formula, 𝑆𝑎𝑐𝑐𝑡 is the peak (valley) value, 𝑆𝑎𝑐𝑐𝑡−𝑘 is the adjacent trough (peak) value, and 𝑘 is the number of sampling points between the two values. 4. Human behavior recognition based on improved 1D-CNNs 4.1. Analysis of 1D-CNNs Convolutional Neural Networks (CNNs) can be used for sensorbased human behavior recognition [20]. This chapter uses 1D-CNNs to classify and recognize human behavior. The structure of the traditional convolutional neural network [21] is shown in Fig. 3. As can be seen from the figure, CNNs mainly include input layer, hidden layer and output layer. The input layer of CNNs can handle one-dimensional or multidimensional data. The hidden layer is composed of convolutional layers, pooling layers, and fully connected layers. The convolutional layer extracts feature from input data; the pool layer receives the eigenvalues from the convolutional layer and carries out feature selection and information filtering; the fully connected layer is equivalent to the hidden layer in the feedforward neural network, which expands the multidimensional data into vectors and transfers them to the next layer by using the excitation function. Finally, the output layer of CNNs uses the softmax function or the logistic function to output the classification results. The calculation formula for the CNNs convolutional layer is as follows:
In the formula, 𝑁 is the number of items in the data set, and 𝜇 is the mean of the data set. (2) Interval of Peaks The walking frequency of the human body is different when walking and running, so the fluctuation frequency of the data drawing is different. Therefore, we choose the interval between two adjacent peaks of the data to distinguish the type of motion. The formula is shown in (4):
Inv𝑙 = 𝑇𝑡 − 𝑇𝑡−1
(5)
(4)
In the equation, 𝑇𝑡−1 is the time at which a peak appears, and 𝑇𝑡 is the time at which the next peak appears. 𝑙 is the number of points sampled in the adjacent wave peaks. (3) Difference between Adjacent Peaks and Troughs Different types of exercise have different ranges. The recognition of human motion state requires high real-time performance, so how to extract real-time information becomes very important when extracting feature points. Therefore, this chapter will combine the peaks and troughs value of the acceleration 𝑆𝑎𝑐𝑐 as the research object. In the data processing process, we compare each newly received sample value with the previous sample value. If there is no turning point in the waveform of the sampling point, continue to observe the new sampling value. If the waveform has a turning point, the point is considered to be the extreme point. In this chapter, we take the difference between adjacent peaks and troughs of 𝑆𝑎𝑐𝑐 as the eigenvalue, as shown in Eq. (5):
⎛∑ ⎞ 𝑋𝑗𝑙 = 𝑓 ⎜ 𝑋𝑖𝑙−1 ∗ 𝐾𝑖𝑗𝑙 + 𝑏𝑙𝑗 ⎟ ⎟ ⎜𝑖∈𝑀 𝑗 ⎝ ⎠
(6)
In the formula, 𝑋𝑗𝑙 represents the 𝑗th feature map of the 𝑙th layer, 𝑀𝑗 represents the set of selected input feature maps, 𝐾𝑖𝑗𝑙 represents the convolution kernel function, 𝑓 (⋅) represents the activation function, and 𝑏𝑙𝑗 represents the offset parameter. 167
Z. Xu, J. Zhao, Y. Yu et al.
Computer Communications 151 (2020) 165–171
Fig. 3. Structure of CNNs.
𝛿𝑗𝑙
Error signal of the 𝑗th feature map of layer 𝑙: ( ) = 𝜔𝑙+1 𝑓 ′ (𝑖𝑝𝑙𝑗 ) ⋅ 𝑢(𝛿𝑗𝑙+1 ) 𝑗
training sample set is artificially set based on past experience, because a good sample learning method can greatly improve the performance of the network. Therefore, this paper proposes a method called Sample Autonomous Learning (SAL) to construct an effective sample training set. The specific method is as follows: In order to improve the efficiency of network classification and reduce the classification training set in limited resources and time, the method gradually adjusts the number of sample sets with incorrect classification to find the optimal classification sample set. The algorithm process is as follows: First, the initial sample 𝐷𝑖 is selected and the number of training samples of 𝐷𝑖 is set, and the initial sample data 𝐷𝑖 is input into the 1D-CNNs network for training. Then, the error classification rate 𝜀𝑖 of each category at the output is calculated separately and compared with the set threshold 𝜃. If 𝜀𝑖 > 𝜃, the number of data of the training sample is expanded until 𝜀𝑖 ≤ 𝜃 or 𝜀𝑖 reaches convergence on the new sample training set. By verifying the 1D-CNNs network, the optimal training sample set can be obtained to avoid the occurrence of over-fitting problem.
(7)
In the formula, 𝜔𝑙+1 represents the weight of the 𝑗th feature map 𝑗 of the 𝑙 + 1th layer, 𝑖𝑝𝑙𝑗 represents the input value of the 𝑗th neuron of the 𝑙th layer, 𝑓 ′ (⋅) represents the partial derivative of the activation function, and 𝑢 (⋅) represents the upsampling function. The weight gradient function of the convolution kernel is as shown in Eq. (8): ∑( ) ( ) 𝜕𝐺 = 𝛿𝑗𝑙 𝑑𝑖𝑙−1 𝑢𝑣 (8) 𝑙 𝑢𝑣 𝜕𝐾𝑖𝑗 𝑢,𝑣 In the equation, 𝑑𝑖𝑙−1 is the block in the 𝑗th feature map of the 𝑙−1th layer, multiplied by 𝐾𝑖𝑗𝑙 in the convolution process, and (𝑢, 𝑣) represents the position of the convolutional feature map. The calculation formula for the pooling layer is as shown in Eq. (9): ( ) ( ) 𝑋𝑗𝑙 = 𝑓 𝜔𝑙𝑗 𝑑 𝑋𝑖𝑙−1 + 𝑏𝑙𝑗 𝑡 (9) In the formula, 𝑑 (⋅) denotes the downsampling function. The gradient function of the pooled layer is similar to Eq. (8). In CNNs, the weight update from time 𝑡 to time 𝑡 + 1 can be expressed as: 𝑊 (𝑡 + 1) = 𝑊 (𝑡) + 𝜎𝛿(𝑡)𝑥(𝑡)
(10)
𝜎 represents the learning rate, 𝛿(𝑡) indicates the error term, and 𝑥(𝑡) represents the input of the neuron. CNNs are a good way to identify simple patterns in data and then generate more complex patterns in more advanced layers. CNNs contain one-dimensional convolution, two-dimensional convolution, and threedimensional convolution. Their processing methods and characteristics are the same, the difference is the dimension of the input data and the way the filter slides between the data. The one-dimensional convolutional neural networks (1D-CNNs) can effectively obtain effective features from the shorter segments of the overall data set, which can be used for the analysis and processing of sensor data. 4.2. Improvementof 1D-CNNs Traditional convolutional neural networks have been able to handle different classification problems better. However, as a neural network for globally optimized supervised learning, 1D-CNNs still have some problems. For example, the sample learning method of 1D-CNNs cannot handle the over-fitting problem very well. In domestic and foreign research, some researchers tried to alleviate the problem by increasing the number of samples. The over-fitting problem means that the training samples are too strict in the training process of the network, so that the trained network has a poor classification effect on the sample data that is slightly different from the training set. The specific performance is that as the number of 1D-CNNs training increases, the error rate of the sample test first decreases and then rises. In the traditional 1D-CNNs algorithm, the
The formula for calculating the error classification rate 𝜀𝑖 is as follows: 𝑒 𝜀𝑖 = 𝑖 (11) 𝐷𝑖 In the formula, 𝑒𝑖 is the number of samples misclassified, and 𝐷𝑖 is the total number of samples classified. The following describes the process of human behavior state recognition based on improved 1D-CNNs: 168
Z. Xu, J. Zhao, Y. Yu et al.
Computer Communications 151 (2020) 165–171
Table 1 Types of activities. Num
Category
Number of experiments
Duration (s)
D01 D02 D03 D04 D05 D06 D07 D08 D09 D010 D011
Slow walking Fast walking Slow running Fast running Slow up and down stairs Fast up and down stairs Stand up slowly from a half-height chair Stand up quickly from a half-height chair Stand up slowly from a low chair Stand up quickly from a low chair Jumping up
1 1 1 1 5 5 55 5 5 5 5
100 100 100 100 25 25 12 12 12 12 12
positive, and 𝐹 𝑁 is the number of activities incorrectly predicted to be negative. (2) Precision: The ratio of the number of activities of a certain type to that of all activities of that type. It is an index for predicting results. 𝑇𝑃 (13) 𝑇𝑃 + 𝐹𝑃 (3) Recall: A certain type of activity is correctly predicted as the ratio of the number of such activities to the original total number of such activities. It is an indicator for the original sample instance. 𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =
𝑇𝑃 (14) 𝑇𝑃 + 𝐹𝑁 Table 2 shows the confusion matrix for identifying 11 activities based on the improved 1D-CNNs experiment. The confusion matrix can directly show the experimental results and help us evaluate the performance of the algorithm. Each row represents the actual behavioral activity data of the human body, and each column represents data in which the actual behavioral activity is identified as the current behavioral activity. For example, the second row of data in the table: the number of people actually performing ‘‘slow walking’’ activities is 38, of which 36 are correctly identified as ‘‘slow walking’’ activities, and 2 are incorrectly identified as ‘‘fast walking’’ activities. Experiments based on improved 1D-CNNs for human behavior recognition have effectively improved the recognition accuracy of 11 human behaviors compared to other traditional classification methods, and achieved better classification and recognition effects. Table 3 shows the final results of human behavior recognition experiments based on improved 1D-CNNs. It can be seen from the table that the average accuracy of the identification of each activity is 92.8%, the average accuracy rate is 98.7%, and the average recall rate is 92.8%. In this method, the possible causes of misidentification are as follows: First, when the human body walks, the speed is too fast and close to the running speed; correspondingly, the human body is slow in running, and the speed is close to the speed of walking fast. Secondly, when the human body is standing up, the range of motion is too large. Thirdly, when jumping, the human body only slightly leaves the 𝑅𝑒𝑐𝑎𝑙𝑙 =
5. Experiments and analysis In order to build a unified evaluation standard, this paper uses the global open SisFall data set [22] for simulation experiments. SisFall is a complete data set containing fall and daily behavioral activities. As shown in Table 1, these data sets are composed of human activity data collected by self-developed embedded devices, and the data can be easily copied for data analysis and simulation experiments. The data set contains 19 types of daily behavioral activities and 15 types of falls. The data collected were from the daily activities of 38 subjects, including 23 adults aged 19–30 and 15 elderly aged 60–75. In this chapter, the improved 1D-CNNs method is analyzed from the accuracy, precision and recall. (1) Accuracy: Ratio of the number of samples correctly classified to the total number of samples. 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =
𝑇𝑃 + 𝑇𝑁 𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁
(12)
In the formula, 𝑇 𝑃 is the number of activities correctly predicted to be positive, 𝑇 𝑁 is the number of activities correctly predicted to be negative, 𝐹 𝑃 is the number of activities incorrectly predicted to be 169
Z. Xu, J. Zhao, Y. Yu et al.
Computer Communications 151 (2020) 165–171 Table 2 Confusion matrix of human behavior recognition.
Table 3 Results of human behavior recognition experiment.
6. Conclusion
Action
Accuracy
Precision
Recall
D01 D02 D03 D04 D05 D06 D07 D08 D09 D010 D011 Average
97.3% 89.7% 89.7% 97.3% 97.3% 94.9% 90.4% 89.3% 92.1% 91.6% 91.3% 92.8%
99.8% 99.5% 99.5% 99.8% 98.9% 99.6% 97.3% 97.8% 96.8% 98.8% 98.0% 98.7%
94.7% 92.1% 92.1% 94.7% 94.7% 97.3% 89.5% 87.9% 92.1% 91.6% 94.2% 92.8%
Human body behavior monitoring based on wearable devices mainly includes: collecting effective human activity data, extracting effective feature values, studying human behavior recognition or classification methods, and establishing a human body behavior monitoring system based on wearable devices. Most human body behavior monitoring methods based on wearable devices contain one or more of the work mentioned above. For behavior recognition, traditional methods can only recognize human activities offline. But in practical application, real-time monitoring results are required. Considering of the above two points, in this paper, we study on efficiently real-time behavior recognition algorithm using a single sensor. Then, we propose a human behavior recognition method based on improved One-Dimensional Convolutional Neural Networks (1D-CNNs). According to the motion characteristics, we extract the eigenvalues which can distinguish the types of activities. At the same time, we propose a sample autonomous learning method, which aims to find the optimal sample training set and avoid over-fitting problems in traditional CNNs. In the recognition of 11 human activities, our method can reach the average accuracy of 98.7%. Compared with other behavior recognition methods in the same dataset, better classification is achieved by this method. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. CRediT authorship contribution statement
Fig. 4. Results comparison of methods in behavior recognition.
Zhiou Xu: Conceptualization, Methodology, Software. Juan Zhao: Data curation, Writing - original draft, Visualization. Yi Yu: Visualization, Investigation. Haijun Zeng: Supervision.
ground, and the movement is more relaxed. Fourth, the sensors are not fixed properly. For human behavior recognition, this chapter selects the methods of [20] and [21] as reference experiments. The comparison method and the method proposed in this paper are run on SisFall, and the recall result is shown in Fig. 4. It can be seen from the figure that the method of [21] is slightly better than the method of this paper in the identification of D07. The reason may be that the literature [21] selects the acceleration dip as the eigenvalue, and D07 is the ‘‘sitting→starting’’ movement. When the human body performs this activity, the body inclination changes greatly, so the method of [21] is sensitive to the recognition of the activity. However, it can be seen from the figure that the recognition effect of this method is better than the methods of [20] and [21,22] in the recognition of other behaviors.
Acknowledgment Foundation item: Special Topic of Independent Innovation for Double-class Construction of Computer College (2018ZZCX14). References [1] S. Bhattacharya, A.M. Krishna, D. Lombardi, et al., Economic MEMS based 3-axis water proof accelerometer for dynamic geo-engineering applications, Soil Dyn. Earthq. Eng. 36 (4) (2012) 111–118. [2] K. Taraldsen, S.F. Chastin, I.I. Riphagen, et al., Physical activity monitoring by use of accelerometer-based body-worn sensors in older adults: a systematic literature review of current knowledge and applications, Maturitas 71 (1) (2012) 13–19. 170
Z. Xu, J. Zhao, Y. Yu et al.
Computer Communications 151 (2020) 165–171 [22] Huijuan Hao, Maoli Wang, Yongwei Tang, Qingdang Li, Research on data fusion of multi-sensors based on fuzzy preference relations, Neural Comput. Appl. 31 (S-1) (2019) 337–346.
[3] J.J. Kavanagh, H.B. Menz, Accelerometry: A technique for quantifying movement patterns 63 during walking, Gait Posture 28 (1) (2008) 1–15. [4] J. Majid, R.F. Atena, R. Katarzyna, et al., A comprehensive analysis on wearable acceleration sensors in human activity recognition, Sensors 17 (3) (2017) 529–554. [5] Y. Cao, Y.Z. Zhang, T. Wen, P. Li, Research on dynamic nonlinear input prediction of fault diagnosis based on fractional differential operator equation in high-speed train control system, Chaos, https://doi.org/10.1063/1.5085397. [6] T. Plötz, Y. Guan, Deep learning for human activity recognition in mobile computing, Computer 51 (5) (2018) 50–59. [7] J. Cheng, O. Amft, G. Bahle, et al., Designing sensitive wearable capacitive sensors for activity recognition, IEEE Sens. J. 13 (10) (2013) 3935–3947. [8] I.D. Dinov, Decision tree divide and conquer classification, Data Sci. Predict. Anal. (2018) 307–343. [9] Yi Ai, Zongping Li, Mi Gan, Yunpeng Zhang, Daben Yu, Wei Chen, Yanni Ju, A deep learning approach on short-term spatiotemporal distribution forecasting of dockless bike-sharing system, Neural Comput. Appl. 31 (5) (2019) 1665–1677. [10] J. Wang, Y. Chen, S. Hao, et al., Deep learning for sensor-based activity recognition: A survey, Pattern Recognit. Lett. 119 (2017) 3–11. [11] H. Jian, H. Chen, A portable falls detection and alerting system based on k-NN algorithm and remote medicine, China Commun. 12 (4) (2015) 23–31. [12] M.H.M. Noor, Z. Salcic, I.K. Wang, Adaptive sliding window segmentation for physical activity recognition using a single tri-axial accelerometer, Pervasive Mob. Comput. 38 (1) (2017) 41–59. [13] G.M. Foody, RVM-based multi-class classification of remotely sensed data, Int. J. Remote Sens. 29 (6) (2008) 1817–1823. [14] D. Anguita, A. Ghio, L. Oneto, et al., Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine, Ambient Assist. Living Home Care (2012) 216–223. [15] H. Martín, A.M. Bernardos, J. Iglesias, et al., Activity logging using lightweight classification techniques in mobile devices, Pers. Ubiquitous Comput. 17 (4) (2013) 675–695. [16] K. Altun, B. Barshan, Human activity recognition using inertial/magnetic sensor units, in: International Conference on Human Behavior Understanding, 2010, pp. 38–51. [17] Y. Cao, H.K. Lu, T. Wen, A safety computer system based on multi-sensor data processing, Sensors 19 (4) (2019) 818. [18] M.N.S. Zainudin, M.N. Sulaiman, N. Mustapha, et al., Activity recognition based on accelerometer sensor using combinational classifiers, in: Open Systems, IEEE, 2015, pp. 68–73. [19] J.L. Reyes Ortiz, L. Oneto, Samà Monsonís, Albert, et al., Transition-aware human activity recognition using smartphones, Neurocomputing 171 (2016) 754–767. [20] M. Zeng, T.N. Le, B. Yu, et al., Convolutional neural networks for human activity recognition using mobile sensors, in: International Conference on Mobile Computing, Applications and Services, IEEE, 2015, pp. 197–205. [21] Y.L. Lecun, L. Bottou, Y. Bengio, et al., Gradient-based learning applied to document recognition, Proc. IEEE 86 (11) (1998) 2278–2324.
Zhiou Xu was born in 1976. He received Ph.D degree in power electronic and electric drive from China University of Mining and Technology in 2011. He works in China University of Mining and Technology since 2011. His research interests include the power electronic technology, smart grid and computer control technology.
Juan Zhao was born in 1981. In 2019, she obtained a master’s degree in electronics and communication engineering from the school of information engineering. Since 2009, she have been working in Henan industrial and trade vocational college. Her research interests include communication networks and information security.
Yi Yu received B.E. degree from Zhejiang University in 1999. He works in Operating Branch of Ningbo Rail Transit Group Co., LTD. His research interests include power supply technology and measure and control systems.
Haijun Zeng received B.E. degree from Lanzhou Jiaotong University in 2002. He works in Operating Branch of Ningbo Rail Transit Group Co., LTD. His research interests include power supply technology and measure and control systems.
171