Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals

Future Generation Computer Systems xxx (xxxx) xxx

Contents lists available at ScienceDirect

Future Generation Computer Systems journal homepage: www.elsevier.com/locate/fgcs

Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals ∗

Feiyun Xiao a , , Decai Yang b , Zhongming Lv a , Xiaohui Guo c , Zhengshi Liu a , Yong Wang a a

School of Mechanical Engineering, Hefei University of Technology, Hefei 230009, China Aerospace System Engineering Shanghai, Shanghai 201109, China c School of Electronics and Information Engineering, Anhui University, Hefei 230009, China b

article

info

Article history: Received 24 July 2019 Received in revised form 3 October 2019 Accepted 18 November 2019 Available online xxxx Keywords: VMD sEMG Permutation entropy Identification of hand movement Classification

a b s t r a c t Research of human hand movements recognition can be applied to artificial limb control, motion recognition of wearable exoskeleton, human–computer interaction in virtual reality and so on. Surface Electromyogram (sEMG) signal is the preferred source. There are many researches on how to extract information from sEMG signal and apply it to human motion recognition. However, how to extract the feature signal from sEMG signal is a difficult problem in the research of human hand movement recognition based on sEMG signal. In this paper, a method based on Variational Mode Decomposition (VMD) and composite permutation entropy index (CPEI) method is proposed for hand motion classification. Previously, the VMD method had not been used in human hand motion recognition studies. The method proposed in this work applies the VMD method to decompose the original sEMG signal into multiple Variational Mode Functions (VMFs) and calculate the corresponding CPEI of each signal component. Three feature selection methods (Infinite Latent Feature Selection (ILFS), ReliefF, and Laplacian Score) were applied to rank the features and remove the unimportant features. Three classifiers (Naive Bayes, K-NN, and Bagging) were used to recognize the hand actions. Ten volunteers participated in the experiment, and the experimental data were used to verify the proposed method. The average accuracy was 94.28 ± 1.26% for the proposed method with Laplacian Score for feature sorting and selection, and Bagging as classifier. Besides, 600 randomly selected hand movements are predicted (CPU is i5-8250U, ram is 8g, processing software is Spyder, python 3.7), and the corresponding execution time of proposed method is 0.56 s. © 2019 Elsevier B.V. All rights reserved.

1. Introduction The research on human hand motion recognition plays a very important role in artificial limb control [1], wearable exoskeleton control [2], human–computer interaction in virtual reality [3] and other fields. The recognition accuracy and real-time performance directly determine the actual effect in the corresponding application scene [4]. In most hand motion recognition studies [4–7], surface electromyogram (sEMG) signal is the most widely used source of human intention signal, which is accompanied by muscle contraction [8]. One movement is corresponding to specific muscles, and different movements are along with different sEMG signals [4]. In addition, for amputees, sEMG signals corresponding ∗ Corresponding author. E-mail addresses: [email protected], [email protected] (F. Xiao), [email protected] (D. Yang), [email protected] (Z. Lv), [email protected] (X. Guo), [email protected] (Z. Liu), [email protected] (Y. Wang).

to the muscles of residual limbs can be mapped to the missing motion function, so as to re-establish the corresponding relationship between muscles and movements, making it easier for patients to learn and adjust [1,4]. The hand movements recognition from sEMG signal include the following main steps: (1) acquisition of sEMG signal; (2) sEMG signal preprocessing; (3) feature extraction of sEMG signal; (4) movement classification and recognition. Since the frequency band of sEMG signal is between 10 Hz and 500 Hz, it is not feasible to directly apply sEMG to identify hand movements [9]. Many scholars proposed to extract feature of sEMG signal, and use methods of feature engineering to extract important information as much as possible to replace the original sEMG signal [10–12]. Many sEMG features (time-domain features: mean absolute value, slope sign change, zero crossing, root mean square, etc.; frequency-domain features: mean frequency, peak frequency, mean power, total power, etc. [11]) have been applied in gesture recognition [13,14], however, the corresponding hand motion recognition effect is still not very well when

https://doi.org/10.1016/j.future.2019.11.025 0167-739X/© 2019 Elsevier B.V. All rights reserved.

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.

2

F. Xiao, D. Yang, Z. Lv et al. / Future Generation Computer Systems xxx (xxxx) xxx

only using time-domain features or frequency-domain features. Therefore, many scholars consider to decompose the sEMG signal, obtain multiple sub-signals, and then extract corresponding feature signals from each sub-signal [2,13,15–18]. Based on existing literatures, there are mainly two decomposition methods for sEMG signal, which are Wavelet decomposition method and Empirical Mode Decomposition (EMD) method respectively. Wavelet decomposition method is a time-frequency analysis method, which can decompose sEMG signal into multiple orthogonal time series with different frequency bands [19]. However, when different mother wavelet functions are selected, the corresponding decomposition effects of wavelet decomposition methods are different [13,15]. The drawback of this approach is that you have to define the mother wavelet function in advance, and different mother wavelet functions will affect the final result, as well as the hard band-limits [20]. EMD proposed by Huang et al. is a nonlinear and non-stationary time-domain decomposition method [21], which is an adaptive, data-driven algorithm. A given signal can be decomposed into multiple intrinsic mode functions (IMFs). Each IMF represents a narrowband frequency– amplitude modulation, which is usually associated with actual specific physical processes. At present, EMD method is widely used in sEMG signal decomposition, which has been applied to gesture recognition [16], sEMG signal denoising [22], interference elimination [23] and so on. The disadvantages of EMD method are that it is sensitive to noise and sampling, the number of decomposed signal components cannot be set artificially, the frequency band of each component may have a large overlap area, and it lacks mathematical theory [20]. In view of these problems of Wavelet decomposition and the EMD decomposition method discussed above, Konstantin Dragomiretskiy proposed a method which is a generalization of the classic Wiener filter into multiple and adaptive bands, that is the Variational Mode Decomposition method (VMD) method [20]. The model looks for an ensemble of modes and their respective center frequencies, such that the modes collectively reproduce the input signal, while each being smooth after demodulation into baseband. The original signal is decomposed into several variational model component, that are the Variational Mode Functions (VMFs). The sum of modes can reproduce the input signal, each VMF signal has its own center frequency, and each mode is smooth after demodulation into the baseband. Finally, these narrow-band VMF components are obtained according to the frequency domain characteristics of the actual signals, completing the adaptive segmentation of the signal frequency band and effectively avoiding modal aliasing. At present, VMD method is mainly used in gear fault diagnosis [24], wind speed prediction [25], neuromuscular disease diagnosis [26] and electro cardio signal decomposition [27,28]. The above advantages of VMD make it be feasible to be applied to sEMG signal decomposition, and then applied to human hand motion recognition research, which may provide some better effects to realize hand motion recognition. However, there is no literature on how to apply VMD method to human hand motion recognition. After the decomposition of sEMG signal, multiple sEMG subsignals are obtained. However, how to apply the decomposed signals and obtain corresponding feature signals is another problem that needs to be considered. Many scholars considered to extract corresponding time-domain signals such as mean absolute value and root mean square from decomposed signals [2,13,15–18]. However, it still has the same problem when replacing sEMG with the time-domain feature of sEMG. Entropy represents signal complexity. Some scholars consider extracting entropy features of sEMG signal, such as sampling entropy [29], fuzzy entropy [30], approximate entropy [31], and permutation entropy [32]. Permutation entropy is an indicator to represent the complexity of

Table 1 The information of participants. Participants

Age

Sex (F/M)

Weight (kg)

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

20 33 51 32 26 44 35 46 36 39

M M M F F F F F M M

64 71 62 49 46 45 52 59 72 76

time series. It is simple in concept, fast in operation, suitable for nonlinear signals, and has high anti-interference ability and good robustness [33]. Literature [34,35] pointed out that the composite permutation entropy index (CPEI) has more information differentiation performance compared with the permutation entropy, which can make full use of the permutation entropy characteristics corresponding to different indicators. Therefore, it is possible to extract the CPEI feature corresponding to the subsignal of sEMG signal and apply it to hand motion recognition, which will may have a better effect. Based on the above analysis, this paper proposed a hand motion recognition algorithm based on the combination of VMD and CPEI. This algorithm uses VMD method to decompose the original sEMG signal, and then calculates the CPEI of each signal component. Different feature selection methods (ILFS, ReliefF, and Laplacian Score) are applied to sort the importance of features and remove unimportant features. Different classifiers (Naive Bayes, K-NN, and Bagging) are introduced to classify the hand motion and the corresponding performance are compared. In this work, three sEMG sensors were used to collect sEMG signals corresponding to flexor carpi radialis, extensor carpi radialis longus and extensor digitorum, which were used to identify six hand movements: forearm pronation (FP), forearm supination (FS), hand closure (HC), hand opening (HO), wrist extension (WE) and wrist flexion (WF). The motivation of this work is to proposed a hand motion classification method with high accuracy and well real-time performance. Compared with existing methods, the advantages of the method proposed in this paper are as follows: (1) Compared with EMD method and wavelet decomposition method, the sEMG signal components decomposed by VMD all have a main frequency and a narrow frequency band, making each component more distinguishable from each other and more conducive to distinguishing different actions. (2) CPEI fully combines the advantages of permutation entropy (PE) under different parameters. Compared with some other features based on entropy theory (such as PE, sample entropy, and fuzzy entropy), the accuracy of hand movement classification is higher. (3) While ensuring real-time performance, the number of electrodes used is small and the accuracy is high. 2. Experiment 2.1. Subjects Ten healthy subjects (20∼51 years old, 5 males, 5 females) participated in this experiment. Specific information is shown in Table 1. Before the beginning of the experiment, each subject signed an ethical agreement and confidentiality agreement with the ethics committee of Hefei University of Technology, so their information was replaced by S1∼S10, as shown in Table 1. All participants were aware of the ethical protocol and personal information confidentiality agreement prior to the experiment and agreed to conduct the experiment.

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.

F. Xiao, D. Yang, Z. Lv et al. / Future Generation Computer Systems xxx (xxxx) xxx

3

Fig. 1. The experimental set-up. Flexor carpi radialis (CH. B), extensor carpi radialis longus (CH. C) and extensor digitorum (CH. A).

2.2. Experimental scheme sEMG signals are generated in the process of muscle contraction, and their frequency band distribution is 10 Hz–500 Hz. The frequency band is wide, so it is not suitable to apply original sEMG signal directly for motion recognition. sEMG applied in gesture recognition has the advantages of no side effects, simple and convenient, and can reflect the degree of muscle contraction, etc. Therefore, sEMG is used as the original signal for gesture recognition and subsequent artificial limb control or hand exoskeleton control. In this work, three sEMG sensor were used. The corresponding muscles were flexor carpi radialis (CH. B), extensor carpi radialis longus (CH. C) and extensor digitorum (CH. A) [1,13]. These were used to identify the FP, FS, HC, HO, WE, WF. In the course of experiment, when the motions are FP, FS, WE, WF, the fingers were relaxed (see Fig. 2). Among them, the flexor carpi radialis is a relatively shallow muscle that runs diagonally across the front of the forearm from the medial epicondyle and connects laterally at the bottom of the second and third metatarsal bones [36]. The extensor carpi radialis longus is basically superficial, attached proximally to the lateral epicondyle above the lateral epicondyle of the supracondylar ridge, and then it extends posteriorly along the lateral forearm, under the two tendons of the thumb, and then under the adductor band of the extensor at the bottom of the second metacarpal bone [36]. The extensor digitorum muscle is a superficial muscle on the posterior forearm and hand. It attaches proximally to the lateral epicondyle of the humerus as part of the common extensor tendon. It passes under the extensor retinaculum to attach distally on the distal phalanx of the second through fifth fingers via the extensor expansion. In the area of the metacarpals are interconnecting bands joining the four extensor digitorum tendons. These interconnecting bands limit independent finger extension [36]. The extensor digitorum muscle is the only common extensor muscle of the fingers. It extends the MCP, PIP, and DIP joints of the second, third, fourth, and fifth fingers. Before the experiment, the corresponding positions were shaved and alcohol was applied to the corresponding electrode area. The Ag/AgCl electrodes (YONGKANGDA Technology Inc, ECG electrodes, circular electrode with 6 mm diameter) were used. Each sEMG sensor corresponding to the triode electrodes (positive electrode and negative electrode, and the reference electrode). The reference electrode, the positive electrode and the negative electrode form an equilateral triangle with the electrode spacing of 20 mm, as shown in Fig. 1. sEMG signals collected are first encoded by MyoScan model SA9503M (Thought Technology Ltd), and then by FlexComp Infiniti encoder (Thought Technology Ltd). The sampled signal is transmitted to the PC via the TTUSB interface module. The sampling rate was 2048 Hz. Each participant repeated each action 50 times in one group with a 30-min rest interval. The number of group was 50. So, there

Fig. 2. The corresponding hand movements. In this work, three sEMG sensor were used. The corresponding muscles were flexor carpi radialis, extensor carpi radialis longus and extensor digitorum. These were used to identify the forearm pronation (FP), forearm supination (FS), hand close (HC), hand open (HO), wrist extension (WE), wrist flexion (WF).

are 15,000 section of sEMG signals per participant. The postprocessing software of experimental data were MATLAB and Spyder (Python 3.7). 3. The proposed method The block diagram of the proposed algorithm in this work is shown in Fig. 3. It is mainly divided into six parts: sEMG signal acquisition, sEMG signal preprocessing, sEMG signal decomposition, sub-signal feature calculation, feature sorting and selection, and hand movement classification. Each section of the diagram is identified by a different color block. Six actions are identified by collecting sEMG signals from three muscles. sEMG signal acquisition has been introduced in Section 2. This chapter mainly introduces the remaining five parts. 3.1. Pre-processing of sEMG signal Firstly, the collected sEMG signal is band-pass filtered. The band-pass filter selected is a fourth-order Butterworth filter, and the passband frequency is 10 Hz–500 Hz. In order to prevent power frequency interference, a 50 Hz Notch filter is adopted for filtering. The Q factor of the designed notch filter is 3. Because the dimensional difference of features will have bad effect on the performance of hand motion recognition, it is necessary to normalize the data set of these features. In this work, a classical isometric maximal voluntary contraction method was applied to normalize the features [37], which means that the subjects should perform with muscle arm position against resistance isometric contraction, then use the measured value corresponding to the maximum sEMG of muscle for the rest of the sEMG signals of corresponding muscles to normalize. Fig. 4 shows an example of sEMG signal and the comparison of its spectrum before and after pretreatment. 3.2. sEMG decomposed with variational mode decomposition The VMD method extends the classical Wiener filter to multiple adaptive bands [20]. The VMD method changes the problem of model estimation to the variational problem, and updates the model and its center frequency constantly, finally the model is transformed by the Inverse Fourier transformation to the time domain. A set of models and their respective center frequency can be obtained from the model, so that the input signal can

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.

4

F. Xiao, D. Yang, Z. Lv et al. / Future Generation Computer Systems xxx (xxxx) xxx

Fig. 3. The block diagram of the proposed method. It is mainly divided into six parts: sEMG signal acquisition, sEMG signal preprocessing, sEMG signal decomposition, sub-signal feature calculation, feature sorting and selection, and hand movement classification. Each section of the diagram is identified by a different color block. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

the following equation.

2 ] ) [( ∑   ∂t δ (t ) + j ∗ uk (t ) e−jwk t    πt 2 k  2 ⟨ ⟩   ∑ ∑   uk + f (t ) − uk (t ) + λ (t ) , f (t ) −  

L ({uk } , {wk } , λ) = α

k

k

2

(3) where α is the regularization factor, λ is the Lagrangian multiplier. By introducing the alternate direction method of multipliers (ADMM) [38,39], the corresponding result is shown as follow. unk +1 Fig. 4. One example of sEMG signal.

be reconstituted from the models. In addition, each mode after demodulation is smooth. The goal of VMD is to decompose a input signal f into several sub signals uk , and the superposition of these sub-signals uk constitute the input signal f as shown in Eq. (1). Where f is the original signal, ∆ represents the noise signal, and rn is the remainder term f =f +∆=

∑

i

uˆ nk +1 (w) =

(1)

k=1

Here, each sub signal uk has a center frequency wk , and it has a limited band. The center frequency wk is determined by the decomposition, and the corresponding constraint condition is Eq. (1). The corresponding constrained variational problem is shown in Eq. (2).

{ 2 } [( ) ] ∑   j −jwk t   min ∂t δ (t ) + π t ∗ uk (t ) e  {uk },{wk } 2 k ∑ s.t. uk = f

(2)

k

By making use of a quadratic penalty term and Lagrangian multipliers to render the problem unconstrained, we can obtain

fˆ (w) −

∑

i̸ =k

uˆ i (w) +

ˆ λ(w) 2

(5)

1 + 2α (w − wk )2

⏐ ⏐2 w ⏐uˆ k (w)⏐ dw = ∫∞⏐ ⏐ ⏐uˆ k (w)⏐2 dw 0 ( ∫∞

w uk + r n

(4)

2

By making use of the Parseval/Plancherel Fourier isometry method, this problem can be solved in spectral domain.

n+1 k

M

⎧ 2 [( ] ) ⎨    j −jwk t  δ t + ∂ = argmin α  e ∗ u t ( ) ( ) t k   πt uk ∈X ⎩ 2  2 ⎫  ⎬ ∑ λ (t )    ui (t ) + + f (t ) −   2  ⎭

0

λˆ n+1 (w) ← λˆ n (w) + τ fˆ (w) −

(6)

) ∑

unk +1

ˆ

(w)

(7)

k

where, τ is the parameter of noise tolerance. The condition of convergence of the above equation is as follow.

 ∑   ˆ n+1 − uˆ n 2 k 2 k uk <ε  n 2 uˆ  k

(8)

2

For the given discriminant accuracy ε > 0, the whole iteration was finished when Eq. (10) is satisfied. Finally, we can obtain M narrow band VMF components and finish the adaptive segmentation of frequency band of signal, thus avoiding the mode mixing.

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.

F. Xiao, D. Yang, Z. Lv et al. / Future Generation Computer Systems xxx (xxxx) xxx

3.3. sEMG feature computation Permutation entropy is a method to detect the randomness and dynamic mutation of time series, which is characterized by simple calculation and strong anti-noise ability [32]. In order to better analyze the signal complexity, CPEI is selected as the sEMG feature to replace the original sEMG signal. The calculation formula of CPEI is as follows. CPEI =

PE1 + PE2 ln (m1 ) + ln (m2 )

(9)

where, PE1 and PE2 represent permutation entropy, while m1 and m2 represent embedding dimension. τ1 is the lag of PE. The solution of the permutation entropy is listed below. (1) Reconstruct the time series u (1) , u (2) , . . . , u (N ) with length of N. The reconstructed signal was represented as X (i) = [u (i) , u (i + τ1 ) , . . . , u (i + (m − 1) τ1 )]. (2) Sort each X (i) with ascending order, that is u (i + (j1 − 1) τ1 ) ≤ u (i + (j2 − 1) τ1 ) ≤ · · · ≤ u (i + (jm − 1) τ1 ). If two elements are equal, sort it with the subscript i of ji . Then the sequence X (i) is mapped to (j1 , j2 , . . . , jm ). This is one of m! arrangement. (3) Obtain the m! symbols probability distribution p1 , p2 , . . . , pk , where k ≤ m! (4) Obtain the PE as follow. PE = −

∑

pi × ln (pi )

(10)

(5) A normalized representation is usually used for convenience. PE =

−

∑

pi × ln (pi )

ln (m!)

(11)

After calculating the features, the features can be sorted according to the importance of the features, and then the first few important features can be selected according to the reordered feature sequence, so as to reduce the amount of computation. Here, three feature ranking methods are selected, including two supervised methods and one unsupervised method. The details are as follows. 3.4. Feature ranking and selection 1. Infinite Latent Feature Selection (ILFS) [40] ILFS performs the ranking step while considering all the possible subsets of features, as paths on a graph, bypassing the combinatorial problem analytically. It includes three steps: preprocessing, graph-weighting, and sorting. 2. ReliefF [41] The ReliefF algorithm is a feature weighting algorithm that assigns different weights to features according to the correlation of each feature and category, and features with a weight smaller than a certain threshold will be removed. The correlation of features and categories in the ReliefF algorithm is based on the ability of features to distinguish close samples. When dealing with multiple types of problems, each time a sample R is randomly taken from the training sample set, and k nearest neighbor samples (near Hits) of R are found from the sample sets of the same kind of R, and k nearest neighbor samples (near Misses) are found from each sample set of different kinds of R, and then the weight of each feature is updated 3. Laplacian Score [42] Laplacian Score feature selection method is an unsupervised method. Laplacian score is computed to reflect its locality preserving power. Laplacian score is based on the observation that two data points are probably related to the same topic if they are close to each other. In fact, in many learning problems such as

5

classification, the local structure of the data space is more important than the global structure. In order to model the local geometric structure, we construct a nearest neighbor graph. Laplacian score seeks those features that respect this graph structure. 3.5. Classification method In this work, two supervised learning methods (Naïve Bayes, Bagging) and one unsupervised learning method (K-NN) were applied. They are all with low computational cost and well classification performance. Naïve Bayes method is based on the Bayes’ theorem [43], the K-NN method has the lowest computational cost [44,45], and the Bagging is one of the ensemble learning algorithm which balances the computational cost and accuracy [46]. 1. Naïve Bayes [43] Naive Bayes predicted classification results by considering feature probability. Naive Bayes classifier is a relatively simple probability classifier based on Bayes’ theorem. Naive refers to the assumption that all features in the model have strong independence without taking into account the correlation between features. In order to estimate the parameters of the feature distribution, it is necessary to assume the feature distribution of the training set or to generate a non-parametric model. The hypothesis of feature distribution is called the event model of naive Bayes classifier. 2. K-NN [44,45] The core idea of K-NN algorithm is that if most of the k most adjacent samples of a sample in the feature space belong to a certain category, then the sample also belongs to this category and has the characteristics of the samples in this category. In determining the classification decision, the classification of the samples to be subdivided is determined only according to the category of the nearest one or several samples. K-NN method is only concerned with a very small number of adjacent samples in category decision. As K-NN method mainly relies on the surrounding limited adjacent samples, rather than the method of discriminating class domain to determine the category, K-NN method is more suitable than other methods for the sample set to be divided with a lot of crossover or overlap of class domain. The model is mainly composed of three basic elements: distance measurement used to describe distance, k value selection and classification decision rules. In other words, as long as these three elements in the K-NN model determine, for a fixed training set, the class to which any new input instance belongs is uniquely determined. 3. Bagging [46] Bagging is short for Bootstrap aggregating. In the ensemble learning algorithm, bagging will construct multiple instances of a kind of black-box estimator on the random subset of the original training set, and then combine the prediction results of these estimators to form the final prediction results. This method reduces the variance of the base estimator (for example, decision tree) by introducing randomness into the model construction. In most cases, the bagging approach provides a very simple way to improve a single model without modifying the underlying algorithm. Because bagging can reduce overfitting, it usually works well on strong classifiers and complex models. In this work, three different feature selection methods and three different classification methods are adopted. Therefore, there are altogether nine cases in the verification process of the method proposed in this work, the details are as follows. (1) Case-A: ILFS method was used for features selection, Naive Bayes method was applied for hand movement classification; (2) Case-B: ILFS method was used for features selection, K-NN Naive Bayes method was applied for hand movement classification;

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.

6

F. Xiao, D. Yang, Z. Lv et al. / Future Generation Computer Systems xxx (xxxx) xxx

Fig. 5. The example of the decomposition of sEMG using VMD method. (a) the decomposed sEMG signals with VMD method, (b) the comparison between the raw sEMG signal and the reconstructed sEMG signal with VMD method.

Fig. 6. The example of the decomposition of sEMG using VMD and EMD methods. (a) the frequency spectrum of decomposed signal of sEMG using VMD method, (b) the frequency spectrum of decomposed signals of sEMG using EMD method.

(3) Case-C: ILFS method was used for features selection, Bagging method was applied for hand movement classification; (4) Case-D: ReliefF method was used for features selection, Naive Bayes method was applied for hand movement classification; (5) Case-E: ReliefF method was used for features selection, K-NN method was applied for hand movement classification; (6) Case-F: ReliefF method was used for features selection, Bagging method was applied for hand movement classification; (7) Case-G: Laplacian Score method was used for features selection, Naive Bayes method was applied for hand movement classification; (8) Case-H: Laplacian Score method was used for features selection, K-NN method was applied for hand movement classification; (9) Case-I: Laplacian Score method was used for features selection, Bagging method was applied for hand movement classification.

3.6. Evaluation index In this work, %Accuracy is used to evaluate the performance of hand movement recognition. As shown in the following formula, Nc represents the number of correctly recognized actions, and Na represents the total number of actions to be recognized. %Accuracy =

Nc Na

(12)

4. Results 4.1. The decomposition performance of VMD method The sample figure of VMD decomposition is shown in Fig. 5. Fig. 5(a) shows that the original sEMG signal is decomposed into twelve sub-signals VMFs. The spectrum diagram corresponding to each sub-signal is shown in Fig. 6(a). As can be seen from the figure, each VMF spectrum has a primary frequency and a narrow band. Fig. 5(b) shows the reconstructed signals obtained by decomposition and the comparison with the original

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.

F. Xiao, D. Yang, Z. Lv et al. / Future Generation Computer Systems xxx (xxxx) xxx

7

Table 4 The average %Accuracy of classification of FS. Subjects Case A Case B Case C Case D Case E Case F Case G Case H Case I S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

Fig. 7. The %Accuracy comparison for various entropy features. Table 2 The average %Accuracy of classification for M = 16.

100.00 97.50 100.00 100.00 100.00 96.97 98.84 98.61 96.15 95.24

97.47 96.25 95.38 95.12 94.38 95.96 95.35 98.61 96.15 93.65

100.00 98.75 100.00 100.00 100.00 100.00 100.00 98.61 100.00 98.41

100.00 95.00 100.00 100.00 98.88 97.98 98.84 98.61 97.44 95.24

98.73 95.00 98.46 100.00 96.63 97.98 95.35 98.61 98.72 95.24

100.00 98.75 100.00 100.00 98.88 97.98 97.67 97.22 97.44 98.41

98.73 93.75 100.00 100.00 98.88 96.97 97.67 97.22 96.15 93.65

96.20 93.75 93.85 96.34 94.38 94.95 90.70 94.44 97.44 92.06

97.47 98.75 100.00 98.78 98.88 98.99 97.67 100.00 98.72 98.41

Table 5 The average %Accuracy of classification of HC.

Subject Case A Case B Case C Case D Case E Case F Case G Case H Case I

Subjects Case A Case B Case C Case D Case E Case F Case G Case H Case I

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 avg std

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

92.29 92.06 92.52 92.05 91.13 90.57 92.05 94.78 91.16 92.06 92.07 1.13

87.98 89.12 89.34 87.06 86.32 87.25 87.25 90.02 88.44 89.34 88.21 1.23

94.33 92.52 94.56 94.09 92.42 94.45 93.16 95.46 94.33 94.10 93.94 0.96

91.84 90.70 92.52 91.87 90.02 90.76 91.50 94.78 91.61 91.84 91.74 1.29

86.85 90.25 89.57 88.72 86.69 85.21 88.72 90.70 88.44 88.44 88.36 1.69

94.56 92.29 94.78 93.53 92.05 94.27 94.09 95.01 95.01 94.78 94.04 1.09

90.93 90.70 92.74 91.68 90.20 91.31 91.50 93.65 92.52 92.06 91.73 1.04

87.30 88.66 89.34 87.25 86.14 87.99 85.95 89.80 87.30 88.66 87.84 1.28

93.20 92.97 94.56 93.90 92.98 95.01 93.16 96.60 95.69 94.78 94.28 1.26

Subjects Case A Case B Case C Case D Case E Case F Case G Case H Case I 88.31 86.21 88.75 89.36 87.88 83.52 90.82 93.62 88.16 89.61

85.71 87.36 86.25 84.04 81.82 82.42 84.69 80.85 86.84 83.12

89.61 88.51 90.00 87.23 88.89 90.11 90.82 93.62 92.11 93.51

85.71 85.06 90.00 87.23 85.86 84.62 88.78 92.55 88.16 87.01

83.12 87.36 86.25 82.98 83.84 79.12 85.71 86.17 86.84 83.12

85.48 86.96 88.06 83.33 84.71 82.72 83.33 85.07 77.05 86.89

91.94 88.41 88.06 96.15 91.76 95.06 91.67 94.03 88.52 90.16

90.32 91.30 88.06 96.15 90.59 91.36 92.86 92.54 85.25 90.16

83.87 89.86 88.06 85.90 84.71 77.78 86.90 82.09 80.33 86.89

95.16 89.86 91.04 96.15 91.76 96.30 94.05 94.03 93.44 90.16

91.94 89.86 89.55 94.87 88.24 91.36 94.05 92.54 90.16 93.44

85.48 88.41 86.57 87.18 82.35 86.42 80.95 89.55 78.69 86.89

91.94 91.30 88.06 96.15 92.94 96.30 92.86 94.03 91.80 90.16

4.3. The comparison results of Case A∼Case I

Table 3 The average %Accuracy of classification of FP. S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

90.32 92.75 86.57 93.59 88.24 90.12 91.67 91.04 83.61 91.80

90.91 87.36 91.25 86.17 88.89 92.31 91.84 94.68 92.11 93.51

80.52 83.91 86.25 81.91 82.83 81.32 85.71 87.23 89.47 85.71

84.42 87.36 90.00 86.17 80.81 83.52 87.76 81.91 85.53 83.12

88.31 87.36 91.25 87.23 89.90 91.21 92.86 94.68 93.42 93.51

signals. It can be seen from Fig. 5(b) that the reconstructed signals are basically the same as the original signals. This indicates that VMD decomposition can guarantee signal integrity. Fig. 6(b) shows the spectrum diagram of each sub-signal decomposed by EMD decomposition method. It can be seen from the figure that, compared with VMD method, the sub-signal decomposed by EMD method has a wide frequency band, such as IMF1∼IMF4. In contrast, VMF1∼VMF12 has a narrow frequency band, which is conducive to subsequent sub-signal feature extraction.

4.2. The comparison between CPEI, PE, SE, and FE

Fig. 7 compares the gesture classification accuracy corresponding to four entropy features (CPEI, PE, sampling entropy and fuzzy entropy) with Case I (Laplacian Score and Bagging method). At present, CPEI is mainly applied to EEG signal feature extraction [34], which is introduced into sEMG signal in this work and applied to hand motion recognition. As mentioned above, CPEI features have more information differentiation, and the complexity of sEMG sub-signals can be better represented by the entropy characteristics of arrangement corresponding to different indicators. According to Fig. 7, its corresponding hand movement recognition accuracy is also the highest.

Since the number of sub-signals of sEMG decomposed by VMD method can be set artificially, and the obtained sub-signals all have narrow frequency band with a single main frequency, great flexibility is provided. Table 2 and Tables A.1–A.7 in the appendix part show the hand movement recognition accuracy corresponding to different number of sub-signals M. According to the comparison results, when the number of sub-signals M is 16, the corresponding Case C, Case I and Case F have the highest accuracy. When M is larger (for example, M is greater than 22), it is possible to further improve the accuracy. However, the larger the M value is, the greater the corresponding computation time is. Moreover, according to Tables A.5–A.7, the increase of M value is relatively limited to the improvement of accuracy. Therefore, the optimal M value is 16. Table 2 shows the accuracy when M is 16. According to the results, Case I (Laplacian Score for feature selection and Bagging for hand movement classification) has the highest accuracy. Fig. 8 shows an example diagram of the corresponding spectrum when M value is 16. The main frequency of the sub-signal is distributed between 25 Hz and 250.3 Hz, and the frequency band of each sub-signal is relatively narrow, and the corresponding main frequency is also different. Tables 3–8 shows the identification accuracy of all actions (FP, FS, HC, HO, WE, WF) corresponding to Case A∼Case I methods. Fig. 9 shows the statistic results of the average recognition accuracy and standard deviation of all actions. According to Fig. 9, FS motion recognition accuracy is the highest. The corresponding Case C method reached 99.58%, and the Case I method had the highest comprehensive recognition accuracy with 90.97 ± 2.69 (FP), 98.77 ± 0.82 (FS), 92.55 ± 2.53 (HC), 96.14 ± 2.79 (HO), 95.51 ± 2.29 (WE), 90.66 ± 3.06 (WF). Fig. 10 shows the confusion matrix diagram corresponding to different hand movements under all cases. In the figure, the main diagonal represents the %Accuracy when the actual action and the predicted action is the same. The darker the color in the figure, the higher the corresponding value. The corresponding statistical analysis (ME and SD) of features for S1∼S4 data are shown in Fig. 11. It seems that the differences

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.

8

F. Xiao, D. Yang, Z. Lv et al. / Future Generation Computer Systems xxx (xxxx) xxx

Fig. 8. The frequency spectrum of decomposed signal of sEMG using VMD with M = 16.

Fig. 9. The average %Accuracy for all corresponding hand movements. Table 6 The average %Accuracy of classification of HO.

Table 7 The average %Accuracy of classification of WE.

Subjects Case A Case B Case C Case D Case E Case F Case G Case H Case I

Subjects Case A Case B Case C Case D Case E Case F Case G Case H Case I

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

89.33 87.06 93.18 87.18 87.61 90.60 92.38 93.24 93.75 89.90

93.33 91.76 94.32 92.31 87.61 94.02 92.38 95.95 93.75 93.94

98.67 92.94 98.86 94.87 92.04 97.44 92.38 97.30 96.25 95.96

94.67 87.06 90.91 86.32 84.96 90.60 89.52 91.89 92.50 92.93

93.33 89.41 94.32 94.02 84.96 92.31 93.33 94.59 90.00 92.93

98.67 90.59 97.73 94.02 91.15 96.58 94.29 97.30 98.75 97.98

92.00 89.41 90.91 88.03 87.61 92.31 92.38 93.24 95.00 89.90

93.33 92.94 92.05 92.31 89.38 94.02 91.43 95.95 93.75 92.93

98.67 92.94 98.86 94.02 91.15 97.44 94.29 97.30 98.75 97.98

of mean values of features were great, but the std values are also large. As the classifier can be used to be trained to neglect the outliers, the corresponding accuracy of classification was not affected a lot. Table 9 shows the calculation time comparison between the proposed method and other relatively advanced methods. In this study, the WNN method combined with discrete wavelet transform in [1] is used for the comparison with the proposed method. The corresponding parameters were chosen according to [1].

93.18 97.18 93.98 93.81 94.44 94.57 88.57 97.33 97.67 96.30

87.50 87.32 90.36 86.60 87.78 85.87 85.71 90.67 94.19 90.12

92.05 94.37 98.80 95.88 92.22 93.48 91.43 96.00 96.51 96.30

92.05 94.37 96.39 93.81 93.33 93.48 88.57 97.33 97.67 95.06

85.23 92.96 86.75 85.57 90.00 81.52 85.71 92.00 86.05 87.65

89.77 94.37 98.80 94.85 92.22 93.48 92.38 96.00 97.67 95.06

92.05 94.37 95.18 93.81 95.56 94.57 90.48 97.33 96.51 96.30

85.23 84.51 89.16 81.44 88.89 82.61 82.86 90.67 86.05 90.12

92.05 94.37 98.80 94.85 95.56 94.57 92.38 97.33 97.67 97.53

This method is abbreviated with WNN with WT in Table 9. The tunable-Q wavelet transform (TQWT) based method with Kraskov entropy (KRE) features, ReliefF and K-NN methods introduced in [2] was also used for comparison. The whole signal was used to extract the CPEI feature value in our proposed method. To be fair, there is one TQWT block to extract the sub-band signals of sEMG cross covariance signals in the TQWT based method. Two of these three sEMG signals were chosen to obtain the corresponding cross covariance signal. Therefore, there are three sEMG cross

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.

F. Xiao, D. Yang, Z. Lv et al. / Future Generation Computer Systems xxx (xxxx) xxx

9

Fig. 10. The %Accuracy confusion matrix plots for all cases.

covariance signals to be input signals of TQWT block. The level of decomposition is set to be 16, the tunable Q factor is 2, the redundancy is equal to be 9. This method is abbreviated with TQWT based method in Table IX. The EMD method was also used to classify motions in [16,47]. In this work, the EMD based method listed in Table IX means that the EMD method is used to replace the VMD method in the proposed method, and the other parts are the same. The BP based method listed in Table 9 means that the BP method [48] is used to replace the Bagging method in the proposed method, and the other parts are the same. The predicted hand movements are 600 randomly selected movements, and the total number of each action is 100. The CPU is i5-8250u and

the memory is 8 g. The processing software adopted is Spyder (python 3.7). The result of corresponding method proposed in this paper is 0.56 s. Table 9 shows the average accuracy of each method, and the accuracy obtained by the method proposed in this paper is the highest, 94.28% ± 1.26%.

5. Discussion and conclusion This paper presents a hand motion recognition method based on VMD and CPEI, and combines different feature selection methods and classifiers for comparative analysis. According to Table 2,

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.

10

F. Xiao, D. Yang, Z. Lv et al. / Future Generation Computer Systems xxx (xxxx) xxx

Fig. 11. The corresponding statistical analysis (ME and SD) of features for S1∼S10 data with proposed method from (a) to (j).

Case I (Laplacian Score method is adopted for feature selection, and Bagging method is used for hand motion classification) combined with VMD and CPEI method acquires the average accuracy with 94.28%. In addition, when it predicted 600 random

hand movements, the corresponding execution time was 0.56 s. Compared with other methods listed in Table 9, the method proposed in this paper has the highest accuracy while ensuring real-time performance. The time-frequency features of the signals

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.

F. Xiao, D. Yang, Z. Lv et al. / Future Generation Computer Systems xxx (xxxx) xxx Table 8 The average %Accuracy of classification of WF. Subjects Case A Case B Case C Case D Case E Case F Case G Case H Case I S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

91.67 93.88 93.10 90.41 89.23 85.25 90.48 94.92 83.33 90.00

75.00 81.63 79.31 78.08 80.00 75.41 79.37 91.53 76.67 86.67

93.33 91.84 89.66 90.41 89.23 86.89 93.65 93.22 90.00 88.33

86.67 93.88 89.66 90.41 87.69 83.61 92.06 96.61 85.00 90.00

73.33 85.71 82.76 82.19 78.46 75.41 84.13 91.53 86.67 83.33

93.33 93.88 87.93 90.41 89.23 85.25 95.24 89.83 88.33 91.67

90.00 95.92 96.55 94.52 89.23 90.16 88.89 96.61 85.00 95.00

76.67 81.63 82.76 78.08 78.46 81.97 79.37 88.14 78.33 85.00

90.00 93.88 87.93 93.15 89.23 88.52 87.30 96.61 91.67 88.33

obtained by VMD decomposition are clearly distinguished from each other [20], and each sub-signal has a main frequency and a narrow frequency band, which makes the features of the subsignals obtained by subsequent extraction have strong differences and can make better use of each sub-signal to enrich the feature set, so as to better distinguish different actions. Fig. 9 shows the average %Accuracy of different hand movements corresponding to different methods (Case A∼Case I). For six different actions (FP, FS, HC, HO, WE, WF), the action recognition accuracy of FS is the highest, while that of FP is the lowest. This is consistent with the experimental results in [1]. Fig. 10 shows the confusion matrixes corresponding to each method between the actual action and the predicted action. It can be seen from the figure that the recognition accuracy of K-NN method (corresponding to Case B, Case E, and Case H) is low, and it is easy to misidentify FP and WF actions. The recognition effect of Naive Bayes method and Bagging method is far better than that of K-NN method. According to Fig. 10(a), it can be seen that it is actually WF. The accuracy of WF prediction is 90.23%, and 7.33% of them are misjudged as FP. The reason is that the two actions of FP and WF are relatively similar, and there is a certain probability of confusion. Moreover, this is also consistent with the results of Ref. [1]. Moreover, our method overcomes the confusion between HC and FP compared with Ref. [1]. According to Table 2, the recognition methods corresponding to Bagging (Case C, Case F, and Case I) have the highest accuracy. The recognition methods corresponding to Naive Bayes method (Case A, Case D, and Case G) has the second highest accuracy, while the recognition methods corresponding to K-NN method (Case B, Case E, and Case H) has the lowest accuracy. The comparison results of the corresponding computation time are given as follow: K-NN > Naive Bayes > Bagging. The comparison results of the required execution time corresponding to the three feature selection methods are given as follow: ILFS < Laplacian Score < ReliefF. Here, take three identification methods corresponding to Bagging method (Case C, Case F, and Case I) as examples. Case C adopted ILFS as the feature selection method, with the shortest execution time and the lowest corresponding identification accuracy compared with Case F and Case I. Case I adopted Laplacian Score method as the feature selection method, with the execution time greater than Case C and smaller than Case F, with the highest corresponding identification accuracy. Case F’s identification accuracy was lower than Case I, with the longest execution time. Therefore, the Laplacian Score method is better than the ReliefF method for feature selection. In addition, according to Tables A.1–A.7, ILFS as the feature selection method

11

with the shortest execution time, the corresponding hand gesture recognition accuracy is close with the Laplacian Score method, even better than it sometimes (Case C and Case F in Table A.7, Case C and Case F in Table A.4). Compared with [2], the proposed approach listed in the Table 9 is better in the accuracy and the real-time performance. The combination of tunable-Q wavelet transform, ReliefF and K-NN methods is adopted in [2], although the decomposed features are sufficient and reasonable, the recognition accuracy of K-NN method is low. The time required by the ReliefF method for feature selection is much longer than the Laplacian Score method adopted in this paper, so the execution time is longer than the method proposed in this paper. As a recognition method, K-NN method has a relatively low recognition accuracy. Moreover, according to Figs. 8 and 9, the maximum misjudgment rate of the corresponding confusion matrix can reach 19.63%, as shown in Fig. 10(b). In this work, VMD method is used for signal decomposition. One advantage of VMD method is that it can independently choose the number of signal decomposition, namely the M value, but it also leads to a problem of determining the optimal M value [20]. According to Tables A.1–A.7 and Table 2, the optimal parameter M is 16 for this work. Too few patterns can lead to insufficient data segmentation. Some components are contained in other patterns, and too many patterns either capture extra noise or cause the pattern to copy. At present, for the selection of the optimal M value, one method is to use EMD to predecompose the signal and obtain the number of IMFs obtained by EMD decomposition to be the M value for VMD [27,49]. For this work, the number of IMFs obtained by EMD decomposition is 12. Fig. 6 shows the spectrum diagram of decomposition signals corresponding to VMD and EMD methods. However, this method is not applicable to human hand movement recognition. According to Table A.3, when M value is 12, the recognition accuracy corresponding to each method (Case A∼Case I) is low, with the highest being 90.75%. Therefore, according to the experimental results, the optimal M value is 16. A corresponding spectrum example is shown in Fig. 8. Each VMF has a different main frequency and a narrow band. CPEI is adopted as feature signal in this paper. According to the [32–35], CPEI can enhance the overall identification robustness compared with PE. The experimental results shown in Fig. 7 also verify its robustness. PE characteristics corresponding to different parameters can be used to enhance the effect by using compound ideas. Comparatively speaking, CPEI features have more information differentiation, and the complexity of sEMG sub-signals can be better represented by the entropy characteristics of arrangement corresponding to different indicators. The article [35] also discusses the advantages of composite entropy. In this work, we refer to [1,13] to measure three muscles with three sEMG electrodes to realize the recognition of six movements. Compared with many existing literatures [3,5,7,11,17], the advantage lies in that as few electrodes as possible are used to realize as much motion recognition as possible and ensure the recognition accuracy. Many researches focus on how to improve the accuracy as much as possible, but neglect how to limit the number of sEMG sensors [3,5,7,11,17]. Increasing the number of sensors can improve accuracy. However, it also tends to affect real-time performance. The limitation of this work is that we have

Table 9 The comparison of average %Accuracy of classification and execution time. Index parameters

WNN with WT

TQWT based method

EMD based method

BP based method

Proposed method(VMD based method)

Execution time (second) %Accuracy

0.79 93.22 ± 1.92

0.64 91.73 ± 1.77

0.48 91.81 ± 1.27

0.31 85.50 ± 2.71

0.56 94.28±1.26

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.

12

F. Xiao, D. Yang, Z. Lv et al. / Future Generation Computer Systems xxx (xxxx) xxx

Table A.1 The average %Accuracy of classification for K = 8.

Table A.3 The average %Accuracy of classification for K = 12.

Subject Case A Case B Case C Case D Case E Case F Case G Case H Case I

Subject Case A Case B Case C Case D Case E Case F Case G Case H Case I

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 avg std

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 avg std

79.15 79.18 79.29 79.03 79.58 79.24 79.27 79.21 78.68 78.86 79.15 0.25

75.04 75.24 74.38 75.01 77.35 74.95 76.07 75.65 75.92 73.07 75.27 1.13

84.92 85.12 84.55 83.32 86.15 85.89 85.03 85.09 85.15 84.98 85.02 0.76

79.15 79.18 79.29 79.03 79.58 79.24 79.27 79.21 78.68 78.86 79.15 0.25

75.04 75.24 74.38 75.01 77.35 74.95 76.07 75.65 75.92 73.07 75.27 1.13

84.78 84.55 84.35 83.26 86.18 85.92 85.06 84.77 85.62 85.42 84.99 0.85

79.15 79.18 79.29 79.03 79.58 79.24 79.27 79.21 78.68 78.86 79.15 0.25

75.04 75.24 74.38 75.01 77.35 74.95 76.07 75.65 75.92 73.07 75.27 1.13

84.78 84.98 84.98 83.46 86.00 85.92 85.18 85.09 85.42 85.50 85.13 0.71

Table A.2 The average %Accuracy of classification for K = 10. Subject Case A Case B Case C Case D Case E Case F Case G Case H Case I S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 avg std

83.78 83.46 86.12 86.74 83.93 80.97 85.02 82.37 77.85 84.09 83.43 2.58

78.32 77.54 80.19 81.59 78.94 79.72 80.03 75.35 83.62 78.16 79.34 2.27

86.58 87.36 89.70 90.02 89.08 86.43 90.17 87.36 88.92 88.46 88.41 1.40

84.24 84.24 86.12 84.40 82.84 82.22 85.65 83.46 85.34 86.27 84.48 1.37

77.07 79.72 79.10 80.50 79.88 77.54 78.16 76.60 77.85 79.88 78.63 1.36

86.27 86.58 89.08 90.95 88.14 85.96 89.86 87.83 89.55 88.46 88.27 1.65

81.90 83.78 86.27 84.87 82.68 82.06 85.34 81.75 83.31 85.49 83.74 1.66

75.20 76.60 77.38 77.85 77.69 76.91 76.13 74.42 77.54 76.29 76.60 1.12

86.27 86.12 89.55 89.55 88.61 86.58 89.24 87.36 88.77 88.30 88.03 1.35

not applied this method to the amputee with artificial limb. In the future, it will be applied to the real motion assistance of patients. All in all, a new hand movement recognition method is proposed in this work. sEMG signal is decomposed by VMD, CPEI features are extracted from each decomposed VMFs signal, features are sorted and selected by Laplacian Score, and hand movements are classified by Bagging classification method. Compared with the existing methods, the experimental results show that the proposed method has the highest accuracy and can ensure the real-time performance. The proposed model could be used to improve the quality of life of amputees, people with impaired hand function and others. In the future, this work will be applied to actual patient prosthesis control [1], wearable hand exoskeleton control [2], virtual reality human–machine interaction [3] and other fields.

86.43 87.68 84.40 85.18 85.34 87.21 86.43 87.83 87.21 85.65 86.33 1.16

79.72 79.10 78.94 79.10 80.66 81.12 79.88 79.41 77.69 78.16 79.38 1.04

91.26 92.67 88.30 89.70 91.11 91.58 90.17 91.26 90.02 91.42 90.75 1.22

87.36 87.68 83.93 87.36 85.49 87.68 85.65 88.77 86.12 87.52 86.76 1.42

82.84 83.62 79.56 84.24 83.15 82.68 79.10 81.28 80.34 78.94 81.58 1.98

89.39 92.20 87.52 90.17 89.39 89.08 89.55 89.70 90.02 90.48 89.75 1.18

87.52 87.68 84.56 86.27 86.43 86.43 85.02 89.24 86.27 86.27 86.57 1.33

78.47 78.16 78.78 81.12 79.72 77.38 77.85 80.34 77.54 76.76 78.61 1.39

90.95 92.98 88.14 89.55 90.48 90.64 90.33 91.26 90.95 90.02 90.53 1.24

Table A.4 The average %Accuracy of classification for K = 14. Subject Case A Case B Case C Case D Case E Case F Case G Case H Case I S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 avg std

87.99 88.61 90.33 89.08 88.30 89.55 88.30 89.86 86.12 88.30 88.64 1.18

85.49 84.87 87.68 85.65 86.27 85.80 86.12 83.93 85.65 85.80 85.73 0.96

89.39 89.39 91.73 91.26 89.39 91.89 90.64 91.89 88.61 90.64 90.48 1.21

85.49 87.36 88.46 88.14 87.05 86.74 85.80 87.05 86.90 87.36 87.04 0.91

86.43 82.22 85.65 87.21 85.65 85.34 84.71 85.18 84.71 85.65 85.27 1.31

89.86 87.52 90.64 89.39 89.39 90.33 89.08 91.11 89.24 90.33 89.69 1.01

87.99 87.21 88.61 87.52 86.43 88.46 87.68 88.14 86.12 87.68 87.58 0.81

83.78 84.24 85.96 86.12 84.71 83.62 84.40 83.78 82.06 85.34 84.40 1.21

89.24 89.86 90.80 90.02 89.24 90.80 91.11 90.95 88.77 90.64 90.14 0.84

Table A.5 The average %Accuracy of classification for K = 18. Subject Case A Case B Case C Case D Case E Case F Case G Case H Case I S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 avg std

92.20 90.17 89.24 90.17 89.39 89.86 89.86 89.70 89.70 88.30 89.86 0.99

87.05 87.68 86.90 87.21 86.90 89.55 85.65 86.27 86.27 86.12 86.96 1.09

92.82 91.58 91.42 91.58 91.73 92.36 91.42 91.58 91.89 90.64 91.70 0.58

90.48 90.48 90.33 91.26 89.86 90.95 89.24 89.55 88.92 88.61 89.97 0.88

86.43 87.21 87.05 84.71 85.18 87.52 85.65 86.12 85.96 86.90 86.27 0.92

92.98 91.58 91.89 92.98 90.80 93.76 92.04 92.20 91.11 91.11 92.04 0.96

91.58 90.02 90.02 92.04 90.48 90.95 90.17 89.39 90.02 89.24 90.39 0.90

87.05 86.43 88.77 87.36 87.83 90.64 86.27 87.52 86.74 87.21 87.58 1.29

93.60 92.20 92.51 92.98 91.73 93.60 92.20 91.73 92.36 91.58 92.45 0.74

Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgments The authors would like to acknowledge all the subjects for participating in this study. The study was funded and supported by National Key R&D Program of China (Grant No. 2018YFB1305400), National Natural Science Foundation (Grant No. U1713210) and Anhui Provincial Natural Science Foundation (1908085QF261).

Table A.6 The average %Accuracy of classification for K = 20. Subject Case A Case B Case C Case D Case E Case F Case G Case H Case I S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 avg std

90.17 91.26 91.58 89.86 90.95 90.17 92.20 89.39 90.02 91.26 90.69 0.89

88.61 86.12 86.58 85.02 84.56 87.36 85.65 85.02 85.18 87.21 86.13 1.30

93.14 91.42 92.67 90.95 91.89 91.42 92.98 92.36 91.73 93.14 92.17 0.79

89.70 91.73 90.64 90.02 90.80 90.02 90.80 90.64 89.08 91.11 90.45 0.76

86.43 87.83 87.05 84.87 87.36 86.12 86.74 86.27 85.96 87.99 86.66 0.94

93.45 92.98 93.60 93.29 93.60 91.89 93.45 92.51 91.58 92.67 92.90 0.73

89.39 90.80 90.17 89.70 91.58 89.86 90.80 89.08 89.70 90.33 90.14 0.75

86.90 85.65 87.05 85.49 86.74 84.40 85.34 83.78 87.52 87.36 86.02 1.29

92.20 92.82 92.51 91.89 92.36 91.89 92.67 92.20 92.04 92.51 92.31 0.32

Appendix See Tables A.1–A.7

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.

F. Xiao, D. Yang, Z. Lv et al. / Future Generation Computer Systems xxx (xxxx) xxx Table A.7 The average %Accuracy of classification for K = 22. Subject Case A Case B Case C Case D Case E Case F Case G Case H Case I S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 avg std

89.24 92.67 93.45 91.89 91.11 90.33 91.89 91.11 90.95 92.82 91.54 1.26

88.77 88.46 89.70 89.55 87.99 89.39 88.14 87.36 88.92 89.08 88.74 0.75

91.89 92.98 94.54 92.98 93.76 92.67 93.60 93.60 91.26 93.76 93.10 0.97

89.70 92.51 92.67 90.02 90.48 90.64 90.64 90.48 90.48 91.89 90.95 1.03

88.14 90.17 91.11 88.30 88.92 89.08 88.61 87.21 90.33 89.70 89.16 1.17

92.36 92.82 93.60 92.51 92.67 92.98 92.82 94.38 92.51 93.92 93.06 0.68

89.86 92.98 90.80 88.92 91.11 90.48 91.42 91.26 90.02 91.42 90.83 1.10

88.30 89.39 92.82 88.92 88.46 89.55 88.30 88.77 88.30 87.68 89.05 1.44

91.89 93.29 95.01 91.89 92.82 92.20 93.14 94.38 92.36 93.45 93.04 1.04

References [1] L.W. Duan Feng, Dai Lili, Chang Wennan, Chen Zengqiang, Zhu Chi, sEMG-based identification of hand motion commands using wavelet neural network combined with discrete wavelet transform, IEEE Trans. Ind. Electron. 63 (2016) 1923–1934, http://dx.doi.org/10.1109/tie.2015. 2497212. [2] A. Nishad, A. Upadhyay, R.B. Pachori, U.R. Acharya, Automated classification of hand movements using tunable-Q wavelet transform based filter-bank with surface electromyogram signals, Future Gener. Comput. Syst. 93 (2019) 96–110, http://dx.doi.org/10.1016/j.future.2018.10.005. [3] I. Mendez, B.W. Hansen, C.M. Grabow, E.J.L. Smedegaard, N.B. Skogberg, X.J. Uth, A. Bruhn, B. Geng, E.N. Kamavuako, Evaluation of the Myo armband for the classification of hand motions, in: IEEE Int. Conf. Rehabil. Robot, 2017, pp. 1211–1214, http://dx.doi.org/10.1109/ICORR.2017.8009414. [4] D. Farina, N. Jiang, H. Rehbaum, A. Holobar, B. Graimann, H. Dietl, O. Aszmann, The extraction of neural information from the surface EMG for the control of upper-limb prostheses: Emerging avenues and challenges, IEEE Trans. Neural Syst. Rehabil. Eng. 22 (2014) 797–809, http://dx.doi.org/ 10.1109/TNSRE.2014.2305111. [5] Ganesh R. Naik, Ali H. Al-Timemy, Hung T. Nguyen, Transradial amputee gesture classification using an optimal number of sEMG sensors: An approach using ICA clustering, IEEE Trans. Neural Syst. Rehabil. Eng. 24 (2016) 837–846, http://dx.doi.org/10.1109/TNSRE.2015.2478138. [6] Y. Guo, G.R. Naik, S. Huang, A. Abraham, H.T. Nguyen, Nonlinear multiscale Maximal Lyapunov Exponent for accurate myoelectric signal classification, Appl. Soft Comput. J. 36 (2015) 633–640, http://dx.doi.org/10.1016/j.asoc. 2015.07.032. [7] K. Xing, P. Yang, J. Huang, Y. Wang, Q. Zhu, A real-time EMG pattern recognition method for virtual myoelectric hand control, Neurocomputing 136 (2014) 345–355, http://dx.doi.org/10.1016/j.neucom.2013.12.010. [8] F. Xiao, Y. Wang, L. He, H. Wang, W. Li, Z. Liu, Motion estimation from surface electromyogram using adaboost regression and average feature values, IEEE Access 7 (2019) 13121–13134, http://dx.doi.org/10.1109/access.2019. 2892780. [9] F. Xiao, Proportional myoelectric and compensating control of a cableconduit mechanism-driven upper limb exoskeleton, ISA Trans. 89 (2019) 245–255, http://dx.doi.org/10.1016/j.isatra.2018.12.028. [10] C. Disselhorst-Klug, T. Schmitz-Rode, G. Rau, Surface electromyography and muscle force: Limits in sEMG-force relationship and new approaches for applications, Clin. Biomech. 24 (2009) 225–235, http://dx.doi.org/10.1016/ j.clinbiomech.2008.08.003. [11] A. Phinyomark, P. Phukpattaranont, C. Limsakul, Feature reduction and selection for EMG signal classification, Expert Syst. Appl. 39 (2012) 7420–7431, http://dx.doi.org/10.1016/j.eswa.2012.01.102. [12] J. Ryu, B.H. Lee, D.H. Kim, sEMG signal-based lower limb human motion detection using a top and slope feature extraction algorithm, IEEE Signal Process. Lett. 24 (2017) 929–932, http://dx.doi.org/10.1109/LSP.2016. 2636320. [13] F. Duan, L. Dai, Recognizing the gradual changes in sEMG characteristics based on incremental learning of wavelet neural network ensemble, IEEE Trans. Ind. Electron. 64 (2017) 4276–4286, http://dx.doi.org/10.1109/TIE. 2016.2593693. [14] J. Rafiee, M.A. Rafiee, F. Yavari, M.P. Schoen, Feature extraction of forearm EMG signals for prosthetics, Expert Syst. Appl. 38 (2011) 4058–4067, http://dx.doi.org/10.1016/j.eswa.2010.09.068. [15] K. Englehart, B. Hudgin, P.A. Parker, A wavelet-based continuous classification scheme for multifunction myoelectric control, IEEE Trans. Biomed. Eng. 48 (2001) 302–311, http://dx.doi.org/10.1109/10.914793. [16] X. Zhang, P. Zhou, Myoelectric pattern identification of stroke survivors using multivariate empirical mode decomposition, J. Healthc. Eng. 5 (2014) 261–274, http://dx.doi.org/10.1260/2040-2295.5.3.261.

13

[17] C. Sapsanis, G. Georgoulas, A. Tzes, EMG based classification of basic hand movements based on time frequency features, in: 2013 21st Mediterr. Conf. Control Autom. MED 2013 - Conf. Proc., 2013, pp. 716–722, http: //dx.doi.org/10.1109/MED.2013.6608802. [18] C. Zhu, Z. Chen, W. Li, F. Duan, L. Dai, W. Chang, sEMG-based identification of hand motion commands using wavelet neural network combined with discrete wavelet transform, IEEE Trans. Ind. Electron. 63 (2015) 1923–1934, http://dx.doi.org/10.1109/tie.2015.2497212. [19] A. Subasi, M. Yilmaz, H.R. Ozcalik, Classification of EMG signals using wavelet neural network, J. Neurosci. Methods 156 (2006) 360–367, http: //dx.doi.org/10.1016/j.jneumeth.2006.03.004. [20] K. Dragomiretskiy, D. Zosso, Variational mode decomposition, IEEE Trans. Signal Process. 62 (2014) 531–544, http://dx.doi.org/10.1007/978-3-31914612-6_15. [21] Y.H. Wang, C.H. Yeh, H.W.V. Young, K. Hu, M.T. Lo, On the computational complexity of the empirical mode decomposition algorithm, Phys. Stat. Mech. Appl. 400 (2014) 159–167, http://dx.doi.org/10.1016/j.physa.2014. 01.020. [22] M. Zhou, H. Zhao, H. Xia, J. Zhang, Z. Liu, C. Liu, F. Gao, De-noising of photoacoustic sensing and imaging based on combined EMD and ICA, J. Biophotonics (2019) e201900042, http://dx.doi.org/10.1002/jbio. 201900042. [23] X. Zhang, P. Zhou, Filtering of surface EMG using ensemble empirical mode decomposition, Med. Eng. Phys. 35 (2013) 537–542, http://dx.doi.org/10. 1016/j.medengphy.2012.10.009. [24] G. Li, G. Tang, G. Luo, H. Wang, Underdetermined blind separation of bearing faults in hyperplane space with variational mode decomposition, Mech. Syst. Signal Process. 120 (2019) 83–97, http://dx.doi.org/10.1016/j. ymssp.2018.10.016. [25] K. Hong, L. Wang, S. Xu, A variational mode decomposition approach for degradation assessment of power transformer windings, IEEE Trans. Instrum. Meas. 68 (2018) 1221–1229, http://dx.doi.org/10.1109/TIM.2018. 2865048. [26] S. Nagineni, S. Taran, V. Bajaj, Features based on variational mode decomposition for identification of neuromuscular disorder using EMG signals, Heal. Inf. Sci. Syst. 6 (2018) 1–10, http://dx.doi.org/10.1007/s13755-0180050-4. [27] P. Singh, G. Pradhan, Variational mode decomposition based ECG denoising using non-local means and wavelet domain filtering, Australas. Phys. Eng. Sci. Med. 41 (2018) 891–904, http://dx.doi.org/10.1007/s13246-018-06850. [28] Y. Kopsinis, S. Mclaughlin, S. Member, Development of EMD-based denoising methods inspired by wavelet thresholding, IEEE Trans. Signal Process. 57 (2009) 1351–1362. [29] X. Zhang, D. Wang, Z. Yu, X. Chen, S. Li, P. Zhou, EMG-torque relation in chronic stroke: A novel EMG complexity representation with a linear electrode array, IEEE J. Biomed. Heal. Inform. 21 (2017) 1562–1572, http: //dx.doi.org/10.1109/JBHI.2016.2626399. [30] Y. Wu, R. Song, Effects of task demands on kinematics and EMG signals during tracking tasks using multiscale entropy, Entropy 19 (2017) http: //dx.doi.org/10.3390/e19070307. [31] J. Hu, P. Wang, Noise robustness analysis of performance for EEG-based driver fatigue detection using different entropy feature sets, Entropy 19 (2017) http://dx.doi.org/10.3390/e19080385. [32] O. Dostál, O. Vysata, L. Pazdera, A. Procházka, J. Kopal, J. Kuchyňka, M. Vališ, Permutation entropy and signal energy increase the accuracy of neuropathic change detection in needle EMG, Comput. Intell. Neurosci. 2018 (2018) 1–5, http://dx.doi.org/10.1155/2018/5276161. [33] Y. Cao, W. wen Tung, J.B. Gao, V.A. Protopopescu, L.M. Hively, Detecting dynamical changes in time series using the permutation entropy, Phys. Rev. E 70 (2004) 7, http://dx.doi.org/10.1103/PhysRevE.70.046217. [34] E. Olofsen, J.W. Sleigh, A. Dahan, Permutation entropy of the electroencephalogram: A measure of anaesthetic drug effect, Br. J. Anaesth. 101 (2008) 810–821, http://dx.doi.org/10.1093/bja/aen290. [35] J. Zheng, Z. Dong, H. Pan, Q. Ni, T. Liu, J. Zhang, Composite multiscale weighted permutation entropy and extreme learning machine based intelligent fault diagnosis for rolling bearing, Meas. J. Int. Meas. Confed. 143 (2019) 69–80, http://dx.doi.org/10.1016/j.measurement.2019.05.002. [36] Lynn S. Lippert, MS Clinical and Anatomy, fourth ed., F.A. Davis Company, Philadelphia, 2006. [37] C.J. De Luca, The use of surface electromyography, J. Appl. Biomech. 13 (1997) 1–38. [38] R.T. Rockafellar, A dual approach to solving nonlinear programming problems by unconstrained optimization, Math. Program. 5 (1973) 354–373, http://dx.doi.org/10.1007/BF01580138. [39] M.R. Hestenes, Multiplier and gradient methods, J. Optim. Theory Appl. 4 (1969) 303–320. [40] G. Roffo, S. Melzi, U. Castellani, A. Vinciarelli, Infinite latent feature selection: A probabilistic latent graph-based ranking approach, in: Proc. IEEE Int. Conf. Comput. Vis. 2017-Octob, 2017, pp. 1407–1415, http://dx. doi.org/10.1109/ICCV.2017.156.

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.

14

F. Xiao, D. Yang, Z. Lv et al. / Future Generation Computer Systems xxx (xxxx) xxx

[41] Marko Robnik-Šikonja, Igor Kononenko, Theoretical and empirical analysis of relieff and rrelieff, Mach. Lear. J. 53 (2003) (2003) 23–69. [42] X. He, D. Cai, P. Niyogi, Laplacian score for feature selection, in: Proc. Adv. Neural Inf. Process. Syst, 2005, pp. 507–514, http://books.nips.cc/papers/ files/nips18/NIPS2005_0149.pdf. [43] Y. Shen, Y. Li, H. Zheng, B. Tang, M. Yang, Enhancing ontology-driven diagnostic reasoning with a symptom-dependency- aware Naïve Bayes classifier, BMC Bioinform. 20 (2019) 1–14. [44] D.W. Aha, D. Kibler, M.K. Albert, Instance-based learning algorithms, Mach. Learn. 6 (1991) 37–66, http://dx.doi.org/10.1023/A:1022689900470. [45] T. Denœux, A k-nearest neighbor classification rule based on dempstershafer theory, IEEE Trans. Syst. Man Cybern. 25 (1995) 804–813, http: //dx.doi.org/10.1007/978-3-540-44792-4_29. [46] C. Croux, K. Joossens, A. Lemmens, Trimmed bagging, Comput. Statist. Data Anal. 52 (2007) 362–368, http://dx.doi.org/10.1016/j.csda.2007.06.012. [47] V.K. Mishra, V. Bajaj, A. Kumar, G.K. Singh, Analysis of ALS and normal EMG signals based on empirical mode decomposition, IET Sci. Meas. Technol. 10 (2016) 963–971, http://dx.doi.org/10.1049/iet-smt.2016.0208. [48] L. Zhang, Y. Shi, W. Wang, Y. Chu, X. Yuan, Real-time and user-independent feature classification of forearm using EMG signals, J. Soc. Inf. Disp. 27 (2019) 101–107, http://dx.doi.org/10.1002/jsid.749. [49] Y. Liu, G. Yang, M. Li, H. Yin, Variational mode decomposition denoising combined the detrended fluctuation analysis, Signal Process. 125 (2016) 349–364, http://dx.doi.org/10.1016/j.sigpro.2016.02.011.

Feiyun Xiao received the B.S. and Ph.D. degrees from Hefei University of Technology (HFUT), Hefei, China, in 2013 and 2018. Feiyun Xiao is a lecturer of HFUT currently. His research interests include mechanism design, sEMG signal processing with upper limb exoskeleton, hand movement classification, pathological tremor suppression.

Decai Yang received the B.S. degree in mechanical engineering from Shanghai Jiao Tong University, Shanghai, China, in 2014, and the M.S. degree in mechanical engineering from the Harbin Institute of Technology, Harbin, China, in 2016. He is currently a Mechanical Design Engineer with Aerospace System Engineering Shanghai, Shanghai. His research interests include flexible actuator with variable stiffness and control, space structure, and mechanism.

Zhongming Lv received his M.S. in Mechanical Engineering from Xinjiang agricultural university, China, in 2016. He is currently working toward the Ph.D. degree in Mechatronic Engineering at Hefei University of Technology, China. His research interest includes bio-signals analysis for controlling assistive and rehabilitative technologies, rehabilitation robotics and human motor control.

Guo Xiaohui received his M. Sc. degree and Ph.D. degree both from Hefei University of Technology in 2015 and 2018, respectively. Currently, he is a lecturer and master supervisor at school of electronics and information engineering of Anhui University. His main research includes sensitive electronics and sensing technology, flexible antenna, embedded control system, signal acquisition and processing, data fusion and intelligent algorithms and so on.

Zhengshi Liu is a profess or of Hefei University of Technology (HFUT), Hefei, China. He received his Ph.D. degree from Hefei University of Technology (HFUT), Hefei, China, in 1996. His research interests include the rehabilitation engineering, force sensor, and signal processing.

Yong Wang is a profess or of Hefei University of Technology (HFUT), Hefei, China. He received his B.S. and Ph.D. degrees from Hefei University of Technology (HFUT), Hefei, China, in 1990 and 2008. His research interests include the rehabilitation engineering, force sensor, and signal processing.

Please cite this article as: F. Xiao, D. Yang, Z. Lv et al., Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals, Future Generation Computer Systems (2019), https://doi.org/10.1016/j.future.2019.11.025.