- Email: [email protected]
Development of myoelectric hand that determines hand posture and estimates grip force simultaneously
Biomedical Signal Processing and Control 38 (2017) 312–321
Contents lists available at ScienceDirect
Biomedical Signal Processing and Control journal homepage: www.elsevier.com/locate/bspc
Development of myoelectric hand that determines hand posture and estimates grip force simultaneously Yusuke Yamanoi a,∗ , Soichiro Morishita b , Ryu Kato a , Hiroshi Yokoi c a b c
Faculty of Engineering, Yokohama National University, 79-1 Tokiwadai, Hodogaya-ku, Yokohama, 240-8501, Japan Brain Science Inspired Life Support Research Center, The University of Electro-Communications, 1-5-1 Chofugaoka, Chofu, Tokyo, 182-8585, Japan Faculty of Informatics and Engineering, The University of Electro-Communications, 1-5-1 Chofugaoka Chofu, Tokyo, 182-8585, Japan
a r t i c l e
i n f o
Article history: Received 18 January 2017 Received in revised form 11 June 2017 Accepted 30 June 2017 Keywords: Myoelectric hand Signal processing Pattern recognition Force estimation
a b s t r a c t The purpose of this study is to develop a myoelectric hand that determines hand posture and estimates grip force simultaneously like human hands. Methods for simultaneous hand posture determination and grip force estimation typically yield unsatisfying results because of the complex characteristics of electromyogram (EMG) signals. In this study, the authors developed a myoelectric hand that is able to control both force and posture using a method proposed in our previous study. Tests of the grasp ability of the myoelectric hand conducted with healthy men and an amputee showed that the proposed method can be used to control both grip force and hand posture simultaneously with sufficiently accuracy in a real environment. In conclusion, it was suggested that the hand that control posture and force simultaneously improve the dexterity of amputees. © 2017 Elsevier Ltd. All rights reserved.
1. Introduction A myoelectric hand is a prosthetic hand controlled by electromyogram (EMG) signals, which are the potentials that occur over muscles when muscles contract. A myoelectric hand is said to be the closest type of prosthetic hand to a human hand in terms of functionality and appearance because a wearer can control a myoelectric hand based on his or her intentions and because most myoelectric hands have five fingers, like human hands. Therefore, many amputees want to use myoelectric hands. EMG signals are measured as convolutional signals of impulses because they are neural signals. It is difficult to use these signals, but EMG signals have some features that can be used to estimate wearers’ intentions. The relationship between EMG signal and grip force is illustrated in Fig. 1. The amplitudes of the EMG signals increase with the grasp force. Furthermore, the type of muscle used and the frequency of EMG signals are changed by the hand posture. Therefore, EMG signals possess information on both the grasp force and the hand posture. However, it is difficult to control both the grasp force and the hand posture of a myoelectric hand by EMG signals because EMG signals are very weak and fragile, and because EMG signals are affected by fatigue, sweating, and many other factors.
∗ Corresponding author. E-mail address: [email protected] (Y. Yamanoi). http://dx.doi.org/10.1016/j.bspc.2017.06.019 1746-8094/© 2017 Elsevier Ltd. All rights reserved.
Thus, with traditional myoelectric hands, a change in the EMG signal is regarded as a change in either force or posture that can be controlled by controlling the EMG signal. The grasp control features of traditional myoelectric hands are summarized in Fig. 2. Most traditional hands belong to either one of the two groups; single-degree-of-freedom (DOF) hands that control the force or multi-DOF hands that control the posture. Geethanjali et al. classified myoelectric control schemes into seven: on-off myoelectric control, proportional myoelectric control, direct myoelectric control, finite state machine control, pattern recognition-based myoelectric control, posture myoelectric control, regression myoelectric control [1]. On about single-DOF hands, on-off myoelectric control and proportional myoelectric control are often used. On-off myoelectric control is the most simple control scheme based on a threshold of EMG amplitudes. Hands can run at a constant speed. In proportional myoelectric control, hands convert the contraction level of EMG signals into the force, the velocity, or the position. One of the most famous myoelectric hands of this scheme is DMC plus, produced by Ottobock [2]. Regression myoelectric control is the control strategy developed recently. This scheme control multiple DOF of hand and wrist proportionally at the same time, but currently, it is not possible to realize a lot of hand postures. On about multi-DOF hands, various studies are being conducted. Direct myoelectric control is the scheme to achieve individual control of fingers. However, it is difficult to control without implantable
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
313
Fig. 1. Relationship between EMG signal and grip force.
Fig. 2. The problem of traditional myoelectric hands.
myoelectric sensors. Finite state machine control predefine the postures as states and transit the states by commands. The combination of postures is limited, because it is necessary to transit the state several times till the desired posture is selected. The SSSA-MyHand and bebionic use this scheme [3,4]. Pattern recognition-based myoelectric control learn the hand posture and then determine the posture by pattern recognition. Many kinds of machine learning like linear discriminant analysis (LDA), artificial neural network (ANN), support vector machine (SVM), and so on are used in this scheme [5,6]. In pattern recognition, the force is also a factor in the patterns, so the pattern determined is affected by the force. Therefore, it needs to keep the force stable in order to identify correctly. In posture myoelectric control, EMG signals are mapped in the principal component domain [7]. This scheme enables simultaneous control of multiple DOF of hand, but it do not considered the force control. There are a few myoelectric hands that attempt to perform both hand posture determination and grip force estimation. However, most of these hands can estimate grip force only when limited postures are considered or can convert between posture control and force control in certain situations [4,5]. The grip control of such myoelectric hands differs from that of human hands, which integrate posture control and force control. It is difficult to divide EMG signals into posture and force features independently. Therefore,
it is important to develop the pattern recognition method with consideration of grasp force. 2. Methods EMG signals are fragile biological signals, so EMG features extracted from EMG signals are often used in the control of myoelectric hands. Two types of EMG features, the mean absolute value (MAV) and the power spectrum (PS), were used in this study. MAV is a time-domain feature that reflects the amplitude average of the EMG signal. We can obtain information on the grasp force from the amplitudes of EMG signals. PS is a frequency-domain feature that reflects the strength of each frequency band. Using frequencydomain features allows determination of many postures by a few sensors. We examined the relationships among three factors: EMG features, grasp force, and hand posture. We measured EMG signals while the hand posture and grasp force were changing. As a result, two types of EMG features appeared to increase monotonically with the grasp force. In addition, the form of the increase was different for each posture shown in Fig. 3. Based on these results, the authors developed the method described below for determining the hand posture and estimating the grip force simultaneously [8]. An overview of this method is presented in Fig. 4. This procedure
314
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
Fig. 3. Example of EMG features for a given posture. Colors mean difference in posture.
Fig. 4. Overview of the developed myoelectric hand system. t
models the relationship between EMG features and grip force for each posture, and determines the hand posture and estimates the grip force simultaneously, using several linear regression models.
s¯ l =
2.1. Preprocessing
Where stl is the signal sequence from sensor l at time t and NFL is the length of the frame size. The PS feature is defined as follows:
Preprocessing is performed at the time that EMG signals are measured. EMG signals are fragile biological signals and sensitive to noise. To minimize the effects of noise, EMG signals were measured using a differential amplifier. Then, 10- to 400-Hz frequency bands were extracted from the signals using a band-pass filter. Noise from power line interference, which is a major source of noise for EMG signals, was also removed [9], using a notch filter cut around 50 Hz of EMG signals.
k ffrq
1 NFL
l st
fmav st
=t−NFL +1
k+Ip
|F n, stl |
(3)
n=k−Ip
2 NFL −j n (m − 1) 1 sl (t − NFL + m) · e NFL NFL
|sl − s¯ l |
(1)
(4)
n = 0, 1, . . ., NFL − 1
MAV and PS were used as EMG features. The MAV feature is defined as follows: t
(2)
m=1
where
1 = NFL
1 = 2Ip
F n, stl =
2.2. Feature extraction
l
sl
=t−NFL +1
k stl : Feature vector X consists of fmav stl and ffrq
1 Nspct X = fmav st1 , . . ., fmav (stNch ) , ffrq st1 , . . ., ffrq (stNch )
T
(5)
Where Nch is the number of channels and Nspct is the number of spectra extracted from a channel.
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
315
2.3. Learning We must model the relationship between the EMG features and the grip force for each posture. The relationship between the grip force y and each EMG feature xi was approximated using a firstorder linear regression model of the following form: y = aωj i xi + bωj i
(6)
The values of the parameters aωj i and bωj i at the time when the hand posture is ωj are calculated by the least squares method. The point on the model Pωj is expressed as follows.
→
OPωj = xˆ 1 , xˆ 2 , . . ., xˆ i , . . ., xˆ N Fig. 5. Feature space of EMG. Where x is each EMG feature, ω means each posture, X is input feature vector, and P is the nearest point of each model from X.
T
(7)
All of xˆ i is expressed by only one norm u, and the model for each posture can be expressed by straight lines in a feature space, as illustrated in Fig. 5. If the force on a posture does not change (e.g., rest and wrist motions), the model is expressed as a point. xˆ i (u) =
aωj 1 aωj i
u+
bωj 1 − bωj i
(8)
aωj i
2.4. Determining the posture and estimating the force Next, the hand posture is determined and the grip force is esti→
mated from the input feature vector OX. →
OX = [x1 , x2 , . . ., xi , . . ., xN ]T
(9)
→
→
The distance between OPωj and OX is expressed as follows, as a function of only one norm u.
N
2 xˆ i (u) − xi |Pω X| = →
(10)
j
i=1
This distance reflects the error of the model from the input feature. Therefore, it is possible to determine the posture ω ˆ by identifying the model that minimizes this error. Fig. 6. Developed myoelectric hand.
ω ˆ =
arg min ωj (j=1,2,...,M)
min
→
|Pωj X|
−∞
(11)
Fig. 7. Grip postures to be determined: (1) power grasp (2) precision grasp (3) lateral grasp (4) rest (5) wrist flexion (6) wrist extension (7) wrist pronation (8) wrist supination.
316
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
Table 1 Average failure rate [%]. Subject
A
B
C
D
E
Ipsilateral Bilateral
22.95 23.75
16.97 19.94
14.93 15.36
8.66 10.00
4.88 12.67
Table 2 Average estimation error [N]. Subject
A
B
C
D
E
Ipsilateral Bilateral
8.50 8.04
4.88 5.63
4.02 10.20
5.52 7.41
5.98 6.51
Table 3 Average failure rate [%]. Subject
A
B
C
D
E
Linear Exponential Composite
22.95 40.17 22.58
16.97 18.09 12.11
14.93 29.58 21.64
8.66 26.73 9.10
4.88 10.11 5.72
The estimated force yˆ is determined at the same time, using a model of the following general form: ˆ i + bωi yˆ = aωi ˆ x ˆ
(12)
In this study, the length of the frame size NFL was 512, the number of channels Nch was 5, and the number of extracted spectra of a channel Nspct was 16. The dimension of the feature vector X was 85. 2.5. Mechanism of hand The authors developed a myoelectric hand to implement this method. This myoelectric hand was based on a hand used in the authors’ laboratory [10]. The myoelectric hand developed is shown in Fig. 6. It has three motors and three DOFs. It is said that 85% of the activities of daily living are accomplished using three basic types of hand postures: the power grasp, the precision grasp, and the lateral grasp [11]. The myoelectric hand achieves these three types of basic hand postures by moving three types of joints: the metacarpal phalangeal joints of the index and middle fingers, the metacarpal phalangeal joints of the ring and little fingers, and the carpometacarpal joint of thumb. The grasping speed of the myoelectric hand was controlled by the estimated force. When another posture, meaning a posture different from the current posture, was determined, the myoelectric hand adopted a pre-grasping posture before the hand grasp. If the estimated force was smaller than a threshold force, the myoelectric hand stopped at the pre-grasping posture rather than at the end posture. 3. Determining the hand posture and estimating the grip force By measuring the EMG signals of five healthy men during changes in the grip force and hand posture, we evaluated the usefulness of the method in terms of the failure rate of the hand posture determination and the error associated with the grip force estimation. Although this method uses the grip force as the teacher signal, the grip force cannot be measures for amputees. However, the grip force of amputees can be estimated by bilateral training [12]. Bilateral training is a task that applies the same force with the right and left hands. By associating the EMG signals of one arm with the grip force of the opposite arm while the subject is performing bilateral training, it becomes possible to estimate the grip force of the same-side arm. We compared two cases, measuring EMG signals and the grip force from the ipsilateral side and the bilateral side.
Fig. 8. Waves presented to subject: red) target force, black) measured force from the dynamometer, blue) geometric mean of the EMG features. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
There are eight types of grip postures involved in grip determination as shown in Fig. 7: three types of basic hand postures, rest, and wrist motion. As mentioned previously, three types of grasps – power, precision, and lateral grasps – account for 85% of the activities of daily living [11]. The subjects changed the force based on the waveform on a screen such as that shown in Fig. 8. The red line indicates the target force, the black line indicates the force measured with the dynamometer, and the blue line indicates the geometric average of the EMG features. The grip force was measured by a hand dynamometer or pinch meter developed by Biometrics, Ltd. The EMG signals were measured by sensors developed in our laboratory. The circuit of the EMG sensor is shown in Fig. 9. The grip force and EMG signals were converted to analog voltages by A/D conversion and downloaded to a computer. The frequencies of EMG signals are said to be in the range of 10–400 Hz. Thus, a sampling frequency of 1600 Hz was used, because a sampling frequency approximately four times greater than the EMG frequency is necessary to obtain sufficient information using a Fourier transform. EMG signals in the range of −10 to 10 V were measured with 16-bit accuracy, and the quantization precision was 0.31 mV. We utilized the failure rate of the hand posture determination and the estimation error of the grip force estimation to evaluate the method. The measured data were divided into three groups, and we evaluated the usefulness of the method by cross-validation with two sets of learning data and one set of test data. Failure rates were defined as time ratios of failure for a whole task. Estimation errors were determined as root mean square errors (RMSE) [13]. The RMSE is the root mean square of the difference between the estimated force yˆ t and the measured forceyt , calculated as follows:
2 N yˆ t − yt
RMSE =
t=1
N
(13)
The average failure rates of the hand posture determination and the estimation errors of the grip force estimation are shown in Tables 1 and 2, respectively. Example results for posture determination and grip estimation for the ipsilateral and bilateral conditions are shown in Figs. 10 and 11, respectively. The black lines indicate the measured forces, the colored lines indicate the estimated forces, and each other color indicates a type of hand posture. These results show that the proposed method can be used to determine the hand posture and estimate the grip force with the same accuracy for both cases (ipsilateral and bilateral). The feasibility of employing this method with amputees whose hand cannot measure the force is therefore demonstrated. Validity of the model was verified using the same data. A distribution of grip force and EMG feature is shown in Fig. 12. When the grip force was strong, the linearity was strong. However, when the grip force is less than 10%MVC, the trend seems to have changed. This also appears as the failure of determination at low grasp force in Figs. 10 and 11. Based on this, linear model, exponential model,
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
Fig. 9. Circuit of EMG sensor.
Fig. 10. Results for ipsilateral condition (power grasp).
Fig. 11. Results for bilateral condition (power grasp).
317
318
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
Fig. 12. (a) A distribution of grip force and EMG feature (b) linear model (c) exponential model (d) composite model.
Fig. 13. Four grip postures to be determined: (1) power grasp. (2) precision grasp (3) lateral grasp (4) rest.
and composite model were compared and evaluated in terms of the failure rate of the hand posture determination and the error associated with the grip force estimation. Composite model consist of linear model above 10%MVC and exponential model below 10%MVC. There are three reasons why exponential model was chose: increase monotonically; possible to calculate the distance between the model and the input feature vector; easy to determine the parameter by least squares method.
Failure rates and force estimation errors are shown in Tables 3 and 4. From the tables, you can see that failure rates and estimation errors of exponential model was worse and there are no differ between linear model and composite model. If a more complex model was used, large calculation cost was required and real-time property is lost. For this reason, linear model was used in this paper.
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
319
Table 4 Average estimation error [N]. Subject
A
B
C
D
E
Linear Exponential Composite
8.50 14.85 8.46
4.88 12.13 5.16
4.02 22.09 4.56
5.52 13.52 5.19
5.98 20.76 5.87
Fig. 16. Layout of grasp objects and tray.
Fig. 14. Experimental environment with a healthy man.
Fig. 15. Experimental environment with an amputee.
4. Evaluation of object grasping function To evaluate the effectiveness of the method, the authors tested the grasp ability of the myoelectric hand with two healthy men and an amputee. Fig. 13 shows the four types of postures determined: a power grasp, a precision grasp, a lateral grasp, and rest. Two methods were compared for determining the hand posture using an ANN and controlling the hand at two continuous grasping speeds (slow and fast). Three layer feedforward neural network with 160 hidden nodes was employed. The input was the same feature vector as the proposed method, and the output nodes were associated with the postures. It is expected that the approach times will be large and the failure times will be small at slow speed, and the approach time will be small and the failure times will be large at fast speed. However, with the proposed method, it is expected that both failure times and approach times will be small because the grasp speed of the myoelectric hand can be controlled. Each subject grasped a series of objects and put the objects on a tray in order from left to right as fast as possible. One set of tests consists of three rounds of grasping each type of object. The experimental environments for the healthy men and the amputee are shown in Figs. 14 and 15 respectively, and the layout of the grasped objects and the tray is shown in Fig. 16. The grasped objects are shown in Fig. 17. The subjects performed three sets of tests for each method (proposed and traditional). The subjects were instructed to
grasp the balls with a power grasp, the cubes with a precision grasp, and the card with a lateral grasp. There were three size of cubes: Large (L) (30 mm), Medium (M) (15 mm), and Small (S) (10 mm), and two height of balls: High (H) (180 mm) and Low (L) (85 mm). The diameter of the ball is 70 mm. The difference of instability of the objects requires the different precision for hand control. The attachment points of the EMG sensors are shown in Fig. 18. For the healthy men, EMG signals were measured at five attachment points: over the flexor digitorum superficialis, the extensor digitorum, the abductor pollicis longus, the extensor indicis, and the extensor digiti minimi. These five muscles meet three requirements for myoelectric hands: they are located in the forearm, they are related to the movement of the fingers, and their positions can be determined by external palpation. For the amputee, EMG signals were measured at three attaching points: two flexor points and one extensor point. The reason that fewer sensors were used with the amputee than with the healthy men is that the forearms of amputees are shorter than those of healthy men. After wearing the prosthesis, the learning data were collected by the same way as in the experiment of chapter 3. The learning data consisted of three sets of 24 s of EMG signals and grip forces for each of four postures. The subjects performed the tests after adequate practices. The order of the methods was swapped in order to reduce learning effects. If a subject broke or dropped an object, we considered the grasp to have failed and restarted the test with the next grasp object, after re-establishing the environment. The subjects were instructed to grasp the objects as quickly as they could. The failure times and approach times were employed to evaluate the grasp ability of the myoelectric hand. The approach time is the time elapsed from the subject starting the approach to grasping the object, except in the cases of failed trials or trials that required long times because of incorrect posture determination. The experimental parameters are summarized in Table 5. The results of the experiment are shown in Tables 6–9. Short failure times and approach times are desirable. The faster the grasp speed of the myoelectric hand is, the smaller the approach times and the larger the failure times are considered to be. Because the grasp speed of the myoelectric hand can be controlled by the proposed method, it is expected that both failure times and approach times will be small. The notation N/A in Table 9 indicates that the subject could not grasp the object accurately in all trials. For both the healthy men and the amputee, the myoelectric hand was able to grasp all of the types of objects when the proposed method was used. This means that the proposed method can control both hand posture and grip force based on the operator’s intentions. For the healthy men, with the traditional method at slow speed, although few failures occurred, the approach times were long. On the other hand, with the traditional method at fast speed, although the approach times were short, the number of failures was high. However, with the proposed method, few failures occurred and the approach times were short. The reason for this is
320
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
Fig. 17. Grasp objects: (a) ball (b) cube (c) card.
Fig. 18. Attachment points of EMG sensors. (a) Healthy man (b) amputee.
Table 5 Experimental parameters. Subject
Hand Posture
Target Force
Feature Dimension
Sampling Frequency
Grasp Object
Number of Trials
Two healthy men and an amputee
power grasp
0–60%MVC
5 electrodes (healthy men) 3 electrodes (amputee) 17 dimensions per electrode
1600 Hz
Cube (large, medium, small) Ball (high, low) Card
9 (3 rounds, 3 sets)
precision grasp lateral grasp rest
Table 6 Average failure times for healthy men [TIMES/9 TIMES]. Cube
Proposed Traditional (slow) Traditional (fast)
Table 8 Failure times for amputee [times/9 times].
Ball
Card
L
M
S
L
H
2.5 1.0 4.5
1.5 2.0 6.0
3.5 4.0 5.0
3.0 1.0 3.0
2.0 1.0 2.0
2.5 3.5 4.0
Total
15.0 12.5 24.5
Table 7 Average approach times for healthy men [S]. Cube
Proposed Traditional (slow) Traditional (fast)
Ball
L
M
S
L
H
3.1 3.1 2.6
3.4 3.6 2.2
3.1 3.7 3.3
2.3 5.0 2.5
2.1 5.1 2.3
Card
Total
2.5 5.2 2.7
16.5 25.7 15.6
Cube
Proposed Traditional (slow) Traditional (fast)
Ball
L
M
S
L
H
0 0 4
5 3 5
3 5 6
2 1 4
3 1 5
Card
Total
6 8 9
19 18 33
that the grasp speed of the myoelectric hand was controlled based on the estimated force in the case of the proposed method. For the amputee, as with the healthy men, few failures occurred with the proposal method and the traditional method at slow speed, whereas more failures occurred with the traditional method at fast speed. However, the approach times were very long with the proposed method. The reason for this is that the failure rates for hand posture determination were high with the traditional methods. Thus, few postures were determined, and the subject had to grip the object using an alternative grasp in many trials. The reason for
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321 Table 9 Approach times for amputee [s].
Appendix A. Supplementary data
Cube
Proposed Traditional (slow) Traditional (fast)
321
Ball
L
M
S
L
H
8.0 5.6 5.2
12.3 6.4 4.3
9.2 7.6 5.1
13.0 4.8 6.2
11.5 7.5 6.1
Card
Total
21.6 N/A N/A
75.6 (31.9) (26.9)
the higher failure rates for the amputee is that fewer sensors were used with the amputee than with the healthy men, so the EMG signals were less stable for the amputee than for the healthy men. It is expected that the success improves by increasing the number of sensors, because different muscle activities were showed by palpation in each posture. Moreover, he was declared to have fatigue with proposed method. Changing the force with different postures was an action that he would not normally do because he had never used a myoelectric hand that determines hand posture and estimates grip force simultaneously. Long-term training may be necessary to use the proposed method’s prosthetic hand better. However, the trends observed in the experiments with the amputee were the same as those observed in the experiments with the healthy men. 5. Conclusion The authors have developed a myoelectric hand capable of simultaneous hand posture determination and grip force estimation. The usefulness of the proposed method for simultaneous hand posture determination and grip force estimation was confirmed experimentally. The grasp ability of the myoelectric hand was tested with two healthy men and an amputee. The myoelectric hand was able to grasp all of the object types considered using the proposed method. For healthy men, the proposed method yielded few failures and short approach times. Satisfactory results were not obtained for the amputee, however, because the failure rate with the traditional method was very high. In this study, the myoelectric hand developed by the author was tested with only a few subjects. In the future, the myoelectric hand should be tested with more subjects. Acknowledgements This research was partially supported by JSPS KAKENHI (A) No. 26242061, (B) No. 15H03051, and the “Brain Machine Interface Development” from Japan Agency for Medical Research and Development (AMED). This manuscript was proofread by a proofreading service of Editage.
Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.bspc.2017.06. 019. References [1] Purushothaman Geethanjali, Myoelectric control of prosthetic hands: state-of-the-art review, Med. Devices 9 (2016) 247–255. [2] Ottobock, 8E33 DMC plus electric gripper, http://www.ottobockus.com/ prosthetics/upper-limb-prosthetics/solution-overview/system-electricgreifer/, (Accessed 07 June 2017) (2017). [3] Marco Controzzi, Francesco Clemente, Diego Barone, Alessio Ghionzoli, Christian Cipriani, The SSSA-MyHand: a dexterous lightweight myoelectric hand prosthesis, IEEE Trans. Neural Syst. Rehabil. Eng. 5 (5) (2017) 459–468. [4] R.S.L. Steeper, The World’s Most Advanced Prosthetic Hand-Bebionic, 2017, http://bebionic.com/ (Accessed 07 June 2017). [5] Toshio Tsuji, Taro Shibanoki, Keisuke Shima, Takeshi Takaki, Akira Otsuka, Takaaki Chin, A novel five-fingered myoelectric prosthetic hand and its muscle synergy-based control, Bull. Japn. Soc. Prosthet. Orthot. Educ. Res. Dev. 26 (2) (2010) 91–96. [6] Marie-Francoise Lucas, Adrien Gaufriau, Sylvain Pascual, Christian Doncarli, Dario Farina, Multi-channel surface EMG classification using support vector machines and signal-based wavelet optimization, Biomed. Signal Process. Control 3 (2) (2008) 169–174. [7] Jacob L. Segil, Richard F.ff. Weir, Design and validation of a morphing myoelectric hand posture controller based on principal component analysis of human grasping, IEEE Trans. Neural Syst. Rehabil. Eng. 22 (2) (2014) 249–257. [8] Yusuke Yamanoi, Soichiro Morishita, Ryu Kato, Hiroshi Yokoi, Selective linear-regression model for hand posture discrimination and grip force estimation using surface electromyogram signals, 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (2015) 4812–4815. [9] Rubana H. Chowdhury, Mamun B.I. Reaz, Mohd Alauddin Bin Mohd Ali, Ashrif A.A Bakar, Kalaivani Chellappan, Tae G. Chang, Surface electromyography signal processing and classification techniques, Sensors 13 (2013) 12431–12466. [10] Suguru Hoshikawa, Shintaro Sakoda, Yusuke Yamanoi, Ryu Kato, Soichiro Morishita, Tatsuhiro Nakamura, Tatsuya Seki, Yinlai Jiang, Hiroshi Yokoi, Development and evaluation of a simplified myoelectric prosthetic hand realizing basic grasp function in ADL, J. Japn. Soc. Fuzzy Theory Intell. Inform. 27 (6) (2015) 885–897. [11] Christian Cipriani, Marco Controzzi, Maria Chiara Carozza, Objectives, criteria and methods for the design of the Smart hand transradial prosthesis, Robotica 28 (6) (2009) 919–927. [12] Ernest Nlandu Kamavuako, Dario Farina, Ken Yoshida, Winnie Jensen, Estimation of grasping force from features of intramuscular EMG signals with mirrored bilateral training, Ann. Biomed. Eng. 40 (3) (2012) 648–656. [13] Shahrul Naim Sidek, Nor Anija Jalaludin, Abu Ubaidah Shamsudin, Surface electromyography (sEMG)-based thumb-tip angle and force estimation using artificial neural network for prosthetic thumb, Procedia Eng. 41 (2012) 650–656.
Contents lists available at ScienceDirect
Biomedical Signal Processing and Control journal homepage: www.elsevier.com/locate/bspc
Development of myoelectric hand that determines hand posture and estimates grip force simultaneously Yusuke Yamanoi a,∗ , Soichiro Morishita b , Ryu Kato a , Hiroshi Yokoi c a b c
Faculty of Engineering, Yokohama National University, 79-1 Tokiwadai, Hodogaya-ku, Yokohama, 240-8501, Japan Brain Science Inspired Life Support Research Center, The University of Electro-Communications, 1-5-1 Chofugaoka, Chofu, Tokyo, 182-8585, Japan Faculty of Informatics and Engineering, The University of Electro-Communications, 1-5-1 Chofugaoka Chofu, Tokyo, 182-8585, Japan
a r t i c l e
i n f o
Article history: Received 18 January 2017 Received in revised form 11 June 2017 Accepted 30 June 2017 Keywords: Myoelectric hand Signal processing Pattern recognition Force estimation
a b s t r a c t The purpose of this study is to develop a myoelectric hand that determines hand posture and estimates grip force simultaneously like human hands. Methods for simultaneous hand posture determination and grip force estimation typically yield unsatisfying results because of the complex characteristics of electromyogram (EMG) signals. In this study, the authors developed a myoelectric hand that is able to control both force and posture using a method proposed in our previous study. Tests of the grasp ability of the myoelectric hand conducted with healthy men and an amputee showed that the proposed method can be used to control both grip force and hand posture simultaneously with sufficiently accuracy in a real environment. In conclusion, it was suggested that the hand that control posture and force simultaneously improve the dexterity of amputees. © 2017 Elsevier Ltd. All rights reserved.
1. Introduction A myoelectric hand is a prosthetic hand controlled by electromyogram (EMG) signals, which are the potentials that occur over muscles when muscles contract. A myoelectric hand is said to be the closest type of prosthetic hand to a human hand in terms of functionality and appearance because a wearer can control a myoelectric hand based on his or her intentions and because most myoelectric hands have five fingers, like human hands. Therefore, many amputees want to use myoelectric hands. EMG signals are measured as convolutional signals of impulses because they are neural signals. It is difficult to use these signals, but EMG signals have some features that can be used to estimate wearers’ intentions. The relationship between EMG signal and grip force is illustrated in Fig. 1. The amplitudes of the EMG signals increase with the grasp force. Furthermore, the type of muscle used and the frequency of EMG signals are changed by the hand posture. Therefore, EMG signals possess information on both the grasp force and the hand posture. However, it is difficult to control both the grasp force and the hand posture of a myoelectric hand by EMG signals because EMG signals are very weak and fragile, and because EMG signals are affected by fatigue, sweating, and many other factors.
∗ Corresponding author. E-mail address: [email protected] (Y. Yamanoi). http://dx.doi.org/10.1016/j.bspc.2017.06.019 1746-8094/© 2017 Elsevier Ltd. All rights reserved.
Thus, with traditional myoelectric hands, a change in the EMG signal is regarded as a change in either force or posture that can be controlled by controlling the EMG signal. The grasp control features of traditional myoelectric hands are summarized in Fig. 2. Most traditional hands belong to either one of the two groups; single-degree-of-freedom (DOF) hands that control the force or multi-DOF hands that control the posture. Geethanjali et al. classified myoelectric control schemes into seven: on-off myoelectric control, proportional myoelectric control, direct myoelectric control, finite state machine control, pattern recognition-based myoelectric control, posture myoelectric control, regression myoelectric control [1]. On about single-DOF hands, on-off myoelectric control and proportional myoelectric control are often used. On-off myoelectric control is the most simple control scheme based on a threshold of EMG amplitudes. Hands can run at a constant speed. In proportional myoelectric control, hands convert the contraction level of EMG signals into the force, the velocity, or the position. One of the most famous myoelectric hands of this scheme is DMC plus, produced by Ottobock [2]. Regression myoelectric control is the control strategy developed recently. This scheme control multiple DOF of hand and wrist proportionally at the same time, but currently, it is not possible to realize a lot of hand postures. On about multi-DOF hands, various studies are being conducted. Direct myoelectric control is the scheme to achieve individual control of fingers. However, it is difficult to control without implantable
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
313
Fig. 1. Relationship between EMG signal and grip force.
Fig. 2. The problem of traditional myoelectric hands.
myoelectric sensors. Finite state machine control predefine the postures as states and transit the states by commands. The combination of postures is limited, because it is necessary to transit the state several times till the desired posture is selected. The SSSA-MyHand and bebionic use this scheme [3,4]. Pattern recognition-based myoelectric control learn the hand posture and then determine the posture by pattern recognition. Many kinds of machine learning like linear discriminant analysis (LDA), artificial neural network (ANN), support vector machine (SVM), and so on are used in this scheme [5,6]. In pattern recognition, the force is also a factor in the patterns, so the pattern determined is affected by the force. Therefore, it needs to keep the force stable in order to identify correctly. In posture myoelectric control, EMG signals are mapped in the principal component domain [7]. This scheme enables simultaneous control of multiple DOF of hand, but it do not considered the force control. There are a few myoelectric hands that attempt to perform both hand posture determination and grip force estimation. However, most of these hands can estimate grip force only when limited postures are considered or can convert between posture control and force control in certain situations [4,5]. The grip control of such myoelectric hands differs from that of human hands, which integrate posture control and force control. It is difficult to divide EMG signals into posture and force features independently. Therefore,
it is important to develop the pattern recognition method with consideration of grasp force. 2. Methods EMG signals are fragile biological signals, so EMG features extracted from EMG signals are often used in the control of myoelectric hands. Two types of EMG features, the mean absolute value (MAV) and the power spectrum (PS), were used in this study. MAV is a time-domain feature that reflects the amplitude average of the EMG signal. We can obtain information on the grasp force from the amplitudes of EMG signals. PS is a frequency-domain feature that reflects the strength of each frequency band. Using frequencydomain features allows determination of many postures by a few sensors. We examined the relationships among three factors: EMG features, grasp force, and hand posture. We measured EMG signals while the hand posture and grasp force were changing. As a result, two types of EMG features appeared to increase monotonically with the grasp force. In addition, the form of the increase was different for each posture shown in Fig. 3. Based on these results, the authors developed the method described below for determining the hand posture and estimating the grip force simultaneously [8]. An overview of this method is presented in Fig. 4. This procedure
314
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
Fig. 3. Example of EMG features for a given posture. Colors mean difference in posture.
Fig. 4. Overview of the developed myoelectric hand system. t
models the relationship between EMG features and grip force for each posture, and determines the hand posture and estimates the grip force simultaneously, using several linear regression models.
s¯ l =
2.1. Preprocessing
Where stl is the signal sequence from sensor l at time t and NFL is the length of the frame size. The PS feature is defined as follows:
Preprocessing is performed at the time that EMG signals are measured. EMG signals are fragile biological signals and sensitive to noise. To minimize the effects of noise, EMG signals were measured using a differential amplifier. Then, 10- to 400-Hz frequency bands were extracted from the signals using a band-pass filter. Noise from power line interference, which is a major source of noise for EMG signals, was also removed [9], using a notch filter cut around 50 Hz of EMG signals.
k ffrq
1 NFL
l st
fmav st
=t−NFL +1
k+Ip
|F n, stl |
(3)
n=k−Ip
2 NFL −j n (m − 1) 1 sl (t − NFL + m) · e NFL NFL
|sl − s¯ l |
(1)
(4)
n = 0, 1, . . ., NFL − 1
MAV and PS were used as EMG features. The MAV feature is defined as follows: t
(2)
m=1
where
1 = NFL
1 = 2Ip
F n, stl =
2.2. Feature extraction
l
sl
=t−NFL +1
k stl : Feature vector X consists of fmav stl and ffrq
1 Nspct X = fmav st1 , . . ., fmav (stNch ) , ffrq st1 , . . ., ffrq (stNch )
T
(5)
Where Nch is the number of channels and Nspct is the number of spectra extracted from a channel.
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
315
2.3. Learning We must model the relationship between the EMG features and the grip force for each posture. The relationship between the grip force y and each EMG feature xi was approximated using a firstorder linear regression model of the following form: y = aωj i xi + bωj i
(6)
The values of the parameters aωj i and bωj i at the time when the hand posture is ωj are calculated by the least squares method. The point on the model Pωj is expressed as follows.
→
OPωj = xˆ 1 , xˆ 2 , . . ., xˆ i , . . ., xˆ N Fig. 5. Feature space of EMG. Where x is each EMG feature, ω means each posture, X is input feature vector, and P is the nearest point of each model from X.
T
(7)
All of xˆ i is expressed by only one norm u, and the model for each posture can be expressed by straight lines in a feature space, as illustrated in Fig. 5. If the force on a posture does not change (e.g., rest and wrist motions), the model is expressed as a point. xˆ i (u) =
aωj 1 aωj i
u+
bωj 1 − bωj i
(8)
aωj i
2.4. Determining the posture and estimating the force Next, the hand posture is determined and the grip force is esti→
mated from the input feature vector OX. →
OX = [x1 , x2 , . . ., xi , . . ., xN ]T
(9)
→
→
The distance between OPωj and OX is expressed as follows, as a function of only one norm u.
N
2 xˆ i (u) − xi |Pω X| = →
(10)
j
i=1
This distance reflects the error of the model from the input feature. Therefore, it is possible to determine the posture ω ˆ by identifying the model that minimizes this error. Fig. 6. Developed myoelectric hand.
ω ˆ =
arg min ωj (j=1,2,...,M)
min
→
|Pωj X|
−∞
(11)
Fig. 7. Grip postures to be determined: (1) power grasp (2) precision grasp (3) lateral grasp (4) rest (5) wrist flexion (6) wrist extension (7) wrist pronation (8) wrist supination.
316
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
Table 1 Average failure rate [%]. Subject
A
B
C
D
E
Ipsilateral Bilateral
22.95 23.75
16.97 19.94
14.93 15.36
8.66 10.00
4.88 12.67
Table 2 Average estimation error [N]. Subject
A
B
C
D
E
Ipsilateral Bilateral
8.50 8.04
4.88 5.63
4.02 10.20
5.52 7.41
5.98 6.51
Table 3 Average failure rate [%]. Subject
A
B
C
D
E
Linear Exponential Composite
22.95 40.17 22.58
16.97 18.09 12.11
14.93 29.58 21.64
8.66 26.73 9.10
4.88 10.11 5.72
The estimated force yˆ is determined at the same time, using a model of the following general form: ˆ i + bωi yˆ = aωi ˆ x ˆ
(12)
In this study, the length of the frame size NFL was 512, the number of channels Nch was 5, and the number of extracted spectra of a channel Nspct was 16. The dimension of the feature vector X was 85. 2.5. Mechanism of hand The authors developed a myoelectric hand to implement this method. This myoelectric hand was based on a hand used in the authors’ laboratory [10]. The myoelectric hand developed is shown in Fig. 6. It has three motors and three DOFs. It is said that 85% of the activities of daily living are accomplished using three basic types of hand postures: the power grasp, the precision grasp, and the lateral grasp [11]. The myoelectric hand achieves these three types of basic hand postures by moving three types of joints: the metacarpal phalangeal joints of the index and middle fingers, the metacarpal phalangeal joints of the ring and little fingers, and the carpometacarpal joint of thumb. The grasping speed of the myoelectric hand was controlled by the estimated force. When another posture, meaning a posture different from the current posture, was determined, the myoelectric hand adopted a pre-grasping posture before the hand grasp. If the estimated force was smaller than a threshold force, the myoelectric hand stopped at the pre-grasping posture rather than at the end posture. 3. Determining the hand posture and estimating the grip force By measuring the EMG signals of five healthy men during changes in the grip force and hand posture, we evaluated the usefulness of the method in terms of the failure rate of the hand posture determination and the error associated with the grip force estimation. Although this method uses the grip force as the teacher signal, the grip force cannot be measures for amputees. However, the grip force of amputees can be estimated by bilateral training [12]. Bilateral training is a task that applies the same force with the right and left hands. By associating the EMG signals of one arm with the grip force of the opposite arm while the subject is performing bilateral training, it becomes possible to estimate the grip force of the same-side arm. We compared two cases, measuring EMG signals and the grip force from the ipsilateral side and the bilateral side.
Fig. 8. Waves presented to subject: red) target force, black) measured force from the dynamometer, blue) geometric mean of the EMG features. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
There are eight types of grip postures involved in grip determination as shown in Fig. 7: three types of basic hand postures, rest, and wrist motion. As mentioned previously, three types of grasps – power, precision, and lateral grasps – account for 85% of the activities of daily living [11]. The subjects changed the force based on the waveform on a screen such as that shown in Fig. 8. The red line indicates the target force, the black line indicates the force measured with the dynamometer, and the blue line indicates the geometric average of the EMG features. The grip force was measured by a hand dynamometer or pinch meter developed by Biometrics, Ltd. The EMG signals were measured by sensors developed in our laboratory. The circuit of the EMG sensor is shown in Fig. 9. The grip force and EMG signals were converted to analog voltages by A/D conversion and downloaded to a computer. The frequencies of EMG signals are said to be in the range of 10–400 Hz. Thus, a sampling frequency of 1600 Hz was used, because a sampling frequency approximately four times greater than the EMG frequency is necessary to obtain sufficient information using a Fourier transform. EMG signals in the range of −10 to 10 V were measured with 16-bit accuracy, and the quantization precision was 0.31 mV. We utilized the failure rate of the hand posture determination and the estimation error of the grip force estimation to evaluate the method. The measured data were divided into three groups, and we evaluated the usefulness of the method by cross-validation with two sets of learning data and one set of test data. Failure rates were defined as time ratios of failure for a whole task. Estimation errors were determined as root mean square errors (RMSE) [13]. The RMSE is the root mean square of the difference between the estimated force yˆ t and the measured forceyt , calculated as follows:
2 N yˆ t − yt
RMSE =
t=1
N
(13)
The average failure rates of the hand posture determination and the estimation errors of the grip force estimation are shown in Tables 1 and 2, respectively. Example results for posture determination and grip estimation for the ipsilateral and bilateral conditions are shown in Figs. 10 and 11, respectively. The black lines indicate the measured forces, the colored lines indicate the estimated forces, and each other color indicates a type of hand posture. These results show that the proposed method can be used to determine the hand posture and estimate the grip force with the same accuracy for both cases (ipsilateral and bilateral). The feasibility of employing this method with amputees whose hand cannot measure the force is therefore demonstrated. Validity of the model was verified using the same data. A distribution of grip force and EMG feature is shown in Fig. 12. When the grip force was strong, the linearity was strong. However, when the grip force is less than 10%MVC, the trend seems to have changed. This also appears as the failure of determination at low grasp force in Figs. 10 and 11. Based on this, linear model, exponential model,
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
Fig. 9. Circuit of EMG sensor.
Fig. 10. Results for ipsilateral condition (power grasp).
Fig. 11. Results for bilateral condition (power grasp).
317
318
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
Fig. 12. (a) A distribution of grip force and EMG feature (b) linear model (c) exponential model (d) composite model.
Fig. 13. Four grip postures to be determined: (1) power grasp. (2) precision grasp (3) lateral grasp (4) rest.
and composite model were compared and evaluated in terms of the failure rate of the hand posture determination and the error associated with the grip force estimation. Composite model consist of linear model above 10%MVC and exponential model below 10%MVC. There are three reasons why exponential model was chose: increase monotonically; possible to calculate the distance between the model and the input feature vector; easy to determine the parameter by least squares method.
Failure rates and force estimation errors are shown in Tables 3 and 4. From the tables, you can see that failure rates and estimation errors of exponential model was worse and there are no differ between linear model and composite model. If a more complex model was used, large calculation cost was required and real-time property is lost. For this reason, linear model was used in this paper.
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
319
Table 4 Average estimation error [N]. Subject
A
B
C
D
E
Linear Exponential Composite
8.50 14.85 8.46
4.88 12.13 5.16
4.02 22.09 4.56
5.52 13.52 5.19
5.98 20.76 5.87
Fig. 16. Layout of grasp objects and tray.
Fig. 14. Experimental environment with a healthy man.
Fig. 15. Experimental environment with an amputee.
4. Evaluation of object grasping function To evaluate the effectiveness of the method, the authors tested the grasp ability of the myoelectric hand with two healthy men and an amputee. Fig. 13 shows the four types of postures determined: a power grasp, a precision grasp, a lateral grasp, and rest. Two methods were compared for determining the hand posture using an ANN and controlling the hand at two continuous grasping speeds (slow and fast). Three layer feedforward neural network with 160 hidden nodes was employed. The input was the same feature vector as the proposed method, and the output nodes were associated with the postures. It is expected that the approach times will be large and the failure times will be small at slow speed, and the approach time will be small and the failure times will be large at fast speed. However, with the proposed method, it is expected that both failure times and approach times will be small because the grasp speed of the myoelectric hand can be controlled. Each subject grasped a series of objects and put the objects on a tray in order from left to right as fast as possible. One set of tests consists of three rounds of grasping each type of object. The experimental environments for the healthy men and the amputee are shown in Figs. 14 and 15 respectively, and the layout of the grasped objects and the tray is shown in Fig. 16. The grasped objects are shown in Fig. 17. The subjects performed three sets of tests for each method (proposed and traditional). The subjects were instructed to
grasp the balls with a power grasp, the cubes with a precision grasp, and the card with a lateral grasp. There were three size of cubes: Large (L) (30 mm), Medium (M) (15 mm), and Small (S) (10 mm), and two height of balls: High (H) (180 mm) and Low (L) (85 mm). The diameter of the ball is 70 mm. The difference of instability of the objects requires the different precision for hand control. The attachment points of the EMG sensors are shown in Fig. 18. For the healthy men, EMG signals were measured at five attachment points: over the flexor digitorum superficialis, the extensor digitorum, the abductor pollicis longus, the extensor indicis, and the extensor digiti minimi. These five muscles meet three requirements for myoelectric hands: they are located in the forearm, they are related to the movement of the fingers, and their positions can be determined by external palpation. For the amputee, EMG signals were measured at three attaching points: two flexor points and one extensor point. The reason that fewer sensors were used with the amputee than with the healthy men is that the forearms of amputees are shorter than those of healthy men. After wearing the prosthesis, the learning data were collected by the same way as in the experiment of chapter 3. The learning data consisted of three sets of 24 s of EMG signals and grip forces for each of four postures. The subjects performed the tests after adequate practices. The order of the methods was swapped in order to reduce learning effects. If a subject broke or dropped an object, we considered the grasp to have failed and restarted the test with the next grasp object, after re-establishing the environment. The subjects were instructed to grasp the objects as quickly as they could. The failure times and approach times were employed to evaluate the grasp ability of the myoelectric hand. The approach time is the time elapsed from the subject starting the approach to grasping the object, except in the cases of failed trials or trials that required long times because of incorrect posture determination. The experimental parameters are summarized in Table 5. The results of the experiment are shown in Tables 6–9. Short failure times and approach times are desirable. The faster the grasp speed of the myoelectric hand is, the smaller the approach times and the larger the failure times are considered to be. Because the grasp speed of the myoelectric hand can be controlled by the proposed method, it is expected that both failure times and approach times will be small. The notation N/A in Table 9 indicates that the subject could not grasp the object accurately in all trials. For both the healthy men and the amputee, the myoelectric hand was able to grasp all of the types of objects when the proposed method was used. This means that the proposed method can control both hand posture and grip force based on the operator’s intentions. For the healthy men, with the traditional method at slow speed, although few failures occurred, the approach times were long. On the other hand, with the traditional method at fast speed, although the approach times were short, the number of failures was high. However, with the proposed method, few failures occurred and the approach times were short. The reason for this is
320
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321
Fig. 17. Grasp objects: (a) ball (b) cube (c) card.
Fig. 18. Attachment points of EMG sensors. (a) Healthy man (b) amputee.
Table 5 Experimental parameters. Subject
Hand Posture
Target Force
Feature Dimension
Sampling Frequency
Grasp Object
Number of Trials
Two healthy men and an amputee
power grasp
0–60%MVC
5 electrodes (healthy men) 3 electrodes (amputee) 17 dimensions per electrode
1600 Hz
Cube (large, medium, small) Ball (high, low) Card
9 (3 rounds, 3 sets)
precision grasp lateral grasp rest
Table 6 Average failure times for healthy men [TIMES/9 TIMES]. Cube
Proposed Traditional (slow) Traditional (fast)
Table 8 Failure times for amputee [times/9 times].
Ball
Card
L
M
S
L
H
2.5 1.0 4.5
1.5 2.0 6.0
3.5 4.0 5.0
3.0 1.0 3.0
2.0 1.0 2.0
2.5 3.5 4.0
Total
15.0 12.5 24.5
Table 7 Average approach times for healthy men [S]. Cube
Proposed Traditional (slow) Traditional (fast)
Ball
L
M
S
L
H
3.1 3.1 2.6
3.4 3.6 2.2
3.1 3.7 3.3
2.3 5.0 2.5
2.1 5.1 2.3
Card
Total
2.5 5.2 2.7
16.5 25.7 15.6
Cube
Proposed Traditional (slow) Traditional (fast)
Ball
L
M
S
L
H
0 0 4
5 3 5
3 5 6
2 1 4
3 1 5
Card
Total
6 8 9
19 18 33
that the grasp speed of the myoelectric hand was controlled based on the estimated force in the case of the proposed method. For the amputee, as with the healthy men, few failures occurred with the proposal method and the traditional method at slow speed, whereas more failures occurred with the traditional method at fast speed. However, the approach times were very long with the proposed method. The reason for this is that the failure rates for hand posture determination were high with the traditional methods. Thus, few postures were determined, and the subject had to grip the object using an alternative grasp in many trials. The reason for
Y. Yamanoi et al. / Biomedical Signal Processing and Control 38 (2017) 312–321 Table 9 Approach times for amputee [s].
Appendix A. Supplementary data
Cube
Proposed Traditional (slow) Traditional (fast)
321
Ball
L
M
S
L
H
8.0 5.6 5.2
12.3 6.4 4.3
9.2 7.6 5.1
13.0 4.8 6.2
11.5 7.5 6.1
Card
Total
21.6 N/A N/A
75.6 (31.9) (26.9)
the higher failure rates for the amputee is that fewer sensors were used with the amputee than with the healthy men, so the EMG signals were less stable for the amputee than for the healthy men. It is expected that the success improves by increasing the number of sensors, because different muscle activities were showed by palpation in each posture. Moreover, he was declared to have fatigue with proposed method. Changing the force with different postures was an action that he would not normally do because he had never used a myoelectric hand that determines hand posture and estimates grip force simultaneously. Long-term training may be necessary to use the proposed method’s prosthetic hand better. However, the trends observed in the experiments with the amputee were the same as those observed in the experiments with the healthy men. 5. Conclusion The authors have developed a myoelectric hand capable of simultaneous hand posture determination and grip force estimation. The usefulness of the proposed method for simultaneous hand posture determination and grip force estimation was confirmed experimentally. The grasp ability of the myoelectric hand was tested with two healthy men and an amputee. The myoelectric hand was able to grasp all of the object types considered using the proposed method. For healthy men, the proposed method yielded few failures and short approach times. Satisfactory results were not obtained for the amputee, however, because the failure rate with the traditional method was very high. In this study, the myoelectric hand developed by the author was tested with only a few subjects. In the future, the myoelectric hand should be tested with more subjects. Acknowledgements This research was partially supported by JSPS KAKENHI (A) No. 26242061, (B) No. 15H03051, and the “Brain Machine Interface Development” from Japan Agency for Medical Research and Development (AMED). This manuscript was proofread by a proofreading service of Editage.
Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.bspc.2017.06. 019. References [1] Purushothaman Geethanjali, Myoelectric control of prosthetic hands: state-of-the-art review, Med. Devices 9 (2016) 247–255. [2] Ottobock, 8E33 DMC plus electric gripper, http://www.ottobockus.com/ prosthetics/upper-limb-prosthetics/solution-overview/system-electricgreifer/, (Accessed 07 June 2017) (2017). [3] Marco Controzzi, Francesco Clemente, Diego Barone, Alessio Ghionzoli, Christian Cipriani, The SSSA-MyHand: a dexterous lightweight myoelectric hand prosthesis, IEEE Trans. Neural Syst. Rehabil. Eng. 5 (5) (2017) 459–468. [4] R.S.L. Steeper, The World’s Most Advanced Prosthetic Hand-Bebionic, 2017, http://bebionic.com/ (Accessed 07 June 2017). [5] Toshio Tsuji, Taro Shibanoki, Keisuke Shima, Takeshi Takaki, Akira Otsuka, Takaaki Chin, A novel five-fingered myoelectric prosthetic hand and its muscle synergy-based control, Bull. Japn. Soc. Prosthet. Orthot. Educ. Res. Dev. 26 (2) (2010) 91–96. [6] Marie-Francoise Lucas, Adrien Gaufriau, Sylvain Pascual, Christian Doncarli, Dario Farina, Multi-channel surface EMG classification using support vector machines and signal-based wavelet optimization, Biomed. Signal Process. Control 3 (2) (2008) 169–174. [7] Jacob L. Segil, Richard F.ff. Weir, Design and validation of a morphing myoelectric hand posture controller based on principal component analysis of human grasping, IEEE Trans. Neural Syst. Rehabil. Eng. 22 (2) (2014) 249–257. [8] Yusuke Yamanoi, Soichiro Morishita, Ryu Kato, Hiroshi Yokoi, Selective linear-regression model for hand posture discrimination and grip force estimation using surface electromyogram signals, 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (2015) 4812–4815. [9] Rubana H. Chowdhury, Mamun B.I. Reaz, Mohd Alauddin Bin Mohd Ali, Ashrif A.A Bakar, Kalaivani Chellappan, Tae G. Chang, Surface electromyography signal processing and classification techniques, Sensors 13 (2013) 12431–12466. [10] Suguru Hoshikawa, Shintaro Sakoda, Yusuke Yamanoi, Ryu Kato, Soichiro Morishita, Tatsuhiro Nakamura, Tatsuya Seki, Yinlai Jiang, Hiroshi Yokoi, Development and evaluation of a simplified myoelectric prosthetic hand realizing basic grasp function in ADL, J. Japn. Soc. Fuzzy Theory Intell. Inform. 27 (6) (2015) 885–897. [11] Christian Cipriani, Marco Controzzi, Maria Chiara Carozza, Objectives, criteria and methods for the design of the Smart hand transradial prosthesis, Robotica 28 (6) (2009) 919–927. [12] Ernest Nlandu Kamavuako, Dario Farina, Ken Yoshida, Winnie Jensen, Estimation of grasping force from features of intramuscular EMG signals with mirrored bilateral training, Ann. Biomed. Eng. 40 (3) (2012) 648–656. [13] Shahrul Naim Sidek, Nor Anija Jalaludin, Abu Ubaidah Shamsudin, Surface electromyography (sEMG)-based thumb-tip angle and force estimation using artificial neural network for prosthetic thumb, Procedia Eng. 41 (2012) 650–656.