Brain Computer Interface system based on indoor semi-autonomous navigation and motor imagery for Unmanned Aerial Vehicle control

Expert Systems with Applications 42 (2015) 4196–4206

Contents lists available at ScienceDirect

Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa

Brain Computer Interface system based on indoor semi-autonomous navigation and motor imagery for Unmanned Aerial Vehicle control Tianwei Shi, Hong Wang ⇑, Chi Zhang Department of Mechanical Engineering and Automation, Northeastern University, 110004 Shenyang, Liaoning, China

a r t i c l e

i n f o

Article history: Available online 24 January 2015 Keywords: Brain Computer Interface Motor imagery Unmanned Aerial Vehicle Cross-correlation Logistic regression Semi-autonomous navigation subsystem

a b s t r a c t This paper proposes a non-invasive Electroencephalogram (EEG)-based Brain Computer Interface (BCI) system to achieve the easy-to-use and stable control of a low speed Unmanned Aerial Vehicle (UAV) for indoor target searching. The BCI system for UAV control consists of two main subsystems responsible for decision and semi-autonomous navigation. The decision subsystem is established based on the analysis of motor imagery (MI) EEG. The improved cross-correlation method (CC) is used to accomplish the MI feature extraction and the logistic regression method (LR) is employed to complete the MI feature classification and decision. The average classification accuracy rate of the BCI system reaches to 94.36%. The semi-autonomous navigation subsystem is utilized to avoid obstacles automatically for UAV and provide feasible directions for decision subsystem. The actual indoor target searching experiment is carried out to verify the performance of this BCI system. The experiment validates the feasibility and effectiveness of this BCI system for low speed UAV control by using MI and semi-autonomous navigation. Ó 2015 Elsevier Ltd. All rights reserved.

1. Introduction In the last decade, Unmanned Aerial Vehicles (UAVs) received an increasing attention from the research community (Angelopoulou & Bouganis, 2014). UAVs are highly suitable when aerial operations are required, and the presence of a pilot is dangerous, impossible, or simply expensive (Sinopoli, Micheli, Donato, & Koo, 2001). This pertains to a wide range of applications, including search and rescue (Varela et al., 2014), aerial mapping (Mesas-Carrascosa, Notario-García, de Larriva, de la Orden, & Porras, 2014), target tracking (Quintero & Hespanha, 2014), flight formation autonomously (Tuna, Nefzi, & Conte, 2014), avoid obstacle automatically (Moon & Prasad, 2011) and disaster recovery (Tuna et al., 2014). Sometimes human remote control is required because of the unexpected complexities in the applications. People’s different operating levels, however, often have different influences on UAV control. To achieve the easy-to-use and stable control, an Electroencephalogram (EEG)-based Brain Computer Interface (BCI) system is proposed in this paper for indoor target searching. It can control a low speed UAV continuously in horizontal dimensions

⇑ Corresponding author. Tel.: +86 24 83681942. E-mail addresses: [email protected] (T. Shi), [email protected] (H. Wang), [email protected] (C. Zhang). http://dx.doi.org/10.1016/j.eswa.2015.01.031 0957-4174/Ó 2015 Elsevier Ltd. All rights reserved.

by decision subsystem based on motor imagery (MI) and a semiautonomous navigation subsystem. The BCI system enables communication between brain activity and devices. The spontaneous electrical activity in the brain can be measured and recorded by means of EEG signals (Ouyang, Dang, Richards, & Li, 2010), these EEG signals are measures of the summed activity of millions neurons lying nearby the recording electrode (Li, Yan, Liu, & Ouyang, 2014). They are widely used in non-invasive BCI system because of its simplicity, inexpensiveness and high temporal resolution (Kayikcioglu & Aydemir, 2010; Zavala-Fernández, Orglmeister, Trahms, & Sander, 2012). Usually, the BCI system can be utilized to restore the motor functions or to offer mobility for the motor disabled individuals by using a BCI controlled device, such as the motorized wheelchairs or service robots (Rebsamen et al., 2006; Ron-Angevin, Velasco-Alvarez, Sancha-Ros, & da Silva-Sauer, 2011; Velasco-Álvarez, RonAngevin, da Silva-Sauer, & Sancha-Ros, 2013). MI task is one of the most studied types of EEG signals in BCI systems (GarcíaLaencina, Rodríguez-Bermudez, & Roca-Dorda, 2014). Most of BCI systems based on MI tasks allow user to control the devices in the virtual or physical environment (Barbosa, Achanccaray, & Meggiolaro, 2010; Millan, Renkens, Mouriño, & Gerstner, 2004; Tsui, Gan, & Roberts, 2009). Virtual environment is a favorable and practical tool to train the subjects and test the BCI systems (Clemente, Rodríguez, Rey, & Alcañiz, 2014). Normally, the simulated device in the virtual environment is in charge of two actions

T. Shi et al. / Expert Systems with Applications 42 (2015) 4196–4206

4197

in response to left- or right-hand MI task (Pfurtscheller, Neuper, Schlogl, & Lugger, 1998; Tsui & Gan, 2007). It is proved that the virtual environment improves the performance of BCI system (Ron-Angevin & Díaz-Estrella, 2009). For the BCI system proposed in this paper, the feature extraction and classifier design are the key steps. The feature extraction is used to get the regular patterns of recordings of the brain activities. An efficient feature extraction method can achieve good classification results. Up to now, several feature extraction methods for EEG signals have been applied in BCI applications, such as the Common Spatial Patterns (CSP) (Fattahi, Nasihatkon, & Boostani, 2013), Wavelet Transform (WT) (Liao, Zhu, & Ding, 2013; Ting, Guozheng, Bang-hua, & Hong, 2008), Power Spectral Density (PSD) (Park et al., 2013) and spatio-spectral patterns (Wu, Gao, Hong, & Gao, 2008). Many researchers have analyzed the linear spatial filtering methods like CSP, such as the Regularized CSP (RCSP), stationary CSP (sCSP), spectrally weighted CSP (SPEC-CSP), Fisher’s common spatio-spectral pattern (FCSSP) and iterative spatio-spectral pattern learning (ISSPL) (Fattahi et al., 2013; Lotte & Guan, 2011; Samek, Vidaurre, Müller, & Kawanabe, 2012; Wu, Lai, Xia, Wu, & Yao, 2008). These methods do not consider the non-stationary and high variable nature on time and frequency of the EEG signals. The EEG signals are assumed homogeneous during collecting. The WT is difficult to select the suitable wavelet. The PSD is quite sensitive to EEG electrode location changes and it is unstable. The spatio-spectral patterns are much difficult to select regularization parameters to realize the reliable classification. Considering the drawbacks of above mentioned feature extraction methods, in this paper, the improved CC method is used for MI tasks feature extraction. It is an effective method to provide the discriminative information for any size (small or large) of EEG signals between two different electrodes and it can reduce noise in EEG signals by using correlation calculation (Li & Wen, 2014). Compared with the original EEG signals, the cross-correlation sequences can provide more useful information. To improve the accuracy of feature extraction and classification, the features of mean, standard deviation, skewness, kurtosis, maximum and minimum are extracted from each cross-correlation sequence. In different BCI systems, normally, the feature extraction methods are used jointly with different classifiers. Classifiers help to predict and identify classification of feature variables in different mental states. In biomedical areas, the LR method is receiving more attention and it is most widely and successfully applied in various fields of pattern recognition. Although LR is similar to the Support Vector Machines (SVM), it has the characteristics of low model complexity and low risk of overfitting (Li & Wen, 2014). Compared with SVM, it has two advantages: first, it is no need to adjust the parameters. They are estimated by the method of maximum likelihood estimation (MLE) automatically; second, the classification result of dichotomous and the probability of class membership are given at the same time. Based on the above advantages, the LR method is used as the classifier for MI tasks. The rest of this paper is organized as follows. Section 2 gives the related work about the literature survey. The methods and experiments used in this paper are explained in Section 3. Section 4 depicts the experimental results. In Sections 5 and 6, the discussion and conclusions are given.

systems. Roberts, Stirling, Zufferey, and Floreano (2007) used ultrasound sensors for controlling a flying vehicle in a structured testing environment. The biggest defect of ultrasonic sensor is low precision. Wendel, Meister, Schlaile, and Trommer (2006) proposed a Global Positioning System (GPS) and Inertial Navigation System (INS) integrated navigation system for UAV. While the periods of GPS outages, the accelerometer and magnetometer are used to provide the approximate measurement of gravity vector and the Earth’s magnetic field. Tuna et al. (2014) used a team of UAVs to set up the aided emergency communications system for rescue operations by using the Geographical Information System (GIS). The UAV acquires the precise latitude and longitude positioning by GPS/INS. Then, the height of UAV can be obtained by comparing with the corresponding point in GIS. GPS and INS cannot provide the feasible directions and avoid obstacles automatically for UAV indoor flight, however, and GIS requires a large amount of data. Templeton, Shim, Geyer, and Sastry (2007) employed the visionbased navigation to accomplish the outdoor terrain mapping. Celik and Somani (2009) presented a vision-based method for indoor localization and mapping by using the monocular camera and ultrasound sensor. Angelopoulou and Bouganis (2014) proposed a UAV egomotion estimation method based on the vision-based navigation. This vision-based navigation was realized by getting sparse optical flow map in two-dimension by featureselection (FS) and feature-tracking (FT) between the two continuous frames. Grzonka, Grisetti, and Burgard (2012) utilized the vision-based navigation to implement the autonomous indoor flight. To autonomously reach the desired location, the map of the environment has to be uploaded to UAV in advance and the environment information can be acquired by using the simultaneous localization and mapping (SLAM) method. In summary, one of the main challenges of autonomous approach is to achieve a balance between the intelligence and low computational cost. For example, in the application of indoor target searching, the fully autonomous system can not immediately identify every obstacle and make a decision at an intersection without prior programming. To realize the indoor target searching using the BCI system, the semi-autonomous navigation subsystem based on the laser range finder and front facing real-time video is used in this paper. The laser range finder is employed to extract environmental information. According to the extracted environmental information, the semi-autonomous navigation subsystem can avoid obstacles automatically and provide feasible directions. The feasible direction information is applied to the decision subsystem to help make decisions. The decision subsystem provides intelligent decision support for the semi-autonomous navigation subsystem. The classifier of the decision subsystem identifies a subject’s intentions by extracting useful information from the multivariate recordings of the brain activities. Compared to the autonomous navigation, the semi-autonomous navigation subsystem is more intelligence because of the intelligent decisions. Since the intelligent decisions are made by human, it reduces the computational burden in the navigation system. The subsystem has the advantages of low computational cost and high control efficiency. These features make the semi-autonomous navigation subsystem more suitable for UAV completing the indoor target searching in this paper.

2. Related work

3. Methods and experiments

In this BCI system, the semi-autonomous navigation subsystem is one of the important components for indoor target searching. In recent decades, more and more scholars and researchers focused on UAV control methods and applications. Usually, the high autonomy of UAV is implemented by different autonomous navigation

3.1. BCI system Fig. 1 shows the architecture of the BCI system for UAV control and UAV components. This BCI system consists of decision subsystem and semi-autonomous navigation subsystem. Subjects

4198

T. Shi et al. / Expert Systems with Applications 42 (2015) 4196–4206

Fig. 1. Architecture of the BCI system for UAV control and UAV components.

perform MI tasks to control the UAV flight in the virtual simulation system and actual indoor environment. Firstly, the MI data (i.e. EEG signals) are collected from a set of channels and inputted into the decision subsystem. Then, the EEG signals are preprocessed to remove the artifacts. The improved CC method is used to extract the useful information from the preprocessed data. Finally, the decision subsystem obtains the classification outputs by using the classification method based on LR. The classification outputs are converted to the control instructions and the control instructions are sent to UAV via Wi-Fi. During flight, the UAV transmits the real-time video to the laptop LCD screen by means of Wi-Fi. The semi-autonomous navigation subsystem extracts the environmental features to avoid obstacles automatically and provide feasible directions for the decision subsystem. The feasible directions are transmitted to the laptop LCD screen through Wi-Fi. According to the cues in the laptop LCD screen, subjects execute corresponding MI tasks (left-hand, right-hand or idle MI tasks) to complete the continuous control of UAV. Six light carbon fiber rods with the same length are evenly distributed around the centre of UAV. The angle between each two rods is 60° and the rotational directions of two adjacent rotors are opposite. UAV has an ARM9 processor with the Linux operating system embedded in. A pressure sensor and the 9 degrees of freedom miniaturized inertial measurement unit with three-axis gyroscope, three-axis accelerometer and three-axis magnetometer are used in the UAV. Additionally, it has a front facing CMOS camera with 90° angle lens and a URG-04LX laser range finder with the distance measurement error of 1%. The laser range finder measures the distance from the surrounding objects in a frontal 180° sector of 4 m radius. It is used to avoid obstacles and provide the feasible directions. The total length and the maximum take-off weight are 0.6 m and 2.5 kg respectively.

3.2. Methods 3.2.1. MI feature extraction and classification 3.2.1.1. MI experiment and data acquisition. Six subjects participated in this experiment (four males and two females, aged 21.4 ± 1.5 years). They were recruited within the university and free of medication at the time of the recording session, and they had no history of neurological diseases. Attached with the EEG electrode cap, they sat comfortably in an armchair looking at a fixed point placed in the centre of a 14-inch laptop LCD screen for a rest period of 5 min. The distance between the subject’s eyes

and the screen was about 50 cm. They were told not to make any significant movements or sounds during the experiment and used virtual simulation system to fulfill the experiment in a random order. The virtual simulation system was designed by using C# and OpenGL and was displayed on the screen in the first person view. The length and width of the virtual scenes were 18 m and 10 m respectively. To simulate the actual indoor flight, in this virtual simulation system, the perspective height, scene movement speed and view angle were set as 1 m, 1 m/s and 90° respectively. Fig. 2 depicts the screen display and scope of the virtual scenes and Fig. 3 depicts the pre-specified paths built in the virtual simulation system. The black arrows denote the flying directions and default trajectories of UAV. This experiment was divided into two parts. Part 1: the subjects were required to accomplish the MI tasks (left-hand imagination movement to turn left, right-hand imagination movement to turn right and idle to flying forward) in accordance with the 60 cues (idle, left- and right-hand each 20 times) appeared in the screen randomly. Part 2: subjects were asked to finish flight tasks along with the pre-specified paths in the virtual simulation system via the left-hand, right-hand and idle MI tasks. They participated in 3 trials for these paths. The subjects accomplished this experiment in the same day and rest for 10 min when they finished each part. This study was approved by the Human Research Protections Program of Northeastern University, and it was performed in accordance with the Declaration of Helsinki. All participants were asked to read and sign an informed consent form before participating in the study. Fig. 4 shows the data collection process during the MI experiment. The single and continuous MI tasks are corresponding to the first and second parts of the experiment respectively. In this experiment, the whole time of the data collection of the single MI task was 5 s. The two periods (0–1 s and 4–5 s) in Fig. 4(a) were not adopted because the EEG signals may not stable or the subjects may not perform the MI task. To eliminate the influence of the response delay, the EEG signals were sampled from 1 to 4 s. In the second part of experiment, MI tasks may be maintained for a long time continuously (e.g. 1 min). The collected EEG signals need to be divided into several segments. Fig. 4(b) shows this process. The EEG signals were obtained by 40 channels mounted on an electrode cap (NuAmps, Neuroscan). According to the 10/20 international system, the electrodes (Ag/AgCl) were attached to the scalp. The EEG signals were recorded from 10 electrodes (FC5, FC1, C3, CP5, CP1, FC2, FC6, C4, CP2, and CP6) uninterruptedly. The linked ears electrodes were used as the

T. Shi et al. / Expert Systems with Applications 42 (2015) 4196–4206

(a) Laptop display from the "Start"

4199

(b) Scope of the virtual scenes

Fig. 2. Laptop screen display and scope of the virtual scenes.

Fig. 3. Top view of pre-specified paths.

(a) Data collection process of single MI task

(b) Data collection process of continuous MI tasks

Fig. 4. Data collection processes of two kinds of MI tasks.

common reference. These EEG signals were amplified and stored as digital at 250 Hz with 22 bit for processing. 3.2.1.2. MI data preprocessing. The recorded EEG signals are contaminated with noise at different frequencies. Generally, the high frequency noise is mainly caused by power frequency noise and the low frequency noise is mainly due to muscle noise, eye movements and heartbeats (Kar, Bhagat, & Routray, 2010; Zhang, Wang,

Wang, & Wu, 2013). The EEG signals were first filtered by a notch filter in 50 Hz to eliminate the power interference. Then, they were filtered by a band pass filter of 0.5–30 Hz to reduce the noise. If the filtered signals were larger than 50–70 lV in amplitude, they were usually treated as artifacts caused by eye movement, blinking and so on (Li, He, Fan, & Fei, 2012). The independent component analysis (ICA) was used to isolate and to remove the artifacts (Jung et al., 1998). ICA is a computational method for separating

4200

T. Shi et al. / Expert Systems with Applications 42 (2015) 4196–4206

multi-channel EEG signals. It assumes the aggregation of EEG signals from different channels is linear and no time delay. It can isolate brain activity effectively and remove artifacts easily (Wang, Chen, & Li, 2014). The data is modeled as follow:

U ¼ WX

ð1Þ

where U is the un-mixing matrix; W is the weight matrix. It projects the mixing independent components back into the original EEG signals; X is the recorded EEG signals. The projection of ith independent component is given by

X clean ðiÞ ¼ W 1 ð:; iÞ  Uði; :Þ

ð2Þ

where each column in W1 indicates the activation weights distributed across the electrodes for each independent component. After ICA method, the EEG signals from all channels are separated and the bad brain components exist in X clean may be the artifacts. 3.2.1.3. MI feature extraction. The main purpose of MI feature extraction is to get the regular patterns of the brain activities from subjects. Since the gathered EEG signals from different channels are closely correlated, in this research, they are unable to provide independent information about electric neuronal activities. In addition, these signals obtained from different scalp areas do not provide the same amount of distinguishable information and are unable to be applied in BCI systems directly. After data preprocessing, the noise cannot be eliminated completely. The adopted improved CC method also has the effect on reducing noise by using cross-correlation calculation. The process of the feature extraction based on improve CC method is shown in Fig. 5. First, the C3 channel is selected as the reference channel because it provides more information and corresponds to the motor cortex area of the brain for MI tasks (Sander et al., 2010). In addition, C3 is very sensitive for supporting the MI tasks. As shown in Fig. 5, channel 1 is considered as the reference channel (C3). Then, the cross-correlation between the reference channel and other channels were calculated by using the Eq. (3).

Rxy ðmÞ ¼

Njmj1 X

xðiÞyði  mÞ;

m ¼ ðN  1Þ; . . . ; ðN  1Þ

ð3Þ

i¼0

where m denotes the time shift lag between each signal, m = (N  1), (N  2), . . . , (N  2), (N  1); Rxy(m) is the cross-correlated sequence of the input signals x(i) and y(i) at mth lag; i is the sequence index. The input signal sequences x(i) and y(i) are

finite, the delay between these two signals is the time difference from the origin to the time where the peak occurs in their correlation. For each signal of x(i) and y(i), consists of N finite samples respectively, the result length of cross-correlation sequence is 2N  1. In Fig. 5, R represents the cross-correlation sequence. R1 is created for the reference channel and the channel 2. R(n1) is generated by the reference channel and channel n. To improve the CC method, the features of mean, standard deviation, skewness, kurtosis, maximum and minimum were extracted from each cross-correlation sequence. These features can reduce the dimension of each cross-correlation sequence and supply more useful information of regular patterns compared with the EEG signals after data preprocessing (Mendenhall, Beaver, & Beaver, 2012). In this paper, the feature sets including the six extracted statistical features were used as the inputs of the classifier. 3.2.1.4. MI classification. Some studies have verified that the twoclass classifier has the best classification accuracy (Kronegg, Chanel, Voloshynovskiy, & Pun, 2007). In this paper, the BCI system is utilized to achieve the continuous control of a low speed UAV in two-dimension. Since the semi-autonomous navigation subsystem can provide the feasible directions and automatic obstacles avoidance, it reduces the required control instructions. The UAV only requires the forward and rotation controls. To ensure the BCI system has a high accuracy, therefore, the multiple control categories are reduced to two-class control categories (idle vs. MI task and left-hand MI task vs. right-hand MI task). LR fits a separating hyper plane that is a linear function of input features between two classes (Li & Wen, 2014). Here, the LR method is used to predict the class labels and the probabilities of the two categories of the MI tasks. The mathematical expression of LR method is given by Eq. (4).

Pn eb0 þ i¼1 bi þxi Pn Pðy ¼ 1jx1 ; x2 ; . . . ; xn Þ ¼ p ¼ 1 þ eb0 þ i¼1 bi þxi

ð4Þ

Assuming that x1, x2, . . . , xn are input feature vectors and they are treated as independent variables; y is the class label and it is treated as a dependent variable; p is a conditional probability of the class 1, namely Pðy ¼ 1jx1 ; x2 ; . . . ; xn Þ: If y belongs to class 0, it can be written as 1  p ¼ 1  Pðy ¼ 1jx1 ; x2 ; . . . ; xn Þ ¼ Pðy ¼ 0jx1 ; x2 ; . . . ; xn Þ; b0 is the intercept; b1 ; b2 ; . . . ; bn are the regression coefficient related to the input feature vectors. These regression coefficients are estimated by maximum likelihood estimation (MLE) (Li & Wen, 2014). The logit model of LR is given as Eq. (5).

Fig. 5. EEG signal processing procedure.

4201

T. Shi et al. / Expert Systems with Applications 42 (2015) 4196–4206

logit ðpÞ ¼ loge



n X p  bi xi ¼ b0 þ 1p i¼1

ð5Þ

In Eq. (5), logit ðpÞ is a linear collection of the independent variables and regression coefficients. In this study, the LR model is employed to classify the MI EEG signals where the feature sets (discussed in Section 3.2.1.3) are used as the inputs to the LR model. The outputs of the LR model are the class labels y of a subject’s mental state (e.g. left- and right-hand MI). It is known that y has two values, represented by 0 or 1. In this paper, classification decision process is divided into two stages. In the stage of MI existence judgment (i.e. the first stage), the MI task is treated as 1 and the idle state is treated as 0. The second stage is the judgment process of leftand right-hand MI. If the MI task is performed, the left-hand MI task is treated as 1 and the right-hand MI task is treated as 0. The feasibility of the improved CC method and LR method are evaluated based on the classification accuracy rate. A total of 360 trials are obtained from the MI experiment. The 240 trials are used as the training set and the other 120 trials are used as the testing set. Finally, the average classification accuracy is evaluated across the training set and the testing set. 3.2.2. Obstacle avoidance and feasible direction estimation The laser range finder scans indoor environment from 90° to 90° in frontal direction (the nose direction of UAV is 0°). The collected data is divided into 9 groups of equal size according to the scanning angle (Perrin, Chavarriaga, Colas, Siegwart, & Millán, 2010) and the maximal and minimal distances in each group are calculated. According to the collected data, the semi-autonomous navigation subsystem extracts the environmental features to avoid obstacles for UAV and provide feasible directions for decision subsystem. Fig. 6 presents the example of the environmental feature extraction. Fig. 6(a) shows the real environment. Fig. 6(b) and (c)

describe the environment arrangement according to the scanning angle of the 9 groups of equal size and the revised result after the polar coordinate transformation respectively. Considering that the UAV could encounter narrow cluttered or wide open spaces during flight, in this research, the mean of measured maximal and minimal distances of each group is used as the threshold (Ta) for obstacle recognition. To ensure the UAV avoid obstacles successfully and safely, the minimum Ta is set as 1 m. If the distances in some directions are less than Ta, the corresponding regions are taken as obstacles and removed. Other regions are treated as pending feasible directions and they are preserved. After that, the semi-autonomous navigation subsystem will rearrange the environment in accordance with the preserved regions. Fig. 7 shows the rearranged environment and the estimated feasible direction. For the opening O1 and O2, the width of Oi is calculated as follow:

Oi ¼

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi s22i1 þ s22i  2s2i1 s2i cos ðt2i  t 2i1 Þ

ð6Þ

where i is the index of openings; s2i1 is the length from the laser range finder to the starting point of Oi; s2i is the length from the laser range finder to the end point of Oi; t2i1 is the degree from 0° to the starting point of Oi; t2i is the degree from 0° to the end point of Oi. This width Oi is then compared with the feasible directions threshold To. The minimum To is set as 1.5 m. If the width Oi exceeds To, the corresponding opening is treated as the feasible direction (O2 is the feasible directions in Fig. 7). According to the current yaw angle of UAV (tU), t2i1 and t2i, the centre of feasible direction is defined as follow:

(

tU þ t2i þt22i1  360; tU þ t2i þt22i1 þ 360;

  t U þ t2i þt22i1 P 180   t U þ t2i þt22i1 < 180

(a) Real environment

(b) Environment arrangement Fig. 6. Example of the environmental feature extraction.

(c) Revised environment

ð7Þ

4202

T. Shi et al. / Expert Systems with Applications 42 (2015) 4196–4206

Fig. 7. The rearranged environment and the estimated feasible direction.

3.3. Actual indoor target searching experiment 3.3.1. Experiment set-up To verify the performance of this BCI system, ten healthy subjects (four males and two females had participated in the MI experiment, aged 21.4 ± 1.5 years; two males and two females hadn’t participated in the MI experiment, aged 21 ± 1.7 years) participated in the actual indoor target searching experiment. The speed of UAV was set as 1 m/s; the rotation speed and radius of UAV was set as 15°/s and 0; the height is limited in the range of 0.5–2.5 m; the initial height and yaw angle were set as 1 m and 0° respectively. The Institute of Mechatronic Engineering of Department of Mechanical Engineering and Automation, Northeastern University Shenyang, Liaoning, China was selected as the indoor environmental site. The length and width of the indoor environmental site were 40 m and 25 m respectively. Fig. 8(a) shows the top view of the indoor environmental site. It includes several T-type crossroads such as the marked areas 1–7. Let ‘‘Start’’ and ‘‘Target’’ as the position of the UAV take-off and searching target respectively. The searching target was placed at an unknown place for subjects and was hanged on the wall. Fig. 8(b) shows the searching target that it was a red ring. The inside diameter was 40 cm and the outside diameter was 65 cm. The subjects were not familiar with the indoor environmental site when they participated in the experiment.

(a) Top view of the site

The indoor target searching task is consisted of two parts: the target searching tasks using the BCI system and without using the BCI system (the subjects used mobile phone to control UAV). For the target searching task using the BCI system, the ten subjects were all took part in this part. The subjects attached the EEG electrode cap and sat comfortably in an armchair looking at a fixed point placed in the centre of a monitor for a rest period of 5 min at the beginning of the experiment. The real-time video and feasible directions transmitted from the UAV were displayed on the 14inch laptop screen in the first person view. The distance between the subject’s eyes and the screen was about 50 cm. They conducted this searching task by using MI tasks. For the target searching task without using the BCI system, the four subjects, who had not participated in the MI experiment, were involved in this part. They performed this task by using the mobile phone. The subjects sat comfortably in an armchair and naturally hold the mobile phone by two hands for a rest period of 1 min at the beginning of the experiment. The control keys and real-time flight video were displayed on the phone screen directly. The distance between the subject’s eyes and phone screen was about 20 cm. To keep the subjects unfamiliar with the site, the position of ‘‘Start’’ and ‘‘Target’’ were changed.

3.3.2. UAV flight control The subjects use single idle MI task to make the UAV take off. Once the UAV takes off, the semi-autonomous navigation subsystem will provide all feasible directions to the subjects and avoid obstacles automatically. The subjects are required to provide choices to these feasible directions by using MI tasks through the decision subsystem (imagination of left-hand movement to choose ‘‘Yes’’, namely choose the current direction; imagination of righthand movement to choose ‘‘No’’, namely not choose the current direction). The feasible directions appear alternately as the current directions to be chosen. If the subjects choose one of the feasible directions, this direction is set as the yaw angle for UAV and the UAV will fly forward along this selected direction automatically. The decision subsystem transmits the control instructions and the interval time of the transmission is 100 ms. In this situation, the single right-hand MI task is utilized to land the UAV. During the process of selecting feasible directions, the UAV keeps on hovering. If the subjects do not select any feasible directions, they must perform MI tasks to control UAV completely by themselves (imagination of left-hand movement to turn left; imagination of right-hand movement to turn right; idle imagination to keep flying forward). In this case, the decision subsystem transmits the control instructions and the interval time of the transmission is 100 ms. The lasting time of

(b) Searching target

Fig. 8. Top view of the indoor environmental site and searching target.

T. Shi et al. / Expert Systems with Applications 42 (2015) 4196–4206 Table 1 Comparison of the classification accuracy rate (%) between the methods used in this paper and the three most recent reported algorithms. Algorithms

S1

S2

S3

S4

S5

S6

Average

Improved CC–LR FCSSP ISSPL CSP

91.6 90.67 91.33 88.33

96.2 93.33 84.2 78.33

91.75 95.67 86.33 75

98.2 93 98.1 80

97.28 94.67 83.33 86.67

91.12 90 87.33 81.67

94.36 92.89 88.44 81.67

the transmission is 3 s. If the UAV does not receive the control instructions in the next 2 s, it will hover and wait for the next control instruction. The subjects can land the UAV by manual control. When the UAV arrives at any crossroads, the semi-autonomous navigation subsystem will provide all feasible directions again. The UAV will hover and wait for the subjects selecting the directions through the above process. During the whole process of UAV flight, if the UAV encounters any obstacles, first, it will hover and turn to the feasible direction for subsequent flight. If the UAV arrives at a dead end, it will rotate 180° automatically. 4. Results

4203

that the combination of improved CC method and the LR method (Improved CC–LR) has relatively high classification accuracy rate. The accuracies of these subjects were all more than 91% and it was significantly higher than other three algorithms for most of the subjects. It can be concluded that the improved CC method and LR method outperforms the other algorithms for the MI tasks classification in this BCI system. After this part of experiment, the significant differences in these three MI tasks were achieved by Wilcoxon test as p < 0.05. For the second part of the experiment, the number of commands and the time spent on the paths were used to analyze the stability of UAV control by using this BCI system. Fig. 9(a) shows the results of the number of commands (mean ± S.D.). Fig. 9(b) describes the results of the time spent on the paths (mean ± S.D.) and Fig. 9(c) shows the success rates of the path selection (mean ± S.D.) of the six subjects. Since the complexities of the four pre-selected paths were different, the number of commands and the time spent on the paths were also significantly different. With the increase of path complexities, the success rates decrease. As can be seen from Fig. 9(c), the success rates are all above 80%. Since the success rates are relatively high, the system is effective and easy-to-use for different subjects.

4.1. Results for MI experiment 4.2. Results for indoor target searching Table 1 displays the comparison of the classification accuracy rate between the methods used in this paper and the three most recent reported algorithms (FCSSP, ISSPL and CSP). It demonstrated

Fig. 10 describes the top view of actual trajectories of the ten subjects in the target searching task. Fig. 10(a) shows the actual

(a) Number of commands

(b) Time spent on the paths

(c) Success rates of the path selection Fig. 9. System performance in the flight tasks along with the pre-specified paths.

4204

T. Shi et al. / Expert Systems with Applications 42 (2015) 4196–4206

(a) Actual trajectories of subjects 1-6

(b) Actual trajectories of subjects 7-10 using the BCI system

(c) Actual trajectories of subjects 7-10 without using the BCI system Fig. 10. Top view of actual trajectories in the indoor target searching.

trajectories of the six subjects who had participated in the MI experiment (subjects 1–6). Fig. 10(b) shows the actual trajectories of the other four subjects, who did not participated in the MI experiment (subjects 7–10), using the BCI system. Fig. 10(c) describes the top view of actual trajectories of subjects 7–10 without using the BCI system. The colors of these curves are used to distinguish different subjects. Compared with Fig. 10(c), the curves in Fig. 10(a) and (b) are smoother and more regular. The trajectories in Fig. 10(b) are approximate to the trajectories in Fig. 10(a). Table 2 describes the comparison of the two searching tasks (using the BCI system and without using the BCI system). Distance errors and rotation errors are used to analyze the stability of UAV control. The distance error is the length between the recorded coordinate point and its mapping point on the ideal path. The rotation error is the yaw angle difference between the recorded coordinate point and its mapping point on the ideal path during the UAV rotated 90°. As shown in Table 2, the flight of the UAV based on the BCI system has the smaller distance errors and rotation errors.

5. Discussion In this paper, a non-invasive EEG-based BCI system for UAV control is proposed. In most of the proposed online BCI systems used in the wheelchair and other devices, subjects mostly control devices through the interface. To achieve a balance between the intelligence and low computational cost, a novel semi-autonomous navigation subsystem is proposed to provide feasible directions and avoid obstacles automatically. The URG-04LX scanning laser range finder from Hokuyo is used to measure the distances from

Table 2 Comparison of the two searching tasks: distance errors using the BCI system (MDEU, mean ± S.D.), rotation errors using the BCI system (MREU, mean ± S.D.), distance errors without using the BCI system (MDEW, mean ± S.D.) and rotation errors without using the BCI system (MREW, mean ± S.D.). Subjects

MDEU (cm)

MDEW (cm)

MREU (deg)

MREW (deg)

7 8 9 10

8.35 ± 3.2 7.79 ± 2.8 10.42 ± 3.1 9.24 ± 3.3

14.37 ± 4.4 13.62 ± 3.9 16.28 ± 4.2 13.21 ± 3.7

8.69 ± 2.6 8.12 ± 2.5 10.41 ± 2.8 10.84 ± 2.7

12.41 ± 4.1 13.25 ± 4.7 15.31 ± 4.6 15.42 ± 4.7

the surrounding objects in a frontal 180° sector of 4 m radius. It is subjected to the detection distance. During the experiments, subjects need to watch the monitor for a long time. To alleviate this tiring effect, a 14-inch laptop LCD screen instead of the CRT was used in this BCI system (Wu et al., 2008). For the MI experiment, first, subjects accepted passive MI tasks calibration. As shown in Table 1, it is noted that the combination of improved CC–LR method has relatively high classification accuracy rate. The highest classification accuracy rate is 98.2% and the average classification accuracy rate is 94.36%. It increased by 1.47%, 5.92% and 12.69% compared with FCSSP, ISSPL and CSP algorithms respectively. Each subject was asked to finish flight tasks along with pre-specified paths in the virtual simulation system by using MI tasks. These subjects did not know when to perform the MI tasks in advance. As can be seen from the Fig. 9(c), the success rates are all above 80%. Since the success rates are relatively high, the system is effective and easy-to-use for different subjects.

T. Shi et al. / Expert Systems with Applications 42 (2015) 4196–4206

In Fig. 10(a), the trajectories are smooth and concentrated on the middle of the corridors. Sometimes, there are small offsets in the actual trajectories, which may be caused by instabilities of hovering and forward flying. The actual trajectories of the subjects 4 and 5 are the best in Fig. 10(a). They spent the shortest time and used the least instructions to accomplish the searching task. There are relative larges offsets, at the first crossroad only for the third subject. These offsets are caused by the uncertainty in the direction selection process. First, the subject selected to turn left and then this subject chose to go straight. Compared with Fig. 10(c), the curves in Fig. 10(a) and (b) are smoother and more concentrated. In addition, the trajectories in Fig. 10(b) are approximate to the trajectories in Fig. 10(a). As shown in Table 2, the flight of the UAV based on the BCI system has the smaller distance errors and rotation errors. It illustrates that the inexperienced subjects can relatively successfully control the UAV to perform the target searching task using the BCI system. In Fig. 10(b), these trajectories are more chaotic than the actual trajectories in Fig. 10(a). Because they were not familiar with MI tasks, the UAV had large deviations in processes of flying along a straight line and taking a rotation, such as the subjects from start to the marked area 1 and 2. At these moments, since subjects did not select the provided feasible directions, they needed to perform MI tasks to control the UAV completely by themselves. Compared to the six subjects, they spent a little more time and performed more MI tasks. Compared with the beginning of this experiment, the results showed that the four subjects mastered the corresponding knowledge on MI tasks and were able to operate the BCI system better. From another aspect, it was proved that the proposed BCI system had the good feasibility and adaptation. The subjects were able to master the BCI system through short time learning. Although they did not reach the accuracy of subjects 1–6, the differences were small. There are three possible key factors: first, the semi-autonomous navigation subsystem is adopted to provide feasible directions; second, the utilized improved CC method for MI feature extraction and LR method for MI classification maximize the separability between the MI tasks; third, this BCI system simplifies the operation process and it is easy-to-use for the subjects. Although the results demonstrated the feasibility of this proposed BCI system, some drawbacks still exist while conducting experiments. It should be noted that, during these experiments, not all subjects were able to generate the available and stable EEG patterns. This may be caused by many factors, such as the motivation of the subjects and differences in electrode cap placements. Another important reason was that the MI experiment did not require subjects to achieve 100% performance in every trial, but the actual flight of the UAV demands almost perfected performance all the time. An error decision needed to be corrected by a series of correct instructions.

6. Conclusions This paper presents a non-invasive BCI system to continuously control an UAV for indoor target searching in two-dimension. It is based on the decision subsystem and the semi-autonomous navigation subsystem. To establish this decision subsystem, the regular EEG patterns of the left-hand, right-hand and idle MI tasks were analyzed. The improved CC method is used to accomplish the feature extraction of MI and the LR method is employed to complete the feature classification and decision. The six extracted features of mean, standard deviation, skewness, kurtosis, maximum and minimum from each cross-correlation sequence are set as the inputs of classifier based on the LR method. To achieve a balance between the intelligence and low computational cost, the semi-autonomous navigation subsystem was adopted in this

4205

paper. It consists of the laser range finder and front facing CMOS camera. It was used to avoid obstacles automatically for UAV and provide feasible directions for decision subsystem. In the MI experiment, the highest accuracy rates of this BCI system were reached to 98.2% and 100% respectively. In the actual indoor target searching experiment, the six subjects who had participated in the MI experiment controlled UAV more reasonably and smoothly than the subjects who did not use the BCI system. The feasibility of this proposed BCI system was proved. In the future work, the proposed BCI system will be expanded and applied in three-dimension. To realize the UAV control on vertical direction without affecting the performance of this BCI system, we plan to extend the improved CC and LR methods for multiclass classification. Another part of the future work is to integrate the multiple inputs to make the system more reliable and flexible for decision, for example, adding EOG, EMG or other physiological signals as the additional decision bases. Since the actual obstacles may not be static, we will improve the real-time performance of this BCI system. Acknowledgments The authors gratefully acknowledge the financial support from the University Innovation Team of Liaoning Province (LT2014006) and National Natural Science Foundation of China (51405073, 61071057). The authors also thank Dr. F Wang, Dr. G Sun and Dr. R Fu for their helps in the experiments. References Angelopoulou, M., & Bouganis, C. (2014). Vision-based egomotion estimation on FPGA for unmanned aerial vehicle navigation. IEEE Transactions on Circuits and Systems for Video Technology, 24(6), 1070–1083. Barbosa, A. O. G., Achanccaray, D. R., & Meggiolaro, M. A. (2010). Activation of a mobile robot through a brain computer interface. In 2010 IEEE international conference on robotics and automation (ICRA) (pp. 4815–4821). IEEE. Celik, K., & Somani, A. K. (2009). Monocular vision SLAM for indoor aerial vehicles. Journal of electrical and computer engineering (Vol. 2, pp. 1566–1573). IEEE Press. Clemente, M., Rodríguez, A., Rey, B., & Alcañiz, M. (2014). Assessment of the influence of navigation control and screen size on the sense of presence in virtual reality using EEG. Expert Systems with Applications, 41(4), 1584–1592. Fattahi, D., Nasihatkon, B., & Boostani, R. (2013). A general framework to estimate spatial and spatio-spectral filters for EEG signal classification. Neurocomputing, 119, 165–174. García-Laencina, P. J., Rodríguez-Bermudez, G., & Roca-Dorda, J. (2014). Exploring dimensionality reduction of EEG features in motor imagery task classification. Expert Systems with Applications, 41(11), 5285–5295. Grzonka, S., Grisetti, G., & Burgard, W. (2012). A fully autonomous indoor quadrotor. IEEE Transactions on Robotics, 28(1), 90–100. Jung, T. P., Humphries, C., Lee, T. W., Makeig, S., McKeown, M. J., Iragui, V., et al. (1998). Extended ICA removes artifacts from electroencephalographic recordings. Advances in Neural Information Processing Systems, 894–900. Kar, S., Bhagat, M., & Routray, A. (2010). EEG signal analysis for the assessment and quantification of driver’s fatigue. Transportation Research Part F: Traffic Psychology and Behaviour, 13(5), 297–306. Kayikcioglu, T., & Aydemir, O. (2010). A polynomial fitting and k-NN based approach for improving classification of motor imagery BCI data. Pattern Recognition Letters, 31(11), 1207–1215. Kronegg, J., Chanel, G., Voloshynovskiy, S., & Pun, T. (2007). EEG-based synchronized brain-computer interfaces: A model for optimizing the number of mental tasks. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 15(1), 50–58. Li, W., He, Q. C., Fan, X. M., & Fei, Z. M. (2012). Evaluation of driver fatigue on two channels of EEG data. Neuroscience Letters, 506(2), 235–239. Li, Y., & Wen, P. (2014). Modified CC–LR algorithm with three diverse feature sets for motor imagery tasks classification in EEG based brain–computer interface. Computer Methods and Programs in Biomedicine, 113(3), 767–780. Li, J., Yan, J., Liu, X., & Ouyang, G. (2014). Using permutation entropy to measure the changes in EEG signals during absence seizures. Entropy, 16(6), 3049–3061. Liao, K., Zhu, M., & Ding, L. (2013). A new wavelet transform to sparsely represent cortical current densities for EEG/MEG inverse problems. Computer Methods and Programs in Biomedicine, 111(2), 376–388. Lotte, F., & Guan, C. (2011). Regularizing common spatial patterns to improve BCI designs: Unified theory and new algorithms. IEEE Transactions on Biomedical Engineering, 58(2), 355–362. Mendenhall, W., Beaver, R., & Beaver, B. (2012). Introduction to probability and statistics. Cengage Learning.

4206

T. Shi et al. / Expert Systems with Applications 42 (2015) 4196–4206

Mesas-Carrascosa, F. J., Notario-García, M. D., de Larriva, J. E. M., de la Orden, M. S., & Porras, A. G. F. (2014). Validation of measurements of land plot area using UAV imagery. International Journal of Applied Earth Observation and Geoinformation, 33, 270–279. Millan, J. R., Renkens, F., Mouriño, J., & Gerstner, W. (2004). Noninvasive brainactuated control of a mobile robot by human EEG. IEEE Transactions on Biomedical Engineering, 51(6), 1026–1033. Moon, J., & Prasad, J. V. R. (2011). Minimum-time approach to obstacle avoidance constrained by envelope protection for autonomous UAVs. Mechatronics, 21(5), 861–875. Ouyang, G., Dang, C., Richards, D. A., & Li, X. (2010). Ordinal pattern based similarity analysis for EEG recordings. Clinical Neurophysiology, 121(5), 694–703. Park, S. A., Hwang, H. J., Lim, J. H., Choi, J. H., Jung, H. K., & Im, C. H. (2013). Evaluation of feature extraction methods for EEG-based brain–computer interfaces in terms of robustness to slight changes in electrode locations. Medical & Biological Engineering & Computing, 51(5), 571–579. Perrin, X., Chavarriaga, R., Colas, F., Siegwart, R., & Millán, J. D. R. (2010). Braincoupled interaction for semi-autonomous navigation of an assistive robot. Robotics and Autonomous Systems, 58(12), 1246–1255. Pfurtscheller, G., Neuper, C., Schlogl, A., & Lugger, K. (1998). Separability of EEG signals recorded during right and left motor imagery using adaptive autoregressive parameters. IEEE Transactions on Rehabilitation Engineering, 6(3), 316–325. Quintero, S. A. P., & Hespanha, J. P. (2014). Vision-based target tracking with a small UAV: Optimization-based control strategies. Control Engineering Practice, 32, 28–42. Rebsamen, B., Burdet, E., Guan, C., Zhang, H., Teo, C. L., Zeng, Q., et al. (2006). A brain-controlled wheelchair based on P300 and path guidance. In The first IEEE/ RAS-EMBS international conference on biomedical robotics and biomechatronics, 2006. BioRob 2006 (pp. 1101–1106). IEEE. Roberts, J., Stirling, T., Zufferey, J. C., & Floreano, D. (2007). Quadrotor using minimal sensing for autonomous indoor flight. In Proceedings of the European micro air vehicle conference and flight competition (EMAV2007). Ron-Angevin, R., & Díaz-Estrella, A. (2009). Brain–computer interface: Changes in performance using virtual reality techniques. Neuroscience Letters, 449(2), 123–127. Ron-Angevin, R., Velasco-Alvarez, F., Sancha-Ros, S., & da Silva-Sauer, L. (2011). A two-class self-paced BCI to control a robot in four directions. In 2011 IEEE international conference on rehabilitation robotics (ICORR) (pp. 1–6). IEEE. Samek, W., Vidaurre, C., Müller, K. R., & Kawanabe, M. (2012). Stationary common spatial patterns for brain–computer interfacing. Journal of Neural Engineering, 9(2), 026013. Sander, T. H., Leistner, S., Wabnitz, H., Mackert, B. M., Macdonald, R., & Trahms, L. (2010). Cross-correlation of motor activity signals from dcmagnetoencephalography, near-infrared spectroscopy, and electromyography. Computational Intelligence and Neuroscience, 2010, 10.

Sinopoli, B., Micheli, M., Donato, G., & Koo, T. J. (2001). Vision based navigation for an unmanned aerial vehicle. IEEE international conference on robotics and automation, 2001. Proceeding 2001 ICRA (Vol. 2, pp. 1757–1764). IEEE. Templeton, T., Shim, D. H., Geyer, C., & Sastry, S. S. (2007). Autonomous vision-based landing and terrain mapping using an MPC-controlled unmanned rotorcraft. In IEEE international conference on robotics and automation, 2007 (pp. 1349–1356). IEEE. Ting, W., Guo-zheng, Y., Bang-hua, Y., & Hong, S. (2008). EEG feature extraction based on wavelet packet decomposition for brain computer interface. Measurement, 41(6), 618–625. Tsui, C. S. L., & Gan, J. Q. (2007). Asynchronous BCI control of a robot simulator with supervised online training. In Intelligent data engineering and automated learning-IDEAL 2007 (pp. 125–134). Berlin Heidelberg: Springer. Tsui, C. S. L., Gan, J. Q., & Roberts, S. J. (2009). A self-paced brain–computer interface for controlling a robot simulator: An online event labelling paradigm and an extended Kalman filter based algorithm for online training. Medical & Biological Engineering & Computing, 47(3), 257–265. Tuna, G., Nefzi, B., & Conte, G. (2014). Unmanned aerial vehicle-aided communications system for disaster recovery. Journal of Network and Computer Applications, 41, 27–36. Varela, G., Caamaño, P., Orjales, F., Deibe, Á., López-Peña, F., & Duro, R. J. (2014). Autonomous UAV based search operations using constrained sampling evolutionary algorithms. Neurocomputing, 132, 54–67. Velasco-Álvarez, F., Ron-Angevin, R., da Silva-Sauer, L., & Sancha-Ros, S. (2013). Audio-cued motor imagery-based brain–computer interface: Navigation through virtual and real environments. Neurocomputing, 121, 89–98. Wang, Y. K., Chen, S. A., & Li, C. T. (2014). An EEG-based brain-computer interface for dual task driving detection. Neurocomputing, 129, 85–93. Wendel, J., Meister, O., Schlaile, C., & Trommer, G. F. (2006). An integrated GPS/ MEMS-IMU navigation system for an autonomous helicopter. Aerospace Science and Technology, 10(6), 527–533. Wu, W., Gao, X., Hong, B., & Gao, S. (2008). Classifying single-trial EEG during motor imagery by iterative spatio-spectral patterns learning (ISSPL). IEEE Transactions on Biomedical Engineering, 55(6), 1733–1743. Wu, Z., Lai, Y., Xia, Y., Wu, D., & Yao, D. (2008). Stimulator selection in SSVEP-based BCI. Medical Engineering & Physics, 30(8), 1079–1088. Zavala-Fernández, H., Orglmeister, R., Trahms, L., & Sander, T. H. (2012). Identification enhancement of auditory evoked potentials in EEG by epoch concatenation and temporal decorrelation. Computer Methods and Programs in Biomedicine, 108(3), 1097–1105. Zhang, C., Wang, H., Wang, H., & Wu, M. H. (2013). EEG-based expert system using complexity measures and probability density function control in alpha subband. Integrated Computer-Aided Engineering, 20(4), 391–405.