Toward automatic detection of brain responses to emotional music through analysis of EEG effective connectivity

Computers in Human Behavior 58 (2016) 231e239

Contents lists available at ScienceDirect

Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh

Full length article

Toward automatic detection of brain responses to emotional music through analysis of EEG effective connectivity Hossein Shahabi b, Sahar Moghimi a, b, c, * a

Rayan Center for Neuroscience and Behavior, Ferdowsi University of Mashhad, Mashhad, Iran Electircal Engineering Department, Ferdowsi University of Mashhad, Mashhad, Iran c Center of Excellence on Soft Computing and Intelligent Information Processing, Ferdowsi University of Mashhad, Mashhad, Iran b

a r t i c l e i n f o

a b s t r a c t

Article history: Received 8 January 2015 Received in revised form 3 January 2016 Accepted 5 January 2016 Available online xxx

The purpose of this study was to investigate the effective brain networks associated with joyful, melancholic, and neutral music. Connectivity patterns among EEG electrodes in different frequency bands were extracted by multivariate autoregressive modeling while 19 nonmusicians listened to selected classical and Iranian musical excerpts. Musical selections were categorized according to the participants' average self-assessment results. Connectivity matrices were analyzed to identify distinct variations in the connectivity indices related to the categorized excerpts. We studied the correlation of inter-/intra-regional connectivity patterns with the self-reported evaluations of the musical selections. The perceived valence was positively correlated with the frontal inter-hemispheric flow, but negatively correlated with the parietal bilateral connectivity. Using the connectivity indices between different cortical areas and a support vector machine, we sought to distinguish trials in terms of the self-reported valence of perceived emotions and the familiarity of the musical genres. For 16 participants, the average classification accuracies in discriminating joyful from neutral, joyful from melancholic and familiar from unfamiliar trials were 93.7% ± 1.06%, 80.43% ± 1.74%, and 83.04% ± 1.47, respectively. Integration of different cortical areas is required for music perception and emotional processing. Thus, by studying the connectivity of brain regions, we may be able to develop a noninvasive assessment tool for investigating musical emotions. © 2016 Elsevier Ltd. All rights reserved.

Keywords: Brain connectivity Electroencephalography (EEG) Directed transfer function Machine learning Multivariate autoregressive modeling Musical emotions

1. Introduction Affective adaptation of the brain-machine interface to the user's mood can have various benefits for society. Many researchers have employed biosignals to develop affective brain-computer interfaces. “Technology [affective computing] can also be improved if it has an intelligent ability to respond to emotion, and technology can be improved by virtue of incorporating principles of emotion learned from biological systems” (Picard, 2010). Accordingly, substantial research has been dedicated to investigating the neural correlates of emotion. The task of listening to music involves various psychological processes, such as perception and multimodal integration, attention, learning and memory, syntactic processing, and the

* Corresponding author. Department of Electrical Engineering, Ferdowsi University of Mashhad, Mashhad, 9177948974, Iran. E-mail address: [email protected] (S. Moghimi). http://dx.doi.org/10.1016/j.chb.2016.01.005 0747-5632/© 2016 Elsevier Ltd. All rights reserved.

processing of meaningful information, action, emotion, and social cognition (Koelsch, 2012). Thus, music is a very powerful tool for investigating functions of the human brain. Music can modulate activities in the limbic and paralimbic brain structures (Koelsch, 2010), and it is well suited for inducing positive or negative emotions (Peretz, Gagnon, & Bouchard, 1998) (although care must be taken to control influential factors, e.g. considering the underlying mechanism of emotion induction (Juslin & V€ astfj€ all, 2008)). Researchers have used different modalities for studying the neural correlates of emotion. For example, using PET, Blood and Zattore found that when participants listened to intensely pleasurable music, meaningful changes in blood flow occurred in circuits involved in reward, motivation, and emotion (Blood & Zatorre, 2001). Salimpour et al. reported the release of endogenous dopamine when listening to music, as a result of emotional arousal (Salimpoor, Benovoy, Larcher, Dagher, & Zatorre, 2011). In an fMRI study, Koelsch et al. found that listening to pleasant or unpleasant music resulted in the activation or deactivation of limbic/paralimbic circuits (Koelsch, Fritz, Müller, & Friederici, 2006). Sammler

232

H. Shahabi, S. Moghimi / Computers in Human Behavior 58 (2016) 231e239

et al. demonstrated that listening to pleasant music increased the theta power of EEG signals in the frontal midline region (Sammler, Grigutsch, Fritz, & Koelsch, 2007), particularly during the second half of musical excerpts. Also, NIRS signals of the prefrontal cortex were used to discriminate positive from negative emotions evoked by listening to different musical excerpts (Moghimi, Kushki, Power, Guerguerian, & Chau, 2012). Several aspects of EEG (e.g., high temporal resolution, portability, and low cost of data recording) make it a suitable candidate for investigating the neural correlates of cognitive functions. In the past two decades, EEG power spectra in different frequency bands have been studied to elucidate alterations with respect to emotional processing. In 1995, Davidson introduced the asymmetry hypothesis, which proposes that positive and negative emotions are mostly processed in the left and right frontal brain regions, respectively (Davidson, 1995). Similar results were reported by Schmidt & Trainor (2001) for the lateralization of EEG activity in the alpha band due to opposite valences. Altenmüller et al. reported bilateral fronto-temporal activation while subjects listened to musical excerpts from different genres (Altenmüller, Schürmann, Lim, & Parlitz, 2002). Various asymmetry measures, not necessarily in the frontal cortex, have been employed to develop quantitative tools for evaluating emotions elicited by visual (Petrantonakis & Hadjileontiadis, 2011) and musical stimuli (Lin et al., 2010). Studies of emotional processing have examined power modulation in different EEG frequency bands (i.e., theta, alpha, beta, and gamma), and researchers have repeatedly considered spectral power changes in different brain regions to be indicators of musical emotion processingdnamely, alpha power activity in the occipital, parietal, frontal, and temporal regions (Baumgartner, €ncke, 2006); beta power in the right parietal-temporal Esslen, & Ja cortex (Aftanas, Reva, Savotina, & Makhnev, 2006); and gamma power in the right parietal region (Balconi & Lucchiari, 2008). Recently, quantitative tools have been developed for recognizing the emotions or positive feelings of subjects while listening to musical excerpts. Lin et al. (2010) used extracted features based on spectral hemispheric asymmetry, derived from all EEG frequency bands, to classify four different emotions. Hadjidimitriou and Hadjileontiadis (2012) discriminated subjects' preferences toward different musical excerpts, using features extracted from time-frequency analyses of EEG signals. Once they considered the familiarity of listeners with the excerpts, the classification accuracy increased for familiar musics (Hadjidimitriou & Hadjileontiadis, 2013). Most EEG studies on the brain emotional response to music have focused on the spectral power of recorded signals. In recent years, many researchers have performed studies on the brain as a complex system (Bullmore & Sporns, 2009) and investigated brain connectivity (He, Yang, Wilke, & Yuan, 2011). As music perception requires integration of different cortical areas (Miskovic & Schmidt, 2010), researchers have tried to use connectivity analysis rrez et al., 2007; (Bhattacharya & Petsche, 2005; Flores-Gutie Karmonik, Brandt, Fung, Grossman, & Frazier, 2013; Kay, Meng, DiFrancesco, Holland, & Szaflarski, 2012; Koelsch & Skouras, 2014) and network theory (Wu et al., 2012) to examine how different brain regions are coactivated and communicate during music listening. Gamma long-range phase synchrony was recently shown to be significantly higher for musicians compared to nonmusicians while listening to classical music. Higher-order longrange phase synchrony was also observed between anterior delta and posterior gamma oscillations in musicians (Bhattacharya & Petsche, 2005). Moreover, increased functional connectivity in the left hemisphere was reported with pleasant music (Floresrrez et al., 2007). Gutie In the current study, we aim to understand the brain emotional

response to music through connectivity analysis of cortical regions. Toward this endeavor, we use multivariate autoregressive (MVAR) modeling to create effective networks of the correlated neural activity. We show that using the connectivity information opposite emotions (in terms of valence), obtained through subjective assessment, can be discriminated. In order to induce emotions we use previously utilized musical excerpts as well as Iranian musical selections. The extracted features demonstrate how the interregional connectivity indices vary from listening to joyful musical selections to melancholic ones and that some of the features are correlated with the behavioral results. 2. Material and methods 2.1. Protocol and recordings We recruited 19 volunteers for this study. Participants were nonmusicians (11 females; age range: 21 ± 3 years) with similar educational backgrounds (undergraduate or MSc students). All subjects were right-handed, had normal hearing, and had no history of any neurological disease. They reported normal nocturnal sleep patterns (7e9 h starting from 10pm to 12am) for the week before the experiment. They had not used caffeine, nicotine, or energy drinks and had not performed excessive exercise in the 24 h before the experiment. None of the subjects had received formal musical training. Stimuli were selected according to the literature to cover both positive and negative musical emotions (Peretz et al., 1998). They comprised the first 60 s of the following six classical music compositions: Mozart, Eine kleine nacht music (Allegro and Rondo Allegro); Vivaldi, Le Quattro stagioni (La primavera); Chopin, Nocturne Op. 9, No. 2; Rodrigo, Concerto de Aranjuez (Adagio); and Grieg, Peer Gynt's Suite no. 2 (Solveigs song). In addition, two Iranian musical excerpts were selected: Shajarian, Rang Shahr Ashoub, and Alizadeh, Neynava. Altogether, these eight pieces were used to induce emotions with different levels of valence and arousal. To ensure that all participants were equally familiar with the pieces, the subjects were presented with the stimuli two to four days before the experiment. They listened to each piece for one time in a calm environment. After listening to the pieces, they were asked whether any was familiar to them. Before the experiment, participants filled out a form to selfassess their levels of comfort, anxiety, and alertness, as well as the presence of any pain. Participants sat in a comfortable chair in dim light. Stimuli were presented via suitable headphones at a comfortable volume. Participants were instructed to listen carefully to the compositions and report their elicited emotion. After listening to each musical excerpt, participants expressed their emotional appraisals by moving the cursor on a two-dimensional arousal-valence (A-V) plane. FEELTRACE (Cowie et al., 2000) was used to record the ratings of A-V. EEG recordings of the scalp were performed with the Emotiv EPOC 14-channel EEG wireless recording headset (Emotiv Systems, Inc., San Francisco, CA), as illustrated in Fig. 1a. This device obtains EEG data with a sampling frequency of 128. Electrodes were placed according to the 10e20 system (Electro Cap International Inc., Eaton, USA), including placement at positions AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4. During the preparation steps, we were careful to place the headset correctly on each subject's head. The reference of the device was changed from the first placement to the mastoids. The recording device is portable and easy to wear, making it a good candidate for BCI applications (Hadjidimitriou & Hadjileontiadis, 2012; Qiang & Sourina, 2013). During each recording session, the participant completed eight consecutive

H. Shahabi, S. Moghimi / Computers in Human Behavior 58 (2016) 231e239

Aðf ÞXðf Þ ¼ Eðf Þ

233

(3)

in which

Aðf Þ ¼

p X

Ar ej2prf Dt

r¼0

where f is frequency and Dt represents sampling interval. Let H(f) be the inverse of A(f). Then, the normalized measure for evaluating the connectivity between each pair of electrodes can be obtained by

jhmn ðf Þj2 gmn ðf Þ ¼ P jhmn ðf Þj2

(4)

n

Fig. 1. (a) Positioning of the electrodes for the Emotiv device according to the 10e20 system. (b) Trial sequence used in the experimental protocol. A beep sound was played between the blocks.

random trials, each consisting of two 5-s intervals of silence, two 5s intervals of white noise, and one 60-s stimulus. One trial sequence is depicted in Fig. 1b. Ratings were recorded, according to the cursor location on the A-V plane, after the musical except was played. 2.2. Directed transfer function Many effective connectivity measures are based on MVAR modeling of time series data (e.g. Granger causality and partial directed coherence (Greenblatt, Pflieger, & Ossadtchi, 2012)). We used the directed transfer function (DTF) technique to investigate causal interactions between EEG time series and to produce features for classifying musical emotions, as described previously for many cognitive processes (Babiloni et al., 2005, 2006). We provide a brief overview of the DTF technique, however detailed information can be found in previous publications (Kaminski & Blinowska, 1991;  ski, Ding, Truccolo, & Bressler, 2001). Throughout the Kamin manuscript, lowercase italic, lowercase bold, and uppercase bold letters are used for presenting scalars, vectors and matrices, respectively. Let x(t) be the sample of signals in the L time series (EEG signals at L locations) at time t, x(t) ¼ [x1(t) x2(t) … xL(t)]T. We can estimate the output of each time series at time t by using p previous samples of all time series

xðtÞ ¼ 

p X

Ar xðt  rÞ þ eðtÞ

(1)

r¼1

where e(t) ¼ N(0,s2) is the Gaussian white noise, and p is the model order. Hence, p X

Ar xðt  rÞ ¼ eðtÞ

(2)

where hmn is an element of the transfer matrix H(f). gmn (f) describes transmission from the nth input to the mth output and, if significantly different from zero, demonstrates that the information in channel n can be used to estimate channel m. Using the above equation, we constructed the adjacency matrices of brain effective networks in four frequency bands: theta (4e8 Hz), alpha (8e13 Hz), beta (13e30 Hz), and gamma (30e42 Hz). 2.3. Data processing For every participant and stimulus, EEG data and individual assessments of the emotional content corresponding to each 1-min musical excerpt were recorded. The data corresponding to three participants were discarded due to abnormal patterns in EEG signals. A band pass 2e42 Hz filter was employed to remove low and high frequency artifacts from the EEG signals, according to related publications (Qiang & Sourina, 2013). The first and last five seconds of EEG signals were discarded, and values >70 mV were removed. An additional manual check was performed to remove any remaining artifacts through visual inspection. Considering the nonstationary nature of EEG data, MVAR modeling was performed in 2-s non-overlapping windows. In each window, data from all channels were normalized to zero mean and unit standard deviation (SD) (Kaminski & Blinowska, 1991). A conventional least-squares method (Ljung, 1998) was used to estimate the elements of the Ar matrices, and p ¼ 6 was selected according to the Akaike information criterion (Akaike, 1974). In each frequency band, the connectivity matrices were calculated and averaged over all windows. Hence, we obtained four matrices for every participant and musical excerpt. We used surrogate data to adjust a threshold for distinguishing significant connectivity indices. Surrogate data were generated by manipulating the EEG signals, with the aim of destroying casual relations between channels. Manipulation was performed by dislocating the blocks of each signal (Babiloni et al., 2005). Using the aforementioned technique, we calculated frequency dependent thresholds, which we used to preserve the significant connectivity values and to change the others to zero. 2.4. Feature extraction and classification

r¼0

where A0 is an L  L identity matrix, and Ar is the L  L MVAR coefficient matrix. Our goal was to identify the elements of matrix A in (2), by using the signal samples at different time steps. The Ar matrices represent weights corresponding to different signal samples for constructing the output signals. By computing the Fourier transform of (2), we obtain

Along with investigating the inter-electrode connectivity indices, we tried to define more general inter-regional features from the electrode ensembles. Six features were extracted from the connectivity matrices to quantify the interactions between different regions: the frontal out-degree (FOD), frontal intrahemispheric (FIAH), frontal inter-hemispheric (FIEH), frontal-toparietal (FP), parietal-to-frontal (PF), and inter/intra-parietal (IP)

234

H. Shahabi, S. Moghimi / Computers in Human Behavior 58 (2016) 231e239

indices (Table 1). The six features were chosen based on visual inspection of the obtained connectivity matrices and their variations for different musical selections. Fig. 2 illustrates how the aforementioned features were calculated over the elements of connectivity matrices. Each feature was defined in all four frequency bands and calculated by averaging over the connectivity indices between the listed source and sink electrodes (Table 1) in the corresponding frequency band. This process was repeated for the networks obtained for every participant, listening to all musical selections. Thus, for each subject, we obtained 24 features for each musical excerpt. We employed an SVM classifier to identify the category of musical selections (in terms of valence) to which the participant was exposed. More precisely, using the aforementioned six features in three separate attempts, we tried to distinguish whether the stimulus belonged to each of the categories defined based on the behavioral results. SVM is a hyper plane classifier,SðxÞ ¼ 〈w; b〉 þ b, designed to solve two-class problems. It has been successfully applied for EEG classification tasks (Garrett, Peterson, Anderson, & Thaut, 2003; cuyer, Lamarche, & Arnaldi, 2007). For nonLotte, Congedo, Le linearly separable data, a nonlinear mapping function (e.g., polynomial function) is employed to map the features into a higher dimensional feature space, where the hyper plane classifier can be applied for discrimination purposes (Shen & Ji, 2009). By using kernel transfer functions, the nonlinear SVM problem becomes (Cristianini & Shawe-Taylor, 2000):

SðxÞ ¼ sgn

X

! ak yk kðxk ; xÞ þ b

(5)

k

where xk2RN is the support vector trained by SVM, and k(xk,x) is a kernel function. Three SVM kernel transfer functions (i.e., linear, polynomial, and Gaussian) were tested for transforming the feature space. Classifiers were trained with different feature subsets. The performance of the SVMs, trained with the features and kernel functions, was evaluated by 8-fold cross validation. All processing steps for the development of connectivity matrices, feature extraction, and classification were performed in MATLAB 2013b (MathWorks, Inc., Natick, MA, USA). 3. Results 3.1. Behavioral performance We recorded the valence and arousal values reported by participants on the FEELTRACE screen after listening to 60 s of each musical excerpt (Table 2). Excerpts were categorized for further processing according to the average reported values. We classified Musical Selection (MS) 1, MS4, and MS6 as “joyful”, MS5 and MS2 as “melancholic,” and MS3 and MS7 as “neutral”. The reported valence values (both positive and negative) were higher for the selected Iranian pieces. This observation motivated us to differentiate

Fig. 2. Features extracted from connectivity matrices. The names of the features are listed in Table 1. Each feature was calculated by averaging over the elements of adjacency matrix included in the defined box.

excerpts based on the subjects' familiarity with the musical genres (not MSs). MS1 and MS5 were classified as unfamiliar, whereas MS2 and MS4 (Iranian excerpts) were classified as familiar. The unfamiliar selections were the classical excerpts with the highest positive and negative mean valence values. 3.2. Connectivity analysis We compared the effective networks of neural activity that corresponded to the musical excerpts, to investigate the effect of music from different emotional categories on the network structure. For each excerpt, we calculated the average of the connectivity matrices among all participants. To emphasize variations from the neutral state, we subtracted the networks corresponding to MS7 from those of MS1 and MS5. Similarly, to explore variations from melancholic to joyful excerpts, we calculated two subtractive networks: MS1eMS5 (classical excerpts) and MS4eMS2 (Iranian excerpts). Subtracting the networks reduced the effects of volume conduction, thus improving the overall results. Fig. 3 provides a topographic representation of the aforementioned subtractive networks corresponding to effective connectivity (averaged over all participants) in the four frequency bands. Among the L  L  L connectivity coefficients, only those at or above the 95th percentile are reported. Some indicative features can be mentioned. A similar pattern in connectivity variations was observed among the frontal and parietal regions in all frequency bands. Specifically, the frontal-to-parietal connectivity in the alpha, beta, and gamma bands increased from MS7 to MS1. This increase

Table 1 Features extracted from connectivity matrices for investigating inter-regional variations while listening to different musical excerpts. No.

Feature name

Source electrodes

Sink electrodes

1 2

Frontal out-degree index (FOD) Frontal intra-hemispheric index (FIAH)

3

Frontal inter-hemispheric index (FIEH)

4 5 6

Frontal-to-parietal index (FP) Parietal-to-frontal index (PF) Inter/intra-parietal index (IP)

AF3, AF4, F3, F4, F7, F8, FC5, FC6 AF3, F3, F7, FC5 AF4, F4, F8, FC6 AF3, F3, F7, FC5 AF4, F4, F8, FC6 AF3, AF4, F3, F4, F7, F8, FC5, FC6 P7, P8, O1, O2 P7, P8, O1, O2

All electrodes AF3, F3, F7, FC5 AF4, F4, F8, FC6 AF4, F4, F8, FC6 AF3, F3, F7, FC5 P7, P8, O1, O2 AF3, AF4, F3, F4, F7, F8, FC5, FC6 P7, P8, O1, O2

H. Shahabi, S. Moghimi / Computers in Human Behavior 58 (2016) 231e239

235

Table 2 Mean valence and arousal values reported by participants for each musical excerpt. No.

Musical selection

Valance (mean ± SD)

1 2 3 4 5 6 7 8

Eine kleine nacht music (Allegro) Neynava Nocturne Op.9, No. 2 Rang Shahr Ashoub Concerto de Aranjuez (Adagio) Eine kleine nacht music (Rondo Allegro) Peer Gynt's Suite no. 2 (Solveigs song) Le Quattro stagioni (La primavera)

0.38 0.46 0.15 0.5 0.21 0.3 0.04 0.33

was more dominant in the alpha and beta bands. The direction of this flow was reversed when the network corresponding to MS7 was subtracted from that of MS5. Increased effective connectivity was observed in the left hemisphere for MS1eMS7 and MS1eMS5. Another noticeable variation from MS7 to MS1 and from MS4 to MS2 was an increased inter-hemispheric frontal connectivity, which mostly manifested in the beta and gamma bands. The network corresponding to MS5eMS7 elicited a right-to-left hemispheric flow in the alpha, beta, and gamma bands. Since the earlier topographic results were only qualitative in nature, we tried to find connectivity indices (graph links) among electrodes in different frequency bands that significantly differed from neutral to joyful or melancholic excerpts. The KruskaleWallis nonparametric test was used to analyze the differences. Table 3 illustrates the source and sink electrodes, along with the frequency bands in which the connectivity value elicited a significant difference (p < 0.05) between musical excerpts. Due to the observed behavioral results corresponding to MS2 and MS4, information for MS4eMS2 is also reported. In almost every case, meaningful differences were only observed in theta and gamma bands. Observing the obtained networks for different musical excerpts and participants, we hypothesized that the reported arousal and valence vectors correlated with the connectivity indices in different regions. To address this hypothesis, we investigated how the variations in calculated features correlated with differences in valence and arousal from MS1 to MS7 and from MS5 to MS7. Table 4 shows the significant correlation values. In both the theta and gamma bands, PF and IP showed significant negative correlations with valence when comparing MS5 with MS7. No significant correlation was observed between the extracted features in different frequency bands and the reported valence when comparing MS1 with MS7. However, in the alpha band, FP, IP, and FOD showed significant correlations with variations in arousal. When comparing MS5 with MS1, FIEH, FOD, and FP in the alpha band showed positive correlations with reported valence values (Table 4 and Fig. 4). Thus, participants with higher FOD, FIEH, and FP in the alpha band expressed more positive emotions while listening to MS1. The direction of connections was reversed while listening to MS5, and the amplitude had a positive correlation with the intensity of the expressed negative emotion. In the beta band, IP and PF were negatively correlated with valence (Table 4).

3.3. Classification After observing the variations in the effective networks from joyful to neutral and melancholic musical selections, as well as the preference of participants for the Iranian musical genre, we tried to classify the obtained features in terms of valence and familiarity. In three separate attempts, classifiers were designed to discriminate musical selections with different contents: namely, neutral (MS3 and MS7) from joyful (MS1 and MS6), melancholic (MS5 and MS2)

± ± ± ± ± ± ± ±

0.16 0.16 0.29 0.15 0.3 0.19 0.35 0.14

Arousal (mean ± SD) 0.28 0.25 0.33 0.48 0.31 0.45 0.22 1.

± ± ± ± ± ± ± ±

0.18 0.25 0.12 0.13 0.24 0.25 0.23 0.31

from joyful (MS1 and MS4), and familiar (MS2 and MS4) from unfamiliar (MS1 and MS5). All six features in the four frequency bands (24 features in total) were examined. Table 5 displays the classification results obtained by using the features in different frequency bands (one feature in every attempt). The best results for all three attempts were obtained when employing FOD, FIAH, and FIEH in the beta and gamma bands. The classification accuracy was improved by employing the brute-force feature-selection technique and testing different kernels with the six features. Fig. 5 shows the variations in classification accuracy when different number of features was allowed during the selection process. For calculating the accuracy values the outliers in 120 attempts were removed. For classifying joyful and neutral excerpts, the best result (93.7% ± 1.06%) was obtained when we used a polynomial kernel of order 2. The selected features were FIAH in gamma, FP in beta, and FOD in theta bands. Joyful excerpts were discriminated from melancholic ones with an accuracy of 80.43% ± 1.74% when we used a Gaussian kernel. The selected features were FP in beta, and FIAH and FIEH in gamma bands. A classification accuracy of 83.04% ± 1.47% was obtained for classifying familiar and unfamiliar musical selections by using a Gaussian kernel. The selected features were FOD in beta and gamma, and FIEH and FIAH in gamma.

4. Discussion The affective processing of complex stimuli involves the integration of various brain sites. We analyzed the effective brain networks associated with musical emotions, to detect the brain response to stimuli with different emotional contents. Recently, researchers have successfully exploited connectivity patterns for recognizing neural states and responses (Billinger, Brunner, & Müller-Putz, 2013). In this study, we performed classification in three different attempts and trained an SVM classifier to solve a two-class problem for joyful vs. neutral, joyful vs. melancholic, and familiar vs. unfamiliar trials. Employing 8-fold cross validation, we classified the aforementioned tasks with average accuracies of 93.7% ± 1.06%, 80.43% þ 1.74%, and 83.04% ± 1.47%, respectively. Our findings indicate that the emotional content of music induces different connectivity patterns, mostly in the frontal and frontoparietal regions, which are detectable through MVAR and DTF analyses and can be utilized for classification purposes. In an EEG-based emotion recognition task the best classification performance was obtained using differential asymmetry defined over pairs of electrodes mostly in the frontal lobe. Using SVM the highest classification results were for the asymmetry features in beta, alpha and theta bands (Lin et al., 2010), which is in agreement with our classification results using FIAH, FIEH and FOD. Due to our definition of classes, we cannot compare our classification accuracies with those of previously published studies (Hadjidimitriou & Hadjileontiadis, 2012; Hadjidimitriou and Hadjileontiadis, 2013; Wang & Sourina, 2013). Nonetheless, our goal was to

236

H. Shahabi, S. Moghimi / Computers in Human Behavior 58 (2016) 231e239

Fig. 3. Effective brain networks (averaged over all participants) in different frequency bands, calculated by subtracting the networks corresponding to MS1 (joyful), MS5 (melancholic), MS7 (neutral), or MS4 and MS2 (Iranian excerpts). Only the connections at or above the 95th percentile are reported.

demonstrate the feasibility of our connectivity-based approach using a network defined by a small number of electrodes and an affordable recording device. Although variations in interelectrode connectivity were observed in different frequency bands (Table 3), the independency of the connectivity-related features from electrode location and musical excerpt demanded more general features. Therefore, we defined features of inter-regional connectivity (Table 1), through a

visual inspection of the network variations related to the emotional content of selected pieces. The previous reports on the asymmetry of neural correlates during music appreciation were based on spectral analysis of EEG signals (Altenmüller et al., 2002; Davidson, 1995; Schmidt & Trainor, 2001) and cannot be compared to the results obtained in this study. However, the observed left lateralization of the effective connectivity in the alpha band during joyful music listening is

H. Shahabi, S. Moghimi / Computers in Human Behavior 58 (2016) 231e239

237

Table 3 Significant variations in inter-electrode connectivity in different frequency bands using the KruskaleWallis test (p < 0.05). C ¼ 1 means that the connectivity value was higher for the first excerpt and vice versa. Source MS1-MS7 O1 FC5 FC5 O1 FC5 P8 F4

Sink F3 T7 T80 FC5 AF3 AF4 P80

Freq. band Theta Theta Theta Theta Theta Theta Gamma

p

C

0.008 0.015 0.027 0.045 0.025 0.037 0.023

1 1 1 1 1 1 1

Source MS5-MS7 O1 O1 O1 FC5 O1 O1 FC5 F4 F4 F4

Sink F3 F7 FC T7 AF3 FC6 F4 P8 O2 FC6

Freq. band Theta Theta Theta Theta Theta Theta Theta Gamma Gamma Gamma

p

C

0.002 0.024 0.034 0.022 0.048 0.025 0.037 0.015 0.041 0.023

1 1 1 1 1 1 1 1 1 1

Source

Sink

Freq. band

p

C

MS4-MS2 O2 AF3 F3 F3 F3 P8 F3 P8 AF4 F3 F3 F3 AF3

AF3 FC5 P8 O1 O1 O1 F8 O2 T8 FC6 AF3 P8 F8

Theta Theta Theta Theta Alpha Beta Gamma Gamma Gamma Gamma Gamma Gamma Gamma

0.043 0.041 0.026 0.045 0.007 0.017 0.019 0.025 0.026 0.011 0.050 0.038 0.026

1 1 1 1 1 1 1 1 1 1 1 1 1

Table 4 Correlation of extracted features with valence and arousal in different frequency bands for different musical excerpts. MS

MS

Frequency band

Feature type

Arousal (A)/valance (V)

p-value

r

MS1 MS1 MS1

MS7 MS7 MS7

Alpha Alpha Alpha

FOD FP IP

A A A

0.039 0.014 0.018

0.519 0.598 þ0.579

MS5 MS5 MS5 MS5 MS5 MS5

MS7 MS7 MS7 MS7 MS7 MS7

Theta Theta Alpha Beta Gamma Gamma

PF IP FP IP PF IP

V V V V V V

0.034 0.004 0.031 0.049 0.012 0.001

0.531 0.669 þ0.538 0.498 0.609 0.716

MS1 MS1 MS1 MS1 MS1

MS5 MS5 MS5 MS5 MS5

Alpha Alpha Alpha Beta Beta

FOD FIEH FP PF IP

V V V V V

0.000 0.001 0.004 0.017 0.010

þ0.764 þ0.730 þ0.665 0.583 0.618

Fig. 4. Difference in FIEH (Table 1) connectivity for networks corresponding to MS1 (joyful)eMS5 (melancholic) versus valence in the alpha frequency band. Correlation analysis statistically verified significant with p < 0.05.

consistent with previous results of coherence analyses (Floresrrez et al., 2007). Gutie Increased fronto-parietal connectivity was detected from the networks of MS7 to MS1 in the alpha, beta, and gamma bands. Also,

the FP index was selected through brute force for discriminating joyful from both neutral and melancholic excerpts (Fig. 5). This result could indicate emotionally modulated attention (Vuilleumier, 2005), as it is believed that participants are more mentally engaged while listening to joyful stimuli compared to neutral ones. Exposure to joyful stimuli, compared to neutral or melancholic ones, elicited increased frontal inter/intrahemispheric connectivity in different frequency bands. Authors have indicated the role of the frontal cortex in the limbic-frontal circuitry during emotion regulation (Banks, Eddy, Angstadt, Nathan, & Phan, 2007; Davidson, 2004). The frontal interhemispheric connectivity also had a positive correlation with perceived valence (Table 4). This relationship between behavioral and neural results can be further explored by considering gender differentiation and neural disorders. Employing FIAH and FIEH in the beta and gamma bands provided the best results (Table 5) for discriminating joyful excerpts from neutral or melancholic ones. This result may motivate the development of machine learning systems for automatic emotion detection based solely on the frontal brain connectivity. Bhattacharya and Petsche (2005) reported that only participants with formal musical training retrieved extensive repertoire of musical pattern, which was accompanied by an enhanced gammaband phase synchrony, while long range delta-band synchrony was significantly higher in nonmusicians. However, in this study gamma band features were successfully used for emotion recognition in nonmusicians. We did not have any EEG recording from musician, and thus, could not compare the dynamic course of

65.43 (1.531) 52.39 (1.112) 50.43 (0.975) 52.83 (1.656)

g

b

a

The bold values represent the highest classification accuracy values.

PF FP

53.04 (1.723) 56.52 (1.693) 72.61 (1.511) 68.91 (1.707) 65.65 (1.463) 65 (2.019) 74.13 (1.926) 66.52 (1.687) 51.3 (1.476) 58.26 (1.871) 65.87 (1.459) 75.65 (1.588)

FIEH FIAH

63.48 (1.390) 57.61 (1.386) 71.96 (1.548) 64.35 (1.606) 63.04 (1.985) 64.35 (1.606) 53.48 (1.659) 51.96 (1.202) 60.65 (1.312) 58.26 (1.224) 60.65 (1.514) 59.78 (1.142)

FOD IP PF FP

59.35 (1.988) 59.13 (1.985) 68.04 (1.530) 61.3 (1.611) 51.96 (1.608) 60.65 (1.930) 68.48 (1.483) 78.48 (1.601)

FIEH FIAH

51.52 (1.781) 59.57 (1.434) 68.91 (1.967) 71.3 (1.807) 55.22 (1.964) 59.13 (1.727) 68.7 (1.948) 69.78 (1.964)

FOD

Fig. 5. Classification accuracy using different numbers of features. Three different attempts were made to differential familiar from unfamiliar (red), joyful from melancholic (green) and joyful from neutral excerpts. For calculating the accuracy values the outliers in 120 attempts were removed. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

62.83 (1.672) 70.65 (1.754) 55.22 (1.000) 50.22 (1.559) 65.87 (1.459) 58.91 (1.549) 55.65 (0.979) 59.57 (1.561) 59.57 (1.942) 65.65 (1.647) 63.04 (1.393) 64.13 (1.855)

IP PF FP FIEH

55 (1.353) 61.96 (1.644) 71.09 (1.531) 73.26 (1.596) 52.39 (1.903) 55.87 (1.491) 71.09 (1.499) 57.39 (1.603)

FIAH FOD

59.57 (1.531) 62.39 (1.862) 71.74 (1.560) 71.52 (1.540)

q

(C) (B) (A)

Table 5 (A) Joyful vs. melancholic, (B) joyful vs. neutral and (C) familiar vs. unfamiliar musical selections. The values in the parentheses represent standard error.

46.96 (1.278) 62.39 (1.756) 54.57 (1.650) 60.43 (1.272)

H. Shahabi, S. Moghimi / Computers in Human Behavior 58 (2016) 231e239

IP

238

signals during music perception and appreciation between musicians and nonmusicians. To eliminate transition effects, we removed the first and last 5s of the EEG signals and the effective networks were estimated using 50-s trials. A recent study reported a significant increase in the brain electrical activity (mostly in the frontal regions) toward the end of pleasant musical pieces, elucidating the temporal dynamics of emotional processing (Sammler et al., 2007). Since we are dealing with stimuli that unfold over time, our results can be improved by considering the time-varying effective networks (Milde et al., 2010; Wilke, Lei, & Bin, 2008) and calculating the time-varying features. All of our participants were nonmusicians. The experimental design did not take into consideration their emotional intelligence (EI) or level of sensitivity to emotional stimuli. Petrides and Furnham demonstrated that participants with high EI trait show greater sensitivity to mood-induction procedures compared to their low EI trait counterparts (Petrides & Furnham, 2003). Our behavioral results also demonstrated participants' preference for the joyful Iranian musical pieces. Grouping a large number of participants according to their preferences toward specific musical genres and their EI, and then processing the neural responses associated with different groups separately might reveal more specific connectivity patterns and improve classification accuracy. 5. Conclusion Using MVAR and DTF analyses, we estimated effective brain networks while participants listened attentively to different musical excerpts. These networks were exploited for the automatic detection of brain response to stimuli with different emotional contents. When participants were exposed to joyful pieces, connectivity was increased in the frontal and frontal-parietal regions. The behavioral results (in terms of perceived valence) showed a positive correlation with frontal inter-hemispheric connectivity and a negative correlation with frontal out-degree connectivity. Employing the features calculated from connectivity matrices, we classified joyful vs. neutral, joyful vs. melancholic, and familiar vs. unfamiliar trials with accuracies of 93.7% ± 1.06%, 80.43% ± 1.74%,

H. Shahabi, S. Moghimi / Computers in Human Behavior 58 (2016) 231e239

and 83.04% ± 1.47, respectively. Because the integration of different cortical areas is required for music perception and emotional processing, by studying the connectivity of brain regions, we may be able to develop a noninvasive assessment tool for the automatic detection of musical emotions. Acknowledgments The work described in this paper was supported by a grant from the research office of Ferdowsi University of Mashhad, Mashhad, Iran. We thank Ms. Fatemeh Hasanzadeh Gharecheshmeh for assistance with data recording. We also thank all the subjects who helped us by voluntarily participating in our study. References Aftanas, L., Reva, N., Savotina, L., & Makhnev, V. (2006). Neurophysiological correlates of induced discrete emotions in humans: an individually oriented analysis. Neuroscience and Behavioral Physiology,, 36, 119e130. Akaike, H. (1974). A new look at the statistical model identification. Automatic Control, IEEE Transactions on, 19, 716e723. Altenmüller, E., Schürmann, K., Lim, V. K., & Parlitz, D. (2002). Hits to the left, flops to the right: different emotions during listening to music are reflected in cortical lateralisation patterns. Neuropsychologia, 40, 2242e2256. Babiloni, F., Cincotti, F., Babiloni, C., Carducci, F., Mattia, D., Astolfi, L., et al. (2005). Estimation of the cortical functional connectivity with the multimodal integration of high-resolution EEG and fMRI data by directed transfer function. Neuroimage, 24, 118e131. Babiloni, C., Vecchio, F., Cappa, S., Pasqualetti, P., Rossi, S., Miniussi, C., et al. (2006). Functional frontoparietal connectivity during encoding and retrieval processes follows HERA model: a high-resolution study. Brain Research Bulletin, 68, 203e212. Balconi, M., & Lucchiari, C. (2008). Consciousness and arousal effects on emotional face processing as revealed by brain oscillations. A gamma band analysis. International Journal of Psychophysiology, 67, 41e46. Banks, S. J., Eddy, K. T., Angstadt, M., Nathan, P. J., & Phan, K. L. (2007). Amygdalaefrontal connectivity during emotion regulation. Social Cognitive and Affective Neuroscience, 2, 303e312. €ncke, L. (2006). From emotion perception to Baumgartner, T., Esslen, M., & Ja emotion experience: emotions evoked by pictures and classical music. International Journal of Psychophysiology, 60, 34e43. Bhattacharya, J., & Petsche, H. (2005). Phase synchrony analysis of EEG during music perception reveals changes in functional connectivity due to musical expertise. Signal Processing, 85, 2161e2177. Billinger, M., Brunner, C., & Müller-Putz, G. R. (2013). Single-trial connectivity estimation for classification of motor imagery data. Journal of Neural Engineering, 10, 046006. Blood, A. J., & Zatorre, R. J. (2001). Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion. Proceedings of the National Academy of Sciences, 98, 11818e11823. Bullmore, E., & Sporns, O. (2009). Complex brain networks: graph theoretical analysis of structural and functional systems. Nature Reviews Neuroscience, 10, 186e198. €der, M. Cowie, R., Douglas-Cowie, E., Savvidou, S., McMahon, E., Sawey, M., & Schro (2000). 'FEELTRACE': an instrument for recording perceived emotion in real time. In ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion. Cristianini, N., & Shawe-Taylor, J. (2000). An introduction to support vector machines and other kernel-based learning methods. Cambridge University Press. Davidson, R. J. (1995). Cerebral asymmetry, emotion, and affective style. The MIT Press. Davidson, R. J. (2004). What does the prefrontal cortex “do” in affect: perspectives on frontal EEG asymmetry research. Biological Psychology, 67, 219e234.  del rrez, E. O., Díaz, J.-L., Barrios, F. A., Favila-Humara, R., Guevara, M.A., Flores-Gutie Río-Portilla, Y., et al. (2007). Metabolic and electric brain patterns during pleasant and unpleasant emotions induced by music masterpieces. International Journal of Psychophysiology, 65, 69e84. Garrett, D., Peterson, D. A., Anderson, C. W., & Thaut, M. H. (2003). Comparison of linear, nonlinear, and feature selection methods for EEG signal classification. Neural Systems and Rehabilitation Engineering, IEEE Transactions on, 11, 141e144. Greenblatt, R., Pflieger, M., & Ossadtchi, A. (2012). Connectivity measures applied to human brain electrophysiological data. Journal of Neuroscience Methods, 207, 1e16. Hadjidimitriou, S. K., & Hadjileontiadis, L. J. (2012). Toward an EEG-based recognition of music liking using time-frequency analysis. Biomedical Engineering, IEEE Transactions on, 59, 3498e3510. Hadjidimitriou, S. K., & Hadjileontiadis, L. J. (2013). EEG-based classification of music appraisal responses using time-frequency analysis and familiarity

239

ratings. Affective Computing, IEEE Transactions on, 4, 161e172. He, B., Yang, L., Wilke, C., & Yuan, H. (2011). Electrophysiological imaging of brain activity and connectivitydchallenges and opportunities. Biomedical Engineering, IEEE Transactions on, 58, 1918e1931. €stfja €ll, D. (2008). Emotional responses to music: the need to Juslin, P. N., & Va consider underlying mechanisms. Behavioral and Brain Sciences, 31, 559e575. Kaminski, M., & Blinowska, K. (1991). A new method of the description of the information flow in the brain structures. Biological Cybernetics, 65, 203e210.  ski, M., Ding, M., Truccolo, W. A., & Bressler, S. L. (2001). Evaluating causal Kamin relations in neural systems: Granger causality, directed transfer function and statistical assessment of significance. Biological Cybernetics, 85, 145e157. Karmonik, C., Brandt, A. K., Fung, S. H., Grossman, R. G., & Frazier, J. T. (2013). Graph theoretical connectivity analysis of the human brain while listening to music with emotional attachment: feasibility study. In Engineering in Medicine and Biology Society (EMBC), 2013 35th Annual International Conference of the IEEE (pp. 6526e6529). Kay, B. P., Meng, X., DiFrancesco, M. W., Holland, S. K., & Szaflarski, J. P. (2012). Moderating effects of music on resting state networks. Brain Research, 1447, 53e64. Koelsch, S. (2010). Towards a neural basis of music-evoked emotions. Trends in Cognitive Sciences, 14, 131e137. Koelsch, S. (2012). Brain and music. John Wiley & Sons. Koelsch, S., Fritz, T., Müller, K., & Friederici, A. D. (2006). Investigating emotion with music: an fMRI study. Human Brain Mapping, 27, 239e250. Koelsch, S., & Skouras, S. (2014). Functional centrality of amygdala, striatum and hypothalamus in a “small-world” network underlying joy: an fMRI study with music. Human Brain Mapping, 35.7, 3485e3498. Lin, Y.-P., Wang, C.-H., Jung, T.-P., Wu, T.-L., Jeng, S.-K., Duann, J.-R., et al. (2010). EEGbased emotion recognition in music listening. Biomedical Engineering, IEEE Transactions on, 57, 1798e1806. Ljung, L. (1998). System identification. Springer. cuyer, A., Lamarche, F., & Arnaldi, B. (2007). A review of Lotte, F., Congedo, M., Le classification algorithms for EEG-based brainecomputer interfaces. Journal of Neural Engineering, 4. Milde, T., Leistritz, L., Astolfi, L., Miltner, W. H., Weiss, T., Babiloni, F., et al. (2010). A new Kalman filter approach for the estimation of high-dimensional timevariant multivariate AR models and its application in analysis of laser-evoked brain potentials. Neuroimage, 50, 960e969. Miskovic, V., & Schmidt, L. A. (2010). Cross-regional cortical synchronization during affective image viewing. Brain Research, 1362, 102e111. Moghimi, S., Kushki, A., Power, S., Guerguerian, A. M., & Chau, T. (2012). Automatic detection of a prefrontal cortical response to emotionally rated music using multi-channel near-infrared spectroscopy. Journal of Neural Engineering, 9, 026022. Peretz, I., Gagnon, L., & Bouchard, B. (1998). Music and emotion: perceptual determinants, immediacy, and isolation after brain damage. Cognition, 68, 111e141. Petrantonakis, P. C., & Hadjileontiadis, L. J. (2011). A novel emotion elicitation index using frontal brain asymmetry for enhanced EEG-based emotion recognition. Information Technology in Biomedicine, IEEE Transactions on, 15, 737e746. Petrides, K., & Furnham, A. (2003). Trait emotional intelligence: behavioural validation in two studies of emotion recognition and reactivity to mood induction. European Journal of Personality, 17, 39e57. Picard, R. W. (2010). Affective computing: from laughter to IEEE. Affective Computing, IEEE Transactions on, 1, 11e17. Qiang, W., & Sourina, O. (2013). Real-time mental arithmetic task recognition from EEG signals. Neural Systems and Rehabilitation Engineering, IEEE Transactions on, 21, 225e232. Salimpoor, V. N., Benovoy, M., Larcher, K., Dagher, A., & Zatorre, R. J. (2011). Anatomically distinct dopamine release during anticipation and experience of peak emotion to music. Nature Neuroscience, 14, 257e262. Sammler, D., Grigutsch, M., Fritz, T., & Koelsch, S. (2007). Music and emotion: electrophysiological correlates of the processing of pleasant and unpleasant music. Psychophysiology, 44, 293e304. Schmidt, L. A., & Trainor, L. J. (2001). Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions. Cognition & Emotion, 15, 487e500. Shen, L. L., & Ji, Z. (2009). Gabor wavelet selection and SVM classification for object recognition. Acta Automatica Sinica, 35, 350e355. Vuilleumier, P. (2005). How brains beware: neural mechanisms of emotional attention. Trends in Cognitive Sciences, 9, 585e594. Wang, Q., & Sourina, O. (2013). Real-time mental arithmetic task recognition from EEG signals. IEEE Transactions on Neural Systems and Rehabilitation Engineering: a Publication of the IEEE Engineering in Medicine and Biology Society, 21, 225e232. Wilke, C., Lei, D., & Bin, H. (2008). Estimation of time-varying connectivity patterns through the use of an adaptive directed transfer function. Biomedical Engineering, IEEE Transactions on, 55, 2557e2564. Wu, J., Zhang, J., Liu, C., Liu, D., Ding, X., & Zhou, C. (2012). Graph theoretical analysis of EEG functional connectivity during music perception. Brain Research, 1483, 71e81.