Detecting Concealed Information with Fused Electroencephalography and Functional Near-infrared Spectroscopy

Accepted Manuscript Research Article Detecting concealed information with fused electroencephalography and functional near-infrared spectroscopy Xiaohong Lin, Liyang Sai, Zhen Yuan PII: DOI: Reference:

S0306-4522(18)30469-X https://doi.org/10.1016/j.neuroscience.2018.06.049 NSC 18540

To appear in:

Neuroscience

Received Date: Revised Date: Accepted Date:

29 December 2017 8 June 2018 29 June 2018

Please cite this article as: X. Lin, L. Sai, Z. Yuan, Detecting concealed information with fused electroencephalography and functional near-infrared spectroscopy, Neuroscience (2018), doi: https://doi.org/ 10.1016/j.neuroscience.2018.06.049

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Running head: Combining EEG and fNIRS for lie detection

Detecting

concealed

electroencephalography

information and

functional

with

fused

near-infrared

spectroscopy Xiaohong Lin1, Liyang Sai2, Zhen Yuan1* 1. Bioimaging Core, Faculty of Health Sciences, University of Macau, Taipa, Macau SAR, China 2. Institutes of Psychological Sciences, Hangzhou Normal University, Hangzhou, China *

Corresponding author at: Bioimaging Core, Faculty of Health Sciences, University of Macau, Taipa, Macau SAR, China. Telephone: 00853-88224989. E-mail address: [email protected] (Z. Yuan).

1

Abstract In this study, fused electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) techniques were utilized to examine the relationship between the ERP (event-related potential) component P300 and fNIRS hemodynamic signals for high-accuracy deception detection. During the performance of a modified concealed information test (CIT) task, a series of Chinese names were presented, which served as the target, irrelevant, or the probe stimuli for both the guilty and innocent groups. For participants in the guilty group, the probe stimulus was their individual name, whereas for the innocent group, the probe stimulus was one irrelevant name. In particular, data from concurrent fNIRS and ERP recordings were carefully inspected for participants from the two groups. Interestingly, we discovered that for the guilty group, the probe stimulus elicited significantly higher P300 amplitude at parietal site and also evoked significantly stronger oxyhemoglobin (HbO) concentration changes in the bilateral superior frontal gyrus and bilateral middle frontal gyrus than the irrelevant stimuli. However, this is not the case for the innocent group, in which participants exhibited no significant differences in both ERP and fNIRS measures between the probe and irrelevant stimuli. More importantly, our findings also demonstrated that the combined ERP and fNIRS feature was able to differentiate the guilty and innocent groups with enhanced sensitivity, in which AUC (the area under Receiver Operating Characteristic curve) is 0.94 for deception detection based on the combined indicator, much higher than that based on the ERP component P300 only (0.85) or HbO measure only (0.84). Key words: Concealed information test, P300, deception detection, functional near-infrared spectroscopy

2

Introduction For centuries, deception detection also referred to lie detection, has attracted widespread attention from various fields such as the psychology, economics, forensic science, and neuroscience. To date, tremendous efforts are being devoted to the detection of deception/concealed information more directly by measuring the neural marks of central nervous system. In particular, functional brain mapping techniques such as EEG/event-related potentials (ERPs), fMRI and fNIRS, have exhibited their unbeatable advantages in unveiling the complex neural mechanism of deception due to their non-invasive nature (Langleben et al., 2002, Kozel et al., 2005, Langleben et al., 2005, Hu et al., 2012, Rosenfeld et al., 2013, Sai et al., 2014a, Sai et al., 2014b, Sai et al., 2016). Interestingly, ERPs are recognized as very powerful tools for deception detection, in which the ERP P300 amplitude has exhibited its significant relationship with concealed information or deception (Lykken, 1959). Specifically, P300 is a positive ERP component with peaks around 300-900 ms after trigger onset (Linden, 2005), which can only be elicited by the rare, recognized, meaningful items.P300 has been successfully adopted as a neural indicator for CIT (concealed information test), which is also known as the guilty knowledge test (Lykken, 1959). In particular, the paradigm for a standard CIT contains at least two categories of stimuli presented to the participants: 1) a rarely presented, crime-related item called “probe” stimulus, which is only sensitive to guilty participants; 2) a series of crime-irrelevant items named “irrelevant” stimuli, which are insensitive to both guilty and innocent participants. The rationale underlying the P300based CIT is that for guilty participants who involve in the crime-related information, the rare and meaningful probe stimulus can elicit a larger P300 amplitude than meaningless irrelevant 3

stimuli. By contrast, for innocent participants who have no relationship with the crime-relevant information (e.g. innocent persons who are not aware of the crime), the probe stimulus is simply another meaningless irrelevant item and the evoked P300 amplitude by the probe stimulus should not exhibit significant difference with that elicited by irrelevant stimuli. In

addition,

different

from

ERPs,

fNIRS

and

fMRI

are

vascular-based

functional neuroimaging techniques, which rely on the hemodynamic responses to infer brain activation. In particular, fNIRS measures the quantitative oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) concentration changes, which plays an important role in the study of cognitive processing in the frontal/prefrontal cortex (Yuan, 2013a, b, Yuan and Ye, 2013) and brain-computer interface (BCI) (Naseer and Hong, 2013, Naseer et al., 2014, Naseer and Hong, 2015b, a, Naseer et al., 2016a, Naseer et al., 2016b, Noori et al., 2017, Qureshi et al., 2017). Compared to fMRI, fNIRS can be operated in a portable, comfortable and quiet way with fewer body constraints (Beurskens et al., 2014, Lee et al., 2014, Naseer and Hong, 2015b). More interestingly, unlike ERPs-based CIT, the rationale underlying those fNIRS recordings was that the truthful responding denotes the default state of brain, whereas the deception involving the executive functions including withholding the truth and response monitoring, can elicit strong brain hemodynamic responses in the frontal cortex. For example, previous fNIRS and fMRI studies showed that compared to truthful responding, the deceptive responding exhibited increased brain activation associated with execution function in the middle frontal gyrus, superior frontal gyrus, and inferior frontal gyrus (Langleben et al., 2002, Kozel et al., 2005, Langleben et al., 2005, Hu et al., 2012, Sai et al., 2014b). More importantly, both EEG and fNIRS techniques have shown the potential for the detection of concealed information/deception although each of them can infer on different aspects of brain

4

activation (ERP component P300 or HbO signals) associated with various brain regions (Rosenfeld et al., 2004, Langleben et al., 2005, Sai et al., 2014b). However, it is still unknown whether the neural indicator that combines P300 and fNIRS measures can exhibit more significant brain activation difference between the guilty and innocent groups, and can also have the ability to detect deception with improved sensitivity. To test this assumption, fused EEGfNIRS techniques were used for concurrent measurements of evoked electrical potentials and hemodynamic responses by using a modified CIT paradigm (Mccarthy et al., 1997, Linden et al., 1999, Kennan et al., 2002). In particular, the fNIRS setup and EEG machine were integrated together to examine whether the accuracy of deception detection was significantly improved compared to that from a single neuroimaging modality only such as EEG or fNIRS. Consequently, the aim of this study is to examine the relationship between the ERP component P300 and hemodynamic signals for high-accuracy deception detection. Meanwhile, the deception detection approach developed by using fused EEG and fNIRS can be considered as a hybrid BCI, which includes the active, passive and reactive BCI. Previous reports demonstrated that the hybrid BCI technique can improve the classification rate of brain cognition (Fazli et al., 2012, Khan et al., 2014, Tomita et al., 2014, Khan and Hong, 2015, Lee et al., 2015, Khan and Hong, 2017). However, the present work can only be considered as an incomplete study on hybrid EEG-fNIRS based BCI for deception decoding since only simple fusion scheme for data classification is used. Later, when more post-processing methods are developed and utilized for the hybrid BCI, we can further this work based on combined EEG and fNIRS recordings for deception decoding. Further, although previous lie detection studies have been performed with the EEG or fNIRS measure (Rosenfeld et al., 2004, Langleben et al., 2005, Sai et al., 2014b), to the best of

5

our knowledge, this is the first study that use combined EEG and fNIRS recordings for deception detection. It is expected that the investigation into deception detection based on fused EEG and fNIRS neuroimaging methods is able to help establish a new way of improving the understanding the neural mechanism of deception and also pave a new avenue for highsensitivity deception detection. Materials and Methods Participants Thirty-two participants (16 females, mean age = 24.22 years, SD=2.67 years) took part in the experimental test. All participants were right-handed with no reported histories of brain trauma, stroke and other psychiatric or neurological diseases. Participants were requested to keep calm and minimize their body movement during the test. The experiment were conducted in accordance with approved guidelines with the Faculty of Health Sciences at the University of Macau. Procedures and Tasks Upon entering the laboratory, each participant was required to sign an informed consent form. And then the 32 participants were separated into the guilty group of 16 and the innocent group of 16. Each participant of the guilty group was informed that his (her) identity was a spy, whereas the participant of innocent group was told that the assigned identity was an innocent person. Although the variant of the differentiation of deception paradigm (Langleben et al., 2002, Kozel et al., 2005, Langleben et al., 2005, Hu et al., 2012, Sai et al., 2014b) was used to examine the brain activation difference between the guilty and innocent groups, a three-stimulus CIT task was adopted for the present study, which contains three categories of triggers: Probe, target and irrelevant stimuli. The target stimulus for the two groups was “Dehua Liu” (Andy Lau, 6

https://en.wikipedia.org/wiki/Andy_Lau), which was the Chinese name of a well-known Hongkong movie star, whereas the irrelevant stimuli for the two groups were eight names which participants were not familiar with. Interestingly, the probe stimulus was their individual name and one irrelevant name for the guilty or innocent group, respectively. Prior to concurrent EEG and fNIRS recordings, participants were given the following instructions: “Now, imagine you are arrested by an immigration officer for being a suspected spy when you are preparing to broad a flight in the airport. You need pass a lie detector test inside the second room on the boarder, in which you can observe a bunch of Chinese names displayed on the personal computer (PC). If you recognize these names, please press the button as soon as possible, whereas do nothing when you don’t know the names. For example, you should press the button when you see a name like “Dehua Liu” because you are telling the truth and you do recognize that. By contrast, you should not press the button for names you do not recognize. However, if you are a participant from a guilty group and discover that your name is presented on the monitor, you should not press the button in order to deny that you are a spy. If that was the case, you are lying since you do know your name.” Subjects sat around 1 m away from the PC. Stimuli including one target, one probe and eight irrelevant ones were repeated 40 times. Consequently, the paradigm consisted of 400 (10 × 40) trials in total. Each of the 400 trials started with the presentation of stimulus with the duration of 300ms, which was followed by the red fixation cross displayed in a PC centre for 2700ms (Figure 1(a)). When stimuli were present, participants needed response as fast as possible by pressing/without pressing the button. Participants could take a rest for every 80 trials (around 4 minutes) and the entire experiment test lasted about 20 minutes.

7

Since it is widely recognized that it need take several seconds for the slow homodynamic responses returning to the baseline, the CIT paradigm is modified to be compatible to fNIRS triggers as well. Figure 1(b) provided the schematic of our experiment design, in which the ERP task contained three categories of stimuli (T, P and I: T represents the target, P represents the probe, and I represents the irrelevant stimuli), whereas the fNIRS task only included two kinds of stimuli (P and I). It was discovered from Fig. 1(b) that the minimum of five irrelevant EEG stimuli with a period of 15s should be placed between successive fNIRS triggers (P or I) to avoid signal interruption from target or probe EEG stimuli (Menon et al., 1997, Linden et al., 1999, Kennan et al., 2002).

Data Acquisition for Concurrent fNIRS and EEG Recordings A home-made EEG/fNIRS patch based on the standard BioSemi EEG cap was used for the concurrent EEG and fNIRS data acquisition. The arrangements of optodes/electrodes along the patch were provided in Fig. 1 (c) for the present study, in which the optodes (four laser sources and eight optical detectors) with support were placed through the holes that were drilled into the EEG cap. The simultaneous EEG and fNIRS recordings were performed with the same stimuli tasks generated by E-prime software. EEG data acquisition was performed with a 74 channel Biosemi Active two system, which had 64 scalp active electrodes and eight non-scalp electrodes. In addition, the 64 active electrodes were put into the BioSemi electrode cap, whereas the eight non-scalp electrodes were utilized to measure the horizontal electrooculogram (HEOG). The sampling rate of EEG recordings was 2000 Hz by using a common average reference (64 EEG channels only). The

8

Common Mode Sense (CMS) electrode and Driven Right Leg electrode (DRL) served as the ground electrodes. For simultaneous fNIRS recordings, four CW laser sources with wavelengths 690nm and 830 nm and eight optical detectors (Fig. 1(d)) were utilized to generate the HbO and HbR concentration changes. The prefrontal cortex (PFC) of both hemispheres was covered by a patch with eight channels (Fig. 1(e)). The distance between each source and each detector was 3cm and the sampling rate of fNIRS data acquisition was 50Hz. In addition, a 3D-magnetic space digitizer Patriot Digitizer (Polhemus Inc.) was employed to capture the 3D spatial information of each optode for each subject. NIRS-SPM software was utilized to access each channel’s 3D location in the Montreal Neurological Institute (MNI) space (Singh et al., 2005). The 3D spatial coordinates of eight fNIRS channels along PFC were plotted in Fig. 1(e). Data Analysis EEG Offline analyses were performed using BASE Version 6 (Brain Electric Source Analysis; MEGIS Software GmbH, Gräfelfing, Germany). ERP data from all electrodes were processed by using an automatic artifact correction to remove eye movements via an internal model of artefact topographies. Continuous ERP data were segmented into two categories of trials according to the probe and irrelevants stimuli, respectively. Each epoch included a 200ms pre-stimuli period and a 1000ms post-stimuli period. ERP data were processed with a 30 Hz low-pass filter (24dB/ct) and then a 0.3 Hz high-pass filter (24d vB/ct). Trials exceeding ± 100µv were defined as artifacts and were excluded for further analysis. For stimulus-locked ERPs, we focused on the analysis of P300 at electrode Pz, which was the ERP component amplitude between 350ms and 550ms after trigger onset (Chen et al., 2017). The statistical analysis was performed by using SPSS 20.0. 9

fNIRS Homer2 software (Huppert et al., 2009) was used for fNIRS data preprocessing. The raw HbR and HbO signals were processed by a low pass filter of 0.2 Hz, followed by a high pass filter of 0.015 Hz. For the present study, only HbO signals were analysed (Zhang et al., 2016) , in which the run-averaged HbO signals were generated for each channel and subsequently the grandaveraged data were calculated for both the probe and irrelevants stimuli cases from the two groups. The peak values of run-averaged HbO data were extracted from each channel of each participant for further statistical analysis. All p values of F-test were corrected by false discovery rate (FDR<0.05) (Singh and Dan, 2006). All statistical analyses were conducted with SPSS 20.0. More Importantly, to exhibit the difference in hemodynamic responses between the two groups associated with various stimuli conditions, the HbO singals were mapped with the BrainNet Viewer tool (Xia et al., 2013).

Results Behavioural Results Behavioural data analysis was performed and the accuracy rate for each category of stimuli was calculated for each group. The mean accuracy rate was entered into two (group: guilty vs. innocent) by two (stimuli type: probe vs. irrelevant) repeated-measures ANOVA. It was discovered that neither the main effect of stimuli type (F (1, 30) = 0.11, p > 0.1, 0.99 ± 0.02 vs. 0.99± 0.05) nor the main effect of group (F (1, 30) = 0.31, p>0.1, 0.99 ± 0.006 vs. 0.99± 0.006) exhibited the conventional level of 10

significance. Our analysis results also showed no significant interaction between the stimuli type and the group (F (1, 30) = 2.17, p >0.1). ERP Results Figure 2 showed the grand-averaged ERP waves at electrode Pz and associated brain topography of P300 component for each of the two conditions (probe vs. irrelevant) from the guilty and innocent groups. A mixed measure ANOVA was performed by using a within-subject variable (stimuli: probe vs. irrelevant) and a between-subject variable (group: guilty vs. innocent). Our results demonstrated that the main effect of stimulus type exhibited the conventional level of significance (F (1, 30) = 22.99, p <0.001, ηp2 = .43), in which the probe stimulus induced higher P300 amplitude than the irrelevants ones (5.25 ± 0.49 µV vs. 3.19 ± 0.49 µV). This main effect was qualified by a significant interaction between the stimulus type and the group (F (1, 30) = 14.17, p = .001, ηp2 = 0.32). Further analysis demonstrated that for the guilty group, the probe stimulus elicited significantly higher P300 amplitude as compared to the irrelevants ones (6.59 ± 0.68 vs. 2.92 ± 0.31 µV, F (1, 30) = 36.63, p < .001, ηp2 = .55). By contrast, the probe stimulus and irrelevant ones exhibited no significant difference in the P300 amplitude for the innocent group (3.91 ± 0.68 vs. 3.46 ± 0.31 µV, F (1, 30) < 1, p = .47).

fNIRS Results Figure 3 showed the grand-averaged HbO signals of each channel for both the probe and irrelevant stimuli from the guilty and innocent groups. The peak values of run-averaged HbO signals from each channel were also extracted for further analysis for each participant in the two groups.

11

A mixed measure ANOVAs was performed using one within-subject variable (stimuli: probe vs. irrelevant) and one between-subject variable (group: guilty vs. innocent). No significant main effect of stimuli type was identified (F ≤ 7.2，p ≥ 0.052，FDR corrected). Interestingly, as shown in Table 1, the statistical analysis manifested significant interaction between the stimuli type and the group in channels 2, 4, 6, and 8( channel 2: F (1, 30) = 11.75, p=0.008, ηp2 = .28, FDR corrected; channel 4: F (1, 30) = 10.85, p =0.008, ηp2 = .27, FDR corrected; channel 6: F (1, 30) = 10.68, p =0.006, ηp2 = .26, FDR corrected; channel 8: F (1, 30) = 14.57, p =0.008, ηp2 = .33, FDR corrected). Further statistical analysis demonstrated that participants in the guilty group exhibited significantly higher HbO concentration changes in response to the probe stimulus compared to that from irrelevant stimuli in channels 2, 4, 6, and 8 (F (1, 30) = 18.67, p <0.001, ηp2 =.38; F (1, 30) = 9.92, p = 0.004, ηp2 =.25; F (1, 30) = 6.88, p = 0.01, ηp2 =.19; F (1, 30) = 20.83, p <0.001, ηp2 =.41). However, no significant difference in the HbO signals between the probe and irrelevant stimuli was revealed for the innocent group (F (1, 30) = 0.28, p > 0.05; F (1, 30) = 2.28, p > 0.05; F (1, 30) = 3.97, p > 0.05; F (1, 30) = 0.70, p > 0.05). To image the brain activation for various stimuli conditions from both the guilty and innocent groups, the HbO images were also generated and displayed on a brain template, as plotted in Fig. 4. The interaction effects between the P300 amplitude and HbO signals were described in Figure 5. In addition, as the brain activation between the probe stimulus and irrelevant ones exhibited significant difference for the guilty group, the grand-averaged HbO concentration changes of each channel were extracted at each time point during the stimulus period, which can describe the brain activation patterns in a dynamic way, as shown in Figs. 6 and 7.

12

Correlation between the P300 amplitude at Pz and HbO Signals We also examined whether the P300 amplitude at Pz exhibited significant relationship with the HbO signals for both the innocent and guilty groups. Consequently, the P300 amplitude difference at Pz between the probe and irrelevant stimuli, and HbO signal differences (channels 2, 4, 6 and 8) between these two conditions were calculated and generated for both the guilty and innocent groups. Statistical analysis results showed that for the guilty group, the P300 amplitude difference at Pz exhibited significant correlation with the HbO signal difference in channel 4 (r (16) = -0.54, p = .031) (see Fig. 8). However, this is not the case for the innocent group, in which no significant correlation was revealed between the P300 amplitude difference at Pz and HbO signal (peak values) difference (r s < .45, p s > .08) , as shown in Table 2. Individual Analysis by Using the Combined EEG and fNIRS Indicator The receiver operating characteristic (ROC) analysis was performed by using the EEG, fNIRS or combined EEG and fNIRS indicator, which was able to examine the accuracy to differentiate between the guilty and innocents groups. The P300 amplitude at Pz and peaks of HbO signals from significant channels were first extracted to generate the various indicators. Importantly, the EEG indicator was defined as the P300 amplitude difference at Pz between the probe and irrelevant stimuli, whereas the fNIRS indicator was denoted as the HbO signal difference in peak values between the probe stimulus and irrelevant ones for the significant channels 2, 4, 6, and 8. More importantly, the combined EEG-fNIRS indicator was denoted as the combined P300 amplitude difference at Pz and HbO signal differences (peak value) associated with the significant channels 2, 4, 6, and 8. To generate the combined EEG-fNIRS indicator, the P300 amplitude difference at Pz and HbO 13

signal peak value differences from significant channels 2, 4, 6 and 8 were transformed into standardized z-scores across all guilty and innocent participants. Then we averaged these zscores into a single indicator for each participant (Sai et al., 2016). For each ROC analysis, one indicator was used as the dependent variable (test variable), and one binary variable (1 for guilty group, 2 for innocent group) was set as state variable in the ROC Curve function of SPSS software (SPSS 20.0). The classification results based on the peak value differences of HbO signals (significant channels 2, 4, 6, and 8), P300 amplitude difference at Pz, and combined EEG-fNIRS indicator were given in Table 3, respectively, in which area under the curve (AUC) was generated based on ROC analysis. Discussion To the best our knowledge, this is the first study to use fused EEG and fNIRS techniques for the investigation of deception detection. Our results showed that the probe stimulus was able to elicit significantly larger P300 amplitude at Pz and significantly higher peak values of HbO signals in the bilateral superior frontal gyrus (SFG) and bilateral middle frontal gyrus than irrelevant stimuli for participants in the guilty group. However, this is not the case for the innocent group, in which the ERP component P300 and hemodynamic responses exhibit no significant difference between the probe stimulus and irrelevant ones. More importantly, our statistical analysis results revealed that the multimodal neuroimaging modality based on concurrent EEG and fNIRS recordings is able to improve the sensitivity of deception detection, in which the accuracy based on the combined EEG-fNIRS indicator is much higher than that based on a single EEG or fNIRS indicator. P300 is an ERP component, which is associated with multiple processes such as attention, working memory, and recognition (Linden, 2005, Polich, 2007). In ERP-based concealed

14

information test/deception detection, P300 reflects the recognition of probe stimulus which is only recognized by the guilty participants (Rosenfeld et al., 1987, Farwell and Donchin, 1991, Rosenfeld et al., 2013). Interestingly, consistent with previous ERP-based CIT studies (Meixner and Rosenfeld, 2014, Sai et al., 2016), our findings showed the probe stimulus can elicit significantly larger P300 amplitude than the irrelevant ones for the guilty group. The elevated P300 amplitude from the probe stimulus in the guilty group can be due to the fact that the guilty participants can recognize the meaningful probe (their own names), whereas participants in the innocent group can’t identify the meaningless probe (random names). In addition, the probe stimulus also elicited significantly higher peak values of HbO signals in channels 4 and 6 (bilateral SFG, BA10, also known as anterior prefrontal cortex or frontapolar) and channels 2 and 8 (bilateral MFG, BA46, also known as dorsolateral prefrontal cortex (DLPFC)) than irrelevant ones for the guilty group although this is also not the case for the innocent group. Our findings were in line with previous fMRI (Langleben et al., 2002, Ganis et al., 2003), PET (Abe, 2009), transcranial direct current stimulation (Karim et al., 2009), and fNIRS results for pathological liars studies (Ding et al., 2013, Ding et al., 2014). More importantly, it is widely recognized that the anterior prefrontal cortex was involved in integrating the outcomes of two or more separated cognitive operations in the pursue of a higher behavioural goal (Ramnani and Owen, 2004). In this study, the guilty participant had to response negatively to his (her) own name to deny the spy identity, in which he (she) needed integrate the outcomes of right (being caught as a spy) and wrong answers (not being caught as a spy), and then lied to the deception machine to avoid possible punishment. Further, the DLPFC plays an essential role in the frontal executive system, which is associated with inhibition control (MacDonald et al., 2000, Greene et al., 2004). Previous studies showed that the deception is also related to

15

inhibition and response control (Nunez et al., 2005). For the present work, the guilty participant had to inhibit the right response in order to cover up the truth, in which the DLPFC was activated for participants in the guilty group. However, this is not the case for the innocent group since the participants needed tell the truth. Meanwhile, we also discovered that increased P300 amplitude at Pz showed significant correlation with enhanced hemodynamic responses in the prefrontal cortex for the guilty group although this is not the case for the innocent group. According to the neurovascular coupling mechanism, the change of local neural activity associated with brain cognition is generally accompanies by subsequent alterations in the local blood flow and blood oxygen, which makes the hemodynamic responses in the DLPFC correlate with P300 amplitudes at the prefrontal cortex. In addition, P300 in the parietal lobe (Pz) reveals the recognition difference between the two conditions for the guilty group, whereas fNIRS signals reflect the difference of executive control in the prefrontal region between the two conditions. We discovered that the recognition showed significant relationship with the executive control during deception detection. As a result, P300 at Pz exhibited significant correlation with hemodynamic responses in the DLPFC. Further, although P300 appears within 1000 ms whereas hemodynamic response emerges within 13s after the onset of the stimuli, both the P300 at Pz and fNIRS signal can distinguish well between these two kinds of stimuli for the guilty group. As such, P300 at Pz also showed the relationship with the hemodynamic response in the right SFG. More interestingly, the highest detection accuracy for deception was achieved (AUC=0.94) when combining the P300 amplitude difference and peak value difference of HbO signal at channel 8. These findings indicated that the fNIRS hemodynamic response of and ERP component P300 were involved in various perspectives of cognitive mechanism associated with

16

deception. For example, the measure of ERP component P300 indicated that the guilty participants could recognize their individual names while the HbO signals in the prefrontal cortex exhibited the executive processing when participants denied their names. When combined the correlated ERP and fNIRS features together, more brain activation associated with deception information was identified, which can aid to enhance the sensitivity of lie detection. Previous studies have demonstrated that multiple measurements can enhance the detection accuracy of deception by using combined ANS measurements (Ben-Shakhar and Elaad, 2002, Gamer et al., 2008) or combined polygraph (ANS) and fNIRS recordings (hemodynamic response) (Bhutta et al., 2015). In this study, a modified CIT paradigm was adopted to conduct concurrent EEG and fNIRS recordings for the deception detection. And a combined EEG-fNIRS indicator was generated and applied to classifying the guilty participants from the innocent ones. We discovered that lie detection accuracy (AUC= 94%) based on fused EEG and fNIRS techniques was significantly improved compared to that from a single EEG or fNIRS neuroimaging method, in which the AUC based on the P300 amplitude difference is about 85% while that based on the peak value difference of HbO signals is about 84%. Compare to previous deception detection studies with a single EEG or fNIRS neuroimaging technique, the advantages of the present study showed that the concurrent EEG and fNIRS recordings based on a modified CIT task was able to achieve much higher classification accuracy. It is expected that the fused EEG-fNIRS neuroimaging technique will pave a new avenue for improving the understanding of neural mechanism of deception and will improve the accuracy for lie detection.

Limitations of the Fused EEG and fNIRS Technique

17

Although fused EEG and fNIRS neuroimaging techniques exhibited superior advantages in deception detection, it had several limitations as well. For example, the penetration depth for both fNIRS and EEG is generally less than 3cm, which are very challenging to detect deep brain activation associated with deception as compared to that from fMRI. In addition, the spatial resolution of both fNIRS and EEG is much lower than that from fMRI and the present fNIRS technique can only identify the hemodynamic response in the prefrontal region, which may also reduce the sensitivity of deception detection. Finally, it is noted that the signal slope and signal peak of both HbO and HbR can serve as effective fNIRS measures (Bhutta et al., 2015, Khan and Hong, 2015, Naseer and Hong, 2015b, Naseer et al., 2016a). However, the peak of HbO signal is routinely used in the fNIRS field for the study of brain activation and brain connectivity. We have not gained sufficient experience in the analysis of fNIRS data by using signal slope and mean as fNIRS features. Further work should be performed to examine the brain cognition and brain disorders by using these measures. However, the peak of HbO signal is the mostly used one in fNIRS field for the study of brain activation and brain connectivity due to its high signal-tonoise ratio. We have no experience in the analysis of fNIRS data by using signal slope and mean as fNIRS features. Further work should be performed to examine the brain cognition and brain disorders by using these measures.

Compliance with Ethical Standards: Funding: This study was funded by grants MYRG 2015-00036-FHS, MYRG 2016-00110-FHS and MYRG2018-00081-FHS from the University of Macau in Macau and FDCT grants FDCT 025/2015/A1 and 0011/2018/A1 from the Macao government.

18

Conflict of Interest: Author Xiaohong Lin declares that she has no conflict of interest. Author Zhen Yuan declares that he has no conflict of interest. Author Liyang Sai declares that he has no conflict of interest. Ethical approval: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors. Informed consent: Informed consent was obtained from all individual participants included in the study.

References Abe N (2009) The neurobiology of deception: evidence from neuroimaging and loss-of-function studies. Curr Opin Neurol 22:594-600. Ben-Shakhar G, Elaad E (2002) Effects of questions' repetition and variation on the efficiency of the Guilty Knowledge Test: A reexamination. J Appl Psychol 87:972. Beurskens R, Helmich I, Rein R, Bock O (2014) Age-related changes in prefrontal activity during walking in dual-task situations: a fNIRS study. Int J Psychophysiol 92:122-128. Bhutta MR, Hong MJ, Kim Y-H, Hong K-S (2015) Single-trial lie detection using a combined fNIRS-polygraph system. Front Psychol 6:709. Chen Y, Ke Y, Meng G, Jiang J, Qi H, Jiao X, Xu M, Zhou P, He F, Ming D (2017) Enhancing Performance of P300-Speller under Mental Workload by Incorporating Dual-task Data during Classifier Training. Comput Meth Prog Bio 152: 35-43. Ding XP, Gao X, Fu G, Lee K (2013) Neural correlates of spontaneous deception: A functional near-infrared spectroscopy (fNIRS) study. Neuropsychologia 51:704-712.

19

Ding XP, Sai L, Fu G, Liu J, Lee K (2014) Neural correlates of second-order verbal deception: A functional near-infrared spectroscopy (fNIRS) study. NeuroImage 87:505-514. Farwell LA, Donchin E (1991) The truth will out: Interrogative polygraphy (“Lie detection”) with Event‐Related brain potentials. Psychophysiology 28:531-547. Fazli S, Mehnert J, Steinbrink J, Curio G, Villringer A, Muller KR, Blankertz B (2012) Enhanced performance by a hybrid NIRS-EEG brain computer interface. NeuroImage 59:519529. Gamer M, Verschuere B, Crombez G, Vossel G (2008) Combining physiological measures in the detection of concealed information. Physiol Behav 95:333-340. Ganis G, Kosslyn SM, Stose S, Thompson W, Yurgelun-Todd DA (2003) Neural correlates of different types of deception: an fMRI investigation. Cereb cortex 13:830-836. Greene JD, Nystrom LE, Engell AD, Darley JM, Cohen JD (2004) The neural bases of cognitive conflict and control in moral judgment. Neuron 44:389-400. Hu X-S, Hong K-S, Ge SS (2012) fNIRS-based online deception decoding. J Neural Eng 9:026012. Huppert TJ, Diamond SG, Franceschini MA, Boas DA (2009) HomER: a review of time-series analysis methods for near-infrared spectroscopy of the brain. Appl Opt 48:D280-D298. Karim AA, Schneider M, Lotze M, Veit R, Sauseng P, Braun C, Birbaumer N (2009) The truth about lying: inhibition of the anterior prefrontal cortex improves deceptive behavior. Cereb Cortex 20:205-213. Kennan RP, Horovitz SG, Maki A, Yamashita Y, Koizumi H, Gore JC (2002) Simultaneous recording of event-related auditory oddball response using transcranial near infrared optical topography and surface EEG. NeuroImage 16:587-592. Khan MJ, Hong KS (2015) Passive BCI based on drowsiness detection: an fNIRS study. Biomed Opt Express 6:4063-4078. Khan MJ, Hong KS (2017) Hybrid EEG-fNIRS-Based Eight-Command Decoding for BCI: Application to Quadcopter Control. Front Neurorobot 11:6. Khan MJ, Hong MJ, Hong KS (2014) Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface. Front Hum Neurosci 8:244. Kozel FA, Johnson KA, Mu Q, Grenesko EL, Laken SJ, George MS (2005) Detecting deception using functional magnetic resonance imaging. Biol Psychiatry 58:605-613. Langleben DD, Loughead JW, Bilker WB, Ruparel K, Childress AR, Busch SI, Gur RC (2005) Telling truth from lie in individual subjects with fast event-related fMRI. Hum Brain Mapp 26:262-272.

20

Langleben DD, Schroeder L, Maldjian J, Gur R, McDonald S, Ragland J, O'brien C, Childress AR (2002) Brain activity during simulated deception: an event-related functional magnetic resonance study. NeuroImage 15:727-732. Lee M-H, Fazli S, Mehnert J, Lee S-W (2015) Subject-dependent classification for robust idle state detection using multi-modal neuroimaging and data-fusion techniques in BCI. Pattern Recognit 48:2725-2737. Lee SH, Jang G, Jin SH, ho Park J, Lee YJ, Lee JM, Kim S-J, An J (2014) Cortical activities during a stand to sit movement using fNIRS. In: Control, Automation and Systems (ICCAS), 2014 14th International Conference on, pp 279-281: IEEE. Linden DE (2005) The P300: where in the brain is it produced and what does it tell us? The Neuroscientist 11:563-576. Linden DE, Prvulovic D, Formisano E, Völlinger M, Zanella FE, Goebel R, Dierks T (1999) The functional neuroanatomy of target detection: an fMRI study of visual and auditory oddball tasks. Cereb Cortex 9:815-823. Lykken DT (1959) The GSR in the detection of guilt. J Appl Psychol 43:385. MacDonald AW, Cohen JD, Stenger VA, Carter CS (2000) Dissociating the role of the dorsolateral prefrontal and anterior cingulate cortex in cognitive control. Science 288:1835-1838. Mccarthy G, Luby M, Gore J, Goldman-Rakic P (1997) Infrequent events transiently activate human prefrontal and parietal cortex as measured by functional MRI. J Neurophysiol 77:16301634. Meixner JB, Rosenfeld JP (2014) Detecting knowledge of incidentally acquired, real-world memories using a P300-based concealed-information test. Psychol Sci 25:1994-2005. Menon V, Ford JM, Lim KO, Glover GH, Pfefferbaum A (1997) Combined event‐related fMRI and EEG evidence for temporal—parietal cortex activation during target detection. Neuroreport 8:3029-3037. Naseer N, Hong K-S (2013) Classification of functional near-infrared spectroscopy signals corresponding to the right-and left-wrist motor imagery for development of a brain–computer interface. Neurosci Lett 553:84-89. Naseer N, Hong K-S (2015a) Decoding answers to four-choice questions using functional near infrared spectroscopy. J Near Infrared Spectrosc 23:23-31. Naseer N, Hong K-S (2015b) fNIRS-based brain-computer interfaces: a review. Front Hum Neurosci 9:3. Naseer N, Hong MJ, Hong K-S (2014) Online binary decision decoding using functional nearinfrared spectroscopy for the development of brain–computer interface. Exp Brain Res 232:555564.

21

Naseer N, Noori FM, Qureshi NK, Hong K-S (2016a) Determining optimal feature-combination for LDA classification of functional near-infrared spectroscopy signals in brain-computer interface application. Front Hum Neurosci 10:237. Naseer N, Qureshi NK, Noori FM, Hong K-S (2016b) Analysis of different classification techniques for two-class functional near-infrared spectroscopy-based brain-computer interface. Comput Intell Neurosci 2016. Noori FM, Naseer N, Qureshi NK, Nazeer H, Khan RA (2017) Optimal feature selection from fNIRS signals using genetic algorithms for BCI. Neurosci Lett 647:61-66. Nunez JM, Casey B, Egner T, Hare T, Hirsch J (2005) Intentional false responding shares neural substrates with response conflict and cognitive control. NeuroImage 25:267-277. Polich J (2007) Updating P300: an integrative theory of P3a and P3b. Clin Neurophysiol 118:2128-2148. Qureshi NK, Naseer N, Noori FM, Nazeer H, Khan RA, Saleem S (2017) Enhancing Classification Performance of Functional Near-Infrared Spectroscopy-Brain–Computer Interface Using Adaptive Estimation of General Linear Model Coefficients. Front Neurorobot 11:33. Ramnani N, Owen AM (2004) Anterior prefrontal cortex: insights into function from anatomy and neuroimaging. Nat Rev Neurosci 5:184-194. Rosenfeld JP, Hu X, Labkovsky E, Meixner J, Winograd MR (2013) Review of recent studies and issues regarding the P300-based complex trial protocol for detection of concealed information. Int J Psychophysiol 90:118-134. Rosenfeld JP, Nasman VT, Whalen R, Cantwell B, Mazzeri L (1987) Late vertex positivity in event-related potentials as a guilty knowledge indicator: A new method of lie detection. Int J Neurosci 34:125-129. Rosenfeld JP, Soskins M, Bosh G, Ryan A (2004) Simple, effective countermeasures to P300‐based tests of detection of concealed information. Psychophysiology 41:205-219. Sai L, Lin X, Hu X, Fu G (2014a) Detecting concealed information using feedback related eventrelated brain potentials. Brain Cogn 90:142-150. Sai L, Lin X, Rosenfeld JP, Sang B, Hu X, Fu G (2016) Novel, ERP-based, concealed information detection: Combining recognition-based and feedback-evoked ERPs. Biol Psychol 114:13-22. Sai L, Zhou X, Ding XP, Fu G, Sang B (2014b) Detecting concealed information using functional near-infrared spectroscopy. Brain Topogr 27:652-662. Singh AK, Dan I (2006) Exploring the false discovery rate in multichannel NIRS. NeuroImage 33:542-549. Singh AK, Okamoto M, Dan H, Jurcak V, Dan I (2005) Spatial registration of multichannel multi-subject fNIRS data to MNI space without MRI. NeuroImage 27:842-851. 22

Xia M, Wang J, He Y (2013) BrainNet Viewer: a network visualization tool for human brain connectomics. PloS one 8:e68910. Yuan Z (2013a) Combining independent component analysis and Granger causality to investigate brain network dynamics with fNIRS measurements. Biomed Opt Express 4:26292643. Yuan Z (2013b) Spatiotemporal and time-frequency analysis of functional near infrared spectroscopy brain signals using independent component analysis. J Biomed Opt. 18:106011106011. Yuan Z, Ye J (2013) Fusion of fNIRS and fMRI data: identifying when and where hemodynamic signals are changing in human brains. Front Hum Neurosci 7. Zhang J, Lin X, Fu G, Sai L, Chen H, Yang J, Wang M, Liu Q, Yang G, Zhang J (2016) Mapping the small-world properties of brain networks in deception with functional near-infrared spectroscopy. Sci Rep 6.

Figure Captions Fig. 1 (a) Schematic of a single EEG trial (trigger) for the CIT task. (b) The schematic of modified CIT paradigm for fused EEG and fNIRS data acquisition. Here the ERP triggers contained all stimuli (P, I and T), whereas the fNIRS triggers only included the red stimuli (P and I). (c) The arrangements of optodes and electrodes along the Biosemi EEG cap. (d) The estimated mean cortical locations of four laser sources and eight fNIRS detectors. (e) The estimated mean locations of eight fNIRS channels along the prefrontal cortex. CIT represents concealed information test, PFC denotes the prefrontal cortex, and T, P and I represents the target, probe, and irrelevant stimuli, respectively. Fig. 2 (a) Grand-averaged ERP waveforms. (b) Scalp topography of ERP component P300. Here, Guilty denotes the guilty group while Innocent defines the innocent group. Probe indicates the probe stimulus while Irrelevant represents the irrelevant stimuli. Fig. 3 The time courses of the grand-averaged hemodynamic (HbO) change associated with the probe (red curve) and irrelevant stimuli (blue curve) for the guilty group and innocent group. Here, Guilty denotes the guilty group while Innocent defines the innocent group. Probe indicates the probe stimulus while Irrelevant represents the irrelevant stimuli. Fig. 4 Mapping of grand-averaged HbO concentration changes associated with the probe stimulus for the guilty group (top left), probe stimulus for the innocent group (bottom left), irrelevant stimuli for the guilty group (top-right), and irrelevant stimuli for the innocent group (bottom-right). We discovered that in the guilty group the probe stimulus elicited higher hemodynamic responses than irrelevant stimuli over the prefrontal cortex, whereas the change is not that obvious for the innocent group. 23

Fig. 5 The interaction effects between the stimuli type and the group. (A) The interaction effect of P300 at Pz. (B) The interaction effect of HbO signals at channel 2. (C) The interaction effect of HbO signal at channel 4. (D) The interaction effect of HbO signals at channel 6. (E)The interaction effect of HbO signals at channel 8. Fig. 6 Dynamic brain activation maps demonstrating the changes in brain activity over time (113s) associated with the probe stimuli for the guilty group. We discovered that HbO concentration changes (especially bilateral middle front gyrus) started to increase at 2s and continued to grow till 7s, sustained for several seconds (7-10 s), and then started to decline and the decrease continued to 13s. Fig. 7 Dynamic brain activation map demonstrating the changes in brain activity over time (113s) associated with irrelevant stimuli for the guilty group. No obvious HbO concentration changes were discovered during the stimuli period although the HbO concentration indeed increased a little bit between 4-7 s after trigger onset. Fig. 8 The relationship between the ERP P300 amplitude difference at Pz and peak value of HbO signal difference at channel 4 between the probe and irreverent stimuli for the guilty group.

24

25

26

27

28

29

30

31

32

Table 1. The statistical analysis results by using the peak values of HbO signals. Channels

MNI coordinates

Brain regions

X

Y

Z

Ch2

44

56

17

Ch4

19

68

Ch6

-10

Ch8

-37

Probability

Statistic values HbO signals Guilty group

Innocent group

Right MFG (BA46) 0.97

F = 11.75** F = 18.67***

F = 0.28

25

Right SFG (BA10)

0.86

F = 10.85** F = 9.92**

F = 2.28

68

25

Left SFG (BA10)

1

F = 10.68** F = 6.88*

F = 3.97

59

20

Left MFG (BA46)

0.90

F = 14.57** F = 20.83***

F = 0.70

33

Table 2. Correlations between P300 and HbO signals Guilty group

P300

Channel 2

Channel 4

Channel 6

Channel 8

-0.02

-0.54*

-0.15

0.19

Channel 2

Channel 4

Channel 6

Channel 8

0.37

0.45

0.33

-0.12

Innocent group

P300

*p < 0.5, n = 16 for each group.

34

Table 3. Receiver operation characteristic (ROC) analyses. AUC

95% confidence intervals

P300

.85**

.70-1.00

Channel 2

.82**

.68-.97

Channel 4

.83**

.68-.99

Channel 6

.79**

.63-.95

Channel 8

.83**

.69-.98

P300 and Channel 2 .92*** .81-1.00 P300 and Channel 4 .91*** .80-1.00 P300 and Channel 6 .90*** .80-1.00 P300 and Channel 8 .94*** .87-1.00 **p < .01, ***p < .001.

35

Highlights: 1. A fused EEG and fNIRS neuroimaging technique was developed for high-sensitivity deception detection. 2. The relationship between the ERP component P300 and fNIRS measure was carefully examined. 3. The neurovascular mechanism associated with deception was inspected by using simultaneous fNIRS and EEG recordings. 4. Deception detection with high accuracy was achieved by using combined ERP and fNIRS indicators.

36