- Email: [email protected]
Contextual effects of surprised expressions on the encoding and recognition of emotional target faces: An event-related potential (ERP) study
Accepted Manuscript Title: Contextual effects of surprised expressions on the encoding and recognition of emotional target faces: an event-related potential (ERP) study Authors: Huiyan Lin, Claudia Schulz, Thomas Straube PII: DOI: Reference:
S0301-0511(17)30246-6 http://dx.doi.org/10.1016/j.biopsycho.2017.09.011 BIOPSY 7440
To appear in: Received date: Revised date: Accepted date:
14-1-2017 27-8-2017 18-9-2017
Please cite this article as: Lin, Huiyan, Schulz, Claudia, Straube, Thomas, Contextual effects of surprised expressions on the encoding and recognition of emotional target faces: an event-related potential (ERP) study.Biological Psychology http://dx.doi.org/10.1016/j.biopsycho.2017.09.011 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Contextual effects of surprised expressions on the encoding and recognition of emotional target faces: an event-related potential (ERP) study Running title: surprised context effects on encoding and recognition Huiyan Lin a, b, c, Claudia Schulz d, Thomas Straube c a
Institute of Applied Psychology, Guangdong University of Finance, 510521
Guangzhou, China b
Laboratory for Behavioral and Regional Finance, Guangdong University of Finance,
510521 Guangzhou, China c
Institute of Medical Psychology and Systems Neuroscience, University of Muenster,
48149 Muenster, Germany d
Department of Clinical Psychology and Psychotherapy, University of Muenster,
48149 Muenster, Germany Correspondence: Huiyan Lin Institute of Applied Psychology Guangdong University of Finance No. 527 Yingfu Road, Tianhe District, 510521, Guangzhou, China Email: [email protected] Tel.: +86-020-37215850
1
Highlights:
Face encoding and recognition are impaired by negative contextual expressions.
Whether other contextual expressions affect face encoding and recognition is unknown.
Surprised compared to neutral context reduced encoding-LPP for happy target faces.
A context effect was shown in N170, but it did not interact with target expression.
In later recognition, there was a context effect in happy-encoded target faces.
Abstract Several studies reported that the encoding and recognition of emotional target faces are modulated by negative contextual expressions. However, it is unknown whether other contextual expressions, such as emotionally ambiguous expressions, affect the encoding and recognition of target faces. To this end, electroencephalography (EEG) was recorded during the presentation of angry or happy target faces after a random sequence of surprised or neutral contextual faces that did not differ in normative valence ratings. Subsequently, participants were asked to perform an unexpected old/new recognition task in which target faces were shown in either the encoded or a non-encoded expression. During the encoding phase, event-related potential (ERP) results showed that surprised as compared to neutral contextual faces led to smaller late positive potentials (LPP) for happy but not for angry target faces. Similar effects were also observed in the N170, even though the interaction of context and target expression failed to reach statistical significance. In the later recognition phase, recognition rates were lower for encoded happy faces when they had been encountered in surprised as compared to neutral context, regardless of whether the target face showed the encoded or a non-encoded expression. However, this context effect was not observed for angry-encoded faces. Taken together, the present study indicates that ambiguous contextual expressions, e.g., surprised faces, affect structural and cognitive encoding and later recognition of happy target faces to a larger extent than neutral contextual faces; whereas angry faces are more resistant to context effects. Keywords: surprised context; angry faces; happy faces; event-related potentials (ERPs); encoding; recognition.
2
Introduction Facial expressions convey a wide variety of interpersonal information about individuals’ emotion and motivation. For example, negative expressions (e.g., fear, sadness, disgust and anger) signify potential harm and threat, a happy expression signals benefits and rewards, and a surprised expression conveys signals of emotional and motivational ambiguity. A number of studies have investigated whether facial expressions influence face encoding and recognition using electrophysiological (e.g., N170 and late positive potential, LPP) and behavioral parameters, respectively (e.g., Balconi & Mazza, 2009; Batty & Taylor, 2003; Calvo & Beltrán, 2013, 2014; Hagemann, Straube, Schulz, 2016; Holmes, Nielsen, & Green, 2008; Johansson, Mecklinger, & Treese, 2004; Morel, Ponz, Mercier, Vuilleumier, & George, 2009; Müller-Bardorff et al., 2016; Lin, Schulz, & Straube, 2015a, 2015b, 2016; Pinabiaux et al., 2013; Schupp et al., 2004; Sessa, Luria, Gotler, Jolicœur, & Dell'Acqua, 2011; Righi et al., 2012; Shimamura, Ross, & Bennett, 2006; Smith, Weinberg, Moran, & Hajcak, 2013). Structural face encoding is often supposed to be reflected by the N170 (Eimer, 2000a, 2000b; Eimer & McCarthy, 1999) in event-related potential (ERP) studies, a face-sensitive component that peaks around 170 ms after stimulus onset and is maximal at occipitotemporal scalp sites. The N170 is relevant to pre-categorical perceptual and structural encoding of faces in face-specific ventral visual areas (e.g., Eimer, 2000a, 2000b; Eimer & McCarthy, 1999; Labuschagne, Croft, Phan, & Nathan, 2010). It has repeatedly been shown that negative faces (e.g., fearful and angry faces) as compared to neutral faces evoke larger N170 amplitudes (e.g., Batty & Taylor, 2003; Calvo & Beltrán, 2013, 2014; Hagemann et al., 2016; Lin et al., 2015a, 2015b, 2016). N170 amplitudes are found to be enlarged also by happy and surprised faces (e.g., Calvo & Beltrán, 2014; Morel et al., 2009), especially for happy faces with strong emotional intensity (Müller-Bardorff et al., 2016). Increased structural encoding of emotional expressions reflected by the N170 has been suggested to be due to the mobilization of attention (Batty & Taylor, 2003; Lin et al., 2015a). The late positive potential (LPP; often overlapping with the P3), a positive deflection which starts at approximately 300 ms after stimulus presentation over frontal-parietal scalp sites, is also supposed to be involved in stimulus encoding (e.g., Dillon, Cooper, Grent, Woldorff, & LaBar, 2006; Michalowski, Pané-Farré, Löw, & Hamm, 2015; Schupp et al., 2004). In contrast to the N170, the LPP is thought to be associated with higher-order and complex cognitive encoding processes (e.g., Labuschagne et al., 2010). Importantly, studies have shown effects of facial expressions on LPP amplitude (e.g., Balconi & Mazza, 2009; Holmes, Nielsen, & Green, 2008; Righi et al., 2012; Schupp et al., 2004; Smith et al., 2013). 3
For example, Righi et al. (2012) found that fearful and happy as compared to neutral faces evoked larger LPP amplitudes. In the study of Balconi and Mazza (2009), surprised faces were also shown to elicit greater LPP amplitudes than neutral faces. Similar to the N170, LPP effects in cognitive encoding of emotional faces have also been attributed to the mobilization of attention (e.g., Polich, 2007; Polich & Criado, 2006). Additionally, although studies have investigated whether recognition of face identity is affected by facial expressions, results are still rather mixed. Some studies reported that negative expressions enhanced face recognition (e.g., Pinabiaux et al., 2013; Sessa et al., 2011), whereas others failed to observe similar effects (e.g., Johansson et al., 2004; Righi et al., 2012) or even showed enhanced recognition of happy or neutral faces (e.g., Hagemann et al., 2016; Shimamura et al., 2006). Discrepancies may be due to whether recognized faces are presented with the encoded expressions or changed into a new expression during the recognition phase (Hagemann et al., 2016; Lin et al., 2015a), whether emotional expressions are shown in the encoding or the recognition phase (Chen et al., 2015), and whether the recognition task is expected or unexpected (e.g., Henson, Shallice, Gorno-Tempini, & Dolan, 2002). In daily life, there are many circumstances where faces are not viewed in isolation, but in the context of other emotional stimuli, such as preceding contextual expressions. Several previous studies have shown that preceding contextual faces with a negative expression modulate the encoding of target faces (Furl, van Rijsbergen, Treves, Friston & Dolan, 2007; Lin et al., 2015a; Richards et al., 2013). For example, fearful compared to neutral contextual expressions reduced the M170 (which is thought to be the magnetoencephalography (MEG) counterpart of the N170; Deffke et al., 2007) or N170 to upcoming target faces, regardless of the expression of target faces (Furl et al., 2007; Lin et al., 2015a). Fearful compared to neutral contextual expressions were also found to evoke smaller LPP amplitudes for target faces irrespective of the expression of target faces, at least for highly anxious individuals (Richards et al., 2013). Moreover, we found that recognition rates were reduced for target faces that had been encoded in the context of fearful compared to neutral expressions. This effect was observed particularly when target faces showed the encoded expression (Lin et al., 2015a). Taken together, these findings indicate that fearful contextual expressions impair the encoding and recognition of target faces, probably due to distraction (Lin et al., 2015a). However, to the best of our knowledge, it is still unknown whether the encoding and recognition of target faces are affected by other contextual expressions (e.g., emotionally ambiguous expressions) and whether contextual effects of ambiguous expressions are 4
affected by the valence of target faces. Therefore, the present study aimed at addressing these questions by using an encoding task and a subsequent recognition task. For the encoding task, participants were asked to indicate the presence of angry or happy target faces in a sequence of emotionally ambiguous (e.g., surprised) or neutral contextual faces. Target and contextual faces were of different identities in order to investigate the recognition effects of target faces in a later recognition task. Considering that some surprised faces are perceived as positive or negative (Neta, Davis, & Whalen, 2011; Neta & Whalen, 2010) and that the present study aimed at investigating effects of emotionally ambiguous contextual faces, we selected surprised faces whose valence was rated neutral, similar to the neutral faces. In addition, electroencephalography (EEG) was recorded during this phase. Subsequently, participants were presented with all target faces and novel faces. Participants were asked to judge whether the prompted face had been presented in the preceding phase (old/target faces) or not (novel faces). Target faces were presented in either the encoded or a non-encoded expression. The recognition task was unexpected. Participants were not informed about the recognition task until the end of the encoding phase. Previous studies have suggested that high arousing negative contextual expressions modulate the encoding and recognition of target faces, possibly as a result of the negative emotion and/or the high arousal of these expressions (Furl et al., 2007; Lin et al., 2015a; Richards et al., 2013). As surprised expressions are also thought to be highly arousing, we predicted that surprised as compared to neutral contextual facial expressions should reduce the encoding of target faces, which is reflected in the N170 and LPP, as well as their later recognition, especially when target faces show the encoded expression. In addition, angry target faces might be less affected by surprised context than happy target faces; as there seems to be a stronger attentional bias for angry than for happy faces among surprised faces (Calvo & Beltrán, 2014).
Methods Participants Twenty-four participants were recruited in Muenster via advertisement and received 10 Euros for participation. Two participants were excluded from statistical analysis due to excessive artifacts in the EEG signal, resulting in a total of 22 participants (20-35 years, M = 24.57, SD = 3.29; 15 females). All participants were right-handed as determined by the Edinburgh Handedness Inventory (Oldfield, 1971). Participants had normal or corrected-tonormal vision and did not report a history of neurological illness. All participants gave written informed consent in line with standard ethical guidelines from the Declaration of 5
Helsinki. The study was approved by the local ethics committee of the University of Muenster. Stimuli Stimuli were 488 digitized color pictures that portrayed 244 identities (122 females and 122 males) with two different expressions each. Of these identities, 242 identities (121 females and 121 males) with angry and happy expressions were selected from FACES (Ebner et al., 2010), Karolinska Directed Emotional Faces (KDEF, Lundqvist et al., 1998), NimStim (Tottenham et al., 2009) and Radboud Faces Database (RaFD, Langner et al., 2010). The other two identities (1 female and 1 male) showing surprised and neutral expressions were selected from RaFD only. Surprised and neutral faces were similar in valence ratings (surprised versus neutral = 3.0 versus 3.1; ranging from 1 “very unpleasant” to 5 “very pleasant”), but higher in arousal ratings (surprised versus neutral = 4.1 versus 3.2; ranging from 1 “very low” to 5 “very high”) according to the supporting materials of the original study (Langner et al., 2010). Mean recognition accuracy for surprised and neutral faces was 100% and 86%, respectively. As facial pictures were derived from different databases, the pictures were cropped similarly around the face outline, and centered so that eyes, nose and mouth were at similar positions for all faces. We also removed external features (e.g., neck, shoulder, distant hair and jewelry), adjusted the size to 7.49 cm (horizontal) × 9.99 cm (vertical) and matched the pictures in luminance, contrast, hue and color using Adobe Photoshop CS6. The two identities with surprised and neutral facial expression served as contextual identities during the encoding phase. The other 122 identities (61 females and 61 males; including 1 female and 1 male for the practice) with angry and happy expressions were used as target identities. The remaining 240 pictures of 120 identities (60 females and 60 males; 120 angry and 120 happy) were used as novel facial pictures during the recognition phase. Target faces in the actual experiment were separated into four sets pseudo-randomly with 30 identities (15 females and 15 males; 30 angry and 30 happy) each. Each set was separated into two sub-sets that were identical in identity (30 identities), but different in facial expression. In the encoding phase, half of the sub-sets with different identities were used to create four experimental conditions: angry targets among surprised context faces, angry targets among neutral context faces, happy targets among surprised context faces and happy targets among neutral context faces. Assignments of sub-sets were counterbalanced across participants. During the recognition phase, all sub-sets were presented to create two conditions for learned faces: changed and unchanged (with regard to facial expressions). 6
Novel identities were also presented with two expressions during the recognition phase. For more details, please refer to Figure 1. (Insert Figure 1 about here) Procedure After informed consent had been given and handedness had been determined, participants were asked to sit in a comfortable chair at a viewing distance of approximately 80 cm from a 15-inch CRT computer screen which resolution was 1280 by 1024 pixel, so that the visual angle of the face stimuli was 5.37° × 7.15°. Stimuli were presented on a black background. Stimulus presentation and behavioral data collection were controlled with EPrime 2.0 software (Psychology Software Tools, Inc., Pittsburgh, PA, USA). Before the experiment started, participants had to familiarize themselves with the four contextual facial pictures. Participants were told that they would be presented with these (contextual) faces and some other (target) faces. They were also told that their task was to indicate the familiarity of the face by pressing the “F” or the “J” key with their left and right index finger, respectively, as quickly and correctly as possible. Response assignments were counterbalanced across participants. Practice trials with feedback on response accuracy and times were performed in order to ensure proper familiarization with the contextual faces and the experimental procedure. Each trial started with a white fixation cross for 200 ms. Following a blank screen for 600 to 1000 ms (M = 800 ms), either a contextual or a target face was presented for 800 ms. The next trial started after another blank screen for 1000 ms. Button presses were allowed from the onset of the face to the offset of the following blank. Faces were presented separately in four different blocks according to the expressions of contextual and target faces (angry targets among surprised context faces, angry targets among neutral context faces, happy targets among surprised context faces and happy targets among neutral context faces). Block sequence was randomized across participants. In each block, each contextual face was presented 90 times and each target face twice. Therefore, the experiment in the encoding phase consisted of a total of 960 (2 × 90 × 4 + 30 × 2 × 4) trials. There were two breaks in each block and the break duration was controlled by the participants. The recognition phase followed the encoding phase after approximately 5 minutes. Participants were first informed about the recognition task at this point in the experiment. Participants were told to indicate whether the identity of a prompted face had been presented in the preceding encoding phase (target or old identity) or not (novel identity) by 7
pressing the keys “F” or “J”. Participants were particularly informed that the old/new judgment was based on facial identity, regardless of the facial expression (e.g., Identity A showed a happy expression in the preceding task. A will either show a happy or an angry expression in the following task. Both happy and angry A should be judged as “old”). The instructions emphasized speed as well as accuracy. Response assignments were counterbalanced across participants. Each trial started with a white fixation cross which was presented for 200 ms. Following a blank screen for 600 to 1000 ms (M = 800 ms), either a target/old face or a novel face was presented for 500 ms. Another blank screen was presented for 1000 ms before the next trial started. Button presses were allowed from the onset of the face to the offset of the following blank. Each facial picture was presented once. Therefore, the experiment in the recognition phase consisted of a total of 480 trials, half of which were target/old faces and the other half novel faces. There were three breaks in this phase and the break duration was determined by participants. The complete experiment with the encoding and the recognition phase (including the preparations and practice) lasted approximately 2 hours. Behavioral recording For the encoding and the recognition phase, accuracy and response times of button presses were recorded from the onset of the target face to the offset of the following blank. EEG recording EEG was continuously recorded with a 32-channel BioSemi Active Two System (BioSemi, Amsterdam, Netherlands). Thirty-two Ag/AgCl active electrodes were placed on the scalp by means of an elastic head cap (BioSemi, Amsterdam), according to the international 10-20 system (FP1, FP2, F7, F3, Fz, F4, F8, FC1, FC5, FC2, FC6, T7, T8, C3, Cz, C4, TP9, TP10, CP1, CP2, P7, P9, P3, Pz, P4, P8, P10, PO9, PO10, O1, Oz, O2). The BioSemi System uses an active electrode (common mode sense, CMS) and a passive electrode (driven right leg, DRL) to form a feedback loop instead of ground and reference (please see http://www.biosemi.com/faq/cms&drl.htm). The horizontal electrooculogram (EOG) was recorded from two electrodes at the outer canthi of both eyes, and the vertical EOG was recorded bipolarly from two electrodes above and below the right eye to monitor eye blinks and movements. All signals were digitized with a sample rate of 2048 Hz and a low-pass fifth order sinc filter with a half-power cutoff at 100 Hz. Impedances were generally kept below 10 kΩ.
8
During offline processing, ocular artifacts were corrected using the algorithm implemented in BESA 6.0 software (www.BESA.de). The continuous EEG was then segmented from -200 to 800 ms relative to onset of the target face during the encoding phase, with the first 200 ms epoch for baseline correction. Artifact rejection was performed with the amplitude threshold of 120 μV, a gradient criterion of 75 μV and low-signal criterion of 0.01 μV. Artifact-free trials were averaged for each channel and experimental condition. Averaged ERPs were re-computed to average reference, excluding vertical and horizontal EOG channels. ERPs were then low-pass filtered at 40 Hz (Butterworth zero phase shift). ERPs were quantified using mean amplitudes for N170 and LPP, both relative to -200 ms baseline. The N170 was measured at the electrodes P7 and P8 and the LPP at the electrodes Fz, FC1, FC2, Cz, CP1, CP2 and Pz. The time windows for the N170 and LPP were 130-180 ms and 350-800 ms, respectively. The time window of the N170 was chosen according to the peak latency identified in the grand waveforms across all conditions (155 ms) and previous studies (e.g., Batty & Taylor, 2003; Lin et al., 2015a, b). The time window of the LPP and the electrodes of interest for the N170 and LPP were chosen based on visual inspection of the grand waveforms and previous studies (e.g., Batty & Taylor, 2003; Meeren, van Heijnsbergen, & de Gelder, 2005; Polich, 2007; Polich & Criado, 2006). Although EEG was also recorded during the recognition phase, ERPs could not be analyzed due to an insufficient number of correct trials (on average about 10 per experimental condition). Data analysis For the encoding and the recognition phase, mean accuracy and response times were calculated for each participant and each experimental condition. For the analysis of response times, only trials with correct responses were included. For the encoding phase, mean accuracy and response times for target faces were entered into 2 × 2 repeated measures analyses of variance (ANOVAs) with “contextual expression” (surprised versus neutral) and “target expression” (angry versus happy) as within-subject factors. For statistical analysis of ERP data, additional within-subject factors “hemisphere” (left versus right; for N170) and “electrode” (Fz versus FC1 versus FC2 versus Cz versus CP1 versus CP2 versus Pz; for LPP) were included. In the recognition phase, accuracy and response times for the target old faces were entered into 2 × 2 × 2 repeated measures ANOVAs with “consistency” (unchanged versus changed), “contextual expression” (surprised versus neutral) and “encoded target expression” (angry versus happy) as within-subject factors. Statistical analyses were performed using IBM SPSS Statistics software (Version 22; SPSS 9
INC., an IBM company, Chicago, Illinois). Greenhouse-Geisser corrections were applied to correct degrees of freedom and p values where appropriate. Bonferroni corrections were used to correct post-hoc t-tests.
Results Encoding phase Behavioral data The analyses of accuracy and response times did not reveal any main effects or interactions (ps. > .05; Table 1). (Insert Table 1 about here) ERP data N170 The analysis of the N170 revealed an interaction between “contextual expression” and “hemisphere” (F(1, 21) = 4.75, p = .041, η2p = .18). Separate analyses for each hemisphere showed that the effect of “contextual expression” was not significant over the left hemisphere (ps. > .05), while right hemispheric N170 amplitudes were larger for target faces in a sequence of neutral as compared to surprised contextual faces (F(1, 21) = 4.59, p = .044, η2p = .18). Other main effects or interactions were not significant (ps. > .05). See Table 2 and Figure 2 and 4 in more detail. As Table 2 and Figure 2 and 4 suggest, the right hemispheric effect of “contextual expression” seems prominent only for happy but not for angry target faces. Indeed, exploratory post-hoc tests for each target expression revealed that the effect of “contextual expression” was only significant for happy target faces (F(1, 21) = 4.41, p = .048, η2p = .17), but not for angry target faces (ps. > .05), suggesting a potentially stronger effect of contextual expression for happy target faces. Table 2 also suggests that the right hemispheric N170 was larger for happy compared to angry target faces particularly in the neutral face context. However, exploratory post-hoc tests did not reach statistical significance (p > 0.05). (Insert Table 2 and Figure 2 about here)
10
LPP The analysis showed main effects of “contextual expression” (F(1, 21) = 5.29, p = .032, η2p = .20) and “electrode” (F(1, 21) = 44.23, p < .001, η2p = .70). The LPP was more positive for target faces in a sequence of neutral as compared to surprised contextual faces. The LPP was also larger at electrode Pz as compared to Fz (p < .001), FC1 (p = .004) and FC2 (p = .004), at CP1 and CP2 as compared to Fz, FC1 and FC2 (all p < .001), and at FC1 and FC2 as compared to Fz (all p < .001). There was a significant interaction between “contextual expression” and “target expression” (F(1, 21) = 4.54, p = .045, η2p = .18). Separate analyses for each category of “target expression” showed that the LPP was larger for happy target faces in a sequence of neutral as compared to surprised contextual faces (F(1, 21) = 6.96, p = .015, η2p = .25); whereas the effect of “contextual expression” was not significant for angry target faces (ps. > .05). Notably, separate analyses for each category of “contextual expression” revealed no significant effects of “target expression” (p > 0.05). See Table 3 and Figure 3 and 4 in more detail.
(Insert Table 3 and Figure 3 and 4 about here)
Recognition phase Accuracy We found a main effect of “consistency” (F(1, 21) = 20.10, p < .001, η2p = .49), with higher accuracy for target faces showing the encoded expression as compared to faces showing a non-encoded expression. More importantly, the interaction between “contextual expression” and “encoded target expression” was significant (F(1, 21) = 4.85, p = .039, η2p = .19). Accuracy was higher for happy encoded target faces in a sequence of neutral as compared to surprised contextual faces (F(1, 21) = 4.96, p = .037, η2p = .19). For angry encoded target faces, however, the effect of “contextual expression” was not significant (ps. > .05). Other main effects or interactions did not reach statistical significance (ps. > .05). For descriptive data, please refer to Table 4.
Response times 11
No main effects or interactions were significant (ps. > .05). For descriptive data, please see Table 4.
(Insert Table 4 about here)
Discussion The present study aimed to investigate whether surprised contextual facial expressions as compared to neutral expressions modulated the encoding and recognition of happy and angry target faces. Results showed that during the encoding phase, the N170 was smaller for happy but not for angry target faces when these faces were presented in a sequence of surprised as compared to neutral contextual expressions. However, the effects of “contextual expression” did not significantly differ between angry and happy target faces. The LPP was smaller for happy target faces within surprised as compared to neutral contextual expressions; whereas the LPP to angry target faces was not affected by the contextual expression. In a later recognition phase, recognition rates were lower when these target faces had been encountered in surprised as compared to neutral contextual expressions, regardless of whether target faces showed the encoded expression or not. However, we did not observe any context effects for angry encoded target faces.
Effects of “contextual expression” on the encoding of target faces Similar to our previous study (Lin et al., 2015b), we did not find context effects on the behavioral classification of target faces during the encoding phase. This might be due to the low task difficulty. However, there was a clear effect on the corresponding ERPs, with smaller N170 and LPP especially for happy target faces after a sequence of surprised as compared to neutral contextual expressions. Given that the N170 and LPP are related to structural and complex cognitive encoding processes, respectively (Dillon et al., 2006; Eimer, 2000a, 2000b; Eimer & McCarthy, 1999; Labuschagne et al., 2010; Michalowski, et al., 2015; Schupp et al., 2004), the findings in the present study indicate that structural and cognitive encoding of happy target faces are reduced when these faces are shown in a sequence of ambiguous (e.g., surprised) as compared to neutral contextual expressions. While the effect of context expression on N170 did not differ between happy and angry target faces, context affected the processing of happy faces in the LPP time window (see below). 12
These context effects may be explained by available attentional resources during the presentation of target faces (Lin et al., 2015b). Considering that attentional resources are occupied by preceding stimuli in general (Flaisch, Stockburger & Schupp, 2008) and that emotional as compared to neutral stimuli are often thought to occupy more attentional resources (e.g., Carretié, Hinojosa, Martín-Loeches, Mercado, & Tapia, 2004; Vuilleumier, 2005), attention should be reduced for target stimuli that are preceded by emotional as compared to neutral stimuli. Furthermore, attention is supposed to affect face encoding as reflected in the N170 (e.g., Crist, Wu, Karp, & Woldorff, 2007; Mohamed, Neumann, & Schweinberger, 2009), with larger amplitudes for faces that are attended as compared to those that are not (e.g., Holmes, Vuilleumier, & Eimer, 2003). Attention also influences stimulus processing shown in the LPP, with enhanced amplitudes for stimuli that capture attentional resources (e.g., novel stimuli; e.g., Polich, 2007; Polich & Criado, 2006). Therefore, in the present study, surprised as compared to neutral contextual faces may occupy more attentional resources even during presentations of target faces, and as a consequence, result in poorer structural and complex cognitive encoding of following target faces. It should be noted that while previous studies attributed emotional ERP effects mainly to the effects evoked by emotional stimuli than those elicited by neutral stimuli; in these studies, positive and/or negative stimuli were used (e.g., Carretié et al., 2004; Vuilleumier, 2005). The present study used surprised faces, which had been rated as emotionally neutral in a prior study (Langner et al., 2010). Therefore, the exact mechanism behind the difference between surprised and neutral contextual faces is still unclear. Given that the present study could not provide an answer to this question, related issues should be carefully addressed in future studies. Another possible interpretation is that our oddball-like task induces strong expectations regarding the occurrences of facial expressions. The expectation of surprised contextual facial expressions might interfere with processing of expectation-incongruent (happy or angry) target faces more than the expectation of neutral contextual facial expressions. However, this interpretation seems to be unlikely, as previous studies found that fearful contextual faces impaired processing of target faces regardless of whether target faces were expectation-congruent or -incongruent (Furl et al., 2007; Lin et al., 2015b). Regarding the N170, the findings in the present study are in line with previous studies (Furl et al., 2007; Lin et al., 2015b) that suggested reduced structural encoding of target faces in a sequence of emotional compared to neutral contextual expressions. Remarkably, these studies only addressed the comparisons between fearful and neutral contextual expressions. Fearful and neutral expressions differ not only in valence but also in some other 13
factors (e.g., arousal). These factors have also been suggested to influence the structural encoding of faces (e.g., Sprengelmeyer & Jentzsch, 2006). Therefore, it was still unclear whether the context effect is based on a negative valence effect or other effects of contextual faces. In the present study, we still observed a similar context effect on the encoding of target faces, although context expressions did not differ in valence. Therefore, our present findings indicate that the encoding of target faces may be not - at least not only - modulated by valence, but by some other factors of contextual facial expressions, such as arousal. While we did not find contextual effects on the N170 contingent on the valence of target faces, there was a stronger effect of context in the LPP for happy as compared to angry target faces. This suggests weaker distractions by (surprised and neutral) contextual faces on the processing of angry target faces. A study by Calvo and Beltrán (2014) suggests that individuals allocate more attentional resources to angry as compared to some other emotional (e.g., surprised, happy and neutral) faces. This might result in less susceptibility of angry faces to be affected by context information. While the effect of surprised compared to neutral contextual expression on the LPP is dependent on “target expression”, a study by Richard et al. (2013) showed that fearful compared to neutral contextual expressions decreased the cognitive encoding reflected in the LPP of target faces, regardless of their expression (fearful, neutral and happy), at least for highly anxious individuals. Unlike surprised expressions that are emotionally ambiguous, fearful expressions signal potential threat and harm. Fearful expressions seem to capture more attentional resources than do threat-unrelated expressions (e.g., Batty & Taylor, 2003; Phelps, Ling & Carrasco, 2006; Pourtois, Grandjean, Sander, & Vuilleumier, 2004), so that fearful contextual expressions may disturb cognitive encoding to a larger extent than surprised faces in the present study do. However, as our present and Richard et al.’s (2013) study differ not only in contextual expressions but also in some other factors (e.g., target expressions and group sample), this explanation remains to be investigated in future studies.
Effects of “contextual expression” on recognition performance of target faces Another important finding in the present study was that later recognition rates were lower for happy encoded target faces after a sequence of surprised as compared to neutral contextual expressions, regardless of whether these target faces showed the encoded or a non-encoded expression during the recognition phase. As the valence of the contextual 14
faces was kept constant, the present finding may suggest an effect of arousal: the subsequent recognition of happy faces is impaired by high arousal of surprised contextual faces more than by low arousal of neutral contextual faces. The finding is partially in line with our previous study (Lin et al., 2015b), in which the recognition of target faces was impaired after a sequence of fearful contextual expressions as compared to neutral contextual expressions when target faces showed the encoded expression. Similar to the context effect related to face encoding, our previous study (Lin et al., 2015b) was not able to indicate whether the context effect on the recognition of target faces depends on negative valence of facial expressions or other factors (e.g., arousal). The present findings show that negative valence or threat relevance of contextual expressions is not necessary to impair later recognition of target faces. However, our previous study reported that the recognition effects of contextual expression did not generalize, when fearfully and neutrally encoded target faces showed neutral and fearful expressions, respectively, during the recognition phase (Lin et al., 2015b). A bias for reporting emotional faces as “old” has been suggested, particularly when faces are difficult to recognize (e.g., when target faces are presented with a new expression; e.g., Sharot, Delgado, & Phelps, 2004). Therefore, in our previous study (Lin et al., 2015b), participants might have had a tendency of reporting fearfully recognized target faces as “old” and neutrally recognized target faces as “new”, irrespectively of the encoded context, resulting in a general reduction of the context effect. However, the present study showed no significant bias for target expressions during the recognition phase (as the interaction “consistency” and “target expression” was not significant), possibly enhancing the difference of the context effect even though target faces showed a new expression during the recognition phase. In addition, surprised compared to neutral context affected recognition rates only for happy but not of angry target faces. As face encoding is suggested to be the first step of face memory (Bruce & Young, 1986), it is an important prerequisite for later steps, such as recognition. In the present study, ERP data suggested that surprised compared to neutral contextual faces resulted in poorer encoding of happy but not of angry target faces. This valence-dependent effect of encoding might have also influenced later steps of face memory, resulting in a similar effect during recognition.
Limitations and future directions 15
As mentioned in the results section, the right hemispheric N170 effect of surprised context was numerically only prominent for happy but not for angry faces. However, this exploratory post-hoc analysis was not justified by a higher-order interaction of “contextual expression”, “target expression” and hemisphere. As we did not determine the required sample size according to potential (but unknown) effects prior to the study, the absence might be due to an insufficient sample size. Future studies might take this into account to further investigate the contextual effects. In addition, according to Tables 2 and 3, there seem to be effects of “target expression” in the neutral face context on the N170 and LPP. However, further analysis did not reveal significant N170 and LPP effects of “target expression” in the neutral face context (please see more in the result section). Although these findings were similar to those in previous studies (e.g., Batty & Taylor, 2003; Calvo & Beltrán, 2013, 2014; Müller-Bardorff et al., 2016), we cannot rule out an absence of differential effects due to the small sample size. Therefore, this issue remains to be further investigated also by expanding the sample size. Previous studies with positive and negative stimuli suggested that the emotional ERP effects are more likely due to enhanced attention attracted by emotional stimuli instead of reduced attention by neutral stimuli. However, unlike positive and negative stimuli, surprised contextual faces in the present study were emotionally neutral. Therefore, the observed effects of context on N170 and LPP cannot definitively be attributed to surprised contextual faces or to neutral contextual faces. Future studies may investigate this issue for more details. Finally, while we discussed that the differences of the contextual LPP effects in our present and Richard et al.’s (2013) study are related to differential distractions between fearful and surprised contextual expressions, no studies - to the best of our knowledge have ever investigated whether fearful as compared to surprised contextual faces influence the LPP to target faces. Future studies may investigate this issue in more detail.
Conclusion The present study investigated the influence of surprised contextual facial expressions as compared to neutral expressions on the encoding and recognition of emotional target faces. We observed that during the encoding phase, surprised as compared to neutral contextual expressions led to smaller N170 for happy target faces but not angry target faces, though the effect of “contextual expression” for happy target faces was not significantly 16
different from the effect for angry faces. The LPP was smaller for happy target faces in a sequence of surprised as compared to neutral contextual faces; whereas this effect was not evident for angry target faces. In the later recognition phase, the recognition of happy encoded target faces was found to be poorer when the faces had been presented in a sequence of surprised as compared to neutral contextual expressions. More importantly, this context effect appeared even when the expression of target faces shifted to angry during the recognition phase. However, there was no context effect for angry encoded target faces. Taken together, our findings indicate that preceding emotionally ambiguous compared to neutral contextual expressions differentially affect the structural and cognitive encoding and subsequent recognition of happy target faces, while angry faces are more resistant to contextual interference.
Acknowledgements This work is supported by Research Program for Humanities and Social Science granted by the Ministry of Education in China (17YJC190016).
References Balconi, M., & Mazza, G. (2009). Consciousness and emotion: ERP modulation and attentive vs. pre-attentive elaboration of emotional facial expressions by backward masking. Motivation and Emotion, 33(2), 113-124. Batty, M., & Taylor, M. J. (2003). Early processing of the six basic facial emotional expressions. Cognitive Brain Research, 17(3), 613-620. Bruce, V., & Young, A. (1986). Understanding face recognition. British Journal of Psychology, 77(3), 305-327. Calvo, M. G., & Beltrán, D. (2013). Recognition advantage of happy faces: tracing the neurocognitive processes. Neuropsychologia, 51(11), 2051-2061. Calvo, M. G., & Beltrán, D. (2014). Brain lateralization of holistic versus analytic processing of emotional facial expressions. NeuroImage, 92, 237-247. Carretié, L., Hinojosa, J. A., Martín-Loeches, M., Mercado, F., & Tapia, M. (2004). Automatic attention to emotional stimuli: neural correlates. Human Brain Mapping, 22(4), 290-299.
17
Crist, R. E., Wu, C. T., Karp, C., & Woldorff, M. G. (2007). Face processing is gated by visual spatial attention. Frontiers in Human Neuroscience, 1, article 10. Deffke, I., Sander, T., Heidenreich, J., Sommer, W., Curio, G., Trahms, L., & Lueschow, A. (2007). MEG/EEG sources of the 170-ms response to faces are co-localized in the fusiform gyrus. NeuroImage, 35(4), 1495-1501. Dillon, D. G., Cooper, J. J., Grent, T., Woldorff, M. G., & LaBar, K. S. (2006). Dissociation of event-related potentials indexing arousal and semantic cohesion during emotional word encoding. Brain and Cognition, 62(1), 43-57. Ebner, N. C., Riediger, M., & Lindenberger, U. (2010). FACES — A database of facial expressions in young, middle-aged, and older women and men: Development and validation. Behavioral Research Methods, 42(1), 351-362. Eimer, M. (2000a). Effects of face inversion on the structural encoding and recognition of faces: Evidence from event-related brain potentials. Cognitive Brain Research, 10(1), 145158. Eimer, M. (2000b). The face-specific N170 component reflects late stages in the structural encoding of faces. Neuroreport, 11(10), 2319-2324. Eimer, M., & McCarthy, R. A. (1999). Prosopagnosia and structural encoding of faces: Evidence from event‐related potentials. Neuroreport, 10(2), 255-259. Flaisch, T., Stockburger, J., & Schupp, H. T. (2008). Affective prime and target picture processing: an ERP analysis of early and late interference effects. Brain Topography, 20(4), 183-191. Furl, N., van Rijsbergen, N. J., Treves, A., Friston, K. J., & Dolan, R. J. (2007). Experiencedependent coding of facial expression in superior temporal sulcus. Proceedings of the National Academy of Sciences, 104(33), 13485-13489. Johansson, M., Mecklinger, A., & Treese, A. C. (2004). Recognition memory for emotional and neutral faces: An event-related potential study. Journal of Cognitive Neuroscience, 16(10), 1840-1853. Hagemann, J., Straube, T., & Schulz, C. (2016). Too bad: Bias for angry faces in social anxiety interferes with identity processing. Neuropsychologia, 84, 136-149.
18
Henson, R. N. A., Shallice, T., Gorno-Tempini, M. L., & Dolan, R. J. (2002). Face repetition effects in implicit and explicit memory tests as measured by fMRI. Cerebral Cortex, 12(2), 178-186. Holmes, A., Nielsen, M. K., & Green, S. (2008). Effects of anxiety on the processing of fearful and happy faces: an event-related potential study. Biological Psychology, 77(2), 159-173. Holmes, A., Vuilleumier, P., & Eimer, M. (2003). The processing of emotional facial expression is gated by spatial attention: evidence from event-related brain potentials. Cognitive Brain Research, 16(2), 174-184. Labuschagne, I., Croft, R. J., Phan, K. L., & Nathan, P. J. (2010). Augmenting serotonin neurotransmission with citalopram modulates emotional expression decoding but not structural encoding of moderate intensity sad facial emotional stimuli: an event-related potential (ERP) investigation. Journal of Psychopharmacology, 24(8), 1153-1164. Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H., Hawk, S. T., & van Knippenberg, A. (2010). Presentation and validation of the Radboud Faces Database. Cognition and Emotion, 24(8), 1377-1388. Lin, H., Schulz, C., & Straube, T. (2015a). Cognitive tasks during expectation affect the congruency ERP effects to facial expressions. Frontiers in Human Neuroscience, 9, article 596. Lin, H., Schulz, C., & Straube, T. (2015b). Fearful contextual expression impairs the encoding and recognition of target faces: an ERP study. Frontiers in Behavioral Neuroscience, 9, article 237. Lin, H., Schulz, C., & Straube, T. (2016). Effects of expectation congruency on event-related potentials (ERPs) to facial expressions depend on cognitive load during the expectation phase. Biological Psychology, 120, 126-136. Lundqvist, D., Flykt, A., & Öhman, A. (1998). The Karolinska directed emotional faces – KDEF. CD-ROM from Department of Clinical Neuroscience, Psychology section. Sweden: Karolinska Institute. Stockholm ([ISBN: 91-630-7164-9]). Meeren, H. K., van Heijnsbergen, C. C., & de Gelder, B. (2005). Rapid perceptual integration of facial expression and emotional body language. Proceedings of the National Academy of Sciences of the United States of America, 102(45), 16518-16523.
19
Michalowski, J. M., Pané-Farré, C. A., Löw, A., & Hamm, A. O. (2015). Brain dynamics of visual attention during anticipation and encoding of threat-and safe-cues in spider-phobic individuals. Social Cognitive and Affective Neuroscience, 10(9), 1177-1186. Mohamed, T. N., Neumann, M. F., & Schweinberger, S. R. (2009). Perceptual load manipulation reveals sensitivity of the face-selective N170 to attention. Neuroreport, 20(8), 782-787. Morel, S., Ponz, A., Mercier, M., Vuilleumier, P., & George, N. (2009). EEG-MEG evidence for early differential repetition effects for fearful, happy and neutral faces. Brain Research, 1254, 84-98. Müller-Bardorff, M., Schulz, C., Peterburs, J., Bruchmann, M., Mothes-Lasch, M., Miltner, W., & Straube, T. (2016). Effects of emotional intensity under perceptual load: An eventrelated potentials (ERPs) study. Biological Psychology, 117, 141-149. Neta, M., Davis, F. C., & Whalen, P. J. (2011). Valence resolution of ambiguous facial expressions using an emotional oddball task. Emotion, 11(6), 1425-1433. Neta, M., & Whalen, P. J. (2010). The primacy of negative interpretations when resolving the valence of ambiguous facial expressions. Psychological Science, 21(7), 901-907. Oldfield, R. C. (1971). The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia, 9(1), 97-113. Pinabiaux, C., Bulteau, C., Fohlen, M., Dorfmüller, G., Chiron, C., Hertz-Pannier, L., ... & Jambaqué, I. (2013). Impaired emotional memory recognition after early temporal lobe epilepsy surgery: the fearful face exception?. Cortex, 49(5), 1386-1393. Phelps, E. A., Ling, S., & Carrasco, M. (2006). Emotion facilitates perception and potentiates the perceptual benefits of attention. Psychological Science, 17(4), 292-299. Polich, J. (2007). Updating P300: an integrative theory of P3a and P3b. Clinical Neurophysiology, 118(10), 2128-2148. Polich, J., & Criado, J. R. (2006). Neuropsychology and neuropharmacology of P3a and P3b. International Journal of Psychophysiology, 60(2), 172-185. Pourtois, G., Grandjean, D., Sander, D., & Vuilleumier, P. (2004). Electrophysiological correlates of rapid spatial orienting towards fearful faces. Cerebral Cortex, 14(6), 619-633.
20
Richards, A., Holmes, A., Pell, P. J., & Bethell, E. J. (2013). Adapting effects of emotional expression in anxiety: Evidence for an enhanced Late Positive Potential. Social Neuroscience, 8(6), 650-664. Righi, S., Marzi, T., Toscani, M., Baldassi, S., Ottonello, S., & Viggiano, M. P. (2012). Fearful expressions enhance recognition memory: electrophysiological evidence. Acta Psychologica, 139(1), 7-18. Ryu, V., An, S. K., Jo, H. H., & Cho, H. S. (2010). Decreased P3 amplitudes elicited by negative facial emotion in manic patients: selective deficits in emotional processing. Neuroscience Letters, 481(2), 92-96. Schubö, A., Schröger, E., & Meinecke, C. (2004). Texture segmentation and visual search for pop-out targets: An ERP study. Cognitive Brain Research, 21(3), 317-334. Schupp, H. T., Öhman, A., Junghöfer, M., Weike, A. I., Stockburger, J., & Hamm, A. O. (2004). The facilitated processing of threatening faces: An ERP Analysis. Emotion, 4(2), 189-200. Sessa, P., Luria, R., Gotler, A., Jolicœur, P., & Dell'Acqua, R. (2011). Interhemispheric ERP asymmetries over inferior parietal cortex reveal differential visual working memory maintenance for fearful versus neutral facial identities. Psychophysiology, 48(2), 187-197. Sharot, T., Delgado, M. R., & Phelps, E. A. (2004). How emotion enhances the feeling of remembering. Nature Neuroscience, 7(12), 1376-1380. Shimamura, A. P., Ross, J. G., & Bennett, H. D. (2006). Memory for facial expressions: The power of a smile. Psychonomic Bulletin & Review, 13(2), 217-222. Smith, E., Weinberg, A., Moran, T., & Hajcak, G. (2013). Electrocortical responses to NIMSTIM facial expressions of emotion. International Journal of Psychophysiology, 88(1), 17-25. Sprengelmeyer, R., & Jentzsch, I. (2006). Event related potentials and the perception of intensity in facial expressions. Neuropsychologia, 44(14), 2899-2906. Tottenham, N., Tanaka, J. W., Leon, A. C., McCarry, T., Nurse, M., Hare, T. A., ... & Nelson, C. (2009). The NimStim set of facial expressions: judgments from untrained research participants. Psychiatry Research, 168(3), 242-249. Vuilleumier, P. (2005). How brains beware: neural mechanisms of emotional attention. Trends in Cognitive Sciences, 9(12), 585-594.
21
Tables and table caption Table 1. Mean of accuracy (%), response times (ms) and their SE for all the experimental conditions during the encoding phase. Contextual Expression Target Expression
Surprised
Neutral
M
SE
M
SE
Angry
96.67
0.81
96.74
0.76
Happy
95.91
0.89
97.12
0.71
Angry
536.84
17.75
538.97
14.49
Happy
516.75
13.76
529.48
17.01
Accuracy
Response times
22
Table 2. Mean of the N170 amplitudes (μV) and its SE for all the experimental conditions during the encoding phase. Contextual Expression Hemisphere
Target Expression
Surprised
Neutral
M
SE
M
SE
Angry
-3.84
0.93
-3.82
0.89
Happy
-3.82
0.84
-3.82
0.94
Angry
-3.64
0.69
-3.91
0.65
Happy
-3.73
0.64
-4.45
0.77
P7
P8
23
Table 3. Mean of the early LPP amplitudes (μV) and its SE for all the experimental conditions during the encoding phase. Contextual Expression Electrode
Target Expression
Surprised
Neutral
M
SE
M
SE
Angry
0.10
3.05
-0.03
2.35
Happy
-0.53
2.31
0.95
2.54
Angry
3.08
3.81
3.11
3.44
Happy
2.32
3.57
4.00
3.41
Angry
3.11
3.62
2.14
3.74
Happy
1.90
3.62
3.38
3.44
Angry
6.41
4.79
5.92
4.18
Happy
5.11
5.40
7.02
4.29
Angry
6.92
3.25
6.86
2.93
Happy
6.03
3.88
7.35
3.21
Angry
6.53
4.18
6.24
3.30
Happy
5.58
3.89
6.88
3.74
Angry
7.00
4.13
6.95
3.08
Happy
6.22
3.99
6.92
2.95
Fz
FC1
FC2
Cz
CP1
CP2
Pz
24
Table 4. Mean of accuracy (%), response times (ms) and their SE for all the experimental conditions during the recognition phase. Contextual Expression Consistency
Target Expression
Surprised
Neutral
M
SE
M
SE
Angry
36.67
2.99
37.42
3.56
Happy
32.88
2.72
38.33
3.55
Angry
31.67
3.30
30.76
2.97
Happy
30.91
2.76
33.64
2.97
Angry
799.94
58.14
864.31
67.38
Happy
796.35
70.55
805.68
64.71
Angry
851.60
74.23
803.66
59.33
Happy
804.90
69.02
837.81
68.48
Accuracy Unchanged
Changed Response times Unchanged
Changed
25
Figure caption Figure 1. A chart regarding the assignments of different faces (i.e., target, contextual and novel faces). (A) Two identities (1 female and 1 male) with surprised and neutral facial expressions served as contextual identities. 120 identities (60 females and 60 males; 120 angry and 120 happy) were used as novel identities. The remaining 122 identities (61 females and 61 males, including 2 practice identities) with angry and happy facial expressions were used as targets. For target identities used in the actual experiment, they were separated randomly into four sets with 30 identities (15 females and 15 males; 30 angry and 30 happy) each. In each set, there were angry and happy sub-sets. (B) In the encoding phase, two angry sub-sets [e.g., (Female and male) - Angry: No. 01-15 and 16-30] and two happy sub-sets [e.g., (Female and male) - Happy: No. 31-45 and 46-60] with different identities were used to be presented with either surprised or neutral contextual faces. In the recognition phase, old faces in all sub-sets, including those in unchanged and changed expressions [e.g., (Female and male) - Happy: No. 01-15 and 16-30 and (Female and male) - Angry: No. 31-45 and 46-60], and novel faces [e.g., (Female and male) - Angry: No. 63-122 and (Female and male) - Happy: No. 63-122] were presented.
Figure 2. Event-related potentials (ERPs) at the electrodes P7 and P8 for all the experimental conditions during the encoding phase. Shaded areas correspond to the time window for the N170 (130-180 ms).
Figure 3. Event-related potentials (ERPs) at electrodes Fz, FC1, FC2, Cz, CP1, CP2 and Pz for all the experimental conditions during the encoding phase. Shaded areas correspond to the time window for LPP (350-800 ms).
Figure 4. Topographical maps based on N170 (130-180 ms) and LPP (350-800 ms) mean amplitudes for all the experimental conditions during the encoding phase.
26
27
28
Figr-1
29
Figr-2
30
Figr-3
31
Figr-4
32
S0301-0511(17)30246-6 http://dx.doi.org/10.1016/j.biopsycho.2017.09.011 BIOPSY 7440
To appear in: Received date: Revised date: Accepted date:
14-1-2017 27-8-2017 18-9-2017
Please cite this article as: Lin, Huiyan, Schulz, Claudia, Straube, Thomas, Contextual effects of surprised expressions on the encoding and recognition of emotional target faces: an event-related potential (ERP) study.Biological Psychology http://dx.doi.org/10.1016/j.biopsycho.2017.09.011 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Contextual effects of surprised expressions on the encoding and recognition of emotional target faces: an event-related potential (ERP) study Running title: surprised context effects on encoding and recognition Huiyan Lin a, b, c, Claudia Schulz d, Thomas Straube c a
Institute of Applied Psychology, Guangdong University of Finance, 510521
Guangzhou, China b
Laboratory for Behavioral and Regional Finance, Guangdong University of Finance,
510521 Guangzhou, China c
Institute of Medical Psychology and Systems Neuroscience, University of Muenster,
48149 Muenster, Germany d
Department of Clinical Psychology and Psychotherapy, University of Muenster,
48149 Muenster, Germany Correspondence: Huiyan Lin Institute of Applied Psychology Guangdong University of Finance No. 527 Yingfu Road, Tianhe District, 510521, Guangzhou, China Email: [email protected] Tel.: +86-020-37215850
1
Highlights:
Face encoding and recognition are impaired by negative contextual expressions.
Whether other contextual expressions affect face encoding and recognition is unknown.
Surprised compared to neutral context reduced encoding-LPP for happy target faces.
A context effect was shown in N170, but it did not interact with target expression.
In later recognition, there was a context effect in happy-encoded target faces.
Abstract Several studies reported that the encoding and recognition of emotional target faces are modulated by negative contextual expressions. However, it is unknown whether other contextual expressions, such as emotionally ambiguous expressions, affect the encoding and recognition of target faces. To this end, electroencephalography (EEG) was recorded during the presentation of angry or happy target faces after a random sequence of surprised or neutral contextual faces that did not differ in normative valence ratings. Subsequently, participants were asked to perform an unexpected old/new recognition task in which target faces were shown in either the encoded or a non-encoded expression. During the encoding phase, event-related potential (ERP) results showed that surprised as compared to neutral contextual faces led to smaller late positive potentials (LPP) for happy but not for angry target faces. Similar effects were also observed in the N170, even though the interaction of context and target expression failed to reach statistical significance. In the later recognition phase, recognition rates were lower for encoded happy faces when they had been encountered in surprised as compared to neutral context, regardless of whether the target face showed the encoded or a non-encoded expression. However, this context effect was not observed for angry-encoded faces. Taken together, the present study indicates that ambiguous contextual expressions, e.g., surprised faces, affect structural and cognitive encoding and later recognition of happy target faces to a larger extent than neutral contextual faces; whereas angry faces are more resistant to context effects. Keywords: surprised context; angry faces; happy faces; event-related potentials (ERPs); encoding; recognition.
2
Introduction Facial expressions convey a wide variety of interpersonal information about individuals’ emotion and motivation. For example, negative expressions (e.g., fear, sadness, disgust and anger) signify potential harm and threat, a happy expression signals benefits and rewards, and a surprised expression conveys signals of emotional and motivational ambiguity. A number of studies have investigated whether facial expressions influence face encoding and recognition using electrophysiological (e.g., N170 and late positive potential, LPP) and behavioral parameters, respectively (e.g., Balconi & Mazza, 2009; Batty & Taylor, 2003; Calvo & Beltrán, 2013, 2014; Hagemann, Straube, Schulz, 2016; Holmes, Nielsen, & Green, 2008; Johansson, Mecklinger, & Treese, 2004; Morel, Ponz, Mercier, Vuilleumier, & George, 2009; Müller-Bardorff et al., 2016; Lin, Schulz, & Straube, 2015a, 2015b, 2016; Pinabiaux et al., 2013; Schupp et al., 2004; Sessa, Luria, Gotler, Jolicœur, & Dell'Acqua, 2011; Righi et al., 2012; Shimamura, Ross, & Bennett, 2006; Smith, Weinberg, Moran, & Hajcak, 2013). Structural face encoding is often supposed to be reflected by the N170 (Eimer, 2000a, 2000b; Eimer & McCarthy, 1999) in event-related potential (ERP) studies, a face-sensitive component that peaks around 170 ms after stimulus onset and is maximal at occipitotemporal scalp sites. The N170 is relevant to pre-categorical perceptual and structural encoding of faces in face-specific ventral visual areas (e.g., Eimer, 2000a, 2000b; Eimer & McCarthy, 1999; Labuschagne, Croft, Phan, & Nathan, 2010). It has repeatedly been shown that negative faces (e.g., fearful and angry faces) as compared to neutral faces evoke larger N170 amplitudes (e.g., Batty & Taylor, 2003; Calvo & Beltrán, 2013, 2014; Hagemann et al., 2016; Lin et al., 2015a, 2015b, 2016). N170 amplitudes are found to be enlarged also by happy and surprised faces (e.g., Calvo & Beltrán, 2014; Morel et al., 2009), especially for happy faces with strong emotional intensity (Müller-Bardorff et al., 2016). Increased structural encoding of emotional expressions reflected by the N170 has been suggested to be due to the mobilization of attention (Batty & Taylor, 2003; Lin et al., 2015a). The late positive potential (LPP; often overlapping with the P3), a positive deflection which starts at approximately 300 ms after stimulus presentation over frontal-parietal scalp sites, is also supposed to be involved in stimulus encoding (e.g., Dillon, Cooper, Grent, Woldorff, & LaBar, 2006; Michalowski, Pané-Farré, Löw, & Hamm, 2015; Schupp et al., 2004). In contrast to the N170, the LPP is thought to be associated with higher-order and complex cognitive encoding processes (e.g., Labuschagne et al., 2010). Importantly, studies have shown effects of facial expressions on LPP amplitude (e.g., Balconi & Mazza, 2009; Holmes, Nielsen, & Green, 2008; Righi et al., 2012; Schupp et al., 2004; Smith et al., 2013). 3
For example, Righi et al. (2012) found that fearful and happy as compared to neutral faces evoked larger LPP amplitudes. In the study of Balconi and Mazza (2009), surprised faces were also shown to elicit greater LPP amplitudes than neutral faces. Similar to the N170, LPP effects in cognitive encoding of emotional faces have also been attributed to the mobilization of attention (e.g., Polich, 2007; Polich & Criado, 2006). Additionally, although studies have investigated whether recognition of face identity is affected by facial expressions, results are still rather mixed. Some studies reported that negative expressions enhanced face recognition (e.g., Pinabiaux et al., 2013; Sessa et al., 2011), whereas others failed to observe similar effects (e.g., Johansson et al., 2004; Righi et al., 2012) or even showed enhanced recognition of happy or neutral faces (e.g., Hagemann et al., 2016; Shimamura et al., 2006). Discrepancies may be due to whether recognized faces are presented with the encoded expressions or changed into a new expression during the recognition phase (Hagemann et al., 2016; Lin et al., 2015a), whether emotional expressions are shown in the encoding or the recognition phase (Chen et al., 2015), and whether the recognition task is expected or unexpected (e.g., Henson, Shallice, Gorno-Tempini, & Dolan, 2002). In daily life, there are many circumstances where faces are not viewed in isolation, but in the context of other emotional stimuli, such as preceding contextual expressions. Several previous studies have shown that preceding contextual faces with a negative expression modulate the encoding of target faces (Furl, van Rijsbergen, Treves, Friston & Dolan, 2007; Lin et al., 2015a; Richards et al., 2013). For example, fearful compared to neutral contextual expressions reduced the M170 (which is thought to be the magnetoencephalography (MEG) counterpart of the N170; Deffke et al., 2007) or N170 to upcoming target faces, regardless of the expression of target faces (Furl et al., 2007; Lin et al., 2015a). Fearful compared to neutral contextual expressions were also found to evoke smaller LPP amplitudes for target faces irrespective of the expression of target faces, at least for highly anxious individuals (Richards et al., 2013). Moreover, we found that recognition rates were reduced for target faces that had been encoded in the context of fearful compared to neutral expressions. This effect was observed particularly when target faces showed the encoded expression (Lin et al., 2015a). Taken together, these findings indicate that fearful contextual expressions impair the encoding and recognition of target faces, probably due to distraction (Lin et al., 2015a). However, to the best of our knowledge, it is still unknown whether the encoding and recognition of target faces are affected by other contextual expressions (e.g., emotionally ambiguous expressions) and whether contextual effects of ambiguous expressions are 4
affected by the valence of target faces. Therefore, the present study aimed at addressing these questions by using an encoding task and a subsequent recognition task. For the encoding task, participants were asked to indicate the presence of angry or happy target faces in a sequence of emotionally ambiguous (e.g., surprised) or neutral contextual faces. Target and contextual faces were of different identities in order to investigate the recognition effects of target faces in a later recognition task. Considering that some surprised faces are perceived as positive or negative (Neta, Davis, & Whalen, 2011; Neta & Whalen, 2010) and that the present study aimed at investigating effects of emotionally ambiguous contextual faces, we selected surprised faces whose valence was rated neutral, similar to the neutral faces. In addition, electroencephalography (EEG) was recorded during this phase. Subsequently, participants were presented with all target faces and novel faces. Participants were asked to judge whether the prompted face had been presented in the preceding phase (old/target faces) or not (novel faces). Target faces were presented in either the encoded or a non-encoded expression. The recognition task was unexpected. Participants were not informed about the recognition task until the end of the encoding phase. Previous studies have suggested that high arousing negative contextual expressions modulate the encoding and recognition of target faces, possibly as a result of the negative emotion and/or the high arousal of these expressions (Furl et al., 2007; Lin et al., 2015a; Richards et al., 2013). As surprised expressions are also thought to be highly arousing, we predicted that surprised as compared to neutral contextual facial expressions should reduce the encoding of target faces, which is reflected in the N170 and LPP, as well as their later recognition, especially when target faces show the encoded expression. In addition, angry target faces might be less affected by surprised context than happy target faces; as there seems to be a stronger attentional bias for angry than for happy faces among surprised faces (Calvo & Beltrán, 2014).
Methods Participants Twenty-four participants were recruited in Muenster via advertisement and received 10 Euros for participation. Two participants were excluded from statistical analysis due to excessive artifacts in the EEG signal, resulting in a total of 22 participants (20-35 years, M = 24.57, SD = 3.29; 15 females). All participants were right-handed as determined by the Edinburgh Handedness Inventory (Oldfield, 1971). Participants had normal or corrected-tonormal vision and did not report a history of neurological illness. All participants gave written informed consent in line with standard ethical guidelines from the Declaration of 5
Helsinki. The study was approved by the local ethics committee of the University of Muenster. Stimuli Stimuli were 488 digitized color pictures that portrayed 244 identities (122 females and 122 males) with two different expressions each. Of these identities, 242 identities (121 females and 121 males) with angry and happy expressions were selected from FACES (Ebner et al., 2010), Karolinska Directed Emotional Faces (KDEF, Lundqvist et al., 1998), NimStim (Tottenham et al., 2009) and Radboud Faces Database (RaFD, Langner et al., 2010). The other two identities (1 female and 1 male) showing surprised and neutral expressions were selected from RaFD only. Surprised and neutral faces were similar in valence ratings (surprised versus neutral = 3.0 versus 3.1; ranging from 1 “very unpleasant” to 5 “very pleasant”), but higher in arousal ratings (surprised versus neutral = 4.1 versus 3.2; ranging from 1 “very low” to 5 “very high”) according to the supporting materials of the original study (Langner et al., 2010). Mean recognition accuracy for surprised and neutral faces was 100% and 86%, respectively. As facial pictures were derived from different databases, the pictures were cropped similarly around the face outline, and centered so that eyes, nose and mouth were at similar positions for all faces. We also removed external features (e.g., neck, shoulder, distant hair and jewelry), adjusted the size to 7.49 cm (horizontal) × 9.99 cm (vertical) and matched the pictures in luminance, contrast, hue and color using Adobe Photoshop CS6. The two identities with surprised and neutral facial expression served as contextual identities during the encoding phase. The other 122 identities (61 females and 61 males; including 1 female and 1 male for the practice) with angry and happy expressions were used as target identities. The remaining 240 pictures of 120 identities (60 females and 60 males; 120 angry and 120 happy) were used as novel facial pictures during the recognition phase. Target faces in the actual experiment were separated into four sets pseudo-randomly with 30 identities (15 females and 15 males; 30 angry and 30 happy) each. Each set was separated into two sub-sets that were identical in identity (30 identities), but different in facial expression. In the encoding phase, half of the sub-sets with different identities were used to create four experimental conditions: angry targets among surprised context faces, angry targets among neutral context faces, happy targets among surprised context faces and happy targets among neutral context faces. Assignments of sub-sets were counterbalanced across participants. During the recognition phase, all sub-sets were presented to create two conditions for learned faces: changed and unchanged (with regard to facial expressions). 6
Novel identities were also presented with two expressions during the recognition phase. For more details, please refer to Figure 1. (Insert Figure 1 about here) Procedure After informed consent had been given and handedness had been determined, participants were asked to sit in a comfortable chair at a viewing distance of approximately 80 cm from a 15-inch CRT computer screen which resolution was 1280 by 1024 pixel, so that the visual angle of the face stimuli was 5.37° × 7.15°. Stimuli were presented on a black background. Stimulus presentation and behavioral data collection were controlled with EPrime 2.0 software (Psychology Software Tools, Inc., Pittsburgh, PA, USA). Before the experiment started, participants had to familiarize themselves with the four contextual facial pictures. Participants were told that they would be presented with these (contextual) faces and some other (target) faces. They were also told that their task was to indicate the familiarity of the face by pressing the “F” or the “J” key with their left and right index finger, respectively, as quickly and correctly as possible. Response assignments were counterbalanced across participants. Practice trials with feedback on response accuracy and times were performed in order to ensure proper familiarization with the contextual faces and the experimental procedure. Each trial started with a white fixation cross for 200 ms. Following a blank screen for 600 to 1000 ms (M = 800 ms), either a contextual or a target face was presented for 800 ms. The next trial started after another blank screen for 1000 ms. Button presses were allowed from the onset of the face to the offset of the following blank. Faces were presented separately in four different blocks according to the expressions of contextual and target faces (angry targets among surprised context faces, angry targets among neutral context faces, happy targets among surprised context faces and happy targets among neutral context faces). Block sequence was randomized across participants. In each block, each contextual face was presented 90 times and each target face twice. Therefore, the experiment in the encoding phase consisted of a total of 960 (2 × 90 × 4 + 30 × 2 × 4) trials. There were two breaks in each block and the break duration was controlled by the participants. The recognition phase followed the encoding phase after approximately 5 minutes. Participants were first informed about the recognition task at this point in the experiment. Participants were told to indicate whether the identity of a prompted face had been presented in the preceding encoding phase (target or old identity) or not (novel identity) by 7
pressing the keys “F” or “J”. Participants were particularly informed that the old/new judgment was based on facial identity, regardless of the facial expression (e.g., Identity A showed a happy expression in the preceding task. A will either show a happy or an angry expression in the following task. Both happy and angry A should be judged as “old”). The instructions emphasized speed as well as accuracy. Response assignments were counterbalanced across participants. Each trial started with a white fixation cross which was presented for 200 ms. Following a blank screen for 600 to 1000 ms (M = 800 ms), either a target/old face or a novel face was presented for 500 ms. Another blank screen was presented for 1000 ms before the next trial started. Button presses were allowed from the onset of the face to the offset of the following blank. Each facial picture was presented once. Therefore, the experiment in the recognition phase consisted of a total of 480 trials, half of which were target/old faces and the other half novel faces. There were three breaks in this phase and the break duration was determined by participants. The complete experiment with the encoding and the recognition phase (including the preparations and practice) lasted approximately 2 hours. Behavioral recording For the encoding and the recognition phase, accuracy and response times of button presses were recorded from the onset of the target face to the offset of the following blank. EEG recording EEG was continuously recorded with a 32-channel BioSemi Active Two System (BioSemi, Amsterdam, Netherlands). Thirty-two Ag/AgCl active electrodes were placed on the scalp by means of an elastic head cap (BioSemi, Amsterdam), according to the international 10-20 system (FP1, FP2, F7, F3, Fz, F4, F8, FC1, FC5, FC2, FC6, T7, T8, C3, Cz, C4, TP9, TP10, CP1, CP2, P7, P9, P3, Pz, P4, P8, P10, PO9, PO10, O1, Oz, O2). The BioSemi System uses an active electrode (common mode sense, CMS) and a passive electrode (driven right leg, DRL) to form a feedback loop instead of ground and reference (please see http://www.biosemi.com/faq/cms&drl.htm). The horizontal electrooculogram (EOG) was recorded from two electrodes at the outer canthi of both eyes, and the vertical EOG was recorded bipolarly from two electrodes above and below the right eye to monitor eye blinks and movements. All signals were digitized with a sample rate of 2048 Hz and a low-pass fifth order sinc filter with a half-power cutoff at 100 Hz. Impedances were generally kept below 10 kΩ.
8
During offline processing, ocular artifacts were corrected using the algorithm implemented in BESA 6.0 software (www.BESA.de). The continuous EEG was then segmented from -200 to 800 ms relative to onset of the target face during the encoding phase, with the first 200 ms epoch for baseline correction. Artifact rejection was performed with the amplitude threshold of 120 μV, a gradient criterion of 75 μV and low-signal criterion of 0.01 μV. Artifact-free trials were averaged for each channel and experimental condition. Averaged ERPs were re-computed to average reference, excluding vertical and horizontal EOG channels. ERPs were then low-pass filtered at 40 Hz (Butterworth zero phase shift). ERPs were quantified using mean amplitudes for N170 and LPP, both relative to -200 ms baseline. The N170 was measured at the electrodes P7 and P8 and the LPP at the electrodes Fz, FC1, FC2, Cz, CP1, CP2 and Pz. The time windows for the N170 and LPP were 130-180 ms and 350-800 ms, respectively. The time window of the N170 was chosen according to the peak latency identified in the grand waveforms across all conditions (155 ms) and previous studies (e.g., Batty & Taylor, 2003; Lin et al., 2015a, b). The time window of the LPP and the electrodes of interest for the N170 and LPP were chosen based on visual inspection of the grand waveforms and previous studies (e.g., Batty & Taylor, 2003; Meeren, van Heijnsbergen, & de Gelder, 2005; Polich, 2007; Polich & Criado, 2006). Although EEG was also recorded during the recognition phase, ERPs could not be analyzed due to an insufficient number of correct trials (on average about 10 per experimental condition). Data analysis For the encoding and the recognition phase, mean accuracy and response times were calculated for each participant and each experimental condition. For the analysis of response times, only trials with correct responses were included. For the encoding phase, mean accuracy and response times for target faces were entered into 2 × 2 repeated measures analyses of variance (ANOVAs) with “contextual expression” (surprised versus neutral) and “target expression” (angry versus happy) as within-subject factors. For statistical analysis of ERP data, additional within-subject factors “hemisphere” (left versus right; for N170) and “electrode” (Fz versus FC1 versus FC2 versus Cz versus CP1 versus CP2 versus Pz; for LPP) were included. In the recognition phase, accuracy and response times for the target old faces were entered into 2 × 2 × 2 repeated measures ANOVAs with “consistency” (unchanged versus changed), “contextual expression” (surprised versus neutral) and “encoded target expression” (angry versus happy) as within-subject factors. Statistical analyses were performed using IBM SPSS Statistics software (Version 22; SPSS 9
INC., an IBM company, Chicago, Illinois). Greenhouse-Geisser corrections were applied to correct degrees of freedom and p values where appropriate. Bonferroni corrections were used to correct post-hoc t-tests.
Results Encoding phase Behavioral data The analyses of accuracy and response times did not reveal any main effects or interactions (ps. > .05; Table 1). (Insert Table 1 about here) ERP data N170 The analysis of the N170 revealed an interaction between “contextual expression” and “hemisphere” (F(1, 21) = 4.75, p = .041, η2p = .18). Separate analyses for each hemisphere showed that the effect of “contextual expression” was not significant over the left hemisphere (ps. > .05), while right hemispheric N170 amplitudes were larger for target faces in a sequence of neutral as compared to surprised contextual faces (F(1, 21) = 4.59, p = .044, η2p = .18). Other main effects or interactions were not significant (ps. > .05). See Table 2 and Figure 2 and 4 in more detail. As Table 2 and Figure 2 and 4 suggest, the right hemispheric effect of “contextual expression” seems prominent only for happy but not for angry target faces. Indeed, exploratory post-hoc tests for each target expression revealed that the effect of “contextual expression” was only significant for happy target faces (F(1, 21) = 4.41, p = .048, η2p = .17), but not for angry target faces (ps. > .05), suggesting a potentially stronger effect of contextual expression for happy target faces. Table 2 also suggests that the right hemispheric N170 was larger for happy compared to angry target faces particularly in the neutral face context. However, exploratory post-hoc tests did not reach statistical significance (p > 0.05). (Insert Table 2 and Figure 2 about here)
10
LPP The analysis showed main effects of “contextual expression” (F(1, 21) = 5.29, p = .032, η2p = .20) and “electrode” (F(1, 21) = 44.23, p < .001, η2p = .70). The LPP was more positive for target faces in a sequence of neutral as compared to surprised contextual faces. The LPP was also larger at electrode Pz as compared to Fz (p < .001), FC1 (p = .004) and FC2 (p = .004), at CP1 and CP2 as compared to Fz, FC1 and FC2 (all p < .001), and at FC1 and FC2 as compared to Fz (all p < .001). There was a significant interaction between “contextual expression” and “target expression” (F(1, 21) = 4.54, p = .045, η2p = .18). Separate analyses for each category of “target expression” showed that the LPP was larger for happy target faces in a sequence of neutral as compared to surprised contextual faces (F(1, 21) = 6.96, p = .015, η2p = .25); whereas the effect of “contextual expression” was not significant for angry target faces (ps. > .05). Notably, separate analyses for each category of “contextual expression” revealed no significant effects of “target expression” (p > 0.05). See Table 3 and Figure 3 and 4 in more detail.
(Insert Table 3 and Figure 3 and 4 about here)
Recognition phase Accuracy We found a main effect of “consistency” (F(1, 21) = 20.10, p < .001, η2p = .49), with higher accuracy for target faces showing the encoded expression as compared to faces showing a non-encoded expression. More importantly, the interaction between “contextual expression” and “encoded target expression” was significant (F(1, 21) = 4.85, p = .039, η2p = .19). Accuracy was higher for happy encoded target faces in a sequence of neutral as compared to surprised contextual faces (F(1, 21) = 4.96, p = .037, η2p = .19). For angry encoded target faces, however, the effect of “contextual expression” was not significant (ps. > .05). Other main effects or interactions did not reach statistical significance (ps. > .05). For descriptive data, please refer to Table 4.
Response times 11
No main effects or interactions were significant (ps. > .05). For descriptive data, please see Table 4.
(Insert Table 4 about here)
Discussion The present study aimed to investigate whether surprised contextual facial expressions as compared to neutral expressions modulated the encoding and recognition of happy and angry target faces. Results showed that during the encoding phase, the N170 was smaller for happy but not for angry target faces when these faces were presented in a sequence of surprised as compared to neutral contextual expressions. However, the effects of “contextual expression” did not significantly differ between angry and happy target faces. The LPP was smaller for happy target faces within surprised as compared to neutral contextual expressions; whereas the LPP to angry target faces was not affected by the contextual expression. In a later recognition phase, recognition rates were lower when these target faces had been encountered in surprised as compared to neutral contextual expressions, regardless of whether target faces showed the encoded expression or not. However, we did not observe any context effects for angry encoded target faces.
Effects of “contextual expression” on the encoding of target faces Similar to our previous study (Lin et al., 2015b), we did not find context effects on the behavioral classification of target faces during the encoding phase. This might be due to the low task difficulty. However, there was a clear effect on the corresponding ERPs, with smaller N170 and LPP especially for happy target faces after a sequence of surprised as compared to neutral contextual expressions. Given that the N170 and LPP are related to structural and complex cognitive encoding processes, respectively (Dillon et al., 2006; Eimer, 2000a, 2000b; Eimer & McCarthy, 1999; Labuschagne et al., 2010; Michalowski, et al., 2015; Schupp et al., 2004), the findings in the present study indicate that structural and cognitive encoding of happy target faces are reduced when these faces are shown in a sequence of ambiguous (e.g., surprised) as compared to neutral contextual expressions. While the effect of context expression on N170 did not differ between happy and angry target faces, context affected the processing of happy faces in the LPP time window (see below). 12
These context effects may be explained by available attentional resources during the presentation of target faces (Lin et al., 2015b). Considering that attentional resources are occupied by preceding stimuli in general (Flaisch, Stockburger & Schupp, 2008) and that emotional as compared to neutral stimuli are often thought to occupy more attentional resources (e.g., Carretié, Hinojosa, Martín-Loeches, Mercado, & Tapia, 2004; Vuilleumier, 2005), attention should be reduced for target stimuli that are preceded by emotional as compared to neutral stimuli. Furthermore, attention is supposed to affect face encoding as reflected in the N170 (e.g., Crist, Wu, Karp, & Woldorff, 2007; Mohamed, Neumann, & Schweinberger, 2009), with larger amplitudes for faces that are attended as compared to those that are not (e.g., Holmes, Vuilleumier, & Eimer, 2003). Attention also influences stimulus processing shown in the LPP, with enhanced amplitudes for stimuli that capture attentional resources (e.g., novel stimuli; e.g., Polich, 2007; Polich & Criado, 2006). Therefore, in the present study, surprised as compared to neutral contextual faces may occupy more attentional resources even during presentations of target faces, and as a consequence, result in poorer structural and complex cognitive encoding of following target faces. It should be noted that while previous studies attributed emotional ERP effects mainly to the effects evoked by emotional stimuli than those elicited by neutral stimuli; in these studies, positive and/or negative stimuli were used (e.g., Carretié et al., 2004; Vuilleumier, 2005). The present study used surprised faces, which had been rated as emotionally neutral in a prior study (Langner et al., 2010). Therefore, the exact mechanism behind the difference between surprised and neutral contextual faces is still unclear. Given that the present study could not provide an answer to this question, related issues should be carefully addressed in future studies. Another possible interpretation is that our oddball-like task induces strong expectations regarding the occurrences of facial expressions. The expectation of surprised contextual facial expressions might interfere with processing of expectation-incongruent (happy or angry) target faces more than the expectation of neutral contextual facial expressions. However, this interpretation seems to be unlikely, as previous studies found that fearful contextual faces impaired processing of target faces regardless of whether target faces were expectation-congruent or -incongruent (Furl et al., 2007; Lin et al., 2015b). Regarding the N170, the findings in the present study are in line with previous studies (Furl et al., 2007; Lin et al., 2015b) that suggested reduced structural encoding of target faces in a sequence of emotional compared to neutral contextual expressions. Remarkably, these studies only addressed the comparisons between fearful and neutral contextual expressions. Fearful and neutral expressions differ not only in valence but also in some other 13
factors (e.g., arousal). These factors have also been suggested to influence the structural encoding of faces (e.g., Sprengelmeyer & Jentzsch, 2006). Therefore, it was still unclear whether the context effect is based on a negative valence effect or other effects of contextual faces. In the present study, we still observed a similar context effect on the encoding of target faces, although context expressions did not differ in valence. Therefore, our present findings indicate that the encoding of target faces may be not - at least not only - modulated by valence, but by some other factors of contextual facial expressions, such as arousal. While we did not find contextual effects on the N170 contingent on the valence of target faces, there was a stronger effect of context in the LPP for happy as compared to angry target faces. This suggests weaker distractions by (surprised and neutral) contextual faces on the processing of angry target faces. A study by Calvo and Beltrán (2014) suggests that individuals allocate more attentional resources to angry as compared to some other emotional (e.g., surprised, happy and neutral) faces. This might result in less susceptibility of angry faces to be affected by context information. While the effect of surprised compared to neutral contextual expression on the LPP is dependent on “target expression”, a study by Richard et al. (2013) showed that fearful compared to neutral contextual expressions decreased the cognitive encoding reflected in the LPP of target faces, regardless of their expression (fearful, neutral and happy), at least for highly anxious individuals. Unlike surprised expressions that are emotionally ambiguous, fearful expressions signal potential threat and harm. Fearful expressions seem to capture more attentional resources than do threat-unrelated expressions (e.g., Batty & Taylor, 2003; Phelps, Ling & Carrasco, 2006; Pourtois, Grandjean, Sander, & Vuilleumier, 2004), so that fearful contextual expressions may disturb cognitive encoding to a larger extent than surprised faces in the present study do. However, as our present and Richard et al.’s (2013) study differ not only in contextual expressions but also in some other factors (e.g., target expressions and group sample), this explanation remains to be investigated in future studies.
Effects of “contextual expression” on recognition performance of target faces Another important finding in the present study was that later recognition rates were lower for happy encoded target faces after a sequence of surprised as compared to neutral contextual expressions, regardless of whether these target faces showed the encoded or a non-encoded expression during the recognition phase. As the valence of the contextual 14
faces was kept constant, the present finding may suggest an effect of arousal: the subsequent recognition of happy faces is impaired by high arousal of surprised contextual faces more than by low arousal of neutral contextual faces. The finding is partially in line with our previous study (Lin et al., 2015b), in which the recognition of target faces was impaired after a sequence of fearful contextual expressions as compared to neutral contextual expressions when target faces showed the encoded expression. Similar to the context effect related to face encoding, our previous study (Lin et al., 2015b) was not able to indicate whether the context effect on the recognition of target faces depends on negative valence of facial expressions or other factors (e.g., arousal). The present findings show that negative valence or threat relevance of contextual expressions is not necessary to impair later recognition of target faces. However, our previous study reported that the recognition effects of contextual expression did not generalize, when fearfully and neutrally encoded target faces showed neutral and fearful expressions, respectively, during the recognition phase (Lin et al., 2015b). A bias for reporting emotional faces as “old” has been suggested, particularly when faces are difficult to recognize (e.g., when target faces are presented with a new expression; e.g., Sharot, Delgado, & Phelps, 2004). Therefore, in our previous study (Lin et al., 2015b), participants might have had a tendency of reporting fearfully recognized target faces as “old” and neutrally recognized target faces as “new”, irrespectively of the encoded context, resulting in a general reduction of the context effect. However, the present study showed no significant bias for target expressions during the recognition phase (as the interaction “consistency” and “target expression” was not significant), possibly enhancing the difference of the context effect even though target faces showed a new expression during the recognition phase. In addition, surprised compared to neutral context affected recognition rates only for happy but not of angry target faces. As face encoding is suggested to be the first step of face memory (Bruce & Young, 1986), it is an important prerequisite for later steps, such as recognition. In the present study, ERP data suggested that surprised compared to neutral contextual faces resulted in poorer encoding of happy but not of angry target faces. This valence-dependent effect of encoding might have also influenced later steps of face memory, resulting in a similar effect during recognition.
Limitations and future directions 15
As mentioned in the results section, the right hemispheric N170 effect of surprised context was numerically only prominent for happy but not for angry faces. However, this exploratory post-hoc analysis was not justified by a higher-order interaction of “contextual expression”, “target expression” and hemisphere. As we did not determine the required sample size according to potential (but unknown) effects prior to the study, the absence might be due to an insufficient sample size. Future studies might take this into account to further investigate the contextual effects. In addition, according to Tables 2 and 3, there seem to be effects of “target expression” in the neutral face context on the N170 and LPP. However, further analysis did not reveal significant N170 and LPP effects of “target expression” in the neutral face context (please see more in the result section). Although these findings were similar to those in previous studies (e.g., Batty & Taylor, 2003; Calvo & Beltrán, 2013, 2014; Müller-Bardorff et al., 2016), we cannot rule out an absence of differential effects due to the small sample size. Therefore, this issue remains to be further investigated also by expanding the sample size. Previous studies with positive and negative stimuli suggested that the emotional ERP effects are more likely due to enhanced attention attracted by emotional stimuli instead of reduced attention by neutral stimuli. However, unlike positive and negative stimuli, surprised contextual faces in the present study were emotionally neutral. Therefore, the observed effects of context on N170 and LPP cannot definitively be attributed to surprised contextual faces or to neutral contextual faces. Future studies may investigate this issue for more details. Finally, while we discussed that the differences of the contextual LPP effects in our present and Richard et al.’s (2013) study are related to differential distractions between fearful and surprised contextual expressions, no studies - to the best of our knowledge have ever investigated whether fearful as compared to surprised contextual faces influence the LPP to target faces. Future studies may investigate this issue in more detail.
Conclusion The present study investigated the influence of surprised contextual facial expressions as compared to neutral expressions on the encoding and recognition of emotional target faces. We observed that during the encoding phase, surprised as compared to neutral contextual expressions led to smaller N170 for happy target faces but not angry target faces, though the effect of “contextual expression” for happy target faces was not significantly 16
different from the effect for angry faces. The LPP was smaller for happy target faces in a sequence of surprised as compared to neutral contextual faces; whereas this effect was not evident for angry target faces. In the later recognition phase, the recognition of happy encoded target faces was found to be poorer when the faces had been presented in a sequence of surprised as compared to neutral contextual expressions. More importantly, this context effect appeared even when the expression of target faces shifted to angry during the recognition phase. However, there was no context effect for angry encoded target faces. Taken together, our findings indicate that preceding emotionally ambiguous compared to neutral contextual expressions differentially affect the structural and cognitive encoding and subsequent recognition of happy target faces, while angry faces are more resistant to contextual interference.
Acknowledgements This work is supported by Research Program for Humanities and Social Science granted by the Ministry of Education in China (17YJC190016).
References Balconi, M., & Mazza, G. (2009). Consciousness and emotion: ERP modulation and attentive vs. pre-attentive elaboration of emotional facial expressions by backward masking. Motivation and Emotion, 33(2), 113-124. Batty, M., & Taylor, M. J. (2003). Early processing of the six basic facial emotional expressions. Cognitive Brain Research, 17(3), 613-620. Bruce, V., & Young, A. (1986). Understanding face recognition. British Journal of Psychology, 77(3), 305-327. Calvo, M. G., & Beltrán, D. (2013). Recognition advantage of happy faces: tracing the neurocognitive processes. Neuropsychologia, 51(11), 2051-2061. Calvo, M. G., & Beltrán, D. (2014). Brain lateralization of holistic versus analytic processing of emotional facial expressions. NeuroImage, 92, 237-247. Carretié, L., Hinojosa, J. A., Martín-Loeches, M., Mercado, F., & Tapia, M. (2004). Automatic attention to emotional stimuli: neural correlates. Human Brain Mapping, 22(4), 290-299.
17
Crist, R. E., Wu, C. T., Karp, C., & Woldorff, M. G. (2007). Face processing is gated by visual spatial attention. Frontiers in Human Neuroscience, 1, article 10. Deffke, I., Sander, T., Heidenreich, J., Sommer, W., Curio, G., Trahms, L., & Lueschow, A. (2007). MEG/EEG sources of the 170-ms response to faces are co-localized in the fusiform gyrus. NeuroImage, 35(4), 1495-1501. Dillon, D. G., Cooper, J. J., Grent, T., Woldorff, M. G., & LaBar, K. S. (2006). Dissociation of event-related potentials indexing arousal and semantic cohesion during emotional word encoding. Brain and Cognition, 62(1), 43-57. Ebner, N. C., Riediger, M., & Lindenberger, U. (2010). FACES — A database of facial expressions in young, middle-aged, and older women and men: Development and validation. Behavioral Research Methods, 42(1), 351-362. Eimer, M. (2000a). Effects of face inversion on the structural encoding and recognition of faces: Evidence from event-related brain potentials. Cognitive Brain Research, 10(1), 145158. Eimer, M. (2000b). The face-specific N170 component reflects late stages in the structural encoding of faces. Neuroreport, 11(10), 2319-2324. Eimer, M., & McCarthy, R. A. (1999). Prosopagnosia and structural encoding of faces: Evidence from event‐related potentials. Neuroreport, 10(2), 255-259. Flaisch, T., Stockburger, J., & Schupp, H. T. (2008). Affective prime and target picture processing: an ERP analysis of early and late interference effects. Brain Topography, 20(4), 183-191. Furl, N., van Rijsbergen, N. J., Treves, A., Friston, K. J., & Dolan, R. J. (2007). Experiencedependent coding of facial expression in superior temporal sulcus. Proceedings of the National Academy of Sciences, 104(33), 13485-13489. Johansson, M., Mecklinger, A., & Treese, A. C. (2004). Recognition memory for emotional and neutral faces: An event-related potential study. Journal of Cognitive Neuroscience, 16(10), 1840-1853. Hagemann, J., Straube, T., & Schulz, C. (2016). Too bad: Bias for angry faces in social anxiety interferes with identity processing. Neuropsychologia, 84, 136-149.
18
Henson, R. N. A., Shallice, T., Gorno-Tempini, M. L., & Dolan, R. J. (2002). Face repetition effects in implicit and explicit memory tests as measured by fMRI. Cerebral Cortex, 12(2), 178-186. Holmes, A., Nielsen, M. K., & Green, S. (2008). Effects of anxiety on the processing of fearful and happy faces: an event-related potential study. Biological Psychology, 77(2), 159-173. Holmes, A., Vuilleumier, P., & Eimer, M. (2003). The processing of emotional facial expression is gated by spatial attention: evidence from event-related brain potentials. Cognitive Brain Research, 16(2), 174-184. Labuschagne, I., Croft, R. J., Phan, K. L., & Nathan, P. J. (2010). Augmenting serotonin neurotransmission with citalopram modulates emotional expression decoding but not structural encoding of moderate intensity sad facial emotional stimuli: an event-related potential (ERP) investigation. Journal of Psychopharmacology, 24(8), 1153-1164. Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H., Hawk, S. T., & van Knippenberg, A. (2010). Presentation and validation of the Radboud Faces Database. Cognition and Emotion, 24(8), 1377-1388. Lin, H., Schulz, C., & Straube, T. (2015a). Cognitive tasks during expectation affect the congruency ERP effects to facial expressions. Frontiers in Human Neuroscience, 9, article 596. Lin, H., Schulz, C., & Straube, T. (2015b). Fearful contextual expression impairs the encoding and recognition of target faces: an ERP study. Frontiers in Behavioral Neuroscience, 9, article 237. Lin, H., Schulz, C., & Straube, T. (2016). Effects of expectation congruency on event-related potentials (ERPs) to facial expressions depend on cognitive load during the expectation phase. Biological Psychology, 120, 126-136. Lundqvist, D., Flykt, A., & Öhman, A. (1998). The Karolinska directed emotional faces – KDEF. CD-ROM from Department of Clinical Neuroscience, Psychology section. Sweden: Karolinska Institute. Stockholm ([ISBN: 91-630-7164-9]). Meeren, H. K., van Heijnsbergen, C. C., & de Gelder, B. (2005). Rapid perceptual integration of facial expression and emotional body language. Proceedings of the National Academy of Sciences of the United States of America, 102(45), 16518-16523.
19
Michalowski, J. M., Pané-Farré, C. A., Löw, A., & Hamm, A. O. (2015). Brain dynamics of visual attention during anticipation and encoding of threat-and safe-cues in spider-phobic individuals. Social Cognitive and Affective Neuroscience, 10(9), 1177-1186. Mohamed, T. N., Neumann, M. F., & Schweinberger, S. R. (2009). Perceptual load manipulation reveals sensitivity of the face-selective N170 to attention. Neuroreport, 20(8), 782-787. Morel, S., Ponz, A., Mercier, M., Vuilleumier, P., & George, N. (2009). EEG-MEG evidence for early differential repetition effects for fearful, happy and neutral faces. Brain Research, 1254, 84-98. Müller-Bardorff, M., Schulz, C., Peterburs, J., Bruchmann, M., Mothes-Lasch, M., Miltner, W., & Straube, T. (2016). Effects of emotional intensity under perceptual load: An eventrelated potentials (ERPs) study. Biological Psychology, 117, 141-149. Neta, M., Davis, F. C., & Whalen, P. J. (2011). Valence resolution of ambiguous facial expressions using an emotional oddball task. Emotion, 11(6), 1425-1433. Neta, M., & Whalen, P. J. (2010). The primacy of negative interpretations when resolving the valence of ambiguous facial expressions. Psychological Science, 21(7), 901-907. Oldfield, R. C. (1971). The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia, 9(1), 97-113. Pinabiaux, C., Bulteau, C., Fohlen, M., Dorfmüller, G., Chiron, C., Hertz-Pannier, L., ... & Jambaqué, I. (2013). Impaired emotional memory recognition after early temporal lobe epilepsy surgery: the fearful face exception?. Cortex, 49(5), 1386-1393. Phelps, E. A., Ling, S., & Carrasco, M. (2006). Emotion facilitates perception and potentiates the perceptual benefits of attention. Psychological Science, 17(4), 292-299. Polich, J. (2007). Updating P300: an integrative theory of P3a and P3b. Clinical Neurophysiology, 118(10), 2128-2148. Polich, J., & Criado, J. R. (2006). Neuropsychology and neuropharmacology of P3a and P3b. International Journal of Psychophysiology, 60(2), 172-185. Pourtois, G., Grandjean, D., Sander, D., & Vuilleumier, P. (2004). Electrophysiological correlates of rapid spatial orienting towards fearful faces. Cerebral Cortex, 14(6), 619-633.
20
Richards, A., Holmes, A., Pell, P. J., & Bethell, E. J. (2013). Adapting effects of emotional expression in anxiety: Evidence for an enhanced Late Positive Potential. Social Neuroscience, 8(6), 650-664. Righi, S., Marzi, T., Toscani, M., Baldassi, S., Ottonello, S., & Viggiano, M. P. (2012). Fearful expressions enhance recognition memory: electrophysiological evidence. Acta Psychologica, 139(1), 7-18. Ryu, V., An, S. K., Jo, H. H., & Cho, H. S. (2010). Decreased P3 amplitudes elicited by negative facial emotion in manic patients: selective deficits in emotional processing. Neuroscience Letters, 481(2), 92-96. Schubö, A., Schröger, E., & Meinecke, C. (2004). Texture segmentation and visual search for pop-out targets: An ERP study. Cognitive Brain Research, 21(3), 317-334. Schupp, H. T., Öhman, A., Junghöfer, M., Weike, A. I., Stockburger, J., & Hamm, A. O. (2004). The facilitated processing of threatening faces: An ERP Analysis. Emotion, 4(2), 189-200. Sessa, P., Luria, R., Gotler, A., Jolicœur, P., & Dell'Acqua, R. (2011). Interhemispheric ERP asymmetries over inferior parietal cortex reveal differential visual working memory maintenance for fearful versus neutral facial identities. Psychophysiology, 48(2), 187-197. Sharot, T., Delgado, M. R., & Phelps, E. A. (2004). How emotion enhances the feeling of remembering. Nature Neuroscience, 7(12), 1376-1380. Shimamura, A. P., Ross, J. G., & Bennett, H. D. (2006). Memory for facial expressions: The power of a smile. Psychonomic Bulletin & Review, 13(2), 217-222. Smith, E., Weinberg, A., Moran, T., & Hajcak, G. (2013). Electrocortical responses to NIMSTIM facial expressions of emotion. International Journal of Psychophysiology, 88(1), 17-25. Sprengelmeyer, R., & Jentzsch, I. (2006). Event related potentials and the perception of intensity in facial expressions. Neuropsychologia, 44(14), 2899-2906. Tottenham, N., Tanaka, J. W., Leon, A. C., McCarry, T., Nurse, M., Hare, T. A., ... & Nelson, C. (2009). The NimStim set of facial expressions: judgments from untrained research participants. Psychiatry Research, 168(3), 242-249. Vuilleumier, P. (2005). How brains beware: neural mechanisms of emotional attention. Trends in Cognitive Sciences, 9(12), 585-594.
21
Tables and table caption Table 1. Mean of accuracy (%), response times (ms) and their SE for all the experimental conditions during the encoding phase. Contextual Expression Target Expression
Surprised
Neutral
M
SE
M
SE
Angry
96.67
0.81
96.74
0.76
Happy
95.91
0.89
97.12
0.71
Angry
536.84
17.75
538.97
14.49
Happy
516.75
13.76
529.48
17.01
Accuracy
Response times
22
Table 2. Mean of the N170 amplitudes (μV) and its SE for all the experimental conditions during the encoding phase. Contextual Expression Hemisphere
Target Expression
Surprised
Neutral
M
SE
M
SE
Angry
-3.84
0.93
-3.82
0.89
Happy
-3.82
0.84
-3.82
0.94
Angry
-3.64
0.69
-3.91
0.65
Happy
-3.73
0.64
-4.45
0.77
P7
P8
23
Table 3. Mean of the early LPP amplitudes (μV) and its SE for all the experimental conditions during the encoding phase. Contextual Expression Electrode
Target Expression
Surprised
Neutral
M
SE
M
SE
Angry
0.10
3.05
-0.03
2.35
Happy
-0.53
2.31
0.95
2.54
Angry
3.08
3.81
3.11
3.44
Happy
2.32
3.57
4.00
3.41
Angry
3.11
3.62
2.14
3.74
Happy
1.90
3.62
3.38
3.44
Angry
6.41
4.79
5.92
4.18
Happy
5.11
5.40
7.02
4.29
Angry
6.92
3.25
6.86
2.93
Happy
6.03
3.88
7.35
3.21
Angry
6.53
4.18
6.24
3.30
Happy
5.58
3.89
6.88
3.74
Angry
7.00
4.13
6.95
3.08
Happy
6.22
3.99
6.92
2.95
Fz
FC1
FC2
Cz
CP1
CP2
Pz
24
Table 4. Mean of accuracy (%), response times (ms) and their SE for all the experimental conditions during the recognition phase. Contextual Expression Consistency
Target Expression
Surprised
Neutral
M
SE
M
SE
Angry
36.67
2.99
37.42
3.56
Happy
32.88
2.72
38.33
3.55
Angry
31.67
3.30
30.76
2.97
Happy
30.91
2.76
33.64
2.97
Angry
799.94
58.14
864.31
67.38
Happy
796.35
70.55
805.68
64.71
Angry
851.60
74.23
803.66
59.33
Happy
804.90
69.02
837.81
68.48
Accuracy Unchanged
Changed Response times Unchanged
Changed
25
Figure caption Figure 1. A chart regarding the assignments of different faces (i.e., target, contextual and novel faces). (A) Two identities (1 female and 1 male) with surprised and neutral facial expressions served as contextual identities. 120 identities (60 females and 60 males; 120 angry and 120 happy) were used as novel identities. The remaining 122 identities (61 females and 61 males, including 2 practice identities) with angry and happy facial expressions were used as targets. For target identities used in the actual experiment, they were separated randomly into four sets with 30 identities (15 females and 15 males; 30 angry and 30 happy) each. In each set, there were angry and happy sub-sets. (B) In the encoding phase, two angry sub-sets [e.g., (Female and male) - Angry: No. 01-15 and 16-30] and two happy sub-sets [e.g., (Female and male) - Happy: No. 31-45 and 46-60] with different identities were used to be presented with either surprised or neutral contextual faces. In the recognition phase, old faces in all sub-sets, including those in unchanged and changed expressions [e.g., (Female and male) - Happy: No. 01-15 and 16-30 and (Female and male) - Angry: No. 31-45 and 46-60], and novel faces [e.g., (Female and male) - Angry: No. 63-122 and (Female and male) - Happy: No. 63-122] were presented.
Figure 2. Event-related potentials (ERPs) at the electrodes P7 and P8 for all the experimental conditions during the encoding phase. Shaded areas correspond to the time window for the N170 (130-180 ms).
Figure 3. Event-related potentials (ERPs) at electrodes Fz, FC1, FC2, Cz, CP1, CP2 and Pz for all the experimental conditions during the encoding phase. Shaded areas correspond to the time window for LPP (350-800 ms).
Figure 4. Topographical maps based on N170 (130-180 ms) and LPP (350-800 ms) mean amplitudes for all the experimental conditions during the encoding phase.
26
27
28
Figr-1
29
Figr-2
30
Figr-3
31
Figr-4
32