- Email: [email protected]
Where perception meets belief updating: Computational evidence for slower updating of visual expectations in anxious individuals
Journal Pre-proof
Where Perception Meets Belief Updating: Computational Evidence for Slower Updating of Visual Expectations in Anxious Individuals Jonathon R. Howlett , Martin P. Paulus PII: DOI: Reference:
S0165-0327(19)32093-2 https://doi.org/10.1016/j.jad.2020.02.012 JAD 11643
To appear in:
Journal of Affective Disorders
Received date: Revised date: Accepted date:
6 August 2019 31 January 2020 1 February 2020
Please cite this article as: Jonathon R. Howlett , Martin P. Paulus , Where Perception Meets Belief Updating: Computational Evidence for Slower Updating of Visual Expectations in Anxious Individuals, Journal of Affective Disorders (2020), doi: https://doi.org/10.1016/j.jad.2020.02.012
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2020 Published by Elsevier B.V.
Highlights
Decision-making and perceptual processing rely on separate belief updating processes
Anxiety is associated with slower updating of perceptual expectations
Anxiety is not associated with altered updating of beliefs for decision-making
Older age was also associated with slower updating of perceptual expectations
1
Where Perception Meets Belief Updating: Computational Evidence for Slower Updating of Visual Expectations in Anxious Individuals
Jonathon R. Howlett 1,2* and Martin P. Paulus1,2
1
Department of Psychiatry, University of California San Diego, La Jolla, CA, USA
2
Laureate Institute for Brain Research, Tulsa, OK, USA
*Corresponding Author: Jonathon R. Howlett, MD University of California San Diego Department of Psychiatry 9500 Gilman Dr La Jolla, CA 92093 [email protected]
2
Where Perception Meets Belief Updating: Computational Evidence for Slower Updating of Visual Expectations in Anxious Individuals
Abstract Background: Surprising events are important sources of internal model updating which adjusts expectations for both decision-making and perceptual processing circuits. Anxious individuals display relatively intact updating of internal models used to make decisions, however how these individuals update their perceptual expectations remains poorly understood. Based on previous work, we hypothesized that anxious individuals experienced exaggerated surprise to predictable events, which imbues them with undue salience. Methods: To model the rate of updating of internal models for both decision-making and perceptual processing, we applied a hybrid Rescorla Wagner (RW)/Drift Diffusion Model (DDM) to a change point detection task in a transdiagnostic group of individuals with mood and anxiety disorders and examined the relationship between learning rates and anxiety and negative affect. Results: Model comparison provided evidence that decision-making and perceptual processing rely on separate internal models with different learning rates. Anxiety and older age were associated with slower updating of models used in perceptual processing, but not those used in decision-making. Limitations: This was a cross-sectional study and lacked neural data to examine the role of specific brain circuits in updating of perceptual predictions.
3
Conclusions: Anxious individuals display slower updating of internal models used in perceptual processing, but not those used in decision-making. This deficit could contribute to exaggerated salience of harmless stimuli in anxiety. The results have implications for the assessment and treatment of basic processing dysfunctions in anxiety.
Keywords: Computational Psychiatry; Anxiety; Rescorla-Wagner; Drift Diffusion Model; Perception; Prediction Error
4
Introduction Surprising events often signal the need for immediate allocation of processing resources, but can also lead us to update our predictive models to avoid being surprised again (O'Reilly et al., 2013). This surprise-driven learning is formalized computationally by the Rescorla-Wagner (RW) model, which holds that the change in expectation after new information is equal to the prediction error scaled by a learning rate (Rescorla and Wagner, 1972). Surprising events are thus thought to generate teaching signals that drive updating of internal predictive models, consistent with empirical evidence that prediction errors are associated with robust activations across a number of brain regions (Maia and Frank, 2011). While initially proposed to account for results from classical conditioning studies, RW is now frequently used to study reward-based decision-making, in which expectations generated by a continuously updated internal model are presumably relayed to action selection circuits in the brain (Maia and Frank, 2011). However, internal expectations are also relayed to perceptual processing regions via top-down neural projections (Rao and Ballard, 1999; Rauss et al., 2011). Deficits in the updating of these perceptual predictions could lead individuals to be repeatedly surprised by predictable events, exaggerating the salience of innocuous stimuli and resulting in excessive allocation of neural resources. Given the conceptualization of anxiety as a disorder of aberrant internal prediction signals (Paulus and Stein, 2006) and the evidence for an overactive salience network in anxiety (Menon, 2011), an examination of the updating of perceptual expectations could contribute to a mechanistic account of dysfunction in anxiety and enable new approaches to assessment and treatment of anxiety-related disorders.
5
Studies of decision-making in anxious individuals have not revealed higher or lower learning rates per se and instead have shown subtler deficits in regulation of learning based on context; in this sense, updating of internal models driving decisions appears relatively intact in anxious individuals (Brown et al., 2018; Browning et al., 2015; Huang et al., 2017). However, the relationship between predictive models that drive decision-making and those that influence perceptual processing are poorly understood, and these may represent fundamentally different neural processes. Dysfunctional updating of perceptual expectations in anxious individuals could help explain the overreaction to harmless stimuli in anxiety and would have important implications for assessment and treatment. An ideal computational method to measure this impairment is to combine RW with a drift diffusion model (DDM), a common approach to studying perceptual decision-making in which presumably noisy evidence is accumulated over time toward one of two decision thresholds, and a decision is made once a decision threshold is reached (Ratcliff et al., 2016). The DDM has been used extensively to model both response accuracy and response times on a variety of 2alternative forced choice tasks, including perceptual decision-making tasks (Ratcliff et al., 2016). The central aspect of a combined RW/DDM model is that prior expectations generated by RW can be incorporated into the DDM through a bias parameter, which allows the starting point for evidence accumulation to be nearer to the boundary representing the expected outcome (Mulder et al., 2012). Recent evidence also indicates that prior information can influence the DDM drift rate parameter, which quantifies the rate of accumulation of perceptual information (Urai et al., 2019). While DDM has recently been employed to study reward-based decision-making (Pedersen et al., 2017),
6
to our knowledge it has never been combined with RW to examine the updating of perceptual expectations via the DDM bias and drift rate parameters in a combined belief updating/perceptual paradigm. We employed this approach in a large transdiagnostic psychiatric sample of healthy volunteers and individuals with mood and anxiety problems, consistent with the National Institute of Mental Health (NIMH) Research Domain Criteria (RDoC) framework, which aims to develop a dimensional, neuroscience-based psychiatric classification system (Insel et al., 2010). RDoC consists of specific domains of functioning, with the negative valence domain being particularly relevant to anxiety-related disorders. Within this domain, fear, which has an arousal component, is distinguished from negative affect more generally, with implications for assessment and treatment of individuals with anxiety-related complaints (McTeague and Lang, 2012). We hypothesized that fearful individuals would exhibit poorer updating of perceptual expectations.
Method Participants The experiment was part of the Tulsa-1000 (T-1000) study (Victor et al., 2018), which is aimed at multilevel assessment and outcome prediction in a large, heterogeneous psychiatric sample. Subjects were recruited from mental health providers or via advertisements. At screening, treatment-seeking individuals were required to have a score on the Patient Health Questionnaire-9 (PHQ-9) ≥ 10 and/or on the Overall Anxiety Severity and Impairment Scale (OASIS) ≥ 8. 306 subjects (age: 35.28 ± 11.31 years; gender: 101 male and 205 female) participated in the experiment. See Supplementary
7
Table 1 for subject diagnoses based on the based on the MINI Neuropsychiatric Interview (MINI) (Sheehan et al., 1998). 167 subjects were medicated (see Supplementary Table 2 for classes of psychotropic medications). 35 subjects were excluded due to low accuracy on the perceptual component of the task (see below). To assess affective state including fear and negative affect, participants completed the Positive and Negative Affect Schedule-Expanded Form (PANAS X) (Watson and Clark, 1999). All study procedures were approved by the Western Institutional Review Board, and all participants provided written informed consent prior to participation.
Experiment Subjects completed a combined belief updating/visual perception task (see Fig.1) (Huang et al., 2017). Subjects attempted to identify the location of a target stimulus, which is one of three random-dot stimulus patches (Gold and Shadlen, 2000) with a specified coherent motion direction (random-dot coherence = 30%). The other two patches were distractors with the opposite motion direction. The specified coherent motion direction was relayed to the subject and kept consistent throughout the task. At the start of each trial, three dots indicated the locations in which the target could appear. Subjects searched locations sequentially by pressing corresponding keyboard keys and made the final decision by pressing the up-arrow key to indicate the last viewed stimulus was the target. Upon selecting a location, the random-dot stimulus patch was shown. If the initially chosen location was correct, subjects could therefore determine this based on the motion direction and terminate the trial without any further searching; if not, subjects would then select another location to continue to search for the target. Subjects were
8
allowed to search until they terminated the trial by indicating that they were viewing the target stimulus. At the end of each trial, the true target location was shown to provide feedback. The target stimulus appeared in the three locations with a relative frequency of 1:3:9, with the reward contingency at the three locations changing every 30 trials on average. See Supplementary Methods for further experiment details.
Model Model fitting was performed in Bayesian hierarchical fashion using the RStan (Stan Development Team, 2018) implementation of the Stan (Carpenter et al., 2017) language. Models were fit using the initial location choice on each trial and the randomdot reaction time for the initially chosen location (indicating whether the location did or did not contain the target). The model assumes that expectations regarding target location, based on the sequence of prior target locations, influences both (a) the initial location choice on a trial and (b) the response and reaction time to the random-dot stimulus shown at the initially chosen location. Regarding location choice, subjects will be more likely to choose an initial location with a greater expectation of choosing the target. Regarding reaction time to the random-dot stimulus, higher expectation that the location is correct will lead to faster choices confirming this (if the location does indeed contain the target) but slower choices disconfirming this (if the location does not contain the target). For both location decisions and random-dot responses, expectations are modeled using an RW model with a corresponding learning rate, which can be fit using location choices, random-dot responses and reaction times, or both.
9
The updating of location expectations based on the true target location on each trial was modeled using an RW model, with a learning rate α parameter for each subject (the learning rate was fit using all trials across the task). Each learning rate was estimated based on subject choices (the decision learning rate), on reaction times and responses to the random-dot stimulus (the perceptual learning rate), or both (see Fig. 2 and Supplementary Fig. S1). For the decision learning rate, subjects’ choices were modeled as being dependent on RW expectations via a softmax decision model with an inverse decision temperature parameter controlling the randomness of decisions. For the perceptual learning rate, RW expectations influenced either the DDM bias parameter, DDM drift rate parameter, or both, depending on the model (with model comparisons performed to determine the best model, as discussed below). The influence of RW expectations on DDM parameters was scaled by ξ parameters determining the strength of influence of expectations on perceptual processing. Each individual learning rate was modeled in Bayesian hierarchical fashion as depending on a group-level mean and also potentially on individual-level predictors such as self-reported fear, gender, and age. See Supplementary Methods for further modeling details.
Model Comparison We performed model comparisons using the widely applicable information criterion (WAIC) with the loo package in R (Vehtari et al., 2017). Model comparisons were performed to select the best performing model based on two aspects: (1) the manner in which RW expectations influenced perceptual processing as modeled by DDM—only via the DDM bias parameter (bias-only model), only via the DDM drift rate parameter
10
(drift-only model), or via both the bias and drift rate parameter (bias and drift model); (2) whether a single updating process, with a single learning rate, supplies expectations to both decision-making and perceptual circuits, or whether there are two separate updating processes with different learning rates. To address this second aspect, we constructed two categories of models. In the first category of models (the single α models), a single learning rate for each subject was fit both to choices (via a softmax decision policy) and to responses and reaction times to the random-dot stimulus (via DDM). The second category of models (dual α models) incorporated two separate updating processes, with two separate learning rates, one fit to choices (the decision learning rate) and the other to responses and reaction times to the random-dot stimulus (the perceptual learning rate). We performed a model comparison of six models (bias-only single α model, biasonly dual α model, drift-only single α model, drift-only dual α model, bias and drift single α model, and bias and drift dual α model). All models predicted both categorical location choices and random-dot reaction times, so they could be compared directly despite the different nature of these two types of data. To assess the relationship between decision learning rate and perceptual learning rate in the dual α model, we calculated the Pearson correlation coefficient of the raw decision learning rate parameter and the raw perceptual learning rate parameter for each draw of the posterior distribution.
Individual-Level Predictors and Model Parameters To determine the relationship between fear and model parameters, we constructed a hierarchical model in which both subject-level learning rates depended on scaled age,
11
gender, and PANAS X Fear. Effect sizes for the relationships between subject-level predictors and subject-level α were calculated by dividing each draw of the posterior distribution of the slope by the posterior distribution of the standard deviation of the raw subject-level α parameter (see Supplementary Methods). This yielded the equivalent of a standardized regression coefficient, i.e. an estimate of the change (in standard deviations) of the dependent variable per standard deviation of the predictor variable. To determine whether the relationship between model parameters and fear was being driven by general negative affect, we also constructed a model in which PANAS X Negative Affect was additionally included as an individual-level predictor. Finally, to supplement our hypothesis-driven analysis of the relationship between perceptual learning rate and fear, we also assessed the relationship between model parameters and higher-level affective dimensions. We constructed a model in which PANAS X Positive Affect and PANAS X Positive Affect (the two higher-order dimensions of the PANAS X scale) were included as individual-level predictors, along with age and gender, but without any specific affects measured by PANAS X (e.g. fear).
Parameter Reliability To estimate parameter reliability, we constructed a different model with separate learning rates and inverse decision temperature parameters for each block (unlike in the main model in which learning rates and inverse decision temperatures were fit using all trials on the task). We then calculated intraclass correlations (ICC) for each parameter.
Results
12
Model Comparison Model comparison using WAIC indicated that the bias and drift dual α model provided the best fit for the observed data compared to the single α model (bias-only single α model WAIC: 141094.4; bias-only dual α model WAIC: 139185.4; drift-only single α model WAIC: 144165.4; drift-only dual α model WAIC: 137661.8; bias and drift single α model WAIC: 138590.1; bias and drift dual α model WAIC: 132962.4). WAIC accounts for model complexity, indicating that the additional complexity of the dual α model is justified by the improvement in predictive accuracy and providing evidence for separate updating processes for decision-making and perceptual processing. Median correlation between the raw decision learning rate parameter and the raw perceptual learning rate parameter was r = .12. The 95% credible interval (i.e. the interval within which 95% of the Bayesian posterior probability distribution fell) was [.05, .18].
Individual-Level Predictors and Model Parameters PANAS X Fear score was negatively associated with perceptual learning rate (median β = -0.15, 95% credible interval β = [-0.27, -0.02]), i.e. individuals who reported the highest fear scores showed the lowest rate of perceptual updating (see Fig. 3a). In contrast, fear was not related to decision learning rate (β = -0.01 [-0.14, 0.12]; Fig. 3b). Similarly, age was also negatively associated with perceptual learning rate (β = -0.40 [0.53, -0.27]) but not decision learning rate (β = -0.08 [-0.21, 0.05]), i.e. older individuals showed slower perceptual but not decisional updating. Male gender was positively associated with perceptual learning rate (β = 0.14 [0.02, 0.27]) but not decision learning rate (β = -0.07 [-0.20, 0.07]).
13
When PANAS X Negative Affect score was added to the model, the median estimate of the association between fear and perceptual learning rate was more negative, but the posterior distribution widened such that 0 was included in the 95% credible interval (β = -0.21 [-0.49, 0.07]; Supplementary Fig. S2). Negative affect was not associated with perceptual learning rate (β = 0.07 [-0.19, 0.36]), indicating that general negative affect does not drive the association between fear and perceptual learning rate. In the model in which PANAS X Positive Affect and PANAS X Negative Affect were included as predictors, the median estimate for relationship between PANAS X Negative Affect and perceptual learning rate was β = -0.11 ([-0.25, 0.03] ; Supplementary Fig. S3), while the median estimate of the relationship with decision learning rate was β = -0.05 ([-0.19, 0.09]). The median estimate for relationship between PANAS X Positive Affect and perceptual learning rate was β = -0.00 ([-0.14, 0.14]), while the median estimate of the relationship with decision learning rate was β = -0.12 ([-0.27, 0.02]).
Parameter Reliability For the decision learning rate, median ICC was .62 (95% credible interval: [.54, .69]. For the perceptual learning rate, median ICC was .80 (95% credible interval: [.75, .85]. For the inverse decision temperature parameter, median ICC was .47 (95% credible interval: [.37, .56].
Discussion The goal of this investigation was to examine the degree to which anxiety and/or negative affect in a transdiagnostic group of individuals with mood and anxiety disorders
14
influence (1) the rate of updating of an internal model that relays expectations to perceptual circuits and (2) the rate of updating of an internal model that relays expectations to decision-making circuits, and to provide evidence for whether these represent two separate updating processes. We found that anxious (and older) individuals exhibit slower updating of the internal model that influences perceptual processing, but not the model that influences decision-making. We found evidence that these two models employ separate updating processes with separate learning rates (a decision learning rate and a perceptual learning rate), which are only weakly correlated. The reliability of estimation of the perceptual learning rate was significantly greater than for the decision learning rate. We did not observe a relationship between fear and learning rate with our standard RW model, based on sequential choices. This result is consistent with previous studies of sequential decision making in anxiety, which have found that anxiety is not associated with alterations in RW learning rate per se. Instead, anxiety has been associated with an impaired ability to adapt learning rate based on environmental context (Browning et al., 2015) and with a higher base learning rate in a more complex model in which the learning rate can increase at certain times during the task (Huang et al., 2017). Similarly, posttraumatic stress disorder (PTSD) has been associated with higher updating of associability (a marker of attentional allocation to unexpected outcomes) in a learning task, but not with higher learning rates (Brown et al., 2018). Taken together, these prior findings and our current results indicate that anxiety may have a selective effect in slowing the updating of expectations that are relayed to perceptual circuits, rather than those relayed to decision-making circuits. Such slow updating could result from
15
hyperprecise prior beliefs, i.e. rigid expectations that are resistant to alteration in light of new information (Paulus, 2019). This slow updating of perceptual predictions could lead anxious individuals to be repeatedly surprised by predictable events. Surprise is fundamentally related to salience (Itti and Baldi, 2009), and surprising events drive immediate reallocation of attentional processing resources (O'Reilly et al., 2013). If anxious individuals are slow to update perceptual expectations, resulting in continued surprise signals driven by predictable events, this could result in overallocation of resources to process possibly threat-related, or even innocuous cues, leading to an increase in tonic arousal levels. Such a processing dysfunction could be a target for individualized assessment and intervention. However, given that our paradigm does not allow us to directly measure surprise, future research will be necessary to confirm this relationship. Functional neuroimaging could help differentiate surprise (which is associated with parietal cortex activation) from model updating (which is associated with dorsal anterior cingulate cortex activation) (O'Reilly et al., 2013). Differential allocation of attention may also play a role in differences in updating of perceptual expectations. Strengths of this study include a large, heterogeneous psychiatric sample and a paradigm and modeling approach that are ideally suited to examine the updating of perceptual expectations. Another strength is the hierarchical Bayesian model-fitting approach, which enables accurate characterization of different sources of variance (Boehm et al., 2018). Limitations include the cross-sectional design, which cannot reveal the relationship between model parameters and symptoms over time, lack of fMRI data to examine the neural mechanisms underlying altered updating, and the need to fit all parameters except learning rate at the group level only to ensure model convergence.
16
Another limitation is that some subjects were medicated, and medication may have influenced performance on the task. Finally, while we focused on the relationship between perceptual learning rate and fear, our sample size did not enable us to determine the extent to which this relationship is specific, or extends to other affects including general negative affect or other specific affects. It is also important to note that our task does not have an explicit fear component. Future research will be necessary to examine the relationship between perceptual updating and other affective processes. In summary, we found that anxious individuals are slower at updating perceptual expectations on a combined sequential decision-making/visual perception task. We additionally found evidence that the internal predictive model for perceptual processing differs from the model for decision-making, and that anxious individuals did not exhibit impairments in the latter process. The results imply that studying belief updating using hybrid decision-making/perceptual paradigms can reveal deficits in anxiety that are obscured by studying decision-making alone, a finding with implications for assessment and treatment of basic processing dysfunctions in anxious individuals.
Author Statement The authors have read and approved the final version of the manuscript. We confirm that it is the authors’ original work, has not received prior publication, and is not under consideration for publication elsewhere.
Jonathon R. Howlett and Martin P. Paulus conceived of, designed, and interpreted the analyses and drafted the manuscript.
17
Disclosure The authors declare no conflicts of interest.
Acknowledgement This work has been supported in part by The William K. Warren Foundation and the National Institute of General Medical Sciences Center Grant Award Number 1P20GM121312. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The http://ClinicalTrials.govidentifier for the clinical protocol associated with data published in the current paper is NCT02450240, “Latent Structure of Multi-level Assessments and Predictors of Outcomes in Psychiatric Disorders”.
The Tulsa 1000 Investigators include the following contributors: Robin Aupperle, Ph.D., Jerzy Bodurka, Ph.D., Yoon-Hee Cha, M.D., Justin Feinstein, Ph.D., Sahib S. Khalsa, M.D., Ph.D., Rayus Kuplicki, Ph.D., Martin P. Paulus, M.D., Jonathan Savitz, Ph.D., Teresa A. Victor, Ph.D.
The funder had no role in study design, in the collection, analysis, and interpretation of data, in the writing of the manuscript, or in the decision to submit the paper for publication.
18
Contributors Jonathon R. Howlett and Martin P. Paulus conceived of, designed, and interpreted the analyses and drafted the manuscript. The authors have read and approved the final version of the manuscript. We confirm that it is the authors’ original work, has not received prior publication, and is not under consideration for publication elsewhere.
Acknowledgement This work has been supported in part by The William K. Warren Foundation and the National Institute of General Medical Sciences Center Grant Award Number 1P20GM121312. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The http://ClinicalTrials.govidentifier for the clinical protocol associated with data published in the current paper is NCT02450240, “Latent Structure of Multi-level Assessments and Predictors of Outcomes in Psychiatric Disorders”.
The Tulsa 1000 Investigators include the following contributors: Robin Aupperle, Ph.D., Jerzy Bodurka, Ph.D., Yoon-Hee Cha, M.D., Justin Feinstein, Ph.D., Sahib S. Khalsa, M.D., Ph.D., Rayus Kuplicki, Ph.D., Martin P. Paulus, M.D., Jonathan Savitz, Ph.D., Teresa A. Victor, Ph.D.
Role of the Funding Source
19
The funder had no role in study design, in the collection, analysis, and interpretation of data, in the writing of the manuscript, or in the decision to submit the paper for publication.
Disclosure The authors declare no conflicts of interest.
Acknowledgement This work has been supported in part by The William K. Warren Foundation and the National Institute of General Medical Sciences Center Grant Award Number 1P20GM121312. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The http://ClinicalTrials.govidentifier for the clinical protocol associated with data published in the current paper is NCT02450240, “Latent Structure of Multi-level Assessments and Predictors of Outcomes in Psychiatric Disorders”.
The Tulsa 1000 Investigators include the following contributors: Robin Aupperle, Ph.D., Jerzy Bodurka, Ph.D., Yoon-Hee Cha, M.D., Justin Feinstein, Ph.D., Sahib S. Khalsa, M.D., Ph.D., Rayus Kuplicki, Ph.D., Martin P. Paulus, M.D., Jonathan Savitz, Ph.D., Teresa A. Victor, Ph.D.
20
References Boehm, U., Marsman, M., Matzke, D., Wagenmakers, E.J., 2018. On the importance of avoiding shortcuts in applying cognitive models to hierarchical data. Behav Res Methods 50, 1614-1631. Brown, V.M., Zhu, L., Wang, J.M., Frueh, B.C., King-Casas, B., Chiu, P.H., 2018. Associability-modulated loss learning is increased in posttraumatic stress disorder. eLife 7, e30150. Browning, M., Behrens, T.E., Jocham, G., O'Reilly, J.X., Bishop, S.J., 2015. Anxious individuals have difficulty learning the causal statistics of aversive environments. Nat Neurosci 18, 590-596. Carpenter, B., Gelman, A., Hoffman, M.D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., Riddell, A., 2017. Stan: A Probabilistic Programming Language. Journal of Statistical Software, Medium: ED; Size: 32 p. Gold, J.I., Shadlen, M.N., 2000. Representation of a perceptual decision in developing oculomotor commands. Nature 404, 390-394. Huang, H., Thompson, W., Paulus, M.P., 2017. Computational Dysfunctions in Anxiety: Failure to Differentiate Signal From Noise. Biol Psychiatry 82, 440-446. Insel, T., Cuthbert, B., Garvey, M., Heinssen, R., Pine, D.S., Quinn, K., Sanislow, C., Wang, P., 2010. Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. Am J Psychiatry 167, 748-751. Itti, L., Baldi, P., 2009. Bayesian surprise attracts human attention. Vision Res 49, 12951306.
21
Maia, T.V., Frank, M.J., 2011. From reinforcement learning models to psychiatric and neurological disorders. Nature Neuroscience 14, 154. McTeague, L.M., Lang, P.J., 2012. The anxiety spectrum and the reflex physiology of defense: from circumscribed fear to broad distress. Depress Anxiety 29, 264-281. Menon, V., 2011. Large-scale brain networks and psychopathology: a unifying triple network model. Trends in Cognitive Sciences 15, 483-506. Mulder, M.J., Wagenmakers, E.-J., Ratcliff, R., Boekel, W., Forstmann, B.U., 2012. Bias in the Brain: A Diffusion Model Analysis of Prior Probability and Potential Payoff. The Journal of Neuroscience 32, 2335. O'Reilly, J.X., Schuffelgen, U., Cuell, S.F., Behrens, T.E., Mars, R.B., Rushworth, M.F., 2013. Dissociable effects of surprise and model update in parietal and anterior cingulate cortex. Proc Natl Acad Sci U S A 110, E3660-3669. Paulus, M.P., 2019. Driven by Pain, Not Gain: Computational Approaches to AversionRelated Decision Making in Psychiatry. Biological Psychiatry. Paulus, M.P., Stein, M.B., 2006. An insular view of anxiety. Biol Psychiatry 60, 383-387. Pedersen, M.L., Frank, M.J., Biele, G., 2017. The drift diffusion model as the choice rule in reinforcement learning. Psychon Bull Rev 24, 1234-1251. Rao, R.P., Ballard, D.H., 1999. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci 2, 79-87. Ratcliff, R., Smith, P.L., Brown, S.D., McKoon, G., 2016. Diffusion Decision Model: Current Issues and History. Trends in Cognitive Sciences 20, 260-281.
22
Rauss, K., Schwartz, S., Pourtois, G., 2011. Top-down effects on early visual processing in humans: A predictive coding framework. Neuroscience & Biobehavioral Reviews 35, 1237-1253. Rescorla, R.A., Wagner, A.R., 1972. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement, In: Black, A.H., Proksasy, W.F. (Eds.), Classical Conditioning II: Current Research and Theory. Appleton-Century Crofts, New York, pp. 64-99. Sheehan, D.V., Lecrubier, Y., Sheehan, K.H., Amorim, P., Janavs, J., Weiller, E., Hergueta, T., Baker, R., Dunbar, G.C., 1998. The Mini-International Neuropsychiatric Interview (MINI): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. Journal of clinical psychiatry 59, 22-33. Stan Development Team, 2018. RStan: the R interface to Stan. Urai, A.E., de Gee, J.W., Tsetsos, K., Donner, T.H., 2019. Choice history biases subsequent evidence accumulation. eLife 8, e46331. Vehtari, A., Gelman, A., Gabry, J., 2017. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing 27, 1413-1432. Victor, T.A., Khalsa, S.S., Simmons, W.K., Feinstein, J.S., Savitz, J., Aupperle, R.L., Yeh, H.W., Bodurka, J., Paulus, M.P., 2018. Tulsa 1000: a naturalistic study protocol for multilevel assessment and outcome prediction in a large psychiatric sample. BMJ Open 8, e016620. Watson, D., Clark, L.A., 1999. The PANAS-X: Manual for the positive and negative affect schedule-expanded form.
23
Figure 1: Combined belief updating/visual perception task. (a) Subjects attempted to locate a target stimulus (one of three random-dot stimulus patches with a specified coherent motion direction) in one of three possible locations. The other two patches were distractors with the opposite motion direction. Subjects searched locations sequentially by pressing corresponding keyboard keys to view the associated random-dot stimuli (target or distractor stimulus) and made the final decision by pressing the up arrow key to indicate the last viewed stimulus was the target. (b) The target stimulus appeared in the three locations with a relative frequency of 1:3:9, with the reward contingency at the three locations changing every 30 trials on average. Figure reused with permission from (Huang et al., 2017).
24
Figure 2: Hybrid Rescorla Wagner-Drift Diffusion Model. On each trial, subjects update the expected value of each location (i.e. probability that the target will appear in each location) based on the outcome of the previous trial. The rate of updating is determined by a subject-level learning rate parameter. This updating of expectations according to the Rescorla-Wagner rule is shown on the left side of the figure. These expectations then influence perceptual processing when subjects view the random-dot stimulus on each trial: expectations determine the starting point for a drift diffusion process, in which noisy perceptual evidence is accumulated until one of two decision boundaries is reached, as shown on the right side of the figure.
25
Figure 3: Relationships Between Fear and Learning Rates. (a) Relationship between perceptual learning rate and PANAS X Fear score, controlling for gender and age. The hierarchical model additionally included relationships between the decision learning rate and the same individual-level predictors (see Figure 3b). For
26
each parameter, the median of the posterior distribution (black circle), 80% credible interval (red line), and 95% credible interval (black line) are shown. Individual-level predictors (i.e. PANAS X Fear score, gender, and age) were scaled prior to model-fitting. PANAS X Fear score was negatively associated with the perceptual learning rate. (b) Relationship between decision learning rate and PANAS X Fear score, controlling for gender and age. The hierarchical model additionally included relationships between the perceptual learning rate and the same individual-level predictors (see Figure 3a). Individual-level predictors were scaled prior to model-fitting. PANAS X Fear score was not associated with the decision learning rate.
27
Where Perception Meets Belief Updating: Computational Evidence for Slower Updating of Visual Expectations in Anxious Individuals Jonathon R. Howlett , Martin P. Paulus PII: DOI: Reference:
S0165-0327(19)32093-2 https://doi.org/10.1016/j.jad.2020.02.012 JAD 11643
To appear in:
Journal of Affective Disorders
Received date: Revised date: Accepted date:
6 August 2019 31 January 2020 1 February 2020
Please cite this article as: Jonathon R. Howlett , Martin P. Paulus , Where Perception Meets Belief Updating: Computational Evidence for Slower Updating of Visual Expectations in Anxious Individuals, Journal of Affective Disorders (2020), doi: https://doi.org/10.1016/j.jad.2020.02.012
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2020 Published by Elsevier B.V.
Highlights
Decision-making and perceptual processing rely on separate belief updating processes
Anxiety is associated with slower updating of perceptual expectations
Anxiety is not associated with altered updating of beliefs for decision-making
Older age was also associated with slower updating of perceptual expectations
1
Where Perception Meets Belief Updating: Computational Evidence for Slower Updating of Visual Expectations in Anxious Individuals
Jonathon R. Howlett 1,2* and Martin P. Paulus1,2
1
Department of Psychiatry, University of California San Diego, La Jolla, CA, USA
2
Laureate Institute for Brain Research, Tulsa, OK, USA
*Corresponding Author: Jonathon R. Howlett, MD University of California San Diego Department of Psychiatry 9500 Gilman Dr La Jolla, CA 92093 [email protected]
2
Where Perception Meets Belief Updating: Computational Evidence for Slower Updating of Visual Expectations in Anxious Individuals
Abstract Background: Surprising events are important sources of internal model updating which adjusts expectations for both decision-making and perceptual processing circuits. Anxious individuals display relatively intact updating of internal models used to make decisions, however how these individuals update their perceptual expectations remains poorly understood. Based on previous work, we hypothesized that anxious individuals experienced exaggerated surprise to predictable events, which imbues them with undue salience. Methods: To model the rate of updating of internal models for both decision-making and perceptual processing, we applied a hybrid Rescorla Wagner (RW)/Drift Diffusion Model (DDM) to a change point detection task in a transdiagnostic group of individuals with mood and anxiety disorders and examined the relationship between learning rates and anxiety and negative affect. Results: Model comparison provided evidence that decision-making and perceptual processing rely on separate internal models with different learning rates. Anxiety and older age were associated with slower updating of models used in perceptual processing, but not those used in decision-making. Limitations: This was a cross-sectional study and lacked neural data to examine the role of specific brain circuits in updating of perceptual predictions.
3
Conclusions: Anxious individuals display slower updating of internal models used in perceptual processing, but not those used in decision-making. This deficit could contribute to exaggerated salience of harmless stimuli in anxiety. The results have implications for the assessment and treatment of basic processing dysfunctions in anxiety.
Keywords: Computational Psychiatry; Anxiety; Rescorla-Wagner; Drift Diffusion Model; Perception; Prediction Error
4
Introduction Surprising events often signal the need for immediate allocation of processing resources, but can also lead us to update our predictive models to avoid being surprised again (O'Reilly et al., 2013). This surprise-driven learning is formalized computationally by the Rescorla-Wagner (RW) model, which holds that the change in expectation after new information is equal to the prediction error scaled by a learning rate (Rescorla and Wagner, 1972). Surprising events are thus thought to generate teaching signals that drive updating of internal predictive models, consistent with empirical evidence that prediction errors are associated with robust activations across a number of brain regions (Maia and Frank, 2011). While initially proposed to account for results from classical conditioning studies, RW is now frequently used to study reward-based decision-making, in which expectations generated by a continuously updated internal model are presumably relayed to action selection circuits in the brain (Maia and Frank, 2011). However, internal expectations are also relayed to perceptual processing regions via top-down neural projections (Rao and Ballard, 1999; Rauss et al., 2011). Deficits in the updating of these perceptual predictions could lead individuals to be repeatedly surprised by predictable events, exaggerating the salience of innocuous stimuli and resulting in excessive allocation of neural resources. Given the conceptualization of anxiety as a disorder of aberrant internal prediction signals (Paulus and Stein, 2006) and the evidence for an overactive salience network in anxiety (Menon, 2011), an examination of the updating of perceptual expectations could contribute to a mechanistic account of dysfunction in anxiety and enable new approaches to assessment and treatment of anxiety-related disorders.
5
Studies of decision-making in anxious individuals have not revealed higher or lower learning rates per se and instead have shown subtler deficits in regulation of learning based on context; in this sense, updating of internal models driving decisions appears relatively intact in anxious individuals (Brown et al., 2018; Browning et al., 2015; Huang et al., 2017). However, the relationship between predictive models that drive decision-making and those that influence perceptual processing are poorly understood, and these may represent fundamentally different neural processes. Dysfunctional updating of perceptual expectations in anxious individuals could help explain the overreaction to harmless stimuli in anxiety and would have important implications for assessment and treatment. An ideal computational method to measure this impairment is to combine RW with a drift diffusion model (DDM), a common approach to studying perceptual decision-making in which presumably noisy evidence is accumulated over time toward one of two decision thresholds, and a decision is made once a decision threshold is reached (Ratcliff et al., 2016). The DDM has been used extensively to model both response accuracy and response times on a variety of 2alternative forced choice tasks, including perceptual decision-making tasks (Ratcliff et al., 2016). The central aspect of a combined RW/DDM model is that prior expectations generated by RW can be incorporated into the DDM through a bias parameter, which allows the starting point for evidence accumulation to be nearer to the boundary representing the expected outcome (Mulder et al., 2012). Recent evidence also indicates that prior information can influence the DDM drift rate parameter, which quantifies the rate of accumulation of perceptual information (Urai et al., 2019). While DDM has recently been employed to study reward-based decision-making (Pedersen et al., 2017),
6
to our knowledge it has never been combined with RW to examine the updating of perceptual expectations via the DDM bias and drift rate parameters in a combined belief updating/perceptual paradigm. We employed this approach in a large transdiagnostic psychiatric sample of healthy volunteers and individuals with mood and anxiety problems, consistent with the National Institute of Mental Health (NIMH) Research Domain Criteria (RDoC) framework, which aims to develop a dimensional, neuroscience-based psychiatric classification system (Insel et al., 2010). RDoC consists of specific domains of functioning, with the negative valence domain being particularly relevant to anxiety-related disorders. Within this domain, fear, which has an arousal component, is distinguished from negative affect more generally, with implications for assessment and treatment of individuals with anxiety-related complaints (McTeague and Lang, 2012). We hypothesized that fearful individuals would exhibit poorer updating of perceptual expectations.
Method Participants The experiment was part of the Tulsa-1000 (T-1000) study (Victor et al., 2018), which is aimed at multilevel assessment and outcome prediction in a large, heterogeneous psychiatric sample. Subjects were recruited from mental health providers or via advertisements. At screening, treatment-seeking individuals were required to have a score on the Patient Health Questionnaire-9 (PHQ-9) ≥ 10 and/or on the Overall Anxiety Severity and Impairment Scale (OASIS) ≥ 8. 306 subjects (age: 35.28 ± 11.31 years; gender: 101 male and 205 female) participated in the experiment. See Supplementary
7
Table 1 for subject diagnoses based on the based on the MINI Neuropsychiatric Interview (MINI) (Sheehan et al., 1998). 167 subjects were medicated (see Supplementary Table 2 for classes of psychotropic medications). 35 subjects were excluded due to low accuracy on the perceptual component of the task (see below). To assess affective state including fear and negative affect, participants completed the Positive and Negative Affect Schedule-Expanded Form (PANAS X) (Watson and Clark, 1999). All study procedures were approved by the Western Institutional Review Board, and all participants provided written informed consent prior to participation.
Experiment Subjects completed a combined belief updating/visual perception task (see Fig.1) (Huang et al., 2017). Subjects attempted to identify the location of a target stimulus, which is one of three random-dot stimulus patches (Gold and Shadlen, 2000) with a specified coherent motion direction (random-dot coherence = 30%). The other two patches were distractors with the opposite motion direction. The specified coherent motion direction was relayed to the subject and kept consistent throughout the task. At the start of each trial, three dots indicated the locations in which the target could appear. Subjects searched locations sequentially by pressing corresponding keyboard keys and made the final decision by pressing the up-arrow key to indicate the last viewed stimulus was the target. Upon selecting a location, the random-dot stimulus patch was shown. If the initially chosen location was correct, subjects could therefore determine this based on the motion direction and terminate the trial without any further searching; if not, subjects would then select another location to continue to search for the target. Subjects were
8
allowed to search until they terminated the trial by indicating that they were viewing the target stimulus. At the end of each trial, the true target location was shown to provide feedback. The target stimulus appeared in the three locations with a relative frequency of 1:3:9, with the reward contingency at the three locations changing every 30 trials on average. See Supplementary Methods for further experiment details.
Model Model fitting was performed in Bayesian hierarchical fashion using the RStan (Stan Development Team, 2018) implementation of the Stan (Carpenter et al., 2017) language. Models were fit using the initial location choice on each trial and the randomdot reaction time for the initially chosen location (indicating whether the location did or did not contain the target). The model assumes that expectations regarding target location, based on the sequence of prior target locations, influences both (a) the initial location choice on a trial and (b) the response and reaction time to the random-dot stimulus shown at the initially chosen location. Regarding location choice, subjects will be more likely to choose an initial location with a greater expectation of choosing the target. Regarding reaction time to the random-dot stimulus, higher expectation that the location is correct will lead to faster choices confirming this (if the location does indeed contain the target) but slower choices disconfirming this (if the location does not contain the target). For both location decisions and random-dot responses, expectations are modeled using an RW model with a corresponding learning rate, which can be fit using location choices, random-dot responses and reaction times, or both.
9
The updating of location expectations based on the true target location on each trial was modeled using an RW model, with a learning rate α parameter for each subject (the learning rate was fit using all trials across the task). Each learning rate was estimated based on subject choices (the decision learning rate), on reaction times and responses to the random-dot stimulus (the perceptual learning rate), or both (see Fig. 2 and Supplementary Fig. S1). For the decision learning rate, subjects’ choices were modeled as being dependent on RW expectations via a softmax decision model with an inverse decision temperature parameter controlling the randomness of decisions. For the perceptual learning rate, RW expectations influenced either the DDM bias parameter, DDM drift rate parameter, or both, depending on the model (with model comparisons performed to determine the best model, as discussed below). The influence of RW expectations on DDM parameters was scaled by ξ parameters determining the strength of influence of expectations on perceptual processing. Each individual learning rate was modeled in Bayesian hierarchical fashion as depending on a group-level mean and also potentially on individual-level predictors such as self-reported fear, gender, and age. See Supplementary Methods for further modeling details.
Model Comparison We performed model comparisons using the widely applicable information criterion (WAIC) with the loo package in R (Vehtari et al., 2017). Model comparisons were performed to select the best performing model based on two aspects: (1) the manner in which RW expectations influenced perceptual processing as modeled by DDM—only via the DDM bias parameter (bias-only model), only via the DDM drift rate parameter
10
(drift-only model), or via both the bias and drift rate parameter (bias and drift model); (2) whether a single updating process, with a single learning rate, supplies expectations to both decision-making and perceptual circuits, or whether there are two separate updating processes with different learning rates. To address this second aspect, we constructed two categories of models. In the first category of models (the single α models), a single learning rate for each subject was fit both to choices (via a softmax decision policy) and to responses and reaction times to the random-dot stimulus (via DDM). The second category of models (dual α models) incorporated two separate updating processes, with two separate learning rates, one fit to choices (the decision learning rate) and the other to responses and reaction times to the random-dot stimulus (the perceptual learning rate). We performed a model comparison of six models (bias-only single α model, biasonly dual α model, drift-only single α model, drift-only dual α model, bias and drift single α model, and bias and drift dual α model). All models predicted both categorical location choices and random-dot reaction times, so they could be compared directly despite the different nature of these two types of data. To assess the relationship between decision learning rate and perceptual learning rate in the dual α model, we calculated the Pearson correlation coefficient of the raw decision learning rate parameter and the raw perceptual learning rate parameter for each draw of the posterior distribution.
Individual-Level Predictors and Model Parameters To determine the relationship between fear and model parameters, we constructed a hierarchical model in which both subject-level learning rates depended on scaled age,
11
gender, and PANAS X Fear. Effect sizes for the relationships between subject-level predictors and subject-level α were calculated by dividing each draw of the posterior distribution of the slope by the posterior distribution of the standard deviation of the raw subject-level α parameter (see Supplementary Methods). This yielded the equivalent of a standardized regression coefficient, i.e. an estimate of the change (in standard deviations) of the dependent variable per standard deviation of the predictor variable. To determine whether the relationship between model parameters and fear was being driven by general negative affect, we also constructed a model in which PANAS X Negative Affect was additionally included as an individual-level predictor. Finally, to supplement our hypothesis-driven analysis of the relationship between perceptual learning rate and fear, we also assessed the relationship between model parameters and higher-level affective dimensions. We constructed a model in which PANAS X Positive Affect and PANAS X Positive Affect (the two higher-order dimensions of the PANAS X scale) were included as individual-level predictors, along with age and gender, but without any specific affects measured by PANAS X (e.g. fear).
Parameter Reliability To estimate parameter reliability, we constructed a different model with separate learning rates and inverse decision temperature parameters for each block (unlike in the main model in which learning rates and inverse decision temperatures were fit using all trials on the task). We then calculated intraclass correlations (ICC) for each parameter.
Results
12
Model Comparison Model comparison using WAIC indicated that the bias and drift dual α model provided the best fit for the observed data compared to the single α model (bias-only single α model WAIC: 141094.4; bias-only dual α model WAIC: 139185.4; drift-only single α model WAIC: 144165.4; drift-only dual α model WAIC: 137661.8; bias and drift single α model WAIC: 138590.1; bias and drift dual α model WAIC: 132962.4). WAIC accounts for model complexity, indicating that the additional complexity of the dual α model is justified by the improvement in predictive accuracy and providing evidence for separate updating processes for decision-making and perceptual processing. Median correlation between the raw decision learning rate parameter and the raw perceptual learning rate parameter was r = .12. The 95% credible interval (i.e. the interval within which 95% of the Bayesian posterior probability distribution fell) was [.05, .18].
Individual-Level Predictors and Model Parameters PANAS X Fear score was negatively associated with perceptual learning rate (median β = -0.15, 95% credible interval β = [-0.27, -0.02]), i.e. individuals who reported the highest fear scores showed the lowest rate of perceptual updating (see Fig. 3a). In contrast, fear was not related to decision learning rate (β = -0.01 [-0.14, 0.12]; Fig. 3b). Similarly, age was also negatively associated with perceptual learning rate (β = -0.40 [0.53, -0.27]) but not decision learning rate (β = -0.08 [-0.21, 0.05]), i.e. older individuals showed slower perceptual but not decisional updating. Male gender was positively associated with perceptual learning rate (β = 0.14 [0.02, 0.27]) but not decision learning rate (β = -0.07 [-0.20, 0.07]).
13
When PANAS X Negative Affect score was added to the model, the median estimate of the association between fear and perceptual learning rate was more negative, but the posterior distribution widened such that 0 was included in the 95% credible interval (β = -0.21 [-0.49, 0.07]; Supplementary Fig. S2). Negative affect was not associated with perceptual learning rate (β = 0.07 [-0.19, 0.36]), indicating that general negative affect does not drive the association between fear and perceptual learning rate. In the model in which PANAS X Positive Affect and PANAS X Negative Affect were included as predictors, the median estimate for relationship between PANAS X Negative Affect and perceptual learning rate was β = -0.11 ([-0.25, 0.03] ; Supplementary Fig. S3), while the median estimate of the relationship with decision learning rate was β = -0.05 ([-0.19, 0.09]). The median estimate for relationship between PANAS X Positive Affect and perceptual learning rate was β = -0.00 ([-0.14, 0.14]), while the median estimate of the relationship with decision learning rate was β = -0.12 ([-0.27, 0.02]).
Parameter Reliability For the decision learning rate, median ICC was .62 (95% credible interval: [.54, .69]. For the perceptual learning rate, median ICC was .80 (95% credible interval: [.75, .85]. For the inverse decision temperature parameter, median ICC was .47 (95% credible interval: [.37, .56].
Discussion The goal of this investigation was to examine the degree to which anxiety and/or negative affect in a transdiagnostic group of individuals with mood and anxiety disorders
14
influence (1) the rate of updating of an internal model that relays expectations to perceptual circuits and (2) the rate of updating of an internal model that relays expectations to decision-making circuits, and to provide evidence for whether these represent two separate updating processes. We found that anxious (and older) individuals exhibit slower updating of the internal model that influences perceptual processing, but not the model that influences decision-making. We found evidence that these two models employ separate updating processes with separate learning rates (a decision learning rate and a perceptual learning rate), which are only weakly correlated. The reliability of estimation of the perceptual learning rate was significantly greater than for the decision learning rate. We did not observe a relationship between fear and learning rate with our standard RW model, based on sequential choices. This result is consistent with previous studies of sequential decision making in anxiety, which have found that anxiety is not associated with alterations in RW learning rate per se. Instead, anxiety has been associated with an impaired ability to adapt learning rate based on environmental context (Browning et al., 2015) and with a higher base learning rate in a more complex model in which the learning rate can increase at certain times during the task (Huang et al., 2017). Similarly, posttraumatic stress disorder (PTSD) has been associated with higher updating of associability (a marker of attentional allocation to unexpected outcomes) in a learning task, but not with higher learning rates (Brown et al., 2018). Taken together, these prior findings and our current results indicate that anxiety may have a selective effect in slowing the updating of expectations that are relayed to perceptual circuits, rather than those relayed to decision-making circuits. Such slow updating could result from
15
hyperprecise prior beliefs, i.e. rigid expectations that are resistant to alteration in light of new information (Paulus, 2019). This slow updating of perceptual predictions could lead anxious individuals to be repeatedly surprised by predictable events. Surprise is fundamentally related to salience (Itti and Baldi, 2009), and surprising events drive immediate reallocation of attentional processing resources (O'Reilly et al., 2013). If anxious individuals are slow to update perceptual expectations, resulting in continued surprise signals driven by predictable events, this could result in overallocation of resources to process possibly threat-related, or even innocuous cues, leading to an increase in tonic arousal levels. Such a processing dysfunction could be a target for individualized assessment and intervention. However, given that our paradigm does not allow us to directly measure surprise, future research will be necessary to confirm this relationship. Functional neuroimaging could help differentiate surprise (which is associated with parietal cortex activation) from model updating (which is associated with dorsal anterior cingulate cortex activation) (O'Reilly et al., 2013). Differential allocation of attention may also play a role in differences in updating of perceptual expectations. Strengths of this study include a large, heterogeneous psychiatric sample and a paradigm and modeling approach that are ideally suited to examine the updating of perceptual expectations. Another strength is the hierarchical Bayesian model-fitting approach, which enables accurate characterization of different sources of variance (Boehm et al., 2018). Limitations include the cross-sectional design, which cannot reveal the relationship between model parameters and symptoms over time, lack of fMRI data to examine the neural mechanisms underlying altered updating, and the need to fit all parameters except learning rate at the group level only to ensure model convergence.
16
Another limitation is that some subjects were medicated, and medication may have influenced performance on the task. Finally, while we focused on the relationship between perceptual learning rate and fear, our sample size did not enable us to determine the extent to which this relationship is specific, or extends to other affects including general negative affect or other specific affects. It is also important to note that our task does not have an explicit fear component. Future research will be necessary to examine the relationship between perceptual updating and other affective processes. In summary, we found that anxious individuals are slower at updating perceptual expectations on a combined sequential decision-making/visual perception task. We additionally found evidence that the internal predictive model for perceptual processing differs from the model for decision-making, and that anxious individuals did not exhibit impairments in the latter process. The results imply that studying belief updating using hybrid decision-making/perceptual paradigms can reveal deficits in anxiety that are obscured by studying decision-making alone, a finding with implications for assessment and treatment of basic processing dysfunctions in anxious individuals.
Author Statement The authors have read and approved the final version of the manuscript. We confirm that it is the authors’ original work, has not received prior publication, and is not under consideration for publication elsewhere.
Jonathon R. Howlett and Martin P. Paulus conceived of, designed, and interpreted the analyses and drafted the manuscript.
17
Disclosure The authors declare no conflicts of interest.
Acknowledgement This work has been supported in part by The William K. Warren Foundation and the National Institute of General Medical Sciences Center Grant Award Number 1P20GM121312. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The http://ClinicalTrials.govidentifier for the clinical protocol associated with data published in the current paper is NCT02450240, “Latent Structure of Multi-level Assessments and Predictors of Outcomes in Psychiatric Disorders”.
The Tulsa 1000 Investigators include the following contributors: Robin Aupperle, Ph.D., Jerzy Bodurka, Ph.D., Yoon-Hee Cha, M.D., Justin Feinstein, Ph.D., Sahib S. Khalsa, M.D., Ph.D., Rayus Kuplicki, Ph.D., Martin P. Paulus, M.D., Jonathan Savitz, Ph.D., Teresa A. Victor, Ph.D.
The funder had no role in study design, in the collection, analysis, and interpretation of data, in the writing of the manuscript, or in the decision to submit the paper for publication.
18
Contributors Jonathon R. Howlett and Martin P. Paulus conceived of, designed, and interpreted the analyses and drafted the manuscript. The authors have read and approved the final version of the manuscript. We confirm that it is the authors’ original work, has not received prior publication, and is not under consideration for publication elsewhere.
Acknowledgement This work has been supported in part by The William K. Warren Foundation and the National Institute of General Medical Sciences Center Grant Award Number 1P20GM121312. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The http://ClinicalTrials.govidentifier for the clinical protocol associated with data published in the current paper is NCT02450240, “Latent Structure of Multi-level Assessments and Predictors of Outcomes in Psychiatric Disorders”.
The Tulsa 1000 Investigators include the following contributors: Robin Aupperle, Ph.D., Jerzy Bodurka, Ph.D., Yoon-Hee Cha, M.D., Justin Feinstein, Ph.D., Sahib S. Khalsa, M.D., Ph.D., Rayus Kuplicki, Ph.D., Martin P. Paulus, M.D., Jonathan Savitz, Ph.D., Teresa A. Victor, Ph.D.
Role of the Funding Source
19
The funder had no role in study design, in the collection, analysis, and interpretation of data, in the writing of the manuscript, or in the decision to submit the paper for publication.
Disclosure The authors declare no conflicts of interest.
Acknowledgement This work has been supported in part by The William K. Warren Foundation and the National Institute of General Medical Sciences Center Grant Award Number 1P20GM121312. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The http://ClinicalTrials.govidentifier for the clinical protocol associated with data published in the current paper is NCT02450240, “Latent Structure of Multi-level Assessments and Predictors of Outcomes in Psychiatric Disorders”.
The Tulsa 1000 Investigators include the following contributors: Robin Aupperle, Ph.D., Jerzy Bodurka, Ph.D., Yoon-Hee Cha, M.D., Justin Feinstein, Ph.D., Sahib S. Khalsa, M.D., Ph.D., Rayus Kuplicki, Ph.D., Martin P. Paulus, M.D., Jonathan Savitz, Ph.D., Teresa A. Victor, Ph.D.
20
References Boehm, U., Marsman, M., Matzke, D., Wagenmakers, E.J., 2018. On the importance of avoiding shortcuts in applying cognitive models to hierarchical data. Behav Res Methods 50, 1614-1631. Brown, V.M., Zhu, L., Wang, J.M., Frueh, B.C., King-Casas, B., Chiu, P.H., 2018. Associability-modulated loss learning is increased in posttraumatic stress disorder. eLife 7, e30150. Browning, M., Behrens, T.E., Jocham, G., O'Reilly, J.X., Bishop, S.J., 2015. Anxious individuals have difficulty learning the causal statistics of aversive environments. Nat Neurosci 18, 590-596. Carpenter, B., Gelman, A., Hoffman, M.D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., Riddell, A., 2017. Stan: A Probabilistic Programming Language. Journal of Statistical Software, Medium: ED; Size: 32 p. Gold, J.I., Shadlen, M.N., 2000. Representation of a perceptual decision in developing oculomotor commands. Nature 404, 390-394. Huang, H., Thompson, W., Paulus, M.P., 2017. Computational Dysfunctions in Anxiety: Failure to Differentiate Signal From Noise. Biol Psychiatry 82, 440-446. Insel, T., Cuthbert, B., Garvey, M., Heinssen, R., Pine, D.S., Quinn, K., Sanislow, C., Wang, P., 2010. Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. Am J Psychiatry 167, 748-751. Itti, L., Baldi, P., 2009. Bayesian surprise attracts human attention. Vision Res 49, 12951306.
21
Maia, T.V., Frank, M.J., 2011. From reinforcement learning models to psychiatric and neurological disorders. Nature Neuroscience 14, 154. McTeague, L.M., Lang, P.J., 2012. The anxiety spectrum and the reflex physiology of defense: from circumscribed fear to broad distress. Depress Anxiety 29, 264-281. Menon, V., 2011. Large-scale brain networks and psychopathology: a unifying triple network model. Trends in Cognitive Sciences 15, 483-506. Mulder, M.J., Wagenmakers, E.-J., Ratcliff, R., Boekel, W., Forstmann, B.U., 2012. Bias in the Brain: A Diffusion Model Analysis of Prior Probability and Potential Payoff. The Journal of Neuroscience 32, 2335. O'Reilly, J.X., Schuffelgen, U., Cuell, S.F., Behrens, T.E., Mars, R.B., Rushworth, M.F., 2013. Dissociable effects of surprise and model update in parietal and anterior cingulate cortex. Proc Natl Acad Sci U S A 110, E3660-3669. Paulus, M.P., 2019. Driven by Pain, Not Gain: Computational Approaches to AversionRelated Decision Making in Psychiatry. Biological Psychiatry. Paulus, M.P., Stein, M.B., 2006. An insular view of anxiety. Biol Psychiatry 60, 383-387. Pedersen, M.L., Frank, M.J., Biele, G., 2017. The drift diffusion model as the choice rule in reinforcement learning. Psychon Bull Rev 24, 1234-1251. Rao, R.P., Ballard, D.H., 1999. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci 2, 79-87. Ratcliff, R., Smith, P.L., Brown, S.D., McKoon, G., 2016. Diffusion Decision Model: Current Issues and History. Trends in Cognitive Sciences 20, 260-281.
22
Rauss, K., Schwartz, S., Pourtois, G., 2011. Top-down effects on early visual processing in humans: A predictive coding framework. Neuroscience & Biobehavioral Reviews 35, 1237-1253. Rescorla, R.A., Wagner, A.R., 1972. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement, In: Black, A.H., Proksasy, W.F. (Eds.), Classical Conditioning II: Current Research and Theory. Appleton-Century Crofts, New York, pp. 64-99. Sheehan, D.V., Lecrubier, Y., Sheehan, K.H., Amorim, P., Janavs, J., Weiller, E., Hergueta, T., Baker, R., Dunbar, G.C., 1998. The Mini-International Neuropsychiatric Interview (MINI): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. Journal of clinical psychiatry 59, 22-33. Stan Development Team, 2018. RStan: the R interface to Stan. Urai, A.E., de Gee, J.W., Tsetsos, K., Donner, T.H., 2019. Choice history biases subsequent evidence accumulation. eLife 8, e46331. Vehtari, A., Gelman, A., Gabry, J., 2017. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing 27, 1413-1432. Victor, T.A., Khalsa, S.S., Simmons, W.K., Feinstein, J.S., Savitz, J., Aupperle, R.L., Yeh, H.W., Bodurka, J., Paulus, M.P., 2018. Tulsa 1000: a naturalistic study protocol for multilevel assessment and outcome prediction in a large psychiatric sample. BMJ Open 8, e016620. Watson, D., Clark, L.A., 1999. The PANAS-X: Manual for the positive and negative affect schedule-expanded form.
23
Figure 1: Combined belief updating/visual perception task. (a) Subjects attempted to locate a target stimulus (one of three random-dot stimulus patches with a specified coherent motion direction) in one of three possible locations. The other two patches were distractors with the opposite motion direction. Subjects searched locations sequentially by pressing corresponding keyboard keys to view the associated random-dot stimuli (target or distractor stimulus) and made the final decision by pressing the up arrow key to indicate the last viewed stimulus was the target. (b) The target stimulus appeared in the three locations with a relative frequency of 1:3:9, with the reward contingency at the three locations changing every 30 trials on average. Figure reused with permission from (Huang et al., 2017).
24
Figure 2: Hybrid Rescorla Wagner-Drift Diffusion Model. On each trial, subjects update the expected value of each location (i.e. probability that the target will appear in each location) based on the outcome of the previous trial. The rate of updating is determined by a subject-level learning rate parameter. This updating of expectations according to the Rescorla-Wagner rule is shown on the left side of the figure. These expectations then influence perceptual processing when subjects view the random-dot stimulus on each trial: expectations determine the starting point for a drift diffusion process, in which noisy perceptual evidence is accumulated until one of two decision boundaries is reached, as shown on the right side of the figure.
25
Figure 3: Relationships Between Fear and Learning Rates. (a) Relationship between perceptual learning rate and PANAS X Fear score, controlling for gender and age. The hierarchical model additionally included relationships between the decision learning rate and the same individual-level predictors (see Figure 3b). For
26
each parameter, the median of the posterior distribution (black circle), 80% credible interval (red line), and 95% credible interval (black line) are shown. Individual-level predictors (i.e. PANAS X Fear score, gender, and age) were scaled prior to model-fitting. PANAS X Fear score was negatively associated with the perceptual learning rate. (b) Relationship between decision learning rate and PANAS X Fear score, controlling for gender and age. The hierarchical model additionally included relationships between the perceptual learning rate and the same individual-level predictors (see Figure 3a). Individual-level predictors were scaled prior to model-fitting. PANAS X Fear score was not associated with the decision learning rate.
27