From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Vision Research xxx (2014) xxx–xxx 1

Contents lists available at ScienceDirect

Vision Research journal homepage: www.elsevier.com/locate/visres 5 6

From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays

3 4 7

Q1

8

Philip L. Smith ⇑, David K. Sewell, Simon D. Lilburn Melbourne School of Psychological Sciences, The University of Melbourne, Australia

9 10

a r t i c l e

1 2 2 4 13 14 15 16

i n f o

Article history: Received 18 June 2014 Received in revised form 25 October 2014 Available online xxxx

17 18 19 20 21 22 23

Keywords: Attention Normalization Shunting equation Salience Decision making

a b s t r a c t Normalization models of visual sensitivity assume that the response of a visual mechanism is scaled divisively by the sum of the activity in the excitatory and inhibitory mechanisms in its neighborhood. Normalization models of attention assume that the weighting of excitatory and inhibitory mechanisms is modulated by attention. Such models have provided explanations of the effects of attention in both behavioral and single-cell recording studies. We show how normalization models can be obtained as the asymptotic solutions of shunting differential equations, in which stimulus inputs and the activity in the mechanism control growth rates multiplicatively rather than additively. The value of the shunting equation approach is that it characterizes the entire time course of the response, not just its asymptotic strength. We describe two models of attention based on shunting dynamics, the integrated system model of Smith and Ratcliff (2009) and the competitive interaction theory of Smith and Sewell (2013). These models assume that attention, stimulus salience, and the observer’s strategy for the task jointly determine the selection of stimuli into visual short-term memory (VSTM) and way in which stimulus representations are weighted. The quality of the VSTM representation determines the speed and accuracy of the decision. The models provide a unified account of a variety of attentional phenomena found in psychophysical tasks using single-element and multi-element displays. Our results show the generality and utility of the normalization approach to modeling attention. Ó 2014 Published by Elsevier Ltd.

25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

44

1. Introduction

45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61

Normalization, or divisive normalization, models offer a simple and powerful formalism for characterizing a variety of visual phenomena, including the effects of covert attention. Normalization models have provided theoretical accounts of such diverse phenomena as lightness adaptation (Sperling & Sondhi, 1968), contrast sensitivity (Heeger, 1991, 1992) and contrast gain control (Geisler & Albrecht, 1997; Ross & Speed, 1991; Scholl, Latimer & Priebe, Q2 2012; Wilson & Kim, 1998), pattern masking (Foley, 1994), efficient, decorrelated encoding of natural images (Schwartz & Simoncelli, 2001), and the psychophysics and neural correlates of attention (Boynton, 2005; Herrmann, Heeger & Carrasco, 2012; Herrmann, Montaser-Kouhsari, Carrasco, & Heeger, 2010; Lee, Itti, Koch, & Braun, 1999; Lee & Maunsell, 2009; Reynolds & Heeger, 2009). Carandini and Heeger (2012) surveyed the range of applications of normalization models and argued that normalization should be viewed as a ‘‘canonical neural computation.’’ ⇑ Corresponding author at: Melbourne School of Psychological Sciences,

Q1

The University of Melbourne, VIC 3010, Australia. E-mail address: [email protected] (P.L. Smith).

Computationally, the idea expressed in normalization models is that the response of a visual mechanism coding a target stimulus is modulated divisively by the sum of the activity in other mechanisms in its neighborhood. In normalization models of lightness adaptation and gain control, the divisive input depends on the contrast energy in the local surround; in models of masking and pattern vision, it depends on the activity in a local population of spatial-frequency and orientation tuned filters. Unlike traditional linear system models of vision (e.g., Campbell & Robson, 1968), in which one stimulus can influence the visual response to another stimulus only if their associated receptive fields have overlapping bandwidths, normalization models allow for a form of global influence that extends outside the classical receptive field (Foley, 1994; Heeger, 1991, 1992). They explain, for example, how the visual response to a grating stimulus can be influenced by the properties of a stimulus oriented at 90° to it. In this article, we investigate the temporal dynamics of normalization and show how the scope of normalization models can be greatly expanded if they are formulated dynamically. The key theoretical insight provided by normalization models of attention is that the sensory representations of stimuli depend on nonlinear interactions between stimuli and their surrounds, and these

http://dx.doi.org/10.1016/j.visres.2014.11.001 0042-6989/Ó 2014 Published by Elsevier Ltd.

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1

2

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

121

interactions can be modulated by attention. We show how normalization models can be obtained computationally as the asymptotic or steady-state solutions of a class of biologically-plausible systems of differential equations known as shunting equations (Grossberg, 1980). Whereas normalization models characterize the steady-state properties of the system, the solutions of the associated shunting equations characterize its entire time course. Formulating a model dynamically makes it possible to view a wide range of attentional phenomena as expressions of a common set of processing mechanisms, of which normalization is one manifestation. In the second part of this article, we review some of these phenomena and describe two models, one proposed by Smith and Ratcliff (2009) and the other proposed by Smith and Sewell (2013), to account for them. Both models use shunting equations to describe the formation of the stimulus representations that support perceptual decision making. Although the models and the experimental phenomena they seek to explain have been discussed in previous articles, our presentation here serves both to highlight the unity of the underlying theoretical and computational principles and to emphasize the relationship among what might otherwise seem a diverse and unrelated set of experimental findings. Some of these phenomena may not seem closely related to normalization, but we argue that they can all be viewed as manifestations of attentionally modulated shunting dynamics. An additional aim of the article is to provide a partial reformulation of the Smith and Ratcliff (2009) and Smith and Sewell (2013) models in order to clarify the relationship between them and to make them consistent with one another. The reformulation also provides an explicit mathematical expression of a theoretical principle that was only represented implicitly in the previous published versions of the models, namely, that stimulus selection and stimulus identification are carried out by visual pathways that code different aspects of the stimulus. Our aim in elaborating the models in this way is to emphasize their relationship to other current normalization models in the literature.

122

2. Normalization and shunting dynamics

123

Normalization models have provided successful descriptions of visual processes at different levels of analysis, ranging from the contrast sensitivity of single neurons to behavioral responses in perceptual judgment tasks. Normalization describes how the response, Ri , of one of a set of mechanisms, i, to a stimulus depends on the responses of other members of the set to the components of the stimulus, denoted Ij . In a typical normalization model, the response is described by an equation of the form

84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120

124 125 126 127 128 129 130

131

Ipi

Ri ¼ 133 134 135 136 137 138 139 140 141 142 143

144 146

ai þ

P

j bj I j

q :

ð1Þ

In this equation, ai is a constant that depends on the mechanism but is independent of the stimulus and the exponents p and q characterize the nonlinearities of the response. Typically they are power functions of low order (e.g., around 2), which may be equal to each other or different, depending on the setting. Probably the simplest normalization model is the ubiquitous Naka–Rushton/Michaelis–Menton model of contrast gain control (Albrecht & Geisler, 1991; Albrecht & Hamilton, 1982; Kaplan, Lee & Shapley, 1990; Ross & Speed, 1991; Sclar, Maunsell & Lennie, 1990; Scholl, Latimer & Priebe, 2012),

RðcÞ ¼ a

c2 : cin þ c2

ð2Þ

In this equation, c is stimulus contrast and RðcÞ is the associated perceptual response. The gain control model states that the contrast response is a nonlinear, saturating function of contrast power or energy, c2 . The constant a determines the saturation point and the constant cin , which represents the aggregated effects of inhibition, determines the horizontal position of the function on the log-contrast axis. The inhibitory constant can be written as cin ¼ c20:5 , where c0:5 is a semisaturation constant that characterizes the contrast at which RðcÞ attains half its maximum value. Heeger (1991, 1992) proposed a normalization model of cortical simple and complex cell responses of the form of Eq. (1). The general form of the equation in Heeger’s model is

147 148 149 150 151 152 153 154 155 156 157 158

159

Ei ðtÞ P Ri ðtÞ ¼ ; ai þ j Ej ðtÞ

ð3Þ

161

where Ei ðtÞ is the time-varying response of a spatiotemporally tuned mechanism. In Heeger’s model simple cell, Ei ðtÞ is the amplitude response of a half-squared linear operator with a specified spatial frequency and orientation tuning and a specified phase. (We have simplified Heeger’s notation a little for ease of exposition.) The half-squared operator behaves like a half-wave rectifier in that the mechanism responds selectively to either positive or negative contrast excursions, depending on its phase, consistent with the fact that neurons only respond either to contrast increments or decrements. The use of a half-squaring operator instead of a half-wave rectifying operator means that the response to low-intensity stimuli is less than would otherwise be the case, which is consistent with both the physiology (Heeger, 1991, Q3 1992) and the psychophysics (Laming, 1986). The sum in the denominator of Eq. (3) is over a set of four mechanisms whose phases vary in steps of 90°. The sum is proportional to the Fourier energy in the stimulus, which means the amplitude response of the model simple cell in Eq. (3) is normalized by stimulus energy. Heeger’s model complex cell is obtained by summing four simple cells with orthogonal phases (0°, 90°, 180° and 270°). The response of the model simple cell depends on the amplitude of the stimulus and is sensitive to its phase or contrast polarity; the response of the model complex cell depends only on the overall stimulus contrast energy. Foley (1994) proposed a masking model influenced by Heeger’s work, which is related to Eq. (1). Foley’s Model 2 has the form

162

 þ p bEc R¼ P q ; þ Zþ j bIj c

188

P

163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187

ð4Þ 190

P

where E ¼ k SE;k ck , and Ij ¼ k SI;jk ck are weighted sums of the k components of the stimulus, which individually have contrast ck . The weights characterize the sensitivities of excitatory and inhibitory mechanisms: SE;k is the excitatory sensitivity to the k-th stimulus component and SI;jk is the inhibitory sensitivity of the j-th divisive input to this same stimulus component. The notation bcþ ¼ maxð:; 0Þ, denotes half-wave rectification, which has similar properties to the half-squaring operator in Heeger’s model, except that the response is a linear rather than a nonlinear function of the stimulus strength. Models related to Eq. (4) continue to be influential in contemporary accounts of masking (e.g., Meese & Holmes, 2010).

191

2.1. Normalization models of attention

203

The effects of attention can be incorporated into normalization models in a very natural way by assuming that the excitatory and inhibitory components of the normalization equation are weighted by attention in some way, where the weights depend on the allocation of attention in space (or feature space). Lee et al.

204

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

192 193 194 195 196 197 198 199 200 201 202

205 206 207 208

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1 209 210 211

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

(1999) proposed an attention model related to Eq. (1), in which Rh;x , the response of a visual mechanism tuned to a stimulus with orientation h and spatial frequency x, is of the form

212

Rh;x ¼ 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235

Sd þ

239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270

h

0

;x0



W hh0 ;xx0 Eh0 ;x0

ð5Þ

d :

The denominator of this equation is a weighted sum of stimulus components, Eh0 ;x0 , in which the weights, W hh0 ;xx0 , express the orientation and spatial frequency bandwidths of the mechanisms contributing to the inhibitory response. Lee et al. called their model a ‘‘winner-take-all’’ model because they found that the main effect of dividing attention by introducing a concurrent discrimination task was to increase the values of the exponents c and d. The theoretical connection between the normalization in Eq. (5) and winnertake-all dynamics is via the properties of Minkowski summation. The p-th order Minkowski sum of a set of variables, xj , is defined P  1=p p to equal ; this sum approaches maxj ðxj Þ as p becomes j xj large. The terms in Eq. (5) perform a form of Minkowski summation on their arguments, so the finding that c and d are increased by attention is consistent with the idea that attention helps the system pick the largest of a set of stimulus components while suppressing the others. Reynolds and Heeger (2009) proposed a normalization model of attention that is similar in spirit to the model of Eq. (5), although the manner in which attention is presumed to act is different. In their model, Rðc; x; hÞ, the contrast response of a mechanism at spatial location x with tuning h, is given by

236

238

P

ðEh;x Þc

 Rðc; x; hÞ ¼

Eðx; h; cÞ r þ Eðx; h; cÞ  sðx; hÞ

þ ;

ð6Þ

where the asterisk denotes the convolution operator. In this equation, Eðx; h; cÞ represents the strength of the excitatory response to a stimulus at location x with tuning h and contrast c, and sðx; hÞ is termed the suppressive field. It expresses an attention-dependent weighting of the inhibitory responses of the mechanisms in the neighborhood of a target mechanism centered at ðx; hÞ. The tuning parameter, h, characterizes tuning across both spatial frequency and orientation. Although the weighting is expressed as convolution in the model, it is perhaps more easily viewed as a weighted sum, or inner product, between the stimulus and a weight function. The operation of taking an inner product of a stimulus sðxÞ with a weight function wðx  x0 Þ, where the latter is parameterized by its center location, x0 , is the same as convolving it with a spatially-reflected impulse response function, hðxÞ ¼ wðxÞ (e.g., Heeger, 1991, p. 120). This interpretation highlights the formal similarities between Eqs. (6) and (1). One of the attractions of the model of Eq. (6) is that it offers a resolution of the question of whether attention affects the contrast response function through contrast gain or response gain (Ling & Q4 Carrasco, 2006a; Ling, Liu & Carrasco, 2009). In the contrast gain model, the contrast response, Rc ðcÞ, to a stimulus of contrast c and attention weight of c, is RðccÞ; when c > 1 the contrast response is compressed on the contrast axis, making an attended stimulus uniformly more detectable than an unattended one. The effect of contrast gain is to produce a horizontal shift of the contrast-response function on the log-contrast axis. In the response gain model, the contrast response function is Rc ðcÞ ¼ cRðcÞ; the effect of attention is to multiplicatively scale the contrast response function, which corresponds to a vertical shift on the log-response axis. The predictions of the normalization model of Eq. (6) depend on the relative sizes of the attended region and the suppressive field

3

in the denominator. When the stimulus is small, with a small suppressive field relative to the size of the attended region of space (or feature space), the attention weight is applied uniformly to all components of the suppressive field and its effects can be factored out of the denominator of Eq. (6), which results in contrast gain. When the stimulus is large, with a large suppressive field relative to the size of the attended region, the attention weight applies unequally to the elements of the suppressive field, which results in response gain. These predictions of the model were confirmed experimentally by Herrmann et al. (2010). Normalization models with similar properties to the model of Eq. (6) have been proposed by Boynton (2005) and Lee and Maunsell (2009). The perceptual template model of Lu and Dosher (1998; Dosher and Lu, 2000) can also be reformulated as a normalization model (Lu & Dosher, 2008). In its more usual, signal-detection theory form, the perceptual template model assumes that sensitivity to a stimulus depends on an attentionweighted combination of additive and multiplicative noise, including internal noise in the observer and external noise in the display. The sum of these various noise sources determines the overall signal-to-noise ratio. Dao, Lu and Dosher (2006) showed that the multiplicative noise in the perceptual template model is mathematically equivalent to a gain control (i.e., normalization) model. A recent tutorial review of normalization and its role in the computational modeling of attention can be found in Carrasco (2014).

271

2.2. Dynamic normalization by shunting differential equations

297

Normalization equations like those in the previous section describe asymptotic, or steady-state, sensitivity as a function of the interactions among the set of mechanisms that code the stimulus, but they do not characterize its time course. (Heeger’s model of contrast sensitivity, discussed earlier, included temporal effects, but they were not essential to its core predictions and have been omitted from subsequent normalization models of attention.) This section shows how normalization equations can be obtained as the asymptotic solution of a class of biologically plausible differential equations called shunting equations. The theoretical link between shunting equations and normalization models was noted by Carandini and Heeger (2012) and emphasized by Foley, Grossberg and Mingolla (2012). The distinguishing feature of shunting equations is that – unlike the more commonly used additive equations of linear systems theory – the inputs to the system or the system’s own responses multiplicatively gate its growth. The Hodgkin–Huxley equations of neural conduction are equations of this type (Tuckwell, 1988). The neurocomputational properties of shunting equations have been studied in detail by Grossberg and colleagues: Collections of their articles can be found in Grossberg (1987a, 1987b) and a survey of equations that have been proposed as formal models of neural computation can be found in Grossberg (1988). The simplest way to introduce time dependency into equations like Eq. (1) is to assume that the temporal response to the stimulus and the strength of the interactions among the mechanisms that respond to it are separable. We assume that the temporal properties of the stimulus are described by a temporal response function, denoted lðtÞ, which characterizes its time course, independent of its intensity. The function lðtÞ will have either low-pass or bandpass filter characteristics, depending on the underlying mechanism, as proposed in models of visual temporal sensitivity

298

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296

299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1 331 332 333 334

335

337 338 339 340 341 342 343 344 345

346

348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369

370

372 373 374 375 376 377 378

4

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

(Watson, 1986). The specific form of the function used in our models is given in the Appendix A.1 The dynamic counterpart of Eq. (1) is a separable, feedforward, differential equation

"

dRi ¼ lðtÞ Ipi  ai Ri ðtÞ  dt

X

!q bj Ij

# Ri ðtÞ :

ð7Þ

j

In this equation, the excitatory input is a power of the stimulus component, Ii , and the inhibitory input is a power of the weighted sum of all of the components, Ij . The distinguishing feature of Eq. (7) that identifies it as a shunting equation is that the inhibitory components are multiplicatively gated by the activity Ri ðtÞ and by the input lðtÞ. There is an additional inhibitory component with weight ai that is also proportional to Ri ðtÞ. Eq. (7) can be solved by separation of variables (see Appendix A) to obtain 2

Ip 6 Ri ðtÞ ¼ 4 P ai þ j bj I j

3 ( " !q ! Z X 7 bj I j q 5 1  exp  ai þ j

t

#)

lðsÞ ds

:

ð8Þ

!

#

X dRi ¼ lðtÞ Ii  Ij Ri ðtÞ : dt j

j Ij

"

X Z 1  exp  Ij j

379

#)

t

lðsÞ ds

:

ð10Þ 381

0

P  The asymptotic solution to Eq. (10) is Ri ð1Þ ¼ Ii = j I j ; in this equation, the excitatory response is normalized by the sum of all stimulus components. The Naka–Rushton contrast gain control P model of Eq. (2) has this form, with Ii ¼ c2 and j–i Ij ¼ cin . One counterintuitive property of Eqs. (8) and (10) is that the rate of growth depends on the sum of excitatory and inhibitory inputs, which leads to the prediction that increasing inhibition will increase the growth rate. In the models we describe later in this article, we consider ways to decouple the growth rate from the asymptotic response. As a precursor to these considerations, we consider two simpler, separable, shunting equations. The first is a one-component simplification of Eq. (9),

382 383 384 385 386 387 388 389 390 391 392 393

394

dRi ðtÞ ¼ lðtÞIi ½1  Ri ðtÞ  lðtÞð1  Ii ÞRi ðtÞ: dt

ð11Þ

396

0

Eq. (8) states that the activity Ri ðtÞ grows exponentially to an asymptote that depends on the ratio of the excitatory and inhibitory terms in the shunting equation. The asymptotic solution, Ri ð1Þ, can be obtained by omitting lðtÞ from Eq. (7), setting dRi =dt ¼ 0 and then solving for Ri ðtÞ. The asymptotic solution is the first term in square brackets on the right and is equal to Eq. (1). This example shows that normalization equations can often be obtained as the asymptotic solutions of excitatory-inhibitory shunting equations. In Eq. (8), the approach to asymptote is controlled by the area under the sensory response function. This area depends on stimulus duration: When stimulus duration is long and the area is large, the solution will approach the theoretical maximum Ri ð1Þ. When stimulus duration is short, the growth in the sensory response will terminate before the theoretical maximum is achieved, because the response only changes while lðtÞ is nonzero. This property is important when attempting to characterize the differing attentional effects that are found with masked and unmasked stimuli, as we discuss subsequently. To help the reader get a feel for the properties of shunting equations, it is useful to consider another equation that is related to Eq. (7), but omits some of its nonessential features,

"

Ri ðtÞ ¼

!(

I Pi

ð9Þ

In this equation, the excitatory part of the response is proportional to the stimulus component Ii and the inhibitory part of the response is proportional to the sum of all stimulus components, Ij . The inhibitory sum is like the suppressive field in the denominator of the Reynolds and Heeger (2009) model, except that its components are not differentially weighted. The solution of Eq. (9) is 1 The sensory response function of Eqs. (24) and (25) parameterizes the time course of backwardly masked and unmasked stimuli, but does not directly characterize the processes that give rise to different time courses for different kinds of stimuli. As discussed by Smith, Ellis, Sewell, and Wolfgang (2010), a deeper insight into backward masking is provided by Francis’s (2000) idea of efficient masking, which he characterized using the analogy of adding cream to hot coffee in a way that maximizes the cooling effect. Maximum cooling is achieved by adding cream after a delay rather than immediately. This exploits the fact that the natural cooling effect is greatest when the coffee is hottest, so the greatest cooling will be obtained if the coffee is allowed to cool for a time before adding cream. An important feature of Francis’s work is that his efficient masking model can be formulated mathematically using a shunting equation and so can be viewed as a form of dynamic normalization. In our modeling work we have compared Eqs. (24) and (25) to the efficient masking model and found them to be indistinguishable in their ability to characterize data and to yield interpretable parameters. We generally prefer the model of Eqs. (24) and (25) as we have found it yields better numerical stability when used as an element of a larger system model.

This equation describes a form of balanced, center-surround, excitation–inhibition to a single stimulus, or stimulus component, Ii . The excitatory term is proportional to Ii ; the inhibitory component is proportional to 1  Ii , and the sum of the excitatory and inhibitory components remains constant (at unity) while the strength of Ii varies. (We assume that the stimulus is scaled in contrast units, so both components vary on the range ½0; 1.) The solution of this equation is

397

 Z Ri ðtÞ ¼ Ii 1  exp 

405

t

 lðsÞ ds :

ð12Þ

0

The normalization term in this equation is equal to Ii þ ð1  Ii Þ ¼ 1, so the asymptotic response is a veridical representation of the stimulus – which may, of course, have been subject to nonlinear transformation earlier in processing. Because the sum of the excitatory and inhibitory terms is constant, Eq. (12) decouples the rate of growth from the asymptotic response strength. We consider one further example of a simple, nonlinear, shunting equation, in which the rate of growth depends on the squared activity in the mechanism. Like Eq. (11), this equation represents a form of balanced, center-surround, excitation–inhibition, but the response depends on the stimulus power or energy rather than its amplitude,

h

i

h i dRi ¼ lðtÞI2i 1  R2i ðtÞ  lðtÞ 1  I2i R2i ðtÞ: dt

398 399 400 401 402 403 404

407 408 409 410 411 412 413 414 415 416 417 418 419

420

ð13Þ

422

Although Eq. (13) is nonlinear, it is nevertheless tractable. It is a special case of a nonlinear equation known as a Riccati equation, whose solution is well known (Polyanin & Zaitsev, 2003, p. 103). Indeed, in this simple case, the equation can be solved by elementary methods (see Appendix A) to yield

423

 Z Ri ðtÞ ¼ jIi j tanh jIi j

428

t

lðsÞ ds :

ð14Þ

0

Stimulus strength in Eq. (14) is encoded veridically, as it is in Eq. (12), but the growth rate follows a hyperbolic tangent function rather than an exponential function. Like the exponential function, the hyperbolic tangent function describes negatively accelerating growth to an asymptote, but, unlike the exponential function, it is odd symmetric: tanhðxÞ ¼  tanhðxÞ. We use functions of this kind subsequently to describe systems that use signed values to encode stimulus identity.2 2 Because the asymptotic response in Eq. (14) is proportional to the absolute value of the stimulus intensity, jIi j, the odd symmetry of the hyperbolic tangent is needed to use signed values to encode stimulus identity. In the case of exponential growth, odd symmetry is not needed because identity can be encoded by the sign of Ii directly.

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

424 425 426 427

430 431 432 433 434 435 436 437 438

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

3. Requirements of a dynamic model of attentional selection and decision-making Attention allows the visual system to select relevant stimuli from the environment while excluding others. Typical attention tasks confront the system with the twofold problem of target selection and target identification. To carry out the task, the system must first locate (i.e., select) the target stimulus, then form a representation of it, and then identify it. The role of a computational model of attention is to characterize how these processes acting in concert support behavior. In the remainder of this article, we review a number of experimental findings on the effects of attention in near-threshold tasks and discuss two models that explain how the problem of stimulus selection and the competition it gives rise to are solved computationally by shunting equations. The first of these models, the integrated system model of Smith and Ratcliff (2009), provided an account of attentional effects in single-element displays, in which a single stimulus is presented at an unknown location in an otherwise empty display. The name of the model reflects the fact that it is comprised of submodels of visual temporal processing, attentional selection, visual short term memory (VSTM), and decision-making. The second model is a competitive interaction generalization of the integrated system model, which was proposed by Smith and Sewell (2013) to account for attentional effects in multi-element displays. In both models, the processes of selecting a target stimulus and forming a representation of it in VSTM are modeled by shunting equations. The shunting equation model of VSTM, which is related to an earlier model of VSTM by Loftus and colleagues (Busey & Loftus, 1994; Loftus & Ruthruff, 1994), solves two important theoretical problems that arise when modeling attention. First, it provides us with a model of the time course of stimulus processing and how it depends on attention and other experimental variables. Second, it provides us with a model of how normalization arises both in the process of stimulus selection and in the asymptotic strengths of the representations of selected stimuli. In general terms, attention tasks may be difficult because targets are difficult to locate and select, or difficult to identify, or both. The difficulty of target selection will depend on target salience and on the observer’s uncertainty about the target location. Target salience will vary as a function of stimulus contrast, the presence or absence of localizing information in the display, and the presence or absence of distractor stimuli, and on target-distractor similarity. The difficulty of target identification will depend on the discriminability of the target alternatives, which will be a function of stimulus similarity and contrast. In the experiments from our laboratory we discuss subsequently, the tasks were designed to decouple the effects of stimulus salience and stimulus discriminability. Fig. 1 shows one way in which we

Fig. 1. Experimentally decoupling salience and discriminability. Salience determines how easy a target is to select; discriminability determines how easy it is to identify. (a) The contrast of a Gabor patch presented directly on a uniform field determines both its salience and its discriminability. (b) Gabor patch presented on top of a luminance pedestal. The contrast of the pedestal determines salience and the contrast of the patch determines discriminability.

5

have done this (Smith, 2000; Smith, Ratcliff & Wolfgang, 2004; Smith & Wolfgang, 2004). The task requires observers to discriminate briefly-presented horizontally and vertically oriented Gabor patches (Carrasco, Penpeci-Talgar & Eckstein, 2000). Instead of presenting the stimuli directly against a uniform field, they are presented on top of suprathreshold (e.g., 15% contrast) luminance pedestals. The contrast of the pedestal controls the salience of the stimulus while the contrast of the patch controls its discriminability. In the following sections we describe, in turn, attentional effects in single-item displays and in multi-element displays and show how these effects can be explained by models based on shunting dynamics.

487

4. Attentional effects in single-element displays

498

Smith’s and Ratcliff’s (2009) integrated system model provided an account of the following attentional phenomena in single-element displays:

499

1. The relationship between RT and accuracy. Attention affects both RT and accuracy, and not always in equivalent ways. In some tasks, the effects are comparable: attended stimuli are identified more rapidly and more accurately than are unattended stimuli. In other tasks, the effects are dissociable: attended stimuli are identified more rapidly than unattended stimuli, but there is no effect on accuracy (Smith, 2000). These effects can only be characterized using a model of decision-making that predicts both RT and accuracy and a model of attentional selection that predicts how the information entering the decision process depends on attention. 2. The role of memory. Psychophysical decision task are not usually thought of as memory tasks, but they nevertheless involve a significant memory component. The RT to identify a low-contrast, 50 ms stimulus will typically be longer than 500 ms and may be longer than one second. In other words, RT can be an order of magnitude longer than stimulus duration and a significant proportion of the time appears to be decision time. Studies of the shapes of RT distributions from tasks using brief, backwardly-masked displays suggest there is little or no decay of stimulus information during the time between stimulus offset and the response (Ratcliff & Rouder, 2000; Smith, Ratcliff & Wolfgang, 2004). This suggests that the decision process operates on durable representations of stimuli in VSTM rather than directly on the decaying perceptual trace. The widespread use of two-interval forced-choice (2IFC) tasks in psychophysics presupposes memory representations of this kind; without them, it would not be possible to compare the stimulus information from two different observation intervals. 3. The role of backward masks. In a series of studies, Smith and colleagues investigated the effects of covert attention on performance in the Posner spatial cuing paradigm (Liu, Wolfgang & Smith, 2009; Smith, 2000; Smith et al., 2010; Smith, Lee, Wolfgang, & Ratcliff, 2009; Smith, Ratcliff & Wolfgang, 2004; Smith & Wolfgang, 2004, 2007; Smith, Wolfgang & Sinclair, 2004). In this paradigm, cues are used to attract attention to particular locations in the display prior to stimulus presentation, allowing performance at attended and unattended locations to be compared. In Smith and colleagues’ studies, the target stimuli were low-contrast Gabor patches that were perceptually well-localized, either by presenting them on top of suprathreshold luminance pedestals, as shown in Fig. 1, or by surrounding them with four arms of a so-called ‘‘fiducial cross’’ (Shimozaki, Eckstein & Abbey, 2003). The key finding, replicated consistently across studies, was that accuracy (signal 0 detection d ) was higher for attended than unattended stimuli only when stimulus processing was terminated by a backward, pattern mask. When stimuli were unmasked, there was no

502

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

488 489 490 491 492 493 494 495 496 497

500 501

503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590

6

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

difference in accuracy between attended and unattended stimuli. However, there was an unconditional effect on RT; RTs were shorter to attended than to unattended stimuli, regardless of whether stimuli were masked, although the effect was larger for masked stimuli (Smith et al., 2010; Smith, Ratcliff & Wolfgang, 2004). Fig. 2 shows the results of Smith, Ratcliff and Wolfgang (2004), which illustrate this pattern of findings. The upper panels on the right are detection sensitivity for unmasked and masked targets as a function of stimulus contrast and cue condition. The lower panels on the right are mean RT. There was no difference in sensitivity for cued and uncued targets when stimuli were unmasked, but a systematic effect at all contrast levels when stimuli were masked. Mean RTs were shorter for cued than for uncued stimuli, regardless of whether stimuli were masked. Although most of the experiments in which we have found mask-dependent cuing effects used Gabor patch stimuli, the results appear to be general ones. Smith et al. (2009) found the same mask dependencies in a discrimination task using masked and unmasked radial-frequency (RF) pattern stimuli and Kerzel, Guach and Buetti (2010) found similar results with letter identification. 4. The role of spatial uncertainty and salience. The mask-dependent cuing effects in accuracy in the preceding paragraph are found only when stimuli are well localized perceptually. When low contrast stimuli are presented without accompanying localizing information, accuracy is higher for attended than for unattended stimuli, regardless of whether they are masked (Cameron, Tai & Carrasco, 2002; Carrasco, Penpeci-Talgar & Eckstein, 2000). Gould, Wolfgang and Smith (2007) showed that the difference in the two sets of results was a result of differences in uncertainty about the target location. They found that when low-contrast, unmasked targets were presented without localizing information, accuracy was higher for cued than uncued stimuli, in agreement with the results of Cameron et al. and Carrasco et al. When stimuli were accompanied by simultaneous localizing information, there was no difference in accuracy for attended and unattended stimuli. As in the masking studies, however, there was an unconditional effect of attention on RT: RTs were shorter to cued than to miscued stimuli, regardless of whether localizing information was

presented – although again, the difference was larger when there was no localizing information. The ways in which attention, masks, and stimulus salience jointly affect speed and accuracy is illustrated in Fig. 3, which summarizes the results of four experiments reported by Liu, Wolfgang and Smith (2009). The experiment used the same stimuli and attentional manipulation as used by Smith, Ratcliff and Wolfgang (2004) but, instead of a using a standard RT task, the data were collected using the response-signal method (Dosher, 1979; Reed, 1976), in which observers are required to respond within a fixed time after an auditory deadline signal. By systematically varying the deadline, the growth of accuracy as a function of processing time can be characterized. The response-signal method has been used by Carrasco, McElree, and colleagues to investigate the effects of attention on the rate of growth of accuracy (Carrasco, Giordano & McElree, 2006; Carrasco & McElree, 2001; Carrasco, McElree, Denisova, & Giordano, 2003). In the experiments in Fig. 3, stimulus salience was varied by presenting stimuli either on top of pedestals or directly against a uniform field. Stimuli that were localized with pedestals were either backwardly masked with checkerboards or unmasked. In the fourth experiment, stimuli were simultaneously masked with random pixel noise. Lu, Dosher, and colleagues have described an attention-dependent external noise exclusion mechanism, which acts only when stimuli are presented in external noise (Dosher & Lu, 2000a, 2000b; Lu & Dosher, 1998, 2000; Lu, Jeon & Dosher, 2004; Lu, Lesmes & Dosher, 2002). The fourth experiment in Fig. 3 was designed to study effects of this kind. In agreement with the previous findings, Fig. 3 shows that when stimuli were made salient by localizing them with pedestals, cues increased accuracy only when stimuli were masked; when they were unmasked, there was a small but significant reversal, with sensitivity to cued stimuli being a little less than to uncued stimuli. This reversal, which we have reported in other studies with unmasked stimuli (e.g., Smith, 2000; Smith & Wolfgang, 2004), may reflect lateral masking of the target by the cue. When stimuli were not localized, cues produced a large increase in accuracy without backward masks, in agreement with results of Gould, Wolfgang and Smith (2007) and as reported by

0

Fig. 2. Mask-dependent cuing effects in accuracy (d ) and mean response time (MRT). Observers discriminated vertical and horizontal Gabor patches presented on top of luminance pedestals for 60 ms, with or without backward, checkerboard masks. The attentional cues were four corner markers presented for 60 ms at a cue-target stimulus onset asynchrony of 140 ms. Targets appeared either at the cued location, a , or at one of two possible uncued locations, a  120 , with equal probability. The data are averaged over six observers. Fig. 4 of Smith, Ratcliff and Wolfgang (2004); copyright 2004; permission to reprint requested.

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

Fig. 3. Effects of attentional cues, stimulus salience, and masking on accuracy and mean RT. Observers discriminated horizontal and vertical Gabor patches presented at cued or uncued locations. Stimuli were either presented on top of luminance pedestals or directly against the uniform field. Pedestal stimuli were either unmasked, backwardly-masked with checkerboards, or simultaneously masked with noise. The horizontal axis shows the effects of cues on mean RT (Cued–Uncued 0 RT); the vertical axis shows the effects on accuracy (Cued–Uncued d ). The data are averages over four observers. The error bars are 95% Bayesian credible intervals. Fig. 7 from Liu, Wolfgang and Smith (2009); copyright 2009; permission to reprint requested.

632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662

Cameron, Tai and Carrasco (2002) and Carrasco, Penpeci-Talgar and Eckstein (2000). There was also a significant cuing effect in accuracy with simultaneous noise masks, in agreement with the external noise results of Lu, Dosher, and colleagues. There appears to be an RT advantage for cued stimuli under all conditions, regardless of accuracy, although the 95% confidence intervals in two of the experiments just cover zero. The overall pattern is similar to the one we have found with standard RT tasks, which is that RTs are shorter to cued stimuli, regardless of whether there is an accompanying accuracy difference, but the RT differences are larger when there is an accuracy difference. 5. The role of onset transients. The increase in accuracy for lowcontrast attended stimuli reported by Cameron, Tai and Carrasco (2002) and Carrasco, Penpeci-Talgar and Eckstein (2000) and replicated by Gould, Wolfgang and Smith (2007) has often be attributed to an attention-dependent mechanism of signal (or stimulus) enhancement (Shiu & Pashler, 1994). Typically, signal enhancement is conceived of as a local increase in contrast sensitivity at cued locations that results from a spatial distribution of attention established prior to stimulus onset by the cue. However, the interaction of the effects of cues with those of masks and uncertainty suggests to us a more dynamic, target selection process, which depends on stimulus salience. Our hypothesis is that localizing information, whether provided by a pedestal or fiducial cross, is a manipulation of stimulus salience, and that stimulus salience affects performance by affecting VSTM selection efficiency. In contrast, signal enhancement predicts that attentional effects will vary in magnitude with stimulus discriminability, but should be unaffected by salience.

663 664 665 666 667 668

To test this, Sewell and Smith (2012) manipulated stimulus salience using a version of Yantis and Jonides’s (1984) no-onset paradigm, adapted for use with Gabor patch stimuli. In their task, observers discriminated the orientations of vertical and horizontal Gabor patches that were presented at cued or uncued locations. In

7

the no-onset condition, four plaid stimuli, consisting of superimposed vertical and horizontal gratings, were presented at each of the possible target locations. The target stimulus was created by removing three of the plaids and one component of the fourth, leaving a single oriented grating. In the abrupt-onset condition, a single, oriented grating was presented at either a cued or an uncued location. Their paradigm and their results are summarized in Fig. 4. Sewell and Smith (2012) found that the effects of attention on both RT and accuracy were larger in the no-onset condition than in the abrupt-onset condition and that they depended on the timing of localizing information. The magnitude of the cuing effect was increased when localizing information (in the form of a fiducial cross) was delayed until 60 ms after stimulus offset, as compared to when the stimulus and the localizing information were simultaneous. The largest cuing effects were found with noonset stimuli and delayed localization. Sewell and Smith argued that these findings are inconsistent with static resource theories (including simple versions of signal enhancement), which predict identical performance with abrupt-onset and no-onset stimuli and do not easily account for the differences between simultaneous and delayed localization. They are, however, consistent with the idea that abrupt onsets and localizing information both affect performance by altering VSTM selection efficiency: abrupt onsets via their attentional capture properties, as Yantis and Jonides (1984) originally proposed, and localizing information via its effects on salience.

669

5. The integrated system model

696

The attentional phenomena described in the previous section are well described by the integrated system model of Smith and Ratcliff (2009). Fig. 5 shows a representation of this model. The model comprises submodels of visual temporal sensitivity, attentional selection, VSTM, and decision-making. In the model, a brief stimulus of intensity I and duration d is encoded by spatiotemporal filters. The encoded intensity is assumed to be perturbed by Gaussian noise, like the noise in signal detection theory. The time course of the stimulus is described by the sensory response function, lðtÞ. When stimuli are unmasked, the decay of the sensory response is comparatively slow and the area under lðtÞ is large; when stimuli are backwardly masked, the sensory response is rapidly suppressed and the area under lðtÞ is small. This feature of the model allows it to account for the mask-dependent cuing effects described earlier. The transient information in the sensory response is gated into VSTM under the control of spatial attention. Attention is represented as a spatiotemporal weighting function, as proposed in the episodic theory of attention of Sperling and Weichselgartner (1995). This representation allows for the possibility that the efficiency with which stimulus information is transferred to VSTM will vary as a function of a spatial distribution of resources established by a cue, and also, that this spatial distribution may shift during the course of a trial as the result of salient events in the visual field. Sewell and Smith (2012) found that their no-onset data were better described by a model in which localizing information caused a shift in attentional resources than one in which the spatial distribution of resources remained constant during the trial. To account for the effects of attention on both RT and accuracy, the integrated system model assumes that decisions are made by a diffusion process (Ratcliff, 1978). Diffusion models assume that successive samples of noise-perturbed stimulus information are accumulated by an evidence process until a criterion quantity of evidence for one or other response has been obtained. The first criterion reached determines the response and the time taken to

697

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695

698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1

8

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

0

Fig. 4. Effects of onset transients and localizing information on accuracy (d ) and mean response time (MRT). Observers discriminated orthogonally-oriented Gabor patches presented for 60 ms at cued or uncued locations. In the no-onset condition (OFF) orthogonal-plaid placeholders were presented at the four possible target locations. Targets were formed by removing three of the plaids and one component of the fourth 150 ms after cue onset. In the abrupt-onset condition (ON), a single Gabor patch was presented 150 ms after cue onset at either the cued location or at an uncued location. The stimuli were localized by fiducial crosses (FID), presented either at the same time as the target (SIM) or delayed until 60 ms after target offset (DEL). Data are averaged over observers and contrast levels. Data from Sewell and Smith (2012).

Fig. 5. Integrated system model of speeded decision making in single-element displays. A brief stimulus of intensity I and duration d is encoded by parallel where (energy) and what (form) pathways. The time course of the stimulus is described by a sensory response function, lðtÞ. Where pathway activity and attention, ci ðtÞ, jointly gate form information carried by the what pathway into visual short-term memory (VSTM). The time course of VSTM trace formation is described by a shunting equation. Successive samples of the noise-perturbed VSTM trace are accumulated to a response criterion by a diffusion process decision mechanism. The stimulus intensity, I, is also subject to trial-to-trial perturbations by noise, like the noise in signal detection theory.

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

reach it determines the decision time. Diffusion models have successfully accounted for accuracy and distributions of RT for correct responses and errors in a variety of perceptual and cognitive tasks (Ratcliff & Smith, 2004). In the integrated system model, the information that drives the decision process is a noise-perturbed representation of the stimulus in VSTM. In addition to between-trial noise added to the stimulus intensity, I, the VSTM trace, denoted mðtÞ, is subject to moment-to-moment perturbation by Gaussian white noise. The decision process accumulates successive samples of the noise-perturbed representation of the stimulus in VSTM until a criterion is reached. The VSTM representation in the integrated system model is not constant, but grows to an asymptote at a rate that depends on attention and stimulus salience, which makes the decision process time-inhomogeneous. Methods for obtaining predictions for time-inhomogeneous diffusion processes were described by Smith (2000) and Smith, Ratcliff and Sewell (2014). To account for differences in the shapes of the RT distributions when stimulus salience and discriminability were varied, Smith and Ratcliff (2009) proposed that stimulus information is encoded in parallel by two pathways, which they called energy and form pathways. Smith and Sewell (2013) subsequently called them where and what pathways, noting that the properties ascribed to them in the model were similar to those attributed to the cortical where and what pathways carried by the dorsal and ventral processing streams. The where pathway responds to the overall energy in the target stimulus, denoted I2 . Like the complex cells in Heeger’s (1991, 1992) model, the where pathway is form-blind; it responds to the presence of target features in the display but it does not encode stimulus identity. Rather, its role is to encode the presence and locations of task-related stimuli and to gate the stimulus information at those locations into VSTM. The what pathway carries information about stimulus identity. Following Heeger, we assume that the encoding of stimulus identity can be represented computationally by a pair of half-squaring operators, in which positive values represent evidence for one stimulus alternative and negative values represent evidence for the other. We denote encoding by a pair of half-squaring operators in Fig. 5 using the notation /ðIÞI2 , where /ðIÞ denotes a scaled, shifted signum function,

772 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796

/ðIÞ ¼

1

if I P 0

9

pathway will depend on the energy in the patch-plus-pedestal compound. The growth of activity in the where and what pathways is described by a coupled pair of shunting equations, similar to Eq. (13) (see Appendix A). The VSTM trace, mðtÞ, is identified with activity in the what pathway. The growth of the VSTM trace is simultaneously gated by energy in the where pathway and by an attention weighting function, whose value can vary both with time and position in the visual field. In many applications, the comparisons of interest will be between an attended (A) and an unattended (U) location under conditions in which the weights do not vary appreciably with time. In these situations, we denote the attention weights by cA and cU , with cA > cU .

797

6. Predictions of the integrated system model

810

6.1. The relationship between RT and accuracy

811

Because the model includes a successful sequential-sampling model of decision making, it simultaneously predicts accuracy, and distributions of RT for correct responses and errors. The distributions of RT predicted by the model agree closely with those that are found experimentally in near-threshold attentional tasks across the entire psychometric function (Gould, Wolfgang & Smith, 2007; Smith et al., 2010; Smith & Ratcliff, 2009; Smith, Ratcliff & Wolfgang, 2004).

812

6.2. The role of memory

820

The shunting equations used to describe the growth of the VSTM trace provide a natural way to model the formation of a durable stimulus representation from a transient stimulus event. As described previously, the model assumes that the rate of growth of the VSTM trace depends multiplicatively on a sensory response function, lðtÞ, that describes the time course of stimulus processing. When the sensory response goes to zero, the VSTM trace stops changing. The asymptotic strength of the VSTM trace determines the quality of the discriminative information entering the decision process. The shunting equations used to model the growth of the VSTM trace are closely related normalization models, as discussed previously.

821

6.3. The role of backward masks

833

The model assumes that backward masks suppress the sensory representations of stimuli, represented mathematically by a reduction in the area under the sensory response function (see the Appendix A for details). When stimuli are unmasked, the area under lðtÞ will be large and the VSTM trace will have time to reach asymptote. When stimuli are masked, the area under lðtÞ is reduced and VSTM trace growth is terminated before asymptote is reached. The rate at which the VSTM trace forms depends on the attention weighting, which is greater for attended than for unattended stimuli (cA > cU ). The VSTM trace grows more rapidly for attended than for unattended stimuli, but, providing the stimulus duration is long enough for the process to run to completion, the asymptotic trace strength for attended and unattended stimuli will be the same. When stimuli are unmasked and the VSTM trace has time to reach asymptote, the model predicts that RTs will be shorter for cued than for uncued stimuli, but accuracy will be the same. When stimuli are masked and VSTM trace growth is terminated before the asymptote is reached, the final trace strength will be greater for cued than for uncued stimuli because of the faster rate of growth associated with the former. Under these circumstances,

834

798 799 800 801 802 803 804 805 806 807 808 809

813 814 815 816 817 818 819

822 823 824 825 826 827 828 829 830 831 832

1 if I < 0:

The function /ðIÞI2 is a symmetric function: Its sign carries information about the identity of the stimulus and its magnitude is proportional to stimulus strength. In Smith and Ratcliff’s (2009) presentation of the model, they assumed the dynamics of the pathways were controlled by stimulus amplitude rather than stimulus power (i.e., by I rather than I2 ). We have reformulated the model to better align with Heeger’s ideas and Smith and Sewell’s (2013) generalization of it. The reader is referred to the Appendix A for details of how the model presented here differs from the one presented by Smith and Ratcliff. The where pathway is responsible for stimulus selection: Activity in the pathway multiplicatively gates the form information carried by the what pathway into VSTM. The rate at which the VSTM trace grows depends jointly on the discriminative information in the stimulus and stimulus salience. Salience determines the rate of VSTM formation via its effects on the activation of the where pathway. In Fig. 5, the inputs to the where and what pathways are both represented by the stimulus energy, I2 , but typically the inputs to the pathways will be different. For example, when a low contrast Gabor patch is presented on top of a suprathreshold contrast pedestal, the activity in the what pathway will depend on the contrast energy in the patch and the activity in the where

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1

10

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

859

the model predicts that RTs will be shorter for cued stimuli and accuracy will be higher. The model predicts interactions between attentional cues and masks at the level of the psychometric functions for accuracy and the RT distributions for correct responses and errors.

860

6.4. The role of spatial uncertainty and salience

861

879

The gated, two-pathway structure of the integrated system model accounts for the effects of stimulus salience on accuracy and RT. Salience, which is affected by the localizing information in pedestals and fiducial crosses, determines the activity in the where pathway, which controls the rate of VSTM growth. Highly salient stimuli are selected more efficiently and produce higher rates of growth. When stimuli are salient and unmasked, VSTM growth will be rapid and the VSTM trace will have time to reach asymptote, resulting in an RT benefit for cued stimuli but no accuracy benefit. Salience will be low when stimulus contrast is low and stimuli are not localized. Under these circumstances, overall VSTM trace growth will be slow (or its onset will be delayed), especially for uncued stimuli. This results in large RT and accuracy benefits for cued stimuli, as shown in Fig. 3. The model successfully accounted for the interaction between attentional cues and the presence or absence of localizing information reported by Gould, Wolfgang and Smith (2007) at the level of the psychometric functions for accuracy and the RT distributions for correct responses and errors (Smith & Ratcliff, 2009).

880

6.5. The role of onset transients

881

Sewell and Smith (2012) showed that cues increased accuracy more for no-onset than for abrupt onset stimuli and that the difference was greater when localizing information was delayed than when it was simultaneous. These effects are accounted for by the two-pathway structure of the integrated system model. The masking of stimulus onsets in Sewell and Smith’s no-onset condition and the timing of localizing information can both be viewed as manipulations of stimulus salience, which determines how efficiently stimuli are selected by the where pathway. A reduction in salience decreases VSTM selection efficiency and the quality of the final VSTM trace. The effects of low salience will be larger at uncued locations than cued locations, which increases the magnitude of the cuing effects in accuracy and RT. Sewell and Smith (2012) compared the fits of two versions of the integrated system model to the psychometric functions and RT distributions for correct responses and errors. They found that performance in the no-onset condition was better described by a model in which there was a shift in attentional resources during the course of the trial initiated by the fiducial marker than one in which the distribution of attentional resources remained constant for the duration of the trial.

855 856 857 858

862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878

882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901

902 903 904 905 906 907 908 909 910 911 912 913

7. External noise normalizes stimulus representations at the target location The experimental findings on the role of attention on the speed and accuracy of performance in single-items displays are welldescribed by a model that assumes that the stimulus representations supporting perceptual decisions are computed by shunting equations. The normalization properties of such equations are not required to explain these findings. Rather, the critical properties involve the modulation of time-varying stimulus representations by attention, masks, and spatial uncertainty, which can be described by interacting where and what pathways. The latter uses a balanced center-surround equation (Eq. (29)), in which the

normalization term is unity. When there is external noise in the display, however, there appears to be a theoretical need for normalization. As discussed previously, Lu, Dosher, and colleagues have identified an external noise exclusion mechanism, which produces large cuing effects in accuracy when target stimuli are presented in external noise (Dosher & Lu, 2000a, 2000b; Lu, Lesmes, and Dosher, 1998). They characterized these effects using their perceptual tem- Q5 plate model, an extended signal detection model that includes various sources of noise, both internal to, and external to the observer. Based on the findings of Ratcliff and Smith (2010), who studied letter discrimination in external noise and found that noise shifted RT distributions to the right along the time axis, Smith and Ratcliff (2009) proposed that external noise slows the rate at which stimulus representations are formed by the what pathway. Consistent with this, Smith, Ratcliff and Sewell (2014) found that the RT distributions and accuracy from the letters-in-noise discrimination task were well described by a time-varying diffusion model in which noise slows the rate at which stimulus information enters the decision process. To test the hypothesis that attention and noise both affect the rate at which VSTM representations are formed, Smith et al. (2010) investigated the effects of adding external noise to the display in a spatial cuing paradigm, similar to the one shown in Fig. 1. Observers discriminated orthogonally oriented Gabor patches, localized by fiducial crosses. The stimuli were presented for 60 ms and were either unmasked, backwardly masked with checkerboards, simultaneously masked with random pixel noise, or masked with both kinds of mask. Fig. 6 summarizes the paradigm and the main findings. There were reliable cuing effects in accuracy with both simultaneous noise masks and backward pattern masks, but the effect was larger with backward masks, in agreement with the results of Liu, Wolfgang and Smith (2009) in Fig. 3. The slight cuing effect in accuracy with unmasked stimuli is atypical for this paradigm and was due largely to one observer. There was a very large cuing effect in accuracy when stimuli were masked with both kinds of mask. This latter effect cannot be attributed to an overall increase in the amount of masking produced by mask pairs, because it was obtained with stimuli whose contrasts were adjusted to equate accuracy. The accuracy data in Fig. 6 support the idea that backward masks and simultaneous noise masks interact with attention in different ways, consistent with the idea that these kinds of masks affect stimulus processing differently (Breitmeyer, 1984). However, the RT data do not support the hypothesis that noise interacts with attention by slowing processing rates: The cuing effects in RT were similar for noise-masked and backwardly-masked stimuli, contrary to the predictions of a rate-based model. Smith et al. (2010) confirmed this by fitting the integrated system model to the psychometric functions and RT distributions for correct responses and errors and found that a pure rate-based model failed to fit the data. They found instead that the data were welldescribed by a model in which there was a noise-dependent inhibition term in the shunting equation for the what pathway, which performed attention-dependent normalization of VSTM trace strength. Either a contrast-gain model or a response-gain model could be obtained, depending on how the what-pathway equation was formulated; these two models provided comparably good accounts of the data. The general finding that the effects of noise can be represented by normalization of the stimulus representation at the target location is consistent with the results of Dao, Lu and Dosher (2006), who showed that Lu and Dosher’s perceptual template model could be reframed as a gain control (i.e., normalization) model.

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1

11

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

0

Fig. 6. Effects of attentional cues on sensitivity (d ) and mean response time (MRT) for simultaneous noise and backward pattern masks. The stimuli were 60 ms, horizontal and vertical Gabor patches, localized by fiducial crosses (FID), presented at cued or uncued locations. Stimuli were either unmasked, or simultaneously masked with random pixel noise, or backwardly masked with checkerboards, or masked with both kinds of masks. The data are averaged over five contrast levels and four observers. The figure is a composite of Figs. 1, 3 and 5 of Smith et al. (2010).

979

8. Attentional effects in multi-element displays

980

Smith and Sewell’s (2013) competitive interaction theory generalized the integrated system model to account for performance in briefly-exposed multi-element displays. Their theory proposed that a number of experimental findings on the effects of attention in such displays could be explained by shunting dynamics and their associated normalization properties. Among the phenomena their theory sought to explain were the following:

981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010

1011 1013 1014 1015 1016

1. Near-threshold search performance is well described by a MAX model. Many investigators have reported that accuracy in near-threshold visual search and spatial cuing tasks is well described by a signal detection MAX model (Baldassi & Burr, 2004; Baldassi & Verghese, 2002; Carrasco, Penpeci-Talgar & Eckstein, 2000; Eckstein, 1998; Eckstein, Thomas, Palmer, & Shimozaki, 2000; Foley & Schwarz, 1998; Morgan, Ward & Castet, 1998; Mulligan & Shaw, 1980; Nachmias, 2002; Palmer, 1994; Palmer, Ames & Lindsey, 1993; Palmer, Verghese & Pavel, 2000; Shaw, 1982; Smith, 1998; Verghese, 2001). The MAX model assumes that the sensory effect associated with each stimulus in a display can be described by a Gaussian strength variable, X i ; i ¼ 1; 2; . . . ; m. The set of strength variables describes trial-to-trial variations in the quality of the encoded representations of the stimuli. The simplest version of the model assumes that targets (signals) have mean l and nontargets have mean zero and a common standard deviation. In target-present/target-absent (yes–no) visual search tasks, the observer simultaneously compares the strength variables at each display location to a decision criterion, c, and responds ‘‘target present’’ if the value of any of the strength variables exceeds the criterion; otherwise, (s)he responds ‘‘target absent.’’ Formally, the probability of a target present, or ‘‘signal’’ response, PðSÞ, is



PðSÞ ¼ P X i P c; X i ¼ maxðX j Þ : j

ð15Þ

Performance in two-alternative forced-choice (2AFC) search tasks, in which the observer is required to identify which of two possible targets was presented, can be described by a

signed-MAX model, which generalizes the simple MAX model in a straightforward way (Baldassi & Burr, 2004). The signedMAX model assumes that the strength of the evidence is carried by the magnitudes of the random variables, X i , and information about stimulus identity is carried by their signs. The model assumes that the alternative targets have means l and l and that nontargets have mean zero. The observer’s decision is based on the absolute maximum of the set of random variables, maxj ðjX j jÞ. If the sign of the random variable with the greatest absolute value is greater than zero, the observer makes one response, say R1 ; if the sign is less than zero, (s)he makes the other response, R2 . Formally, the probability of an R1 response is

"

#

PðR1 Þ ¼ P X i P 0; X i ¼ argmaxðjX j jÞ :

1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028

1029

ð16Þ

j

1031

In this equation, the argmax function picks out the argument at which the absolute value function attains its maximum value, namely, the largest of the set of strength variables. The theoretical significance of the MAX model is that it shows that performance in near-threshold search tasks is often well described by a model in which there are no attentional capacity limitations of any kind – whether conceived of either as a processing bottleneck (Deutsch & Deutsch, 1963; Norman, 1968; Treisman, 1960), or as a limitation in the capacity to activate neural structure (Kahneman, 1973). The only source of performance limitations in the MAX model is noise in the stimulus representations (i.e., the random variables X i ). The model predicts that performance will be worse with larger displays because of an increase in the noise entering the decision process. Strikingly, the model gives a good account of performance in conjunction search tasks, in which the decision is based on combinations of simple stimulus features (Eckstein et al., 2000). Conjunction search tasks have historically been regarded as paradigm examples of serial, or near-serial, search tasks (Treisman & Gelade, 1980), but the success of the MAX model shows that performance in these tasks can often be explained by the effects of decision noise alone. 2. The double-target detection deficit. Moray (1969), in a dichotic listening, signal-detection, task, compared performance under conditions in which target stimuli (defined as small increases in the amplitude of pure tones) were presented either to the left

1032

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122

12

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

or the right ear, but never simultaneously, to conditions in which targets could occur on the left or the right or on both sides simultaneously. He found a large double-target detection deficit: Detection accuracy was significantly reduced when two targets, both of which had to be identified, were presented simultaneously, as compared to when targets appeared only on one side or the other. Sorkin and colleagues studied this phenomenon in detail using signal detection theory, which allowed them to distinguish sensitivity from bias effects (Gilliom & Sorkin, 1974; Sorkin & Pohlmann, 1973; Sorkin, Pohlmann & Gilliom, 1973; Sorkin, Pohlmann & Woods, 1976). They confirmed Moray’s original observations and showed that there 0 was a reduction in sensitivity (d ) for targets that were accompanied by contralateral signal responses, that is, when the observer made a ‘‘signal’’ response to the stimulus presented to the other ear, regardless of whether it was a signal or not. Duncan (1980) subsequently reported a double-target deficit in vision in a task requiring observers to detect either one or two digit targets in four-element arrays of letters, which were either presented simultaneously or sequentially in two pairs. These findings implicate the process of selecting the contralateral stimulus for further processing as the basis of the deficit in processing a simultaneous target. 3. Redundancy costs in the post-stimulus probe task. In the poststimulus probe task, the observer reports the properties of a single display location that is probed after display offset. The post-stimulus probe method is similar to Sperling’s original (1960) partial-report method, except that only a single probed item is reported on any trial. It was first used in attention research by Downing (1988) to investigate whether spatial cues produce local improvements in detection sensitivity once the effects of noise at nontarget locations are controlled for. The assumption is that noise at other locations has a negligible effect on performance when the contents of only one location must be reported. Following Downing, the method was used in covert attention tasks by a variety of other investigators (Hawkins et al., 1990; Luck et al., 1994; Müller & Humphreys, 1991; Smith, 1998). These studies all found that covert attention increased sensitivity – although, critically, all of them used backwardly-masked displays (see the preceding section). An important feature of the post-stimulus probe paradigm, first reported by Müller and Humphreys (1991) and subsequently by Smith (1998, 2010), is that it leads to redundancy costs: Sensitiv0 ity (d ) to the contents of a probed location is reduced when there are target stimuli elsewhere in the display and decreases progressively as the number of redundant targets increases. This is the opposite to what to be expected in a task in which observers are asked to indicate whether or not the display contains a target, irrespective of its location. The redundancy costs in the post-stimulus probe paradigm likely reflect the same limited-capacity target selection mechanism that underlies the double-target detection deficit: The presence of stimuli that could be targets, but which are not, impairs the report of the contents of the probed location. 4. The information capacity of visual short-term memory. The use of a post-stimulus probe effectively turns the task from a search task into a memory task. As noted earlier, there is probably a VSTM component in all psychophysical decision tasks, but the post-stimulus probe methodology makes the memory component of the task overt, because the relevant stimulus is not known until after display offset. Palmer (1990) reported that performance in such tasks is well described by a sample-size Q6 model (Bonnel & Hafter, 1998; Miller and Bonnel, 1994; Lindsay, Taylor & Forbes, 1968; Shaw, 1980; Taylor, Lindsay & Forbes, 1967). The sample-size model assumes that the observer has available a fixed number, n, of noisy stimulus samples

to represent the stimuli in the display, where n depends on exposure duration. When there is only a single item in the display, all n samples can be used to represent this one item; when there are m items in the display and there is no preferential weighting of items by attention, the samples are distributed equally among items, so that each item is represented by n=m samples. The signature of the sample-size model is the invariance of P  0 2 across displays of different sizes, up to a presumed i di VSTM item-capacity limit of around four items (Cowan, 2001; 0 Luck & Vogel, 1997). Symbolically, if d1 is the sensitivity when there is a single item in the display, the model predicts that

1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134

1135 0 2 ðdm Þ

¼

0 2 ðd1 Þ

m

;

m ¼ 2; 3; 4;

ð17Þ

or equivalently,

1137 1138

1139 m X 0 2 ðdi Þ ¼ c

ðconstÞ:

ð18Þ

i¼1

This prediction follows straightforwardly from elementary sampling theory: The mean of the item representations in VSTM is proportional to n=m; the standard deviation is proportional to pffiffiffiffiffiffiffiffiffiffi 0 n=m, and d , which is the ratio of the mean pffiffiffiffiffiffiffiffiffiffiand the standard deviation, is likewise proportional to n=m. Squaring gives Eqs. (17) and (18). Equivalently, the model predicts that thresholds increase linearly with display size, with slope 1/2, when data are plotted in logarithmic coordinates. This was the form in which the relationship was reported by Palmer (1990). Assuming a fixed sampling rate, the model also predicts linearity 0 2 of ðd Þ as a function of stimulus exposure time up to the sampling capacity of VSTM.

1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154

The standard interpretation of the sample-size model is that it represents an encoding capacity limitation; that is, it reflects a limitation on the observer’s ability to form simultaneous representations of multiple stimuli. This interpretation is consistent with the model’s derivation from sampling theory and is the way it was interpreted by Bonnel and Hafter (1998) and Bonnel and Miller (1994), who found it gave a good account of performance in concurrent tasks, in which observers are required to divide their attention between simultaneous visual and auditory tasks. In contrast, Palmer (1990) interpreted the sample-size properties as a reflection of the information capacity of VSTM itself. Sewell, Lilburn and Smith (2014) reported a post-stimulus probe study in which observers discriminated between horizontally and vertically oriented Gabor patches that were presented in noise for 50, 100, 150, or 200 ms, and then backwardly masked by high-contrast checkerboards. In order to avoid any item capacity limitations of VSTM, they limited their task to displays of one, two, three, or four items. They found that the sample-size model provided an essentially parameter-free description of the display size effect at all exposure durations. To test whether their sample-size effects were due to an encoding capacity limitation or a VSTM capacity limitation, Sewell et al. compared performance with simultaneous and sequential stimulus presentation. In the sequential condition, there was only ever one stimulus in the display at a time, so if the sample-size effects found with simultaneous presentation were due to encoding capacity limitations, they should have been eliminated with sequential presentation. Performance with simultaneously and sequentially presented stimuli was essentially indistinguishable, and was well described by the sample-size model in either instance, which implicates VSTM as the locus of the effect. Consequently, a sampling interpretation of P  0 2 i di -invariance would need to assume that the set of stimulus samples is a reflection of the representational capacity properties

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210

of VSTM itself; and, moreover, it would need to assume that the available VSTM capacity is dynamically redistributed among items as new items are added to the memory system. This can be accomplished by an appropriate normalization model, as we show subsequently. As noted previously, in addition to the information capacity limitations of VSTM, there is a substantial literature suggesting that VSTM has an item capacity limitation, often estimated to be around four items (Anderson, Vogel & Awh, 2011; Cowan, 2001; Q7 Luck & Vogel, 1997; Zhang & Luck, 2008) – although some authors have reported that the item capacity of VSTM depends on item complexity (Alvarez & Cavanagh, 2004) and others have argued that VSTM capacity in some tasks is better described by models in which there are no item limits whatsoever (van den Berg, Awh & Ma, 2014). If VSTM has an item-capacity limit, then, when the display size exceeds the available item capacity or number of ‘‘slots’’ in VSTM, observers must guess once the available slots are full. Consistent with this, ROC analysis suggests that observers do indeed appear to guess on a proportion of trials when larger displays are used (Donkin, Nosofsky, Gold, & Shiffrin, 2013; Rouder Q8 et al., 2008). Guessing leads to a high-threshold model of decision-making, which predicts distinctive, linear ROC functions (Green & Swets, 1966, pp. 127–130).

1211

9. Competitive interaction theory of attention

1212

Smith and Sewell’s (2013) multi-channel generalization of the integrated system model assumes that stimuli compete to enter a limited-capacity VSTM stage. The nature of the competition varies, depending on whether the observer is seeking to determining whether the display contains a designated target (as occurs in visual search) or seeking to encode as much information about the display as possible (as occurs in the post-stimulus probe task). The theory assumes that the competitive interactions are biased by attention, which gives attended stimuli an advantage over unattended stimuli. Desimone and Duncan (1995) argued that attention operates by a process of biased competition; Smith and Sewell’s competitive interaction model embodies the principles of biased competition in a computationally explicit way. In our presentation here, we reformulate the model to better emphasize the different properties of the where and what pathways and to align the model with Heeger’s (1991, 1992) proposal that cortical simple cells can be modeled as half-squaring operators. The Appendix A lists the specific ways in which the model presented here differs from the one described by Smith and Sewell. Mathematically, VSTM trace formation in Smith and Sewell’s (2013) competitive interaction theory is described by three systems of shunting equations, which characterize, respectively, activity in the where pathway, the what pathway, and the VSTM trace itself. In Smith and Ratcliff’s (2009) model, the VSTM trace was identified with activity in the what (form) pathway; but Smith and Sewell (2013) found they needed to distinguish the computations carried out by the what pathway from those carried out in VSTM in order to account for the results of Sewell, Lilburn and Smith (2014), who showed that performance with simultaneously and sequentially presented stimuli was identical. Nodes in the where pathway (which represent stimulus locations) are denoted by ti ðtÞ; i ¼ 1; 2; . . . ; m (Greek upsilon); nodes in the what pathway are denoted by mi ðtÞ (Greek nu), and nodes in VSTM are denoted xi ðtÞ (Greek omega). The properties of the three systems of equations are depicted in Fig. 7. The figure depicts a display with two stimuli, but the model extends to displays of arbitrary size. The activity in the nodes are continuous functions of time, but the time dependency is omitted from the notation in the figure for simplicity.

1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250

13

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

Fig. 7. Competitive interaction theory of attentional selection in multi-element displays. The lower panel shows the where pathway; the upper panel shows the what pathway and VSTM. The where pathway activity gates stimulus identity information carried by the what pathway into VSTM. Nodes in the where pathway are activated by external stimuli, I2i , and self-excite by nonlinear feedback, f ðti Þ, and mutually inhibit one another. The form of the feedback function determines whether the system performs winner-take-all selection or selects multiple stimuli into VSTM. The what pathway forms veridical representations of the stimuli selected by the where pathway. VSTM has both an item capacity and an information capacity. The information capacity limitations are expressed via trace strength normalization and realize the predictions of the sample-size model.

9.1. Where pathway

1251

Activity in the where pathway, ti ðtÞ, is described by a system of shunting differential equations:

1252

dti dt

  i  ¼ kti þ ci lðtÞI2i þ f t2i 1  t2i ðtÞ  t2i ðtÞ

m h X

 i

cj lðtÞI2j þ f t2j ; i ¼ 1; . . . ; m:

ð19Þ

j–i

This equation states that activity in node i in the where pathway grows as a rate that depends on the contrast energy in the stimulus, I2i . The time course of the stimulus is described by the sensory response function lðtÞ. Once active, the node also self-excites via the nonlinear feedback function, f ðti Þ. The feedback function determines the type of selection carried out by the pathway, as we discuss subsequently. In the absence of inhibition, activity in node i will grow until saturation, which is determined by the term

 1  t2i ðtÞ . The choice of unity as a saturation point is arbitrary and does not affect the generality of the model.

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

1253

1254

h

1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1

14

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

1278

As well as being excited by the external stimulus, node i receives feedforward inhibition from all of the other stimuli in the display, j ¼ 1; 2; . . . ; m; j – i. The feedforward inhibition is gated by shunting interactions via the term t2i ðtÞ. The competition among nodes, as represented by the balance of excitatory and inhibitory influences, is biased by the attention weights, ci , which represent the spatial distribution of attentional resources across the visual field. Stronger stimuli (larger values of I2i ) and attended stimuli (larger values of ci ) are advantaged in the competition. The activity in the node also decays with rate k. Decay is needed to ensure the selection process carried out by the feedback function operates correctly.

1279

9.2. What pathway

1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277

1280 1281 1282 1283 1284 1285 1286

1287

1289

The where pathway responds to features in the display that signal the presence of targets, but does not carry information about stimulus identity. Stimulus identity is coded by the what pathway. For simple, two-valued stimulus alternatives, stimulus identity information is encoded by the sign function, /ðIÞ. The activity in the what pathway, mi ðtÞ, is described by a second system of m independent, uncoupled shunting equations:

n o

 dmi ¼ ati ðtÞ/ðIÞ lðtÞI2i 1  m2i ðtÞ  lðtÞð1  I2i Þm2i ðtÞ i ¼ 1; .. .; m: dt ð20Þ

1310

In these equations, I2i is the power of the stimulus at location i, as encoded by the what pathway, and lðtÞ describes its time course, as expressed by the sensory response function. Stimulus encoding by the what pathway is gated multiplicatively by the strength of activation in the where pathway, ti ðtÞ. The greater the activity in the where pathway, the more rapidly the associated node in the what pathway will become active. The what pathway forms representations of only those stimuli that are selected by the where pathway. Mathematically, the what pathway exhibits the kind of balanced center-surround, excitation–inhibition that was discussed previously in relation to Eq. (13). Because the normalization constant in such equations is unity, the what pathway forms veridical stimulus representations. The sign term, /ðIi Þ, in Eq. (20) determines the direction of growth of mi ðtÞ. As noted in relation to Eq. (13), growth in such Riccati-type equations is described by the odd-symmetric hyperbolic tangent function. Consequently, the asymptotic solution of Eq. (20) represents both the strength of stimulus i and its identity: mi ð1Þ ¼ Ii or mi ð1Þ ¼ Ii , depending on the value of /ðIi Þ. The rate at which stimuli are encoded by the what pathway depends on a rate constant a, which is not assumed to depend on attention.

1311

9.3. VSTM trace

1312

Each of the nodes in VSTM, xi ðtÞ, is driven by its corresponding what pathway activity, mi ðtÞ. VSTM activity is described by a set of additive-multiplicative shunting equations:

1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309

1313 1314

1315

      dxi ¼ /ðmi Þ ci m2i ðtÞ h2  x2i ðtÞ  1  m2i ðtÞ x2i ðtÞ dt ) " !# " # m m     X X cj # x2j x2i ðtÞ  1  # # x2j  K ;  j–i

1317 1318 1319 1320 1321 1322

i ¼ 1; . . . ; m:

j–i

ð21Þ

In these equations, the additive inhibition terms (the terms in braces) express the information capacity of VSTM; the multiplicative inhibition terms express its item capacity. The constants, ci are attention gain terms that determine the attention allocated to the stimulus at location i; h determines the asymptotic trace

strength, and K determines the item capacity of VSTM. The function #ðx2 Þ, is a so-called ‘‘soft threshold,’’ that is, a smooth approximation to a step function (see Appendix A). Because of the threshold-like properties of #ðÞ, when the attention weights are all the same the

additive part ofEq. (21) is approximately equal to dxi =dt ¼ c h2 m2i ðtÞ  mx2i ðtÞ . The asymptotic pffiffiffiffiffi pffiffiffiffiffi solution of this equation is mx2i ¼ h2 m2i or xi ¼ hmi = m ¼ hIi = m. In other words, the strength of the VSTM trace is reduced in proportion to the square root of the number of stimuli selected into VSTM. The normalization of VSTM trace strength by the number of items in VSTM is the basis of the model’s ability to reproduce  0 2 the invariance of Ri di predicted by the sample-size model. The evidence of an item capacity limitation in VSTM, together with the success of the sample-size model in tasks in which the item capacity of VSTM was probably not exceeded, suggests that item and information capacities are complementary properties of the memory system, and that performance may be determined by both of these capacity limitations operating simultaneously. The item capacity of VSTM, termed the K-capacity, is determined by the multiplicative term on the right of Eq. (21). The item capacity is controlled by a double nonlinearity, realized by two applications of the threshold function, #ðÞ. As in the additive inhibition P 2 term, the summation over items, m j–i #ðxj Þ, counts the number of other stimuli, not including the one at location i, that are seeking to access VSTM. This number is compared to a number K that is strictly less than the item capacity of VSTM and again thresholded and then subtracted from 1.0. (We use K ¼ 3:5 in our model to implement an item capacity of four items.) The term as a whole will be equal to 1.0 if there are fewer than four other items in VSTM with trace strengths above some threshold value and equal to zero if there are more than four items. When there are fewer than four items, the K-capacity term has no effect on the growth of VSTM; when there are more than four items, dxi =dt goes to zero and the trace for item i stops growing. This means that once four items are established in VSTM, the memory is full and no new items will be admitted. The parameter  determines the minimum trace strength required for an item to become established in VSTM. The presence of the sign term, /ðmi Þ, means that the activity in xi ðtÞ increases or decreases according to whether mi ðtÞ is positive or negative. As a result, the VSTM trace encodes information about both the strength of the stimulus encoded by the what pathway and its identity.3

1323

10. Predictions of competitive interaction theory

1365

The predictions of the competitive interaction theory depend critically on the properties of the feedback function, f ðti Þ, which determine the kind of selection performed by the where pathway. Grossberg (1987a, 1987b, 1988) has shown that when the feedback function is increasing and positively accelerating, like a quadratic or other power function, the where pathway selects the strongest, or most target-like, stimulus in the display and suppresses all others; that is, it performs winner-takes-all selection. When the feedback function has a positively accelerating portion followed by a negatively accelerating portion, like a sigmoid, all stimuli above a certain threshold strength are selected and all stimuli below it are suppressed. That is, the system attempts to form representations of a subset of the strongest items.

1366

3 There is no requirement that the item capacity limit in Eq. 21 must remain invariant across trials. It seems plausible to us that it might depend on the observer’s state of preparation and vigilance, as envisaged in Kahneman’s (1973) effort-based theory of attentional capacity. Our main reason for including it in the model was to show how item-capacity limitations are compatible with normalization-dependent information-capacity limitations and can co-exist with them.

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364

1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404

1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

In the competitive interaction theory, the choice of the feedback function characterizes the observer’s strategy, or mode of information acquisition. A power-law feedback function puts the system into search mode. In search mode, the system attempts to form a VSTM representation of the single, most target-like stimulus in the display. A sigmoid feedback function puts the system into acquisition mode. In acquisition mode, the system attempts to form VSTM representations of as many task-relevant stimuli as it can, up to the item-capacity limits of VSTM. Because the where pathway dynamics are controlled by t2i , the squared pathway activity, the system is in search mode by default. The feedback function is assumed to be under the observer’s cognitive control: The observer can change the mode of acquisition by switching between a quadratic and a sigmoid feedback function. Psychologically, the feedback function determines how the system’s resources are allocated during the course of the trial as a function of partial information extracted from the display. In search mode, evidence for a target at location i leads all of the resources to be concentrated at that location, expressed by a quadratic feedback function. In acquisition mode, evidence for a target at location i leads to an initial commitment of resources to that location, followed by a withdrawal of resources, which allows the system to process targets at other locations as well. The feedback function thus provides a place in the model where bottom-up, stimulus-driven processing can interact with top-down cognitively-controlled processing.

10.1. Near-threshold search performance is well described by a MAX model Fig. 8 shows the predictions of the model when the feedback function, f, is quadratic, which puts the system into search mode. As noted in the previous section, because the dynamics of the where pathway are controlled by the squared amplitude ti ðtÞ2 , search mode is the default mode of the system. The predictions in Fig. 8 represent performance with an eight-item display with four signals (targets) and four noise stimuli (nontargets). To generate these predictions, we assumed, arbitrarily, that signals have a mean intensity of 0.5 and noise stimuli have a mean of 0.1. Because the model assumes separable stimulus representations, the predictions are relatively unaffected by these choices. The specific values

15

of stimulus intensity used to generate the predictions in Fig. 8 are given in the vector, I, in the caption. The sensory response function, lðtÞ, was chosen to represent an unmasked display, which allows activity in all of the pathways to reach asymptote before the stimulus decays. The values of the model parameters used to generate Q9 the predictions are given in the Appendix A (see Fig. 9). The three columns in Fig. 8 represent activity in the where pathway, the what pathway, and the VSTM trace, respectively. As shown in Fig. 8, the integrated system model of Smith and Ratcliff (2009) assumes (minimally) that there are two sources of noise in the model: between-trial noise and within-trial noise. The between-trial noise reflects variations in the encoding of nominally equivalent stimuli and is represented computationally by the variations of the intensity values, Ii , around their nominal means. The within-trial noise represents moment-to-moment perturbations in the VSTM trace strengths that drive the decision process. Fig. 8 shows two versions of the predictions, with and without within-trial noise. In the top row of the figure, within-trial noise on a 1 ms time base has been added to the where pathway, the what pathway, and the VSTM trace, in addition to the between-trial noise in the input, which is represented by variability among the values of the vector I. As a comparison with the bottom row of the figure shows, the predictions of the model are robust to the effects of added noise, up to the point where differences between stimuli become submerged by the noise within the system. However, the predictions of the model are somewhat easier to appreciate in the absence of noise. We focus specifically on the asymptotic 0 VSTM trace, xi ð1Þ, as the computational basis for d . As we discussed earlier, the integrated system model assumes a diffusion process decision stage, which allows it to predict accuracy and distributions of RT (Sewell & Smith, 2012; Smith et al., 2010; Smith & Ratcliff, 2009). Because our benchmark phenomena for multi-element displays involve accuracy rather than RT, we focus here on the model’s accuracy predictions. The left-hand column of Fig. 8 shows that, when the system is in search mode, the where pathway rapidly selects the largest of the stimulus inputs, I1 ¼ 0:53. It also initially selects the three weaker signals as well, but their representations are rapidly suppressed by t1 ðtÞ via competitive interaction. Noise (nontarget) stimuli are not selected; in the absence of within-trial trial noise, their associated nodes remain inactive.

Fig. 8. Search mode predictions of competitive interaction theory. Predictions for an eight-item display containing four signal (target) stimuli and four noise (nontarget) stimuli. The columns show activity in the where pathway, the what pathway, and VSTM, respectively, for noisy (top row) and noiseless (bottom row) pathways. The vector of stimulus inputs was I ¼ f0:53; 0:50; 0:49; 0:47; 0:13; 0:11; 0:09; 0:07g. The selected stimulus, x1 ð1Þ 0:5, is Ið1Þ, the largest of the signals.

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1

16

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

Fig. 9. Double-target deficit predictions. Predicted VSTM trace strengths for a backwardly-masked, four-item display containing either one target, I ¼ f0:81; 0:10; 0:10; 0:10g, or two targets, I ¼ f0:81; 0:79; 0:10; 0:10g. (Asymptotic trace strengths are less than the stimulus strengths because of the effects of backward masking.) The system was in search mode (quadratic feedback function) for the single-target task and acquisition mode (sigmoid feedback function) for the two-target task. The panel on the right shows 0 the predictions of the model superimposed on the d values from Duncan’s (1980) single-target and two-target tasks.

The middle column of Fig. 8 shows the strength of the activity in the what pathway. The what pathway carries information about 1462 both stimulus strength (i.e., the quality of the encoded discrimina1463 tive information) and stimulus identity – the latter represented by 1464 the sign of the activity in the pathway. However, to make it easier 1465 to compare variations in pathway activation across conditions, we 1466 assumed here that all of the stimuli had the same sign. We show an 1467 example in which this is not the case in Fig. 11. The figure shows 1468 that the system forms a near-veridical representation of stimulus 1469 I1 : its asymptotic activation is close to its nominal value of 0.53. 1470 The representations of the remaining three signals are, in compar1471 ison, much weaker; the mean of the asymptotic activations of 1472 these stimuli is around 0.14. 1473 The right-hand column of Fig. 8 shows that the VSTM trace 1474 strengths reflect the stimulus information carried by the what 1475 pathway. The system forms a near-veridical representation of the 1476 largest signal (x1 ð1Þ ¼ :49) and weaker representations of the 1477 remaining three signals. The nonsignal stimuli do not enter VSTM 1478 at all. The asymptotic VSTM trace strengths of the three weaker 1479 signals is somewhat less than their where pathway representations 1480 because of the competitive interactions in VSTM. 1481 The significance of the results in Fig. 8 is that they show that, 1482 when the system is in search mode, it performs winner-take-all 1483 selection on its stimulus inputs. The winner is veridically trans1484 ferred to VSTM and forms the basis for the subsequent decision. 1485 Winner-take-all selection is the computational equivalent of the 1486 signal-detection MAX model. If the vector of inputs, I, is regarded 1487 as a set of random variables drawn from a multivariate distribution 1488 of sensory effect, then the correspondence between winner-take-all 1489 selection and the MAX model is exact. Indeed, because the where 1490 pathway and VSTM jointly represent stimulus quality and stimulus 1491 identity, the model realizes the predictions of the more general 1492 signed-MAX model of Eq. (16): Stimuli are selected on the basis 1493 of strength (I2i ), but the decision is based on stimulus identity. 1494 The MAX model represents one end of a continuum of search 1495 efficiency identified by Wolfe (1998), in which targets can be 1496 selected in parallel across the visual field with no evidence of 1497 capacity limitations. Duncan and Humphreys (1989) showed that 1498 search efficiency depends both on the similarity between targets 1499 and distractors and on the similarity among distractors. As the dis1500 tractor set becomes more heterogeneous (i.e., the distractors 1501 become more dissimilar to one another), search becomes more dif1502Q10 ficult. Dosher, Han, and Lu (2010) and Mazyar, van den Berg and 1503 Ma (2012) reported large distractor heterogeneity effects in orien1504 tation discrimination (using line segments and Gabor patches, 1505 respectively). Dosher et al. found that performance with heteroge1506 neous distractors was well described by an unlimited-capacity par1507 allel model, in which the observer’s decision is based on 1508 independent decisions about each of the m items in the display, 1460

1461

and in which heterogeneity increases the probability that targets and distractors will be confused with each other and slows the rates at which stimulus representations are formed. Dosher et al.’s (2010) results are in broad agreement with our competitive interaction theory, which assumes that the efficiency of attentional selection depends upon the similarity among stimuli. Smith and Sewell (2013) showed that efficient selection, as represented by the MAX model and as realized computationally by the where pathway, breaks down if the competition between stimuli in Eq. (19) is allowed to depend on distractor similarity. They showed that, as distractors become more dissimilar, more and more items are selected into VSTM. Instead of basing decisions on the single, most-target-like item in the display, they must be based on all of the items in VSTM. This increases the noise in the decision process and reduces accuracy. Increasing distractor heterogeneity also reduces where-pathway activation for selected items, which decreases the rate at which their VSTM representations are formed, in agreement with Dosher et al.’s finding that heterogeneity decreases processing rates. Although Dosher et al. found their search data could be described by an unlimited capacity parallel model, our theoretical preference is for a competitive interaction model, because it accounts for other phenomena that cannot easily be explained by unlimited capacity parallel models.4

1509

10.2. The double-target detection deficit

1532

Double target studies suggest that there are costs associated with dividing attention over multiple sources of information, but, providing two targets do not occur simultaneously, the costs are no greater than would be predicted by an increase in noise alone and do not imply capacity limitations. However, there are large costs when two targets occur at the same time. This suggests that the system efficiently rejects nontargets in parallel, but that candidate targets require more extensive processing in a

1533

4 One substantive difference between Dosher et al.’s (2010) unlimited capacity parallel model and the competitive interaction theory of Smith and Sewell (2013) is that the former assumes the observer’s decision is based on all items in the display, whereas the latter assumes it is based only on those items that enter VSTM. If observers are required to report on items not represented in VSTM, they have to guess. As the competitive interaction theory has not been fitted to data from Dosher et al.’s paradigm, we do not know whether this difference between models would allow them to be distinguished, particularly as Dosher’s et al.’s model also included guessing parameters – although they allowed guessing rates to vary with heterogeneity but not with display size. Recent work by van den Berg, Awh and Ma (2014) has highlighted the difficulty in distinguishing between guessing and low-fidelity, unlimited-capacity VSTM encoding. They showed that findings that other researchers have attributed to item-capacity limitations could be explained by variable-precision unlimited capacity models, in which some items are represented in VSTM with zero or near-zero precision, which leads to guessing-like performance. The weak trace strengths associated with unselected stimuli in Fig. 8 could be interpreted as the computational substrate of low-precision representations of this kind.

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531

1534 1535 1536 1537 1538 1539 1540

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1 1541 1542 1543 1544 1545 1546 1547

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

limited-capacity system, as Duncan (1980) argued. In our theory, VSTM is a such limited capacity system, whose capacity limitations are expressed via VSTM trace strength normalization. The relevant data are from Duncan’s (1980) Experiment 2, which compared 0 vs. 1 target discrimination and 1 vs. 2 target discrimination for simultaneously presented arrays of four items. Our qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi  0 2 ffi 0 competitive interaction theory predicts that d2 ¼ d1 =2, where 0

0

d1 and d2 are sensitivities for single and dual targets, respectively. 0 0 1549 Duncan found d1 ¼ 1:88 and d2 ¼ 1:07, whereas the theory pre0 1550 dicts a double-target sensitivity of d2 ¼ 1:33, which slightly under1551 estimates the observed deficit. Smith and Sewell (2013) showed 1552 that the model could predict a double-target deficit of the same 1553 magnitude as the one in the data via a combination of VSTM 1554 strength normalization and decision noise. If each item in the dis1555 play contributes an independent source of noise to the decision 0 1556 process, then the predicted deficit is increased by around 0.3 d 1557 units. This is consistent with Dosher et al.’s (2010) finding that 1558 decisions about targets in heterogeneous, multielement displays 1559 are made by aggregating the results of noisy decisions about indi1560 vidual items. For the purposes of this article, the important finding 1561 is that the double-target deficit can be predicted by a process of 1562 VSTM trace-strength normalization acting on the squared trace 1563Q11 strengths.5 1548

1564

10.3. Redundancy costs in the post-stimulus probe task

1565

The left-hand panel of Fig. 10 shows data from an experiment reported by Smith (2010), which was designed, in part, to replicate the findings of Müller and Humphreys (1991). The observer’s task was to detect a 60 ms, vertical, Gabor patch target in an eight-item array. The nontargets were blank locations and all locations were masked with checkerboards after stimulus offset. One of the locations was cued 140 ms before stimulus onset; 500 ms after the onset of the mask array, one of the locations was probed, instructing observers to report the contents of that location. On 75% of trials the cued location was probed; on the remaining 25% of trials one of the other locations was probed. In addition to a possible target at the probed location, there were 0, 1, 2, or 4 targets at other, unprobed locations. As Fig. 10 shows, there was a large cuing effect 0 in this task: Sensitivity to cued targets was around two d units greater than to uncued targets. There was also a systematic redundancy cost for both cued and uncued targets: As the number of redundant targets increased, sensitivity at the probed location progressively decreased. Smith and Sewell (2013) attributed the redundancy costs to VSTM selection competition. To predict the large cuing effects in Fig. 10, Smith and Sewell (2013) asumed that, in addition to biasing the where-pathway selection process, attention differentially weights the VSTM trace strengths. This gives attended stimuli an advantage in the competition for VSTM capacity, not just for VSTM selection. The attention weights, ci , in the VSTM equation (Eq. (21)) represent this second locus of attentional involvement. The right-hand panel of Fig. 10 shows the predictions of the model when attention affects both where-pathway selection and VSTM trace normalization. The predictions were obtained using a standard set of model parameters (see Appendix A); they were not obtained by fitting the model to

1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594

17

5

Smith and Sewell (2013) showed that the double-target deficit in Duncan’s (1980) data could also be predicted by their competitive interaction model if it is assumed that the single-target and double-target tasks are performed in different information acquisition modes, specifically, if the single-target task is performed in search mode and the double-target task is performed in acquisition mode. The greater efficiency of VSTM selection in search mode leads to a greater difference between single-target and double-target performance than is predicted by trace-strength normalization 0 alone – again, by about 0.3 d units – in agreement with the size of the effect in Duncan’s data.

Fig. 10. Redundancy costs in the post-stimulus probe task. Observers reported the presence or absence of a Gabor patch target at a probed location in a backwardlymasked, eight-item display. Targets could occur at the cued location or at one of the seven uncued locations. There were 0, 1, 2, or 4 redundant targets at the other 0 (uncued and unprobed) locations. The left-hand panel shows sensitivity (d ), averaged over five observers; the right-hand panel shows the predictions of competitive interaction theory for the task. Circles are cued stimuli; triangles are uncued stimuli.

the data. Nevertheless, the predictions of the model and the data agree fairly well: There is a large cuing effect and a systematic redundant target cost for cued and miscued stimuli. These costs arise because redundant targets compete with the contents of the probed location for entry to VSTM and the amount of competition increases with their number. Competition leads to weighted normalization of VSTM trace strengths, as described in Section 9.3.

1595

10.4. The information capacity of visual short-term memory

1602

Sewell, Lilburn and Smith (2014) investigated the information capacity of VSTM using displays of 1, 2, 3, or 4 vertically or horizontally oriented Gabor patches, as shown in the upper panel of P  0 2 Fig. 11. They found invariance of i di for all exposure durations and for simultaneously and sequentially presented stimuli, which implicates the representational capacity of VSTM rather than VSTM encoding as the locus of the capacity limitations. As discussed in Section 9.3, our competitive interaction theory predicts invariance P  0 2 of i di . The dynamics of the VSTM equation, Eq. (21), are such that trace strengths are automatically renormalized when new items are added to VSTM. As a result, the model predicts that asymptotic trace strengths will be the same for simultaneous and sequential presentation, as Smith and Sewell (2013) and Sewell, Lilburn and Smith (2014) reported. The 12 panels on the right-hand side of Fig. 11 show the predicted activity in the where pathway, the what pathway, and VSTM, for 1, 2, 3, and 4 item displays when the system is in acquisition mode. In acquisition mode, the system attempts to form representations of as many target items as it can, up to the item capacity of VSTM (set at four in our simulations). To generate these predictions we assumed unmasked displays and a long (100 ms) stimulus exposure duration to allow the VSTM trace formation process to run to completion. With a single item in the display, VSTM forms a veridical representation of the stimulus, x1 ð1Þ ¼ I1 ¼ 0:50. As more items are added to the display, VSTM trace q strengths ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi are normalized according to the equation xm ¼ ðx1 Þ2 =m. (Subscripts here indicate the number of stimuli in the display rather than the stimulus location.) This gives x2 ð1Þ ¼ 0:35; x3 ð1Þ ¼ 0:29, and x4 ð1Þ ¼ 0:25. If d0 is assumed to be proportional to VSTM trace strength, these predictions realize those of the sample size model. The panel at the lower left of Fig. 11 shows signed VSTM traces predicted by Eq. (21) for a four-item display with three stimuli of one kind and one of the other. This figure shows how the VSTM trace simultaneously encodes information about stimulus identity and the strength of the evidence for that stimulus. We added noise

1603

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

1596 1597 1598 1599 1600 1601

1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1

18

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

Fig. 11. The information capacity limits of VSTM. The predictions are for a two-choice, post-stimulus probe task, like the one in the upper panel on the left. The 12 panels on the right show where pathway and what pathway activity and VSTM trace strengths for display sizes (m) of one, two, three, or four items. In acquisition mode, the model P  0 2 realizes the predictions of the sample-size model, i di ¼ constant, as shown by the numbers on the far right-hand side of the figure. The inset at the lower left shows noisy, signed VSTM traces for a four-item display containing three stimuli of one kind (coded as positive) and one stimulus of the other kind (coded as negative). The inset shows how VSTM simultaneously carries information about stimulus identity and strength and that it is robust to pathway noise.

1655

to the where and what pathways and to the VSTM trace to generate these predictions, as we did in Fig. 8, to show that the coding of stimulus identity in the model is robust to noise. In applications of the model to RT and accuracy data, the VSTM trace determines the drift of a diffusion process: A positive trace is evidence for one response; a negative trace is evidence for the other response. Because of the item-capacity limits in the VSTM equation (Eq. P 0 2 (21)), the model predicts invariance of only up to the i di assumed item-capacity limits, which we set at four items. As shown by Smith and Sewell (2013) and Sewell, Lilburn and Smith (2014), the dynamics of VSTM growth are such that once memory traces for four items are established in VSTM, the magnitudes of which are controlled by the threshold parameter , the memory is full and no new items can be added. Within the overall item constraints of memory, normalization distributes the representational capacity of VSTM following the predictions of the sample-size model.

1656

11. Conclusion

1657

In this article, we have sought to characterize the link between normalization models and shunting differential equations and to

1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654

1658

outline some of the theoretical implications of that link. Normalization models have provided accounts of a variety of visual phenomena, including the effects of covert attention. We have shown that normalization models can be obtained as the asymptotic solutions of shunting differential equations, which characterize how the response of a visual mechanism depends on interactions among its excitatory and inhibitory inputs. The theoretical value of the shunting equation formalism is that it provides a characterization of the entire time course of the response, not just its asymptotic strength. Using the models of Smith and Ratcliff (2009) and Smith and Sewell (2013) as examples, we have shown that dynamic models based on shunting equations can provide an account of the processes of attentional selection and VSTM trace formation, where the latter provides the stimulus representations used in perceptual decision-making. We have shown that these models provide a unified account of a variety of attentional phenomena. They provide an account of how attention, in combination with other experimental variables, affects the speed and accuracy of perceptual decision-making. They also provide an explicit computational account of attentional selection processes, the capacity limitations of VSTM, the double-target detection deficit, and the effects of target redundancy in probed

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1 1681 1682 1683 1684

1685

19

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

detection tasks. In view of the diversity of phenomena accounted for by the shunting/normalization framework, we strongly endorse Carandini and Heeger’s (2012) characterization of normalization as a canonical neural computation.

h i9 8 R <1  exp 2jIi j 0t lðsÞ ds = h i Ri ðtÞ ¼ jIi j :1 þ exp 2jI j R t lðsÞ ds ; i 0  Z t lðsÞ ds ; ¼ jIi j tanh jIi j

1739

ð23Þ

0

12. Uncited references

Buffalo, Fries, Landman, Liang, and Desimone (2010) and 1687Q12 Gradshteyn and Ryzhik (1965)

1741

which is Eq. (14).

1742

A.2. Model implementation

1743

This section provides details of the implementation of the integrated system model and the competitive interaction model that were not presented in the text. It also describes the ways in which the models in the text differ from the models as presented by Smith and Ratcliff (2009) and Smith and Sewell (2013).

1744

A.3. Sensory response function

1749

The sensory response to a brief pulsed stimulus of duration d, denoted lðtÞ, is of the form

1750

1686

1688

Acknowledgments

1689

The research in this article was supported by Australian Research Council Discovery Grants DP110103406 and DP140102970 to Philip Smith, a Discovery Early Career Award DE140100772 to David Sewell, and an Australian Postdoctoral Award to Simon Lilburn. P.S. would like to express his appreciation for the hospitality of the Department of Psychology at Vanderbilt University, where work on the preparation of this article was carried out.

1690 1691 1692 1693 1694 1695 1696

lðtÞ ¼ Cðt; bon ; nÞ½1  Cðt  d; boff ; nÞ;

1697

Appendix A

1698

A.1. Solutions of Eqs. (7) and (13)

1699

1700

1702

where Cðt; b; nÞ is the output of a linear filter composed of n identical exponential stages,

We write Eq. (7) as

dRi ¼ lðtÞ½A  BRi ðtÞ; dt Ipi

Cðt; b; nÞ ¼ 1  ebt q

1704

with A ¼ and B ¼ ai þ . After separating variables and j bj I j multiplying both sides by B we obtain

1707

BdRi ¼ BlðtÞdt; A  BRi

1708

which is integrable. Integrating both sides yields

1703

1705

1709

1711

lnðA  BRi Þ ¼ 

Z

BlðtÞ dt þ C:

1713

With the initial condition Ri ð0Þ ¼ 0, the constant of integration is C ¼ ln A. Exponentiating both sides and solving for Ri gives

1716

 Z t  A Ri ðtÞ ¼ 1  exp B lðsÞ ds ; B 0

1712

1714

1717 1718 1719

1720

1722 1723

1724

which is Eq. (8). To solve Eq. (13), we eliminate the common and write it as

1727

which can be written as

dR 1

R2

¼ jIi jlðtÞ dt;

after making the substitution R ¼ Ri =jIi j. Using the fact that the inte1 gral of ð1  x2 Þ is ð1=2Þ ln½ð1 þ xÞ=ð1  xÞ (Gradshteyn and Ryzhik, 1733Q13 1994, p. 63), we integrate both sides of this equation to obtain 1731 1732

1734 1736

1737 1738

1 1þR ln ¼ jIi j 2 1R

Z

j!

:

lðtÞ dt þ C:

With the initial condition Ri ð0Þ ¼ 0, the constant of integration is zero. Solving for R, and then substituting for Ri ðtÞ, we obtain

1746 1747 1748

1751

1752 1754 1755 1756

1757

ð25Þ 1759

The onset (rise) and offset (decay) times of the filter depend on the time constants bon and boff , respectively. When stimuli are unmasked, bon > boff (slow decay); when they are masked, boff > bon (fast decay).

1760

A.4. Where pathway feedback function

1764

The where pathway growth rate is controlled by the square of the activity in the pathway, ti ðtÞ2 , so when the feedback function is the identity, f ½ti ðtÞ2  ¼ ti ðtÞ2 , the feedback is quadratic and the system is in search mode by default. The inclusion of a switchable sigmoid feedback function puts the system into acquisition mode,

1765

f

 2

ti

n h   b io ¼ c1 1  exp  at2i ;

ð26Þ

1761 1762 1763

1766 1767 1768 1769

1770 1772

where a; b, and c1 are constants (Table A1).

1773

A.5. Soft threshold function

1774

The soft threshold function, #ðx2i Þ, is a smooth approximation to a step function, which is used in the VSTM equationto count the number of active traces in memory. In our model, # x2i was implemented as

1775

  

 1 # x2i ¼ 1 þ exp c2 x2i  x20 ;

¼ lðtÞ dt;

I2i  R2i

1730

lðtÞI2i Ri ðtÞ terms

Separation of variables then yields

1726 1728

ð22Þ

h i dRi ¼ lðtÞ I2i  R2i ðtÞ : dt dRi

n1 X ðbtÞj j¼0

P

ð24Þ

1745

ð27Þ

1776 1777 1778

1779 1781

where c2 , and x0 are constants (Table A1).

1782

A.6. Differences from the models of Smith and Ratcliff (2009) and Smith and Sewell (2013)

1783

Smith and Ratcliff (2009) assumed that the growth rates in their energy (where) and form (what) pathways were controlled by the amplitude rather than the power (i.e., the squared amplitude) of the stimulus and the activity in the pathways. In the notation of this article, their energy pathway equation was

1785

  dti ¼ ci Icomp lðtÞ½1  ti ðtÞ  ½1  Icomp lðtÞti ðtÞ ; dt

ð28Þ

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

1784

1786 1787 1788 1789

1790 1792

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806

20

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

where ci 2 fcA ; cU g are attention weights and Icomp is the square root of the contrast energy in the stimulus compound. The stimulus compound is the combination of the Gabor patch and pedestal when a pedestal is used (Fig. 1). Eq. (28) has the same form as the balanced, center-surround, shunting equation of Eq. (11). When stimuli are localized using a fiducial cross, the energy in the fiducial cross is also assumed to contribute to the stimulus compound and to affect salience. Unlike the pedestal task, however, in which the energy in the compound is dominated by the pedestal and remains roughly constant across stimulus conditions, in the fiducial-cross task the energy in the compound varies appreciably with stimulus contrast. These assumptions allowed Smith and Ratcliff to explain the differences in the RT distributions in the two kinds of tasks. Smith and Ratcliff’s (2009) form (what) pathway equation was

1807 1809

dmi ¼ ci ti ðtÞfIi lðtÞ½1  mi ðtÞ  ½1  Ii lðtÞmi ðtÞg: dt

ð29Þ

1835

1836

A.7. Model parameters

1837

The competitive interaction model was implemented in C++, called from Matlab. Eqs. (19)–(21) were solved numerically using the Runge–Kutta method (Kreyszig, 1979, p. 797). Table A1 gives the model parameters used to generate the simulations used in this article. Although the predictions of the model depend on a relatively large number of numerical constants, many of them – such as those that control selection by the where pathway and those that control the threshold properties of VSTM – should be viewed as fixed properties of the model architecture rather than as free parameters used to optimize the fit to data. Sewell, Lilburn and Smith (2014) found that a simplified form of the model provided a good fit to the data from their VSTM paradigm, in which display size and exposure duration were varied factorially, and to data from a similar paradigm by Vogel, Woodman and Luck (2006). The fits to these data required four and six parameters, respectively, making the model comparably parsimonious to other models in the literature. The values of c2 and x2 , the parameters of the threshold function #ðÞ in Eq. (21) given in Table A1 differ from those given by

1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834

1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855

Symbol

Parameter

Value

bon boff boff n

Sensory response function onset rate Sensory response offset rate (unmasked) Sensory response offset rate (masked) Number of stages in sensory response function Attention gain (attended stimuli) Attention gain (unattended stimuli) What pathway rate Quadratic feedback scale (where pathway) Sigmoid feedback exponent (where pathway) Sigmoid feedback amplitude (where pathway) VSTM discriminability parameter VSTM threshold scale VSTM threshold location VSTM selection threshold VSTM item capacity Where pathway decay Noise standard deviation Noise perturbation interval (s)

200.0 30.0 200.0 3 25.0 12.5 50.0 50.0 2.0 2.0 1.0 120.0 0.225 5.0 3.5 50.0 0.1 .001

cA cU a a b c1 h c2

x0



This equation is again a balanced center-surround equation, in which the growth rate is controlled by amplitude rather than power. The growth rate depends on the product of the attention gain, ci , and the activity in the where pathway, ti ðtÞ. The stimulus input, Ii , is the contrast energy in the Gabor patch, rather than the compound, which is the basis for the discriminative judgment. Smith and Sewell (2013) chose to express their pathway equations in power rather than amplitude terms. Their main reason for doing so was because a VSTM normalization equation controlled by power P  0 2 naturally expresses the additivity and invariance of i di . Our other reason for characterizing the model in terms of power rather than amplitude was to link it to Heeger’s (1991, 1992) idea that form is coded by pairs of half-squaring (i.e., power or energy based) operators. The other main difference between the model presented in this article and the model of Smith and Sewell (2013) is the inclusion of the sign term, /ðIÞ, in the equations for the what pathway and VSTM, which allows them to code both stimulus identity and strength. Because the VSTM traces, xi ðtÞ, can either be positive or negative in the revised model, we write the threshold function of Eq. (27) as #ðx2i Þ instead of #ðxi Þ, which ensures that VSTM normalization depends only on the strengths of the traces and not on their signs. The form of the threshold function in Eq. (27) is the same as the one assumed by Smith and Sewell, but its parameters are different, to reflect the fact that its argument is x2i rather than xi .

1810

Table A1 Model parameters.

K k

r D

Smith and Sewell (2013) because we have written it as a function of x2i ðtÞ rather than xi ðtÞ.

1856

References

1858

Albrecht, D. G., & Geisler, W. S. (1991). Motion selectivity and the contrast-response function of simple cells in the visual cortex. Visual Neuroscience, 7, 531–546. Albrecht, D. G., & Hamilton, D. B. (1982). Striate cortex of monkey and cat: Contrast response function. Journal of Neurophysiology, 48, 217–237. Anderson, D. E., Vogel, E. K., & Awh, E. (2011). Precision in visual working memory reaches a stable plateau when individual item limits are exceeded. Journal of Neuroscience, 31, 1128–1138. Baldassi, S., & Burr, D. C. (2004). Pop-out of targets modulated in luminance or colour: The effects of intrinsic and extrinsic uncertainty. Vision Research, 44, 1227–1233. Baldassi, S., & Verghese, P. (2002). Comparing integration rules in visual search. Journal of Vision, 2, 559–570. Bonnel, A. M., & Hafter, E. R. (1998). Divided attention between simultaneous auditory and visual signals. Perception and Psychophysics, 60, 179–190. Bonnel, A. M., & Miller, J. (1994). Attentional effects on concurrent psychophysical discriminations: Investigations of a sample-size model. Perception & Psychophysics, 55, 162–179. Boynton, G. (2005). Attention and visual perception. Current Opinion in Neurobiology, 15, 465–469. Breitmeyer, B. G. (1984). Visual masking: An integrative approach. Oxford UK: Clarendon Press. Buffalo, E. A., Fries, P., Landman, R., Liang, H., & Desimone, R. (2010). A backward progression of attentional effects in the ventral stream. Proceedings of the National Academy of Sciences of the United States of America, 107, 361–365. Busey, T. A., & Loftus, G. R. (1994). Sensory and cognitive components of visual information acquisition. Psychological Review, 101, 446–469. Cameron, E. L., Tai, J. C., & Carrasco, M. (2002). Covert attention affects the psychometric function of contrast sensitivity. Vision Research, 42, 949–967. Campbell, F. W., & Robson, J. (1968). Application of Fourier analysis to the visibility of gratings. Journal of Physiology, 197, 551–566. Carandini, M., & Heeger, D. J. (2012). Normalization as a canonical neural computation. Nature Reviews Neuroscience, 13, 51–62. Carrasco, M. (2014). Spatial attention: Perceptual modulation. In S. Kastner & A. C. Nobre (Eds.), The oxford handbook of attention (pp. 183–230). Oxford (UK): Oxford University Press. Carrasco, M., Giordano, A. M., & McElree, B. (2006). Attention speeds processing across eccentricity: Feature and conjunction searches. Vision Research, 46, 2028–2040. Carrasco, M., & McElree, B. (2001). Covert attention speeds the accrual of visual information. Proceedings of the National Academy of Science, 98, 5363–5367. Carrasco, M., McElree, B., Denisova, K., & Giordano, A. M. (2003). Speed of visual processing increases with eccentricity. Nature Neuroscience, 6, 699–700. Carrasco, M., Penpeci-Talgar, C., & Eckstein, M. (2000). Spatial covert attention increases contrast sensitivity across the CSF: Support for signal enhancement. Vision Research, 40, 1203–1215. Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87–185. Dao, D. Y., Lu, Z.-L., & Dosher, B. A. (2006). Adaptation to sine-wave gratings selectively reduces the contrast gain of the adapted stimuli. Journal of Vision, 6, 739–759. Desimone, R., & Duncan, J. (1995). Neural mechanisms of selective visual attention. Annual Review of Neuroscience, 18, 193–222.

1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

1857

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978Q14 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx Deutsch, J. A., & Deutsch, D. (1963). Attention: Some theoretical considerations. Psychological Review, 70, 80–90. Donkin, C., Nosofsky, R. M., Gold, J. M., & Shiffrin, R. M. (2013). Discrete-slots models of visual working-memory response times. Psychological Review, 120, 873–902. Dosher, B. A. (1979). Empirical approaches to information processing: Speedaccuracy tradeoff functions or reaction time – a reply. Acta Psychologica, 43, 347–359. Dosher, B. A., & Lu, Z.-L. (2000a). Mechanisms of perceptual attention in precuing of location. Vision Research, 40, 1269–1292. Dosher, B. A., & Lu, Z.-L. (2000b). Noise exclusion in spatial attention. Psychological Science, 11, 139–146. Downing, C. J. (1988). Expectancy and visual-spatial attention: Effects on perceptual quality. Journal of Experimental Psychology: Human Perception and Performance, 14, 188–202. Duncan, J. (1980). The locus of interference in the perception of simultaneous stimuli. Psychological Review, 87, 272–300. Duncan, J., & Humphreys, G. W. (1989). Visual search and stimulus similarity. Psychological Review, 96, 433–458. Eckstein, M. P. (1998). The lower efficiency for conjunctions is due to noise and not serial attentional processing. Psychological Science, 9, 111–118. Eckstein, M. P., Thomas, J. P., Palmer, J., & Shimozaki, S. (2000). A signal detection model predicts the effects of set size on visual search accuracy for feature, conjunction, triple conjunction, and disjunction displays. Perception & Psychophysics, 62, 425–451. Foley, J. M. (1994). Human luminance pattern-vision mechanisms: Masking experiments require a new model. Journal of the Optical Society of America, A, 11, 1710–1719. Foley, N. C., Grossberg, S., & Mingolla, E. (2012). Neural dynamics of object-based multifocal visual spatial attention and priming: Object cuing, useful-field-ofview, and crowding. Cognitive Psychology, 65, 77–117. Foley, J. M., & Schwarz, W. (1998). Spatial attention: Effect of position uncertainty and number of distractor patterns on the threshold-versus-contrast function for contrast discrimination. Journal of the Optical Society of America, A, 15, 1036–1047. Francis, G. (2000). Quantitative theories of metacontrast masking. Psychological Review, 107, 768–785. Geisler, W. S., & Albrecht, D. G. (1997). Visual cortex neurons in monkeys and cats: Detection, discrimination, and identification. Visual Neuroscience, 14, 897–919. Gilliom, J. D., & Sorkin, R. D. (1974). Sequential vs. simultaneous two-channel signal detection: More evidence for a high-level interrupt theory. Journal of the Acoustical Society of America, 56, 157–164. Gould, I. C., Wolfgang, B. J., & Smith, P. L. (2007). Spatial uncertainty explains exogenous and endogenous attentional cuing effects in visual signal detection. Journal of Vision, 7(13:4), 1–17. Gradshteyn, I. S., & Ryzhik, I. M. (1965). Tables of integrals, series, and productions. San Diego, CA: Academic Press. Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley. Grossberg, S. (1980). How does a brain build a cognitive code? Psychological Review, 87, 1–51. Grossberg, S. (Ed.). (1987a). The adaptive brain, I. Amsterdam. The Netherlands: Elsevier. Grossberg, S. (Ed.). (1987b). The adaptive brain, II. Amsterdam. The Netherlands: Elsevier. Grossberg, S. (1988). Nonlinear neural networks: Principles, mechanisms, and architectures. Neural Networks, 1, 17–61. Hawkins, H. L., Hillyard, S. A., Luck, S. J., Mouloua, M., Downing, C. J., & Woodward, D. P. (1990). Visual attention modulates signal detectability. Journal of Experimental Psychology: Human Perception and Performance, 16, 802–811. Heeger, D. J. (1991). Nonlinear model of neural responses in cat visual cortex. In M. S. Landy & J. A. Movshon (Eds.), Computational models of visual processing (pp. 119–133). Cambridge (MA): MIT Press. Heeger, D. (1992). Normalization of cell responses in cat striate cortex. Visual Neuroscience, 9, 181–197. Herrmann, K., Heeger, D. J., & Carrasco, M. (2012). Feature-based attention enhances performance by increasing response gain. Vision Research, 74, 10–20. Herrmann, K., Montaser-Kouhsari Carrasco, M., & Heeger, D. J. (2010). When size matters: Attention affects performance by contrast or response gain. Nature Neuroscience, 13, 1554–1559. Kahneman, D. (1973). Attention and effort. Englewood Cliffs (NJ): Prentice Hall. Kaplan, E., Lee, B. B., & Shapley, R. M. (1990). New views of primate retinal function. In Progress in retinal research. In N. N. Osborne & G. J. Chader (Eds.) (Vol. 9, pp. 273–336). Oxford (England): Pergamon Press. Kerzel, D., Gauch, A., & Buetti, S. (2010). Involuntary attention with uncertainty: Peripheral cues improve perception of masked letters, but may impair perception of low-contrast letters. Journal of Vision, 10(12), 1–13. Kreyszig, E. (1979). Advanced engineering mathematics (4th ed.). New York (NY): Wiley. Laming, D. R. J. (1986). Sensory analysis. Orlando (FL): Academic Press. Lee, D. K., Itti, L., Koch, C., & Braun, J. (1999). Attention activates winner-take-all competition among visual filters. Nature Neuroscience, 2, 375–381. Lee, J., & Maunsell, J. H. R. (2009). A normalization model of attentional modulation of single unit responses. PLoS One, 4(2), 1–13. Lindsay, P. H., Taylor, M. M., & Forbes, S. M. (1968). Attention and multidimensional discrimination. Perception & Psychophysics, 4, 113–117.

21

Ling, S., & Carrasco, M. (2006a). Sustained and transient covert attention enhance the signal via different contrast response functions. Vision Research, 46, 1210–1220. Ling, S., Liu, T., & Carrasco, M. (2009). How spatial and feature-based attention affect the gain and tuning of population responses. Vision Research, 4, 1194–1204. Liu, C. C., Wolfgang, B. J., & Smith, P. L. (2009). Attentional mechanisms in simple visual detection: A speed-accuracy trade-off analysis. Journal of Experimental Psychology: Human Perception and Performance, 35, 1329–1345. Loftus, G. R., & Ruthruff, E. (1994). A theory of visual information acquisition and visual memory with special application to intensity-duration trade-offs. Journal of Experimental Psychology: Human Perception and Performance, 20, 33–49. Luck, S. J., Hillyard, S. A., Mouloua, M., Woldorff, M. G., Clark, V. P., & Hawkins, H. L. (1994). Effects of spatial cuing on luminance detectability: Psychophysical and electrophysical evidence. Journal of Experimental Psychology: Human Perception and Performance, 20, 887–904. Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature, 390, 279–281. Lu, Z.-L., & Dosher, B. A. (1998). External noise distinguishes attention mechanisms. Vision Research, 38, 1183–1198. Lu, Z.-L., & Dosher, B. A. (2000). Spatial attention: Different mechanisms for central and peripheral temporal precues? Journal of Experimental Psychology: Human Perception and Performance, 26, 1534–1548. Lu, Z.-L., & Dosher, B. A. (2008). Characterizing observers using external noise and observer models: Assessing internal representations with external noise. Psychological Review, 115, 44–82. Lu, Z.-L., Jeon, S.-T., & Dosher, B. A. (2004). Temporal tuning characteristics of the perceptual template and endogeneous cuing of spatial attention. Vision Research, 44, 1333–1350. Lu, Z.-L., Lesmes, L. A., & Dosher, B. A. (2002). Spatial attention excludes noise at the target location. Journal of Vision, 2, 312–323. Mazyar, H., van den Berg, R., & Ma, W. J. (2012). Does precision decrease with set size? Journal of Vision, 12(6:1), 1–16. Meese, T. S., & Holmes, D. J. (2010). Orientation masking and cross-orientation suppression (XOS): Implications for estimates of filter bandwidths. Journal of Vision, 10(12:9), 1–20. Moray, N. (1969). Selective processes in vision and hearing. London (UK): Hutchinson. Morgan, M. J., Ward, R. M., & Castet, E. (1998). Visual search for a tilted target: Tests of spatial uncertainty models. The Quarterly Journal of Experimental Psychology A: Human Experimental Psychology, 51A, 347–370. Müller, H. J., & Humphreys, G. W. (1991). Luminance-increment detection: Capacity-limited or not? Journal of Experimental Psychology: Human Perception and Performance, 17, 107–124. Mulligan, R. M., & Shaw, M. L. (1980). Multimodal signal detection: Independent decisions vs. integration. Perception & Psychophysics, 28, 471–478. Nachmias, J. (2002). Contrast discrimination with and without spatial uncertainty. Vision Research, 42, 41–48. Norman, D. A. (1968). Towards a theory of memory and attention. Psychological Review, 75, 522–536. Palmer, J. (1990). Attentional limits on the perception and memory of visual information. Journal of Experimental Psychology: Human Perception and Performance, 16, 332–350. Palmer, J. (1994). Set-size effects in visual search: The effect of attention is independent of the stimulus for simple tasks. Vision Research, 34, 1703–1721. Palmer, J., Ames, C. T., & Lindsey, D. T. (1993). Measuring the effect of attention on simple visual search. Journal of Experimental Psychology: Human Perception and Performance, 19, 108–130. Palmer, J., Verghese, P., & Pavel, M. (2000). The psychophysics of visual search. Vision Research, 40, 1227–1268. Polyanin, A. D., & Zaitsev, V. F. (2003). Handbook of exact solutions for ordinary differential equations (2nd ed.). Boca Raton (FL): Chapman and Hall/CRC. Ratcliff, R. (1978). A theory of memory retrieval. Psychological Review, 85, 59–108. Ratcliff, R., & Rouder, J. (2000). A diffusion model account of masking in two-choice letter identification. Journal of Experimental Psychology: Human Perception and Performance, 26, 127–140. Ratcliff, R., & Smith, P. L. (2004). A comparison of sequential-sampling models for two choice reaction time. Psychological Review, 111, 333–367. Ratcliff, R., & Smith, P. L. (2010). Perceptual discrimination in static and dynamic noise: The temporal relationship between perceptual encoding and decision making. Journal of Experimental Psychology: General, 139, 70–94. Reed, A. V. (1976). List length and the time course of recognition in human memory. Memory & Cognition, 4, 16–30. Reynolds, J. H., & Heeger, J. H. (2009). The normalization model of attention. Neuron, 61, 168–185. Ross, J., & Speed, H. D. (1991). Contrast adaptation and contrast masking in human vision. Proceedings of the Royal Society (Series B), 246, 61–69. Scholl, B., Latimer, K. W., & Priebe, N. J. (2012). A retinal source of spatial contrast gain control. Journal of Neuroscience, 32, 9824–9830. Schwartz, O., & Simoncelli, E. P. (2001). Natural signal statistics and sensory gain control. Nature Neuroscience, 4, 819–825. Sclar, G., Maunsell, J. H. R., & Lennie, P. (1990). Coding of image contrast in central visual pathways of the macaque monkey. Vision Research, 30, 1–10. Sewell, D. K., Lilburn, S. D., & Smith, P. L. (2014). An information capacity limit of visual short-term memory. Journal of Experimental Psychology: Human Perception and Performance.. Published online, September 15.

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078

VR 6930

No. of Pages 22, Model 5G

12 November 2014 Q1 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123

22

P.L. Smith et al. / Vision Research xxx (2014) xxx–xxx

Sewell, D. K., & Smith, P. L. (2012). Attentional control in visual signal detection: Effects of abrupt-onset and no-onset stimuli. Journal of Experimental Psychology: Human Perception and Performance, 38, 1043–1068. Shaw, M. L. (1980). Identifying attentional and decision-making components in information processing. In R. S. Nickerson (Ed.), Attention and performance VIII (pp. 227–296). Hillsdale (NJ): Erlbaum. Shaw, M. I. (1982). Attending to multiple sources of information: I. The integration of information in decision making. Cognitive Psychology, 14, 353–409. Shimozaki, S. S., Eckstein, M. P., & Abbey, C. K. (2003). Comparison of two weighted integration models for the cueing task: Linear and likelihood. Journal of Vision, 3, 2009–2229. Shiu, L., & Pashler, H. (1994). Negligible effects of spatial precuing on identification of single digits. Journal of Experimental Psychology: Human Perception and Performance, 20, 1037–1054. Smith, P. L. (1998). Attention and luminance detection: A quantitative analysis. Journal of Experimental Psychology: Human Perception and Performance, 24, 105–133. Smith, P. L. (2000). Attention and luminance detection: Effects of cues, masks, and pedestals. Journal of Experimental Psychology: Human Perception and Performance, 26, 1401–1420. Smith, P. L. (2010). Spatial attention and the detection of weak visual signals. In V. Coltheart (Ed.), Tutorials in visual cognition (pp. 211–259). New York (NY): Psychology Press. Smith, P. L., Ellis, R., Sewell, D. K., & Wolfgang, B. J. (2010). Cued detection with compound integration–interruption masks reveals multiple attentional mechanisms. Journal of Vision, 10, 1–28. Smith, P. L., Lee, Y.-E., Wolfgang, B. J., & Ratcliff, R. (2009). Attention and the detection of masked radial frequency patterns: Data and model. Vision Research, 49, 1363–1377. Smith, P. L., & Ratcliff, R. (2009). An integrated theory of attention and decision making in visual signal detection. Psychological Review, 116, 283–317. Smith, P. L., Ratcliff, R., & Sewell, D. K. (2014). Modeling perceptual discrimination in dynamic noise: Time-changed diffusion and release from inhibition. Journal of Mathematical Psychology, 59, 95–113. Smith, P. L., Ratcliff, R., & Wolfgang, B. J. (2004). Attention orienting and the time course of perceptual decisions: Response time distributions with masked and unmasked displays. Vision Research, 44, 1297–1320. Smith, P. L., & Sewell, D. K. (2013). A competitive interaction theory of attentional selection and decision making, in brief, multielement displays. Psychological Review, 120, 589–625. Smith, P. L., & Wolfgang, B. J. (2004). The attentional dynamics of masked detection. Journal of Experimental Psychology: Human Perception and Performance, 30, 119–136. Smith, P. L., & Wolfgang, B. J. (2007). Attentional mechanisms in visual signal detection: The effects of simultaneous and delayed noise and pattern masks. Perception & Psychophysics, 69, 1093–1104.

Smith, P. L., Wolfgang, B. J., & Sinclair, A. (2004). Mask-dependent attentional cuing effects in visual signal detection: The psychometric function for contrast. Perception & Psychophysics, 66, 1056–1075. Sorkin, R. D., & Pohlmann, L. D. (1973). Some models of observer behavior in twochannel auditory signal detection. Perception & Psychophysics, 14, 101–109. Sorkin, R. D., Pohlmann, L. D., & Gilliom, J. (1973). Simultaneous two-channel signal detection: III. 603 and 1400 Hz signals. Journal of the Acoustical Society of America, 53, 1045–1051. Sorkin, R. D., Pohlmann, L. D., & Woods, D. D. (1976). Decision interaction between auditory channels. Perception & Psychophysics, 19, 290–295. Sperling, G. (1960). The information available in brief stimulus presentations. Psychological Monographs, 74, 1–29. Sperling, G., & Sondhi, M. M. (1968). Model for visual luminance discrimination and flicker detection. Journal of the Optical Society of America, A, 58, 1133–1145. Sperling, G., & Weichselgartner, E. (1995). Episodic theory of the dynamics of spatial attention. Psychological Review, 102, 503–532. Taylor, M. M., Lindsay, P. H., & Forbes, S. M. (1967). Quantification of shared capacity processing in auditory and visual discrimination. Acta Psychologica, 27, 223–229. Treisman, A. M. (1960). Contextual cues in selective listening. Quarterly Journal of Experimental Psychology, 12, 242–248. Treisman, A. M., & Gelade, G. (1980). A feature integration theory of attention. Cognitive Psychology, 12, 97–136. Tuckwell, H. (1988). Introduction to theoretical neurobiology, Nonlinear and stochastic theories (Vol. 2). Cambridge UK: Cambridge. van den Berg, R., Awh, E., & Ma, W. J. (2014). Factorial comparison of working memory models. Psychological Review, 121, 124–149. Verghese, P. (2001). Visual search and attention: A signal detection approach. Neuron, 31, 523–535. Vogel, E. K., Woodman, G. F., & Luck, S. J. (2006). The time course of consolidation in visual working memory. Journal of Experimental Psychology: Human Perception and Performance, 32, 1436–1451. Watson, A. B. (1986). Temporal sensitivity. In K. R. Boff, L. Kaufman, & J. P. Thomas (Eds.). Handbook of perception and performance (Vol. 1, pp. 6.1–6.85). New York: Wiley. Wilson, H. R., & Kim, J. (1998). Dynamics of a divisive gain control in human vision. Vision Research, 38, 2735–2741. Wolfe, J. M. (1998). What can 1 million trials tell us about visual search? Psychological Science, 9, 33–39. Yantis, S., & Jonides, J. (1984). Abrupt visual onsets and selective attention: Evidence from visual search. Journal of Experimental Psychology: Human Perception & Performance, 10, 601–621.

Q1 Please cite this article in press as: Smith, P. L., et al. From shunting inhibition to dynamic normalization: Attentional selection and decision-making in brief visual displays. Vision Research (2014), http://dx.doi.org/10.1016/j.visres.2014.11.001

2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166