- Email: [email protected]
To bind or not to bind, that's the wrong question: Features and objects coexist in visual short-term memory
Acta Psychologica 167 (2016) 45–51
Contents lists available at ScienceDirect
Acta Psychologica journal homepage: www.elsevier.com/locate/actpsy
To bind or not to bind, that's the wrong question: Features and objects coexist in visual short-term memory Shriradha Geigerman a, Paul Verhaeghen a, John Cerella b a b
Georgia Institute of Technology, United States Syracuse University, United States
a r t i c l e
i n f o
Article history: Received 17 March 2015 Received in revised form 11 April 2016 Accepted 13 April 2016 Available online xxxx Keywords: Working memory Visual short-term memory Binding Objects Features
a b s t r a c t In three experiments, we investigated whether features and whole-objects can be represented simultaneously in visual short-term memory (VSTM). Participants were presented with a memory set of colored shapes; we probed either for the constituent features or for the whole object, and analyzed retrieval dynamics (cumulative response time distributions). In our first experiment, we used whole-object probes that recombined features from the memory display; we found that subjects' data conformed to a kitchen-line model, showing that they used whole-object representations for the matching process. In the second experiment, we encouraged independent-feature representations by using probes that used features not present in the memory display; subjects' data conformed to the race-model inequality, showing that they used independent-feature representations for the matching process. In a final experiment, we used both types of probes; subjects now used both types of representations, depending on the nature of the probe. Combined, our three experiments suggest that both feature and whole-object representations can coexist in VSTM. © 2016 Published by Elsevier B.V.
1. Introduction Visual short-term memory (VSTM) stores visual information for a brief period of time (a few seconds) in the interest of ongoing tasks (Baddeley & Hitch, 1974). There is considerable consensus that a severe capacity limit exists on the number of items it can represent – typically no more than 3 or 4 at a time (Cowan, 2001; Luck & Vogel, 1997). A question central to our understanding of the observed capacity limit within VSTM is qualitative: What is the format of representation of information in VSTM (e.g., Suchow, Fougnie, Brady, & Alvarez, 2014) that determines its capacity limits? There are three points of view on this matter. According to the object-driven view, VSTM stores information in the form of integrated objects; capacity is then measured as the number of (multi-feature) objects that can be maintained simultaneously (Lee & Chun, 2001; Luck & Vogel, 1997; Vogel, Woodman, & Luck, 2001). Alternatively, the feature-driven view of VSTM posits that capacity is influenced by the total number of features (such as colors, shapes, or orientations) that constitute a visual scene (Bays, Catalao, & Husain, 2009; Bays, Wu, & Husain, 2011; Wheeler & Treisman, 2002), independent of the number of objects across which those features are distributed. The third view is a hybrid: The object-and-features-driven view proposes dual representation of both objects and features in VSTM. According to this view, the VSTM capacity limit is a function of both the number of objects and the number of features that can be accurately maintained at a given time (e.g. Hardman & Cowan, 2015; Oberauer & Eichenberger, 2013; Vergauwe & Cowan, 2015; Wheeler & Treisman, 2002).
http://dx.doi.org/10.1016/j.actpsy.2016.04.004 0001-6918/© 2016 Published by Elsevier B.V.
Support for the object-driven view rests on findings from the change detection paradigm. In this paradigm, participants are presented with two consecutive displays separated by either a blank display or a mask (colored shapes are a favorite type of stimulus). The two displays are either identical, or they differ in one item; participants indicate whether or not a change has occurred. The typical result is that accuracy is high for 3 to 4 items, but declines sharply at larger set sizes (e.g. Cowan, 2001; Luck & Vogel, 1997; Vogel et al., 2001; Wheeler & Treisman, 2002). Research that supports the feature-driven view, on the other hand, typically uses a reconstruction paradigm. In this paradigm, features of a more continuous nature are used (e.g., shades of green, line orientations, Gabor patches, and the like), and participants have to reconstruct one or more items from the memory array by ‘dialing in’ their response (Bays & Husain, 2008; Bays et al., 2011; Wilken & Ma, 2004). Finally, objects-and-features studies tend to use the changedetection paradigm, now manipulating feature characteristics, such as complexity (Alvarez & Cavanagh, 2004) or the number of withinobject features (Oberauer & Eichenberger, 2013) in addition to the number of objects. Some recent research has made it increasingly clear that the strict object-driven and feature-driven views are too restrictive. Notably, at least four papers (all using the change detection paradigm) have provided evidence for the position that people can selectively encode, store, and/or retrieve either features or objects in VSTM, depending on the requirements of the task. These studies thus directly refute the notion that memory representations consist only of objects, or only of unbound features. The first study to demonstrate selectivity was by
46
S. Geigerman et al. / Acta Psychologica 167 (2016) 45–51
Woodman and Vogel (2008). In their experiment, participants were shown colored bars in different orientations (their Expt. 1) or colored objects of different shapes (their Expt. 2); they were instructed to remember either color or orientation or both (their Expt. 1), or either color or shape or both (their Expt. 2). The main result was that memory was better for the color-feature alone condition than for the conditions where color was combined with either orientation or shape. This strongly suggests that when participants are instructed to hold on to only a single feature, they can do so. Kondo and Saiki (2012) obtained a similar result. They asked their subjects to pay attention to the conjunction of two of three features of stimuli varying in color, shape, and location (color-shape, shape-location, or color-location combinations). Subjects were able to block interference from the task-irrelevant feature (only, however, if it was shape or color, not if it was location), suggesting that they were able to selectively filter out some types of information (viz., color or shape) as required. Morey, Guérard, and Tremblay (2013) compared memory for either the color or the letter identity of colored letters under three conditions: a reinstated stimulus (e.g., if the display contained a red ‘M’ and a blue ‘P’, a letter test match could be a red ‘m’); a recombined stimulus (in the example, a letter test match could be a blue ‘m’), or a correct feature bound with an extralist feature (in the example, a green ‘m’). They found no difference in response times between these three conditions, suggesting that features were not obligatorily bound into objects – if they were, people would be faster when the whole stimulus (the object) was reinstated. Instead, subjects seemed to only retrieve the particular type of feature necessary for accurate performance. Finally, Vergauwe and Cowan (2015) found that the representation of colored shapes could be biased towards a whole-object or a separate-feature representation simply by explicitly instructing the subjects to do so. There are two possible explanations for these findings: (a) multiple representations are possible – that is, the objects-and-features driven view is correct – or (b) people can flexibly adjust representations during encoding, favoring one type over the other. If the former explanation were true, then these results further suggest that participants can bias their retrieval efforts towards one specific aspect of the representation. In the present set of three experiments, we aimed to disentangle these two possibilities by investigating the dynamics of retrieval from VSTM. We measured cumulative response-time (RT) distributions for the retrieval of whole objects, and compared these to theorized distributions derived from expectations about the nature of the VSTM representations (more detail on these predictions follows below); these theorized distributions were based on the empirical distributions for single features. (Cumulative RT distributions represent the probability that an RT is less than or equal to some specific time.)
Fig. 1. The 36 colored shapes used in Experiments 1, 2, and 3.
The basic paradigm in our experiments is simple. We showed participants a memory set of three two-featured objects (colored shapes); after a delay, they were probed with a single comparison item – either an individual feature (a shapeless color or a colorless shape, tested in different sessions) or a compound stimulus, that is, the whole object. The crucial manipulation involved the compound-object comparisons. In our first experiment, we forced participants to process the object as a conjunction of features by presenting them with recombinant lures (i.e., objects composed of features drawn from different objects in the memory set). In our second experiment, we used extralist lures, that is, lures that were constructed from a shape and a color neither of which was presented in the memory set. In this instance, binding of features to objects is completely unnecessary. In our third experiment, both types of lures were combined in an unpredictable manner. The advantage of recording cumulative RT distributions is that distinct predictions can be made for a feature representation versus a compound-object representation. Experiment 2, with its extralist lures, is the simplest case: In this experiment, we probed memory with a stimulus that is built from two features that were either both present (match probe) or both absent (mismatch probe) in the memory display. If subjects keep features stored independently, a response can be emitted as soon as either of the features can be recognized or rejected. For instance, if the display contained a blue square, a red spiral, and a yellow star, and the match probe is a red spiral, the subject can respond as soon as either the color red or the spiral shape is matched with the memory representation. Conversely, if the mismatch probe is a green disc, the subject can respond as soon as it is clear that neither the color green nor the disc shape match the memory representation. The expected response time can then be modeled as a horse race between the detection times for each of the features – shape or color – separately; within psychology, this model is known as the race model inequality (Miller, 1982; Ulrich, Miller, & Schroeder, 2007). In Experiment 1, the mismatch probe is a recombination of two features from the memory display. (Returning to the previous example, a blue star would be an instance of a recombinant mismatch probe.) If features were kept separate, a response would have to wait at least until both features have been retrieved; it can therefore be no faster than the detection time for the feature that is retrieved most slowly. We are not aware of such a model within psychology, but it reminded us of the process by which the assembly line in a restaurant kitchen operates: A dish can be served only when all its components are ready, and thus the limiting factor is the component that takes the longest time to prepare. Therefore, we will label this model the ‘kitchen-line model’. Both models thus make clear mathematical predictions for the cumulative RT distribution of the compound mismatch probes, based on the cumulative RT distribution of the single-feature mismatch probes. We would expect to observe data in line with the race model inequality in Experiment 2, signifying that features are kept separate; for Experiment 1, we expect that the data will be considerably faster than the kitchen line model prediction if the whole probe was matched to a compound-object representation. The critical comparison is in Experiment 3, where we randomly intermingled recombinant lures (as in Expt. 1) and extralist lures (as in Expt. 2). In this experiment (as in Expt. 1) subjects would need to maintain bound-object representations if they wanted to be able to respond to the recombinant probes correctly. The informative data point would be mismatch probes. If participants would maintain bound-representations only, and extract features from them as needed for the extralist probes, then the results for mismatch probes in Experiment 3 should look like those from Experiment 1 for both types of probes, because the advantage of keeping unbound features available (as in Expt. 2) would have disappeared. If, however, both feature and object representations were to coexist, the recombinant lures would yield results similar to those of Experiment 1, but the extralist lures would conform to those of Experiment 2, because now subjects would have access to unbound features as well.
S. Geigerman et al. / Acta Psychologica 167 (2016) 45–51
47
Fig. 2. Sequence of events in a mismatch trial (left: Experiment 1, middle: Experiment 2, right: Experiment 3) in whole object conditions. The example trial from experiment 3 shows a mismatch probe which is a recombination of an intra-list color and an intra-list shape.
2. Method 2.1. Subjects Twelve college students between the ages of 18 and 30 (mean age 19.94 years) participated in each experiment (Expt. 1: 7 women; Expt. 2: 3 women; Expt. 3: 7 women). 2.2. Stimuli Stimuli were 36 color-shape combinations (Fig. 1), projected on a light gray background. The six shapes were adapted from the stimulus set used by Wheeler and Treisman (2002), designed to be difficult to label. The six colors (default Microsoft Paint palette) were red, blue, green, yellow, pink and brown. The memory sets consisted of three stimuli presented in a random subset of three cells of a 2 × 2 invisible matrix at the center of the monitor, subtending 1.5 degrees of visual angle. 2.3. Procedure A trial (Fig. 2) started with a 200 ms central fixation cross, followed by a 500 ms blank interval. Next a three-digit number was presented in font Courier New, size 18 at the center of the screen for 1000 ms. The participant recited this number aloud, as individual digits, without pausing between successive repetitions, until a response for that particular trial was emitted. This articulatory suppression task was imposed to interfere with verbal encoding of the stimuli. A memory set of three stimuli appeared 1000 ms after the 3-digit number. The stimuli were chosen randomly from the pool of 36 with the restriction that each of the color and shape members of the array should be different from one another. Stimulus presentation time was 754 ms, to ensure adequate encoding. The stimuli were
immediately masked by a 200 ms pattern mask composed of colored curved lines in a mixture of the 6 colors used in the stimulus color palette. A single central probe1 followed the mask; it stayed onscreen until the participant pressed either the right arrow key to indicate a match, or the left arrow key to indicate a mismatch. At this point, the participant could stop repeating the three-digit number. To check on compliance with the articulatory suppression task, all sessions were tape recorded with full knowledge of the participant. Examples of probes are provided in Fig. 3. The nature of the probe differed across the three conditions within each experiment. In the single-feature-color condition, the probe was a colored cloud of pixels; the color was either a color presented in the memory set (match probe) or one of the six possible colors, but not one that was presented in the memory set (mismatch probe). In the single-feature-shape condition, probes consisted of a gray shape; the shape was either a shape presented in the memory set (match probe) or one of the six possible shapes, but not one that was presented in the memory set (mismatch probe). Match probes for the whole-object conditions simply reinstated one of the stimuli from the memory set. Experiment 1 used recombinant mismatch probes (mis-pairing a memory-list shape with a memory-list color); Experiment 2 used
1 One important but somewhat neglected finding from the flexible-resource research program is that object files are originally bound by attention, and that these representations need time to be transferred into the actual memory store. Thus, if the memory display is followed by a test display which reinstates all objects in their original locations, performance is better than when a single centrally located probe is used, but only when the delay between memory and test display is short – less than about a second (e.g., Treisman & Zhang, 2006). This result suggests that a clear distinction needs to be made between fleeting storage in a perceptual and attentional store, and more lasting storage into ‘true’ VSTM (Logie, Brockmole, & Jaswal, 2011). Therefore, to avoid confusion between perceptual/attentional encoding and encoding into ‘true’, location-independent memory, our experiments all used a single, centrally located probe (a procedure also used by Morey et al., 2013).
48
S. Geigerman et al. / Acta Psychologica 167 (2016) 45–51
mismatch probes combining two extralist features. Experiment 3 used both types of mismatch probes (recombinant and with two extralist features) randomly. Satisfactory performance in Experiment 1, therefore, required binding of features into objects. In Experiment 2, probes could be answered correctly based on either feature alone. Experiment 3 required participants to bind features into objects throughout the experiment if they wanted to maintain high levels of performance. In each of the three experiments, participants were tested in a single session. Experiment 1 had 90 trials for each of the three conditions. Experiment 2 had 120 trials (to allow an equal number of single-feature trials for color and shape to be compared with an equal number of whole-object trials). Experiment 3 consisted of 360 trials, out which 180 were single-feature trials, and the rest were whole-object trials. Half of the whole-object trials used recombinant probes, and the other half used extralist probes. 2.4. Model fitting and testing As explained in the Introduction, we fit two models to the data: the race model inequality (e.g. Miller, 1982; Ulrich et al., 2007) and the kitchen-line model. Retrieval benefits might be expected when whole-object probes can be answered on the basis of a single feature (as is the case in extralist feature probes, as in Expt. 2), but only if retrieval of each of the features occurs independently – that is, when representations consist of loose features, not bound objects. In that case, completion time for a wholeobject match probe consisting of two features can be predicted from the completion times of the individual features using a race model. Under the race model inequality, the two features of the whole-object
probe are retrieved independently; the first one to be retrieved successfully ‘wins’, and its retrieval time determines RT: RTnovelfeature probe ¼ min RTcolor ; RTshape : Under this scenario of parallel independent feature channels, response time to a whole object would on average be faster than that to any of the individual features. The best way to evaluate these predictions is at the level of the cumulative RT distribution (Ulrich et al., 2007). If the observations conform to the race model inequality predictions, we can conclude that the individual features are indeed independently accessible. If the cumulative distribution of the whole-object condition, however, were to be slower than expected under the race model inequality, we would conclude that features are not racing independently, and that the item's representation is likely one where features are bound into objects. Conversely, recombinant probes, which consist of rearranged memory-set features (as in Expt. 1), would benefit from whole-object representations, because at the time of retrieval whole-object probes could be matched directly to whole-object representations. If features of the memory representation were kept separate, memory matching would by necessity involve a comparison of each probe feature with the corresponding features of the representations (i.e., the color of the probe is matched to the colors of the representations; the shape of the probe is matched to the shapes of the representations), and the matching process would terminate only when both features are matched and, conceivably, some form of binding tag is retrieved as well. This, then, follows a kitchen-line model for whole-object match probes – like in restaurants, where a dish can be served only when all its components are finished: RTwhole object ≥ max RTcolor ; RTshape :
Fig. 3. Examples of match and mismatch probes in Experiments 1, 2, and 3.
S. Geigerman et al. / Acta Psychologica 167 (2016) 45–51
(The ‘greater-than’ part of the equation is to allow extra time for binding information to be processed alongside the feature information.) Here, again, the difference between the predictions and the observations is useful. If the observations follow the predictions of the kitchen-line model, whole-object comparisons are likely driven by the same feature information as single-feature comparisons. If, however, the observations are faster than predicted by the kitchenline model, the suggestion is that whole-object comparisons are done on the basis of a whole-object match, rather than on the basis of its constituent features. To determine the expected cumulative RT distributions for the kitchen-line model and the race model inequality, we adopted a Monte Carlo approach, recording minimum and maximum values from each individual's single-feature RT distributions through repeated sampling (500 times) with replacement. Specifically, for each individual, we drew a random value from the observed RT distribution of each single-feature condition (one value for color, one for shape); the slowest of these two RTs was used as the expected value under the kitchen-line model. The fastest of the pair was used as the expected value for the race model inequality. For the purposes of statistical testing, the resulting distributions of expected values were summarized by deciles within each experiment. The fit of a model was evaluated across participants at each decile using two-tailed paired samples t-tests, treating the predicted and the empirically obtained whole object response times for each participant as a pair (as in Ulrich et al., 2007).
49
3. Results Fig. 4 shows the cumulative RT distributions for the three experiments (only RTs for correct answers were used), separated out by match and mismatch probes. The whole-object model predictions are overlaid on the data – the kitchen-line model for Experiment 1, the race model inequality for Experiment 2, and both these models for Experiment 3. In the interest of readability of the graphs, we have included error bars for the whole-object conditions only; these error bars designate 95% confidence intervals, and thus allow us to directly evaluate how the whole-object conditions compare to the model predictions. For match probes in Experiment 1, the observed RTs were almost uniformly faster than predicted from the kitchen-line model, t (11) ranges between −1.74 and −3.67, p b 0.05 for the nine fastest deciles. The difference is rather large: Across all deciles, subjects were, on average, 159 ms faster (average SD over deciles = 180 ms) than predicted by the kitchen-line model in the whole-object condition. For mismatch probes, we obtained a similar result – on average and across deciles, subjects were 196 ms faster (average SD over deciles = 270 ms) in the whole-object condition than in the kitchen-line predictions – this difference was significant for the nine fastest deciles; t (11)-values ranged from −1.31 to −4.01. The observed data for the match probes in Experiment 2 conform quite well to the predictions from the race model inequality, as can be seen in the figure; t-values for paired RT differences (M = −56 ms, average SD over deciles = 234 ms, signifying that subjects are nominally
Fig. 4. Comparison between cumulative reaction time distributions observed in Experiments 1, 2, and 3 and corresponding model-based predictions for match and mismatch probes; error bars for whole-object conditions are 95% confidence intervals.
50
S. Geigerman et al. / Acta Psychologica 167 (2016) 45–51
actually a bit faster than predicted under the race model) range between − 1.19 and 0.61, all ns. The observed whole-object mismatch data were faster than the predictions from the race model inequality: The mean of the paired differences at each decile was −73 ms (average SD over all deciles = 92 ms) for mismatch (whole object B) probes. The t(11)-values ranged from −3.62 to 0.92, significant for all but two the two slowest deciles. In Experiment 3, mismatch probes become the truly informative data point, as explained above. That is, if participants would maintain bound-representations only, and extract features from them as needed for the extralist probes, then the results for mismatch probes in Experiment 3 should look like those from Experiment 1 for both types of probes, because the advantage of keeping unbound features available (as in Expt. 2) would have disappeared. If, however, both feature and object representations were to coexist, the recombinant lures would yield results similar to those of Experiment 1, but the extralist lures would conform to those of Experiment 2, because now subjects would have access to unbound features as well. For reason of completeness, we also analyzed the match probes. For the test against the kitchenline model, the observed RTs for match probes were consistently and significantly faster at all deciles from the model prediction with a mean paired difference of − 269 ms (average SD over all deciles = 259 ms), t (11) ranges between −2.50 and −4.94, all p b 0.05. Comparison of RTs from the whole object condition with the predictions from the race model inequality yielded t(11)-values ranging between − 0.87 and 2.74, significant only for the three slowest deciles, where the whole-object data were significantly slower than predicted under the race model inequality. The mean paired difference across deciles was 48 ms (average SD over all deciles = 159 ms). Turning to mismatch probes, the observed RTs in the cumulative distribution function for recombinant probes were faster than predicted by the kitchen-line model (average difference −187 ms, average SD over all deciles = 459), significant for eight deciles, with t(11) values ranging from − 1.73 to − 2.64. The cumulative RT distribution for extralist probes, however, conformed to the race model inequality predictions (average difference 30 ms, average SD over all deciles = 213); t (11) values ranged between −0.45 and 1.63, p N 0.05 for all deciles. 4. Discussion The current study fits in the debate about the nature of VSTM representations – are objects stored as objects, or as a collection of their constituent features? Previous research (Kondo & Saiki, 2012; Morey et al., 2013; Vergauwe & Cowan, 2015; Woodman & Vogel, 2008) has shown that participants can, to some extent, access either feature representations or whole-object representations in VSTM, depending on which type of representation is advantageous for the task, or depending on which type of representation they are instructed to use. The question we investigated is the underlying reason for this apparent flexibility: Is this flexibility due to differential encoding, that is, do participants, when confronted with a memory set, flexibly adjust what aspect of a stimulus to encode, thus favoring one type over the other (a position taken by, amongst other, Morey et al., 2013, and Vergauwe & Cowan, 2015), or are objects simply stored in both formats, objects and features (a position taken by, amongst others, Hardman & Cowan, 2015; Oberauer & Eichenberger, 2013; Vergauwe & Cowan, 2015; Wheeler & Treisman, 2002)? To answer this question, we presented our participants with colored shapes, and examined the retrieval dynamics (viz., cumulative RT distributions) for individual features (colors or shapes) and for whole objects (colored shapes). Our manipulation was to bias the participants towards one type of representation over the other by changing the nature of the whole-object mismatch probe. In Experiment 1, the probe was a combination of two features from the memory list; in Experiment 2, the probe consisted wholly of features not presented in the memory list; in Experiment 3, we mixed the two types of probe in a random
fashion. Experiment 1 should bias representations towards whole objects. If participants did not use whole-object representations, but independent features, their response would be delayed until information from both features has been retrieved – an expectation we dubbed the kitchen-line model. Response times faster than those predicted by the kitchen-line model would suggest that features are not being retrieved independently, but synergistically, that is, as bound representations. Experiment 2 should bias representations towards independent features, and the data should confirm to the well-known race model inequality (Miller, 1982), with a statistical facilitation effect. Crucially, Experiment 3 was designed to arbitrate between the two positions. In this experiment, the participant should be biased towards whole-object representations (otherwise, recombinant mismatch probes cannot be responded to adequately). If this then leads the participants to encode and store only whole-object representations, the data would look like those for Experiment 1. If, however, both representations were to coexist, the nature of the probe would matter, and the results for recombinant mismatch probes should look like those from Experiment 1; those for extralist mismatch probes should look like those from Experiment 2. The figures for the match data clearly show the anticipated statistical facilitation (for the race model inequality) and penalty (for the kitchenline model) compared to the single-feature cumulative RT functions.: The function indicating predictions from the kitchen-line model lies considerably to the right of the slower single-feature-shape function (in Experiment 1), and the function standing for predictions from the race model inequality lies considerably to the left of the faster singlefeature-color function (in Experiment 2). For Experiment 3, the kitchen line model and the race model bound the observed data: The race model function is significantly on the left from the faster single-feature-color function and the kitchen line function is on the right from the slower single-feature-shape function. For match probes in Experiment 1, the observed RTs were uniformly faster than predicted from the kitchen-line model. This suggests that participants in Experiment 1, as expected, used a compound, bound representation of colors and shapes while responding in the whole-object condition. The match-probe data for Experiment 2 conform quite well to the predictions from the race model inequality. These data suggest, again as expected, that in Experiment 2 individual features – colors and shapes – rather than compound representations were matched to the memory set.2 Some unexpected support for this notion comes from the mismatch stimuli, which were answered more quickly than expected under the race model inequality; the difference was 62 ms. One possible explanation is that participants set a more liberal response criterion for responding on the basis of a single feature in the wholeobject condition, which would indicates that the cognitive system ‘knows’ it can utilize evidence from a single feature, and is willing to do so, and place a sharper bet on the accumulating evidence. In Experiment 3, the recombinant mismatch probes were answered much faster than the kitchen-line model predicted, as in Experiment 1, suggesting the presence of whole-object representations. The cumulative RT distribution for extralist mismatch probes, however, conformed to the race model inequality predictions. This suggests that for this type of whole-object probes, participants used a comparison process based on independent features, as in Experiment 2. We note here that the same conclusion can be drawn from the qualitative pattern of the data, without recourse to explicit modeling. That is, in Experiment 1, the line for whole-object condition mismatch probes (i.e., the line for recombinant mismatch probes) is located right between the color and shape lines – this is also the case for the recombinant mismatch probe
2 One reviewer pointed out, correctly, that we are affirming the null hypothesis here. Our data do conform to the race model inequality, but this does not necessarily imply that the race model inequality is the correct model for these data. It is hard to imagine, however, what the alternative would be, that is, what different mechanism could produce a fast whole-object condition that would, statistically and qualitatively, conform so nicely to the prediction of independent memory traces inherent in the race model inequality.
S. Geigerman et al. / Acta Psychologica 167 (2016) 45–51
line for Experiment 3. Likewise, the pattern for Experiment 2, where responses in the whole-object condition (which uses extralist probes) are much faster than responses for either color or shape, is repeated for the extralist probes in Experiment 3. Taken together, Experiments 1 and 2 support the notion that participants had access to memory representations that differed across tasks. When the task demands would be best met by the representation of individual features (Expt. 2), individual features were matched to the memory probe; when binding was necessary (Expt. 1), whole-object matching occurred. The results from Experiment 3 tell us that this result is not indicative of a flexible, selective system, but rather that the two types of representations coexist. Note that in Experiment 3 the two types of probes were intermingled randomly, so that participants could not strategically prepare their mode of encoding. This finding strongly suggests that the representation used for retrieval is dependent not on any strategic encoding on the part of the participant, but simply on the nature of the individual retrieval probes. This in turn suggests that both whole-object and separate-feature representations are available and accessible in VSTM. We note that this result, clearly in favor of the coexistent featureand-objects view, does not imply that both representations are also and necessarily encoded simultaneously. In fact, experiments restricting encoding time either by manipulating presentation time or by using transcranial magnetic stimulation suggest that object representations are available to the memory system a little (about 100 ms) later than feature representations (Braet & Humphreys, 2009; Oberauer & Eichenberger, 2013), suggesting the existence of a time-consuming binding process (e.g., Wheeler & Treisman, 2002). Our results do suggest that after binding is completed, the constituent feature representations are still available for retrieval, without going through an unbinding process. If an unbinding process (sometimes called the features-fromobjects view; Wheeler & Treisman, 2002) were operating, the results from Experiment 3 should then look like those from Experiment 1 for both types of probes, because the advantage of keeping unbound features available (as in Expt. 2) would have disappeared. This was not the case. References Alvarez, G. A., & Cavanagh, P. (2004). The capacity of visual short-term memory is set by both visual information load and number of objects. Psychological Science, 15, 106–111.
51
Baddeley, A. D., & Hitch, G. J. (1974). Working memory. In G. A. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory. Vol. 8. (pp. 47–89). New York, NY: Academic Press. Bays, P. M., & Husain, M. (2008). Dynamic shifts of limited working memory resources in human vision. Science, 321, 851–854. Bays, P. M., Catalao, R. F. G., & Husain, M. (2009). The precision of visual working memory is set by allocation of a shared resource. Journal of Vision, 9, 1–11. Bays, P. M., Wu, E. Y., & Husain, M. (2011). Storage and binding of object features in visual working memory. Neuropsychologia, 49, 1622–1631. Braet, W., & Humphreys, G. W. (2009). The role of reentrant processes in feature binding: Evidence from neuropsychology and TMS on late onset illusory conjunctions. Visual Cognition, 17, 25–47. Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87–114. Hardman, K., & Cowan, N. (2015). Remembering complex objects in visual working memory: Do capacity limits restrict objects or features? Journal of Experimental Psychology: Learning, Memory, and Cognition, 41, 325–347. Kondo, A., & Saiki, J. (2012). Feature-specific encoding flexibility in visual working memory. PloS One, 7, e50962. Lee, D., & Chun, M. M. (2001). What are the units of visual short-term memory: Objects or spatial locations? Perception & Psychophysics, 63, 253–257. Logie, R. H., Brockmole, J. R., & Jaswal, S. (2011). Feature binding in visual short-term memory is unaffected by task-irrelevant changes of location, shape, and color. Memory & Cognition, 39, 24–36. Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature, 390, 279–281. Miller, J. O. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14, 247–279. Morey, C. C., Guérard, K., & Tremblay, S. (2013). Neither separate nor equivalent: Relationships between feature representations within bound objects. Acta Psychologica, 144, 279–290. Oberauer, K., & Eichenberger, S. (2013). Visual working memory declines when more features be remembered for each object. Memory & Cognition, 41, 1212–1227. Suchow, J., Fougnie, D., Brady, T. F., & Alvarez, G. A. (2014). Terms of the debate on the format and structure of visual memory. Attention, Perception & Psychophysics, 76, 2071–2079. Treisman, A., & Zhang, W. (2006). Location and binding in visual working memory. Memory & Cognition, 34, 1704–1719. Ulrich, R., Miller, J. O., & Schroeder, H. (2007). Testing the race model inequality: An algorithm and computer programs. Behavior Research Methods & Instrumentation, 39, 291–302. Vergauwe, E., & Cowan, N. (2015). Working memory units are all in your head: Factors that influence whether features or objects are the favored units. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41, 1404–1406. Vogel, E. K., Woodman, G. F., & Luck, S. J. (2001). Storage of features, conjunctions, and objects in visual working memory. Journal of Experimental Psychology: Human Perception and Performance, 27, 92–114. Wheeler, M. E., & Treisman, A. M. (2002). Binding in short-term visual memory. Journal of Experimental Psychology: General, 131, 48–64. Wilken, P., & Ma, W. J. (2004). A detection theory account of change detection. Journal of Vision, 4, 1120–1135. Woodman, G. F., & Vogel, E. K. (2008). Selective storage and maintenance of an object's features in visual working memory. Psychonomic Bulletin & Review, 15, 223–229.
Contents lists available at ScienceDirect
Acta Psychologica journal homepage: www.elsevier.com/locate/actpsy
To bind or not to bind, that's the wrong question: Features and objects coexist in visual short-term memory Shriradha Geigerman a, Paul Verhaeghen a, John Cerella b a b
Georgia Institute of Technology, United States Syracuse University, United States
a r t i c l e
i n f o
Article history: Received 17 March 2015 Received in revised form 11 April 2016 Accepted 13 April 2016 Available online xxxx Keywords: Working memory Visual short-term memory Binding Objects Features
a b s t r a c t In three experiments, we investigated whether features and whole-objects can be represented simultaneously in visual short-term memory (VSTM). Participants were presented with a memory set of colored shapes; we probed either for the constituent features or for the whole object, and analyzed retrieval dynamics (cumulative response time distributions). In our first experiment, we used whole-object probes that recombined features from the memory display; we found that subjects' data conformed to a kitchen-line model, showing that they used whole-object representations for the matching process. In the second experiment, we encouraged independent-feature representations by using probes that used features not present in the memory display; subjects' data conformed to the race-model inequality, showing that they used independent-feature representations for the matching process. In a final experiment, we used both types of probes; subjects now used both types of representations, depending on the nature of the probe. Combined, our three experiments suggest that both feature and whole-object representations can coexist in VSTM. © 2016 Published by Elsevier B.V.
1. Introduction Visual short-term memory (VSTM) stores visual information for a brief period of time (a few seconds) in the interest of ongoing tasks (Baddeley & Hitch, 1974). There is considerable consensus that a severe capacity limit exists on the number of items it can represent – typically no more than 3 or 4 at a time (Cowan, 2001; Luck & Vogel, 1997). A question central to our understanding of the observed capacity limit within VSTM is qualitative: What is the format of representation of information in VSTM (e.g., Suchow, Fougnie, Brady, & Alvarez, 2014) that determines its capacity limits? There are three points of view on this matter. According to the object-driven view, VSTM stores information in the form of integrated objects; capacity is then measured as the number of (multi-feature) objects that can be maintained simultaneously (Lee & Chun, 2001; Luck & Vogel, 1997; Vogel, Woodman, & Luck, 2001). Alternatively, the feature-driven view of VSTM posits that capacity is influenced by the total number of features (such as colors, shapes, or orientations) that constitute a visual scene (Bays, Catalao, & Husain, 2009; Bays, Wu, & Husain, 2011; Wheeler & Treisman, 2002), independent of the number of objects across which those features are distributed. The third view is a hybrid: The object-and-features-driven view proposes dual representation of both objects and features in VSTM. According to this view, the VSTM capacity limit is a function of both the number of objects and the number of features that can be accurately maintained at a given time (e.g. Hardman & Cowan, 2015; Oberauer & Eichenberger, 2013; Vergauwe & Cowan, 2015; Wheeler & Treisman, 2002).
http://dx.doi.org/10.1016/j.actpsy.2016.04.004 0001-6918/© 2016 Published by Elsevier B.V.
Support for the object-driven view rests on findings from the change detection paradigm. In this paradigm, participants are presented with two consecutive displays separated by either a blank display or a mask (colored shapes are a favorite type of stimulus). The two displays are either identical, or they differ in one item; participants indicate whether or not a change has occurred. The typical result is that accuracy is high for 3 to 4 items, but declines sharply at larger set sizes (e.g. Cowan, 2001; Luck & Vogel, 1997; Vogel et al., 2001; Wheeler & Treisman, 2002). Research that supports the feature-driven view, on the other hand, typically uses a reconstruction paradigm. In this paradigm, features of a more continuous nature are used (e.g., shades of green, line orientations, Gabor patches, and the like), and participants have to reconstruct one or more items from the memory array by ‘dialing in’ their response (Bays & Husain, 2008; Bays et al., 2011; Wilken & Ma, 2004). Finally, objects-and-features studies tend to use the changedetection paradigm, now manipulating feature characteristics, such as complexity (Alvarez & Cavanagh, 2004) or the number of withinobject features (Oberauer & Eichenberger, 2013) in addition to the number of objects. Some recent research has made it increasingly clear that the strict object-driven and feature-driven views are too restrictive. Notably, at least four papers (all using the change detection paradigm) have provided evidence for the position that people can selectively encode, store, and/or retrieve either features or objects in VSTM, depending on the requirements of the task. These studies thus directly refute the notion that memory representations consist only of objects, or only of unbound features. The first study to demonstrate selectivity was by
46
S. Geigerman et al. / Acta Psychologica 167 (2016) 45–51
Woodman and Vogel (2008). In their experiment, participants were shown colored bars in different orientations (their Expt. 1) or colored objects of different shapes (their Expt. 2); they were instructed to remember either color or orientation or both (their Expt. 1), or either color or shape or both (their Expt. 2). The main result was that memory was better for the color-feature alone condition than for the conditions where color was combined with either orientation or shape. This strongly suggests that when participants are instructed to hold on to only a single feature, they can do so. Kondo and Saiki (2012) obtained a similar result. They asked their subjects to pay attention to the conjunction of two of three features of stimuli varying in color, shape, and location (color-shape, shape-location, or color-location combinations). Subjects were able to block interference from the task-irrelevant feature (only, however, if it was shape or color, not if it was location), suggesting that they were able to selectively filter out some types of information (viz., color or shape) as required. Morey, Guérard, and Tremblay (2013) compared memory for either the color or the letter identity of colored letters under three conditions: a reinstated stimulus (e.g., if the display contained a red ‘M’ and a blue ‘P’, a letter test match could be a red ‘m’); a recombined stimulus (in the example, a letter test match could be a blue ‘m’), or a correct feature bound with an extralist feature (in the example, a green ‘m’). They found no difference in response times between these three conditions, suggesting that features were not obligatorily bound into objects – if they were, people would be faster when the whole stimulus (the object) was reinstated. Instead, subjects seemed to only retrieve the particular type of feature necessary for accurate performance. Finally, Vergauwe and Cowan (2015) found that the representation of colored shapes could be biased towards a whole-object or a separate-feature representation simply by explicitly instructing the subjects to do so. There are two possible explanations for these findings: (a) multiple representations are possible – that is, the objects-and-features driven view is correct – or (b) people can flexibly adjust representations during encoding, favoring one type over the other. If the former explanation were true, then these results further suggest that participants can bias their retrieval efforts towards one specific aspect of the representation. In the present set of three experiments, we aimed to disentangle these two possibilities by investigating the dynamics of retrieval from VSTM. We measured cumulative response-time (RT) distributions for the retrieval of whole objects, and compared these to theorized distributions derived from expectations about the nature of the VSTM representations (more detail on these predictions follows below); these theorized distributions were based on the empirical distributions for single features. (Cumulative RT distributions represent the probability that an RT is less than or equal to some specific time.)
Fig. 1. The 36 colored shapes used in Experiments 1, 2, and 3.
The basic paradigm in our experiments is simple. We showed participants a memory set of three two-featured objects (colored shapes); after a delay, they were probed with a single comparison item – either an individual feature (a shapeless color or a colorless shape, tested in different sessions) or a compound stimulus, that is, the whole object. The crucial manipulation involved the compound-object comparisons. In our first experiment, we forced participants to process the object as a conjunction of features by presenting them with recombinant lures (i.e., objects composed of features drawn from different objects in the memory set). In our second experiment, we used extralist lures, that is, lures that were constructed from a shape and a color neither of which was presented in the memory set. In this instance, binding of features to objects is completely unnecessary. In our third experiment, both types of lures were combined in an unpredictable manner. The advantage of recording cumulative RT distributions is that distinct predictions can be made for a feature representation versus a compound-object representation. Experiment 2, with its extralist lures, is the simplest case: In this experiment, we probed memory with a stimulus that is built from two features that were either both present (match probe) or both absent (mismatch probe) in the memory display. If subjects keep features stored independently, a response can be emitted as soon as either of the features can be recognized or rejected. For instance, if the display contained a blue square, a red spiral, and a yellow star, and the match probe is a red spiral, the subject can respond as soon as either the color red or the spiral shape is matched with the memory representation. Conversely, if the mismatch probe is a green disc, the subject can respond as soon as it is clear that neither the color green nor the disc shape match the memory representation. The expected response time can then be modeled as a horse race between the detection times for each of the features – shape or color – separately; within psychology, this model is known as the race model inequality (Miller, 1982; Ulrich, Miller, & Schroeder, 2007). In Experiment 1, the mismatch probe is a recombination of two features from the memory display. (Returning to the previous example, a blue star would be an instance of a recombinant mismatch probe.) If features were kept separate, a response would have to wait at least until both features have been retrieved; it can therefore be no faster than the detection time for the feature that is retrieved most slowly. We are not aware of such a model within psychology, but it reminded us of the process by which the assembly line in a restaurant kitchen operates: A dish can be served only when all its components are ready, and thus the limiting factor is the component that takes the longest time to prepare. Therefore, we will label this model the ‘kitchen-line model’. Both models thus make clear mathematical predictions for the cumulative RT distribution of the compound mismatch probes, based on the cumulative RT distribution of the single-feature mismatch probes. We would expect to observe data in line with the race model inequality in Experiment 2, signifying that features are kept separate; for Experiment 1, we expect that the data will be considerably faster than the kitchen line model prediction if the whole probe was matched to a compound-object representation. The critical comparison is in Experiment 3, where we randomly intermingled recombinant lures (as in Expt. 1) and extralist lures (as in Expt. 2). In this experiment (as in Expt. 1) subjects would need to maintain bound-object representations if they wanted to be able to respond to the recombinant probes correctly. The informative data point would be mismatch probes. If participants would maintain bound-representations only, and extract features from them as needed for the extralist probes, then the results for mismatch probes in Experiment 3 should look like those from Experiment 1 for both types of probes, because the advantage of keeping unbound features available (as in Expt. 2) would have disappeared. If, however, both feature and object representations were to coexist, the recombinant lures would yield results similar to those of Experiment 1, but the extralist lures would conform to those of Experiment 2, because now subjects would have access to unbound features as well.
S. Geigerman et al. / Acta Psychologica 167 (2016) 45–51
47
Fig. 2. Sequence of events in a mismatch trial (left: Experiment 1, middle: Experiment 2, right: Experiment 3) in whole object conditions. The example trial from experiment 3 shows a mismatch probe which is a recombination of an intra-list color and an intra-list shape.
2. Method 2.1. Subjects Twelve college students between the ages of 18 and 30 (mean age 19.94 years) participated in each experiment (Expt. 1: 7 women; Expt. 2: 3 women; Expt. 3: 7 women). 2.2. Stimuli Stimuli were 36 color-shape combinations (Fig. 1), projected on a light gray background. The six shapes were adapted from the stimulus set used by Wheeler and Treisman (2002), designed to be difficult to label. The six colors (default Microsoft Paint palette) were red, blue, green, yellow, pink and brown. The memory sets consisted of three stimuli presented in a random subset of three cells of a 2 × 2 invisible matrix at the center of the monitor, subtending 1.5 degrees of visual angle. 2.3. Procedure A trial (Fig. 2) started with a 200 ms central fixation cross, followed by a 500 ms blank interval. Next a three-digit number was presented in font Courier New, size 18 at the center of the screen for 1000 ms. The participant recited this number aloud, as individual digits, without pausing between successive repetitions, until a response for that particular trial was emitted. This articulatory suppression task was imposed to interfere with verbal encoding of the stimuli. A memory set of three stimuli appeared 1000 ms after the 3-digit number. The stimuli were chosen randomly from the pool of 36 with the restriction that each of the color and shape members of the array should be different from one another. Stimulus presentation time was 754 ms, to ensure adequate encoding. The stimuli were
immediately masked by a 200 ms pattern mask composed of colored curved lines in a mixture of the 6 colors used in the stimulus color palette. A single central probe1 followed the mask; it stayed onscreen until the participant pressed either the right arrow key to indicate a match, or the left arrow key to indicate a mismatch. At this point, the participant could stop repeating the three-digit number. To check on compliance with the articulatory suppression task, all sessions were tape recorded with full knowledge of the participant. Examples of probes are provided in Fig. 3. The nature of the probe differed across the three conditions within each experiment. In the single-feature-color condition, the probe was a colored cloud of pixels; the color was either a color presented in the memory set (match probe) or one of the six possible colors, but not one that was presented in the memory set (mismatch probe). In the single-feature-shape condition, probes consisted of a gray shape; the shape was either a shape presented in the memory set (match probe) or one of the six possible shapes, but not one that was presented in the memory set (mismatch probe). Match probes for the whole-object conditions simply reinstated one of the stimuli from the memory set. Experiment 1 used recombinant mismatch probes (mis-pairing a memory-list shape with a memory-list color); Experiment 2 used
1 One important but somewhat neglected finding from the flexible-resource research program is that object files are originally bound by attention, and that these representations need time to be transferred into the actual memory store. Thus, if the memory display is followed by a test display which reinstates all objects in their original locations, performance is better than when a single centrally located probe is used, but only when the delay between memory and test display is short – less than about a second (e.g., Treisman & Zhang, 2006). This result suggests that a clear distinction needs to be made between fleeting storage in a perceptual and attentional store, and more lasting storage into ‘true’ VSTM (Logie, Brockmole, & Jaswal, 2011). Therefore, to avoid confusion between perceptual/attentional encoding and encoding into ‘true’, location-independent memory, our experiments all used a single, centrally located probe (a procedure also used by Morey et al., 2013).
48
S. Geigerman et al. / Acta Psychologica 167 (2016) 45–51
mismatch probes combining two extralist features. Experiment 3 used both types of mismatch probes (recombinant and with two extralist features) randomly. Satisfactory performance in Experiment 1, therefore, required binding of features into objects. In Experiment 2, probes could be answered correctly based on either feature alone. Experiment 3 required participants to bind features into objects throughout the experiment if they wanted to maintain high levels of performance. In each of the three experiments, participants were tested in a single session. Experiment 1 had 90 trials for each of the three conditions. Experiment 2 had 120 trials (to allow an equal number of single-feature trials for color and shape to be compared with an equal number of whole-object trials). Experiment 3 consisted of 360 trials, out which 180 were single-feature trials, and the rest were whole-object trials. Half of the whole-object trials used recombinant probes, and the other half used extralist probes. 2.4. Model fitting and testing As explained in the Introduction, we fit two models to the data: the race model inequality (e.g. Miller, 1982; Ulrich et al., 2007) and the kitchen-line model. Retrieval benefits might be expected when whole-object probes can be answered on the basis of a single feature (as is the case in extralist feature probes, as in Expt. 2), but only if retrieval of each of the features occurs independently – that is, when representations consist of loose features, not bound objects. In that case, completion time for a wholeobject match probe consisting of two features can be predicted from the completion times of the individual features using a race model. Under the race model inequality, the two features of the whole-object
probe are retrieved independently; the first one to be retrieved successfully ‘wins’, and its retrieval time determines RT: RTnovelfeature probe ¼ min RTcolor ; RTshape : Under this scenario of parallel independent feature channels, response time to a whole object would on average be faster than that to any of the individual features. The best way to evaluate these predictions is at the level of the cumulative RT distribution (Ulrich et al., 2007). If the observations conform to the race model inequality predictions, we can conclude that the individual features are indeed independently accessible. If the cumulative distribution of the whole-object condition, however, were to be slower than expected under the race model inequality, we would conclude that features are not racing independently, and that the item's representation is likely one where features are bound into objects. Conversely, recombinant probes, which consist of rearranged memory-set features (as in Expt. 1), would benefit from whole-object representations, because at the time of retrieval whole-object probes could be matched directly to whole-object representations. If features of the memory representation were kept separate, memory matching would by necessity involve a comparison of each probe feature with the corresponding features of the representations (i.e., the color of the probe is matched to the colors of the representations; the shape of the probe is matched to the shapes of the representations), and the matching process would terminate only when both features are matched and, conceivably, some form of binding tag is retrieved as well. This, then, follows a kitchen-line model for whole-object match probes – like in restaurants, where a dish can be served only when all its components are finished: RTwhole object ≥ max RTcolor ; RTshape :
Fig. 3. Examples of match and mismatch probes in Experiments 1, 2, and 3.
S. Geigerman et al. / Acta Psychologica 167 (2016) 45–51
(The ‘greater-than’ part of the equation is to allow extra time for binding information to be processed alongside the feature information.) Here, again, the difference between the predictions and the observations is useful. If the observations follow the predictions of the kitchen-line model, whole-object comparisons are likely driven by the same feature information as single-feature comparisons. If, however, the observations are faster than predicted by the kitchenline model, the suggestion is that whole-object comparisons are done on the basis of a whole-object match, rather than on the basis of its constituent features. To determine the expected cumulative RT distributions for the kitchen-line model and the race model inequality, we adopted a Monte Carlo approach, recording minimum and maximum values from each individual's single-feature RT distributions through repeated sampling (500 times) with replacement. Specifically, for each individual, we drew a random value from the observed RT distribution of each single-feature condition (one value for color, one for shape); the slowest of these two RTs was used as the expected value under the kitchen-line model. The fastest of the pair was used as the expected value for the race model inequality. For the purposes of statistical testing, the resulting distributions of expected values were summarized by deciles within each experiment. The fit of a model was evaluated across participants at each decile using two-tailed paired samples t-tests, treating the predicted and the empirically obtained whole object response times for each participant as a pair (as in Ulrich et al., 2007).
49
3. Results Fig. 4 shows the cumulative RT distributions for the three experiments (only RTs for correct answers were used), separated out by match and mismatch probes. The whole-object model predictions are overlaid on the data – the kitchen-line model for Experiment 1, the race model inequality for Experiment 2, and both these models for Experiment 3. In the interest of readability of the graphs, we have included error bars for the whole-object conditions only; these error bars designate 95% confidence intervals, and thus allow us to directly evaluate how the whole-object conditions compare to the model predictions. For match probes in Experiment 1, the observed RTs were almost uniformly faster than predicted from the kitchen-line model, t (11) ranges between −1.74 and −3.67, p b 0.05 for the nine fastest deciles. The difference is rather large: Across all deciles, subjects were, on average, 159 ms faster (average SD over deciles = 180 ms) than predicted by the kitchen-line model in the whole-object condition. For mismatch probes, we obtained a similar result – on average and across deciles, subjects were 196 ms faster (average SD over deciles = 270 ms) in the whole-object condition than in the kitchen-line predictions – this difference was significant for the nine fastest deciles; t (11)-values ranged from −1.31 to −4.01. The observed data for the match probes in Experiment 2 conform quite well to the predictions from the race model inequality, as can be seen in the figure; t-values for paired RT differences (M = −56 ms, average SD over deciles = 234 ms, signifying that subjects are nominally
Fig. 4. Comparison between cumulative reaction time distributions observed in Experiments 1, 2, and 3 and corresponding model-based predictions for match and mismatch probes; error bars for whole-object conditions are 95% confidence intervals.
50
S. Geigerman et al. / Acta Psychologica 167 (2016) 45–51
actually a bit faster than predicted under the race model) range between − 1.19 and 0.61, all ns. The observed whole-object mismatch data were faster than the predictions from the race model inequality: The mean of the paired differences at each decile was −73 ms (average SD over all deciles = 92 ms) for mismatch (whole object B) probes. The t(11)-values ranged from −3.62 to 0.92, significant for all but two the two slowest deciles. In Experiment 3, mismatch probes become the truly informative data point, as explained above. That is, if participants would maintain bound-representations only, and extract features from them as needed for the extralist probes, then the results for mismatch probes in Experiment 3 should look like those from Experiment 1 for both types of probes, because the advantage of keeping unbound features available (as in Expt. 2) would have disappeared. If, however, both feature and object representations were to coexist, the recombinant lures would yield results similar to those of Experiment 1, but the extralist lures would conform to those of Experiment 2, because now subjects would have access to unbound features as well. For reason of completeness, we also analyzed the match probes. For the test against the kitchenline model, the observed RTs for match probes were consistently and significantly faster at all deciles from the model prediction with a mean paired difference of − 269 ms (average SD over all deciles = 259 ms), t (11) ranges between −2.50 and −4.94, all p b 0.05. Comparison of RTs from the whole object condition with the predictions from the race model inequality yielded t(11)-values ranging between − 0.87 and 2.74, significant only for the three slowest deciles, where the whole-object data were significantly slower than predicted under the race model inequality. The mean paired difference across deciles was 48 ms (average SD over all deciles = 159 ms). Turning to mismatch probes, the observed RTs in the cumulative distribution function for recombinant probes were faster than predicted by the kitchen-line model (average difference −187 ms, average SD over all deciles = 459), significant for eight deciles, with t(11) values ranging from − 1.73 to − 2.64. The cumulative RT distribution for extralist probes, however, conformed to the race model inequality predictions (average difference 30 ms, average SD over all deciles = 213); t (11) values ranged between −0.45 and 1.63, p N 0.05 for all deciles. 4. Discussion The current study fits in the debate about the nature of VSTM representations – are objects stored as objects, or as a collection of their constituent features? Previous research (Kondo & Saiki, 2012; Morey et al., 2013; Vergauwe & Cowan, 2015; Woodman & Vogel, 2008) has shown that participants can, to some extent, access either feature representations or whole-object representations in VSTM, depending on which type of representation is advantageous for the task, or depending on which type of representation they are instructed to use. The question we investigated is the underlying reason for this apparent flexibility: Is this flexibility due to differential encoding, that is, do participants, when confronted with a memory set, flexibly adjust what aspect of a stimulus to encode, thus favoring one type over the other (a position taken by, amongst other, Morey et al., 2013, and Vergauwe & Cowan, 2015), or are objects simply stored in both formats, objects and features (a position taken by, amongst others, Hardman & Cowan, 2015; Oberauer & Eichenberger, 2013; Vergauwe & Cowan, 2015; Wheeler & Treisman, 2002)? To answer this question, we presented our participants with colored shapes, and examined the retrieval dynamics (viz., cumulative RT distributions) for individual features (colors or shapes) and for whole objects (colored shapes). Our manipulation was to bias the participants towards one type of representation over the other by changing the nature of the whole-object mismatch probe. In Experiment 1, the probe was a combination of two features from the memory list; in Experiment 2, the probe consisted wholly of features not presented in the memory list; in Experiment 3, we mixed the two types of probe in a random
fashion. Experiment 1 should bias representations towards whole objects. If participants did not use whole-object representations, but independent features, their response would be delayed until information from both features has been retrieved – an expectation we dubbed the kitchen-line model. Response times faster than those predicted by the kitchen-line model would suggest that features are not being retrieved independently, but synergistically, that is, as bound representations. Experiment 2 should bias representations towards independent features, and the data should confirm to the well-known race model inequality (Miller, 1982), with a statistical facilitation effect. Crucially, Experiment 3 was designed to arbitrate between the two positions. In this experiment, the participant should be biased towards whole-object representations (otherwise, recombinant mismatch probes cannot be responded to adequately). If this then leads the participants to encode and store only whole-object representations, the data would look like those for Experiment 1. If, however, both representations were to coexist, the nature of the probe would matter, and the results for recombinant mismatch probes should look like those from Experiment 1; those for extralist mismatch probes should look like those from Experiment 2. The figures for the match data clearly show the anticipated statistical facilitation (for the race model inequality) and penalty (for the kitchenline model) compared to the single-feature cumulative RT functions.: The function indicating predictions from the kitchen-line model lies considerably to the right of the slower single-feature-shape function (in Experiment 1), and the function standing for predictions from the race model inequality lies considerably to the left of the faster singlefeature-color function (in Experiment 2). For Experiment 3, the kitchen line model and the race model bound the observed data: The race model function is significantly on the left from the faster single-feature-color function and the kitchen line function is on the right from the slower single-feature-shape function. For match probes in Experiment 1, the observed RTs were uniformly faster than predicted from the kitchen-line model. This suggests that participants in Experiment 1, as expected, used a compound, bound representation of colors and shapes while responding in the whole-object condition. The match-probe data for Experiment 2 conform quite well to the predictions from the race model inequality. These data suggest, again as expected, that in Experiment 2 individual features – colors and shapes – rather than compound representations were matched to the memory set.2 Some unexpected support for this notion comes from the mismatch stimuli, which were answered more quickly than expected under the race model inequality; the difference was 62 ms. One possible explanation is that participants set a more liberal response criterion for responding on the basis of a single feature in the wholeobject condition, which would indicates that the cognitive system ‘knows’ it can utilize evidence from a single feature, and is willing to do so, and place a sharper bet on the accumulating evidence. In Experiment 3, the recombinant mismatch probes were answered much faster than the kitchen-line model predicted, as in Experiment 1, suggesting the presence of whole-object representations. The cumulative RT distribution for extralist mismatch probes, however, conformed to the race model inequality predictions. This suggests that for this type of whole-object probes, participants used a comparison process based on independent features, as in Experiment 2. We note here that the same conclusion can be drawn from the qualitative pattern of the data, without recourse to explicit modeling. That is, in Experiment 1, the line for whole-object condition mismatch probes (i.e., the line for recombinant mismatch probes) is located right between the color and shape lines – this is also the case for the recombinant mismatch probe
2 One reviewer pointed out, correctly, that we are affirming the null hypothesis here. Our data do conform to the race model inequality, but this does not necessarily imply that the race model inequality is the correct model for these data. It is hard to imagine, however, what the alternative would be, that is, what different mechanism could produce a fast whole-object condition that would, statistically and qualitatively, conform so nicely to the prediction of independent memory traces inherent in the race model inequality.
S. Geigerman et al. / Acta Psychologica 167 (2016) 45–51
line for Experiment 3. Likewise, the pattern for Experiment 2, where responses in the whole-object condition (which uses extralist probes) are much faster than responses for either color or shape, is repeated for the extralist probes in Experiment 3. Taken together, Experiments 1 and 2 support the notion that participants had access to memory representations that differed across tasks. When the task demands would be best met by the representation of individual features (Expt. 2), individual features were matched to the memory probe; when binding was necessary (Expt. 1), whole-object matching occurred. The results from Experiment 3 tell us that this result is not indicative of a flexible, selective system, but rather that the two types of representations coexist. Note that in Experiment 3 the two types of probes were intermingled randomly, so that participants could not strategically prepare their mode of encoding. This finding strongly suggests that the representation used for retrieval is dependent not on any strategic encoding on the part of the participant, but simply on the nature of the individual retrieval probes. This in turn suggests that both whole-object and separate-feature representations are available and accessible in VSTM. We note that this result, clearly in favor of the coexistent featureand-objects view, does not imply that both representations are also and necessarily encoded simultaneously. In fact, experiments restricting encoding time either by manipulating presentation time or by using transcranial magnetic stimulation suggest that object representations are available to the memory system a little (about 100 ms) later than feature representations (Braet & Humphreys, 2009; Oberauer & Eichenberger, 2013), suggesting the existence of a time-consuming binding process (e.g., Wheeler & Treisman, 2002). Our results do suggest that after binding is completed, the constituent feature representations are still available for retrieval, without going through an unbinding process. If an unbinding process (sometimes called the features-fromobjects view; Wheeler & Treisman, 2002) were operating, the results from Experiment 3 should then look like those from Experiment 1 for both types of probes, because the advantage of keeping unbound features available (as in Expt. 2) would have disappeared. This was not the case. References Alvarez, G. A., & Cavanagh, P. (2004). The capacity of visual short-term memory is set by both visual information load and number of objects. Psychological Science, 15, 106–111.
51
Baddeley, A. D., & Hitch, G. J. (1974). Working memory. In G. A. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory. Vol. 8. (pp. 47–89). New York, NY: Academic Press. Bays, P. M., & Husain, M. (2008). Dynamic shifts of limited working memory resources in human vision. Science, 321, 851–854. Bays, P. M., Catalao, R. F. G., & Husain, M. (2009). The precision of visual working memory is set by allocation of a shared resource. Journal of Vision, 9, 1–11. Bays, P. M., Wu, E. Y., & Husain, M. (2011). Storage and binding of object features in visual working memory. Neuropsychologia, 49, 1622–1631. Braet, W., & Humphreys, G. W. (2009). The role of reentrant processes in feature binding: Evidence from neuropsychology and TMS on late onset illusory conjunctions. Visual Cognition, 17, 25–47. Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87–114. Hardman, K., & Cowan, N. (2015). Remembering complex objects in visual working memory: Do capacity limits restrict objects or features? Journal of Experimental Psychology: Learning, Memory, and Cognition, 41, 325–347. Kondo, A., & Saiki, J. (2012). Feature-specific encoding flexibility in visual working memory. PloS One, 7, e50962. Lee, D., & Chun, M. M. (2001). What are the units of visual short-term memory: Objects or spatial locations? Perception & Psychophysics, 63, 253–257. Logie, R. H., Brockmole, J. R., & Jaswal, S. (2011). Feature binding in visual short-term memory is unaffected by task-irrelevant changes of location, shape, and color. Memory & Cognition, 39, 24–36. Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature, 390, 279–281. Miller, J. O. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14, 247–279. Morey, C. C., Guérard, K., & Tremblay, S. (2013). Neither separate nor equivalent: Relationships between feature representations within bound objects. Acta Psychologica, 144, 279–290. Oberauer, K., & Eichenberger, S. (2013). Visual working memory declines when more features be remembered for each object. Memory & Cognition, 41, 1212–1227. Suchow, J., Fougnie, D., Brady, T. F., & Alvarez, G. A. (2014). Terms of the debate on the format and structure of visual memory. Attention, Perception & Psychophysics, 76, 2071–2079. Treisman, A., & Zhang, W. (2006). Location and binding in visual working memory. Memory & Cognition, 34, 1704–1719. Ulrich, R., Miller, J. O., & Schroeder, H. (2007). Testing the race model inequality: An algorithm and computer programs. Behavior Research Methods & Instrumentation, 39, 291–302. Vergauwe, E., & Cowan, N. (2015). Working memory units are all in your head: Factors that influence whether features or objects are the favored units. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41, 1404–1406. Vogel, E. K., Woodman, G. F., & Luck, S. J. (2001). Storage of features, conjunctions, and objects in visual working memory. Journal of Experimental Psychology: Human Perception and Performance, 27, 92–114. Wheeler, M. E., & Treisman, A. M. (2002). Binding in short-term visual memory. Journal of Experimental Psychology: General, 131, 48–64. Wilken, P., & Ma, W. J. (2004). A detection theory account of change detection. Journal of Vision, 4, 1120–1135. Woodman, G. F., & Vogel, E. K. (2008). Selective storage and maintenance of an object's features in visual working memory. Psychonomic Bulletin & Review, 15, 223–229.