- Email: [email protected]
State-dependent learning and suboptimal choice: when starlings prefer long over short delays to food
ANIMAL BEHAVIOUR, 2005, 70, 571–578 doi:10.1016/j.anbehav.2004.12.009
State-dependent learning and suboptimal choice: when starlings prefer long over short delays to food LORENA PO MPILIO & A LEX K ACELNIK
Department of Zoology, University of Oxford (Received 30 July 2004; initial acceptance 3 September 2004; final acceptance 22 December 2004; published online 11 July 2005; MS. number: 8228)
Recent studies have used labels such as ‘work ethics’, ‘sunk costs’ and ‘state-dependent preferences’ for apparent anomalies in animals’ choices. They suggest that preference between options relates to the options’ history, rather than depending exclusively on the expected payoffs. For instance, European starlings, Sturnus vulgaris, trained to obtain identical food rewards from two sources while in two levels of hunger preferred the food source previously associated with higher hunger, regardless of the birds’ state at the time of testing. We extended this experimentally and theoretically by studying starlings choosing between sources that differed not only in history but also in the objective properties (delay until reward) of the payoffs they delivered. Two options (PF and H) were initially presented in single-option sessions when subjects were, respectively, prefed or hungry. While option PF offered a delay until reward of 10 s in all treatments, option H delivered delays of 10, 12.5, 15 and 17.5 s in four treatments. When training was completed, we tested preference between the options. When delays in both options were equal (10 s), the birds strongly preferred H. When delay in H was 17.5 s, the birds were indifferent, with intermediate results for intermediate treatments. Preference was not mediated by disrupted knowledge of the delays. Thus, preferences were driven by past state-dependent gains, rather than by the joint effect of the birds’ state at the time of choice and knowledge of the absolute properties of each alternative, as assumed in state-dependent, path-independent models of optimal choice. Ó 2005 The Association for the Study of Animal Behaviour. Published by Elsevier Ltd. All rights reserved.
Several recent studies of avian decision making have dealt with two related and somewhat counterintuitive observations. The first is that when subjects are initially trained to obtain equally sized rewards after either hard or light work, and are then offered a choice without the work requirement, they prefer the reward previously associated with harder work (Clement et al. 2000; Kacelnik & Marsh 2002; originally reported by Marsh 1999). The second is that when subjects are trained with equal rewards in different states of deprivation, and are then offered a choice, they prefer the reward previously associated with greater deprivation (Marsh et al. 2004). Enhanced preference for stimuli associated with greater need is probably far from restricted to birds: locusts, Locusta migratoria, too prefer a signal (an odour in this case) that has been previously paired with food encountered under a state of greater need, regardless of state at the time of testing (Pompilio 2004; L. Pompilio, A. Kacelnik & S. T.
Correspondence: A. Kacelnik, Department of Zoology, Oxford University, South Parks Road, Oxford OX1 3PS, U.K. (email: alex.kacelnik@ zoo.ox.ac.uk). 0003–3472/04/$30.00/0
Bahmar, unpublished data). Here we discuss the findings on birds, propose some theoretical clarifications, and present an experimental study aimed at establishing the strength of the effect and the relevance of previously proposed theoretical models. Clement et al. (2000) referred to the first phenomenon as ‘work ethic’, but the label should not be seen to imply that the birds would prefer to work harder, as this was not the case: the preference was between the outcomes of the two levels of work and not between the levels of work themselves. Kacelnik & Marsh (2002) instead related their findings to the notions of ‘sunk costs’ and ‘concorde fallacy’ that are used to describe cases when subjects (suboptimally) attribute value to options in relation to their history instead of to their future payoffs (Dawkins & Carlisle 1976; Thaler 1980). In terms of the mechanisms that may lead to these phenomena, Clement et al. (2000) proposed that the contrast between the hedonic state of the individual immediately prior to reward and at the time of reward may be greater after greater effort than after less effort. Hedonic state is a concept often used in reference to the agents’ wellbeing (see e.g. Berridge & Robinson 1998), but
571 Ó 2005 The Association for the Study of Animal Behaviour. Published by Elsevier Ltd. All rights reserved.
572
ANIMAL BEHAVIOUR, 70, 3
to our knowledge there is no agreed operational definition. The proposed mechanism may occur if delivery of reward always brings the animal to the same hedonic state, while the previous condition (the amount of work) leads to a different state prior to the reward being delivered. Thus, the differential increase in hedonic state is greater because harder work is more hedonically negative. Marsh (1999) and Kacelnik & Marsh (2002) proposed a similar mechanism inspired by functional considerations, and without the assumption that the rewards left the animals in the same hedonic state regardless of the starting point. In Marsh’s (1999, page 79) words, ‘Rewards realised after much effort could be locally assessed as being more valuable because of depressed energy reserves at the time of reward payoff. This reasoning is based on the assumption that an otherwise equivalent food payoff has more fitness value when an individual is in poorer condition than when it is in better condition, and that such relative considerations influence the way that outcome value is represented in memory’. If what the subject finds reinforcing is a correlate of fitness gain, then a food source that (because of the circumstances in which it was typically encountered) has previously yielded higher gains than an alternative would be preferred even when the circumstances have changed so that the gains from the two sources at the time of the choice are identical. In Marsh et al.’s (2004) experiment, the differential need reduction during acquisition was caused not by extra work but by the experimenters manipulating directly the energetic state of their subjects. They proposed that two payoffs of equal size would be valued differently if one typically occurred when reserves were lower, based on the assumption that the fitness versus reserves function is most often concave (e.g. McNamara & Houston 1982). In their experiment, Marsh et al. (2004) trained European starlings, Sturnus vulgaris, to peck differently coloured keys offering equal food rewards. The two food sources were encountered in different sessions, while the birds were either food deprived or fed. After training for an equal number of reinforcements in each state, the birds progressed into choice tests in which they were presented simultaneously with the two food sources. Regardless of whether they were hungry or fed at the time of the choice, the starlings significantly preferred the source that had previously been met under higher deprivation, suggesting that it is indeed plausible that the earlier results obtained by changing the work requirement might have been mediated by some effect of work on the need reduction afforded by the reward. Before revising the ideas involved, it is worth revisiting the bases on which normative models of choice are developed. We do this below. Normative models of decision making are characterized by referring to a scale of value, or ‘utility’, dependent on (but distinct from) the metrics of the events involved. The following statement, although written over two and a half centuries ago, makes the case for the relevance of this scale as clearly as anything written since: ‘The price of an item is dependent only on the thing itself and is equal for everyone; the utility, however, is dependent on the particular circumstances of the person making the
estimate. Thus, there is no doubt that a gain of one thousand ducats is more significant to a pauper than to a rich man though both gain the same amount.’ (Bernoulli 1738, page 24). Bernoulli’s use of the term ‘utility’ is not the same as that used in modern microeconomics, and is also different from the use of fitness in evolutionary biology. However, for the present purposes what his text highlights is the distinction between metrics of outcomes and metrics of value. For instance, a bee may experience sugar solutions that are 20, 40 or 60% in concentration: it may be possible to show a monotonic scale for the perception of concentration and a nonmonotonic scale for preference for concentration. In Bernoulli’s example, there is no suggestion that the pauper or the rich man perceive the sum as anything different from a thousand ducats, but they are said to value the same amount differently. Both the objective events and their perceived representations (which are treated as causal factors in the decision process) may be thought of as having dimensions of state. For instance, the monetary rewards can be referred to a scale of wealth, while the sugar in a solution has dimensions of energetic reserves. Both the pauper and the rich man become a thousand ducats better off from the prize, and they may both be aware that this is the case. In contrast, the value assigned to the events or to their perceived representations has dimensions of either fitness or utility, or of some correlated measure of motivational attractiveness, and hence they differ between the two subjects. For once, the pauper has more reasons to celebrate than the rich man, even if their final states will preserve their initial difference in wealth. Crucially, the relation between the perception of absolute properties of the reward and its utility need not be linear and may not even be monotonic (although it normally would be for the case of money). In behavioural ecology, the ideal dependent variable is evolutionary fitness and the independent variables are metrics of the physical properties of rewards. However, both dimensions are often replaced by operational (‘cognitive’) dimensions, where the dependent variable is some motivational correlate of fitness (i.e. ‘pleasure’) and the independent variable is the subject’s knowledge of the event, for instance, the bee’s receptor’s intensity of response to concentration or its knowledge of the size of a reward or the length of handling time. In the present study we manipulated the state of the starlings (as done by Marsh et al. 2004) and systematically varied, in different treatments, the objective properties (delay until food) of the option associated with the state of higher deprivation. This manipulation allows us to test several hypotheses, which we present using Fig. 1 as a framework. The figure presents a positive but decelerated relation between a dependent ‘value’ ordinate and an independent ‘state’ abscissa. The abscissa is an approximate representation of energetic state, so that a hungrier animal is placed closer to the origin. However, its dimensions are the same as the metrics of the reward, so that consuming a reward has the effect of displacing the animal’s state to the right. The reward metric is not confined to reward size, and in relation to the present experiment it is not. We
POMPILIO & KACELNIK: STATE, LEARNING AND CHOICE
Value (fitness/pleasure)
ˆ M H sPF vPF
ˆ M PF
MPF sH
vH
MH
H
PF State
Figure 1. Value (measured in fitness or in a motivational correlate) as a function of state of the subject. The outcome of each interaction with a food source produces a state change proportional to the outcome’s magnitude in some metric of state. Outcomes with greater immediacy (delay1) have lower opportunity cost and cause larger displacement. The subject encounters two food sources, each when it (the subject) is in one of two states, hungry (H) or prefed (PF). The outcomes are labelled MH and MPF and their metrics are represented as arrows that cause a positive state displacement. The ^ H and M ^ PF ) is shown in subject’s representation of these metrics (M the inset. The value of each outcome (vH and vPF) is the vertical displacement that corresponds to each change in state. The first derivative (marginal rate) of the value versus state function at each initial state is indicated by the slopes of the tangents sH and sPF. In this example the objective metrics of the outcomes are equal, but the subject’s representation differs because it sees the outcome obtained when hungrier as larger (or having greater immediacy) than the alternative.
consider that rewards delivered with greater immediacy cause a lower opportunity loss and hence a higher net payoff, so we represent all outcomes as causing a displacement along this axis, and more immediate rewards as causing a larger displacement than more delayed rewards. Figure 1 represents a subject that could be found in two states, which we label with the suffixes H (for ‘hungry’) and PF (for ‘prefed’). Rewards of magnitude MH or MPF cause positive displacements in the abscissa and upward displacements vH and vPF in the ordinate. The slope of the value versus state function at the points of encounter with the rewards (namely the first derivative or marginal rate of the function) is indicated by the slopes of the tangents sH and sPF. Using this representation, we may consider four hypotheses for mechanisms that could explain the reported effects of past experience. These are described below. (1) Magnitude priority: preferences are driven primarily by reward metrics (Mi), and secondarily by additional criteria. This is a lexicographic rule. In lexicographic models such as this one, the dimensions of the targets are judged in order, and the highest-order dimension in which options differ controls the choice. In terms of Fig. 1, differences in the horizontal arrows showing reward metrics would have primary and even exclusive control. This is compatible both with previous observations and with conventional reward maximizing, because in the published experiments the rewards were equal, so that the
comparison among magnitudes would have yielded no difference, and control could have passed to second-order dimensions such as the history of each source. The prediction of this hypothesis is that preference will always be for source H when MH O MPF and for source PF when the reverse is true, regardless of the strength of the inequality. Other, normally irrelevant dimensions would enter the scene when MH and MPF are strictly equal as in the experiments reported previously. (2) Value priority: preferences are driven by the value (vi) provided by each option at the time of learning. As already mentioned, two payoffs of equal size would be remembered as having different value if one typically occurred when reserves were lower, based on the assumption that the fitness versus reserves function is concave. In functional terms, this is equivalent to choosing based on the past fitness gain. This is compatible with previous reports because either harder work or greater deprivation placed the animals under conditions where the value function was steeper, and, since rewards were of equal size, the vertical displacement (vH) was larger for the hard work or hungry option. The present hypothesis differs from the previous one in that the impact of reward metrics is mediated through the consequences in value, so that the animal will prefer option H whenever vH O vPF. Under this hypothesis preference for the H option can persist even if MH ! MPF (namely if the reward from source H is of lower magnitude). The strength of this preference, however, should decrease as the inequality (vH O vPF) becomes less extreme, mediated through the effect of magnitude on value. (3) State priority: preferences are driven exclusively by the state of need when subjects learned about the options. In terms of Fig. 1, preferences would depend on the slopes of the tangents (sH, sPF). This could occur because natural selection may have calibrated the valuation mechanism to the marginal fitness gains of the value versus state function. This mechanism would sustain preference for option H regardless of the magnitude of its reward (MH). Unlike hypothesis 2, Mi would not even have an indirect effect. (4) Perceptual distortion: preferences are driven by the subject’s internal representation of the metrics of the ^ i ) and these representations are affected by rewards (M state at the time of learning. In Fig. 1, this is represented by the cartoon inset showing a bird that remembers the metric of one reward as smaller than the alternative even if the reward magnitudes were objectively equal. This idea is inspired by Killeen’s (1984) hypothesis that temporal intervals may be measured and represented internally by means of a ‘clock’ that varies its rate depending on state of arousal. In this hypothesis, if a subject learns about two stimuli SH and SPF both lasting 10 s but it meets these stimuli under low and high levels of arousal, respectively, it would store more ‘ticks’ of the clock for the duration of SPF because it is encountered when the clock runs faster. If the internally represented intervals are measured later on (for instance using a behavioural peak procedure) under equal arousal, one would observe a shorter interval for SH than for SPF. If the preference is for the option with the most favourable internal representation, the bird will
573
574
ANIMAL BEHAVIOUR, 70, 3
choose SH because it perceives it as yielding a shorter delay. Thus, in this case time of reward expectation should be the best predictor of preference. For instance, if subjects prefer option H when it yields 15 s of delay over option PF that always yields 10 s of delay then their pecking behaviour should show that the time of expectation of reward is shorter for option H, thus revealing a perceptual distortion caused by the training conditions. In the present study we designed the experimental procedure so as to establish as much as possible differences between the predictions of these four hypotheses. To distinguish between the four hypothetical mechanisms described above, we used four treatments, one of which reproduced the conditions of Marsh et al. (2004), whereas in the others the rewards were no longer equal.
METHODS
Subjects Our subjects were six wild-caught European starlings with extensive experimental histories. The birds were kept in individual cages (120 ! 60 cm and 50 cm high) that served as housing and test chambers. They were visually but not acoustically isolated on a light:dark cycle of 12:12 h and at a temperature of 16–18 C. Fresh drinking water was always available and bathing trays were provided at least twice a week. Birds were also permitted to feed ad libitum on turkey starter crumbs and supplementary mealworms (Tenebrio sp.) for 2.5 h at the end of the last experimental session. From previous experience this regime is known to allow the starlings’ body weights to remain stable at approximately 90% of their free feeding value (Bateson 1993). The birds were caught during 2002 and kept under an English Nature licence. The experiments took place in March 2003. All the birds remained healthy and were released into the wild (University Parks, Oxford) during the following summer.
Apparatus Each cage had a panel with a central food hopper and two square response keys. The keys could be illuminated with yellow, red, blue, orange, violet, white, pink or green. Computers running Arachnid language (Paul Fray Ltd., Cambridge, U.K.) served for control and data collection. The size of food rewards was fixed for all treatments at 2 units of semicrushed and sieved Orlux pellets (ca. 0.02 g/ unit) and delivered at a rate of 1 unit/s by automatic pellet dispensers (Campden Instruments, Cambridge, U.K.). The delays to delivery were varied as explained below.
Experimental Protocol We used a within-subject design in which each bird was exposed to four treatments (the order of the treatments was balanced across birds). Each treatment lasted for 6 days (5 days of training plus 1 day of testing). In all the
treatments, on the test day the starlings had to choose between two options (signalled with differently coloured keys). During training one of the options (option H, for ‘hungry’) had been encountered in a higher level of deprivation (the subjects were food deprived for 3 h before the session started). The other option (option PF, for ‘prefed’) had been encountered in a lower level of deprivation (the subjects were deprived as in the other group but were then fed for 10 min before the session started). Subjects were exposed to only forced trials during training and only test trials during testing. In forced trials only one key was active. The trial started with the key blinking (0.7 s ON–0.3 s OFF). The first peck caused the light to stay steadily ON and the programmed delay started running. The first peck after the programmed delay had elapsed triggered the delivery of the programmed amount of food, followed by a fixed intertrial interval (ITI) of 80 s during which all keys were OFF. If no peck was registered during the 5 s after the programmed delay had elapsed the bird lost the reward, a new ITI started, and the same option was presented after the ITI. This 5-s limited hold was aimed at reducing differences between experienced and programmed delay. Thus, in operant terms the schedule could be described as a discrete-trials, limited-hold, response-initiated, fixed interval. Choice trials started with two keys that simultaneously blinked. The first peck on either of them caused that key to turn steadily ON and the other to turn OFF. After that, the trial continued as in forced trials. During test trials we recorded the number of pecks/s during the delay programmed for each option. Option PF was programmed as a fixed interval (FI) of 10 s in all the treatments. In the alternative (option H), we manipulated the delay to food across treatments: FI10 s, FI12.5 s, FI15 s and FI17.5 s in treatments TR1, TR2, TR3 and TR4, respectively. Regardless of the preference of the subjects, the energetic rate of gain or immediacy offered by the options can be ranked in inverse order of length of the delay. We used this metric as the objective measure of the outcome of each option (equivalent to Mi in Fig. 1).
Pretraining All subjects were given a ‘reminder’ pretraining to eat from the hopper and peck at each of the eight hues (red, green, yellow, blue, orange, violet, white and pink) randomly presented on each of the pecking keys. Initially, each peck was followed by reinforcement. On the next 3 days of pretraining, the fixed interval was gradually increased to 17.5 s.
Training During this stage, the birds experienced only forced trials to allow them to learn about the properties of the options. They experienced two sessions per day, one of each type: Hungry and Prefed. Each session consisted of the following parts, in sequential order: (1) a food deprivation period (2 h 50 min); (2) a ‘manipulation’ period (10 min); (3) a ‘key-pecking’ period, where the birds had to complete 20 forced trials (this period could last up to
POMPILIO & KACELNIK: STATE, LEARNING AND CHOICE
50 min, depending on how long it took the birds to complete 20 trials); and (4) an ad libitum food period (turkey crumbs, for the final 15 min of each session) designed to reduce carry-over influences across sessions and equalize state for the following session. The manipulation determined the kind of session: in Prefed sessions, the experimenter entered the room and provided ad libitum food (turkey crumbs) for 10 min. In Hungry sessions, the experimenter entered the room at the same times as in Prefed sessions, but no food was provided. In the key-pecking period that immediately followed, a coloured lamp (red, green, yellow, violet, pink, orange, white or blue) corresponding with the session type and treatment was illuminated behind one of the pecking keys on either side (randomly determined across trials) of the panel. The association of colours with session types was balanced across birds and treatments, but for each bird one colour was always associated with one option type (e.g. red with PF and green with H) and a different pair of colours were used for each treatment. The key-pecking period was programmed to terminate after the subject completed 20 forced trials or after 50 min, whichever came first (in fact, the birds always completed the 20 forced trials in less than 50 min). The two daily sessions began at 0800 and 1200 hours, respectively. For each subject, one session of each type was given each day, with order of presentation balanced across subjects and reversing between days.
Test After 5 days of training, the birds experienced 1 day with two choice sessions. These sessions consisted of 15 choice trials each. In choice trials, subjects expressed their preference between options H and PF. The state of the subjects at the time of testing was manipulated as during the training period and the order in which subjects experienced the two states was balanced across birds: within the group, three of the subjects had the morning session in the Hungry state and the afternoon session in the Prefed state. The remaining three birds within each group had the opposite order. Thus, each subject contributed 30 choices, 15 in each state at the time of testing.
Data Analysis Preferences as revealed by choices The proportion of choices for each option during the choice sessions was considered as the measure of preference. Because preference is analysed in the form of proportions, the data were arcsine square-root transformed and residuals were inspected for normality and homogeneity of variance (Zar 1999). None of these assumptions were violated (Jarque–Bera test of normality and Bartlett’s test of the hypothesis that error variance of the dependent variable was equal across the different treatments: P O 0.05 in both cases).
Pecking patterns We analysed the pecking patterns during choice trials, converted into 1-s time bins. To estimate whether the
subjects could differentiate between H and PF options, we compared the relative percentage of accumulated pecks at bin 10 between the two options comprising each treatment (hereafter, R(10)). To obtain R(10) we first calculated individual values as follows: CHb Rb 10Z CHb CCPFb
ð1Þ
where Rb(10) is the score for bird b. CHb and CPFb are the mean cumulative numbers of pecks up to 10 s into the delay to food for the H and the PF options, respectively, for bird b. We calculated one proportion per bird, per treatment, and averaged the results across subjects to obtain R(10).
RESULTS
Preferences as Revealed by Choices We are interested in the effect of several factors on preference: state during training, state during testing, and metrics of the options. To simplify the language, we conducted all the analysis in terms of the proportion of choices for option H, the one that had been experienced during training in sessions when the subject was hungrier. We refer to this proportion as PH. Figure 2 shows PH in choice sessions when the subjects were, at the time of testing, in the Prefed state or in the Hungry state and also the results pooling all the data. Qualitatively, it can be seen that (1) all of the bars lie above the indifference line, indicating that there was a preference for the option associated with the Hungry state during training, even though this option was either equal to or worse than the alternative; (2) there is no major difference between the left and centre two groups of bars, indicating that state at the time of testing was largely ineffective; and (3) the strength of preference for option H decreased towards indifference as the delay to food in this option increased from 10 s (when it was equal to the alternative) to 17.5 s. For statistical analysis we considered two measures of preference: the proportion of subjects that favoured the H option (regardless of strength of preference) and the proportion of choices for this option across subjects and treatments (TR1–TR4). The proportion of subjects was tested as follows. Of the 30 choice opportunities each bird had (two sessions of 15 trials each, one in each of the energetic states), all six birds preferred the H option (i.e. PH O 0.5) in TR1–TR3. In TR4, PH was greater than 0.5 for four of the six birds. Thus, the proportion of subjects preferring H is significantly higher than random for TR1, TR2 and TR3 (binomial test: P Z 0.031), and nonsignificantly so for TR4 (P Z 0.688). For the second measure of preference, that is, to test the magnitude of preference, we carried out four one-sample t tests, one per treatment, comparing PH across birds against a random expectation of 0.5. The result indicates that the preference for H is significantly higher than 0.5 for TR1, TR2 and TR3 (TR1: t1,5 Z 6.386, P Z 0.001; TR2: t1,5 Z 5.263, P Z 0.003; TR3: t1,5 Z 5.181, P Z 0.004) and nonsignificant for TR4 (t1,5 Z 1.418, P O 0.215).
575
ANIMAL BEHAVIOUR, 70, 3
1 0.9 Proportion of choices
576
*
Tested when prefed
*
0.8
Overall
Tested when hungry
* **
*
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
FI10 s FI12.5 s FI15 s FI17.5 s
FI10 s FI12.5 s FI15 s FI17.5 s H option
FI10 s FI12.5 s FI15 s FI17.5 s
Figure 2. Mean proportion of choices G SD for the H options (i.e. those experienced during training when the subject was hungrier than in the prefed option). Whereas the PF option always had a programmed delay to food of 10 s, the H option always had a programmed delay of 10, 12.5, 15 or 17.5 s according to treatment. The subjects were tested either hungry or prefed. ‘Overall’ shows the data after pooling across testing states. The horizontal line indicate indifference. Asterisks indicate a significant (P ! 0.05) difference from random.
We examined the influences of state during testing and of treatment on choice proportions by running a GLM model, with PH as response variable and subject (from 1 to 6) as a random factor, state during testing (Prefed or Hungry) as a fixed factor and delay until food associated with the H option (10 s, 12.5 s, 15 s, 17.5 s) as a continuous variable. State during testing did not have significant effects (F1,24 Z 0.33, P Z 0.592) and there were no significant interactions between state during testing and treatment (F1,24 Z 0.15, P Z 0.698) or between subjects and state during testing (F5,24 Z 0.33, P Z 0.891). There were, however, significant effects of treatment. PH decreased as the delay to food in the H option increased (F1,24 Z 32.23, slope: 0.10008, P ! 0.0001). We did not find interaction effects between treatment and subjects (F5,24 Z 1.09, P Z 0.390).
had a longer delay. A one-sample t test per treatment comparing R(10) against a random expectation of 0.5 yielded significant differences from 0.5 for treatments TR3 and TR4 tested under the Prefed condition. Comparison with the choice results shown in Fig. 2 shows interesting effects. For TR1 and TR2, H was preferred, but R(10) did not differ from indifference; for TR3 H was preferred but R(10) was significantly less than 0.5, and for TR4 the birds were neutral but R(10) was significantly lower than 0.5. The most striking comparison is for TR3; the birds significantly preferred H when it delivered a delay 50% longer than the alternative PF, but the pecking patterns show that they could distinguish the options in terms of their programmed delay until reward.
DISCUSSION
Pecking Patterns The analysis of choice showed that the birds preferred delays of 12.5 s and 15 s to delays of 10 s or were indifferent between a delay of 17.5 s and a delay of 10 s provided that the history of the first option was associated with a state of greater need. This could be mediated by a distorted perception of the delays (hypothesis 4: perceptual distortion). To test this hypothesis, we analysed the pecking pattern during test trials and compared the pecks accumulated up to the 10th second between each pair of options presented in each treatment. The assumption is that this number of pecks should be greater for the option in which the reward is perceived as being more imminent, and this should be expressed by the index R(10) (see Methods). Figure 3 shows the R(10) index recorded under both energetic states and pooled during testing. The pooled data show that R(10) was almost exactly 0.5 when both options had delays of 10 s and was lower than 0.5 when the H option
Our subjects were allowed to learn the properties of two options while in two different states, and after extensive training they were allowed to choose between the options. While one option (PF) was always experienced when the subject was not very hungry and always offered a delay of 10 s, the other (H) was encountered when the subject was hungrier, and delivered delays of 10, 12.5, 15 and 17.5 s in four treatments. Similarly to Marsh et al. (2004), we found that subjects facing two options of equal objective value preferred the one associated with a state of higher deprivation during training. The preference for H declined in inverse relation to the delay in this option, reaching neutrality when H offered a delay of 17.5 s. The state of the animals at the time of expressing their preference had no detectable effect. Of the four hypotheses presented in the introduction, we can discard the first, third and fourth, for the following reasons. Hypothesis 1 postulated an absolute priority for the objective metrics of the reward, explaining previous
POMPILIO & KACELNIK: STATE, LEARNING AND CHOICE
1 Tested when prefed
Relative proportion of pecks accumulated at bin 10
0.9
Tested when hungry
Overall
0.8 0.7 0.6 0.5
*
*
0.4 0.3 0.2 0.1 0
FI10 s FI12.5 s FI15 s FI17.5 s
FI10 s FI12.5 s FI15 s FI17.5 s
FI10 s FI12.5 s FI15 s FI17.5 s
H option Figure 3. Mean G SD relative proportion of accumulated pecks for H options (i.e. those experienced during training when the subject was hungrier than in the prefed option) at bin 10 in test trials. For further details see legend of Fig. 2.
effects of state (Clement et al. 2000; Marsh et al. 2004) by the fact that the subjects (pigeons, Columba livia, and starlings) incurred no costs, as the options were objectively identical. This is not the case here, where starlings preferred an option yielding a delay of 15 s over another yielding a delay of 10 s. Hypothesis 3 postulated that only state at the time of training was influential, as the subject would pay attention only to its state-dependent marginal rate of gain with independence of reward metrics. This hypothesis can be discarded here because the preference for option H declined as the loss resulting from choosing it increased. Finally, hypothesis 4 postulated that the paradoxical preference was due to a distorted perception of the delays to food. If this were true, time-dependent pecking rate (which is known to reflect reward expectation) should have paralleled preference. In this study we did not collect data in probe trials where the precise timing of reward expectation could be judged, but extensive evidence published elsewhere (see, for instance, Kacelnik & Brunner 2002) proves that starlings do learn these intervals accurately. In addition, the analysis of relative cumulative number of pecks indicated that reward expectation after 10 s of waiting was lower in some cases when preference was higher. Critically, when H offered a delay of 15 s, preference for H was significantly higher than 50%, whereas reward expectation in the same option as measured by relative cumulative pecking rate was significantly lower. This leaves us with hypothesis 2, which predicates that preference is driven by the remembered ‘value’ of previous outcomes of each source and raises the issue (which we cannot fully explain) of why (in functional terms) the choice mechanism should have evolved these properties. Clearly, this mechanism would produce sensible choices whenever the probability of encountering each option is uncorrelated with the subject’s state. The animal would learn that source A yields higher gains than source B in
some metric correlated with fitness, and then will choose A over B whenever the choice occurs. However, as is the case in our experimental situation, the mechanism can produce paradoxical consequences if fitness gain is state dependent and encounters with each option occur on systematically different states. These possibly costly paradoxes may be counterbalanced if, first, they are rare in nature and, second, the mechanism has advantages over possible alternatives in a sufficiently large number of other circumstances. We do not know how frequently situations such as those implemented here (where the subject’s state correlates with probability of encounter) are encountered in nature, and arguing about this point may have to wait until formal modelling is implemented, but we may speculate about the possible benefits such systems may yield. Consider the possibility that (as in our hypothesis 1) decision makers remember the metrics of each alternative and then, whenever a choice is required, pick the alternative with more favourable parameters (such as shorter delay or larger reward). The drawback of this mechanism, in comparison to remembering a correlated variable of fitness gains, is that it may be harder to choose between actions whose consequences have different dimensions, such as amount and delay, mean and variance, or nutritional value and predation risk. A mechanism sensitive to past gains, instead, would simply allow the animal to compare alternatives in terms of the resulting benefits in some common currency, a hedonic scale related to fitness (McFarland & Sibly 1975; Cabanac 1992). In summary, while state-dependent valuation learning can lead to paradoxical and even costly choices in some situations, it may in fact be a very efficient way to act in comparison with other realistic possibilities. Although this is just a speculation, the present findings highlight the importance of the agent’s history. While for good theoretical reasons normative models of decision making tend to be path independent (Houston & McNamara 1999), so
577
578
ANIMAL BEHAVIOUR, 70, 3
that only present state and future consequences are included, empirical findings such as the present ones and those related to other paradoxical observations (such as framing, Marsh & Kacelnik 2002) suggest that pathdependent models may be due for an important contribution. In fact, subtle differences in state when experiencing options may have lasting cognitive consequences capable of explaining future differences in preferences between individuals in the same energetic state and this may have to be incorporated in the models. Two themes appear appropriate for future research. First, formal evolutionary modelling may be used to evaluate the likelihood that a mechanism such as the one we predicate may evolve in competition with other proposed alternatives. Second, the model itself should be tested empirically to judge the extent to which it may explain a wider range of anomalies found across diverse paradigms of animal choice.
Acknowledgments We thank Cynthia Schuck-Paim and Martin Rossi for advice, discussion and comments on the manuscript. L.P. was supported by a Clarendon Fund Scholarship from the University of Oxford. The research was partially funded by the BBSRC Grant 43/S13483 to A.K. References Bateson, M. 1993. Currencies for decision making: the foraging starling as a model animal. D.Phil. thesis, University of Oxford. Bernoulli, D. 1738. Specimen Theoriae novae de Mensura Sortis [Exposition of a new theory on the measurement of risk]. Translation Econometrica, 22, 23–36, January 1954. Berridge, K. C. & Robinson, T. E. 1998. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Research Reviews, 28, 309–369.
Cabanac, M. 1992. Pleasure: the common currency. Journal of Theoretical Biology, 155, 173–200. Clement, T. S., Feltus, J. R., Kaiser, D. H. & Zentall, T. R. 2000. ‘Work ethic’ in pigeons: reward value is directly related to the effort or time required to obtain the reward. Psychonomic Bulletin Review, 7, 100–106. Dawkins, R. & Carlisle, T. R. 1976. Parental investment, mate desertion and a fallacy. Nature, 161, 131–133. Houston, A. I. & McNamara, J. M. 1999. Models of Adaptive Behaviour: An Approach Based on State. Cambridge: Cambridge University Press. Kacelnik, A. & Brunner, D. 2002. Timing and foraging: Gibbon’s scalar expectancy theory and optimal patch exploitation. Learning and Motivation, 33, 177–195. Kacelnik, A. & Marsh, B. 2002. Cost can increase preference in starlings. Animal Behaviour, 63, 245–250. Killeen, P. R. 1984. Incentive theory III: adaptive clocks. In: Timing and Time Perception (Ed. by J. Gibbon & L. Allan), pp. 515–527. New York: New York Academy of Sciences. McFarland, D. J. & Sibly, R. M. 1975. The behavioural final common path. Philosophical Transactions of the Royal Society of London, Series B, 270, 265–293. McNamara, J. M. & Houston, A. I. 1982. Short-term behaviour and lifetime fitness. In: Functional Ontogeny (Ed. by D. J. McFarland), pp. 60–87. London: Pitman. Marsh, B. 1999. Making the best choice: judgement and strategic decision making under conditions of risk and uncertainty. D.Phil. thesis, University of Oxford. Marsh, B. & Kacelnik, A. 2002. Framing effects and risky decisions in starlings. Proceedings of the National Academy of Sciences, U.S.A., 99, 3352–3355. Marsh, B., Schuck-Paim, C. & Kacelnik, A. 2004. State-dependent learning affects foraging choices in starlings. Behavioral Ecology, 15, 396–399. Pompilio, L. 2004. Animal choice and the construction of preferences. D.Phil. thesis, University of Oxford. Thaler, R. 1980. Towards a positive theory of consumer choice. Journal of Economic Behaviour and Organization, 1, 39–60. Zar, J. M. 1999. Biostatistical Analysis, 4th edn. Englewood cliff: Prentice Hall.
State-dependent learning and suboptimal choice: when starlings prefer long over short delays to food LORENA PO MPILIO & A LEX K ACELNIK
Department of Zoology, University of Oxford (Received 30 July 2004; initial acceptance 3 September 2004; final acceptance 22 December 2004; published online 11 July 2005; MS. number: 8228)
Recent studies have used labels such as ‘work ethics’, ‘sunk costs’ and ‘state-dependent preferences’ for apparent anomalies in animals’ choices. They suggest that preference between options relates to the options’ history, rather than depending exclusively on the expected payoffs. For instance, European starlings, Sturnus vulgaris, trained to obtain identical food rewards from two sources while in two levels of hunger preferred the food source previously associated with higher hunger, regardless of the birds’ state at the time of testing. We extended this experimentally and theoretically by studying starlings choosing between sources that differed not only in history but also in the objective properties (delay until reward) of the payoffs they delivered. Two options (PF and H) were initially presented in single-option sessions when subjects were, respectively, prefed or hungry. While option PF offered a delay until reward of 10 s in all treatments, option H delivered delays of 10, 12.5, 15 and 17.5 s in four treatments. When training was completed, we tested preference between the options. When delays in both options were equal (10 s), the birds strongly preferred H. When delay in H was 17.5 s, the birds were indifferent, with intermediate results for intermediate treatments. Preference was not mediated by disrupted knowledge of the delays. Thus, preferences were driven by past state-dependent gains, rather than by the joint effect of the birds’ state at the time of choice and knowledge of the absolute properties of each alternative, as assumed in state-dependent, path-independent models of optimal choice. Ó 2005 The Association for the Study of Animal Behaviour. Published by Elsevier Ltd. All rights reserved.
Several recent studies of avian decision making have dealt with two related and somewhat counterintuitive observations. The first is that when subjects are initially trained to obtain equally sized rewards after either hard or light work, and are then offered a choice without the work requirement, they prefer the reward previously associated with harder work (Clement et al. 2000; Kacelnik & Marsh 2002; originally reported by Marsh 1999). The second is that when subjects are trained with equal rewards in different states of deprivation, and are then offered a choice, they prefer the reward previously associated with greater deprivation (Marsh et al. 2004). Enhanced preference for stimuli associated with greater need is probably far from restricted to birds: locusts, Locusta migratoria, too prefer a signal (an odour in this case) that has been previously paired with food encountered under a state of greater need, regardless of state at the time of testing (Pompilio 2004; L. Pompilio, A. Kacelnik & S. T.
Correspondence: A. Kacelnik, Department of Zoology, Oxford University, South Parks Road, Oxford OX1 3PS, U.K. (email: alex.kacelnik@ zoo.ox.ac.uk). 0003–3472/04/$30.00/0
Bahmar, unpublished data). Here we discuss the findings on birds, propose some theoretical clarifications, and present an experimental study aimed at establishing the strength of the effect and the relevance of previously proposed theoretical models. Clement et al. (2000) referred to the first phenomenon as ‘work ethic’, but the label should not be seen to imply that the birds would prefer to work harder, as this was not the case: the preference was between the outcomes of the two levels of work and not between the levels of work themselves. Kacelnik & Marsh (2002) instead related their findings to the notions of ‘sunk costs’ and ‘concorde fallacy’ that are used to describe cases when subjects (suboptimally) attribute value to options in relation to their history instead of to their future payoffs (Dawkins & Carlisle 1976; Thaler 1980). In terms of the mechanisms that may lead to these phenomena, Clement et al. (2000) proposed that the contrast between the hedonic state of the individual immediately prior to reward and at the time of reward may be greater after greater effort than after less effort. Hedonic state is a concept often used in reference to the agents’ wellbeing (see e.g. Berridge & Robinson 1998), but
571 Ó 2005 The Association for the Study of Animal Behaviour. Published by Elsevier Ltd. All rights reserved.
572
ANIMAL BEHAVIOUR, 70, 3
to our knowledge there is no agreed operational definition. The proposed mechanism may occur if delivery of reward always brings the animal to the same hedonic state, while the previous condition (the amount of work) leads to a different state prior to the reward being delivered. Thus, the differential increase in hedonic state is greater because harder work is more hedonically negative. Marsh (1999) and Kacelnik & Marsh (2002) proposed a similar mechanism inspired by functional considerations, and without the assumption that the rewards left the animals in the same hedonic state regardless of the starting point. In Marsh’s (1999, page 79) words, ‘Rewards realised after much effort could be locally assessed as being more valuable because of depressed energy reserves at the time of reward payoff. This reasoning is based on the assumption that an otherwise equivalent food payoff has more fitness value when an individual is in poorer condition than when it is in better condition, and that such relative considerations influence the way that outcome value is represented in memory’. If what the subject finds reinforcing is a correlate of fitness gain, then a food source that (because of the circumstances in which it was typically encountered) has previously yielded higher gains than an alternative would be preferred even when the circumstances have changed so that the gains from the two sources at the time of the choice are identical. In Marsh et al.’s (2004) experiment, the differential need reduction during acquisition was caused not by extra work but by the experimenters manipulating directly the energetic state of their subjects. They proposed that two payoffs of equal size would be valued differently if one typically occurred when reserves were lower, based on the assumption that the fitness versus reserves function is most often concave (e.g. McNamara & Houston 1982). In their experiment, Marsh et al. (2004) trained European starlings, Sturnus vulgaris, to peck differently coloured keys offering equal food rewards. The two food sources were encountered in different sessions, while the birds were either food deprived or fed. After training for an equal number of reinforcements in each state, the birds progressed into choice tests in which they were presented simultaneously with the two food sources. Regardless of whether they were hungry or fed at the time of the choice, the starlings significantly preferred the source that had previously been met under higher deprivation, suggesting that it is indeed plausible that the earlier results obtained by changing the work requirement might have been mediated by some effect of work on the need reduction afforded by the reward. Before revising the ideas involved, it is worth revisiting the bases on which normative models of choice are developed. We do this below. Normative models of decision making are characterized by referring to a scale of value, or ‘utility’, dependent on (but distinct from) the metrics of the events involved. The following statement, although written over two and a half centuries ago, makes the case for the relevance of this scale as clearly as anything written since: ‘The price of an item is dependent only on the thing itself and is equal for everyone; the utility, however, is dependent on the particular circumstances of the person making the
estimate. Thus, there is no doubt that a gain of one thousand ducats is more significant to a pauper than to a rich man though both gain the same amount.’ (Bernoulli 1738, page 24). Bernoulli’s use of the term ‘utility’ is not the same as that used in modern microeconomics, and is also different from the use of fitness in evolutionary biology. However, for the present purposes what his text highlights is the distinction between metrics of outcomes and metrics of value. For instance, a bee may experience sugar solutions that are 20, 40 or 60% in concentration: it may be possible to show a monotonic scale for the perception of concentration and a nonmonotonic scale for preference for concentration. In Bernoulli’s example, there is no suggestion that the pauper or the rich man perceive the sum as anything different from a thousand ducats, but they are said to value the same amount differently. Both the objective events and their perceived representations (which are treated as causal factors in the decision process) may be thought of as having dimensions of state. For instance, the monetary rewards can be referred to a scale of wealth, while the sugar in a solution has dimensions of energetic reserves. Both the pauper and the rich man become a thousand ducats better off from the prize, and they may both be aware that this is the case. In contrast, the value assigned to the events or to their perceived representations has dimensions of either fitness or utility, or of some correlated measure of motivational attractiveness, and hence they differ between the two subjects. For once, the pauper has more reasons to celebrate than the rich man, even if their final states will preserve their initial difference in wealth. Crucially, the relation between the perception of absolute properties of the reward and its utility need not be linear and may not even be monotonic (although it normally would be for the case of money). In behavioural ecology, the ideal dependent variable is evolutionary fitness and the independent variables are metrics of the physical properties of rewards. However, both dimensions are often replaced by operational (‘cognitive’) dimensions, where the dependent variable is some motivational correlate of fitness (i.e. ‘pleasure’) and the independent variable is the subject’s knowledge of the event, for instance, the bee’s receptor’s intensity of response to concentration or its knowledge of the size of a reward or the length of handling time. In the present study we manipulated the state of the starlings (as done by Marsh et al. 2004) and systematically varied, in different treatments, the objective properties (delay until food) of the option associated with the state of higher deprivation. This manipulation allows us to test several hypotheses, which we present using Fig. 1 as a framework. The figure presents a positive but decelerated relation between a dependent ‘value’ ordinate and an independent ‘state’ abscissa. The abscissa is an approximate representation of energetic state, so that a hungrier animal is placed closer to the origin. However, its dimensions are the same as the metrics of the reward, so that consuming a reward has the effect of displacing the animal’s state to the right. The reward metric is not confined to reward size, and in relation to the present experiment it is not. We
POMPILIO & KACELNIK: STATE, LEARNING AND CHOICE
Value (fitness/pleasure)
ˆ M H sPF vPF
ˆ M PF
MPF sH
vH
MH
H
PF State
Figure 1. Value (measured in fitness or in a motivational correlate) as a function of state of the subject. The outcome of each interaction with a food source produces a state change proportional to the outcome’s magnitude in some metric of state. Outcomes with greater immediacy (delay1) have lower opportunity cost and cause larger displacement. The subject encounters two food sources, each when it (the subject) is in one of two states, hungry (H) or prefed (PF). The outcomes are labelled MH and MPF and their metrics are represented as arrows that cause a positive state displacement. The ^ H and M ^ PF ) is shown in subject’s representation of these metrics (M the inset. The value of each outcome (vH and vPF) is the vertical displacement that corresponds to each change in state. The first derivative (marginal rate) of the value versus state function at each initial state is indicated by the slopes of the tangents sH and sPF. In this example the objective metrics of the outcomes are equal, but the subject’s representation differs because it sees the outcome obtained when hungrier as larger (or having greater immediacy) than the alternative.
consider that rewards delivered with greater immediacy cause a lower opportunity loss and hence a higher net payoff, so we represent all outcomes as causing a displacement along this axis, and more immediate rewards as causing a larger displacement than more delayed rewards. Figure 1 represents a subject that could be found in two states, which we label with the suffixes H (for ‘hungry’) and PF (for ‘prefed’). Rewards of magnitude MH or MPF cause positive displacements in the abscissa and upward displacements vH and vPF in the ordinate. The slope of the value versus state function at the points of encounter with the rewards (namely the first derivative or marginal rate of the function) is indicated by the slopes of the tangents sH and sPF. Using this representation, we may consider four hypotheses for mechanisms that could explain the reported effects of past experience. These are described below. (1) Magnitude priority: preferences are driven primarily by reward metrics (Mi), and secondarily by additional criteria. This is a lexicographic rule. In lexicographic models such as this one, the dimensions of the targets are judged in order, and the highest-order dimension in which options differ controls the choice. In terms of Fig. 1, differences in the horizontal arrows showing reward metrics would have primary and even exclusive control. This is compatible both with previous observations and with conventional reward maximizing, because in the published experiments the rewards were equal, so that the
comparison among magnitudes would have yielded no difference, and control could have passed to second-order dimensions such as the history of each source. The prediction of this hypothesis is that preference will always be for source H when MH O MPF and for source PF when the reverse is true, regardless of the strength of the inequality. Other, normally irrelevant dimensions would enter the scene when MH and MPF are strictly equal as in the experiments reported previously. (2) Value priority: preferences are driven by the value (vi) provided by each option at the time of learning. As already mentioned, two payoffs of equal size would be remembered as having different value if one typically occurred when reserves were lower, based on the assumption that the fitness versus reserves function is concave. In functional terms, this is equivalent to choosing based on the past fitness gain. This is compatible with previous reports because either harder work or greater deprivation placed the animals under conditions where the value function was steeper, and, since rewards were of equal size, the vertical displacement (vH) was larger for the hard work or hungry option. The present hypothesis differs from the previous one in that the impact of reward metrics is mediated through the consequences in value, so that the animal will prefer option H whenever vH O vPF. Under this hypothesis preference for the H option can persist even if MH ! MPF (namely if the reward from source H is of lower magnitude). The strength of this preference, however, should decrease as the inequality (vH O vPF) becomes less extreme, mediated through the effect of magnitude on value. (3) State priority: preferences are driven exclusively by the state of need when subjects learned about the options. In terms of Fig. 1, preferences would depend on the slopes of the tangents (sH, sPF). This could occur because natural selection may have calibrated the valuation mechanism to the marginal fitness gains of the value versus state function. This mechanism would sustain preference for option H regardless of the magnitude of its reward (MH). Unlike hypothesis 2, Mi would not even have an indirect effect. (4) Perceptual distortion: preferences are driven by the subject’s internal representation of the metrics of the ^ i ) and these representations are affected by rewards (M state at the time of learning. In Fig. 1, this is represented by the cartoon inset showing a bird that remembers the metric of one reward as smaller than the alternative even if the reward magnitudes were objectively equal. This idea is inspired by Killeen’s (1984) hypothesis that temporal intervals may be measured and represented internally by means of a ‘clock’ that varies its rate depending on state of arousal. In this hypothesis, if a subject learns about two stimuli SH and SPF both lasting 10 s but it meets these stimuli under low and high levels of arousal, respectively, it would store more ‘ticks’ of the clock for the duration of SPF because it is encountered when the clock runs faster. If the internally represented intervals are measured later on (for instance using a behavioural peak procedure) under equal arousal, one would observe a shorter interval for SH than for SPF. If the preference is for the option with the most favourable internal representation, the bird will
573
574
ANIMAL BEHAVIOUR, 70, 3
choose SH because it perceives it as yielding a shorter delay. Thus, in this case time of reward expectation should be the best predictor of preference. For instance, if subjects prefer option H when it yields 15 s of delay over option PF that always yields 10 s of delay then their pecking behaviour should show that the time of expectation of reward is shorter for option H, thus revealing a perceptual distortion caused by the training conditions. In the present study we designed the experimental procedure so as to establish as much as possible differences between the predictions of these four hypotheses. To distinguish between the four hypothetical mechanisms described above, we used four treatments, one of which reproduced the conditions of Marsh et al. (2004), whereas in the others the rewards were no longer equal.
METHODS
Subjects Our subjects were six wild-caught European starlings with extensive experimental histories. The birds were kept in individual cages (120 ! 60 cm and 50 cm high) that served as housing and test chambers. They were visually but not acoustically isolated on a light:dark cycle of 12:12 h and at a temperature of 16–18 C. Fresh drinking water was always available and bathing trays were provided at least twice a week. Birds were also permitted to feed ad libitum on turkey starter crumbs and supplementary mealworms (Tenebrio sp.) for 2.5 h at the end of the last experimental session. From previous experience this regime is known to allow the starlings’ body weights to remain stable at approximately 90% of their free feeding value (Bateson 1993). The birds were caught during 2002 and kept under an English Nature licence. The experiments took place in March 2003. All the birds remained healthy and were released into the wild (University Parks, Oxford) during the following summer.
Apparatus Each cage had a panel with a central food hopper and two square response keys. The keys could be illuminated with yellow, red, blue, orange, violet, white, pink or green. Computers running Arachnid language (Paul Fray Ltd., Cambridge, U.K.) served for control and data collection. The size of food rewards was fixed for all treatments at 2 units of semicrushed and sieved Orlux pellets (ca. 0.02 g/ unit) and delivered at a rate of 1 unit/s by automatic pellet dispensers (Campden Instruments, Cambridge, U.K.). The delays to delivery were varied as explained below.
Experimental Protocol We used a within-subject design in which each bird was exposed to four treatments (the order of the treatments was balanced across birds). Each treatment lasted for 6 days (5 days of training plus 1 day of testing). In all the
treatments, on the test day the starlings had to choose between two options (signalled with differently coloured keys). During training one of the options (option H, for ‘hungry’) had been encountered in a higher level of deprivation (the subjects were food deprived for 3 h before the session started). The other option (option PF, for ‘prefed’) had been encountered in a lower level of deprivation (the subjects were deprived as in the other group but were then fed for 10 min before the session started). Subjects were exposed to only forced trials during training and only test trials during testing. In forced trials only one key was active. The trial started with the key blinking (0.7 s ON–0.3 s OFF). The first peck caused the light to stay steadily ON and the programmed delay started running. The first peck after the programmed delay had elapsed triggered the delivery of the programmed amount of food, followed by a fixed intertrial interval (ITI) of 80 s during which all keys were OFF. If no peck was registered during the 5 s after the programmed delay had elapsed the bird lost the reward, a new ITI started, and the same option was presented after the ITI. This 5-s limited hold was aimed at reducing differences between experienced and programmed delay. Thus, in operant terms the schedule could be described as a discrete-trials, limited-hold, response-initiated, fixed interval. Choice trials started with two keys that simultaneously blinked. The first peck on either of them caused that key to turn steadily ON and the other to turn OFF. After that, the trial continued as in forced trials. During test trials we recorded the number of pecks/s during the delay programmed for each option. Option PF was programmed as a fixed interval (FI) of 10 s in all the treatments. In the alternative (option H), we manipulated the delay to food across treatments: FI10 s, FI12.5 s, FI15 s and FI17.5 s in treatments TR1, TR2, TR3 and TR4, respectively. Regardless of the preference of the subjects, the energetic rate of gain or immediacy offered by the options can be ranked in inverse order of length of the delay. We used this metric as the objective measure of the outcome of each option (equivalent to Mi in Fig. 1).
Pretraining All subjects were given a ‘reminder’ pretraining to eat from the hopper and peck at each of the eight hues (red, green, yellow, blue, orange, violet, white and pink) randomly presented on each of the pecking keys. Initially, each peck was followed by reinforcement. On the next 3 days of pretraining, the fixed interval was gradually increased to 17.5 s.
Training During this stage, the birds experienced only forced trials to allow them to learn about the properties of the options. They experienced two sessions per day, one of each type: Hungry and Prefed. Each session consisted of the following parts, in sequential order: (1) a food deprivation period (2 h 50 min); (2) a ‘manipulation’ period (10 min); (3) a ‘key-pecking’ period, where the birds had to complete 20 forced trials (this period could last up to
POMPILIO & KACELNIK: STATE, LEARNING AND CHOICE
50 min, depending on how long it took the birds to complete 20 trials); and (4) an ad libitum food period (turkey crumbs, for the final 15 min of each session) designed to reduce carry-over influences across sessions and equalize state for the following session. The manipulation determined the kind of session: in Prefed sessions, the experimenter entered the room and provided ad libitum food (turkey crumbs) for 10 min. In Hungry sessions, the experimenter entered the room at the same times as in Prefed sessions, but no food was provided. In the key-pecking period that immediately followed, a coloured lamp (red, green, yellow, violet, pink, orange, white or blue) corresponding with the session type and treatment was illuminated behind one of the pecking keys on either side (randomly determined across trials) of the panel. The association of colours with session types was balanced across birds and treatments, but for each bird one colour was always associated with one option type (e.g. red with PF and green with H) and a different pair of colours were used for each treatment. The key-pecking period was programmed to terminate after the subject completed 20 forced trials or after 50 min, whichever came first (in fact, the birds always completed the 20 forced trials in less than 50 min). The two daily sessions began at 0800 and 1200 hours, respectively. For each subject, one session of each type was given each day, with order of presentation balanced across subjects and reversing between days.
Test After 5 days of training, the birds experienced 1 day with two choice sessions. These sessions consisted of 15 choice trials each. In choice trials, subjects expressed their preference between options H and PF. The state of the subjects at the time of testing was manipulated as during the training period and the order in which subjects experienced the two states was balanced across birds: within the group, three of the subjects had the morning session in the Hungry state and the afternoon session in the Prefed state. The remaining three birds within each group had the opposite order. Thus, each subject contributed 30 choices, 15 in each state at the time of testing.
Data Analysis Preferences as revealed by choices The proportion of choices for each option during the choice sessions was considered as the measure of preference. Because preference is analysed in the form of proportions, the data were arcsine square-root transformed and residuals were inspected for normality and homogeneity of variance (Zar 1999). None of these assumptions were violated (Jarque–Bera test of normality and Bartlett’s test of the hypothesis that error variance of the dependent variable was equal across the different treatments: P O 0.05 in both cases).
Pecking patterns We analysed the pecking patterns during choice trials, converted into 1-s time bins. To estimate whether the
subjects could differentiate between H and PF options, we compared the relative percentage of accumulated pecks at bin 10 between the two options comprising each treatment (hereafter, R(10)). To obtain R(10) we first calculated individual values as follows: CHb Rb 10Z CHb CCPFb
ð1Þ
where Rb(10) is the score for bird b. CHb and CPFb are the mean cumulative numbers of pecks up to 10 s into the delay to food for the H and the PF options, respectively, for bird b. We calculated one proportion per bird, per treatment, and averaged the results across subjects to obtain R(10).
RESULTS
Preferences as Revealed by Choices We are interested in the effect of several factors on preference: state during training, state during testing, and metrics of the options. To simplify the language, we conducted all the analysis in terms of the proportion of choices for option H, the one that had been experienced during training in sessions when the subject was hungrier. We refer to this proportion as PH. Figure 2 shows PH in choice sessions when the subjects were, at the time of testing, in the Prefed state or in the Hungry state and also the results pooling all the data. Qualitatively, it can be seen that (1) all of the bars lie above the indifference line, indicating that there was a preference for the option associated with the Hungry state during training, even though this option was either equal to or worse than the alternative; (2) there is no major difference between the left and centre two groups of bars, indicating that state at the time of testing was largely ineffective; and (3) the strength of preference for option H decreased towards indifference as the delay to food in this option increased from 10 s (when it was equal to the alternative) to 17.5 s. For statistical analysis we considered two measures of preference: the proportion of subjects that favoured the H option (regardless of strength of preference) and the proportion of choices for this option across subjects and treatments (TR1–TR4). The proportion of subjects was tested as follows. Of the 30 choice opportunities each bird had (two sessions of 15 trials each, one in each of the energetic states), all six birds preferred the H option (i.e. PH O 0.5) in TR1–TR3. In TR4, PH was greater than 0.5 for four of the six birds. Thus, the proportion of subjects preferring H is significantly higher than random for TR1, TR2 and TR3 (binomial test: P Z 0.031), and nonsignificantly so for TR4 (P Z 0.688). For the second measure of preference, that is, to test the magnitude of preference, we carried out four one-sample t tests, one per treatment, comparing PH across birds against a random expectation of 0.5. The result indicates that the preference for H is significantly higher than 0.5 for TR1, TR2 and TR3 (TR1: t1,5 Z 6.386, P Z 0.001; TR2: t1,5 Z 5.263, P Z 0.003; TR3: t1,5 Z 5.181, P Z 0.004) and nonsignificant for TR4 (t1,5 Z 1.418, P O 0.215).
575
ANIMAL BEHAVIOUR, 70, 3
1 0.9 Proportion of choices
576
*
Tested when prefed
*
0.8
Overall
Tested when hungry
* **
*
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
FI10 s FI12.5 s FI15 s FI17.5 s
FI10 s FI12.5 s FI15 s FI17.5 s H option
FI10 s FI12.5 s FI15 s FI17.5 s
Figure 2. Mean proportion of choices G SD for the H options (i.e. those experienced during training when the subject was hungrier than in the prefed option). Whereas the PF option always had a programmed delay to food of 10 s, the H option always had a programmed delay of 10, 12.5, 15 or 17.5 s according to treatment. The subjects were tested either hungry or prefed. ‘Overall’ shows the data after pooling across testing states. The horizontal line indicate indifference. Asterisks indicate a significant (P ! 0.05) difference from random.
We examined the influences of state during testing and of treatment on choice proportions by running a GLM model, with PH as response variable and subject (from 1 to 6) as a random factor, state during testing (Prefed or Hungry) as a fixed factor and delay until food associated with the H option (10 s, 12.5 s, 15 s, 17.5 s) as a continuous variable. State during testing did not have significant effects (F1,24 Z 0.33, P Z 0.592) and there were no significant interactions between state during testing and treatment (F1,24 Z 0.15, P Z 0.698) or between subjects and state during testing (F5,24 Z 0.33, P Z 0.891). There were, however, significant effects of treatment. PH decreased as the delay to food in the H option increased (F1,24 Z 32.23, slope: 0.10008, P ! 0.0001). We did not find interaction effects between treatment and subjects (F5,24 Z 1.09, P Z 0.390).
had a longer delay. A one-sample t test per treatment comparing R(10) against a random expectation of 0.5 yielded significant differences from 0.5 for treatments TR3 and TR4 tested under the Prefed condition. Comparison with the choice results shown in Fig. 2 shows interesting effects. For TR1 and TR2, H was preferred, but R(10) did not differ from indifference; for TR3 H was preferred but R(10) was significantly less than 0.5, and for TR4 the birds were neutral but R(10) was significantly lower than 0.5. The most striking comparison is for TR3; the birds significantly preferred H when it delivered a delay 50% longer than the alternative PF, but the pecking patterns show that they could distinguish the options in terms of their programmed delay until reward.
DISCUSSION
Pecking Patterns The analysis of choice showed that the birds preferred delays of 12.5 s and 15 s to delays of 10 s or were indifferent between a delay of 17.5 s and a delay of 10 s provided that the history of the first option was associated with a state of greater need. This could be mediated by a distorted perception of the delays (hypothesis 4: perceptual distortion). To test this hypothesis, we analysed the pecking pattern during test trials and compared the pecks accumulated up to the 10th second between each pair of options presented in each treatment. The assumption is that this number of pecks should be greater for the option in which the reward is perceived as being more imminent, and this should be expressed by the index R(10) (see Methods). Figure 3 shows the R(10) index recorded under both energetic states and pooled during testing. The pooled data show that R(10) was almost exactly 0.5 when both options had delays of 10 s and was lower than 0.5 when the H option
Our subjects were allowed to learn the properties of two options while in two different states, and after extensive training they were allowed to choose between the options. While one option (PF) was always experienced when the subject was not very hungry and always offered a delay of 10 s, the other (H) was encountered when the subject was hungrier, and delivered delays of 10, 12.5, 15 and 17.5 s in four treatments. Similarly to Marsh et al. (2004), we found that subjects facing two options of equal objective value preferred the one associated with a state of higher deprivation during training. The preference for H declined in inverse relation to the delay in this option, reaching neutrality when H offered a delay of 17.5 s. The state of the animals at the time of expressing their preference had no detectable effect. Of the four hypotheses presented in the introduction, we can discard the first, third and fourth, for the following reasons. Hypothesis 1 postulated an absolute priority for the objective metrics of the reward, explaining previous
POMPILIO & KACELNIK: STATE, LEARNING AND CHOICE
1 Tested when prefed
Relative proportion of pecks accumulated at bin 10
0.9
Tested when hungry
Overall
0.8 0.7 0.6 0.5
*
*
0.4 0.3 0.2 0.1 0
FI10 s FI12.5 s FI15 s FI17.5 s
FI10 s FI12.5 s FI15 s FI17.5 s
FI10 s FI12.5 s FI15 s FI17.5 s
H option Figure 3. Mean G SD relative proportion of accumulated pecks for H options (i.e. those experienced during training when the subject was hungrier than in the prefed option) at bin 10 in test trials. For further details see legend of Fig. 2.
effects of state (Clement et al. 2000; Marsh et al. 2004) by the fact that the subjects (pigeons, Columba livia, and starlings) incurred no costs, as the options were objectively identical. This is not the case here, where starlings preferred an option yielding a delay of 15 s over another yielding a delay of 10 s. Hypothesis 3 postulated that only state at the time of training was influential, as the subject would pay attention only to its state-dependent marginal rate of gain with independence of reward metrics. This hypothesis can be discarded here because the preference for option H declined as the loss resulting from choosing it increased. Finally, hypothesis 4 postulated that the paradoxical preference was due to a distorted perception of the delays to food. If this were true, time-dependent pecking rate (which is known to reflect reward expectation) should have paralleled preference. In this study we did not collect data in probe trials where the precise timing of reward expectation could be judged, but extensive evidence published elsewhere (see, for instance, Kacelnik & Brunner 2002) proves that starlings do learn these intervals accurately. In addition, the analysis of relative cumulative number of pecks indicated that reward expectation after 10 s of waiting was lower in some cases when preference was higher. Critically, when H offered a delay of 15 s, preference for H was significantly higher than 50%, whereas reward expectation in the same option as measured by relative cumulative pecking rate was significantly lower. This leaves us with hypothesis 2, which predicates that preference is driven by the remembered ‘value’ of previous outcomes of each source and raises the issue (which we cannot fully explain) of why (in functional terms) the choice mechanism should have evolved these properties. Clearly, this mechanism would produce sensible choices whenever the probability of encountering each option is uncorrelated with the subject’s state. The animal would learn that source A yields higher gains than source B in
some metric correlated with fitness, and then will choose A over B whenever the choice occurs. However, as is the case in our experimental situation, the mechanism can produce paradoxical consequences if fitness gain is state dependent and encounters with each option occur on systematically different states. These possibly costly paradoxes may be counterbalanced if, first, they are rare in nature and, second, the mechanism has advantages over possible alternatives in a sufficiently large number of other circumstances. We do not know how frequently situations such as those implemented here (where the subject’s state correlates with probability of encounter) are encountered in nature, and arguing about this point may have to wait until formal modelling is implemented, but we may speculate about the possible benefits such systems may yield. Consider the possibility that (as in our hypothesis 1) decision makers remember the metrics of each alternative and then, whenever a choice is required, pick the alternative with more favourable parameters (such as shorter delay or larger reward). The drawback of this mechanism, in comparison to remembering a correlated variable of fitness gains, is that it may be harder to choose between actions whose consequences have different dimensions, such as amount and delay, mean and variance, or nutritional value and predation risk. A mechanism sensitive to past gains, instead, would simply allow the animal to compare alternatives in terms of the resulting benefits in some common currency, a hedonic scale related to fitness (McFarland & Sibly 1975; Cabanac 1992). In summary, while state-dependent valuation learning can lead to paradoxical and even costly choices in some situations, it may in fact be a very efficient way to act in comparison with other realistic possibilities. Although this is just a speculation, the present findings highlight the importance of the agent’s history. While for good theoretical reasons normative models of decision making tend to be path independent (Houston & McNamara 1999), so
577
578
ANIMAL BEHAVIOUR, 70, 3
that only present state and future consequences are included, empirical findings such as the present ones and those related to other paradoxical observations (such as framing, Marsh & Kacelnik 2002) suggest that pathdependent models may be due for an important contribution. In fact, subtle differences in state when experiencing options may have lasting cognitive consequences capable of explaining future differences in preferences between individuals in the same energetic state and this may have to be incorporated in the models. Two themes appear appropriate for future research. First, formal evolutionary modelling may be used to evaluate the likelihood that a mechanism such as the one we predicate may evolve in competition with other proposed alternatives. Second, the model itself should be tested empirically to judge the extent to which it may explain a wider range of anomalies found across diverse paradigms of animal choice.
Acknowledgments We thank Cynthia Schuck-Paim and Martin Rossi for advice, discussion and comments on the manuscript. L.P. was supported by a Clarendon Fund Scholarship from the University of Oxford. The research was partially funded by the BBSRC Grant 43/S13483 to A.K. References Bateson, M. 1993. Currencies for decision making: the foraging starling as a model animal. D.Phil. thesis, University of Oxford. Bernoulli, D. 1738. Specimen Theoriae novae de Mensura Sortis [Exposition of a new theory on the measurement of risk]. Translation Econometrica, 22, 23–36, January 1954. Berridge, K. C. & Robinson, T. E. 1998. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Research Reviews, 28, 309–369.
Cabanac, M. 1992. Pleasure: the common currency. Journal of Theoretical Biology, 155, 173–200. Clement, T. S., Feltus, J. R., Kaiser, D. H. & Zentall, T. R. 2000. ‘Work ethic’ in pigeons: reward value is directly related to the effort or time required to obtain the reward. Psychonomic Bulletin Review, 7, 100–106. Dawkins, R. & Carlisle, T. R. 1976. Parental investment, mate desertion and a fallacy. Nature, 161, 131–133. Houston, A. I. & McNamara, J. M. 1999. Models of Adaptive Behaviour: An Approach Based on State. Cambridge: Cambridge University Press. Kacelnik, A. & Brunner, D. 2002. Timing and foraging: Gibbon’s scalar expectancy theory and optimal patch exploitation. Learning and Motivation, 33, 177–195. Kacelnik, A. & Marsh, B. 2002. Cost can increase preference in starlings. Animal Behaviour, 63, 245–250. Killeen, P. R. 1984. Incentive theory III: adaptive clocks. In: Timing and Time Perception (Ed. by J. Gibbon & L. Allan), pp. 515–527. New York: New York Academy of Sciences. McFarland, D. J. & Sibly, R. M. 1975. The behavioural final common path. Philosophical Transactions of the Royal Society of London, Series B, 270, 265–293. McNamara, J. M. & Houston, A. I. 1982. Short-term behaviour and lifetime fitness. In: Functional Ontogeny (Ed. by D. J. McFarland), pp. 60–87. London: Pitman. Marsh, B. 1999. Making the best choice: judgement and strategic decision making under conditions of risk and uncertainty. D.Phil. thesis, University of Oxford. Marsh, B. & Kacelnik, A. 2002. Framing effects and risky decisions in starlings. Proceedings of the National Academy of Sciences, U.S.A., 99, 3352–3355. Marsh, B., Schuck-Paim, C. & Kacelnik, A. 2004. State-dependent learning affects foraging choices in starlings. Behavioral Ecology, 15, 396–399. Pompilio, L. 2004. Animal choice and the construction of preferences. D.Phil. thesis, University of Oxford. Thaler, R. 1980. Towards a positive theory of consumer choice. Journal of Economic Behaviour and Organization, 1, 39–60. Zar, J. M. 1999. Biostatistical Analysis, 4th edn. Englewood cliff: Prentice Hall.