- Email: [email protected]
Relative Time and Delay of Reinforcement
LEARNING AND MOTIVATION ARTICLE NO.
29, 236–248 (1998)
LM970999
Relative Time and Delay of Reinforcement Ben A. Williams University of California, San Diego Rats were trained on a conditional discrimination in which either a light or a noise cued whether the left or right lever was the correct choice. Faster learning occurred when shorter delays were imposed between presentation of the stimuli and the choice opportunity, and the effect of delay variation was dependent on the value of the intertrial interval (ITI): the effects of the retention interval were smaller with longer ITI’s. When the delays were inserted between the choice response and its consequence, main effects of delay and ITI occurred, but there was no interaction between them. The occurrence of relative time effects with delayed stimulus control, but not with delay-of-reinforcement procedures, suggests that different types of delay intervals depend upon different psychological processes. 1998 Academic Press
A finding of apparent generality in several different learning procedures is that behavior controlled by temporal intervals of various types is not determined by the absolute time values, but instead by the value of the temporal interval relative to the average interreinforcement interval in the situation. One example of this ‘‘relative time’’ effect is the rate of responding maintained by fixed-interval (FI) schedules. When response rates maintained by different Fl values are plotted in terms of the proportions of their maximum rates at various points within the interval (e.g., quartiles), these proportions overlap almost perfectly. That is, regardless of the absolute value of the FI, the proportion of the interval that will produce some fraction of the maximum rate is approximately constant (Dews, 1970). A second example of this relative time effect has been reported by Gibbon et al. (1977) with autoshaping. Although it had been well known that both the CS-US interval and the intertrial interval (ITI) have important effects on the rate of learning in Pavlovian conditioning, Gibbon et al. demonstrated that the ratio of the CS-US interval to the ITI, rather than the absolute value of either in isolation, was the controlling variable. This research was supported by NSF and NIMH grants to the University of California, San Diego. Reprint requests should be addressed to the author, Department of Psychology, University of California, San Diego, La Jolla, CA 92093-0109. E-mail: [email protected] 236 0023-9690/98 $25.00
Copyright 1998 by Academic Press All rights of reproduction in any form reserved.
REINFORCEMENT DELAY AND RELATIVE TIME
237
A powerful example of the interaction between ITI and delay interval has been reported by Kaplan (1984) in his analysis of the effect of different CSUS trace intervals in an autoshaping procedure. While the trace interval was held constant, Kaplan demonstrated that either conditioned excitation or conditioned inhibition could be produced by variation of the ITI from long to short. Balsam (1984) has elaborated on this effect and has argued that such interactions can be explained in terms of the same general model as that relating CS-US interval to ITI in the more typical autoshaping procedure in which the CS and US are temporally contiguous. Still other results consistent with the notion of relative time have been reported with a delayed matching-to-sample procedure, in which the effects of different delays between sample and choice stimuli were smaller with long ITI’s than with short ITI’s, with the critical variable being the ratio between the retention interval to the ITI (e.g., Roberts & Kraemer, 1982. Also see Wixted, 1989). It should be noted that this procedure involves utilization of a discriminative cue to control steady-state behavior rather than the acquisition of a new response. The diversity of procedures that have produced a relative time effect makes a strong case for its generality, and encourages an investigation of its applicability to still other procedures. A literature that invites such investigation is that involving how instrumental behavior is controlled by delayed reinforcement. Traditional treatments of this problem (e.g., Mowrer, 1960) have assumed that the temporal separation between the response and reinforcer weakens the association between them because the memory trace of the response decays regularly with increasing temporal separation. However, this interpretation seems simplistic given the enormous variability in the size of delay effects across different experiments. For example, Williams (1976) demonstrated that unsignalled delays as brief as three seconds reduced response rate maintained on a VI schedule by 70–90% with pigeons as subjects, while Lattal and Gleason (1990) demonstrated that pigeons could acquire a new response with no prior training with delays of 30 s between the response and reinforcer. A possible source of such varying sensitivity to delay effects is that the effects of response-reinforcer delays occur with respect to a relative, rather than an absolute scale. Thus, delayed reinforcers occurring in a context in which the frequency of reinforcement is very low will be more effective in strengthening behavior that when occurring in a context of more frequent reinforcement. To investigate this issue, a simple factorial design was used in which the delay of reinforcement was varied as one factor and the ITI was varied as the second orthogonal factor. The task was the acquisition of a simple two-choice discrimination. Rats were presented one of two stimuli, followed by a choice between one of two responses, with the correct response defined by which stimulus was presented on that trial. Because the conditioning preparation involves rats learning a conditional
238
BEN A. WILLIAMS
discrimination, and almost all demonstrations of relative-time effects have occurred with pigeons in different types of procedures, Experiment 1 was conducted to demonstrate relative-time effects involving stimulus-stimulus relations using our new preparation. Rats were presented the same conditional discrimination that was to be used in later experiments varying the response-reinforcer delay, but here the delay that was varied was between the discriminative stimulus and the response opportunity. At issue was whether the effects of this delay was affected by the size of the ITI. EXPERIMENT 1
Method Subjects. Forty male Sprague-Dawley albino rats, ranging in age from 3 to 12 months, served as subjects. All subjects had participated in other experiments of various types, but none had experience with the particular discriminative stimuli used here. All subjects were housed in individual cages with a 14 : 10-h light/dark cycle. Food deprivation was maintained by 90-min. access to standard lab chow approximately 5 min. following the experimental session. Water was continuously available in the home cage. Apparatus. A standard two-lever rat chamber, with glass side walls, sheet metal ceiling and front and rear walls, and a grid floor, was housed inside a sound-attenuating larger chamber equipped with a ventilating fan. The interior of the chamber was 30.5 m wide by 20.3 cm high by 22.9 cm long. Two retractable levers (BRS/LVE model RRL-015), which protruded 1.5 cm into the chamber when extended and which required a minimum force of 0.3 N for depression, were mounted on the front wall of the chamber, spaced 9 cm apart measured from side to side. The only feedback for a lever press was the sound of the microswitch inside the lever housing. Directly between and 6.5 cm below the levers was a food receptacle into which dropped 45mg Noyes pellets (improved Formula A) that served as the reinforcer. Between and 3.3. cm above the levers was a 28-V pilot light, covered by a glass translucent cover. In the center of the ceiling was mounted a speaker through which a 82-db white noise could be presented. Ambient noise level in the absence of the noise was 72 db. Procedure. The experiment was conducted in three separate replications, with the subjects within each replication having similar experimental histories, but with different histories for the different replications. Subjects within each replication were randomly assigned to each of the four experimental conditions. The procedure began with the ITI, after which a trial began with the onset of one or the other conditional cue. The white pilot light on the front panel of chamber indicated that the right lever was correct; the white noise indicated that the left lever was correct. The two cues were quasi-randomly alternated across trials. Each conditional cue was presented for 5 s, followed by
REINFORCEMENT DELAY AND RELATIVE TIME
239
FIG. 1. Proportion of choices that were correct for each of the four groups in Experiment 1, in which the delay interval was between the conditional cues and the choice opportunity.
the delay interval with the same stimulus conditions as during the ITI, and then the presentation of the two levers. The first response to either lever defined whether the trial was correct, and also withdrew the levers from the chamber. Correct responses were followed immediately by the food pellet and the onset of the ITI. Incorrect responses were followed only by the onset of the ITI. Four different experimental conditions were defined by the delay interval between the conditional cue and the choice opportunity and the ITI, according to a factorial design. Two groups received a 15-s ITI and two received a 45-s ITI. One half of each ITI assignment received a 2-s delay between the conditional cue and the choice and the remaining half received a 6-s delay. Training continued on each session for 100 trials. A total of 12 sessions was presented to each subject. Results Figure 1 shows the acquisition functions for the four experimental conditions. The most evident observation is that the group with the 6-s delay combined with the 15-s ITI acquired the discrimination more slowly than the other three groups. In addition, faster acquisition occurred with the 2-s delays than with the 6-s delays, for both ITI values, and faster acquisition also occurred with the 45-s ITI than with the 15-s ITI for both delay values. The results were analyzed with a mixed ANOVA, with replication, delay, and ITI as between-subject factors, and blocks of training a within-subject factor. The significance level for all comparisons was .05. The effect of replication was significant, F(2, 28) ⫽ 11.8, but replication did not interact with either delay or the ITI, and so will be ignored in further discussion. The
240
BEN A. WILLIAMS
effect of ITI was significant, F(1, 28) ⫽ 45.7, as was the effect of delay, F(1, 28) ⫽ 23.95. Most importantly, the interaction between the ITI and delay was significant, F(1, 28) ⫽ 8.63. The effect of blocks was significant, F(5, 140) ⫽ 341.0, as was the interaction between blocks and ITI, F(5, 140) ⫽ 6.03, and the interaction between blocks and delay, F(5, 140) ⫽ 4.95. The triple interaction between blocks, ITI, and delay approached but did not achieve significance, F(5, 140) ⫽ 1.96, p ⬍ .09. The pattern of results seen in Fig. 1 is potentially due in part to differences in the relative sizes of the main effects of delay versus ITI. The significant interaction argues against this interpretation, in that it shows that the size of a main effect was significantly influenced by the value of the other variable. Nevertheless, a more stringent test of the relative time account of the results shown in Figure 1 may be useful. The strongest version of that concept is that the rate of acquisition should be constant whenever the ratio of retention interval to ITI is constant, regardless of the absolute values of either variable. To test this strong version of the relative time hypothesis, a second analysis was conducted involving only the condition with the 2-s delay combined with the 15-s ITI, and the condition with the 6-s delay combined with the 45-s ITI. The main effect of the absolute time value was not significant, F(1, 14) ⫽ 1.39. The main effect of blocks was signficant, F(5, 70) ⫽ 150.1, while the interaction between the absolute time values and blocks was not significant, F(5, 70) ⫽ 1.78. Thus, given equal ratios of the retention interval to the ITI, there was no evidence that the absolute values of the temporal parameters played a significant role. Discussion The results shown in Fig. 1 extend the domain of relative-time effects by showing that the effect of the delay interval between the stimulus and choice opportunity varied systematically with the size of the ITI. Independent effects of delay and ITI occurred in their own right, but, in addition, the effects of delay were smaller with the longer ITI condition. The results are thus similar to those reported by other investigators using delayed matching to sample procedures with pigeon subjects (for a partial review see Wixted, 1989), although those procedures have not studied the acquisition of the discrimination but rather the utilization of the conditional cue in maintaining discrimination performance. Given that relative time effects do occur with the acquisition procedure used in the present study, the issue now addressed is whether comparable relative time effects occur with respect to the delay interval between the choice response and the response consequence. EXPERIMENT 2
The design of Experiment 2 was similar to that of Experiment 1 except for the location of the delay interval. Instead of the delay separating the discriminative cues and the choice, as in Experiment 1, here it separated
REINFORCEMENT DELAY AND RELATIVE TIME
241
the choice from the reinforcer. The same delay values and ITI’s were again repeated. The issue was whether the effect of the delay interval would depend upon the ITI value, as it did in Experiment 1. Method Subjects. Forty-eight albino rats, maintained in all respects as those in Experiment 1, and with similar histories, served as subjects. Apparatus. The same experimental chamber as in Experiment 1 was also used. Procedure. The procedure was again run in three separate replications, with subjects in each replication having different experimental histories. Subjects within each replication were again randomly assigned to the four experimental conditions. The procedure began with the ITI, followed by the onset of either the light or the noise at the start of the trial. Simultaneous with the onset of the discriminative cue the levers were extended into the chamber. Responses to either lever had no effect until a 4-s interval had elapsed, at which time the next response defined whether the trial was counted as correct or incorrect. Correct responses began both the next ITI and the delay interval to the reinforcer, which entailed that the delay-of-reinforcement interval was counted as part of the ITI. Incorrect responses began only the ITI. After either type of response, the levers were withdrawn until the start of the next trial. The same four conditions as in Experiment 1 were studied here with a factorial design of delay X ITI. The delay values were again 2 versus 6 s, while the ITI values were again 15 versus 45 s. Each session continued for 100 trials, and again 12 sessions were presented to all subjects. Results The acquisition functions for the four groups are shown in Fig. 2. Again, subjects with the shorter delay value appear to have learned the discrimination more rapidly, as did subjects with the longer ITI. The results were again analyzed with a mixed ANOVA. Preliminary analysis showed that the replication factor was not significant, so it was excluded from the analysis. The between-subject factors were the delay value and the ITI, while the withinsubject factor was blocks of training. The main effect of ITI was significant, F(1, 44) ⫽ 4.65, as was the main effect of delay, F(1, 44) ⫽ 11.29. The main effect of blocks was also significant, F(5, 220) ⫽ 113.9, as was the interaction between delay and blocks, F(5, 220) ⫽ 6.73. The interaction between ITI and blocks was not significant (F ⬍ 1). Most importantly, the interaction between ITI and delay was not significant (F ⬍ 1), nor was the triple interaction between ITI, delay, and blocks, F(5, 220) ⫽ 1.69. Thus, unlike Experiment 1, there was no evidence that the effect of the delay value depended on the ITI value.
242
BEN A. WILLIAMS
FIG. 2 Proportion of choices that were correct for each of the four groups in Experiment 2, in which the delay interval was between the choice and the outcome.
The critical evidence regarding the applicability of the relative time concept to delay of reinforcement is the absence of a significant interaction between the delay and ITI variables. Interactions in factorial designs may depend critically on the response scale employed, in that transformations of the dependent variable may yield significant interactions while the raw data do not. Hence it is important to establish the generality of the preceding statistical analysis. One potential problem with percentage correct as the dependent variable is that it is bounded at the upper end, which may have the possible effect of truncating the spread between the functions. To address this possibility, the results were transformed into logit p scores: log (proportion correct/proportion incorrect), which is unbounded, and the two-factor ANOVA was repeated. The results were entirely similar to those from the percentage correct measure. The main effect of ITI was significant, F(1, 43) ⫽ 4.23, as was the main effect of delay, F(1, 43) ⫽ 14.78, while the interaction was not significant, F ⬍ 1. The interaction between ITI and blocks was not significant, F(5, 215) ⫽ 1.64, while the interaction between delay and blocks was significant, F(5, 215) ⫽ 12.17. The triple interaction between ITI, delay, and blocks was again not significant, F(5, 215) ⫽ 1.65. As in Experiment 1, it is also possible to determine whether the absolute values of the ITI and delay parameters played a role when their ratio was constant. A separate ANOVA was conducted comparing the condition with the 45-s ITI with the 6-s delay, and the condition with the 15-s ITI with the 2-s delay. Here the main effect of the absolute time values was not significant, F(1, 22) ⬍ 1; the effect of blocks was significant F(5, 110) ⫽ 65.9, while the interaction between absolute time value and blocks was also significant, F(5, 110) ⫽ 2.43, p ⬍ .05. Examination of the results in Fig. 1 suggest that the basis of the significant interaction term was that the effects
REINFORCEMENT DELAY AND RELATIVE TIME
243
of delay were more potent than those of ITI in determining the rate of discrimination acquisition. Discussion Despite identical procedures except for the location of the delay interval, the pattern of results in Experiment 2 differed from that in Experiment 1, in that here the effect of the delay did not depend on the value of the ITI. In both experiments there were main effects of both ITI and delay interval, indicating sensitivity to both variables, but only in Experiment 1 was the size of the delay effect dependent on the ITI. The results thus suggest that a delay interval between the choice response and the reinforcer has different functional properties than the delay interval between the discriminative stimulus and the choice opportunity. Because this conclusion depends upon the acceptance of a null finding, however, it is important to replicate its generality. EXPERIMENT 3
Experiment 3 was a direct replication of Experiment 2 but with different parameter values. The delay values were here extended to 4 and 10 s, with the aim of exploring a portion of the delay-of-reinforcement parameter range that might cause considerable retardation of learning, thus allowing some assessment of whether different ITI’s had different effects in that range than with the shorter delays used in Experiment 2. The ITI values employed were 15 and 60 s. Method Subjects and apparatus. Twenty-four albino rats were maintained as described in Experiment 1. All had similar experimental histories to the subjects used in the earlier experiments. The same conditioning chamber was also used. Procedure. The procedure was identical with the exception of the different parameter values. Six subjects were assigned to each of four groups, defined by a factorial design with the 4- vs 10-s delays as one factor and the 15- vs 60-s ITI’s as the other factor. Sessions continued for 50 trials, in contrast to the 100 trials/session used in Experiments 1 and 2. Training continued for 20 sessions. Results Figure 3 shows the acquisition functions for the four groups of subjects. Once again the discrimination was learned most quickly by the group with the shorter delay in combination with the longer ITI, and slowest by the group with the longer delay combined with the shorter ITI. The results shown
244
BEN A. WILLIAMS
FIG. 3. Proportion of choices that were correct for the four groups in Experiment 3, in which the delay interval was between the choice and the outcome.
in Fig. 3 were analyzed with a two-factor ANOVA. The main effect of ITI was significant, F(1, 20) ⫽ 13.3, as was the main effect of delay, F(1, 20) ⫽ 8.74. The interaction between delay and ITI was not significant, F ⬍ 1. The interaction between ITI and blocks was not significant, F(9, 180) ⫽ 1.28, while the interaction between delay and blocks was significant, F(9, 180) ⫽ 5.99. The triple interaction was not significant, F ⬍ 1. The ANOVA was repeated after transforming the percentage correct measure into the logit p measure described in Experiment 2. The main effect of ITI was again significant, F(1, 20) ⫽ 16.96, as was the main effect of delay, F(1, 20) ⫽ 10.78. The interaction was again not significant, F ⬍ 1. The interaction between ITI and blocks was now significant, F(4, 80) ⫽ 4.99, as was the interaction between delay and blocks, F(4, 80) ⫽ 10.36. The triple interaction was again not significant (F ⬍ 1). Discussion The results of Experiment 3 replicated the pattern obtained in Experiment 2. The failure to find an interaction between the effects of delay and of the ITI thus does not depend on a limited set of parameters, although obviously the values employed here represent only a small portion of the possible values that could be used. The failure to find a significant interaction also seems unlikely to be due to poor sensitivity in the measures that were employed, in that both experiments found strong effects of the ITI and delay variables in their own right. Moreover, the F values for the interaction term were all less than 1, indicating little likelihood that using more subjects to increase the power of the statistical test would reveal a different outcome.
REINFORCEMENT DELAY AND RELATIVE TIME
245
GENERAL DISCUSSION
There are three major findings in the present series of experiments. First, the relative time effect, in which the effect of a given temporal interval on behavior is relative to the interreinforcement interval, has been extended to a new procedure in which the delay was varied between a discriminative stimulus and choice opportunity in the acquisition of a simultaneous discrimination. Previous demonstrations of such delayed stimulus control effects have occurred with the maintenance of behavior, rather than its acquisition. The present data thus provide still more evidence for the theoretical importance of the relative time principle for a wide variety of situations. The procedure of Experiment 1 shares some similarity with the occasionsetting procedure investigated by Holland (1995). In his study the interval between the feature and target was varied, as was the ITI. Longer ITI’s increased the rate of discrimination learning, as did shorter intervals between the feature and target. When conditions were compared as a function of their ratio of ITI/delay, the rate of discrimination acquisition was not constant for a given ratio value, indicating that the strong form of the ratio invariance effect did not hold. However, the interaction between the ITI and featuretarget interval was significant, indicating that the effects of the feature-target interval were at least partially dependent on the value of the ITI. The second major finding of the present study is that the relative time principle does not apply to one type of temporal interval—the delay between a response and its contingent reinforcer. In two different acquisition procedures, involving variation of different delay-of-reinforcement values, significant main effects of delay and of the interreinforcement interval were obtained, but these did not interact. In other words, the effect of delay was not relative to the interreinforcement interval. This failure to find a relative time effect with delay of reinforcement is surprising, given that such an effect was obtained with delays between the stimulus and response opportunity, as noted above, in the same procedure. The issue is how this dissociation is to be interpreted. One perspective on the difference between Experiment 1 vs Experiments 2 and 3 results from considering the memory requirements of the different tasks. In Experiment 1, the subject was reinforced immediately for its choice response, which meant that only the discriminative stimulus had to be retained in memory. In contrast, Experiments 2 and 3 required the subject to remember the combination of the discriminative stimulus and response selection. Unfortunately, there is no obvious reason why the different requirements should produce different patterns of interaction with the ITI. One hypothesis is that delays between the response and reinforcer may fail to show the relative time effect because the animal controls the time of response emission. Long response latencies after presentation of the levers
246
BEN A. WILLIAMS
were possible, and it was also possible for the rats to make incipient responses to the locations of the lever during the ITI even though the levers themselves were not actually present. The result should be a degradation of the signal value of the response as a temporal predictor of reinforcement especially if there is considerable variability of the temporal properties of the response. However, there was no obvious difference between Experiment 1 vs Experiments 2 and 3 in such variability. The second possible interpretation of the failure to find relative time effects with delay-of-reinforcement procedures is that real functional differences occur for different types of temporal intervals, such that delay of reinforcement cannot properly be conceptualized as simply the result of the association between the response and reinforcer in a fashion analogous to that between a stimulus and reinforcer. Various frameworks for why such differences should occur are available. One salient possibility is that delays of reinforcement between the response and reward produce both motivational and associative effects, and these different types of effects have different functional characteristics (cf., Cox & D’Amato, 1977). A second possibility is that time, per se, whether in relative or absolute terms, is not the critical determinant of the effectiveness of delay-of-reinforcement contingencies. Considerable evidence now exists that the effects of a given delay interval depend on the events occurring during the delay interval that compete with the response for association with the reinforcer (e.g., Williams, 1982; Williams et al., 1990), and there is no necessary reason that such ‘‘blocking’’ effects are functionally related to the average interreinforcement interval in the situation. The failure to obtain a relative time effect with delay of reinforcement is especially surprising given that other data do suggest that a given delay-ofreinforcement interval may have different effects depending on the reinforcement context. Dickinson, Watt, and Griffiths (1992) reported that a 64-s delay-of-reinforcement interval in a response acquisition procedure like that used by Lattal and Gleason (1990) produced no learning when instituted at the start of 30-min sessions, but did produce learning if the rats were exposed to the chamber with the response lever absent for 30 min prior to the session in which the lever and the delay contingency were made available. Whether this prior period of nonreinforcement facilitated acquisition by increasing the perceived interreinforcement interval, by reducing associative competition from other cues in the chamber, or by some other mechanism, is now uncertain. It should be noted that the present failure to find relative time effects with delay-of-reinforcement procedures does cause considerable difficulty for some recent accounts of relative time phenomena. Killeen (1994) has proposed a general theory to encompass many different findings in instrumental learning, based in part on the behavioral theory of timing (Killeen and Fetterman, 1988). The critical assumptions of this account are that reinforce-
REINFORCEMENT DELAY AND RELATIVE TIME
247
ment occurs with respect to whatever is present in the animal’s ‘‘memory window’’ at the time the reinforcer is presented, and that the rate at which events pass through the memory window is directly related to the rate of reinforcement in the situation. The result is that short-term memory declines more rapidly in situations of high reinforcement rate than of low reinforcement rates, which predicts the interaction between ITI and delay interval that was found in Experiment 1. However, the account has no basis for distinguishing between different types of short-term memory effects, and thus should predict a similar interaction in the delay-of-reinforcement procedures as well, contrary to the obtained results. The third finding of the present experiments that deserves attention is the consistent finding of substantial ITI effects in the simultaneous discrimination procedures. Acquisition occurred more rapidly with longer ITI’s in all of the experiments, quite apart from the issue of the interaction between ITI and delay interval. This finding is noteworthy because ITI effects have not generally occurred with respect to the acquisition of simple simultaneous discriminations (e.g., Biederman, 1967; also see Williams, 1977). Effects of ITI have been reported with conditional discriminations like those used here, but primarily with respect to the maintenance of performance, not the initial acquisition of the discrimination. Holt and Shafer (1973) did compare the effects of ITI values ranging from 0 to 60 s on the acquisition of simultaneous matching to sample, and reported that such acquisition was greatly retarded with a 0-s ITI but learned similarly under all of the other values (5, 15, 25, and 60 s). However, only two subjects per condition were studied under each ITI condition, so it is uncertain whether greater effects would have occurred with a more sensitive assessment. In any event, the present results show strong effects of differences between substantially longer ITI’s. Whether this effect is specific to procedures involving delay of reinforcement remains to be determined. REFERENCES Balsam, P. (1984). Relative time in trace conditioning. Annals of the New York Academy of Sciences, 423, 211–227. Biederman, G. (1967). Simultaneous discrimination: Parameters of reinforcement and ITI. Psychonomic Science, 8, 215–216. Cox, J. K., & D’Amato, M. R. (1977). Disruption of overlearned discriminative behavior in monkeys (Cebus appella) by delay of reward. Animal Learning & Behavior, 5, 93–98. Dews, P. B. (1970). The theory of fixed-interval responding. In W. N. Schoenfeld (Ed.). The theory of reinforcement schedules. New York: Appleton-Century-Crofts. Dickinson, A., Watt, A., & Griffiths, W. J. H. (1992). Free-operant acquisition with delayed reinforcement. Quarterly Journal of Experimental Psychology, 45B, 241–258. Gibbon, J., Baldock, M. D., Locurto, C., Gold, L., & Terrace, H. S. (1977). Trial and intertrial durations in autoshaping. Journal of Experimental Psychology: Animal Behavior Processes, 3, 264–284. Gibbon, J., & Balsam, P. (1981). Spreading association in time. In C. M. Locurto, H. S.
248
BEN A. WILLIAMS
Terrace, & J. Gibbon (Eds.), Autoshaping and conditioning theory (pp. 219–253). New York: Academic Press. Holland, P. C. (1995). The effects of intertrial and feature-target intervals on operant serial feature-positive discrimination learning. Animal Learning & Behavior, 23, 411–428. Holt, G. L., & Shafer, J. N. (1973). Function of intertrial interval in matching to sample. Journal of the Experimental Analysis of Behavior, 19, 181–186. Kaplan, P. S. (1984). The importance of relative temporal parameters in trace autoshaping: From excitation to inhibition. Journal of Experimental Psychology: Animal Behavior Processes, 10, 113–126. Killeen, P. (1994). Mathematical principles of reinforcement based on the correlation of behavior with incentives in short-term memory. Behavioral and Brain Sciences, 17, 105–172. Killeen, P., & Fetterman, G. (1988). A behavioral theory of timing. Psychological Review, 95, 274–295. Lattal, K. A., & Gleeson, S. (1990). Response acquisition with delayed reinforcement. Journal of Experimental Psychology: Animal Behavior Processes, 16, 27–39. Mowrer, O. H. (1960). Learning theory and behavior. New York: John Wiley & Sons. Roberts, W. A., & Kraemer, P. J. (1982). Some observations of the effect of intertrial interval and delay on delayed matching to sample in pigeons. Journal of Experimental Psychology: Animal Behavior Processes, 8, 342, 353. Williams, B. A. (1976). The effects of unsignalled delayed reinforcement. Journal of the Experimental Analysis of Behavior, 26, 441–449. Williams, B. A. (1977). Contrast effects in simultaneous discrimination learning. Animal Learning & Behavior, 5, 47–50. Williams, B. A. (1982). Blocking the response-reinforcer association. In M. Commons, R. Herrnstein, & A. Wagner (Eds.). Quantitative Analyses of Behavior: Volume 3. Acquisition (pp. 427–445). Cambridge, MA: Ballinger Publishing Company. Williams, B. A., Preston, R. A., & DeKervor, D. (1990). Blocking of the response-reinforcer association: Additional evidence. Learning and Motivation, 21, 379–398. Wixted, J. T. (1989). Nonhuman short-term memory: a quantitative reanalysis of selected findings. Journal of the Experimental Analysis of Behavior, 52, 409–426. Received September 12, 1997 Revised December 9, 1997
29, 236–248 (1998)
LM970999
Relative Time and Delay of Reinforcement Ben A. Williams University of California, San Diego Rats were trained on a conditional discrimination in which either a light or a noise cued whether the left or right lever was the correct choice. Faster learning occurred when shorter delays were imposed between presentation of the stimuli and the choice opportunity, and the effect of delay variation was dependent on the value of the intertrial interval (ITI): the effects of the retention interval were smaller with longer ITI’s. When the delays were inserted between the choice response and its consequence, main effects of delay and ITI occurred, but there was no interaction between them. The occurrence of relative time effects with delayed stimulus control, but not with delay-of-reinforcement procedures, suggests that different types of delay intervals depend upon different psychological processes. 1998 Academic Press
A finding of apparent generality in several different learning procedures is that behavior controlled by temporal intervals of various types is not determined by the absolute time values, but instead by the value of the temporal interval relative to the average interreinforcement interval in the situation. One example of this ‘‘relative time’’ effect is the rate of responding maintained by fixed-interval (FI) schedules. When response rates maintained by different Fl values are plotted in terms of the proportions of their maximum rates at various points within the interval (e.g., quartiles), these proportions overlap almost perfectly. That is, regardless of the absolute value of the FI, the proportion of the interval that will produce some fraction of the maximum rate is approximately constant (Dews, 1970). A second example of this relative time effect has been reported by Gibbon et al. (1977) with autoshaping. Although it had been well known that both the CS-US interval and the intertrial interval (ITI) have important effects on the rate of learning in Pavlovian conditioning, Gibbon et al. demonstrated that the ratio of the CS-US interval to the ITI, rather than the absolute value of either in isolation, was the controlling variable. This research was supported by NSF and NIMH grants to the University of California, San Diego. Reprint requests should be addressed to the author, Department of Psychology, University of California, San Diego, La Jolla, CA 92093-0109. E-mail: [email protected] 236 0023-9690/98 $25.00
Copyright 1998 by Academic Press All rights of reproduction in any form reserved.
REINFORCEMENT DELAY AND RELATIVE TIME
237
A powerful example of the interaction between ITI and delay interval has been reported by Kaplan (1984) in his analysis of the effect of different CSUS trace intervals in an autoshaping procedure. While the trace interval was held constant, Kaplan demonstrated that either conditioned excitation or conditioned inhibition could be produced by variation of the ITI from long to short. Balsam (1984) has elaborated on this effect and has argued that such interactions can be explained in terms of the same general model as that relating CS-US interval to ITI in the more typical autoshaping procedure in which the CS and US are temporally contiguous. Still other results consistent with the notion of relative time have been reported with a delayed matching-to-sample procedure, in which the effects of different delays between sample and choice stimuli were smaller with long ITI’s than with short ITI’s, with the critical variable being the ratio between the retention interval to the ITI (e.g., Roberts & Kraemer, 1982. Also see Wixted, 1989). It should be noted that this procedure involves utilization of a discriminative cue to control steady-state behavior rather than the acquisition of a new response. The diversity of procedures that have produced a relative time effect makes a strong case for its generality, and encourages an investigation of its applicability to still other procedures. A literature that invites such investigation is that involving how instrumental behavior is controlled by delayed reinforcement. Traditional treatments of this problem (e.g., Mowrer, 1960) have assumed that the temporal separation between the response and reinforcer weakens the association between them because the memory trace of the response decays regularly with increasing temporal separation. However, this interpretation seems simplistic given the enormous variability in the size of delay effects across different experiments. For example, Williams (1976) demonstrated that unsignalled delays as brief as three seconds reduced response rate maintained on a VI schedule by 70–90% with pigeons as subjects, while Lattal and Gleason (1990) demonstrated that pigeons could acquire a new response with no prior training with delays of 30 s between the response and reinforcer. A possible source of such varying sensitivity to delay effects is that the effects of response-reinforcer delays occur with respect to a relative, rather than an absolute scale. Thus, delayed reinforcers occurring in a context in which the frequency of reinforcement is very low will be more effective in strengthening behavior that when occurring in a context of more frequent reinforcement. To investigate this issue, a simple factorial design was used in which the delay of reinforcement was varied as one factor and the ITI was varied as the second orthogonal factor. The task was the acquisition of a simple two-choice discrimination. Rats were presented one of two stimuli, followed by a choice between one of two responses, with the correct response defined by which stimulus was presented on that trial. Because the conditioning preparation involves rats learning a conditional
238
BEN A. WILLIAMS
discrimination, and almost all demonstrations of relative-time effects have occurred with pigeons in different types of procedures, Experiment 1 was conducted to demonstrate relative-time effects involving stimulus-stimulus relations using our new preparation. Rats were presented the same conditional discrimination that was to be used in later experiments varying the response-reinforcer delay, but here the delay that was varied was between the discriminative stimulus and the response opportunity. At issue was whether the effects of this delay was affected by the size of the ITI. EXPERIMENT 1
Method Subjects. Forty male Sprague-Dawley albino rats, ranging in age from 3 to 12 months, served as subjects. All subjects had participated in other experiments of various types, but none had experience with the particular discriminative stimuli used here. All subjects were housed in individual cages with a 14 : 10-h light/dark cycle. Food deprivation was maintained by 90-min. access to standard lab chow approximately 5 min. following the experimental session. Water was continuously available in the home cage. Apparatus. A standard two-lever rat chamber, with glass side walls, sheet metal ceiling and front and rear walls, and a grid floor, was housed inside a sound-attenuating larger chamber equipped with a ventilating fan. The interior of the chamber was 30.5 m wide by 20.3 cm high by 22.9 cm long. Two retractable levers (BRS/LVE model RRL-015), which protruded 1.5 cm into the chamber when extended and which required a minimum force of 0.3 N for depression, were mounted on the front wall of the chamber, spaced 9 cm apart measured from side to side. The only feedback for a lever press was the sound of the microswitch inside the lever housing. Directly between and 6.5 cm below the levers was a food receptacle into which dropped 45mg Noyes pellets (improved Formula A) that served as the reinforcer. Between and 3.3. cm above the levers was a 28-V pilot light, covered by a glass translucent cover. In the center of the ceiling was mounted a speaker through which a 82-db white noise could be presented. Ambient noise level in the absence of the noise was 72 db. Procedure. The experiment was conducted in three separate replications, with the subjects within each replication having similar experimental histories, but with different histories for the different replications. Subjects within each replication were randomly assigned to each of the four experimental conditions. The procedure began with the ITI, after which a trial began with the onset of one or the other conditional cue. The white pilot light on the front panel of chamber indicated that the right lever was correct; the white noise indicated that the left lever was correct. The two cues were quasi-randomly alternated across trials. Each conditional cue was presented for 5 s, followed by
REINFORCEMENT DELAY AND RELATIVE TIME
239
FIG. 1. Proportion of choices that were correct for each of the four groups in Experiment 1, in which the delay interval was between the conditional cues and the choice opportunity.
the delay interval with the same stimulus conditions as during the ITI, and then the presentation of the two levers. The first response to either lever defined whether the trial was correct, and also withdrew the levers from the chamber. Correct responses were followed immediately by the food pellet and the onset of the ITI. Incorrect responses were followed only by the onset of the ITI. Four different experimental conditions were defined by the delay interval between the conditional cue and the choice opportunity and the ITI, according to a factorial design. Two groups received a 15-s ITI and two received a 45-s ITI. One half of each ITI assignment received a 2-s delay between the conditional cue and the choice and the remaining half received a 6-s delay. Training continued on each session for 100 trials. A total of 12 sessions was presented to each subject. Results Figure 1 shows the acquisition functions for the four experimental conditions. The most evident observation is that the group with the 6-s delay combined with the 15-s ITI acquired the discrimination more slowly than the other three groups. In addition, faster acquisition occurred with the 2-s delays than with the 6-s delays, for both ITI values, and faster acquisition also occurred with the 45-s ITI than with the 15-s ITI for both delay values. The results were analyzed with a mixed ANOVA, with replication, delay, and ITI as between-subject factors, and blocks of training a within-subject factor. The significance level for all comparisons was .05. The effect of replication was significant, F(2, 28) ⫽ 11.8, but replication did not interact with either delay or the ITI, and so will be ignored in further discussion. The
240
BEN A. WILLIAMS
effect of ITI was significant, F(1, 28) ⫽ 45.7, as was the effect of delay, F(1, 28) ⫽ 23.95. Most importantly, the interaction between the ITI and delay was significant, F(1, 28) ⫽ 8.63. The effect of blocks was significant, F(5, 140) ⫽ 341.0, as was the interaction between blocks and ITI, F(5, 140) ⫽ 6.03, and the interaction between blocks and delay, F(5, 140) ⫽ 4.95. The triple interaction between blocks, ITI, and delay approached but did not achieve significance, F(5, 140) ⫽ 1.96, p ⬍ .09. The pattern of results seen in Fig. 1 is potentially due in part to differences in the relative sizes of the main effects of delay versus ITI. The significant interaction argues against this interpretation, in that it shows that the size of a main effect was significantly influenced by the value of the other variable. Nevertheless, a more stringent test of the relative time account of the results shown in Figure 1 may be useful. The strongest version of that concept is that the rate of acquisition should be constant whenever the ratio of retention interval to ITI is constant, regardless of the absolute values of either variable. To test this strong version of the relative time hypothesis, a second analysis was conducted involving only the condition with the 2-s delay combined with the 15-s ITI, and the condition with the 6-s delay combined with the 45-s ITI. The main effect of the absolute time value was not significant, F(1, 14) ⫽ 1.39. The main effect of blocks was signficant, F(5, 70) ⫽ 150.1, while the interaction between the absolute time values and blocks was not significant, F(5, 70) ⫽ 1.78. Thus, given equal ratios of the retention interval to the ITI, there was no evidence that the absolute values of the temporal parameters played a significant role. Discussion The results shown in Fig. 1 extend the domain of relative-time effects by showing that the effect of the delay interval between the stimulus and choice opportunity varied systematically with the size of the ITI. Independent effects of delay and ITI occurred in their own right, but, in addition, the effects of delay were smaller with the longer ITI condition. The results are thus similar to those reported by other investigators using delayed matching to sample procedures with pigeon subjects (for a partial review see Wixted, 1989), although those procedures have not studied the acquisition of the discrimination but rather the utilization of the conditional cue in maintaining discrimination performance. Given that relative time effects do occur with the acquisition procedure used in the present study, the issue now addressed is whether comparable relative time effects occur with respect to the delay interval between the choice response and the response consequence. EXPERIMENT 2
The design of Experiment 2 was similar to that of Experiment 1 except for the location of the delay interval. Instead of the delay separating the discriminative cues and the choice, as in Experiment 1, here it separated
REINFORCEMENT DELAY AND RELATIVE TIME
241
the choice from the reinforcer. The same delay values and ITI’s were again repeated. The issue was whether the effect of the delay interval would depend upon the ITI value, as it did in Experiment 1. Method Subjects. Forty-eight albino rats, maintained in all respects as those in Experiment 1, and with similar histories, served as subjects. Apparatus. The same experimental chamber as in Experiment 1 was also used. Procedure. The procedure was again run in three separate replications, with subjects in each replication having different experimental histories. Subjects within each replication were again randomly assigned to the four experimental conditions. The procedure began with the ITI, followed by the onset of either the light or the noise at the start of the trial. Simultaneous with the onset of the discriminative cue the levers were extended into the chamber. Responses to either lever had no effect until a 4-s interval had elapsed, at which time the next response defined whether the trial was counted as correct or incorrect. Correct responses began both the next ITI and the delay interval to the reinforcer, which entailed that the delay-of-reinforcement interval was counted as part of the ITI. Incorrect responses began only the ITI. After either type of response, the levers were withdrawn until the start of the next trial. The same four conditions as in Experiment 1 were studied here with a factorial design of delay X ITI. The delay values were again 2 versus 6 s, while the ITI values were again 15 versus 45 s. Each session continued for 100 trials, and again 12 sessions were presented to all subjects. Results The acquisition functions for the four groups are shown in Fig. 2. Again, subjects with the shorter delay value appear to have learned the discrimination more rapidly, as did subjects with the longer ITI. The results were again analyzed with a mixed ANOVA. Preliminary analysis showed that the replication factor was not significant, so it was excluded from the analysis. The between-subject factors were the delay value and the ITI, while the withinsubject factor was blocks of training. The main effect of ITI was significant, F(1, 44) ⫽ 4.65, as was the main effect of delay, F(1, 44) ⫽ 11.29. The main effect of blocks was also significant, F(5, 220) ⫽ 113.9, as was the interaction between delay and blocks, F(5, 220) ⫽ 6.73. The interaction between ITI and blocks was not significant (F ⬍ 1). Most importantly, the interaction between ITI and delay was not significant (F ⬍ 1), nor was the triple interaction between ITI, delay, and blocks, F(5, 220) ⫽ 1.69. Thus, unlike Experiment 1, there was no evidence that the effect of the delay value depended on the ITI value.
242
BEN A. WILLIAMS
FIG. 2 Proportion of choices that were correct for each of the four groups in Experiment 2, in which the delay interval was between the choice and the outcome.
The critical evidence regarding the applicability of the relative time concept to delay of reinforcement is the absence of a significant interaction between the delay and ITI variables. Interactions in factorial designs may depend critically on the response scale employed, in that transformations of the dependent variable may yield significant interactions while the raw data do not. Hence it is important to establish the generality of the preceding statistical analysis. One potential problem with percentage correct as the dependent variable is that it is bounded at the upper end, which may have the possible effect of truncating the spread between the functions. To address this possibility, the results were transformed into logit p scores: log (proportion correct/proportion incorrect), which is unbounded, and the two-factor ANOVA was repeated. The results were entirely similar to those from the percentage correct measure. The main effect of ITI was significant, F(1, 43) ⫽ 4.23, as was the main effect of delay, F(1, 43) ⫽ 14.78, while the interaction was not significant, F ⬍ 1. The interaction between ITI and blocks was not significant, F(5, 215) ⫽ 1.64, while the interaction between delay and blocks was significant, F(5, 215) ⫽ 12.17. The triple interaction between ITI, delay, and blocks was again not significant, F(5, 215) ⫽ 1.65. As in Experiment 1, it is also possible to determine whether the absolute values of the ITI and delay parameters played a role when their ratio was constant. A separate ANOVA was conducted comparing the condition with the 45-s ITI with the 6-s delay, and the condition with the 15-s ITI with the 2-s delay. Here the main effect of the absolute time values was not significant, F(1, 22) ⬍ 1; the effect of blocks was significant F(5, 110) ⫽ 65.9, while the interaction between absolute time value and blocks was also significant, F(5, 110) ⫽ 2.43, p ⬍ .05. Examination of the results in Fig. 1 suggest that the basis of the significant interaction term was that the effects
REINFORCEMENT DELAY AND RELATIVE TIME
243
of delay were more potent than those of ITI in determining the rate of discrimination acquisition. Discussion Despite identical procedures except for the location of the delay interval, the pattern of results in Experiment 2 differed from that in Experiment 1, in that here the effect of the delay did not depend on the value of the ITI. In both experiments there were main effects of both ITI and delay interval, indicating sensitivity to both variables, but only in Experiment 1 was the size of the delay effect dependent on the ITI. The results thus suggest that a delay interval between the choice response and the reinforcer has different functional properties than the delay interval between the discriminative stimulus and the choice opportunity. Because this conclusion depends upon the acceptance of a null finding, however, it is important to replicate its generality. EXPERIMENT 3
Experiment 3 was a direct replication of Experiment 2 but with different parameter values. The delay values were here extended to 4 and 10 s, with the aim of exploring a portion of the delay-of-reinforcement parameter range that might cause considerable retardation of learning, thus allowing some assessment of whether different ITI’s had different effects in that range than with the shorter delays used in Experiment 2. The ITI values employed were 15 and 60 s. Method Subjects and apparatus. Twenty-four albino rats were maintained as described in Experiment 1. All had similar experimental histories to the subjects used in the earlier experiments. The same conditioning chamber was also used. Procedure. The procedure was identical with the exception of the different parameter values. Six subjects were assigned to each of four groups, defined by a factorial design with the 4- vs 10-s delays as one factor and the 15- vs 60-s ITI’s as the other factor. Sessions continued for 50 trials, in contrast to the 100 trials/session used in Experiments 1 and 2. Training continued for 20 sessions. Results Figure 3 shows the acquisition functions for the four groups of subjects. Once again the discrimination was learned most quickly by the group with the shorter delay in combination with the longer ITI, and slowest by the group with the longer delay combined with the shorter ITI. The results shown
244
BEN A. WILLIAMS
FIG. 3. Proportion of choices that were correct for the four groups in Experiment 3, in which the delay interval was between the choice and the outcome.
in Fig. 3 were analyzed with a two-factor ANOVA. The main effect of ITI was significant, F(1, 20) ⫽ 13.3, as was the main effect of delay, F(1, 20) ⫽ 8.74. The interaction between delay and ITI was not significant, F ⬍ 1. The interaction between ITI and blocks was not significant, F(9, 180) ⫽ 1.28, while the interaction between delay and blocks was significant, F(9, 180) ⫽ 5.99. The triple interaction was not significant, F ⬍ 1. The ANOVA was repeated after transforming the percentage correct measure into the logit p measure described in Experiment 2. The main effect of ITI was again significant, F(1, 20) ⫽ 16.96, as was the main effect of delay, F(1, 20) ⫽ 10.78. The interaction was again not significant, F ⬍ 1. The interaction between ITI and blocks was now significant, F(4, 80) ⫽ 4.99, as was the interaction between delay and blocks, F(4, 80) ⫽ 10.36. The triple interaction was again not significant (F ⬍ 1). Discussion The results of Experiment 3 replicated the pattern obtained in Experiment 2. The failure to find an interaction between the effects of delay and of the ITI thus does not depend on a limited set of parameters, although obviously the values employed here represent only a small portion of the possible values that could be used. The failure to find a significant interaction also seems unlikely to be due to poor sensitivity in the measures that were employed, in that both experiments found strong effects of the ITI and delay variables in their own right. Moreover, the F values for the interaction term were all less than 1, indicating little likelihood that using more subjects to increase the power of the statistical test would reveal a different outcome.
REINFORCEMENT DELAY AND RELATIVE TIME
245
GENERAL DISCUSSION
There are three major findings in the present series of experiments. First, the relative time effect, in which the effect of a given temporal interval on behavior is relative to the interreinforcement interval, has been extended to a new procedure in which the delay was varied between a discriminative stimulus and choice opportunity in the acquisition of a simultaneous discrimination. Previous demonstrations of such delayed stimulus control effects have occurred with the maintenance of behavior, rather than its acquisition. The present data thus provide still more evidence for the theoretical importance of the relative time principle for a wide variety of situations. The procedure of Experiment 1 shares some similarity with the occasionsetting procedure investigated by Holland (1995). In his study the interval between the feature and target was varied, as was the ITI. Longer ITI’s increased the rate of discrimination learning, as did shorter intervals between the feature and target. When conditions were compared as a function of their ratio of ITI/delay, the rate of discrimination acquisition was not constant for a given ratio value, indicating that the strong form of the ratio invariance effect did not hold. However, the interaction between the ITI and featuretarget interval was significant, indicating that the effects of the feature-target interval were at least partially dependent on the value of the ITI. The second major finding of the present study is that the relative time principle does not apply to one type of temporal interval—the delay between a response and its contingent reinforcer. In two different acquisition procedures, involving variation of different delay-of-reinforcement values, significant main effects of delay and of the interreinforcement interval were obtained, but these did not interact. In other words, the effect of delay was not relative to the interreinforcement interval. This failure to find a relative time effect with delay of reinforcement is surprising, given that such an effect was obtained with delays between the stimulus and response opportunity, as noted above, in the same procedure. The issue is how this dissociation is to be interpreted. One perspective on the difference between Experiment 1 vs Experiments 2 and 3 results from considering the memory requirements of the different tasks. In Experiment 1, the subject was reinforced immediately for its choice response, which meant that only the discriminative stimulus had to be retained in memory. In contrast, Experiments 2 and 3 required the subject to remember the combination of the discriminative stimulus and response selection. Unfortunately, there is no obvious reason why the different requirements should produce different patterns of interaction with the ITI. One hypothesis is that delays between the response and reinforcer may fail to show the relative time effect because the animal controls the time of response emission. Long response latencies after presentation of the levers
246
BEN A. WILLIAMS
were possible, and it was also possible for the rats to make incipient responses to the locations of the lever during the ITI even though the levers themselves were not actually present. The result should be a degradation of the signal value of the response as a temporal predictor of reinforcement especially if there is considerable variability of the temporal properties of the response. However, there was no obvious difference between Experiment 1 vs Experiments 2 and 3 in such variability. The second possible interpretation of the failure to find relative time effects with delay-of-reinforcement procedures is that real functional differences occur for different types of temporal intervals, such that delay of reinforcement cannot properly be conceptualized as simply the result of the association between the response and reinforcer in a fashion analogous to that between a stimulus and reinforcer. Various frameworks for why such differences should occur are available. One salient possibility is that delays of reinforcement between the response and reward produce both motivational and associative effects, and these different types of effects have different functional characteristics (cf., Cox & D’Amato, 1977). A second possibility is that time, per se, whether in relative or absolute terms, is not the critical determinant of the effectiveness of delay-of-reinforcement contingencies. Considerable evidence now exists that the effects of a given delay interval depend on the events occurring during the delay interval that compete with the response for association with the reinforcer (e.g., Williams, 1982; Williams et al., 1990), and there is no necessary reason that such ‘‘blocking’’ effects are functionally related to the average interreinforcement interval in the situation. The failure to obtain a relative time effect with delay of reinforcement is especially surprising given that other data do suggest that a given delay-ofreinforcement interval may have different effects depending on the reinforcement context. Dickinson, Watt, and Griffiths (1992) reported that a 64-s delay-of-reinforcement interval in a response acquisition procedure like that used by Lattal and Gleason (1990) produced no learning when instituted at the start of 30-min sessions, but did produce learning if the rats were exposed to the chamber with the response lever absent for 30 min prior to the session in which the lever and the delay contingency were made available. Whether this prior period of nonreinforcement facilitated acquisition by increasing the perceived interreinforcement interval, by reducing associative competition from other cues in the chamber, or by some other mechanism, is now uncertain. It should be noted that the present failure to find relative time effects with delay-of-reinforcement procedures does cause considerable difficulty for some recent accounts of relative time phenomena. Killeen (1994) has proposed a general theory to encompass many different findings in instrumental learning, based in part on the behavioral theory of timing (Killeen and Fetterman, 1988). The critical assumptions of this account are that reinforce-
REINFORCEMENT DELAY AND RELATIVE TIME
247
ment occurs with respect to whatever is present in the animal’s ‘‘memory window’’ at the time the reinforcer is presented, and that the rate at which events pass through the memory window is directly related to the rate of reinforcement in the situation. The result is that short-term memory declines more rapidly in situations of high reinforcement rate than of low reinforcement rates, which predicts the interaction between ITI and delay interval that was found in Experiment 1. However, the account has no basis for distinguishing between different types of short-term memory effects, and thus should predict a similar interaction in the delay-of-reinforcement procedures as well, contrary to the obtained results. The third finding of the present experiments that deserves attention is the consistent finding of substantial ITI effects in the simultaneous discrimination procedures. Acquisition occurred more rapidly with longer ITI’s in all of the experiments, quite apart from the issue of the interaction between ITI and delay interval. This finding is noteworthy because ITI effects have not generally occurred with respect to the acquisition of simple simultaneous discriminations (e.g., Biederman, 1967; also see Williams, 1977). Effects of ITI have been reported with conditional discriminations like those used here, but primarily with respect to the maintenance of performance, not the initial acquisition of the discrimination. Holt and Shafer (1973) did compare the effects of ITI values ranging from 0 to 60 s on the acquisition of simultaneous matching to sample, and reported that such acquisition was greatly retarded with a 0-s ITI but learned similarly under all of the other values (5, 15, 25, and 60 s). However, only two subjects per condition were studied under each ITI condition, so it is uncertain whether greater effects would have occurred with a more sensitive assessment. In any event, the present results show strong effects of differences between substantially longer ITI’s. Whether this effect is specific to procedures involving delay of reinforcement remains to be determined. REFERENCES Balsam, P. (1984). Relative time in trace conditioning. Annals of the New York Academy of Sciences, 423, 211–227. Biederman, G. (1967). Simultaneous discrimination: Parameters of reinforcement and ITI. Psychonomic Science, 8, 215–216. Cox, J. K., & D’Amato, M. R. (1977). Disruption of overlearned discriminative behavior in monkeys (Cebus appella) by delay of reward. Animal Learning & Behavior, 5, 93–98. Dews, P. B. (1970). The theory of fixed-interval responding. In W. N. Schoenfeld (Ed.). The theory of reinforcement schedules. New York: Appleton-Century-Crofts. Dickinson, A., Watt, A., & Griffiths, W. J. H. (1992). Free-operant acquisition with delayed reinforcement. Quarterly Journal of Experimental Psychology, 45B, 241–258. Gibbon, J., Baldock, M. D., Locurto, C., Gold, L., & Terrace, H. S. (1977). Trial and intertrial durations in autoshaping. Journal of Experimental Psychology: Animal Behavior Processes, 3, 264–284. Gibbon, J., & Balsam, P. (1981). Spreading association in time. In C. M. Locurto, H. S.
248
BEN A. WILLIAMS
Terrace, & J. Gibbon (Eds.), Autoshaping and conditioning theory (pp. 219–253). New York: Academic Press. Holland, P. C. (1995). The effects of intertrial and feature-target intervals on operant serial feature-positive discrimination learning. Animal Learning & Behavior, 23, 411–428. Holt, G. L., & Shafer, J. N. (1973). Function of intertrial interval in matching to sample. Journal of the Experimental Analysis of Behavior, 19, 181–186. Kaplan, P. S. (1984). The importance of relative temporal parameters in trace autoshaping: From excitation to inhibition. Journal of Experimental Psychology: Animal Behavior Processes, 10, 113–126. Killeen, P. (1994). Mathematical principles of reinforcement based on the correlation of behavior with incentives in short-term memory. Behavioral and Brain Sciences, 17, 105–172. Killeen, P., & Fetterman, G. (1988). A behavioral theory of timing. Psychological Review, 95, 274–295. Lattal, K. A., & Gleeson, S. (1990). Response acquisition with delayed reinforcement. Journal of Experimental Psychology: Animal Behavior Processes, 16, 27–39. Mowrer, O. H. (1960). Learning theory and behavior. New York: John Wiley & Sons. Roberts, W. A., & Kraemer, P. J. (1982). Some observations of the effect of intertrial interval and delay on delayed matching to sample in pigeons. Journal of Experimental Psychology: Animal Behavior Processes, 8, 342, 353. Williams, B. A. (1976). The effects of unsignalled delayed reinforcement. Journal of the Experimental Analysis of Behavior, 26, 441–449. Williams, B. A. (1977). Contrast effects in simultaneous discrimination learning. Animal Learning & Behavior, 5, 47–50. Williams, B. A. (1982). Blocking the response-reinforcer association. In M. Commons, R. Herrnstein, & A. Wagner (Eds.). Quantitative Analyses of Behavior: Volume 3. Acquisition (pp. 427–445). Cambridge, MA: Ballinger Publishing Company. Williams, B. A., Preston, R. A., & DeKervor, D. (1990). Blocking of the response-reinforcer association: Additional evidence. Learning and Motivation, 21, 379–398. Wixted, J. T. (1989). Nonhuman short-term memory: a quantitative reanalysis of selected findings. Journal of the Experimental Analysis of Behavior, 52, 409–426. Received September 12, 1997 Revised December 9, 1997