- Email: [email protected]
The costs of changing an intended action: Movement planning, but not execution, interferes with verbal working memory
Neuroscience Letters 509 (2012) 82–86
Contents lists available at SciVerse ScienceDirect
Neuroscience Letters journal homepage: www.elsevier.com/locate/neulet
The costs of changing an intended action: Movement planning, but not execution, interferes with verbal working memory M.A. Spiegel a,b,∗ , D. Koester a,b , M. Weigelt c , T. Schack a,b,d a
Neurocognition and Action – Biomechanics Research Group, Bielefeld University, Germany Center of Excellence – Cognitive Interaction Technology (CITEC), Bielefeld University, Germany Sport Psychology Group, Department of Sport and Health, University of Paderborn, Germany d CoR-Lab Research Institute for Cognition and Robotics, Bielefeld University, Germany b c
a r t i c l e
i n f o
Article history: Received 12 October 2011 Received in revised form 4 December 2011 Accepted 17 December 2011 Keywords: Verbal working memory Dual-task Action planning Action modification Grasping
a b s t r a c t How much cognitive effort does it take to change a movement plan? In previous studies, it has been shown that humans plan and represent actions in advance, but it remains unclear whether or not action planning and verbal working memory share cognitive resources. Using a novel experimental paradigm, we combined in two experiments a grasp-to-place task with a verbal working memory task. Participants planned a placing movement toward one of two target positions and subsequently encoded and maintained visually presented letters. Both experiments revealed that re-planning the intended action reduced letter recall performance; execution time, however, was not influenced by action modifications. The results of Experiment 2 suggest that the action’s interference with verbal working memory arose during the planning rather than the execution phase of the movement. Together, our results strongly suggest that movement planning and verbal working memory share common cognitive resources. © 2011 Elsevier Ireland Ltd. All rights reserved.
A major finding in cognitive psychology is that human thought and action is often guided by a plan, a mental representation that structures complex behavior [18]. There is behavioral and neurophysiological evidence that actions are mentally represented prior to motor execution and that these representations include, for example, the goals and consequences of the action [12]. Action sequences can be planned and covertly represented up to the third movement, before beginning the sequence [8], and it seems that both the beginning and end of a movement are represented in considerable detail [21]. Additionally, activated action representations affect forthcoming behavior. For example, people grasp an object differently depending on the current goal, e.g. fit vs. throw an object [17]. Although prior research provided evidence that actions are planned and represented prior to movement initiation, surprisingly little research has focused on the re-planning of actions. How much cognitive effort is required to modify a movement plan? The present study seeks to extend the existing literature by investigating the (dual-task) costs of re-planning an intended action. Cross-talk between action and cognition processes has been demonstrated for several cognitive domains, such as perception [11], language [13], emotion [1] and memory [22]. These studies
∗ Corresponding author at: Neurocognition and Action – Biomechanics – Research Group, Faculty of Psychology and Sport Sciences, Bielefeld University, Room N3-101, 33501 Bielefeld, Germany. Tel.: +49 521 106 5129; fax: +49 521 6432. E-mail address: marnie [email protected] (M.A. Spiegel). 0304-3940/$ – see front matter © 2011 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.neulet.2011.12.033
suggest that processes involved in action and cognition tasks share common cognitive resources. Here, we use the term cross-talk in a metaphorical sense; the term is not meant to imply a strictly modular architecture of cognition. Weigelt et al. [25] demonstrated interactions between movement planning and memory processes by combining a motor task (sequential opening of drawers at different heights) with a memory task (letter recall). Weigelt et al. [25] reported two major findings: First, reducing the effort needed for the memory task (i.e. when using free recall instead of serial recall) resulted in stronger movement planning effects, as indicated by the longer persistence of previous action plans. Second, the simultaneous motor task apparently abolished the recency effect, signifying the tendency of recent items to be recalled better than earlier items, which is a well-studied and otherwise stable serial-position effect in working memory research [20]. Logan and Fischman [15] suggest that the abolition of the recency effect is a basic concurrence cost of motor and memory tasks. Such demonstrations of cross-talk between motor and memory processes raise questions about the locus of the interaction. Does the effort of planning the movement, controlling it during execution, or a combination of both, interfere with the concurrent cognitive process? Glover et al. [7] reported an influence of word labels attached to rectangular target objects on early but not late stages of grasping. Participants showed larger initial grasp apertures after reading words representing relatively large objects (e.g., “APPLE”) than after reading words representing smaller objects (e.g., “GRAPE”), suggesting a semantic influence on action
M.A. Spiegel et al. / Neuroscience Letters 509 (2012) 82–86
83
Fig. 1. (A) Full view of the experimental apparatus including task board, PC, monitor, and video camera. (B) Schematic overview of the processing demands in a re-planning trial for both, the planning and the control phase. Note. This scheme only depicts trials in which the initial movement was planned to the right target. In the experiment, initial planning direction (left vs. right) was counterbalanced within subjects; low = low planning effort/execution demands; high = high planning effort/execution demands; t = time.
planning. Interestingly, this effect was on-line corrected as the hand approached the target. This continuously decreasing effect is consistent with the view of planning and online-control as being distinct stages of motor actions, as originally suggested by Woodworth [26]. The planning-control distinction assumes that both systems rely on distinct visual representations [6,19], with the planning representations being susceptible to interference from cognitive and perceptual variables, and the control representation being more independent of these interferences [5,7]. Although the ability to re-plan an intended action is crucial in allowing flexible adaptations to changing environments, to our knowledge, no study has systematically assessed the dual-task costs of action modifications. Moreover, it is not known whether dual-task costs are greater for re-planning high versus low accuracy movements. This issue was tested in Experiment 1. In the second Experiment we investigated the locus of motor-memory interactions by assessing whether dual-task costs arise from constraints of the motor system’s output during execution, or rather from cognitive effort involved in planning the movement. In both studies, a motor task (unimanual grasp-to-place task) was combined with a memory task (letter recall). In Experiment 1, we asked whether dual-task costs are greater for re-planning high versus low accuracy movements. Subjects prepared to move the sphere toward one of two identical target positions, which required either low or high accuracy, and subsequently encoded and maintained visually presented letters. Before they executed the placing movement and reported the letters, the planned movement direction was either confirmed or reversed by one of two auditory cues. Hence, two conditions were compared: In the prepared movement condition, participants executed the movement as planned. In the re-planning condition, the participant had to re-plan their movement to the other target position after the presentation of the to-be-memorized letter matrix. We hypothesized that movement re-planning would require cognitive resources which would interact with verbal working memory. Therefore, we predicted superior memory performance for prepared compared to re-planned movements. It has been shown that high precision movements require increased programming effort because the motor system’s output is more constrained
due to a larger and/or more precise muscle synergy recruitment pattern [23]. Therefore, we predicted a stronger decrease in memory performance for high compared to low precision placing movements in the re-planning condition. We assume that verbal working memory shares resources mostly with the movement planning phase and should not significantly affect the control phase. Thus, we predicted only a main effect of motor precision for execution time; however, the control phase may well be influenced by motor precision demands. Forty-eight (24 female, 24 male, Mage = 25.2 years, SD = 3.9, age range: 19–35) right-handed German students with normal or corrected-to-normal vision participated. Participants were compensated with either 5D or 1 h of participation credits. Subjects were randomly assigned to either the high motor precision group (N = 24) or the low motor precision group (N = 24). The task board (4 × 60 × 28 cm) included a starting position and two interchangeable targets (Fig. 1A). All positions were equipped with pressure-sensitive micro switches in order to record execution times, and to allow self-paced trial beginnings. In Experiment 1, homogeneous motor targets were used: for the high motor precision group, both sides of the task board were equipped with a stick (10 cm high, 0.5 cm wide). For the low motor precision group, both sides of the task board were equipped with a bowl (10 cm high, 10 cm in diameter). The targets were positioned 15 cm horizontally from the centre of the setup, which was marked by a yellow cross. A sphere (6 cm in diameter, furnished with a hole of 10 mm in diameter) either had to be fit onto the stick or put into the bowl, respectively. The left and right directional arrows were of identical size (2.5 × 1.5 cm). The 3 × 3 letter matrices (14 × 10.5 cm) contained nine random consonants of the Latin alphabet (adapted from Sperling [24]). The acoustic stimuli which signalled whether to re-plan or not, were a low and a high sound with the fundamental frequency of 436 Hz and 1280 Hz, respectively. At the beginning of each trial, the sphere was placed on the starting position. A fixation cross was displayed in the centre of a 17 ’monitor with integrated speakers. The self-paced lifting of the sphere triggered the presentation of a fixed sequence of stimulus events (Fig. 2). During a 1000 ms interstimulus interval (ISI), participants moved the sphere above the centre of the setup. Participants
84
M.A. Spiegel et al. / Neuroscience Letters 509 (2012) 82–86
Fig. 2. Timing of the stimulus sequence for one trial. Stimulus durations are presented in milliseconds. At the beginning of each trial, a fixation cross was presented. The lifting of the sphere from the starting position triggered the beginning of the stimulus sequence: ISI = interstimulus interval, preparation cue, letter matrix, auditory cue and “Report” (trigger to place the sphere on the correct target and then report the letters in written form). The time counter stopped as soon as the sphere hit the micro-switch of a target. The placing of the sphere back on the starting position initiated the next trial.
were instructed to hold the sphere above the yellow cross until the last of four stimuli was presented. A preparation cue (arrow left/right) was shown for 250 ms, pointing either to the right or the left target. Then, a 3 × 3 letter matrix appeared for 500 ms, followed by an auditory cue (high or low). Depending on the type of sound, either the prepared or the re-planned placing movement was the correct response. Finally, the imperative cue “Berichten” (German for “Report”) was presented. Participants then executed the placing movement and reported the letters in written form. Instructions did not prioritize one task over the other. Participants were encouraged to memorize as many letters as possible and to move the sphere quickly, but at a comfortable speed to the correct target. Placing the sphere back on the starting position initiated the next trial. Prior to the experimental trials, participants completed 10 practice trials to familiarize themselves with the task. One hundred experimental trials were presented in two blocks of 50 trials, separated by a break. 80 trials did not require movement re-planning (80%) whereas 20 trials did (20%). All trials were fully randomized and auditory cues were counterbalanced across subjects. The experiment lasted about 30 min. For further analyses of the grasp-and placing movements, a Sony HDR-SR11 camcorder positioned on a shelf of 223 cm recorded the experimental trials. Three dependent variables were of interest: Memory Performance was defined as the number of correctly reported letters, independent of their position within the matrix. Execution Time (ET) was measured from the onset of the auditory cue until target hit. As an additional movement-related measure for the planning phase, Response Preparation (RP) was analyzed from the video data. Operationally, RP was defined as the distance (in cm) of the sphere’s centre from the centre of the setup (yellow cross) at auditory cue onset. Considering that we instructed the participants to hold the sphere over the yellow cross, we interpret RP as the participant’s tendency to overtly prepare the subsequent placing movement
in horizontal direction (left vs. right) according to the directional arrow. Mean memory performance was 4.35 letters (SD = 0.77), which is in line with findings of other authors, who estimate the storage capacity of working memory to about four items [2,16]. A twoway ANOVA with the factors Motor Precision (high vs. low) and Movement Planning (prepared vs. re-planned) with repeated measures on Movement Planning revealed a significant interaction, F(1, 46) = 4.29, p = .04, p2 = .085 (Fig. 3). This interaction suggests that re-planning high precision placing movements compared to replanning low precision placing movements was associated with a larger reduction in memory performance. There was a main effect of Movement Planning, F(1, 46) = 68.60, p < .001, p2 = .599, but not of Motor Precision (F < 1). Video analyses revealed a prepared response effect. That is, participants planned and prepared the subsequent placing movement according to the preparation cue. A two-way ANOVA with the factors Motor Precision (high vs. low) and Arrow Direction (left vs. right) revealed a main effect of Arrow Direction, F(1, 47) = 3.18, p = .081, p2 = .064, but no interaction (F < 1). Thus, the Response Preparation (RP) effect appeared stable for both groups of Motor Precision. This result is in line with the assumption that stimulus–response mappings are established in working memory, in which the stimulus can directly elicit the associated motor response [10]. As a measure for sensory-motor control processes, Execution Time (ET) was analyzed. Trials that deviated more than 2.5 SD from individual mean ET were excluded (3.9% of the data). A two-way ANOVA with the factors Motor Precision and Movement Planning revealed neither an interaction nor a main effect of Movement Planning (both F < 1). That is, movement re-planning did not affect ET. The main effect of Motor Precision, F(1, 46) = 8.96, p < .004, p2 = .163, reflects that mean ET increased for high (2428 ms) compared to low precision placing movements (2057 ms). This
Fig. 3. Memory performance for Experiments 1 and 2 for all four experimental conditions (prepared vs. re-planned placing movements; high vs. low precision motor task).
M.A. Spiegel et al. / Neuroscience Letters 509 (2012) 82–86
difference is in line with Fitts’ Law [4], in that ET depends on the size of the target. Therefore, it is argued that as both target distances were identical, the longer ET most likely reflects precision requirements during the sensory-motor phase. Importantly, there was no evidence that ET was prolonged by re-planning processes, suggesting that once the placing movement had been initiated there was no dual-task interference from the concurrent memory task on ET. Analyses of RP and ET are compatible with the assumption that the cross-talk between cognitive processes and sensory-motor processes originates during movement planning rather than during motor control processes. However, the cross-talk effect, i.e. the interaction between Movement Planning and Motor Precision for memory performance, might be explained by either processing effort during the movement planning phase (two times low for low precision, two times high for high precision demands) or sensory-motor demands during movement execution (low for low precision, high for high precision demands; see Fig. 1B). Experiment 2 was conducted to distinguish between these two alternative explanations. In Experiment 2, the task board was changed to a heterogeneous layout, i.e. the left and the right target had different motor precision demands. Consequently, in re-planned trials movement execution with low precision demands required the planning of a high precision movement and the (subsequent) re-planning into a low precision movement. In contrast, movement execution with high precision demands required the planning of a low precision movement and the re-planning into a high precision movement. Therefore, re-planning trials required the planning of a low and a high precision movement, but only a high or a low precision movement was executed (cf. Fig. 1B). Accordingly, planning processes should be of comparable difficulty (low plus high) regardless of the demands of the actual target movement (low vs. high motor precision). This dissociation permits us to distinguish between the two alternative explanations: If the cross-talk between the motor and the memory task arose from sensory-motor demands during movement execution, we should again find an interaction between movement planning and motor precision for memory performance. In contrast, if the cross-talk originated during the planning phase, we expected no interaction. Predictions for Execution Time (ET) were the same as in Experiment 1. Forty-eight (30 female, 18 male, Mage = 24.8 years, SD = 4.1, age range: 19–35) right-handed healthy German students participated. The procedure for Experiment 2 was the same as described in Experiment 1. Out of a total of 150 trials, 120 trials (80%) did not require modification of the current action plan, whereas 30 trials (20%) did. The positions of the high and the low precision target on the setup were counterbalanced between subjects. Again, the grasp and placing movements of the experimental trials were videotaped. Mean memory performance was 4.15 letters (SD = .72). A twoway ANOVA with the factors Motor Precision (high vs. low) and Movement Planning (prepared vs. re-planned) with repeated measures on both factors did not detect an interaction between Motor Precision and Movement Planning on memory performance, F(1, 47) = 1.09, p = .30, p2 = .023 (Fig. 3). There was a significant main effect of Movement Planning, F(1, 47) = 93.74, p < .001, p2 = .666, but not of Motor Precision (F < 1). That is, as a consequence of replanning, memory performance decreased. Video data for one participant had to be excluded from Response Preparation (RP) analyses due to a malfunction of the video recording system. A two-way ANOVA with the factors Motor Precision (high vs. low) and Arrow Direction (left vs. right) showed a main effect of Arrow Direction, F(1, 46) = 4.10, p = .049, p2 = .082, confirming the RP-effect seen in Experiment 1. That is, participants planned the subsequent placing movement, and therefore deviated from the midline, according to the preparation cue. There was
85
neither a main effect nor an interaction involving Motor Precision (all Fs < 1). As a measure for sensory-motor control processes, Execution Time (ET) was analyzed. Trials that deviated more than 2.5 SD from the individual mean ET were excluded (4.5% of the data). A two-way ANOVA with the factors Motor Precision and Movement Planning revealed neither an interaction nor a main effect of Movement Planning (all Fs < 1). The main effect of Motor Precision, F(1, 47) = 162.36, p < .001, p2 = .776, reflects that mean ET increased for high (2257 ms) compared to low motor precision demands (1949 ms). In Experiment 2 we found no differences in dual-task costs for the re-planning of high vs. low precision placing movements after experimentally inducing a comparable cognitive planning effort. This result suggests that the degree of dual-task costs depends on cognitive planning effort rather than on motor execution demands. In two experiments using a dual-task scenario (grasp-to-place task vs. letter recall), we investigated whether action planning and verbal working memory share cognitive resources. The present study demonstrates that, if the modification of an intended action is required, the planning but not the control phase of the action shares cognitive resources with verbal working memory. These findings provide new insights into the architecture of goal-directed actions. Our data support and considerably extend earlier findings of actions reducing memory performance [15,25]. Measures of Response Preparation (RP) and Execution Time (ET) provide a direct assessment of the planning and control phases of actions, respectively. The RP-effect reflects that participants planned the subsequent placing movement according to the preparation cue. However, modifying the activated action representation did not affect ET – a measure for the control component of the objectdirected action. This finding is in line with prior findings on action-cognition cross-talks. A study by Liu et al. [14], for example, showed that conscious perception interfered with the initiation but not speed or accuracy of an ensuing pointing action. Another recent study by Hesse and Deubel [9] demonstrated that a concurrent perceptual identification task resulted in less accurate grip adaptations in early movement phases, while movement times were unaffected. Furthermore, this result is consistent with the assumption that planning and control are two components of human motor actions [26] (for a review see [3]). It has been shown that planning rather than control representations are susceptible to interference from cognitive and perceptual variables (e.g. [5,7]). Here, we demonstrate that conversely, movement planning activity can selectively interfere with concomitant working memory processes. This result supports the assumption that processes involved in action and cognition tasks share common resources which may lead to bidirectional interactions. Experiment 1 demonstrated that movement re-planning can influence concurrent cognitive processes very specifically: Replanning high precision movements resulted in greater dual-task costs than re-planning low precision movements. This result pattern is compatible with the finding that a greater accuracy demand of the motor target involves increased programming effort, because it requires a more constrained motor system output [23]. Experiment 2 localized the source of this cross-talk to the planning phase. Here, we observed no differences in dual-task costs after matching the cognitive load for the re-planning of both placing movements. Critically, one cannot fully exclude a much smaller effect of movement control on memory processes based on a statistical null effect for the interaction in question. Future research may test for an influence of movement control on memory processes by varying the timing of the presentation of the letter matrix so that it may coincide with the movement control phase. In summary, we show that modifying an intended action requires working memory resources that overlap with capacities
86
M.A. Spiegel et al. / Neuroscience Letters 509 (2012) 82–86
needed for the processing of a simultaneous memory task, reducing its performance. Together, the results of both experiments provide evidence that it is not the precision demands during movement execution per se, but rather the cognitive demands for planning the movement which determine the amount of dual-task costs. Hence, the cross-talk between action (re-)planning and working memory appears to be a basic process of action organisation. Acknowledgements This research was supported by German Research Foundation Grant DFG EXC 277 “Cognitive Interaction Technology” (CITEC). The authors thank two anonymous reviewers for their constructive comments and Alexander Koehn for programming assistance and supplying MeVisLab Public SDK (© MMS MeVis Medical Solutions) for video analyses. References [1] S.A. Coombes, T. Higgins, K.M. Gamble, J.H. Cauraugh, C.M. Janelle, Attentional control theory. anxiety, emotion, and motor planning, Journal of Anxiety Disorders 23 (8) (2009) 1072–1079. [2] N. Cowan, The magical number 4 in short-term memory: a reconsideration of mental storage capacity, Behavioral and Brain Sciences 24 (1) (2000) 87–114, discussion 114-85. [3] D. Elliott, W.F. Helsen, R. Chua, A century later. Woodworth’s (1899) twocomponent model of goal-directed aiming, Psychological Bulletin 127 (3) (2001) 342–357. [4] P.M. Fitts, The information capacity of the human motor system in controlling the amplitude of movement, Journal of Experimental Psychology 47 (6) (1954) 381–391. [5] S. Glover, P. Dixon, Semantics affect the planning but not control of grasping, Experimental Brain Research 146 (3) (2002) 383–387. [6] S. Glover, Separate visual representations in the planning and control of action, Behavioral and Brain Sciences 27 (1) (2004) 3–24, discussion 24–78. [7] S. Glover, D.A. Rosenbaum, J. Graham, P. Dixon, Grasping the meaning of words, Experimental Brain Research (154) (2004) 103–108. [8] P. Haggard, Planning of action sequences, Acta Psychologica 99 (2) (1998) 201–215.
[9] C. Hesse, H. Deubel, Efficient grasping requires attentional resources, Vision Research 51 (11) (2011) 1223–1231. [10] B. Hommel, The prepared reflex. Automaticity and control in stimulus-response translation, in: S. Monsell, J. Driver (Eds.), Control of Cognitive Processes. Attention and Performance, MIT, Press, Cambridge, 2000, pp. 247–274. [11] B. Hommel, J. Müsseler, G. Aschersleben, W. Prinz, The Theory of Event Coding (TEC): a framework for perception and action planning, Behavioral and Brain Sciences 24 (5) (2001) 849–878 (discussion 878–937). [12] M. Jeannerod, Neural simulation of action: a unifying mechanism for motor cognition, Neuroimage 14 (2001) 103–109. [13] O. Lindemann, P. Stenneken, H.T. van Schie, H. Bekkering, Semantic activation in action planning, Journal of Experimental Psychology. Human Perception and Performance 32 (3) (2006) 633–643. [14] G. Liu, R. Chua, J.T. Enns, Attention for perception and action: task interference for action planning, but not for online control, Experimental Brain Research 185 (4) (2008) 709–717. [15] S.W. Logan, M.G. Fischman, The relationship between end-state comfort effects and memory performance in serial and free recall, Acta Psychologica 137 (2011) 292–299. [16] S.J. Luck, E.K. Vogel, The capacity of visual working memory for features and conjunctions, Nature 390 (1997) 279–281. [17] R.G. Marteniuk, C.L. MacKenzie, M. Jeannerod, S. Athenes, C. Dugas, Constraints on human arm movement trajectories, Canadian Journal of Psychology 41 (3) (1987) 365–378. [18] G.A. Miller, E. Galanter, K.H. Pribram, Plans and the Structure of Behavior, Holt Rinehart & Winston (A Holt International Edition), Great Britain, London, 1960. [19] D. Milner, M. Goodale, The Visual Brain in Action, Oxford University Press, London, 1995. [20] Murdock Jr., The serial position effect of free recall, Journal of Experimental Psychology 64 (5) (1962) 482–488. [21] D.A. Rosenbaum, E.S. Halloran, R.G. Cohen, Grasping movement plans, Psychonomic Bulletin & Review 13 (5) (2006) 918–922. [22] T. Schack, F. Mechsner, Representation of motor skills in human long-term memory, Neuroscience Letters 391 (3) (2006) 77–81. [23] B. Sidaway, R.W. Christina, J.B. Shea, A movement constraint interpretation of the response complexity effect on programming time, in: A. Colley, J. Beech (Eds.), Cognition and Action in Skilled Behaviour, Elsevier Science Publisher B.V, Amsterdam, North-Holland, 1988, pp. 87–102. [24] G. Sperling, The information available in brief visual presentations, Psychological Monographs 74 (1960) 1–29. [25] M. Weigelt, D.A. Rosenbaum, S. Huelshorst, T. Schack, Moving and memorizing: motor planning modulates the recency effect in serial and free recall, Acta Psychologica 132 (1) (2009) 68–79. [26] R.S. Woodworth, The accuracy of voluntary movement, Psychological Review 3 (3(Suppl. 13)) (1899) 1–119.
Contents lists available at SciVerse ScienceDirect
Neuroscience Letters journal homepage: www.elsevier.com/locate/neulet
The costs of changing an intended action: Movement planning, but not execution, interferes with verbal working memory M.A. Spiegel a,b,∗ , D. Koester a,b , M. Weigelt c , T. Schack a,b,d a
Neurocognition and Action – Biomechanics Research Group, Bielefeld University, Germany Center of Excellence – Cognitive Interaction Technology (CITEC), Bielefeld University, Germany Sport Psychology Group, Department of Sport and Health, University of Paderborn, Germany d CoR-Lab Research Institute for Cognition and Robotics, Bielefeld University, Germany b c
a r t i c l e
i n f o
Article history: Received 12 October 2011 Received in revised form 4 December 2011 Accepted 17 December 2011 Keywords: Verbal working memory Dual-task Action planning Action modification Grasping
a b s t r a c t How much cognitive effort does it take to change a movement plan? In previous studies, it has been shown that humans plan and represent actions in advance, but it remains unclear whether or not action planning and verbal working memory share cognitive resources. Using a novel experimental paradigm, we combined in two experiments a grasp-to-place task with a verbal working memory task. Participants planned a placing movement toward one of two target positions and subsequently encoded and maintained visually presented letters. Both experiments revealed that re-planning the intended action reduced letter recall performance; execution time, however, was not influenced by action modifications. The results of Experiment 2 suggest that the action’s interference with verbal working memory arose during the planning rather than the execution phase of the movement. Together, our results strongly suggest that movement planning and verbal working memory share common cognitive resources. © 2011 Elsevier Ireland Ltd. All rights reserved.
A major finding in cognitive psychology is that human thought and action is often guided by a plan, a mental representation that structures complex behavior [18]. There is behavioral and neurophysiological evidence that actions are mentally represented prior to motor execution and that these representations include, for example, the goals and consequences of the action [12]. Action sequences can be planned and covertly represented up to the third movement, before beginning the sequence [8], and it seems that both the beginning and end of a movement are represented in considerable detail [21]. Additionally, activated action representations affect forthcoming behavior. For example, people grasp an object differently depending on the current goal, e.g. fit vs. throw an object [17]. Although prior research provided evidence that actions are planned and represented prior to movement initiation, surprisingly little research has focused on the re-planning of actions. How much cognitive effort is required to modify a movement plan? The present study seeks to extend the existing literature by investigating the (dual-task) costs of re-planning an intended action. Cross-talk between action and cognition processes has been demonstrated for several cognitive domains, such as perception [11], language [13], emotion [1] and memory [22]. These studies
∗ Corresponding author at: Neurocognition and Action – Biomechanics – Research Group, Faculty of Psychology and Sport Sciences, Bielefeld University, Room N3-101, 33501 Bielefeld, Germany. Tel.: +49 521 106 5129; fax: +49 521 6432. E-mail address: marnie [email protected] (M.A. Spiegel). 0304-3940/$ – see front matter © 2011 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.neulet.2011.12.033
suggest that processes involved in action and cognition tasks share common cognitive resources. Here, we use the term cross-talk in a metaphorical sense; the term is not meant to imply a strictly modular architecture of cognition. Weigelt et al. [25] demonstrated interactions between movement planning and memory processes by combining a motor task (sequential opening of drawers at different heights) with a memory task (letter recall). Weigelt et al. [25] reported two major findings: First, reducing the effort needed for the memory task (i.e. when using free recall instead of serial recall) resulted in stronger movement planning effects, as indicated by the longer persistence of previous action plans. Second, the simultaneous motor task apparently abolished the recency effect, signifying the tendency of recent items to be recalled better than earlier items, which is a well-studied and otherwise stable serial-position effect in working memory research [20]. Logan and Fischman [15] suggest that the abolition of the recency effect is a basic concurrence cost of motor and memory tasks. Such demonstrations of cross-talk between motor and memory processes raise questions about the locus of the interaction. Does the effort of planning the movement, controlling it during execution, or a combination of both, interfere with the concurrent cognitive process? Glover et al. [7] reported an influence of word labels attached to rectangular target objects on early but not late stages of grasping. Participants showed larger initial grasp apertures after reading words representing relatively large objects (e.g., “APPLE”) than after reading words representing smaller objects (e.g., “GRAPE”), suggesting a semantic influence on action
M.A. Spiegel et al. / Neuroscience Letters 509 (2012) 82–86
83
Fig. 1. (A) Full view of the experimental apparatus including task board, PC, monitor, and video camera. (B) Schematic overview of the processing demands in a re-planning trial for both, the planning and the control phase. Note. This scheme only depicts trials in which the initial movement was planned to the right target. In the experiment, initial planning direction (left vs. right) was counterbalanced within subjects; low = low planning effort/execution demands; high = high planning effort/execution demands; t = time.
planning. Interestingly, this effect was on-line corrected as the hand approached the target. This continuously decreasing effect is consistent with the view of planning and online-control as being distinct stages of motor actions, as originally suggested by Woodworth [26]. The planning-control distinction assumes that both systems rely on distinct visual representations [6,19], with the planning representations being susceptible to interference from cognitive and perceptual variables, and the control representation being more independent of these interferences [5,7]. Although the ability to re-plan an intended action is crucial in allowing flexible adaptations to changing environments, to our knowledge, no study has systematically assessed the dual-task costs of action modifications. Moreover, it is not known whether dual-task costs are greater for re-planning high versus low accuracy movements. This issue was tested in Experiment 1. In the second Experiment we investigated the locus of motor-memory interactions by assessing whether dual-task costs arise from constraints of the motor system’s output during execution, or rather from cognitive effort involved in planning the movement. In both studies, a motor task (unimanual grasp-to-place task) was combined with a memory task (letter recall). In Experiment 1, we asked whether dual-task costs are greater for re-planning high versus low accuracy movements. Subjects prepared to move the sphere toward one of two identical target positions, which required either low or high accuracy, and subsequently encoded and maintained visually presented letters. Before they executed the placing movement and reported the letters, the planned movement direction was either confirmed or reversed by one of two auditory cues. Hence, two conditions were compared: In the prepared movement condition, participants executed the movement as planned. In the re-planning condition, the participant had to re-plan their movement to the other target position after the presentation of the to-be-memorized letter matrix. We hypothesized that movement re-planning would require cognitive resources which would interact with verbal working memory. Therefore, we predicted superior memory performance for prepared compared to re-planned movements. It has been shown that high precision movements require increased programming effort because the motor system’s output is more constrained
due to a larger and/or more precise muscle synergy recruitment pattern [23]. Therefore, we predicted a stronger decrease in memory performance for high compared to low precision placing movements in the re-planning condition. We assume that verbal working memory shares resources mostly with the movement planning phase and should not significantly affect the control phase. Thus, we predicted only a main effect of motor precision for execution time; however, the control phase may well be influenced by motor precision demands. Forty-eight (24 female, 24 male, Mage = 25.2 years, SD = 3.9, age range: 19–35) right-handed German students with normal or corrected-to-normal vision participated. Participants were compensated with either 5D or 1 h of participation credits. Subjects were randomly assigned to either the high motor precision group (N = 24) or the low motor precision group (N = 24). The task board (4 × 60 × 28 cm) included a starting position and two interchangeable targets (Fig. 1A). All positions were equipped with pressure-sensitive micro switches in order to record execution times, and to allow self-paced trial beginnings. In Experiment 1, homogeneous motor targets were used: for the high motor precision group, both sides of the task board were equipped with a stick (10 cm high, 0.5 cm wide). For the low motor precision group, both sides of the task board were equipped with a bowl (10 cm high, 10 cm in diameter). The targets were positioned 15 cm horizontally from the centre of the setup, which was marked by a yellow cross. A sphere (6 cm in diameter, furnished with a hole of 10 mm in diameter) either had to be fit onto the stick or put into the bowl, respectively. The left and right directional arrows were of identical size (2.5 × 1.5 cm). The 3 × 3 letter matrices (14 × 10.5 cm) contained nine random consonants of the Latin alphabet (adapted from Sperling [24]). The acoustic stimuli which signalled whether to re-plan or not, were a low and a high sound with the fundamental frequency of 436 Hz and 1280 Hz, respectively. At the beginning of each trial, the sphere was placed on the starting position. A fixation cross was displayed in the centre of a 17 ’monitor with integrated speakers. The self-paced lifting of the sphere triggered the presentation of a fixed sequence of stimulus events (Fig. 2). During a 1000 ms interstimulus interval (ISI), participants moved the sphere above the centre of the setup. Participants
84
M.A. Spiegel et al. / Neuroscience Letters 509 (2012) 82–86
Fig. 2. Timing of the stimulus sequence for one trial. Stimulus durations are presented in milliseconds. At the beginning of each trial, a fixation cross was presented. The lifting of the sphere from the starting position triggered the beginning of the stimulus sequence: ISI = interstimulus interval, preparation cue, letter matrix, auditory cue and “Report” (trigger to place the sphere on the correct target and then report the letters in written form). The time counter stopped as soon as the sphere hit the micro-switch of a target. The placing of the sphere back on the starting position initiated the next trial.
were instructed to hold the sphere above the yellow cross until the last of four stimuli was presented. A preparation cue (arrow left/right) was shown for 250 ms, pointing either to the right or the left target. Then, a 3 × 3 letter matrix appeared for 500 ms, followed by an auditory cue (high or low). Depending on the type of sound, either the prepared or the re-planned placing movement was the correct response. Finally, the imperative cue “Berichten” (German for “Report”) was presented. Participants then executed the placing movement and reported the letters in written form. Instructions did not prioritize one task over the other. Participants were encouraged to memorize as many letters as possible and to move the sphere quickly, but at a comfortable speed to the correct target. Placing the sphere back on the starting position initiated the next trial. Prior to the experimental trials, participants completed 10 practice trials to familiarize themselves with the task. One hundred experimental trials were presented in two blocks of 50 trials, separated by a break. 80 trials did not require movement re-planning (80%) whereas 20 trials did (20%). All trials were fully randomized and auditory cues were counterbalanced across subjects. The experiment lasted about 30 min. For further analyses of the grasp-and placing movements, a Sony HDR-SR11 camcorder positioned on a shelf of 223 cm recorded the experimental trials. Three dependent variables were of interest: Memory Performance was defined as the number of correctly reported letters, independent of their position within the matrix. Execution Time (ET) was measured from the onset of the auditory cue until target hit. As an additional movement-related measure for the planning phase, Response Preparation (RP) was analyzed from the video data. Operationally, RP was defined as the distance (in cm) of the sphere’s centre from the centre of the setup (yellow cross) at auditory cue onset. Considering that we instructed the participants to hold the sphere over the yellow cross, we interpret RP as the participant’s tendency to overtly prepare the subsequent placing movement
in horizontal direction (left vs. right) according to the directional arrow. Mean memory performance was 4.35 letters (SD = 0.77), which is in line with findings of other authors, who estimate the storage capacity of working memory to about four items [2,16]. A twoway ANOVA with the factors Motor Precision (high vs. low) and Movement Planning (prepared vs. re-planned) with repeated measures on Movement Planning revealed a significant interaction, F(1, 46) = 4.29, p = .04, p2 = .085 (Fig. 3). This interaction suggests that re-planning high precision placing movements compared to replanning low precision placing movements was associated with a larger reduction in memory performance. There was a main effect of Movement Planning, F(1, 46) = 68.60, p < .001, p2 = .599, but not of Motor Precision (F < 1). Video analyses revealed a prepared response effect. That is, participants planned and prepared the subsequent placing movement according to the preparation cue. A two-way ANOVA with the factors Motor Precision (high vs. low) and Arrow Direction (left vs. right) revealed a main effect of Arrow Direction, F(1, 47) = 3.18, p = .081, p2 = .064, but no interaction (F < 1). Thus, the Response Preparation (RP) effect appeared stable for both groups of Motor Precision. This result is in line with the assumption that stimulus–response mappings are established in working memory, in which the stimulus can directly elicit the associated motor response [10]. As a measure for sensory-motor control processes, Execution Time (ET) was analyzed. Trials that deviated more than 2.5 SD from individual mean ET were excluded (3.9% of the data). A two-way ANOVA with the factors Motor Precision and Movement Planning revealed neither an interaction nor a main effect of Movement Planning (both F < 1). That is, movement re-planning did not affect ET. The main effect of Motor Precision, F(1, 46) = 8.96, p < .004, p2 = .163, reflects that mean ET increased for high (2428 ms) compared to low precision placing movements (2057 ms). This
Fig. 3. Memory performance for Experiments 1 and 2 for all four experimental conditions (prepared vs. re-planned placing movements; high vs. low precision motor task).
M.A. Spiegel et al. / Neuroscience Letters 509 (2012) 82–86
difference is in line with Fitts’ Law [4], in that ET depends on the size of the target. Therefore, it is argued that as both target distances were identical, the longer ET most likely reflects precision requirements during the sensory-motor phase. Importantly, there was no evidence that ET was prolonged by re-planning processes, suggesting that once the placing movement had been initiated there was no dual-task interference from the concurrent memory task on ET. Analyses of RP and ET are compatible with the assumption that the cross-talk between cognitive processes and sensory-motor processes originates during movement planning rather than during motor control processes. However, the cross-talk effect, i.e. the interaction between Movement Planning and Motor Precision for memory performance, might be explained by either processing effort during the movement planning phase (two times low for low precision, two times high for high precision demands) or sensory-motor demands during movement execution (low for low precision, high for high precision demands; see Fig. 1B). Experiment 2 was conducted to distinguish between these two alternative explanations. In Experiment 2, the task board was changed to a heterogeneous layout, i.e. the left and the right target had different motor precision demands. Consequently, in re-planned trials movement execution with low precision demands required the planning of a high precision movement and the (subsequent) re-planning into a low precision movement. In contrast, movement execution with high precision demands required the planning of a low precision movement and the re-planning into a high precision movement. Therefore, re-planning trials required the planning of a low and a high precision movement, but only a high or a low precision movement was executed (cf. Fig. 1B). Accordingly, planning processes should be of comparable difficulty (low plus high) regardless of the demands of the actual target movement (low vs. high motor precision). This dissociation permits us to distinguish between the two alternative explanations: If the cross-talk between the motor and the memory task arose from sensory-motor demands during movement execution, we should again find an interaction between movement planning and motor precision for memory performance. In contrast, if the cross-talk originated during the planning phase, we expected no interaction. Predictions for Execution Time (ET) were the same as in Experiment 1. Forty-eight (30 female, 18 male, Mage = 24.8 years, SD = 4.1, age range: 19–35) right-handed healthy German students participated. The procedure for Experiment 2 was the same as described in Experiment 1. Out of a total of 150 trials, 120 trials (80%) did not require modification of the current action plan, whereas 30 trials (20%) did. The positions of the high and the low precision target on the setup were counterbalanced between subjects. Again, the grasp and placing movements of the experimental trials were videotaped. Mean memory performance was 4.15 letters (SD = .72). A twoway ANOVA with the factors Motor Precision (high vs. low) and Movement Planning (prepared vs. re-planned) with repeated measures on both factors did not detect an interaction between Motor Precision and Movement Planning on memory performance, F(1, 47) = 1.09, p = .30, p2 = .023 (Fig. 3). There was a significant main effect of Movement Planning, F(1, 47) = 93.74, p < .001, p2 = .666, but not of Motor Precision (F < 1). That is, as a consequence of replanning, memory performance decreased. Video data for one participant had to be excluded from Response Preparation (RP) analyses due to a malfunction of the video recording system. A two-way ANOVA with the factors Motor Precision (high vs. low) and Arrow Direction (left vs. right) showed a main effect of Arrow Direction, F(1, 46) = 4.10, p = .049, p2 = .082, confirming the RP-effect seen in Experiment 1. That is, participants planned the subsequent placing movement, and therefore deviated from the midline, according to the preparation cue. There was
85
neither a main effect nor an interaction involving Motor Precision (all Fs < 1). As a measure for sensory-motor control processes, Execution Time (ET) was analyzed. Trials that deviated more than 2.5 SD from the individual mean ET were excluded (4.5% of the data). A two-way ANOVA with the factors Motor Precision and Movement Planning revealed neither an interaction nor a main effect of Movement Planning (all Fs < 1). The main effect of Motor Precision, F(1, 47) = 162.36, p < .001, p2 = .776, reflects that mean ET increased for high (2257 ms) compared to low motor precision demands (1949 ms). In Experiment 2 we found no differences in dual-task costs for the re-planning of high vs. low precision placing movements after experimentally inducing a comparable cognitive planning effort. This result suggests that the degree of dual-task costs depends on cognitive planning effort rather than on motor execution demands. In two experiments using a dual-task scenario (grasp-to-place task vs. letter recall), we investigated whether action planning and verbal working memory share cognitive resources. The present study demonstrates that, if the modification of an intended action is required, the planning but not the control phase of the action shares cognitive resources with verbal working memory. These findings provide new insights into the architecture of goal-directed actions. Our data support and considerably extend earlier findings of actions reducing memory performance [15,25]. Measures of Response Preparation (RP) and Execution Time (ET) provide a direct assessment of the planning and control phases of actions, respectively. The RP-effect reflects that participants planned the subsequent placing movement according to the preparation cue. However, modifying the activated action representation did not affect ET – a measure for the control component of the objectdirected action. This finding is in line with prior findings on action-cognition cross-talks. A study by Liu et al. [14], for example, showed that conscious perception interfered with the initiation but not speed or accuracy of an ensuing pointing action. Another recent study by Hesse and Deubel [9] demonstrated that a concurrent perceptual identification task resulted in less accurate grip adaptations in early movement phases, while movement times were unaffected. Furthermore, this result is consistent with the assumption that planning and control are two components of human motor actions [26] (for a review see [3]). It has been shown that planning rather than control representations are susceptible to interference from cognitive and perceptual variables (e.g. [5,7]). Here, we demonstrate that conversely, movement planning activity can selectively interfere with concomitant working memory processes. This result supports the assumption that processes involved in action and cognition tasks share common resources which may lead to bidirectional interactions. Experiment 1 demonstrated that movement re-planning can influence concurrent cognitive processes very specifically: Replanning high precision movements resulted in greater dual-task costs than re-planning low precision movements. This result pattern is compatible with the finding that a greater accuracy demand of the motor target involves increased programming effort, because it requires a more constrained motor system output [23]. Experiment 2 localized the source of this cross-talk to the planning phase. Here, we observed no differences in dual-task costs after matching the cognitive load for the re-planning of both placing movements. Critically, one cannot fully exclude a much smaller effect of movement control on memory processes based on a statistical null effect for the interaction in question. Future research may test for an influence of movement control on memory processes by varying the timing of the presentation of the letter matrix so that it may coincide with the movement control phase. In summary, we show that modifying an intended action requires working memory resources that overlap with capacities
86
M.A. Spiegel et al. / Neuroscience Letters 509 (2012) 82–86
needed for the processing of a simultaneous memory task, reducing its performance. Together, the results of both experiments provide evidence that it is not the precision demands during movement execution per se, but rather the cognitive demands for planning the movement which determine the amount of dual-task costs. Hence, the cross-talk between action (re-)planning and working memory appears to be a basic process of action organisation. Acknowledgements This research was supported by German Research Foundation Grant DFG EXC 277 “Cognitive Interaction Technology” (CITEC). The authors thank two anonymous reviewers for their constructive comments and Alexander Koehn for programming assistance and supplying MeVisLab Public SDK (© MMS MeVis Medical Solutions) for video analyses. References [1] S.A. Coombes, T. Higgins, K.M. Gamble, J.H. Cauraugh, C.M. Janelle, Attentional control theory. anxiety, emotion, and motor planning, Journal of Anxiety Disorders 23 (8) (2009) 1072–1079. [2] N. Cowan, The magical number 4 in short-term memory: a reconsideration of mental storage capacity, Behavioral and Brain Sciences 24 (1) (2000) 87–114, discussion 114-85. [3] D. Elliott, W.F. Helsen, R. Chua, A century later. Woodworth’s (1899) twocomponent model of goal-directed aiming, Psychological Bulletin 127 (3) (2001) 342–357. [4] P.M. Fitts, The information capacity of the human motor system in controlling the amplitude of movement, Journal of Experimental Psychology 47 (6) (1954) 381–391. [5] S. Glover, P. Dixon, Semantics affect the planning but not control of grasping, Experimental Brain Research 146 (3) (2002) 383–387. [6] S. Glover, Separate visual representations in the planning and control of action, Behavioral and Brain Sciences 27 (1) (2004) 3–24, discussion 24–78. [7] S. Glover, D.A. Rosenbaum, J. Graham, P. Dixon, Grasping the meaning of words, Experimental Brain Research (154) (2004) 103–108. [8] P. Haggard, Planning of action sequences, Acta Psychologica 99 (2) (1998) 201–215.
[9] C. Hesse, H. Deubel, Efficient grasping requires attentional resources, Vision Research 51 (11) (2011) 1223–1231. [10] B. Hommel, The prepared reflex. Automaticity and control in stimulus-response translation, in: S. Monsell, J. Driver (Eds.), Control of Cognitive Processes. Attention and Performance, MIT, Press, Cambridge, 2000, pp. 247–274. [11] B. Hommel, J. Müsseler, G. Aschersleben, W. Prinz, The Theory of Event Coding (TEC): a framework for perception and action planning, Behavioral and Brain Sciences 24 (5) (2001) 849–878 (discussion 878–937). [12] M. Jeannerod, Neural simulation of action: a unifying mechanism for motor cognition, Neuroimage 14 (2001) 103–109. [13] O. Lindemann, P. Stenneken, H.T. van Schie, H. Bekkering, Semantic activation in action planning, Journal of Experimental Psychology. Human Perception and Performance 32 (3) (2006) 633–643. [14] G. Liu, R. Chua, J.T. Enns, Attention for perception and action: task interference for action planning, but not for online control, Experimental Brain Research 185 (4) (2008) 709–717. [15] S.W. Logan, M.G. Fischman, The relationship between end-state comfort effects and memory performance in serial and free recall, Acta Psychologica 137 (2011) 292–299. [16] S.J. Luck, E.K. Vogel, The capacity of visual working memory for features and conjunctions, Nature 390 (1997) 279–281. [17] R.G. Marteniuk, C.L. MacKenzie, M. Jeannerod, S. Athenes, C. Dugas, Constraints on human arm movement trajectories, Canadian Journal of Psychology 41 (3) (1987) 365–378. [18] G.A. Miller, E. Galanter, K.H. Pribram, Plans and the Structure of Behavior, Holt Rinehart & Winston (A Holt International Edition), Great Britain, London, 1960. [19] D. Milner, M. Goodale, The Visual Brain in Action, Oxford University Press, London, 1995. [20] Murdock Jr., The serial position effect of free recall, Journal of Experimental Psychology 64 (5) (1962) 482–488. [21] D.A. Rosenbaum, E.S. Halloran, R.G. Cohen, Grasping movement plans, Psychonomic Bulletin & Review 13 (5) (2006) 918–922. [22] T. Schack, F. Mechsner, Representation of motor skills in human long-term memory, Neuroscience Letters 391 (3) (2006) 77–81. [23] B. Sidaway, R.W. Christina, J.B. Shea, A movement constraint interpretation of the response complexity effect on programming time, in: A. Colley, J. Beech (Eds.), Cognition and Action in Skilled Behaviour, Elsevier Science Publisher B.V, Amsterdam, North-Holland, 1988, pp. 87–102. [24] G. Sperling, The information available in brief visual presentations, Psychological Monographs 74 (1960) 1–29. [25] M. Weigelt, D.A. Rosenbaum, S. Huelshorst, T. Schack, Moving and memorizing: motor planning modulates the recency effect in serial and free recall, Acta Psychologica 132 (1) (2009) 68–79. [26] R.S. Woodworth, The accuracy of voluntary movement, Psychological Review 3 (3(Suppl. 13)) (1899) 1–119.