Levels of sensorimotor representation

JOURNAL

OF MATHEMATICAL

PSYCHOLOGY

Levels

20,91-163

of Sensorimotor ELLIOT

Department

(1979)

of Kinesiology,

Representation’ SALTZMAN

Child Development and Mental University of Washington

Retardation

Center,

Problems of planning coordinated sensorimotor actions are examined within a cohesive framework based on seven levels of movement representation (conceptual, environmental spatial, effector, body spatial, joint motion, joint torque, and muscle) and the relationships existing among these levels. Emphasis is placed on discussions of the spatial motion, joint motion, and joint torque levels in the context of mathematical treatments of the degrees of freedom problem in motor control and the incorporation of environmentally supplied forces into movement planning procedures. Implications of this approach for understanding the nature of skilled actions are discussed, and applications of the mathematical treatments for simulation and experimental studies of coordinated movement are suggested.

CONTENTS I. Introduction A. II.

The

Seven Levels of Representation: 1. The Action Plan

An Overview

Levels

A. Conceptual

Level

B. Environmental Space Motion, Effector Levels 1. Effector-nonspecific Environmental Space 2. Effector System Selection 3. Effector-Specific Environmental Space C.

Body Space, Joint Motion Levels 1. Body Segment Space (Segment Space) a. The direct kinematic transform b. Body spatial trajectory planning c. Body space-work/task-space mapping d. Body space-global invariant coordination:

Postural

control

1 An earlier version of this paper was submitted to the University of Minnesota in partial fulfillment of the requirements of the Ph.D. degree, December, 1977. Portions of this paper were presented at the 1st Joint Meeting of the Psychometric Society and the Society for Mathematical Psychology, Hamilton, Ontario, August 1978. The preparation of this manuscript was partially supported by a grant (HDO-2274) from the National Institute of Child Health and Human Development awarded to the Child Development and Mental Retardation Center, University of Washington. I would like to express my appreciation to the following colleagues for their helpful comments on an earlier draft of this paper and for valuable discussions concerning several of the topics therein: Louise Carter-Saltzman, Roger Enoka, Dedre Gentner, Berthold Horn, Earl Hunt, Robert Hutton, Beth Kerr, Colin MacLeod, Michael Maratsos, Doris Miller, Herbert Pick, Jr., John Rieser, and Michael Turvey.

91 0022-2496/79/050091-73$02.00/O Copyright 0 1979 All rights of reproduction

by Academic Press, Inc. in any form reserved.

92

ELLIOT

SALTZMAN

2. Sensorimotor Coordination: The Degrees of Freedom a. Degrees of freedom; generalized coordinates b. The inverse kinematic transform c. Matrix inversions: Singularities d. Redundant degrees of freedom e. Sensorimotor coordination: Kinematics f. The functional synergy matrix g. Representation of skills: I. Kinematics; coordinative h. Sensorimotor timing D. Joint Torque Level 1. Lagrangian Overview; Linkage Kinetics a. Lagrangian analysis b. Linkage-mechanical effects (kinetic energy terms) 2. Interaction with Environmental Forces a. Gravity effects (potential energy terms; generalized b. Contact effects: Normal reaction force c. Contact effects: Tangential reaction force (friction) 3. Three-Dimensional Motion; Computational Complexity a. Sensorimotor state/configuration space b. Representation of skills: II. Kinematics and kinetics E. Muscle Level 1. Muscle Forces a. Anatomy matrix; muscle b. Representation of skills: c. Skill acquisition 2. Muscle Innervation III.

Summary A.

synergy matrix III. Muscle, torque,

Problem

structures

forces)

and motion

and Conclusions

Concluding

Remarks

Appendixes A. B. C. D. E. F. G. H.

Direct Kinematic Transform Orientation Angles The Velocity Matrix The Jacobian Matrix Linkage-Mechanical Effects: Kinetic Energy Kinetics: Gravity Effects Kinetics: Normal Contact Effects Kinetics: Tangential Contact Effects-Friction

Terms

In at least one important sense, the problems encountered in dealing with the human sensorimotor system are similar to those encountered in studying any highly complex, goal-oriented, adaptive system. In each case, one is confronted with a multiplicity of observable variables and with the attendant difficulties of interpretation once observations are made. Thus, for example, one can code the act of picking up and drinking from a container of liquid as: (a) a set of electromyographic signals; (b) a set of skeletal joint rotations; (c) a set of spatial trajectories for each involved body segment; (d) a member of a symbolically classified group of lifting, carrying, and ingesting actions; or (e) a

SENSORIMOTOR

REPRESENTATION

93

member of a more abstract category based on the action’s relevance to the actor (e.g., nutrition, intoxication, or poisoning). Clearly, a given action can be representedin many ways, and the choice of a particular type of description will depend on the observer’s chosen level of analysis. Although quite general, this multilevel perspective offers profound insights into the nature of human sensorimotor coordination (cf. Bernstein, 1967; Gelfand, Gurfinkel, Fomin, & Tsetlin, 1971; Greene, 1972; Turvey, 1977; Turvey, Shaw, & Mace, 1978; Greene, Note 1, Note 2), and can be used to guide the formulation of questions that are appropriate to the examination of a given set of sensorimotorproblems, e.g., infant reaching, the nature of motor “programs,” and adaptation to prolonged sensory distortion. The following sectionselaborate on someof the possiblelevels of representationfor a given action and on the relations that exist among these levels. Problems of coordinated movement are analyzed within a cohesive framework which combines the mathematical rigor of approaches taken in the robotics, biomechanics, and prosthetics literature with the behavioral relevance of psychological and physiological approaches.In addition, a set of descriptive and mathematicalterms are recommendedwhich, hopefully, may serveas the basisfor a common languagein future discussionsand analysisof sensorimotorissues. Finally, it should be noted that severalissuesconsideredin this paper are meant to apply equally as well to the actions of both complex biological and man-madesystems.These include the use of a levels of representation approach, the use of synergies,descriptions of the physical constraints relevant to sensorimotoractivity, and trade-offs betweenstorage spaceand computation time in action planning procedures.The particular mathematical formalisms and storage/retrieval techniques used to deal with these issuesmay or may not be those used by biological organisms. However, these formalisms and techniques have been and should continue to be useful in planning movements for man-made devices and for simulation studies of biological movement.

Seven Levels of Representation:

An Overview

In discussingthe control of a particular type of movement (e.g., writing the letter Z), one must distinguish between the representation of the movement in long-term memory and the “working” representation usedto control the movement in the situation at hand (Bartlett, 1932; Bernstein, 1967;Pew, 1974;Schmidt, 1975; Turvey, 1977).This working representation, or action plan (Turvey, 1977),2is quite specific and is closely related to task demands as well as to the current state of the environment and musculoskeletal system. The long-term memorial representation appearsto be more abstract. This is suggestedby two considerations: limitations on memory storage capacity and the proz Turvey (1977) was actually ambivalent in his use of the term “action plan” which at diierent times referred to a “statement of consequences” (p. 215), a description of the “contours and timing of the movement” (p. 218) or a multileveled structure representing an action at different levels of abstraction, i.e., as a structure that undergoes “phases of change, phases that constitute a tailoring of the plan to the current kinematics and environmental contingencies” (p. 215). The term “action plan” is used in the present paper in the last sense described.

94

ELLIOT

SALTZMAN

duction of novel movements. For instance, the phenomenon of motor ep~ivalence(Hebb, 1949; Lashley, 1930) can be used to demonstrate that one can write the recognizably same2 with a wide range of effector systems,e.g., with either hand or foot, or even with a pencil clamped betweenone’steeth. Additionally, even within the sameeffector system the same spatiotemporal pattern of muscle contractions is rarely used from one performance to the next; the particular setsof musclesactivated will dependon task demands (e.g., writing on a small piece of paper or acrossa large blackboard), the physical state of the peripheral musculoskeletalapparatus (e.g., the initial position and velocity of the limbs) and the environment (e.g., the writing surface’s location and orientation relative to the writer). Thus, it would be extremely inefficient for the nervous system to store all the possible variations for every movement type in the actor’s behavioral repertoire. This leadsto the secondconsideration, which is that such an inelegant system would faiI to capture a central property of motor behavior: An actor has the ability to generate novel instancesof a particular movement type with which he has had only limited experience. In most situations, therefore, it appearsthat the actual performance of an action involves the generation and use of an action plan. This action plan incorporates both abstract information from long-term memory and specific information about current task demands,the physical state of the environment, and the physical state of the peripheral motor apparatus. The above considerationsare not meant to imply that long-term memory representations have nothing to do with the concrete details of a given action (effector systems, spatial and force requirements, etc.). The existence of skills implies that an actor has somesort of stored knowledge, specific to certain task requirements and effector systems, that is used to guide very familiar actions. Thus, writing is an upper limb-specific skill while walking is a lower limb-specific skill. It is important to realize that this statement refers to skills, not capabilities. One can write with one’s feet or walk on one’s hands, but these will generally be less-skilledactions. The topic of long-term representation of skilled actions will be discussedfurther below, where suggestionswill be made concerning the possiblecontent of such representations. The present analysisfocuseson the nature of action plans for movementsin particular situations, and begins at the stage when decisions have been made to execute these actions with respect to specific objects, surfaces,and events in the actor’s immediately perceivable environment. Neither the motives behind such actions nor the processes used to control ongoing movements (e.g., open-loop vs closed-loop processes)will be discussed.For the sake of brevity, discussionwill be limited to tasks of lifting, transporting, and manipulating. Examples of such tasks include situations in which an actor already hasdecided to transport somepeason a fork from dinner plate to mouth or write a seriesof letters on a blackboard. Furthermore, analysiswill be limited to those movements during which the actor maintains a constant baseof environmental support (e.g., by keeping his feet planted firmly on the ground), and acts within the immediately surrounding interaction space. Thus, discussionswill not include actions that involve imparting or absorbing forces during brief periods of impact, as in batting or catching a ball; nor will they include actions which involve planned shifts in the actor’s baseof support, as in walking through a larger-scalelocomotion space,or as in simple jumping.

SENSORIMOTOR

REPRESENTATION

95

However, the principles outlined in the analyses below may be modified and extended to these other types of movements without significant distortion. The Action

Plan

The action plan is viewed as a complex structure with seven major levels. In an order of generally decreasingabstractness,these are labeled: conceptual, environmental-space motion, effector, body-spacemotion, joint motion, joint torque, and muscle. For illustration, a brief and intentionally oversimplified example will be presented. Consider an experimental situation where a subject is standing in front of a table on which there are two blocks of identical size and weight. One block is red and rests at locus A, a second block is green and rests at locus B, and the subject’s right hand rests on the table at locus C. The subject has received instructions to stack the red block on top of the green block as quickly and accurately as possiblewhen he hears a tone. This stacking action might be representedat each of the above levels in the following way: (1) Conceptual-The act is defined in terms of symbolically coded and interrelated component actions, e.g., “reach” for the red block, “lift” and “transport” it above the green block such that it can be “lowered into place” on top of the greenblock. Movement representationsof this type are similar to what Turvey (1977) called action concepts. (2) EnvironmentaLspace motion-The blocks are located and oriented relative to a spatial referencesystemdefined with respectto the table surface,and the spatiotemporal trajectory of the red block’s motion is outlined in these table-surface-centered coordinates. For the sakeof simplicity, this spatial reference system is assumedto be defined in Cartesiancoordinates.

(3) EESfector-The acting member of the body, the body segmentsthat link it to the trunk, and the set of musclesor muscle groups associatedwith the control of this effector system are specified. To move the red block, the right hand-forearm-upper arm-shoulder girdle system is selectedalong with the relevant set of musclesor muscle groups. (4) Body-spacemotion-Assume that the actor’s spatial relationship to the table is known. The locations, orientations, and outlined trajectories of the block and the acting hand then may be specified with respect to a reference system defined, for example, relative to the frontal, longitudinal, and sagittal planesof the actor’s trunk. At this point, the action may be represented as a specification of the right-hand’s spatial trajectory relative to the trunk as a function of time. (5) Joint motion-In order to enable the hand to follow a particular body-spatial path at a particular speed,the changesof the joint angles,+j , in the joints between the hand and the trunk (i.e., wrist, radioulnar, elbow, shoulder, shoulder girdle) must be specified as functions of time. Thus, at this level the movement is representedas a set of joint angles (+j(t)), angular velocities ($j = (d/d) +j(t)), and angular accelerations (d;j

=

(d2/dt2)

+j(t))*

96

ELLIOT

SALTZMAN

(6) Joint torque-Given the desired set of & , q$ , and C& in the hand system, the resultant torques ( Ti(t)) which must be actively produced at each joint are computed. The weights of the red block and the dynamics of the arm enter into these calculations. (7) Muscle-There are two sublevels within this category. The first deals with the calculation of the muscle forces needed to produce a desired torque at a given joint. The second deals with the computation of the amount of muscle innerwation (neural input) required to produce the desired muscle forces. Two important points should be mentioned here. The first is that there is no necessarily fixed, one-to-one relationship between representations of the same movement in different levels of the action plan. Thus, one can write the letter 2 (conceptual level) with either the right or left hand (effector level), and one can use the right hand (effector level) to write the letter 2 or to scratch one’s back (conceptual level). The second point is that each level is posited to receive general instructions from other levels and to then operate relatively autonomously within its own set of computational procedures and domains of information (Greene, 1972; Turvey, 1977; Greene, Note 1, Note 2). For instance, in the above example of block stacking, the operations performed at the conceptual level would be concerned with the presence and identity of “graspable” and “transportable” objects in the environment but would be relatively uninterested in the particular effector used to accomplish the intended action. Operations performed at the spatial level, on the other hand, would be concerned with an object’s size and location, but would not be especially interested in whether it was a red block or a cube-shaped cookie jar. The next section examines in greater detail both the nature of the individual levels and the many-to-one relationships existing among them. As a general working principle, I have kept discussions of the higher, more abstract levels as uncluttered as possible with the lower-level details of a given movement. Such movement specifics have been added to the evolving action plan at the lowest possible level of movement planning. Additionally, in the interest of simplicity, I have considered only those indeterminacies introduced as more abstract representations are transformed into more concrete representations; that is, I will consider only those indeterminacies involved in a “top-down” sequence of transformations. Emphasis will be placed on characterizing the nature of the computational procedures used to transform movement representations from body-spatial to joint-motion levels and from joint-motion to joint-torque levels of the action plan. In the final section, the general approach presented in this paper will be reviewed, and applications relevant to studies of human sensorimotor control and development will be brieff y discussed.

THE

LEVELS

Bernstein (1967) has defined sensorimotor coordination as redundant degrees offreedom of the moving organ, in other words its system” (p. 127). Indeed, the highest stage of a coordinated skill of coordination at which the organism is not only unafraid of

“the process of master&g conversion to a controllable “corresponds to a degree reactive phenomena in a

SENSORIMOTOR

REPRESENTATION

97

system with many degrees of freedom, but is able to structure its movements so as to entirely the reactive phenomena which arise” (p. 109). The remainder of this essay is devoted to an unpacking, elaboration, and extension of these extremely reasonable ideas. Central to the present approach are the concepts of “redundancy,” “degrees of freedom,” and “reactive phenomena.” The first two terms becomemore understandable when viewed in conjunction with the body-spatial and joint-motion levels of the action plan. These levels of representationare discussedin the framework of a kinematic analysis of movement, which is the study of rectilinear and angular motion (and their derivatives with respect to time) without reference to the forces and torques producing motion (Meriam, 1966). The term “reactive phenomena” will be interpreted according to its role in the computation of actively produced resultant torques at the joints. These calculations take place at the joint-torque level of the action plan and are discussedin the context of a Kineticmovement analysis:the study of changesin motion related to the action of forces and torques (Meriam, 1966). I n accordancewith these views, this paper will present a detailed examination of the spatial-motion, joint-motion and joint-torque levels of movement representation. In contrast, the conceptual, effector, and muscle levels will be discussedin a relatively brief way, although it is not my intention to slight the importance of these latter types of representationin the production of human movement. Certainly, they warrant separate essaysdevoted to problems specific to their individual domainsof information and computational procedures. In the text that follows, the levels of representationin the action plan will be discussed in the order of their original presentation: conceptual, environmental-space motion, effector, body-space motion, joint motion, joint torque, and muscle. utilize

Conceptual

Level

The problems of representing an action at the conceptual level have much in common with efforts to specify the semantic structure of actions or action verbs in the fields of psychology (e.g., Abrahamson, 1975; Gentner, 1978; Miller & Johnson-Laird, 1976), linguistics (e.g., Talmy, 1975) and artificial intelligence (e.g., Schank, 1973; Winograd, 1973). Such representations appear to involve highly abstract symbolic components integrated within a logical or propositional framework. This suggestedanalogy between action concepts and semantic representation of verbs can be illustrated with reference to Fig. 1, a tentative representation of the verb “stir” developedby Gentner and Green0 (presented in Gentner, 1978) in accordance with the general approach presented by Norman and Rumelhart (1975). For a particular situation (e.g., where “Elliot is stirring somespaghetti saucewith a wooden spoon”), the diagram in Fig. 1 representsthe action at its highest node by “Agent (X = Elliot) stirs Object (Y = sauce) with Implement (2 = spoon).” The node “stir” includes three semantic subpredicate nodes, each of which stands for a distinct relational concept. An arrow originating from a given node terminates on an entity that is linked through the relation expressedat the arrow’s node of origin. Thus, the two predicates “Move” and Motion” are the arguments for the predicate “Cause,” and the labels “event” and “result” indicate the roles that “Move”

98

ELLIOT

SALTZMAN

1

J

is when

CAUSE

MOVE A

0

MOTION figure rate medium

x

11011

LOWI

FIG. 1. Semantic representation of the verb “stir” (adapted from “On relational acquisition of verb meaning” by D. Gentner, Child Development, 1978, 49, 988-998. 1978 by the Society for Research in Child Development. Used with permission.).

meaning: Copyright

the 0

and “Motion” play, respectively, in relation to “Cause.” What this representsis that Agent X’s (Elliot’s) hand movement is an event that causesImplement2 (spoon) to move with respect to Object Y (sauce) in a certain spatiotemporal manner. Additionally, the hand and spoon are related as implement and object, respectively, through the “Grasp” node; and the sauceis located (L) with respect to a bowl through the “Contained In” node. In terms of the present analysis, only one modification of this representation is suggested,and that is to replace the anatomically defined effector “hand” variable with a goal directed function, or e#ectz%ity (Turvey, et uZ., 1978), “gripper/transporter” variable. This would leave the selection of the effector system to the discretion of the actor, operating under the constraints imposedby the task environment. For example, one could choose(for some unspecified reason) to grasp the spoon with one’s toes or teeth. Such a decisionwould not be made until the effector level of the action plan. Having rendered the action concept effector-nonspecific, the representation becomes an abstract specification of a goal in terms of desired relationships between objects and surfaces in the environment. Such relationships are expressed functionally, spatially, and temporally with respect to the action situation. Specifically, the conceptsof “figure” and “ground” are used in Talmy’s (1975) senseto describe situations where one object (figure) is moving or is located with respect to another object (ground); and the spatial and temporal aspectsof motion are expressedonly in an abstract qualitative senserather

SENSORIMOTOR

99

REPRESENTATION

than metrically or quantitatively (Bernstein, 1967; Turvey, 1977). Thus, in our example, the rotary motion of a spoon is specified relative to some sauce inside a bowl; however, the precise trajectory of the spoon, the shapes of the spoon and bowl, and the exact environmental location and orientation of the bowl-spoon complex remain indefinite. What is important is that the representations at this level involve only those spatial characteristics that allow identification of objects and actions as well as action-relevant properties of objects and surfaces. These latter properties include “graspability,” “transportability, ” “stand-on-ability,” etc., and fall under the concepts of both uffordunces (Gibson, 1977) and munipuZandu (Tolman, 1932).3I n a similar way, the temporal parameters of a movement are not defined precisely at this level, but are stated in rather qualitative terms, e.g., the “medium-slow” motion of the spoon. It is interesting to note that the spatial and temporal aspectsof the motion have beenrepresentedindependently at the conceptual level. I will return to this point later in discussingthe spatial-motion levels of the action plan. Environmental

Space Motion,

Efector

Levels

The environmental spaceand effector levels will be treated as a unit since it seems reasonableto posit that the specific environmental trajectory of a transported object will depend on the effector systemusedin a given task situation. Three substageswill therefore be discussedin this section: (a) effector-nonspecific environmental space, (b) effector system selection, and (c) effector-specific environmental space. Effector-Nonspecific

Environmental

Space

At this level of the action plan, the shapes,sizes,locations, and orientations of objects and surfacesin the environment (interaction space)are specified relative to one another in quantitative metric terms as opposedto the more qualitative topological descriptions used at the conceptual level. These “geographic” (Pick, Yonas, & Rieser, 1978) spatial representations are defined, therefore, independently of the actor. Thus, the bowl in the action concept of “stir” discussedabove would be elaborated into a representation involving a container of a specific shape, height, and diameter which is located in a particular position relative to a supporting surface (e.g., a table top). In addition, the support surface (and hence the bowl) would be specified as being oriented parallel to a The definitions of affordance and manipulandum overlap to an impressive degree. The affordance of an object or surface is “a specific combination of the properties of its substance and its surfaces taken with reference to an animal” (Gibson, 1977, p. 67). These properties, although defined physically, are also defined by their relations to the action systems of a given animal. Thus, the affordance of “walk-on-ability ” is perceived in relation to the capacities of the animal’s locomotor system. Similarly, manipulanda are those properties of objects and surfaces which support sensorimotor action. They “derive in character from the independent physical character of the environmental object and from the response organ make-up of the given organism... [and are defined]... in terms of the range and refinements of manipulations which they will support in the given organism. They are stand-on-able-messes, pick-up-able-nesses, sit-in-able-nesses, etc., etc.” (Tolman, 1932, p. 448).

100

ELLIOT

SALTZMAN

the floor and perpendicular to the line of gravity (assuming that rational architects and contractors were involved in construction of the room in which we would like to stir). Thus, two types of coordinate systems are implied in this analysis. The first type of reference frame is a task/work space based on object- or surfacecentered coordinate systems (Agin & Binford, 1973; Nevatia & Binford, 1973; Paul, 1972; Marr, Note 3; Marr & Nishihara, Note 4) establishedfor each of the objects and surfacesrelevant to the action situation. For the sake of simplicity it is assumedthat such systemsare Cartesian, that the origins of thesereferenceframes lie at the geometric center of the objects and surfacesof interest, and that the orthogonal coordinate axes are oriented parallel to any existing axesof symmetry whenever possible.The spatial relations between any two objects or surfacescan then be specified by a vector from one frame’s origin to that of the next frame together with a description of the orientations of one frame’s axes relative to those of the next frame. Additionally, one object may be related to a secondwhich is related to a third, allowing the first object to be related to the third in a transitive manner. Referring again to our example of stirring, separate coordinate systemswould be defined for the table surface,bowl, and spoon, and spatiai interrelationships would be described with reference to the relationships existing between their respective reference frames. The question of which object’s or surface’s coordinate systemwill serve asthe primary reference frame for the task is specifiedby the “figureground” structure supplied in the conceptual level of the action plan. For example, the stirring action concept definesthe spoonand sauceasfigure and ground, respectively. Thus, the stirring action of the spoon is defined relative to the coordinates of the sauce which are essentiallydetermined with respect to the bowl’s coordinate system. Another example is the situation described earlier that involved stacking a red cube (figure) on top of a green cube (ground). Here the red cube’s motion would be defined relative to the green cube’s coordinate axes. The second type of reference system involved in this analysisis basedon the globaZ inmriants of the environment, i.e., the direction of gravity, the slant of the ground plane, and, if visible, the terrestrial horizon (Shaw & Mcintyre, 1974; Turvey, et al., 1978). The reference frame defined in this casewould have its z-axis parallel to gravity and its X- and y-axes defined in a plane perpendicular to the line of grvity. There is no a priori basis,however, for locating the system’sorigin at any particular point along the z-axis, or for that matter, for specifying the z-axis’ point of intersection in the xy plane; these axes will therefore be useful for defining only the relative orientations of other objector surface-centeredaxes to these global axes, but not for defining direction or distance relationships.For example, the task of transporting a full glassof water from one location to another on a table would involve, at this level of representation, a specification to maintain the open rim of the glassparallel to the global xy plane, as well as a surfacecentered description of the initial and final Iocations of the glassin table coordinates. Thus, the “ground” coordinate system in the object- or surface-centered description of a given task is ultimately oriented as a “figure” relative to the global system’saxes. The amount of trajectory detail generated at this level will depend on the nature of the task and the constraints imposed by the global invariants as well as on the layout of environmental surfacesand objects. Moving a cube from one location to another on

SENSORIMOTOR

REPRESENTATION

101

an uncluttered table top will involve only the definition of the initial and goal locations and orientations of the cube’s coordinate system relative to table space. The exact route taken and the movement’s precise speed will most likely depend on the strength and dexterity of the particular effector system used to grasp and transport the cube (e.g., hands versus feet versus teeth). A similar situation exists for moving a glass of water except that the orientation of the glass relative to gravity is additionally specified throughout the trajectory. If the table is cluttered, the locations of any obstacles will be described as locations to be avoided. An example of the most constrained type of spatial trajectory would be one that involved an object that could only be moved along a fixed track. An example of the most constrained type of temporal trajectory would be one which involved a movement whose timing was inflexibly regulated by metronome. Efjrector System Selection

On the basis of current task demands (e.g., instructions, environmental constraints) and previous experience in similar situations, a given effector system will be selectedto to accomplish a particular task. In addition, the coordinate axes of objects to be manipulated (“figures”; Talmy 1975) will be labeled in an actor-relevant manner determined by the specific task and the effector system being used. Thus, for the action of grasping a cylindrical container of liquid, the “approach and grasp” direction when the hand is used would most likely be any radially directed line toward the container’s longitudinal axis of symmetry. However, if the teeth are used, the approach and grasp would be initiated from above the cup and would proceed parallel to the sidesof the container toward any point on the lip of the container. Similarly, asan example of different tasks involving common effector systems,consider playing a piano versus pushing it to a new location in a room. Here each task entails different actor-relevant labels of the piano’s coordinate axeseven though the hands are usedin both casesto exert forces on the object itself. Effector-Speci$c

Environmental

Space

At this level of the action plan, an overall time frame is defined for the motion of the objects and surfaces specified as “figures” in the task space; also, the general spatial relationship of the actor to the work space is described. The former is specified with reference to both task constraints and knowledge of the strength, speed, and dexterity of the choseneffector system. For example, a given object may be carried faster if grasped with the hand than with the teeth. Similarly, if conservation of effort is a goal, a heavy object may be carried more slowly than a light object (Greene, Note 5) since lessforce is neededto initiate and deceleratethe motion if the path is traversed slowly. Specifying the actor’s spatial relationship to the work space involves defining the location and shape(area) of the actor’s base of support in task spacecoordinates.The base of support is that area of the supporting surface whoselimits are defined by the areasof contact with given parts of the body (or artificial extensions of the body) and which includes all intervening areas(Wells, 1971). Thus, the baseof support increaseswhen the feet are spread apart, and this area can be further increasedby the use of a cane or

102

ELLIOT

SALTZMAN

walking staff. Unless an actor can maintain his line of grvaity (i.e., the vertical projection of his center of gravity) within the base of support (or within a given region near its boundaries) he will lose his balance and totally disrupt any ongoing task-oriented activity. This problem of postural control with respect to the global invariant of gravity will be discussed further under the body-spatial motion section below; for now, it is sufficient to note that a wise choice of a support base is essential for successful completion of a given task. The location and shape of the base of support are specified with respect to the height of the task space from the ground surface, the anatomical range of motion of the effector chosen for the planned action, and the masses of the body segments included in this effector system. For instance, consider the task of moving an object from one place to another on a table top which is located at waist height in front of a standing actor. If the support base is located close to the table, if the object is light, and if the effector system involves only the hands and arms, the action will cause only a slight displacement of the line of gravity relative to the base of support and balance will be easily preserved. However if the support base is located farther from the table such that the actor must lean considerably forward with his trunk to reach the object and/or the object is much heavier, the trunk-plus-arm action will cause a rather large displacement of the line of gravity. If the line of gravity passes too far out of the support base, balance will be lost. These examples illustrates the factors that enter into the computation of the relative distance of the support base from the work space. The following examples deal with the specification of the shape of the support base for the same task, but in a situation where the work space is located on the floor. If the hands are the effecters and the feet are planted sufficiently close to the object, the action can be accomplished from a squatting position; that is, the shape of the base of support is defined by the boundary of the floor’s contact with the balls of the feet and any intervening area. However, if the grasping and transporting actions are to be done with the teeth, the hands and knees will most likely be used to maintain balance. The shape of the support base will consequently be outlined by the contact areas of the feet, knees, and hands with the floor. Essentially, then, the specification of the shape and distance of the actor’s base of support in work-space coordinates involves a knowledge of the outer limits of the posturally “safe” movements for the selected effector system, given the general constraints of the task-space trajectory and a crude estimate of the weights of objects to be manipulated. As was noted earlier, the spatial and temporal aspects of the motion are represented independently at the conceptual level. This separation has also been maintained throughout the treatment of environmental-space motion. However, although it is still rather abstract, the description of an intended action may now be considered more “meaningful” in the sense that “motion has meaning only when measured relative to a system of reference, which requires a well-defined system of coordinates and a timemeasuring device” (Meirovitch, 1970, p. 2). Both requirements are present in the representation of an action at this stageof the environmental-spacelevel, either explicitly (i.e., the Cartesian reference system defined for the work space)or implicitly (i.e., the timing device necessaryto specify the time frame of the action). Before proceeding to the next levels, it should be noted that the spatial and temporal

SENSORIMOTOR

103

REPRESENTATION

aspects of the motion will eventually be functionally related. This most probably occurs at some point between the body-spatial/joint-motion levels and the joint-torque level of the action plan because, as was mentioned earlier, the input to the joint-torque level is a set of “n” angular accelerations (n = the number of joints involved in the action) expressed as functions of time. At this point, several convenient ways to relate the spatial and temporal aspects of a given type of movement will be examined. Consider, for example, a circular motion with radius, L, in the task-spacemyplane. The spatial aspectof this trajectory is defined by the setof points in the planethat satisfy the function f(~, y) = x2+ y2 - L2 = 0. This curve can be related to time using any one of several possibleparameters, each of which has a unique value at every point in the curve. The most obvious choice of parameters is the parameter of time, t. Thus, x and y could be expressedas x = x(t), y = y(t), and the position vector, r, defined by r = r(t) = x(t)i + y(t)&” Another equally plausible choice of parameterswould be h, a measureof the distance along the curve from a given starting point. In this casex and y could be expressedasx = x(h), y = y(h), and the position vector defined by r = r(X) = x(h)i + y(h)j; x’s and y’s dependenceon time could then be specifiedindirectly by defining X asa function of time, i.e., X = h(t). We are now ready to proceed to the body-space and joint-motion levels of the action plan.

Body-Space,

Joint-Motion

Levels

In discussingthe nature of movement representationsin the conceptual, environmental, and effector levels of the action plan, it was evident that a great deal of potential indeterminacy existed among them. Thus, the conceptual level specified neither effector nor environmental trajectory (path and/or time frame), a given spatial location could be reached by several effecters, the sameeffector could reach a number of spatial locations at a variety of speeds,and so on. This representationalindeterminacy appearsto be one of the main sourcesof the action system’sflexibility. This flexibility can be seenin the phenomenonof motor equivalence, i.e., “a variability of specific muscular responses,with circumstance, in such a way as to produce a single result” (Hebb, 1949, pp. 153-154). The indeterminacy existing between the body-space and joint levels of the action plan is yet a further source of such variability. Consider this relatively standard example-the sameeffector (the hand) can draw the letter 2 on a piece of paper either with finger and wrist motion or by holding the arm relatively stiff and moving the hand via trunk and shoulder activity. There are alsolessobvious instancesof spatial-joint motor equivalence. Golani (1976) has studied certain of these types of actions, referred to as “fixations,” in the movement sequencesof the golden jackal (Canis aweus). During fixations, a “joint” is formed either physically or abstractly between the animal’s body part and either another portion of its body or an environmental target (i.e., an object, another animal, or an abstractly defined spatial position). Thus, an animal may be seenas “keeping its * In this expression and in the rest of the text, the following notation will be used in vector expressions: (a) Any boldface small or capital letter denotes a vector; and (b) i, 1, & denote the unit vectors corresponding to the X, y, z axes, respectively, of a Cartesian coordinate system.

104

ELLIOT

SALTZMAN

own head in contact with a particular point on its own hindquarters, and then steadily maintaining this contact by simultaneous‘twisting’ of the head, neck, and the torso” (p. 92). A more abstract example would be seenwhen “as a result of raising the neck in relation to the torso, and of lowering the headin relation to the neck for the sameamount of movement, the head may be ‘fixated’ .... i.e., it will steadily maintain its initial angle to the ground” (p. 92). A similar type of activity has been examined in human subjects during the task of aiming a pistol at a target, when the goal is to achieve and maintain a stablepositional relationship betweenthe sightsof the gun and the target (Arutyunyan, Gurfinkel, & Mirskii, 1968, 1969). The following sectionsdeal with the types of computational procedures that may be involved in this sort of motor equivalence. I will attempt to characterize the nature of the transformations between body-spatial and joint-motion representations.In doing so, I will discussalsothe interaction of postural and task-related movements, and the central role of redundant degreesof freedom in executing a coordinated action. These problems are examined in the context of actions that involve reaching for objects and transporting them through space. Such actions correspond to both Whitney’s (1969) rearrangement tasksand Smith and Smith’s (1962) transport tasks. Body SegmentSpace (Segment

Space)

Once an effector system has been selectedand the shapeand location of the support basehas been specifiedin work-space coordinates, the task of transforming the environmental spatiotemporal action trajectory into body-relevant terms begins. In order to separatethe reaching problem from that of maintaining postural stability, I will temporarily adopt the simplifying assumptionthat the shoulder girdle is spatially fixed relative to the trunk which is itself fixed relative to the baseof support. This is a highly artificial situation. For a standing actor this might correspond to a casein which the feet were glued to the ground, a body cast prevented any motion in the joints between the feet and shoulder, and external bracing prevented the massive edifice from toppling during object reaching and manipulation. However, this assumption does allow the environmental-spatialaction representation, defined for the objects and surfacein the work space, to be specified unambiguously with respect to a trunk segment- (or shoulder girdle-) centered spatial coordinate system. To reach for and successfully grasp or manipulate any of theseobjects and surfaces,the hand must alsobe located and oriented in shoulder space.Once known, the spatial relationship between hand and environmental target may be computed and used in planning an appropriate set of spatial reaching commands which, in turn, must be transformed into joint-motion commandsbefore the action can begin. To perceive the hand’s location, orientation, linear velocity, and angular velocity relative to the shoulder girdle (or any body segment)by a meansother than vision, one must be able to translate proprioceptively specified joint information into a spatially meaningful form. The “perceptual” transformation from joint information to body space is a direct kinematic transform, while the “motor” transformation from body spaceto joint commandsis an inverse kinematic transform (Wylie, 1975).

SENSORIMOTOR

REPRESENTATION

105

The direct kinematic transform. The discussionof spatial and joint-motion relations begins by considering a direct transformation method which is widely used to describe the kinematics of robot manipulator arms (Kahn, 1969; Kahn & Roth, 1971; Lewis, 1974; Paul, 1972; Whitney, 1969, 1972; Waters, Note 6) and which is equally applicable to human or prosthetic arm motion. This method is based on a notation developed by Denavit and Hartenberg (1955) to represent the spatial relationships amongthe different segmentsof kinematic chains5(see Fig. 2). In dealing with the problem of locating the

‘i

FIG. 2. Three-dimensional (adapted from “Modelling, R. Paul, AIM-177, Artificial by The Artificial Intelligence

linkage system labeled with Denavit-Hartenberg parameters trajectory calculation and servoing of a computer controlled arm” by Intelligence Laboratory, Stanford University, 1972. Copyright 0 1972 Laboratory, Stanford University. Used with permission.).

hand in shoulder girdle spatial coordinates as a function of arm joint variables, the arm is viewed as an n + 1 segment(j = 0, l,..., n) chain in which the hand and shoulder girdle are the nth (most distal) and 0th (most proximal) segments,respectively. Additionally, eachjointi (located between segmentjel and segmenti)is assumedto be frictionlessand to contain either one revolute (i.e., a hinge joint allowing relative rotation of two segmentsabout a rotation axis) or one prismatic (i.e., a sliding joint allowing relative translation, but no rotation, between two segmentsalong a translation axis) degree of freedom.6 Since I am treating the shoulder girdle as fixed relative to the stationary base 5 A kinematic chain consists of a series of links (i.e., rigid bodies, body segments, etc.) joined together to transmit motion or force. Limbs (robot, prosthetic, or human) are therefore prime examples. B A human arm contains joints with two (wrist) and three (shoulder) approximately orthogonal rotation axes. For simplification I am considering a model system in which each joint contains only one degree of freedom. However, the approach presented can be extended to joints containing two or three degrees of freedom with relative ease (Denavit & Hartenberg, 19.55; Sheth & Uicker, 1971). Human arms do not contain prismatic joints. The reason for introducing this type of joint is that robot manipulator arms will be discussed which may contain prism joints in addition to revolute joints (e.g., the “Scheinman arm,” Dobrotin & Scheinman, 1973). 480/20/2-2

106

ELLIOT

SALTZMAN

of support, the shoulder girdle provides a stable reference frame for the motion of segments 1 through n. In this direct kinematic transformation system, each segmentj has its own orthogonal spatial coordinate system xyq fixed relative to itself: zi is given by the axis of rotation of jointi+l ; X, is the extension of the common perpendicular constructed from zi-r to zj ; yj completes the right-handed coordinate system; and Oj is the origin of systemj . The spatial relationships between the coordinate systems of two successive segments (i.e., xyq-r and xy,q) are completely specified by the following four parameters: q = the distance

from ziel to zj measured

& = the angle from xjel to xi measured sj = the distance

along zj-r

along xj ;

in a right-handed

sense about aj-1 ;

from xj-r to xj ;

aj = the angle from xj-r to zj measured

in a right-handed

sense about xi .

Figure2 illustrates theway these parameters can be used to relate one segment spatially to an adjacent segment in a three-dimensional kinematic chain with revolute joints. A simplified two-dimensional example of a chain with three revolute joints can be seen in Fig. 3A. Here the cuj’s = 0 since all z-axes are perpendicular to the plane of the paper

A

FIG.

3.

Hartenberg segment.

and hence zj’s at the paper. All and z, . One can

Two-dimensional parameters: (A)

linkage parameters

B

system with 4 segments for segments 0 through

(bodies) 2; (B)

labeled with Denavitparameters for terminal

parallel to each other. Similarly, the Q’S = 0 since the 5’s each intersect the same z-coordinate, i.e., at the intersection of the q’s with the plane of the q’s are in the plane of the paper and measure the distance between zjel readily

use the Denavit-Hartenberg

notation

to express in xyzjP1 coordinates

SENSORIMOTOR

REPRESENTATION

107

the location of points fixed relative to bodyj and given in xy,Zj coordinates (see Fig. 2). These spatial relationships can be written in matrix form’ as: j-1R

=

(j-lAj)

. jR,

(1)

where j-lR = [xjP1 , yj-l , zjel , l]r, a 4 X I column vector expressing points in framejel coordinates; ‘R = [xj 7yj , Zj 7 I]r, a 4 X 1 column vector expressing points in segmentj in framei coordinates; aj cos 4j cos $bj -cos aj sin C#J~ sin aj sin $i sin I$~ cos aj cos $j -sin 01~ cos $? aj sin & j-lAj = 0 sin aj cos aj sj ? 0 I 0 1 i 0 a 4 x 4 matrix associatedwith jointj which transforms systemj coordinatesto systemjPl coordinates. In fact, the upper three elementsin the right-most column of this matrix can be viewed asa 3 x 1 column vector locating the origin of systemjin terms of systemjel coordinates. Similarly, the upper-left 3 x 3 submatrix representsthe rotation of framei relative to frame i-l . Thus, the first (i.e., left-most), second, and third columns can be viewed as 3 x 1 column vectors expressingthe directions of xj , yj , and zj , respectively, reIative to frame i-l . A more detaiIed derivation of the elementsin this matrix is presented in Appendix A. Equation (1) expressesthe spatial relationships between the coordinate systems of any two adjacent segmentsin a kinematic chain. However, since arms have more than two segments,it is particularly important to find a means of expressing the location and orientation of the most distal link (i.e., the hand) in terms of the most proximal link’s coordinate system(i.e., the shoulder or “base” reference frame). Describing the method for relating hand and shoulder coordinates requires, in turn, a preliminary discussion of the nature of the hand coordinate system, asit is usually defined in terms that are a bit different from other links’ coordinate systems. The main difference between the hand-oriented coordinate system ~yz~~and more proximal frames is that there is no associatedjoint $h+l . Consequently, the hand’s z-axis, z,, , can be specified according to what one considersthe hand’s “natural” func’ Readers unfamiliar with matrix representations of spatial transformations among coordinate systems are advised to consult any of several mechanical engineering texts (e.g., Groesberg, 1968; Meriam, 1966, Tuma, 1974) for overviews of these techniques. Additionally, regarding the direct kinematic equation (Eq. l), it should be noted that: (a) In the interest of notational compactness, the 4 x 1 column vectors have been expressed as the equivalent 1 i: 4 row vector transposes. The transpose operation is denoted by the superscript “T,” and will be used in other vector expressions presented later in the text; and (b) the “0001” in the last row of the 4 x 4 transform matrix and the “1”‘s in the last rows of the 4 x 1 column vectors are not essential components of the underlying 3 x 4 rotation/translation matrix and the 3 x 1 spatial vectors. These fourth rows have been added in order to facilitate finding the inverse spatial matrix transformation (see Eq. 5). Matrix inversion methods are described in Wylie (1975) or in any matrix algebra text.

108

ELLIOT

SALTZMAN

tional axes. Once z, is defined, however, the common perpendicular from xhVl to xh can be found and xh , yh , ah, $h , sh , and ah can be measured in the manner discussed above. As a result one can define the matrix transform associated with joint, , h-1,4h , which maps hand coordinates into the adjacent proximal segmenth-r’s coordinates. Figure 4 illustrates the manner in which hand-oriented coordinates have been defined for a robot manipulator capable of general motion in three-dimensional space (Lewis, 1974; Whitney, 1969, 1972; Waters, Note 6). Here one can see that the yh (reach, R; or approach) axis points in the direction of approach of the hand, and a rotation about R is called a twist; the xh (sweep, S; or sliding) axis points in the direction of finger (i.e., robot jaws or pincers) motion as the hand opens and closes, and a rotation about S is called a tilt; the zh (lift, L; or normal) axis points orthogonally to the plane of finger motion, and a rotation about L is called a turn; and the origin Oh of the hand-oriented frame is located in the plane of finger motion between the hand’s grasping surfaces. A two-dimensional example of a hand-oriented system may be seen in Fig. 3B.

FIG. 4. Hand-oriented coordinate system for a three-dimensional manipulator: xyz,, = base 0 = base frame origin, H = hand frame origin, W = wrist frame frame, xyz, = hand frame, origin, P,, = hand position vector, P, = wrist position vector, V, = linear velocity vector, &, = angle (adapted from “Mechanical arm control” by R. C. angular velocity vector, & = orientation Waters, Vision Flash 42, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1973. Copyright 0 1973 by The Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Used with permission.).

We are now in a position to discussa method for expressingthe hand’s location and orientation in shoulder or base-frame coordinates. This reducesto finding the location of Oh in shoulder coordinates and the relative orientation of the xyz, hand axes with respect to the xyz, shoulder axes. This samemethod will be equally useful for locating a point relative to hand coordinatesthat had initially been defined as fixed with respect to shoulder coordinates, and vice versa. These spatial relationshipsare relatively easy to specify and the matrix transform relating any two nonadjacent segmental coordinate systemscan be obtained by multiplying the matrix transforms associatedwith all inter-

SENSORIMOTOR

REPRESENTATION

109

mediate joints. Thus, in an arm with segments 0 through n and joints 1 through n, in which the nth segment is the hand, the position and orientation of systemi (j = 1, 2,..., n) in system, (base or shoulder frame) coordinates can be found by computing the product, Bj , of j 4 X 4 matrices: B, z “A, . IA, ... j-lAj,

j = 1, 2,. .. , 11.

(2)

Forj = n we have B, , a 4 x 4 matrix transform relating hand to shoulder coordinates. The upper three elements in the right-most column of the matrix can be viewed as a 3 x 1 vector locating the hand frame’s origin in shoulder coordinates. This vector corresponds to the vector PH in Fig. 4. The relative orientation of the hand axes with respect to the shoulder frame is given by the upper-left 3 x 3 submatrix. Thus, the first (or left-most), second, and third columns of this submatrix can be viewed as 3 x 1 column vectors expressing the directions of xh (s, or sweep axis), yh (R, or reach axis), and zh (L, or lift axis), respectively, relative to the shoulder frame. For example, if i, , io, k,, then and I,, jh , i(, are the xyz unit vectors of the base and hand frames, respectively, the first column in the rotation submatrix will consist of the xyzo components of the ih vector. Thus, the topmost element in this 3 x 1 column vector will be the x0 component, the middle element in this vector will be the y. component, and the bottom element will be the x0 component of the ih vector, i.e., ih = (ih)s0 i, + (ih)YOi. + (i& k, . It is useful to express the relative orientation of the hand and shoulder coordinate systems in terms of the set of orient&on angles, pH (Waters, Note 6), seen in Fig. 4, where: (flH)zO is the angle between the y. and yh axes measured about x0 ; (pH),O is the angle between the z. and zh axes measured about y. ; and (/3H)z0is the angle between the x0 and xh axes measured about z. . These angles may be expressed as trigonometric functions of the base frame components of the hand frame’s unit vectors: @H)%,

=

tan-‘((j,)z,/(jh>,,),

(PHL~

=

tan-l((rz,),/(kh)z,),

@H)z,

=

tan-‘((;h>,,/(i,),).

(3)

As was mentioned above, these components are obtainable from the rotation submatrix. Details of these derivations are presented in Appendix B. In addition to expressing the hand’s origin and unit vectors in base frame coordinates, one can locate in base coordinates a point in the hand that had been initially specified relative to the hand by OR = B, . “R,

(4)

where OR and “R are defined as in Eq. (1). C onversely, one can locate a point in hand coordinates that had been initially specified as fixed relative to base coordinates by “R = Ho . OR, where Ho = (B&l.

(5)

110

ELLIOT

SALTZMAN

The present discussion of hand-oriented coordinates and the earlier treatment of object-centered coordinates suggest that the manipulation of objects may be described as an interaction between these two types of reference systems. Specifically, it seems plausible that when one learns to use a tool (Whitney, 1972) or sports implement in a skillful manner, one constructs a “new” effector system centered in the coordinate system of the object being used. Thus, given an object with known dimensions and “functional” axes, and given a known grip type (e.g., Napier’s (1956) power versus precision grip) and grip angle (e.g., forehand versus backhand tennis grip), one can locate and orient the object’s functional axes in base frame (and thereby task-space) coordinates. In effect, the object becomes a new effector, an extension of one’s self. To quote Polanyi (1964): When we use a hammer to drive in a nail, we attend to both nail and hammer, but in a different req. We watch the effect of our strokes on the nail and try to wield the hammer so as to hit the nail most effectively. When we bring down the hammer we do not feel its handle has struck our palm but that its head has struck the nail. (p. 55) Our subsidiary awareness of tools and probes can be regarded now as the act of making them form a part of our own body. The way we use a hammer or a blind man uses his stick, shows in fact that in both cases we shift outward the points at which we make contact with the things that we observe as objects outside ourselves. (p. 59)

Of course, skilled tool use involves more than the ability to use the above type of kinematic object knowledge. It also requires a knowledge of the kinetic (or inertial) properties of the objects, the ways in which these objects respond to applied forces and torques. Such kinetic information, however, becomes interpretable only within a kinematic context. Let us return now to a discussion of the significance of the B, matrix. Since matrix B, is the result of the multiplication of matrices OA, . lA, ... +lA, , each element of B, will be some function of the Denavit-Hartenberg (1955) parameters aj , sj , Us , and di (j = 1, 2,..., n). For an arm with only revolute joints, aj , si , and aj will be constant and & will vary as the limb configuration changes; for an arm that includes prismatic joints as well, aj , ai , and 4j will be constant for such joints while si will vary. In general, the joint variables in a given chain, whether c,$ or sj , are labeled pi . Human limbs can be treated as having revolute joints only and consequently the joint angles, $j , will be the only variable elements, qj , in the B, matrix for a given limb. Thus, one could keep track of the position and orientation of the hand in base frame coordinates as limb configuration changed over time by monitoring the B, matrix, whose component handorigin vector and hand-axis orientation submatrix would be updated continuously according to current joint angle information. Following Kahn (1969) and Waters (Note 6), the base frame velocity of a point fixed in body,‘s coordinate system can be obtained by differentiating with respect to time the expression OR zzz Bj . jR (j = 1, 2 )..., n), (6) which is the general form of Eq. (4). Th e result of this differentiation Ofi = vj . jR

is the expression

(7)

SENSORIMOTOR

REPRESENTATION

111

where

and Vi = a 4

X

4 “velocity

matrix”

=-$Bj

Details of this derivation are provided in Appendix C. However, the interpretation of the Vj matrix follows a by now familiar pattern. In the matrix V, , the velocity matrix expressing hand parameters in terms of base frame coordinates: (a) the rightmost column contains a 3 x 1 vector expressing the velocity of the hand frame’s origin relative to the base frame; (b) the first (or leftmost), second, and third columns of the upper-left 3 x 3 submatrix can be viewed as 3 x 1 column vectors expressing the directional changes over time of the hand’s unit vectors relative to the base frame; and (c) the angular velocity components (&),o , (&dllg and @& may be expressed as trigonometric functions of the base frame components of the hand frame’s unit vector time derivatives. An alternative means of expressing the hand’s linear and angular velocity relative to the base frame as a function of the joint variables is to use the Jacobian of the direct kinematic transform (Lang, 1973; Wylie, 1975; Whitney, 1969, 1972). Since the Jacobian is crucial to later discussions of the inverse kinematic transform, let us now examine some of its main features. To begin, one needs six parameters in order to specify the spatial location and orientation of a rigid body in a given Cartesian reference system: three to locate a point fixed in the body relative to the base frame’s xyz, axes, and three orientation angles to express the body’s relative rotation about the xyz, axes. Treating the hand as a rigid body with its own coordinate system fixed within itself, the hand’s location and orientation relative to a base frame may be expressed by the location of the hand system’s origin plus three orientation angles. In the above discussion, it was shown that these values could be obtained from the B, matrix as separate functions of the joint variables, fi(dj), where i = 1, 2 ,..., m, and i = 1, 2 ,..., n. Referring once again to Fig. 4, these spatial values are labeled PH (with components (PH)rO , (PH)yO, (P,)zO) and /3H (with components @& , &A,, TG%&J Th ese six variables may be combined into an m x 1 column vector, S, where m = 6. In a similar fashion, the joint variables may be grouped together to form an 1z x 1 column vector, 4, where 12equals the number of revolute joints in the (humanoid) arm under consideration. It can then be shown (see Appendix D for details) that differentiating S with respect to time yields

112

ELLIOT

SALTZMAN

abbreviated as

IYH I MT; 4 = [$&,$2,...,$4JT; and J(4) = the Jacobian of the direct transformation

with respect to 4;

afi = K’

of partial derivatives, where Jij = afJa& is the element [ 1anin them xithn matrix row andjth column (i = 1, 2 ,..., m; j = 1, 2 ,..., n), and represents the ith j # K. In later by the symbol resulting from

component of $ discussions in the $/q$j, and refer a unit change in

per unit di when all other & = 0 for text, these elements are also represented to the changes in the spatial variable $ the joint variable I$~ .

Using this method, one could monitor the linear and angular velocity of a given point in a given segment relative to the base frame by: (a) using the appropriate B-matrix to define a set of 6 fi’s; (b) computing the Jacobian matrix; (c) feeding current joint angle and angular velocity information into J(4) and 4, respectively; and (d) computing the components of $l via Eq. (8). In anticipation of discussions below concerning the inverse kinematic transform, it is sufficient to note at this point that the utility of the Jacobian method lies in the possibility of computing the Jacobian inverse, J-l(4). If this can be found, one can calculate readily the set of joint velocities, 4, required to bring about a desired body-spatial motion, !%, of a given effector. Body spatial trajectory planning. Let us now return to the earlier, simplified example of reaching. Here, the problem of postural stability was “solved” by arbitrarily specifying that the shoulder girdle be spatially fixed relative to the base of support and hence also relative to the environmental task space. As a result, the intended reaching action did not require any additional postural activity to maintain the actor’s balance. In this situation the environmental-spatial movement representation was specified relative to the base or shoulder reference frame. Having discussed the direct kinematic transform, one can see how the spatial position, orientation, linear velocity, and angular velocity of the hand may also be perceived (nonvisually) in shoulder space. Since the shoulder girdle is fixed relative to the task space, these values are readily expressed in work-space coordinates. Once the spatial relationships between the effector and the task space are determined in shoulder frame and hence task-space coordinates, a relatively precise spatial path for the hand (and, therefore, the objects and surfaces labeled as “figures” in the environment; Talmy, 1975) may be specified in the work space via the shoulder frame. This procedure is usually referred to as trajectory planning (Arbib, 1975; Lewis & Bejczy, 1973; Paul, 1972; Waters, Note 6). Such planning must deal with the problem of selecting or generating a single spatiotemporal path from among the infinite number of paths that could possibly be taken. This could be done by attempting to minimize, for example, the variable of distance between the initial and goal locations while avoiding any intermediate obstacles.CAdditionally, an attempt could be made to minimize the effort expended

SENSORIMOTOR

REPRESENTATION

113

during the motion. Greene (Note 5) points out that a heavy object may be carried closer to the body than a light object because less active torque is required at the shoulder joint to support a closely held object against gravity. This is due to the fact that, when held close to the body, an object’s moment arm is relatively short. The movement path ultimately chosen will most likely involve a compromise among several such minimization criteria and, once the planning is complete, a single path is defined for the effector in shoulder space (and hence, task frame) coordinates over a given interval of time. The trajectory may also involve a path for intermediate portions of the arm (e.g., the elbow) such that they avoid any obstacles in the environment during this same time period. Body space-work/task-space mapping. In the above example, reaching was described relative to a set of spatial coordinates fixed in the shoulder girdle, which was in turn assumedto be spatially fixed relative to the baseof support, which wasitself fixed relative to the work space.There are several ways to locate and orient the shoulder frame with respect to the object- and surface-centered coordinate systemsin the work space (excluding tactile-haptic exploration which presupposesthe reaching activity we are attempting to explain). First, since the location and orientation of the support basewas initially specified in task-spacecoordinates, the spatial relation of the shoulder to either foot segmentvia matrix transforms could supply the required work space-shoulderspace information. Alternatively (or additionally), the location and orientation of the task space relative to the head segment may be provided visually or auditorily, and matrix transforms could again be used to compute the desired values. Unless the head were stationary relative to the trunk, however, the task description in head-centeredcoordinates would vary with head-trunk angle. This variability would ultimately be compensatedfor by the changing terms in the shoulder-head matrix, reflecting the shoulder’s objective fixed relationship to the task space. Body-segment space is thus defined relative to each individual body segment (e.g., trunk, foot, head, etc.). The spatial relationshipsbetween any given segmentand another are at leastpotentially available via the type of direct kinematic transformations discussed above. Segment spaceis thus “egocentric” in that spatial information is coded relative to coordinate systemsthat remain fixed in the body segmentsdespite changing relationshipsbetween the body (or individual segments)and the environment. Such relationships between body-centered and environmental coordinate systemshave been investigated and reviewed by many authors, notable among whom are Attneave and his colleagues (e.g., Attneave & Benson, 1969;Attneave & Olson, 1968; Attneave & Reid, 1967),Howard and Templeton (1966), Pick and his colleagues(e.g., Pick, Yonas, & Reiser, 1978; Rieser& Pick, 1976; Pick, Note 7) and Rock (1973). Body space-global invariant coordination: Postural control. Earlier discussionsof reaching in a standing posture involved the simplifying assumption that the shoulder girdle was artificially immobilized relative to the task space.Parts A and B of Fig. 5 illustrate (for this artificially immobilized state) the problem of reaching in a situation in which the actor wishesto place his hand at a height a1from the ground and at a frontal distance x1 from his heels. If balance were artificially maintained, he could do this via inverse kinematic procedures involving arm joint motion only. However, his line of

114

ELLIOT

SALTZMAN

FIG. 5. Interaction of instrumental and postural control during a reaching task, where G = line of gravity: (A) stationary standing actor; (B) arm raised to xizr , shoulder stabilized, G is displaced; (C) arm joint angles are the same as in (B), ankle and hip rotation compensate for displacement of G; (D) final solution of reaching problem with ankle and hip joint angles as in (C), but with new arm joint angle.

gravity, G, will be shifted by this redistribution of body mass to a region outside his base of support. (In Fig. 5B, the displacement of the line of gravity has been exaggerated for illustrative purposes.) If he were not artificially supported, he would fall over. It is clear, therefore, that a reaching or transporting action involves the displacement of the arm segment masses relative to the trunk, and entails a potentially destabilizing motion of the center of gravity relative to the stationary support base. For a freely standing actor, these movements of the body mass center are automatically compensated for by the actions of a “postural fixation” (Martin, 1967) or “pedate” (Fomin & Shtilkind, 1972, 1973; Turvey, et al., 1978) control system. The resultant postural corrections that are involved in raising one’s arms forward to the horizontal position while standing may be seen in Fig. 6A, B. In these illustrations, it is evident that the forward motion of the gravity center which would have resulted from arm movement has been cancelled

FIG. 6. Compensatory postural control: (A) normal standing position viewed relative to an environmentally fixed vertical reference line; (B) arms raised to horizontal position, body has moved radially backwards; (C) 34 lb. book held horizontally at arm’s length, body has moved further backward radially than in (B) (adapted from The basalganglia andposture by J. P. Martin. Copyright 0 I967 by Pitman Medical Publishing Company. Used with permission.).

SENSORIMOTOR

REPRESENTATION

115

by backward rotation of the body as a whole. This compensation is even more pronounced when the actor holds a 34 pound book in his hands (see Fig. 6C). Such compensatory postural control is an extremely useful property of the sensorimotorsystem and serves to maintain the actor’s spatial orientation to the global invariant of gravity. However, this type of control doesintroduce someproblemswith respectto calculating the joint motions required to move the hand through its desired task-spatial trajectory (i.e., the inverse kinematic proceduresdiscussedabove). If one assumesthat the actor in Fig. 5A is standing freely, the task of moving his hand to xi~i becomesmore complex and involves inverse kinematics defined over the ankle and hip joints as well as the arm joints (Fig. 5C, D). Figure 5C illustrates postural compensationat the ankle and hip joints while keeping the arm joints at the samevalues asin Fig. 5B. The hand is then at xaza , and additional arm motion must be usedto move the hand to the desiredxix1 location. Figure 5D showsonejoint angleconfiguration which satisfiesthe dual constraints of postural stability and desired hand position. Voluntary movement, therefore, appears to reflect both postural and task demands. Hess (1954; cf. also Jung & Hassler, 1960) emphasized this point and functionally distinguished two mechanismsinvolved in voluntary movement: (a) a “teleokinetic” (teleos= an end) mechanisminvolved in the voluntary, task-related aspectsof the movement; and (b) an “ereismatic” (ereisma = support) mechanism involved in providing postural stability during the movement. These different aspectsof movement may be referred to more simply as instrumental and postural, respectively. It is evident that the type of postural activity illustrated in Figs. 5 and 6 involves compensatory motion of the leg and trunk segmentsto offset the displacementof the center of gravity that accompaniesthe instrumental motion of the arm segments.In addition, however, several investigators (Belen’kii, Gurfinkel, & Pal’tsev, 1967; El’ner, 1973; Pal’tsev & El’ner, 1967) have demonstrated task-specific patterns of anticipatory postural activity which precede various types of instrumental activity. Such anticipatory activity indicates that the postural consequencesof instrumental movements are taken into account during the planning stagesof a given action. Sensorimotor

Coordination:

The Degrees of Freedom Problem

Let us consider the example of a freely standing actor whose support baseis specified in the task spaceand who wishesto reach for a nearby object. The only body segments that can be relied upon to remain stationary relative to the task space(barring loss of balance) are the supporting segmentsin contact with the ground plane. Consequently, the inverse kinematic transform is defined over the highly complex kinematic chain which has the feet as its base frame segmentsand the hand as its terminal segment. The reaching task thus requires simultaneous control over the angular motion of all the joints lying between the feet and the hand segments(i.e., the joint motions must be coordinated with one another). As mentioned in the Introduction, Bernstein (1967) characterized coordinated movement as involving the mastery of a kinematic chain’s redundant degreesof freedom. The following sectionsoutline severalmeansof achieving such coordination in the context of inverse kinematics.

116

ELLIOT

SALTZMAN

Degrees of freedom; generalized coordinates. The degreesof freedom for a given system are the least number of independent coordinatesrequired to specify the positions of the system elements without violating any geometric constraints (Groesberg, 1968; Timoshenko& Young, 1948). For example, if we describethe positionsof the particles A and B in Fig. 7A in terms of Cartesian coordinates, we need four independent coordinates: x1 , ya and xa , ya . Since these are independent (i.e., changing one will not affect any of the others), there are four degreesof freedom in this system. Figure 7B illustrates a

A

B

C

Degrees of freedom in two-dimensional systems with increasing degrees of constraint: (A) no constraint; (B) one constraint; (C) two constraints (adapted from Advanced dynamics by S. Timoshenko & D. H. Young. Copyright 0 1948 by McGraw-Hill Book Company. Used with permission.). FIG.

7.

simple hypothetical rigid body consistingof the particles A and B connected by a rigid, weightlessrod of length L. In this case,the Cartesian coordinates are not independent since they must satisfy the following equation of constraint:

(x2 - 0 + (3%- YIY = 0.

(9)

Thus, if three of the xy coordinatesare specified, the selectionof the fourth is constrained by the geometry of the system according to Eq. (9), and the system has three degreesof freedom. In Fig. 7C the samebar is attached at particle A to the fixed point 0 by another bar of length M. In this case,there are two equations of constraint, Eq. (9) and xl2 + y12 = M2,

(10)

and the system has two degreesof freedom. These examplesinvolve systemsdefined in two-dimensional Cartesianspace.In general, the number of degreesof freedom of such systemsis 2N - c, where N = the number of particles in the system and c = the number of equations of constraint. For systems defined in three-dimensional Cartesian space (i.e., each particle has its own xyz coordinates), the number of degreesof freedom is in general 3N - c.* s Note that a hinge joint introduces two constraints. When dealing with two-dimensional motion, one of these constraints is implicit since segmental motion is restricted by definition to a constant z-axis value. The other is explicit, and deals with constant interjoint or particle-joint distances (e.g., Eqs. (9), (IO) in the text). When dealing with three-dimensional motion, one must supply both types of equations of constraint if hinge joints are used (Horn, B.K.P., Note 15).

SENSORIMOTOR

REPRESENTATION

117

One can choose a set of independent generalized coordinates to specify any system’s configuration. The number of generalized coordinates will equal the number of degrees of freedom in the system. Thus, the system in Fig. 7A may be described by the set of generalized coordinatesx1 , yr , xa , ya ; the systemin Fig. 7B by the x and y coordinates of any point on the bar (including particles A and B) plus an orientation angle between the bar and either the x or y axis; and the systemin Fig. 7C by the two orientation angles between each bar and either the x or y axes. Figure 7C showsa system similar to the kinematic chain seenin Fig. 3. The segments’orientation angleswith respectto the base frame’s .r-axis are related to the joint angles(&) in a manner illustrated in Fig. 8. In this diagram, segment l’s orientation angle is $r , while segment2’s orientation angle is (+r + 4,). One can consider the generalized coordinates of any such open kinematic chain (i.e., only a single segmentis fixed to the baseframe) to consist of the set of joint variables associatedwith that chain.

FIG. 8. Two-dimensional linkage system with two segments and two joints; although not illustrated, each segmenti has mass (m,), length (ZJ, center of mass (corni), and a moment of inertia about its center of mass (lCOmi = $5 mJi2).

In all previous sections, we have been considering joints with only a single axis of either rotation (a hinge joint) or of translation (a prismatic joint). Such joints have one degree of freedom each, and therefore the total number of degreesof freedom in such systemsis simply equal to the number of joints. In human arms, somejoints are biaxial (e.g., the wrist joint) or triaxial (e.g., the ball and socket joint of the shoulder). Such joints have two and three degreesof freedom, respectively. In general, then, the number of degreesof freedom in a given open kinematic chain is equal to the sum of the degrees of freedom in each of the joints. The inverse kinematic transform. In the earlier example of a standing and reaching actor, the inverse kinematic problem consistedof finding the set of joint angle changes necessaryto move the hand segment along a given task-spatial trajectory. The position and orientation of the hand segmentin spaceis uniquely described by six spatial coordinates (three define the location of the segment’sorigin; three define the orientation

118

ELLIOT

SALTZMAN

of the segment’s axes to the base frame). After trajectory planning is complete, these six coordinates are specified as known functions of time or path distance (cf. the discussion at the end of the effector-speci$c environmental space section). From the direct kinematic transform matrix for the hand frame in base frame coordinates, a set of six equations may be derived expressing these spatial variables as separatefunctions of the n joint variables (cf. the discussionof the Jacobian matrix in the direct kinematic transform section). Essentially, the inverse problem reduces to solving a system of m equations (m = the number of spatial variables controlled during an action) for n unknowns (n = the number of joint variables, or more strictly speaking,the number of degreesof freedom in the kinematic chain). As long as m = n, it is generally possibleto solve for the joint variables. However, if the number of degreesof freedom in the kinematic chain (rz) is greater than the number of specifiedspatial variables (m), there are an infinite number of possiblesolutions, and further constraints must be imposedto selecta unique solution for a given action. In such cases,the system is indeterminate and the kinematic chain is said to contain redundant degreesof freedom. Since there is a maximum of six spatial variables for describing the kinematics of the hand segment, it is clear that any open kinematic chain with more than six degreesof freedom is redundant by definition. Most robot manipulators are designed with six degreesof freedom and are hence nonredundant. In these types of arms there are a finite number of joint angle configurations, typically a very small number, that will be associatedwith each point along a desired spatial (position and orientation) hand trajectory. For example, there will only be two sets of joint anglesfor most positions and orientations of the hand in the rightor left-handed Scheinman robot-manipulator arm (Lewis, 1974). This doesnot provide much flexibility for avoiding obstaclesthat may be cluttering up the task space; e.g., the hand may be able to reach the target, but the elbow may be obstructed due to the lack of a sufficient number of alternative arm configurations. Redundant arms, however, are highly flexible sincethere are an infinite number of joint angleconfigurations associatedwith each point along most spatial hand trajectories. Thus, during both target acquisition and maintenance, a redundant arm should be able to avoid successfullymost obstaclesin the task space.Clearly, then, redundancy is extremely valuable in a practical , sense. Human arms(to say nothing of the kinematic chain from the support baseto the hand segment)are highly redundant, with the degreesof freedom distributed in the following way: wrist motion (2), forearm pronation-supination (l), elbow flexion-extension (1), shoulder joint motion (3), and shoulder girdle motion (3). Thus, the human arm has a total of 10 degreesof freedom if one includes the limited motion of the shoulder girdle. How might one solve for the set of joint anglesrequired to move the hand along a given spatial trajectory for the relatively “simple” kinematic chain consisting of arm segments only ? To answer this question, consider again the earlier discussionof Whitney’s (1969, 1972) work and the Jacobian matrix. Equation (8) used the Jacobian matrix to expressthe direct kinematic transformation from known (sensed)joint velocities to a set of spatial linear and angular velocities for the hand segment.If oneis given insteadthe desiredspatialvariables as known quantities (i.e., as after spatial trajectory planning), and the number of unknown joint variables is

SENSORIMOTOR

REPRESENTATION

119

equal to the number of spatial variables (i.e., 1z = m), the system of equations may be solved by inverting the Jacobian matrix (Wylie, 1975), and using the following equation: 4 = J-l($) J-i(+)

* s, where 4 and $ are defined as in Eq. 8; and

(11)

= the inverse of the Jacobian matrix, =Qj an n x m matrix, where J;l = &/si is the element in thejth row and ith [ si I ’ column (i = 1, 2 ,..., n; i = 1, 2 ,..., m) and representsthejth component of C$per unit Si , when all S, equal zero for i # R. In other words, dj/Si is a symbolic representation of the changein the joint variabledi resulting from a unit change in the spatial variable si . Alternatively, dj/Si may also be interpreted as the change required in the joint variable & to produce, in combination with changesin the other joint variables, a unit change in the spatial variable si .

Thus, there are two major aspectsto the inverse kinematic transform: (a) inverting the matrix, and (b) solving the redundancy problem. Matrix inversions: Singularities. For the case where the number of joint variables equalsthe number of spatial variables, J(4) will be invertible and Eq. (11) will provide the required joint velocity vector only if the J(4) matrix in nonsingular, i.e., if the Jacobian determinant is different from zero at all points, or nearly all points (Symon, 1971). In general, the matrix will be nonsingular. However, the Jacobian is a function of 4, and it is possiblethat the arm will assumea joint angle configuration such that the determinant will equal zero and the matrix will have no inverse. Such singularities correspond to configurations in which, for any combination of the components of 4, at least one type of motion in !$ is either totally unobtainable (Whitney, 1972) or obtainable only in combination with other types of motion in 3. Mathematically, this correspondsto cases in which either all the elementsof at leastone row of the Jacobianequal zero or in which at least one row of the Jacobianis a linear combination of the other rows. These types of matrices are singular by definition (Wylie, 1975). For example, such singularities will occur at the boundaries of the arm’s reaching space,i.e., when the arm is outstretched, further reaching is impossible and the only bi-directional, independently obtainable spatial hand motion is a twist about its reach axis. There is no way to overcomethis type of singularity. Other types may be circumvented (Whitney, 1972) by: (a) reducing the size of the Jacobian to eliminate the “blocked” spatial commandsand inessentialjoint variables; or (b) by recruiting an extra degree of freedom (if the arm is redundant) in the region of the singularity. Finally, one should remember that since the Jacobian is a function of c$,new values must be found for the elementsof both J(4) and J-l(#) at each stage of the trajectory according to the current joint angle configuration. Redundant degrees of freedom. If the number of joint variables (unknowns) is greater than the number of spatial variables (equations) in the inverse problem, one cannot

120

ELLIOT

SALTZMAN

directly solve for 4. One can overcome this difficulty by “freezing” a number of joints i.e., reducing the number of unknowns), adding hand or “elbow” coordinates to s I.i.e., increasing the number of equations), or by additionally specifying optimality criteria to be satisfied during the movement (Whitney, 1969). One can also reduce the number of unknowns by specifying functional relationships between the joint velocities, e.g., by expressing one joint velocity as a ratio or as a more complex function of another joint velocity over a given spatial or temporal interval during the hand’s trajectory. Let us consider the methods of freezing and ratio specification in further detail. For a seven degree-of-freedom arm, if one wishes to freeze a given joint, the remaining joint velocities may be calculated by: (a) computing the corresponding 6 x 7 Jacobian matrix, (b) striking out the column corresponding to the frozen joint, and (c) computing the inverse matrix for solving the remaining six joint velocities. If one instead wishes to constrain a given joint’s velocity, dc (where c denotes constraint), as a ratio of another joint’s velocity, & (where d denotesthe “driving” joint), one again starts by computing the 6 x 7 Jacobian. However, the Jid elementsof the column correspondingto 4d do not simply depend on motion at & . Since & motion drives joint +c , the ith component of $I per unit & will depend not only on q& but also on 4, . Thus, each element $/bd will be expressedas (12) where K = the proportionality constant of the velocity ratio, && , and representsthe change in & accompanying a unit changein & . In other words, the column vector corresponding to Cc is multiplied by the scalar K and added to the column vector correspondingto & . The “old” & column is then struck out, and the inversematrix is computed to solve for the six free variables. Once calculated, the entire set of free and constrained variables may be recovered in a fashion similar to that used by Jacobsenand Mann (Note 8) via

9&e = c . &ree 3 [ 4:----I constrained where the term on the left of the equation is an n x 1 vector containing the free and constrainedjoint velocities; C is an n x (n - c) matrix containing the functional relationship between the free and constrainedvariables,where c equalsthe number of functional constraints; and the right-most term is an (n - c) x 1 vector containing the free joint variables. As an overly simplified example, consider a set of three joint velocities where 4, and da are free, and & is constrained to equal 24, . Then, if the free variables were calculated, we would have

SENSORIMOTOR

REPRESENTATION

121

In the case where the third joint is frozen, this expression becomes

[-km] =Eg$]. Let us now consider Bernstein’s (1967) Sensorimotor coordination: Kinematics. definition of coordination as “the process of mastering redundant degrees of freedom of the moving organ, in other words its conversion to a controllable system” (p. 127). Having defined degreesof freedom and redundancy, and having discussedsome of the ways redundancy might be incorporated into an inverse kinematic transformation, we are now in a better position to understand this statement.The major methodsexaminedfor dealing with the issue of redundancy were joint fixation and the specification of joint velocity velocity ratios. These techniques were emphasizedintentionally since they appear to be reflected in human sensorimotorbehavior. Several types of joint fixation can be distinguished. One types seemsto be structural in the sensethat it deals with consistent findings of fixation under certain anatomical conditions. For example, the degree of freedom corresponding to upward tilt of the shoulder girdle is fixed unlessthere is hyperextension at the shoulderjoint (Wells, 1971). Therefore, if one were planning to move the arm, one would eliminate this degree of freedom from the inverse transform unless one planned to hyperextend the shoulder. A secondtype may be referred to as a functional fixation, since it appearsto be related more closely to strategy and experience than to anatomy. This occurs during the early stagesof skill acquisition in adults (e.g., Bernstein, 1967) and children (e.g., Bernstein, 1967; Wickstrom, 1977). Thus, in the early stagesof a skill, the actor tends to keep much of his body rigidly fixed and reducesthe movement to a relatively simplekinematic form. For example, compare the left-handed and right-handed throwing patterns of a righthanded actor. Contrasted to the right arm, the left arm’s performance will appear rather stilted and rigid. There are alsoseveraltypes of constraints that relate motions correspondingto different kinematic degreesof freedom. Again, one type might be described as structural in the samesenseas was used for joint fixation above. As the shoulder girdle fixation example shows, upward tilt of the shoulder girdle will accompany any planned shoulder hyperextension. Additionally, sideward or forward elevation at the shoulder is always accompanied by shoulder girdle upward rotation (Wells, 1971).Waterland and Munson (1964) have discoveredmutual dependenciesbetween the motions of the head, shoulder girdle, and humerus (upper arm). These latter motions, however, seemlessclearly related to anatomical constraints than do the immediately preceding examples.Such linkages seem to be instances of functional synergies (Kots, Krinskiy, Naydin, & Shik, 1971), in the sensethat corresponding to each characteristic form of motive activity is a subdivision of all joints participating in the movement into a small number of connected groups of the sort that for the control of each of them one degree of freedom of the control system is enough (functional synergies). (p. 373) @O/20/2-3

122

ELLIOT

SALTZMAN

It can be seen that functional synergies involve the specification of relationships between the motions in a given set of joints and, as such, they effectively serve to reduce the number of degrees of freedom in a complex kinematic chain. Kots and Syrovegin (1966) p rovided evidence for a type of functional synergy that was more abstractly defined (i.e., defined in terms of actual joint velocity ratios) than that demonstrated by Waterland and Munson (1964). In their study, Kots and Syrovegin (1966) asked subjects to perform simultaneous movements at the elbow and wrist joints. Each subject participated in four conditions that involved all possible combinations of wrist and elbow flexion-extension, e.g., wrist flexion and elbow flexion, wrist extension and elbow flexion, etc. During each of the tasks, angular motion at each of the joints was measured. The major findings were that: (a) most movements included portions in which the joint velocity ratios were constant; (b) each subject had a constant preferred set of 5 & 2 distinct ratios used across all tasks; (c) each subject tended to use different but overlapping subsets of these ratios in each of the individual conditions; and (d) these ratios did not appear to be dependent on anatomical coupling between the joints. Kots and Syrovegin (1966) interpreted these ratios as evidence of functional synergies that serve to simplify the control problem associated with simultaneous coordination of the two joints involved in the task. However, other interpretations of Kots and Syrovegin’s data have been proposed by Bishop and Harrison (1977), and the reader should consult this latter study for further details of this controversy. The functional synergy matrix. One may hypothesize in the context of the previous discussionof inverse kinematics that functional synergies are the source of the joint velocity ratios usedto reduce the Jacobian to a square6 x 6 (or smaller) matrix. It then becomesconvenient to replace the joint velocity variables with joint “motor” velocity variables, reflecting a distinction suggestedby Whitney (1972). fn the caseof the seven degree-of-freedom arm examined earlier in Eqs. (12) and (13), we may begin again by computing the 6 x 7 Jacobian. Now, however, we will replace each 4j with a motor velocity variable, iI$j , such that there are separatemotors controlling the angular velocities of eachjoint. We may then define a set of elementary matrix transformations on the columns of this matrix to construct an equivalent 6 x 7 matrix (Wylie, 1975)that includes the effectsof the velocities of “functional synergy motors,” &ZF, that causesimultaneousmotions at severaljoints. Specifically, we will apply the elementarytransformation of adding a multiple of the elementsof a given column to the correspondingelementsof another column. This is the sameprocedure illustrated in Eq. (12) with joint velocity variablesstill being usedin the Jacobian. For illustration, considerthe following simplified example of a 3 x 4 Jacobian defined by three spatial and four joint-motor velocities:

SENSORIMOTOR

123

REPRESENTATION

Here, the second column from the left represents the effects of joint-motor Ma on the three spatial variables. If we now multiply the third column by K and the fourth column by P, and then add each of these columns to the second column, we

get s, ni, s, M4 s, M4

Now, the second column reflects the effects of functional synergy M,, on the three spatial variables. The matrix may be rewritten as

We may now strike out column one, three, or four. This results in the “freezing” of the corresponding motor, but not necessarily the corresponding joint, since the synergy motor may cause motion at this joint. After inverting this 3 x 3 matrix, we may solve for the desired a and MF values. Once computed, the entire set of n;ir variables (and therefore 4 variables) may be recovered via a C-matrix similar to that used in Eq. (13). The synergy matrix approach has several interesting features. First, it combines in a single construct the concept of functional synergies with the mathematics of inverse kinematics. Thus, each functional synergy matrix (or variant) consists of a distinct set of functional synergy and joint-motor variables which are represented in separate columns of the matrix. In the inverted form of a given matrix, each functional synergy and jointmotor variable is represented in a separate row of the matrix. These row elements take the form of il?ZJgi, and represent the change required in the functional synergy or joint motor variable, il?lj, to contribute to the production of a unit change in the spatial variable, si (see Eq. (11)). The relative contribution of a particular motor variable to a given spatial variable is a function of the contributions of the other motor variables associated with the inverted functional synergy matrix. Since different matrices consist of different sets of functional synergy and tuning variables, the relative contribution of a particular motor variable to a given spatial variable in one matrix is most probably different from the contribution of that same motor variable in a different matrix. Thus, for a given desired spatial hand trajectory in a redundant arm, the joint angle motions (and hence configurations) associated with each point along this trajectory will be different for each distinct functional synergy matrix used to plan the motion, Consequently, the

124

ELLIOT

SALTZMAN

joint angle configuration associated with the hand’s final spatial target also will vary as a function of the functional synergy matrix used in motion planning. A second interesting feature of the synergy matrix approach is that it allows several functional synergies, operating on distinct or overlapping sets of joints, to be represented in the MFj columnsof a singlematrix. Especially intriguing in this regard is the possibility of describing the simultaneousoperation of postural and instrumental synergiesdefined over a set of partially overlapping joints. Finally, the organization and function of the synergy matrix appearsto reflect many of the properties that Greene (1971; 1972; 1973; Note 1; Note 2) has used to characterize the action “style” of the sensorimotorsystem. Let us examine theseproperties in further detail. Greene (Note 1) distinguished between main, tuning, and jixed joint variables on the basisof their respective roles in the performance of a given action: Main variables tend to be parts of standardized “recipes” used to approximate desired movement patterns, and are controlled by low-level “function generators;” tuning variables are used to improve the movement approximation provided by the main variables; and fixed variables are those which are held constant. In the course of a movement, any given joint variable might be reassignedaccording to its role in the ongoing movement. Referring to the synergy matrix discussedabove, it appearsthat main variables correspondto the synergy columns of the matrix, with the function generators represented by the functional synergy “motors.” The tuning variables for a given function generator correspond to those matrix columns not associatedwith the output of that functional synergy. The fixed variables are those not representedas columns in the matrix. Thus, given a set of main variables controlled by a single function generator, these variables will be representedin a functional synergy motor column of the matrix. The tuning variables will be representedin the remaining columns. Once inverted and applied to the spatial velocity vector, the matrix will output the set of motor variables (functional synergy and single joint) which will produce the desired movement. Thus, given a functional synergy and an appropriate set of additional joint variables, the movement approximation provided by the functional synergy will be automatically tuned via operation of the inverted functional synergy matrix. It is significant to note that Greene (1973) included joint velocity ratios as instancesof low-level function generators, and also stated that tuning can act to modify these ratios. This latter statement implies that a given joint variable can be both a tuning and a main variable; i.e., the motion of a given joint may be simultaneously affected by the operation of a functional synergy motor and a joint motor, which is in agreement with our earlier discussionof the properties of the functional synergy matrix. In addition, the use of the functional synergy matrix allows one to maintain the notion of underlying resemblancesbetween actions sharing a common set of recipesor function generators.Thus, movementsdistinguished only by different tunings of the samerecipe may be describedas “cousins” (Greene, 1973), and their computation is thought to be simplified upon the recognition of this common structure. Some hypothesesconcerning the nature of this recognition processwill be discussedin the following section. Finally, Greene (1973) has also discussedthe need for transitionfunctions that help to smooth the transition when control passesfrom the operation of one function generator

SENSORIMOTOR

REPRESENTATION

125

to the next in the course of a given movement. As with tuning, these smoothing procedures are posited to occur automatically. In terms of the functional synergy matrix planning procedures described above, however, it seems unnecessary to distinguish between tuning and smoothing functions. For example, consider a hypothetical desired spatial trajectory for the hand segment involving the use of two different function generators along adjacent, nonoverlapping trajectory segments. The required joint motion plan will be generated via two functional synergy matrices with different functional synergy motor columns and (possibly) different joint-motor columns. These planned joint motions, if properly executed, will provide a smooth spatial hand motion in the region of transition between function generators by virtue of the accompanying patterns of generated main and tuning variables. Thus, transition functions per se would be unnecessary for movement planning procedures. However, as suggested by Greene (1973), they are probably necessary components in sequentially controlling the relative offsetsand onsetsof function generators operating adjacently in real-time. Representation of skills: I. Kinematics; coordinate structures. By considering the performance of a familiar action in the context of levels of representation in the action plan, it becomespossibleto arrive at somereasonablehypothesesconcerning the nature of sensorimotor skill. In the present section, we will consider those factors that seemto contribute to the kinematic nature of skill, without discussingthe kinetic “knowledge” that is alsoseenin a truly skilled act. This latter aspectof skill will be explored under the joint torque sectionsbelow. If one accepts the possibility that the inverse kinematic transform involves processes similar to Jacobian inversion, and if one agreesthat functional synergiesmight indeed be useful for mastering redundant degreesof freedom, then one is left with (at least) two puzzling questions. First, what criteria are usedto select a particular joint velocity ratio in constructing the functional synergy Jacobianfor a familiar type of action under given task conditions? And second, how might the selection processitself be characterized? The following admittedly speculative schemeis proposed as one solution to the former question:

(a) A given functional synergy is defined with respect to a set of simultaneous motions produced acrossa given set of joints. Particular setsof functional synergiesare associatedwith the performance of particular types of actions; e.g., functional synergies producing cyclic lower-limb motion are involved in locomotion and bicycling, while arm and hand functional synergiesare involved in writing; (b) each functional synergy is internally represented in an abstract model of the actor’s body spaceand the actor’s relation to the task space.This is the beginning of the type of “complex data structure” that appearsto be involved in organizing the use of functional synergiesin sensorimotoractivity (Greene, Note 1, Note 2);Q 9 It should be mentioned in the concept of different those discussed by Greene

at this point that the data structures discussed in this paper are rooted levels of action representation and hence appear to be different from (Note I, Note 2). In other aspects, however, they are quite similar.

126

ELLIOT

SALTZMAN

(c) each internal functional synergy representation is defined at several levels of the action plan by predictively modeling the level-specific consequences of activating the functional synergy. Such activation would represent a given pattern of joint-motor velocities in the abstract motors of the functional synergy model.lO The consequences of these hypothetical activations for the various levels would be the joint motions corresponding to these motor velocities, the associated spatial motion via direct velocity transforms of the effector being controlled (e.g., the hand in a reaching task), and possibly, the conceptual relation between the controlled segment and the environmental goal (e.g., approach vs avoidance); (4 each functional synergy model is associated with a set of motor velocities in which each element in the set is defined by a distinct joint-velocity pattern acrossthe joints involved, with eachpattern thus being representedat the joint motion level asa set of relatively fixed joint velocity ratios (cf. Kots & Syrovegin, 1966). Thus, each different ratio would be represented by a different proportionality constant of the type seenin Eq. (12). At the spatial level, each elementwould be interpreted asa vector specifying the direction and speed of effector motion which would result from the activation of the functional synergy element; and (e) at a given point in the effector’s desired spatial trajectory, the particular joint velocity ratio usedto construct the functional synergy matrix is selectedon the basisof the degreeto which its associatedspatial vector approximates the desiredspatial vector. For example, in a reaching task, the set of relevant synergiesis specified at the effector level of the action plan. Each of thesefunctional synergieshas at any point in time a set internally representedvariants associatedwith it, such that each variant is represented at the spatial level by a vector and at the joint motion level by a joint velocity ratio. The spatial vector closestto the desired vector will be selectedand the correspondingjoint velocity ratio will be used in the inverse transformation. The above schemedescribesthe way functional synergy information may be organized in order to permit an appropriate choice of joint velocity ratio for the functional synergy matrix. It doesnot deal, however, with the manner in which this choice is made. We will return to this point after distinguishingamongthe related conceptsof functional synergies, coordinate structures, and muscle synergies. First, the joint-related notion of functional Synergyis seenas distinct from the complex data structure involved in abstractly representing and updating the modeled effects of a functional synergy’s activation at the different levels of the action plans. This active data structure will be labeled the coordinative structure associatedwith a given functional synergy or one of its variants. And, second,the reader should note that the term coordinative structure has been used in a different senseby several authors (Easton, 1972; Turvey, 1977; Turvey, et al., 1978) lo In examining type of predictive Velocity outputs however.

the muscle level of the action plan below, modeling that is based on force outputs of the modeled functional synergy motors

I will discuss the possibility of another in abstract models of muscle activity. will suffice for the present discussion,

SENSORIMOTOR

REPRESENTATION

127

to describe what we will refer to as muscle synergies (as distinct from functional synergies) in the section below dealing with the muscle level of the action plan. The term “muscle synergy” will then be used to label a “group of muscles, often spanning many joints, that is constrained to act as a unit” (Turvey, et al., 1978; this was the authors’ definition of coordinate structure). This labeling change was introduced to better capture the relative complexity and simplicity of abstract data structures versus concrete muscle groupings, respectively. Having distinguished among functional synergies, coordinative structures, and muscle synergies, let us turn to the issue of how a given joint velocity ratio is selected for inclusion in the functional synergy matrix. There appear to be two distinct possibilities. The first is that the desired spatial vector and the set of coordinative structure spatial vectors (i.e., those vectors provided by the coordinative structures associated with the functional synergy variants) are related via the stimulus-response mapping operations of a limitedcapacity central processor or “executive.” As such, these operations should demand attention or cognitive effort (Kahneman, 1973; Posner & Klein, 1973). Alternatively, it is also possible that this mapping process is automatized, in the sense that “low level detectors” (Greene, 1971, 1972, 1973; Note 1, Note 2) associated with each coordinative structure might signal when the desired spatial vector lies close to the output spatial vector of a given functional synergy variant. In the case where more than one coordinative structure “decides” that its functional synergy variant is appropriate for producing the desired output, there must be some interaction network defined among coordinative structures such that a single functional synergy variant is ultimately activated with no higher-level executive intervention. Presumably, the coordinative structures could become more sophisticated due to experience with certain types of actions in a manner which would allow automatized functional synergy variant selection at even higher levels of the action plan. Thus, in a reaching task, a coordinative structure might make its decision on the basis of the conceptual level appropriateness of the output of its associated functional synergy variant (e.g., potential for successful approach or avoidance). One should note, however, that even if the executive were responsible for spatial stimulusresponse mapping, it would still be ignorant of the particular functional synergy jointvelocity ratio selected for inclusion in the functional synergy matrix. That is, the executive could make its decision on the basis of information presented in purely spatial form, and would not necessarily have to be aware of the lower-level joint-motion consequences of its decision in order to produce an appropriate action. It is possible that the predicted consequences of activating a given functional synergy would be to match the desired spatial output. In such a case, no tuning would be required; the functional synergy matrix step could be bypassed; and the functional synergy velocity ratio could be directly employed. Such situations may be quite rare. However, there are other contexts in which the construction of the functional synergy matrix might quite regularly be bypassed. For example, if the precision requirements of a task were low enough such that a rather loosely defined movement would suffice, functional synergies could be directly used. Another example would be in situations where time pressures might dictate a fast, crude, and approximate action but might not allow a slower and more precise action.

128

ELLIOT

SALTZMAN

Sensorimotor timing. We should briefly consider the manner in which temporal constraints may be integrated with the spatial aspects of movement. In earlier discussions of the environmental space level of the action plan, we noted that a separation of spatial and temporal motion parameters had been maintained throughout that level. It was possible to represent an effector’s planned Cartesian spatial coordinates as functions of the distance along the desired path (i.e., x(h), w h ere h = path distance). Path distance could be expressed independently as a function of time (i.e., h = A(t)) and the relation between desired Cartesian coordinates and time would then be indirect (i.e., x = x(h(t))). It was also mentioned that the spatial and temporal aspects of movement would be related at some point between the body-spatial/joint-motion levels and the joint-torque level of the action plan. Functional independence between spatial and temporal motion parameters in the evolving action plan may be maintained throughout the inverse kinematic transform stage. If an effector’s planned spatial trajectory is specified as a function of distance along the desired path, the required joint angle trajectories also may be computed as functions of path distance (h) by rewriting Eq. (11) as

where d#/dh and dS/dh are the derivatives with respect to h of # and S, respectively; and J-l($) is defined as in Eq. (11). In the direct kinematic transform, currently monitored joint motion and effector spatial trajectory information may be expressed with respect to desired path distance only after the inverse kinematics stage. Therefore, spatial effector and joint motion information in the direct transform probably will be expressed directly with respect to time, but only indirectly with respect to path distance. Desired joint motions must eventually be expressed, however, as angular velocities and accelerations with respect to time, which serve as the inputs to the joint torque level of the action plan. Such joint information must be expressed temporally because, among other things, it will be used to represent the action in kinetic energy terms, and would thus be useless unless expressed as a function of time.

Joint Torque Level At the joint motion level of the action plan, a movement can be represented kinematitally as a set of joint positions, velocities, and accelerations expressed as desired functions of time across the set of joints involved in the motion. Although such information appears to be necessary in order to execute an intended action skillfully, it is not sufficient. There are two reasons for this. The first deals with reactive mechanical phenomena and the second deals with environmentally supplied forces arising in the course of movement. Reactive mechanical phenomena are those torques induced at a given joint as passive dynamic consequences of velocities and accelerations produced at remote joints. As mentioned in the Introduction, Bernstein (1967) described the highest degree of skill as

SENSORIMOTOR

REPRESENTATION

129

the stage in which these reactive phenomena are actively predicted and incorporated into the action plan. Yet, even if these reactive torques are utilized in a given action, the resulting description of movement will still be incomplete. Consider, for example, a standing actor who moves his arm from a position at his side into a forward, horizontal position with respect to his trunk. If this action is performed while standing upright, shoulder flexion torque must be exerted against the torque supplied by gravity. On the other hand, if the action is performed while upside down, the gravity torque will assist the motion; and if a controlled rate of arm motion is desired, shoulder extension torque must be exerted against the gravity torque. Likewise, the pattern of joint torques required to maintain a given static arm-joint configuration will vary according to the trunk segment’s orientation to gravity. Thus, the same kinematic description of a movement or static posture will realized as different patterns of exerted joint torques depending on the actor’s relation to the environment. Finally, it should be noted that physical contact between an effector and environmental objects (and surfaces) requires the environment to “exert” reaction forces and torques on the effector that are equal and opposite to the forces and torques exerted by the effector on the environment (Newton’s third law). Thus, a given kinematically defined movement will have multiple kinetic descriptions corresponding to the patterns of forces and torques exerted at the environment-effector interface. As Turvey and Fowler (1978) h ave emphasized, a theory of movement based solely on kinematics is not a theory of action, but one of mime. An adequate representation of a skilled action should therefore integrate both linkage and environmental kinetics with a strictly kinematic description of the intended motion. One way such integration might occur will be discussed in this section of the paper, which describes a unified mathematical treatment of linkage-mechanical, gravity, and contact effects arising in the course of movement. Discussions will focus on the nature of skill representations, rather than on stages of development of such representations occurring during skill acquisition. Thus, emphasis is placed on both the environmental and body-linkage knowledge which a skilled actor brings to a given task situation, and on the way this knowledge might be used to plan a particular skilled action. The mathematical description of movement kinetics presented below relies on a Lagrangian approach to dynamics as opposed to a Newtonian one. The reason for choosing the Lagrangian approach is that it is advantageous for dealing with systems having many degrees of freedom, since it involves formulating movement equations directly in terms of generalized coordinates (Meriam, 1966). This clearly entails a considerable simplification of movement analysis, as the geometric constraints of the system are implicitly incorporated into the statement of the problem (cf. the earlier discussion of degrees of freedom). For example, the spatial locations and orientations of the segments and joints, the internal mechanical forces within the segments, and the reaction forces and torques between segments may all be ignored since these constraints have already been considered in formulating the problem in terms of generalized coordinates. A Newtonian approach must explicitly express these in the analyses. In addition, the Lagrangian method involves only scalar functions of the generalized coordinates. The scalar analysis is simpler mathematically, since it eliminates the complications of vector analyses (e.g., keeping track of the signs of vector components or using the right-hand rule to label

130

ELLIOT

SALTZMAN

correctly the sign of a cross-product term (Groesberg, 1968)). Finally, the Lagrangian approach is more compact mathematically since all equations of motion are systematically derived from a single comprehensive scalar function called the Lugrangian function or the K&&c potential. On the other hand, the Newtonian method derives equations of motion through separate considerations of each segment in the system (Groesberg, 1968; Meriam, 1966). Let us now examine the Lagrangian approach (Groesberg, 1968; Meriam, 1966; McCuskey, 1959; Symon, 1971; Wells, 1967; Horn, Note 9) in more detail, as applied to the kinetic aspects of sensorimotor activity. Lagrangian Overview; Linkage Kinetics For the purpose of simplicity, discussion will be restricted to a simple two-dimensional linkage system in which there are two rod segments of length Z1and la , and mass m, and ms , respectively; each segment’s mass is uniformly distributed along its length; and each joint is revolute, having one degree of freedom about an axis perpendicular to the xy plane (see Fig. 8). The system, therefore, has two degrees of freedom, which allows the tip of segment 2 to be placed at any position in the reaching space. Orientation, however, cannot simultaneously be freely controlled. Similarly, two degrees of freedom are sufficient to allow the production of an arbitrarily chosen force (magnitude and direction) at the linkage tip, but not to allow comparably arbitrary, simultaneous torque production. This system has been examined by Horn (Note 9) with a Lagrangian analysis of linkagemechanical and gravity effects. The treatment of these topics in the present paper is essentially identical to Horn’s analyses, except for the omission of Horn’s simplifying assumption that m, = ma = m and li = 1, = 1.In the discussions below, this assumption is not made, since it appears desirable to distinguish the relative roles of each link in the derived equations of motion. The Lagrangian analyses of environmental contact reaction forces, however, were derived explicitly for this paper. Before proceeding further, one should note that the Lagrangian method becomes much more complex computationally (although not theoretically) when extended to threedimensional motion. Without additional modifications (e.g., such as those introduced by Horn, Hirokawa, & Vazirani, Note lo), it appears that this method may be too unwieldly to provide a practical, real-time solution to the control of sensorimotor behavior. I will return to this issue below, and will outline a promising approach to this problem (Raibert, 1976, 1978; Horn & Raibert, Note 11; Raibert, Note 12, Note 13). For now, however, a straightforward Lagrangian analysis of two-dimensional motion will suffice to describe the main principles of movement kinetics stated in terms of generalized coordinates. Lagrangian analysis. As mentioned above, the Lagrangian approach to describing the kinetics of a linkage system is based on the concept of the Lagrangian function or kinetic potential (L), which is equal to the difference between the kinetic energy function (K) and the potential energy function (P) of the entire system expressed in terms of generalized coordinates. Once L is formulated, the equations of motion follow directly from the following expression: j = 1, 2,..., n;

(15)

SENSORIMOTOR

REPRESENTATION

131

where: (a)

There is one equation for each of the n degrees of freedom in the system;

(b)

qj = the joint position variable at the jth joint = 4) ;

(c) & = the joint velocity variable at the jth joint with respect to time = & ; (d) &!,j&& , aL/aq, = the partial derivatives of L with respectto dj and qj , respectively; (e) the terms on the left side of the equation represent the effects of the kinetic and potential energy of the system. All conservative forces acting on the system (e.g., gravity, pure elastic forces, etc.) are included here; and (f) the terms on the right represent the effects of all the nonconservative forces (f = 1) 2,..., M) acting on the system (e.g., muscle force, friction, etc.). C,“=, Qzfj are the generalized forces associatedwith eachof the system’sdegreesof freedom. It should be noted at this point that for a linkage in which all degreesof freedom are revoiute, Qrj will have the dimensionsof torque. Thus, the generalizedforce corresponding to muscle activity at a given joint is the resultant muscletorque, TMj , at that joint. Similarly, the Qrj correspondingto the normal contact reaction force of an environmental surface onto the tip of the linkage is TNj , the resultant torque that is induced at each of the joints by this component of the contact force. One can alsospeakof the generalized tangential (e.g., frictional) contact reaction force, TTj , corresponding to the torques induced as a result of the tangential componentsof contact forces encountered by the tip in moving across an environmental surface. Finally, the torques induced by gravity, TGj , may also be treated as generalized forces on the right side of Eq. (15), provided one does not include gravitational potential energy in the formulation of L on the left side of the equation. We are now ready to analyze the motion of the systemin Fig. 8 accordingto Lagrangian methods, and will start by deriving a value of L for the system that includes only the kinetic energy terms. The aim is to begin with a treatment of strictly linkage-mechanical effects, which may be derived exclusively from the kinetic energy terms, before attempting to deal with environmentally produced kinetic effects. It should be noted that the treatment presented below is rather abbreviated, and includes only the major steps of the derivation. A more detailed analysisis provided in Appendix E. Linkage-mechanical effects (kinetic energy terms). In earlier sections, it was noted that motion at a given joint could be influenced by reactive torques induced by motion at remotejoints. For example, if one desiredto flex the elbow and therefore only exerted an elbow flexion torque, one would find that an undesiredshoulderextension accompanied the elbow flexion. Conversely, if one exerted only shoulder flexion torque, shoulder flexion would be accompaniedby undesired elbow extension. Such unplanned motions are the resultsof dynamic coupling between motions at eachof the joints, and are functions of joint acceleration and velocity. These linkage-mechanical effects can be shown to derive from the kinetic energy terms in the Langrangian function. We will now examine these effects in the context of the linkage system in Fig. 8.

132

ELLIOT

SALTZMAN

The kinetic energy of each segment may be expressed as the sum of two components (Horn, Note 9) each expressed with respect to an inertial reference frame (e.g., the xy frame in Fig. 8). The first reflects the instantaneous linear translation of the segment’s center of mass. Since the segments in Fig, 8 are thin rod segments with uniform mass distributions, the center of mass of each is located at the geometric center of the segment. This linear contribution is simply &mvz. The secondcomponent reflects the instantaneous angular motion of each segmentabout its center of mass,and equals41~~. In this expression, w is the segment’sangular velocity about its masscenter, and I is the moment of inertia about the masscenter, which equals&ml2 for the rod segmentsbeing considered. The angular velocities of links 1 and 2 are & and (dr + $,), respectively. The linear velocities of their respective centers of massare 1$Z,$, 1 and 1Z& + +Z,(r+$+ d2)1.The total kinetic energiesfor links 1 and 2 are Kl = 9(mli42) d12,

(164

and K2

=

+Hw$~(Z,~

+

3h2

+

412

~0s

d2) +

m29Ad2(3~22

U2 ~0s ~5~)+ %4220122>).

+

WI

The sum of Kl and K, expressesthe total kinetic energy of the system.In this example, then, L = Kl + K, and one can expressL, as LK = %h2(m2(k2 + 3122+ 412~0s +2) + +

m19k2)

+

d142(m2(3Z22

+

U2 ~0s 42)) (17)

+2”(m2W2”)))a

Performing the operations on L specified in Eq. (15) with respect to the generalized coordinates (i.e., the joint variables associatedwith joints 1 and 2: C&, 42 , C&, $a), one arrives at the following expressions:

--d aL - - aL = T hfl = (m2V12 + V22+ 412cos +2) + v dt ( ah ) a+, +

(m2W2” --

+

W2

~0s

542ii2

B

-

(3m2V2

mlV12)~1

sin

$2)d22

---

d aL - (- - aL = T h42= (m2(*Z22+ W2 cos d2>)41 -dt a+, ) a+, E +

*$2

+ F

(4m2412

sin

42> d12.

V3b)

G

Using these equations, one may compute the resultant muscle torques required to produce a desired pattern of joint accelerations.In doing so, one needsto specify the current joint position and velocity variables as well as the desired joint accelerations.

SENSORIMOTOR

REPRESENTATION

133

One may segregate the terms in the above expressions to distinguish between dynamic coupling effects due to acceleration and those due to velocity. The velocity effects are due to centripetal, centrifugal, and coriolis forces in the multijoint system. The acceleration effects may be described by constructing an acceleration sensitivity matrix (Horn, Note 9) which has several interesting properties. This matrix may be expressed as: a 11 = A [ a 21 -I? -

uI2 = B az2 -F’ - I

(1%

where A, B, E, F are the joint acceleration coefficients in Eq. (lSa, b). If one examines this matrix, one can determine that each element aii is expressed in units of moments of inertia (Ti&). Th e main diagonal terms represent the moments of inertia for link 2’s rotation about joint 2 (a&, and for the rotation of links I and 2 taken as a unit about joint 1 (all). The off-diagonal terms represent the dynamic coupling between the joints due to joint acceleration, and are symmetrical across the main diagonal; i.e., B = E in Eq. (lSa, b). It should be noted that each off-diagonal term, aij (i # j), represents the muscle torque required at joint i to compensate for the undesired torque at joint i induced per unit of acceleration at joint j. Thus, if one ignores velocity product terms for the moment, one may write

where .. 3 and. TM are expressed as column vectors, TM1 = u,i& + (II,& , and TM2 = u& + CL&~ . Referring to the earlier example of producing shoulder flexion without elbow extension, one can see that if joints 1 and 2 are labeled as the shoulder (flexionextension degree of freedom only) and elbow, respectively, then this desired pattern corresponds to 4, = +X (positive sign indicating flexion) and d;s = 0. From Eq. (20), this results in TM1 = qIX, TM2 = a21X; i.e., the elbow-flexion torque required to negate unwanted linkage-mechanical elbow extension is “automatically” provided via operation of the acceleration sensitivity matrix. Finally, referring to Eq. (18a, b), it becomes evident that each term in the acceleration sensitivity matrix depends on angular position but not velocity variables. For the main diagonal terms (for the 2-link system being discussed, only a,, is dependent on +), this means that the moments of inertia for rotation at a given joint vary as a function of I$. Thus, a,, in Eqs. (19), (20) will be largest when $a = 0” (cos& = fl) and smallest when +a = 180” (cos 4s = -l), corresponding to linkage configurations for which the moment of inertia with respect to joint 1 is maximal and minimal, respectively. For the off-diagonal terms, the dependence of the elements on 4 means that the extent of acceleration-related dynamic coupling varies as a function of joint angle configuration. Interaction with Environmental Forces Gravity effects (potential energy terms; generalized forces). In earlier discussions of action planning, gravity information was used at the environmental space level of the

134

ELLIOT

SALTZMAN

action plan to specify certain constraints on the positions and motions of task-space objects (e.g., orientation of a full glass of liquid), and was used at the body-space level to orient the actor posturally to the global environmental invariants (i.e., the line of gravity, the plane normal to this line). It is possible that one can sense the direction of gravity kinesthetically (Fitger, 1976). This could be accomplished by monitoring the relations between patterns of joint torques when maintaining or changing limb configurations in a gravitational field. We shall soon discuss how this might be done. First, however, we will discuss how gravity torque effects are incorporated into muscle torque computations when the direction of gravity is known. Again, derivations will be brief as details are presented in Appendix F. If one assumes that the line of gravity is parallel to the y-axis and directed toward the x-axis in Fig. 8, then one may express the potential energy (P) of each segment as: (gravity force acting at the segment’s center of mass) x (the y-coordinate of the segment mass center). Thus, where g = the acceleration of gravity:

Pl = (w)(% sinv4), pi?= ew)Vlsin+,+ t&JWA + dd*

(2la) @lb)

The sum of PI and P, represents the total potential energy of the system. For the system under consideration, one may now express the Lagrangian (L) with respect to both kinetic and potential energy:

L = LK- (flg(%sinA)+ wf(4sinA + Q&, sin($, + M),

(22)

where L, is given in Eq. (17). Per f orming the operations specified in Eq. (15), one arrives at

TAO = E$l+

F& + Gq$2 - g(&m& COS(+~+ &)), TGZ

(234

where A through G are as in Eq. (18a, b). TG1 and TG2 represent the torques induced by gravity at joints 1 and 2, respectively, which must be taken into account in computing T M1 and T,,,,2 when movements are performed in a gravitational field. As mentioned earlier, gravity torques may be treated as generalized forces on the right side of Eq. (15) rather than as potential energy terms on the left side of this equation. The formula for a generalized force (Groesberg, 1968) associated with degree-of-freedom j is

SENSORIMOTOR

135

REPRESENTATION

where Fk = the external force vector applied to the system; rk: = the position vector of points at which F acts on the system; lz = the number of points at which F acts on the system; and the dot denotesthe scalarproduct operation. If, asin the above example, the direction of gravity is known to be parallel to the y-axis, the acceleration of gravity may be expressedas g = (gz, gJ, with g, = g and g, = 0. Since gravity forces act on the system at the masscenter of each segment,N = 2 and rk denotesthe position vectors of the mass centers expressedin xy component vectors. Using these values, one may compute values for QG1(= TG1)and QG2(= TG2)which are identical to those in Eq. (23a, b). If, however, the direction of gravity is unknown relative to the xy axes, one may reformulate Eq. (24) as

QG~= 5 (F, + F,)k - +,

3

k=l

where F, and F, denote the unknown x and y component vectors of the gravity forces applied to each segment.From Eq. (29, one arrives at the following:

TG, = gd-hS4 \ + bM& TG~

=

gz(--m& --

sin 41) - Wl sin4J / \/ - (m& sin& + dd> cosA> + (mJl CLv91) + (m& coG + A))), W4

+

~4))

+

ihkktl,

COS(~~

.I

+

74)).

K

These equations may be rewritten in matrix form as 1

(27)

where H, 1, J, K are the coefficients of g, and g, in Eq. (26a, b). If this coefficient matrix can be inverted (i.e., if the determinant #O), one may proceed with the inversion and solve for the unknown g, and g, in terms of sensedjoint torques using (28) Consider the action of drawing a line on a blackboard. If one could measurethe force exerted at the point of contact from the tip of the chalk onto the surface of the board, one could resolve it into two components, one normal or perpendicular to the surface and one tangential to the surface. These environmentally directed force componentsmay be labeled Fe,, and FET , respectively. If one could then measurethe reaction force exerted from the surface onto the tip of chalk, one could again resolve this into normal and tangential components relative to the surface. These linkage-directed reaction force components may be labeled FRN Contact

eflects: Normal

reaction

force.

136

ELLIOT

SALTZI+N

andFRT, respectively. Assuming the surface is rigid, one knows by Newton’s third law that FRN = -FEN . Assuming the surface is dry, it is more difficult, however, to specify FRT since FRT = FF , where FF is the frictional force and is given by

In this equation, pLdis the dynamic coefficient for dry friction between two sliding surfaces, and depends on the nature of both surfaces (e.g., surface “texture,” the degree of roughness or smoothness). It should be noted that FF acts in a direction opposite to that of the linkage tip velocity at the contact point, and is relatively independent of both the speed of motion and the area of contact (Wells, 1967). In the present discussion of normal contact reaction effects, it will be assumed temporarily for the sake of simplicity that pd = 0 and, hence, that FRT = 0. The tangential contact effects of frictional forces will be examined in a later section. In this simplified version of “writing” with a frictionless blackboard and piece of chalk, the only existing contact reaction force is FRN (note: Tracing is a more appropriate term now, since writing is impossible without friction). Consider an actor who traces out a curve on the blackboard. If one then removes the blackboard, the actor may also trace out an identical curve on an imaginary “blackboard” coincident with the original. Kinematically, these actions are identical, and the same joint positions, velocities, and accelerations as functions of time may be used to describe each action. However, the actions will be different kinetically in that when writing on the real blackboard, the actor may exert a range of FEN’s which will be matched by the FRN’s supplied by the board. Thus, the FEN’s are prevented from causing any undesired motion of the linkage tip into the plane of the board by the constraint forces, F,, , provided by the board’s surface. On the other hand, when tracing the same trajectory on an imaginary surface, such constraint forces are no longer available and any FEN exerted at the tip will produce undesired motion into the plane of the fictitious surface. It is evident, then, that the constraint forces easethe problem of controlling the pattern of joint torques (at a given point in time) that result in producing FENat the linkage tip, by allowing the motion to be successfully accomplished with less-precisecontrol of the magnitude of this joint torque pattern. During contact with the real blackboard (i.e., as long asFEN is exerted into the plane of the board), the linkage tip is no longer free to move in three-dimensionalspaceand the system no longer has three degreesof freedom. Rather, the linkage tip is constrained to move on the two-dimensional surface provided by the blackboard and the system is thereby reduced from three to two degreesof freedom. From the earlier discussionof generalized coordinates, it is known that one can represent this loss of one degree of freedom during contact by introducing an equation of constraint into the description of the system that relates the original coordinates via an equation representing the geometry of the board’s surface (i.e., shape, location, and orientation of the board in body space).Similarly, the motions of turning a crank or of moving an object along a mechanical track have only one degree of freedom. There are thus two equations of constraint associatedwith each movement. For cranking, the equations are those des-

SENSORIMOTOR

REPRESENTATION

137

cribing the geometry of the plane in which cranking takes place plus the circular path taken by the hand in this pIane; for motion along a track, the equations once again describe the geometry of the surface in which the motion occurs and also the path taken on that surface. In the same way that eliminating a degree of freedom in three dimensions is accomplished by restricting motion to a two-dimensional surface, one can eliminate a degree of freedom in two dimensions by restricting motion to a one-dimensional line. This situation is illustrated in Fig. 9 (note: A straight line was chosen for ease of illustration, but any curved line would provide a similar constraint to one-dimensional motion),

FIG. 9. Two-dimensional linkage system with two segments, two joints, and with the linkage tip constrained to move along a const~uint line; F RN = normal reaction force, FEN = normally directed force exerted by linkage tip, Ff = tangentially directed friction reaction force, V = velocity of linkage tip.

The reduction of the system’s degrees of freedom can be represented by an equation of constraint describing the line in terms of x and y coordinates. Examination of this figure shows that this loss of a spatial degree of freedom is accompanied by the loss of a degree of freedom corresponding to one of the joint variables. Thus, as long as contact is maintained with the line, the joint variables are no longer independent. If one angle is freely chosen, then the other is constrained to take on a value which is a function of the first joint’s angle and the constants associated with the new system (e.g., the lines y-intercept and slope). As discussed earlier, one property of a Lagrangian analysis is that the constraint forces do not explicitly appear in the derived equations of motion. Since the number of derived equations of motion equals the total degrees of freedom in the system, the constrained two-dimensional system will be described by only one equation of motion corresponding to the single degree of freedom represented by the freely varying joint angle. This introduces a problem, because one is interested in formulating two equations to solve for the muscle torques required at both joints, given torques induced by the normal reaction force at the linkage tip. Fortunately, the Lagrangian method may be modified to include situations in which one is interested in the constraint forces themselves (Groesberg, 1968; McCuskey, 1959). Essentially, a fictitious degree of freedom corresponding 480/20/z-4

138

ELLIOT

SALTZMAN

to the direction of the constraint force’s action is added to the system and the constraint force is included as that force which is necessary to prevent the motion from violating the system constraints. Since the fictitious degree of freedom is redundant with respect to the true number of generalized coordinates, an equation of constraint is required for the final description of the system. In the constrained two-dimensional linkage example, this allows one to establish the desired two equations of motion and to compute the generalized forces associated with the normal contact reaction force. These generalized forces (QNj = TNj) may be expressed as (see Appendix G for details):

where FRN and F,, denote the x and y component vectors of the normal contact reaction fo&e (FRN) a”t the linkage tip. In Eq. (15), one can see that these generalized forces are computed on the right side of the equation. Shifting them to the left side, one may then write TM = A$, + B4, + @a2 + D&j2 TM = J% + F+;, f G@ -

TG2 -

TG1 -

TN2

TN1 ,

(314 (31b)

where A through G, TG1 and T,, are as in Eq. (23a, b). TN1 and TN2 represent the torque induced at joints 1 and 2, respectively, by FRN . Since FRN = -FEN , one arrives at an interesting result: If one knows the x andy components of the force one desires to produce normal to an environmental surface at the contact point, one may insert the negative of these values into Eq. (30a, b), and compute the TN1 and TN2 terms. Equation (31a, b) then provides the muscle torque required to produce a desired motion with a desired normal environmental contact force under the influence of gravity. Being able to specify FEN (and hence FRN = -FEN) requires that the actor know or perceive: (a) the spatial parameters of the constraint surface or line (location, orientation, and shape)-these are used in order to compute the direction of the desired FEN at a given point in time; and (b) the “material” qualities of the objects and surfaces that come into contact during the motion-these can be used to specify the magnitude of the desired FEN in terms of its permissible upper and lower bounds (i.e., the strength of a blackboard or piece of chalk will determine the maximum normal force that each can sustain before breaking). An intriguing point about the possibility of controlling the force to be exerted on the environment (FEN) by the predicted consequences of the intended action (FRN) is that this position is consistent with the principle of ideomotor action proposed by William James (1890) and espoused by Greenwald (1970). According to this principle, “an anticipatory image of feedback from an action participates in the selection and initiation of that action” (Greenwald, 1970, pp. 91). With respect to normal contact reaction force planning, this statement describes the procedures discussed above quite well.

SENSORIMOTOR

139

REPRESENTATION

Contact eflects: Tangential reaction force (friction). In the previous section, the simplifying assumption was made that contact between linkage and environment was frictionless and that only normal forces were experienced during motion across environmental surfaces. This assumption is highly unrealistic and will now be lifted. Equation (29) expresses the fact that the component of contact reaction force tangential to an object’s dry surface, FRT , is due to friction (FF). In turn, FF depends on the magnitude of the normal contact reaction force (FRN) and on the nature of the contacting surfaces (via pd); is directed opposite to the linkage tip velocity; and is relatively independent of the contact area and the speed of motion. Returning to the two-dimensional linkage in Fig. 9, the existence of FF means that additional force must be exerted in the direction of the tip’s motion to counteract the opposing frictional force. These frictional forces at the linkage tip can be represented as generalized forces (QFi = TFj) associated with dry sliding friction (see Appendix H for details): TF~ = -~~FRN(v2)-1’2(w(Z12 TF~

=

-~~Zm&~F~‘~(d2~2~

+ Z22+ W2 ~0s $2) + $2(4h ~0s +2 + 4% +

41W2

~0s 42

+

(324 (32b)

Z22>>>

where v is the velocity of the linkage tip. Since the TFj’s are calculated on the right side of Eq. (15), one may shift them to the left side and write T,,.,l = A$‘, + Tnr2

= E+, +

32 F$;,

+ Cd2 + + &I2

-

-

D&j2 T,,

-

TN2

To, -

TN1 -

Tm ,

Tr-1,

(334 Wb)

where A through G, T,, , TG2 , TN1 , TN2 are as in Eq. (31a, b). TF1 and TF2 represent the the torques induced in joints 1 and 2, respectively, by FF at the linkage tip. These results can be interpreted in the context of the earlier example of blackboard writing. If an actor knows the amount of friction required for the chalk to leave a trace on the board (FF), and he knows or can perceive the hardness/softness of the chalk and the surface texture of the board (& he can then calculate the corresponding normal reaction force (FRN) via Eq. (29). The values for pd and F,, can be plugged into Eq. (32a, b) to provide TF1 and TF2 , and FRN can be plugged into Eq. (30a, b) to provide TN1 and TN2 . These terms may then be inserted into Eq. (33a, b) to compute the resultant muscle torques needed to produce a desired motion under the influence of gravity with the required degree of friction between chalk and board. This task involves the direct control of friction, with normal force being controlled in service of friction production during the course of a movement. Other tasks involve the control of normal force for its own sake with secondary compensation for frictional resistance (e.g., smoothing out a wrinkled piece of paper, ironing, etc.); still other tasks involve primary control of both (e.g., wood carving, clay sculpting, etc.). Three-Dimensional Motion; Computational Complexity It was stated above that Lagrangian analyses become highly complex computationally when applied to three-dimensional kinetics. As was noted by Raibert (1976) the Lagran-

140

ELLIOT

SALTZMAN

gian equations for the three-dimensional motion of the three-link, three-joint arm analyzed by Kahn (1969) involve approximately 1600 terms and 13,000 multiplications. One can get a better idea of the increase in complexity when one link, one joint and one spatial motion dimension are added to the planar example in Fig. 8 by referring to earlier discussions of the acceleration sensitivity matrix (see Eqs. (18)-(20)). Specifically, in Eq. (18a) the coefficient for +I is quite simple. However, the corresponding expression for the coefficient of +I in Kahn’s (1969) example fills approximately 12 text pages with small computer print. Sensorimotor statelconjiguration space. Raibert (1976, 1978; Note 12, Note 13) proposed an ingeniousmethod of dealing with the problem of computational complexity. Essentially, he capitalized on the fact that if one divides a motion trajectory into very short temporal intervals (“time slices”), then the joint angleposition and velocity of each joint in the limb may be consideredto be constant during a given time slice. Thus, the vector and matrix elementsrepresenting the coefficients of the motion equations, which normally vary with the state of the system (i.e., the position and velocity at all the joints in the limb), becomeconstantswithin a given time slice. During a particular movement, the problem of finding the values of these elementscan becomenot a problem of lengthy computation, but one of retrieving a set of precomputed constants stored at an address corresponding to a given limb state. Since the movement consistsof a series of time slices,the joint torque calculationsthus involve retrieving and using the setsof constants corresponding to the states assumedby the system over the entire range of time slices. It is clear, however, that although this approach vastly simplifies the computation problem, it potentially introduces an even worse storageand retrieval problem. That is, since each joint in the system can have infinitely many positions and velocities, the number of states of the system is infinite. Therefore, it is impossible to store all the constants associatedwith all the statesin a finite capacity memory. Raibert (1976, 1978; Note 12, Note 13) deals with this problem by proposing a discrete state spacein which eachdimensionof the space(the position and velocity parametersof eachjoint) is divided into M intervals. The state spacethereby consistsof 2N dimensions(position, velocity times the number of joints), and is partitioned into iW* subregionsor hypercubes. Thus, the retrieval problem is reduced to that of retrieving the constants associatedwith the subregionof the state spacewhich includes the state of the arm during a given time slice. Recently, however, Horn and Raibert (Note 11) have proposedan alternative configuration spacemodel which provides a more optimal balance between problems of computation time and storage space.This method is similar to the state spaceapproach, except that motion constants are indexed in memory according to the configuration (joint angles only) of the linkage system. At the cost of a little more computation, the advantage of configuration spacelies in its economy of storage,i.e., it requires only iV dimensionsand MN subregionsor hypercubes of memory. (The interested reader is referred to Raibert’s (1976, 1978; Note 12, Note 13) papers for further details on the computation of the hypercube constants, and the way these constants may be recalibrated according to the results of movementsexperience; cf. alsothe work of Albus (1975a,b) for an alternative method of dealing with movement-related problems of storage and retrieval.)

SENSORIMOTOR

REPRESENTATION

141

The state space (Raibert, 1976, 1978; Note 12, Note 13) and configuration space (Horn & Raibert, Note 11) approaches may also be used to eliminate the matrix inversion step in the inverse kinematic transform. Such elimination of an explicit kinematic inverse was demonstrated experimentally by Raibert (Note 13) for a nonredundant arm. In the present context, the same method might be used to eliminate the Jacobian matrix inversion step in the inverse kinematic transform. Since the elements in the inverted matrix depend on the limb configuration, these elements can be stored in a given region of the state/configuration space and retrieved when appropriate during computation of the joint velocities required for a particular movement. It should be noted, however, that this technique only applies to the use of nonredundant arms and/or redundant arms for which a functional synergy matrix has been constructed. Thus, with a redundant arm, an unreduced matrix relating known spatial variables to unknown joint velocity variables will contain elements which are not constants for a given arm configuration. Since redundant arms are indeterminate with respect to the inverse kinematic relationship, the elements of such a hypothetical unreduced “inverse transform” matrix would have an infinite number of solutions, and would hence be unsuitable for storage in a finitestate/configuration space memory. Since most actions are performed by redundant arms, one is led to the conclusion that a state/configuration space approach to inverse kinematics primarily involves the elimination of the functional synergy matrix inversion step. Recall, then, that each inverted functional synergy (or variant) matrix is associated with a distinct set of functional synergy and tuning variables which are represented in separate rows of the matrix (see the above functional synergy matrix section). For a given limb state/configuration, the elements,Mj/si , of such a matrix are constants.Recall alsothat the relative contribution of a particular functional synergy or tuning motor variable to a unit changein a given spatial variable is a function of the other motor variables associatedwith the inverted matrix. Since different matrices representthe effects of different setsof functional synergy and tuning variables, a particular motor variable’s relative contribution to a given spatial variable in one matrix is most probably different from that samemotor variable’scontribution in a different matrix. For a given limb state/configuration (i.e., for a given subregion of the state/configuration space),each distinct inverted functional synergy matrix can be represented by a distinct set of constants associatedwith the corresponding set of functional synergy and tuning variables. Thus, if it is reasonableto use a state/configuration space approach to describe sensorimotor behavior, it is conceivable that a given subregionof the state/configuration spacewould be associatedwith constantsrepresenting: (a) the terms in the matrices and vectors usedto analyze motion kinetics, and (b) the terms in the inverted functional synergy matrices corresponding to those sets of functional synergy and tuning variables which are frequently used near or within that region of the state/configuration space. Representation of skills: II. Kinematics and kinetics. One can consider the above suggestionsin the context of the specialnature of the representationof skilled acts. Thus, a well-practiced action will be associatedwith a given set of functional synergies.In the construction of functional synergy matrices, certain functional synergy and tuning

142

ELLIOT

SALTZMAN

variables may characteristically be associated with one another in given regions of the state/configuration space. The sets of constants corresponding to the elements in the inverted functional synergy matrices for a given state/configuration space subregion may be stored at this state/configuration space address. Thus, each region of the state/ configuration space may have different sets of constants associated with it, corresponding to those combinations of functional synergy and tuning variables that are characteristically used during inverse kinematic procedures within and near those regionsrl For reasons described above, these inverse kinematic constants are defined only in the context of functional synergy matrices. These matrices, in turn, are constructed in order to solve the degrees of freedom problem that exists for the inverse kinematic transform. At the joint torque level of the action plan, however, there is no analogous degrees of freedom problem. The number of equations (one equation for each desired angular acceleration at a given joint) is equal to the number of unknowns (the joint torques required to produce these accelerations). Hence, there is no need at the joint torque level for a construct similar to that of functional synergy, which would serve to functionally relate the torque produced at one joint to the torque produced at another joint. In the section below on the muscle level of the action plan, it will be suggested that joint-torque ratios may in fact exist. However, they only appear to be defined at the joint-torque level indirectly as the result of the construction of muscle syrzergies at the muscle level. Therefore, those constants at a given state/configuration space address, which are related to joint-torque computations, need only relate torque at a given joint to movement at the same or another joint. The process of converting a movement representation from spatial to torque form may be described in the following manner: (a)

If the linkage is redundant, a functional synergy matrix is constructed; (b) if the action is an unfamiliar one (e.g., new functional synergies must be constructed; old functional synergies are used in unfamiliar regions of the statelconfiguration space; etc.), the matrix must be inverted computationally; if the action is a wellpracticed one, the constants for the inverted matrix may be retrieved directly from the current state/configuration space address. This availability of kinematic constants for skilled actions may be one source of the superior performance of familiar versus unfamiliar actions; (4 once the set of functional synergy and tuning variable “motor” rates have been calculated, the corresponding individual joint velocities are recovered (see the above redundant degrees of freedom section) and a set of individual joint accelerationsis derived; (d) using this set of joint accelerationsand the set of joint-torque level constants stored at the current state/configuration spaceaddress,a set of individual joint torques is r1 Conversely,

it is also possible

that

each

functional

synergy/tuning

variable

combination

is

associated with its own smallerstate/configuration space.This smallerspacewould be definedby those subregions of the more global state/configuration space in which it is characteristically used. Such synergy spaces might be included in a “multiple spaces model” (Raibert, Note 12, Note 13) of sensl rimotor activity to reduce further the size of the state/configuration space memory involved in a given performance of a particular action type.

SENSORIMOTOR

REPRESENTATION

143

derived. It should be noted, however, that the set of joints for which such constants are retrieved is not limited to those joints at which motion is desired. Rather, constants are retrieved for both this “motion” set of joints and the “frozen” joints left out of the functional synergy matrix. If the latter joints are to remain truly immobile, the reactive torques induced in them by the moving joints must be counteracted. According to the properties of the state space proposed by Raibert (1976, 1978; Note 12, Note 13) the constants associated with “well-used” regions of the state space (and by implication the configuration space of Horn and Raibert (Note 11)) reflect the true mechanical properties of the linkage more accurately than the constants associated with less-used regions. Therefore, a second advantage of a skilled versus an unskilled action is that the former will be better adapted to the kinetic requirements of a movement at the joint torque level. Muscle Level There are two sublevels within the muscle level of the action plan. The first deals with the relation between desired joint torques and the muscle forces required to produce these torques, while the second deals with the amount of neural input required to produce these muscle forces. Muscle Forces The resultant torque which is actively produced at a given joint is a function of the forces exerted by many muscles. Therefore, the problem of calculating the muscle contractions required to produce a desired joint torque is an indeterminate one; i.e., the joint torque to muscle force transformation involves solving a system of equations (related to the number of joints at which torque is to be controlled) for a set of unknowns (equal to the number of muscles acting at the joints in question) in which there are more unknowns than equations. This is analogous to the degrees of freedom problem involved in the inverse kinematic transform, and the solution is similar. Thus, one can express the total muscularly produced torque ( TM1) at joint j as a function of the forces exerted by a given subset of the muscles of the system (F,,&. It should be noted, however, that the output of the joint torque level of the action plan is a set of resultant muscularly produced joint torques. Each such resultant torque can be produced by an infinite number of flexion and extension torque-production combinations. It is therefore proposed that, prior to the torque-force transformation, each resultant joint torque is resolved into a given flexion (T&) and extension (5“~~) torque combination. Anatomy matrix; muscle synergy matrix. The relation between the force exerted by a muscle and the torque produced at a given joint depends on the angle of the muscle’s insertion with respect to the joint. This angle determines the size of the moment arm in the Torque = (moment arm) x (force) equation, and depends in turn on the joint angle configuration of the linkage. One may thus write the following series of equations, representing the direct transform from muscle force to joint torque:

144

ELLIOT

Gn

=

rhn-lh

Tiin

=

sd’~1+

F Mlf

SALTZMAN

Ykw-1)2 ‘(zn)zFm

F M2 **.

*** + +

F MD

~(2n-1h

~(,,Q’M~

3

,

where the Tz-‘s refer to flexion and extension muscle torques, the FM’s represent muscle forces, and the Y’S designate the moment arms of the muscle forces. More specifically, this system of 2n equations (i.e., one equation each for the flexion and extension torques at joint j; i = 1, 2,..., n) can be represented in matrix form via an anatomy matrix (Jacobsen & Mann, Note 8, Note 14): (34) where T$:: F Mk

= a 2n x 1 column vector representing the set of flexion (+) and extension (-) torques at each jointj(j = 1,2,..., n); =

a p x 1 column vector representing (K = 1, 2,...,p); and

the set of forces exerted by each muscle K

A(+) = [r@], the 2n X p anatomy matrix (g = 1, 2,..., 2n; K = 1,2...., p) whose elements are function of the linkage configuration; where rgk is the flexion (if g = an odd integer) or extension (if g = an even integer) torque per unit force produced by muscle K at joint (g + 1)/2 or g/2, respectively, when all other FM, = 0 for K # h. In order to provide linear independence between the rows of the matrix that correspond to Aexion and extension torques at a given joint, the rgk’s in rows corresponding to flexion (or extension) torques are assigned values of zero when associated with muscles exerting extension (or Aexion) torques at that joint. The A(4) matrix may be reduced to a square matrix with the same methods that were used to construct the functional synergy matrix by: (a) defining a set of elementary matrix transformations on the columns of A(+) to construct an equivalent 2n x p matrix that includes the effects of muscle synergies and tuning muscles. These synergies involve groups of muscles which often span many joints and are constrained to act as units (to paraphrase Turvey, et al., 1978) exerting simultaneous forces across these joints; and (b)

striking out excess columns to produce a 2n x 2n muscle synergy matrix.

If this resulting synergy matrix is invertible, one may compute the inverse and then solve for the flexion and extension, synergy and tuning muscle forces required to produce a desired set of joint torques. Alternatively, since the elements in the anatomy matrix are functions of limb configuration, it should also be possible to use a state/configuration

SENSORIMOTOR

REPRESENTATION

145

space approach to muscle force planning. 12 In a manner analogous to inverse kinematic planning (see the above sensorimotor state/conjiguratio-n space section), a distinct set of muscle synergy and tuning variables would be representedby a distinct musclesynergy matrix. For a given limb configuration, each inverted musclesynergy matrix should then, contain a set of constant elements associatedwith the corresponding combination of muscle synergy and tuning variables. Thus, each region of the state/configuration space would be associatedwith distinct setsof constants,correspondingto those setsof synergy and tuning variables that are usedcharacteristically within or near those regions. Conversely (seefootnote ll), each distinct inverted muscle synergy matrix may be associated with its own smaller state/configuration space, defined by the subregionsof the larger spacein which the correspondingsynergy and tuning musclecombination is characteristically used. One should remember that the musclesynergy and tuning musclevariablesare defined with respect to muscle forces, and are distinct from functional synergy and joint tuning variables which are defined with respect to joint motions. However, interesting relationships may exist between these muscle- and joint-centered concepts. Representation of skills: III. Muscle, torque, and motion. In earlier discussionsof functional synergies, it was proposed that coordinative structures were associatedwith each functional synergy, and were responqiblefor abstractly representing and updating the modeled effectsof a given functional synergy’s activation at severallevels of the action plan. Such abstract activation took the form of a set of joint-motor velocities defined in specific ratios across the joints included in the representation of a given functional synergy or variant. These motor-velocity ratios were transformed by the coordinative structures into movement representationsfor the spatial (and possibly conceptual) levels of the action plan to provide a meansof selectingthe functional synergy appropriate to the task at hand. According to this formulation, the origins of the joint-motor velocity ratios that defined the functional synergies were left unspecified. It is now proposed that these ratios derive from the modeled consequencesfor the joint motion level of activating the muscle force outputs associatedwith a given muscle synergy. These modeled effects are generated by a more complex type of coordinative structure which provides the level-specific, predicted consequencesof activating a given muscle synergy to the joint torque, joint motion, spatial motion, and (possibly) conceptual levels of the action plan. As such, this new type of muscle synergy coordinative structure subsumes the function of the earlier type defined strictly for functional synergies. I2 It seems evident, therefore, that information concerning the state/configuration of the body’s effector systems plays a major role in providing an updated context for sensorimotor action planning procedures. Several investigators have posited such a role for proprioceptive/kinesthetic afference in both limb movements (e.g., Polit & Bizzi, 1979) and speech production (e.g., Lindblom, Lubker, & Gay, 1979). In the framework of the present paper, afferent information is used to provide the state-lconfiguationdependent elements in the matrices and vectors involved in transformations of movement representations between levels of the action plan. This information is essential for both computational and state/configuration space approaches to planning coordinated sensorimotor movement (e.g., as in the use of the functional synergy, acceleration sensitivity, and muscle synergy matrices described in the text).

146

ELLIOT

SALTZMAN

The coordinative structure for a given synergy models the joint torque effects associated with a ratio of forces applied across the set of muscles included in that synergy. The set of muscle synergy coordinative structures associated with a particular skill provides a group of synergy-labeled torque representations, which can then be used in the selection of the synergy(s) involved in constructing the muscle synergy matrix. In addition, these joint-torque representations can be transformed into sets of joint accelerations if the corresponding kinetic equations of motion (see Eq. (33a, b)) are invertible, or if the appropriate constants have been stored in the sensorimotor state/configuration space. From a joint acceleration representation, the muscle synergy model’s effects may be readily transformed into a set of joint velocities, and a corresponding spatial vector and conceptual label may be derived as with the earlier functional synergy coordinative structures. A given muscle synergy is thus represented at the joint motion level by a joint velocity ratio which depends on the effects of the modeled muscle force ratio associated with that synergy. The velocity ratios previously discussed in conjunction with functional synergies remained constant despite variations in the state of the linkage. On the other hand, those velocity ratios associated with muscle synergy force ratios should vary with the linkage state, since the modeled muscle forces will have both different torque and acceleration effects depending on the link state and external loading conditions. However, if a movement is performed under constant conditions, the joint velocity ratio associated with a given muscle synergy should be relatively invariant from one trial to the next. Since an action ultimately depends on the correct selection, sequential ordering, and timing of muscle contractions, it would be advantageous to select the participating muscle groups as early as possible in the planning process. In earlier sections of the paper, when coordinative structures were used to select among functional synergies only, the joint velocity ratio which was finally chosen might not necessarily map directly onto a muscle-defined synergy. Now, however, since coordinative structures are defined with respect to muscle synergies, the selection among velocity ratios (functional synergies) at the joint motion level should more closely correspond to a selection among muscle synergies at the muscle level. Finally, it is likely that muscle synergy coordinative structures, which represent such synergies from the muscle to conceptual levels of the action plan, will increase the probability that different instances of the same action will maintain an underlying resemblance to one another. Referring to the earlier discussion of functional synergies, one may say that actions having common muscle synergies but different tuning muscles are “cousins.” Skill acquisition. Having introduced and discussed the concept of muscle synergies, it becomes appropriate to inquire briefly into their origins. For most forms of sensorimotor behavior, the development of synergies apparently requires training (Gelfand, Gurfinkel, Tsetlin, & Shik, 1971; Gurfinkel, Kots, Paltsev, & Feldman, 1971). Such synergies appear to be made each time not on an empty

place

but on the framework

of a small

number

of basic synergies

SENSORIMOTOR

and inherent neurophysiological parameters of the controlled

mechanisms system. (Gelfand

147

REPRESENTATION

which lower et al., 1971b,

the number p. 332).

of independent

Several authors (e.g., Easton, 1972; Hayes & Marteniuk, 1976)have reviewed evidence suggesting that reflexes may be included in the set of “basic synergies and inherent neurophysiological mechanisms” referred to by Gelfand, et al., (1971b). If one accepts this hypothesis, then the coordinative structures associatedwith a set of reflexes seem to sharemany of the properties attributed by Piaget (1952) to reflex schemes, i.e., abstract, relatively Butonomousinternal representationsof reflex actions. Thus, during movement training, an actor might use a body space-joint motion strategy of combining joint fixation and functional synergy selection to construct a functional synergy matrix. This matrix would then be used to compute the set of joint velocities that kinematically approximate the desired action. The functional synergy velocity ratios involved in this selection process would have been provided by the coordinative structures associated with the muscle synergies already in the actor’s behavioral repertoire. With practice (and through some unspecified pattern recognition process), the actor would then discover or construct a new muscle synergy, internally represented by an associated coordinative structure which is relatively autonomousin function. Muscle Innervation

This section is relatively brief, and servesonly to outline the major problems faced in calculating the neural input to the appropriate motoneurons pools (i.e., the group of motoneurons controlling a given muscle’scontraction) in order to produce the desired pattern of muscleforces for those musclesselectedin a given action. Every muscle consistsof a population of motor units, each of which may be defined as a set of muscle fibers controlled by a single alpha motoneuron. A muscle’s force output can be controlled either by increasing the number of contracting motor units (recruitment) and/or by changing the firing rates of a given set of motor units (rate coding) (cf. Stein, 1974). The maximal force produced by a given motor unit depends on the size of the muscle fibers in the unit. Additionally, the force output of a muscle firing at a given rate is a function of both musclelength and the velocity of shortening or lengthening (e.g., Joyce & Rack, 1969; Joyce, Rack & Westbury, 1969;Rack& Westbury, 1969). Furthermore, the force exerted by a muscledependsnot only on the neural input, but also on the parallel elastic and viscoelastic properties of the musculotendinous tissues. These passive forces may assistor oppose the production of a desired muscle force, and are themselvesdependenton the limb state(e.g., Long, Thomas, & Crochetiere, 1966). Finally, once the required degree of muscle activation (i.e., level of alpha motoneuron innervation) has been calculated, the actual neural input to a given motoneuron pool will depend on the amount of background activity already present in the pool. A promising approach for dealing with the physiological control of muscle force production has been developed recently. This approach treats each muscle as a spring with a physiologically controllable equilibrium length (Asatryan & Feldman, 1965; B&i, Dev, Morasso, & Polit, 1978; B&i, Polit, & Morasso, 1976; Crago, Houk, & Hasan, 1976; Feldman, 1966a, 1966b, 1974a, 197415,1976; Fowler, 1977; Houk, 1979; Kelso,

148

ELLIOT

SALTZMAN

1977; Polit & Bizzi, 1979). For a given muscle length in an isometric situation, specifying a certain equilibrium length results in producing a force that is proportional to the difference between the current length and the newly established equilibrium length. If unhindered, the muscle will contract until it reaches this new equilibrium length. Control of angular position at a particular joint involves controlling the forces of agonist and antagonist muscle groups to produce opposing torques at the joint. A given set of specified equilibrium lengths for the agonist and antagonist muscle “springs” results in the joint attaining a final angular position at which the agonist torque is balanced by the antagonist torque. Two highly desirable features of this model are that, for a given set of agonist and antagonist equilibrium lengths, the joint will achieve and maintain a desired final position regardless of both (1) the initial joint angle and (2) any perturbations imposed during angular motion toward the target position. It is sometimes tempting to hypothesize that motor commands to achieve and maintain a desired hand position in body space, from any initial arm configuration, are based on the set of spring parameters associated in memory with the final limb configuration. Stated this simply, however, the hypothesis is inadequate and/or incorrect for several reasons. First, for a given limb configuration the required pattern of resultant, muscle-produced joint torques (and hence agonist-antagonist spring parameters) will vary according to the body’s orientation to gravity (e.g., Feldman, 1974a; see also the introduction to the above joint torque level section). Second, for a given limb configuration the sameset of resultant, muscle-produced joint torques may be produced with different degrees of agonist-antagonist cocontraction (e.g., Asatryan & Feldman, 1965; Crago et al, 1976; Polit & Bizzi, 1979; seealso the above muscleforces section). Third, a given set of joint anglechangesmay occur with different angular velocity trajectories(e.g., Feldman, 1966b, 1974a, 1974b; Polit & Bizzi, 1979). Fourth, for redundant arms there are an infinite number of possiblelimb configurations (and hence spring parameters) associatedwith a given spatial hand position (see the above inverse kinematic transform section). However, a distinct final limb configuration will be associatedwith the corresponding set(s) of functional synergy and tuning variables used to attain the spatial hand target (see the above functiona synergy matrix section). Finally, and in a similar fashion, there are an infinite number of possiblemuscleforce patterns associatedwith a given set of resultant, muscle-produced joint torques. However, a distinct pattern of muscle forces will be associatedwith the corresponding set(s)of musclesynergiesand tuning variables usedto attain the desired torque pattern (see the above anatomy matrix; muscle synergy matrix section). An adequatemodel for the generalcontrol of limb position and movement using the muscle-spring approach should therefore qualify the process of selecting spring parametersfrom memory according to these relevant considerations. Thus, the problems faced at the muscleinnervation level of the action plan are complex ones.As such, they are no different from those faced at other levels. There is, however, a major distinction to be drawn between problems faced here as compared to those at other levels: The results of computations at the innervation level of the action plan provide physiologically relevant information that may be directly related to the control of the peripheral musculoskeletalapparatus.It is sobering to realize that, until the last few paragraphs,all discussionshave beenfocussedon aspectsof action planning presuma-

SENSORIMOTOR REPRESENTATION

149

bly involved in transforming a symbolically representedmovement into muscle-relevant form. It should be noted that even in such “physiological” form, however, this information offers only a sketchy outline of a solution to the musculoskeletalcontrol problem, e.g., control of motor unit recruitment and firing rate, coordination of alpha and gamma motoneuron activation, damping of undesired oscillatory behavior, etc. The investigation of such topics falls chiefly under the domains of motor physiology and control system theory, and will not be pursued further in this paper.

SUMMARY AND CONCLUSIONS

The approach taken in this paper with respect to analyzing sensorimotorbehavior has been to examine a series of relatively simple actions (e.g., block stacking, reaching, transporting, writing on a blackboard, etc.) in terms of the ways such actions are represented at different levels of the action plan. In addition, attention was focussedon the nature of the transformations that were defined with respectto the different levels of representation of a given action. This perspective appearsto afford certain insights into the nature of movement that might have been overlooked had other approachesbeen adopted. In the present section, several major features of sensorimotor activity that emerged from these analysesare reviewed. Emphasis is placed on the roles of object geometry and gravity, on the degreesof freedom problem, and on the use of certain mathematical techniques for describing and simulating human movement. (1) Object Geometry. The geometry of environmental objects and surfaces was used: at the conceptual level to identify objects and to detect their action-relevant properties (affordances); at the environmental spaceand effector levels, to define a set of object coordinate axes labeled with respect to the type of action being performed; at the body-space and joint motion levels, to enter into the calculation of the motion trajectory of the effector with respect to a given object or surface; and at the joint-torque level, to incorporate adaptively normal and friction reaction forces into a movement involving contact between the effector and a given object or surface. (2) Gravity. At the environmental spacelevel, gravity information was used to constrain the placement and movements of certain objects in the task space,e.g., as in transporting peas on a fork from dinner plate to mouth. In the body-spatial and joint motion levels, the pedate system maintained the actor’s balancewith respect to gravity. Finally, at the joint-torque level, gravity information was usedto plan movementsof the actor in a gravitational field. (3) Degrees of Freedom. The degrees of freedom problem was defined with respect to situations in which one was faced with solving a system of m equationsfor n unknowns when n > m. Such situations appeared to occur both during the inverse kinematic (spatialmotion to joint motion) and the joint torque to muscleforce transformation procedures. In each case,one solution to the problem wasbasedon the construction

150

ELLIOT

SALTZMAN

of functional synergy (joint-velocity ratios) and muscle synergy(muscle force ratios) matrices at the joint motion and muscle force levels, respectively. Additionally, it was proposed that these functional and muscle synergieswere related via the operations of coordinative structures. These coordinative structures were described as complex data structures used to provide, for all levels of the action plan, the predictively modeled effects of activating the muscle synergies. Implications of functional synergies, muscle synergies, and coordinative structures for the representation of skilled action were discussedin the text. Finally, it was noted that the joint motion to joint torque transformation did not involve a degreesof freedom problem. (4) Denavit-Hartennberg Notation. The Denavit-Hartenberg (1955) notation has proved useful for the construction and control of robot manipulators. It is alsoapplicable to computer simulation studies of human movement. For example, in all discussions in the text, the coordinate axes for the linkage segmentswere defined in a right-handed sense.If one wished to study bimanual movements, one could define the right limb’s coordinate axes in a right-handed senseand the left limb’s axes in a left-handed sense. It should be noted that such an arrangement would result in homologous(mirror image) responsesof the right and left limbs if the sameset of joint velocity commands were applied to each limb given the sameinitial limb states; if both limb’s axes were define identically (both right- or both left-handed), however, the same set of joint velocity commandsapplied to eachlimb would produce nonhomologousmovements. Additionally, the use of these representationaltechniques would facilitate simulation studies of the movements of redundant arms for which different functional synergy matricesare defined acrossvarying task and environmental situations. In a complementary fashion, one could examine experimentally the joint-velocity ratios found under different initial postural, loading, and task conditions (cf. Bishop & Harrison, 1977; Kots & Syrovegin, 1966) to seeif one could teaseapart the contributions of functional synergy and tuning variables. Such an approach should prove quite valuable in studies of skill acquisition, where new synergies are presumably constructed or discovered. Finally, these methods offer a novel and rigorous approach to examining the developmental issueof what an infant is learning when it begins to reach for and grasp a nearby object.

Concluding

Remarks

A levels-of-representation approach to sensorimotor phenomena offers a clearly defined conceptual framework within which both to analyze movement and to guide the formulation of experimental approachesto problems of movement regulation and coordination. The framework, as outlined in the text, is accessibleto researchersin psychology, physiology, biomechanics,and artificial intelligence. Contributions from each discipline can be infused at appropriate levels. A true interdisciplinary synthesis in the field of sensorimotor behavior and development will depend upon rigorous theoretical and empirical work in a variety of fields. It is hoped that the present perspective will contribute to this synthesis.

151

SENSORIMOTOR REPRESENTATION

APPENDIX

A:

DIRECT KINEMATIC

TRANSFORM

In Eq. (l), j-l/& is a 4 x 4 matrix associatedwith jointi which transforms systemi coordinates to systemj, coordinates (see Fig. 2). This transformation is accomplished through the following operations on systemj : 1st = rotation by -01~about xj , aligning zj and zj-r ; 2nd = translation by aj along xi , bringing Zj and .zjel into coincidence; 3rd = translation along zjdl by Sj , bringing xyj into the xyj-r plane; and 4th = rotation by -&

about zj-r , bringing the two frames into full coincidence.

These transformations can be written in matrix form as:

2nd rotation

COS #j

=

sin & 0 0

2nd translation

-cos 01~ sin & cos aj cos+j sin aj 0

APPENDIX

1st translation

1st rotation

sin aJsin dj aj cosqSiT -sin aj COS q5j aj sin $j cos OIj sj 0 1

B:

ORIENTATION

ANGLES

The set of orientation angles, pH (Waters, Note 6) of the hand frame (xh , yh , zh) relative to the baseframe (x0 , y,, , z,,) are (P& = t he angle betweeny,, and yh measuredabout x,, , (/3H)Y0= the angle between z,, and .zhmeasuredabout y,-,, (/lH),O = the angle between x0 and xh measuredabout z,, . The anglesmay be examined by referring to Fig. 10. Thus, for: (1) (pH)zO: Treat OP as jh , and project jh into the y,z, plane. Then (/3H)z0= the angle about x0 in the y,,za plane from y,, to this projection. Thererefore,

tan(B& = GJzol(hJyo, and ObLo = tan-l(AJz,/(iJ~o .

152

ELLIOT

FIG.

10.

Three-dimensional

SALTZMAN

coordinate

system

used to define

orientation

angles.

(2) (pH)y, : Treat OP as k, , and project kh into the z,,z,, plane. Then (/3H),0 = the angle about y0 in the z,,xO plane from z0 to this projection. Therefore,

(3) (/3H)z0 : Treat OP as i, , and project ih into the x,,y,, plane. Then (pH),O = the angle about z, in the x,,y,, plane from x0 to this projection. Therefore,

and

APPENDIX

C:

THE

VELOCITY

MATRIXES

Equation 6 in the text may be differentiated with respect to time: ;

(OR) = OlEi= $ (Bj jR) z $ & . jR + B, . $R,

where (d/&) JR = 0, since iR is a constant expressing a fixed point relative to body j. Thus, O~=-$B,.jR. I3 Adapted

from

Kahn,

1969;

Waters,

Note

6.

SENSORIMOTOR

153

REPRESENTATION

Since I$ is a function of the joint variables, q1 , (z’ = 1, 2,..., n), one may define a “velocity matrix,” V, , by

Vj-,$$ = f Edpi i-1aqi dt

where gi is Ji , the angular velocity at joint i. In the above equation, aBj/aqi = 0 for i > j, since Bj only includes matrices involving joint variables from ql to qj . Thus

One should note that it is relatively easy to calculate aBj/aqi , since

aBj -

OA

aqi

where a j-lA,/aq,

1.

= 0 for j #

. ..

'4

@-l/J . __

i-2&

8%

.

iAi+l

. . . j-lAj,

i.

APPENDIX

D:

THE

JACOBIAN

MATRIX~*

The direct kinematic transform defines a mapping from joint variables (4) to spatial variables (S) via a set of functions, fi(4j), where i = 1, 2 ,..., m, and j = 1, 2 ,..., n. Assuming that the partial derivatives of each function, fi , exist, one can form the matrix of partial derivatives, [af&#j]. Th is matrix is expressed in Eq. 8 in the text, and is called the Jacobian matrix of the direct transform, J(4). One can differentiate S with respect to time in the following manner:

=

_- afi [

a+j I

.-

d 93 dt

Finally, one knows by a theorem (Wylie, 1975, p. 713) that since the direct kinematic transform has the Jacobian, I(+), then the Jacobian of the inverse kinematic transform is J-i. Thus, if J is invertible, one can find J-1 and, hence, solve the inverse kinematic problem. I* Lang, 480/20/2-5

1973;

Wylie,

1975.

154

ELLIOT

APPENDIX (1)

E:

SALTZMAN

LINKAGE-MECHANICAL

Angular velocity about center of

EFFECTS: mass

KINETIC

ENERGY

TERMS~~

(corn).

link, : $I , link, : q$ + & . (2) Linear velocity (magnitude)of centerof mass.

1% : I !&y$ I, link2: I hy$ + B12(&+ d2>l. (3) Linear veZocitysquared:) v 12. Note that vector multiplication is defined by the scalarproduct operation. link, : v12= 1&2&2 1 = @$,? link, : v22

=

I 4312

=

h2412

=

d12K2

+

@)(+d lJ2

+

4

* ~2&&

w29$(~1

Q122 +

+42> +

942)

1112 c-$2)

a~22(d,

+

+

952),

I

42)2

c-+2

+

$122)

Total kinetic energy, link 1.

(5)

linear

K

Total kinetic energy, link 2.

=

4(i%2~22)(4~

+

5J2)'

+

*m2(v2">

angularX =

B(i%2122)d12

=

h2qi2(k2

(6) LK

+

&~2"(41

+~142(4~2

8ngularK

K2

+

$&v

+

(4)

+

linear +

+

$12"

i5~m2~2~d142 +

u2

~0s

K +

42) +

!C(ih2z22)422

+

$m241d2(fi~22

tm2(v22) +

412cos d2) +

Bm2d22(5122)a

Total K, link 1 + 2: L = K - P = K, sinceP = 0 in this example. =

4(@(m2(h2 +

I6 See Fig.

d22(m2(+~2W

8.

+

*122

+

412

c-42)

+

m16h2)

+

4d2(m2(8~22

+

412

COS#~))

SENSORIMOTOR

(7)

155

REPRESENTATION

Pa&at deriwatives ojL.

Wa+2 = t~)t-sin~2)tm2Z112~12) + M-sin

d2)(m2k12&d2)

= M-sin 42) d1m~4l2td1+ A> = -*(sin $*) fjlm211Z2(qG + d2>, w+,

=

(:)(qm,(gh* +

+

B(m*(i32”

+

312” 1112 yJ”

aL/af$, = (fr)(m,(#z** + llZZcos

+

1112 cos 42))

42

42)

+

m1%h2>

$1

,

+2M1+

tGw*wN

d2

(9) Finally:

= -& + F+*+ (Bm21112(--sin $,)&$,) - (- &442(sin~2>~1<$& + 42,)

APPENDIX

F:

KINETICS:

GRAVITY

EFFECTS

(1) Potential energy terms (P). parallel

Assuming that the direction of g is known to the y axis, the total potential energy of links 1 and 2 is P = m&l1

I6 See

Fig.

8.

sin&

+ m&l1

sin 4, + 31, sin(&

+ +*I).

to be

156

ELLIOT

For a conservative

force like gravity,

SALTZMAN

one can write

Qj=-$.

(Groesberg,

1968)

3

Thus,

Qdl= Tel = = = Qd2= Tea= =

-awl -(+w& cosA + mad1~0sA + m&4 COS(#~ + &)) -& cos+4(Qml + ma)+ $6Jaco+j$ + da)). -w+, -(+%&2 co+h + 74))= -g(ihl, cos(dl+ +J).

(2) Generalized force terms. formulate the problem according

Assuming the direction to Eq. (25) in the text:

QG~ = f

of g is unknown,

one may

(F, + F,)lc * 2, 3

R=l

where r, = the position vectors of corn, and comz (N = 2) in the base frame, expressedas xy component vectors. Thus: (a)

link 1: r, = iZ1 cos+,i

+ &ll sin&j,

link 2: r2 = (11 cm dl + $1, cos(&

(b) &d,= =

+ &))i

+ (I1 sin+,

+ &lz sin(+,

+ &))j.

TGI m,g

* 2

+

m,g

- 2

= (wd + w,j) . (-ii& sin42 + iill cosAi) + (m& + mag,j) . (C-4 sinA - 812Sin(A+ &))i + (4 ~0s& + &laCOS($~ + &))j) = ((mlgz)(-tll sinA> + hg,)(% ~0s4,)) + (@w&-4 SinA - 41,SWA + &I) + @-%)(4cos41+ & cos(41+ 42))) = ge(-(ml~ll sin9J - (4, sinA> - (m& sin(A + +J)) + g,((m& cosA) + (mallcosA) + (ma&cosC++ A))). (C> Qm,= TG:! = (ml& + mlg&(O)+ (wd + fw,U - (-44 WA + AY + +lacos(A+ Ca>i) = gd--m& sin& + 9%))+ gdm& cos(A+ 5%)).

SENSORIMOTOR

APPENDIX

G:

157

REPRESENTATION

KINETICS:

NORMAL

CONTACT

EFFECTS~’

One may rewrite Eq. (24) in the text as:

QN~ = 5 (FRN, + FRN& - 2

3

k=l

>

where rk = the position vector of the linkage tip (N = 1) in the base frame, expressed as xy component vectors. Thus:

(1) r = r, + r, = (4 ~0s& + Z2co+A + +2>)i+ (4 sin A + I2SW1 + +2>>i(2) Qm,= TN~ = (FRN,) * GXr, + ~,M%> + FRN,) * @(rz + r&i%> Qm,=

(3)

TN~

APPENDIX

=

-FRN,(zlsin+l

=

(FRNJ

=

-FRN,&

H:

+ * Gk S%ib

KINETICS:

+

z2sin(+l

+h>>

r,>&U -k

+FR~y(zlCos+l

+

&>)

+

FRNJ

FRNy(z2

TANGENTIAL

* @(rz cosbbl

CONTACT

f +

f

z2cos(~lf+2))~

r,M%>

+d)-

EFFECTS-FRICTIONAL

The frictional forces at the linkage tip can be represented as generalized forces (QFj = TFj) associated with dry sliding friction (Wells, 1967):

where P, is the “power

function”

associated with friction, PF = i

and may be expressed as

akvL.

k=l

In this expression, N = 1 since there is only one point of contact between the linkage and the environmental surface, a = /L~FR,, and v = the velocity of the linkage tip. (1)

23 = j v 12 =

I 441

+

12(41

+

d2,12

=

~1$12

+

w2

c42td1,c&

+

$2,

+

4@1

+$2)2

= q42(42 + G2 + 2ZJ, cos 42) + 2&$2(z,z2 CO842 + I,“) + &2Z2”. Once can then express v as (~a)l/~. I7 See Fig. I* See Fig.

9. 9.

158

ELLIOT

(2)

PF

(4)

QF+,

=

SALTZMAN

,%FRN(v2)1’2

=

Tp2

=

-

3

=

-pd17Rj.J(v2)-“2(~2z22

2 +

4#J2

cos

$2

+

z22))*

REFERENCES

A. A. Experimental analysis of the semantics of movement. In D. A. Norman & D. E. Rumelhart (Eds.), Explorations in cognition. San Francisco: Freeman, 1975. AGIN, G. J., & BINFORD, T. 0. Computer description of curved objects. In Third International Joint Conference on Artificial Intelligence, Advance Papers of the Conference. Menlo Park, Calif.: Stanford Research Institute Publications Department, 1973. ALBUS, J. S.A new approach to manipulator control: The cerebellar model articulation controller (CMAC). ASME Journal of Dynamic Systems, Measurement, and Control, 1975a, 97, 220-227. ALBUS, J. S. Data storage in the cerebellar model articulation controller (CMAC). ASME JournuZ of Dynamic Systems, Measurement, and Control, 1975b, 97, 228-233. ARBIB, M. A. Artificial intelligence and brain theory: unities and diversities. Annals of Biomedical Engineering, 1975, 3, 228-274. ARUTYUNYAN, G. A., GURFINKEL, V. S., & MIRSKII, M. L. Investigation of aiming at a target. Biophysics, 1968, 13, 642-645. ARUTYUNYAN, G. A., GURFINKEL, V. S., & MIRSKII, M. L. Organization of movements on execution by man of an exact postural task. Biophysics, 1969, 14, 1162-l 167. ASZATRYAN, D. G., & FELDMAN, A. G. Functional tuning of nervous system with control of movement or maintenance of a steady posture. I. Mechanographic analysis of the work of the joint on execution of a postural task. Biophysics, 1965, 10, 925-935. ATTNEAVE, F., & BENSON, B. Spatial coding of tactual stimulation. Journal of Experimental Psychology, 1969, 81, 216-222. ATPNEAVE, F., & OLSON, R. K. Discrimination of stimuli varying in physical and retinal orientation. Journal of Experimental Psychology, 1967, 74, 149-l 57. ATTNEAVE, F., & REID, K. W. Voluntary control of frame of reference and slope equivalence under head rotation. Journal of Experimental Psychology, 1968, 78, 153-159. BARTLETT, F. C. Remembering. Cambridge: Cambridge Univ. Press, 1932. BELEN’KII, V. YE., GURFINKEL, V. S., &PAL’TSEV, YE. I. Elements of control of voluntary movements. Biophysics, 1967, 12, 154-161. BERNSTEIN, N. The coordination and regulation of movements. New York: Pergamon, 1967. BISHOP, A., & HARRISON, A. Kots and Syrovegin (1966)-a demonstration of modular units in motor programming ? Journal of Human Movement Studies, 1977, 3, 99-109. BIZZI, E., Dnv, P., Mowo, P., & POLIT, A. Effect of load disturbances during centrally initiated movements. Journcd of Neurophysiology, 1978, 41, 542-556. BIZZI, E., POLIT, A., & MO-O, P. Mechanisms underlying achievement of final head position. Journal of Neurophysiology, 1976, 39, 435-444. ABRAHAMSON,

SENSORIMOTOR

REPRESENTATION

159

CRAGO, P. E., HOUK, J. C., & HASAN, Z. Regulatory actions of human stretch reflex. Journal of Neurophysiology, 1976, 39, 925-935. DENAVIT, J., & HART-ERG, R. S. A kinematic notation for lower pair mechanisms based on matrices. ASME Journal of Applied Mechanics, 1955, 77, 215-221. DOBROTIN, B. M., & SCHEINMAN, V. D. Design of a computer controlled manipulator for robot research. In Third International Joint Conference on ArtiJicial Intelligence, Advance papers of the conference. Menlo Park, California: Stanford Research Institute Publications Department, 1973. EASTON, T. A. On the normal use of reflexes. American Scientist, 1972, 60, 591-599. EL’NER, A. N. Possibilities of correcting the urgent voluntary movements and the associated postural activity of human muscles. Biophysics, 1973, 18, 966-971. FELDMAN, A. G. Functional tuning of the nervous system with control of movement or maintenance of a steady posture. II. Controllable parameters of the muscles. Biophysics, 1966a, 11, 565-578. FELDMAN, A. G. Functional tuning of the nervous system during control of movement or maintenance of a steady posture. III. Mechanographic analysis of the execution by man of the simplest motor tasks. Biophysics, 196613, 1 I, 766-775. FELDMAN, A. G. Change in the length of the muscle as a consequence of a shift in equilibrium in the muscle-load system. Biophysics, 1974a, 19, 544-548. FELDMAN, A. G. Control of the length of the muscle. Biophysics, 1974b, 19, 766-771. FELDMAN, A. G. Control of the postural length and strength of a muscle: advantages of the central co-activation of the oi and static y motoneurones. Biophysics, 1976, 21, 188-l 90. FITGER, C. Tactile-kinesthetic space estimation: the influence of gravity. PsychoZogicaZ Research, 1976, 39, 113-135. FOMIN, S. V., & SHTIL’KIND, T. I. The concept of equilibrium of systems having legs. Biophysics, 1972, 17, 137-141. FOMIN, S. V., & STILLKIND, T. I. On the dynamic equilibrium of the pedatar systems. In A. A. Gydikov, N. T. Tankov, & D. S. Kosarov (Eds.), Motor control. New York: Plenum, 1973. FOWLER, C. A. Timing control in speech production. Bloomington, Indiana: Indiana University Linguistics Club, 1977. GELFAND, I. M., V. S. Gurfinkel, S. V. Fomin, & M. L. Tsetlin (Eds.), Models of the structuralfunctional organization of certain biological systems. Cambridge, Mass.: Massachusetts Institute of Technology Press, 1971 a. GELFAND, I. M., GURFINKEL, V. S., TSETLIN, M. L., & SHIK, M. L. Some problems in the analysis of movements. In I. M. Gelfand, V. S. Gurfinkel, S. V., Fomin, & M. L. Tsetlin (Eds.), Models of the structural-functional organization of certain biological systems. Cambridge, Mass.: Massachusetts Institute of Technology Press, 1971 b. GENTNER, D. On relational meaning: the acquisition of verb meaning. Child Development, 1978, 49, 988-998. GIBSON, J. J. The theory of affordances. In R. Shaw & J. Bransford (Eds.), Perceiving, acting, and knowing. Hillsdale, N. J.: Erlbaum, 1977. GOLANI, I. Homeostatic motor processes in mammalian interactions: a choreography of display. In P. P. G. Bateson & P. H. Klopfer (Eds.), Perspectives in ethology (Vol. 2). New York: Plenum, 1976. GREENE, P. H. Introduction. In I. M. Gelfand, V. S., Gurfinkel, S. V. Fomin, & M. L. Tsetlin (Eds.), Models of the structural-functional organization of certain biological systems. Cambridge, Mass.: Massachusetts Institute of Technology Press, 1971. GREENE, P. H. Problems of organization of motor systems. In R. Rosen & F. M. Snell (Eds.), Progress in theoretical biology (Vol. 2). New York: Academic Press, 1972. GREENE, P. H. Hierarchical hybrid control of manipulators, artificial intelligence in LSI. In E. Heer (Ed.), Remotely manned systems: exploration and operation in space. Pasadena, Ca.: California Institute of Technology Press, 1973. GREENWALD, A. G. Sensory feedback mechanisms in performance control: with special reference to the ideo-motor mechanism. Psychological Review, 1970, 77, 73-99.

160

ELLIOT

SALTZMAN

GROESBERG, S. W. Adwanced mechanics. New York: Wiley, 1968. GURFINKEL, V. S., KOTS, YA. M., PALTSEV, E. I., & FELDMAN, A. G. The compensation of respiratory disturbances of the erect posture of man as an example of the organization of interarticular interaction. In I. M. Gelfand, V. S. Gurfmkel, S. V. Fomin, & M. L. Taetlin (Eds.), Models of the structural-functional organization of certain biological systems. Cambridge, Mass.: Massachusetts Institute of Technology Press, 1971. HAYES, K. C., & MARTENIUK, R. G. Dimensions of motor task complexity. In G. E. Stelmach (Ed.), Motor control: Issues U trends. New York: Academic Press, 1976. HEBB, D. 0. The organization of behuwior. New York: Wiley, 1949. HESS, W. R. Diencephalon: autonomic and extrapyramidal functions. New York: Grune & Stratton, 1954. HOUK, J. C. Regulation of stiffness by skeletomotor reflexes. Annual Review of Physiology, 1979, 41, 99-l 14. HOWARD, I. P., & TEMPLETON, W. B. Human spatial orientation. New York: Wiley, 1966. JAMES, W. The principles of psychology (Vol. 2). New York: Holt, 1890. JOYCE, C. G., & RACK, P. M. H. Isotonic lengthening and shortening movements of cat soleus muscle. Journal of Physiology, 1969, 204, 475-491. JOYCE, G. C., RACK, P. M. H., & WESTBURY, D. R. The mechanical properties of cat soleua muscle during controlled lengthening and shortening movements. Journal of Physiology, 1969, 204, 461-474. JUNG, R., & HASSLER, R. The extrapyramidal motor system. In J. Field, H. W. Magoun, & V. E. Hall (Eds.), Handbook of physiology, Section I. Neurophysiology, VoZ. 11. Washington, D.C.: American Physiological Society, 1960. KAHN, M. E. The near-minimum-time control of open-loop articulated kinematic chains (AIM-106). Artificial Intelligence Project, Stanford University, 1969. (NTIS No. AD-708 076) KAHN, M. E., & ROTH, B. The near-minimum-time control of open-loop articulated kinematic chains. ASME Journal of Dynamic Systems, Measurement and Control, 1971, 93, 164-172. KAHNEMAN, D. Attention and effort. Englewood Cliffs, N.J.: Prentice-Hall, 1973. KELSO, J. A. S. Motor control mechanisms underlying human movement reproduction. Journal of Experimental Psychology: Human Perception and Performance, 1977, 3, 529-543. KOTS, YA. M., KRINSKIY, V. I., NAMIN, V. L., & SHIK, M. L. The control of movements of the joints and kinesthetic afferentation. In I. M. Gelfand, V. S. Gurfinkel, S. V. Fomin, & M. L. Tsetlin (Eds.), Models of the structural-functional organization of certain biological systems. Cambridge, Mass.: Massachusetts Institute of Technology Press, 1971. KOTS, YA. M., & SYROVEGIN, A. V. Fixed set of variants of interaction of the muscles of two joints used in the execution of simple voluntary movements. Biophysics, 1966, 11, 1212-1219. LANG, S. Calculus of several variables. Reading, Mass.: Addison-Wesley, 1973. LASHLEY, K. S. Basic neural mechanisms in behavior. Psychological Review, 1930, 37, l-24. LEWIS, R. A. Autonomous manipulation on a robot: summary of manipulator software functions (JPL-TM-33-679). Jet Propulsion Laboratory, California Institute of Technology, 1974. (NTIS No. N74-20023) LEWIS, R. A., & BEJCZY, A. K. Planning considerations for a roving robot with arm. In Third International Joint Conference on Artificial Intelligence, Advance papers of the conference. Menlo Park, Ca.: Standard Research Institute Publications Department, 1973. LINDBLOM, B., LUBKER, J., & GAY, T. Formant frequencies of some fixed-mandible vowels and a model of speech motor programming by predictive simulation. Journal of Phonetics, 1979, 7, 147-161. LONG, C., THOMAS, D., & CROCHETIERE, W. J. Viscoelastic factors in hand control. Proceedings of IV International Congress of Physical Medicine, Excerpta Medica international congress series no. 107. New York: Excepta Medica, 1966. MARTIN, J. P. The basal ganglia and posture. London: Pitman, 1967. MCCVSKEY, S. W. An introduction to advanced dynamics. Reading, Mass.: Addison-Wesley, 1959.

SENSORIMOTOR

REPRESENTATION

161

L. Methods of annlytical dynamics. New York: McGraw-Hill, 1970. J. L. Dynamics. New York: Wiley, 1966. G. A., & JOHNSON-LAIRD, P. N. Language and perception. Cambridge, Mass.: The Belknap Press of Harvard Univ. Press, 1976. NAPIER, J. R. The prehensile movements of the human hand. Journal of Bone and Joint Surgery, 1956, 38B, 902-913. NEVATIA, R., & BINFORD, T. 0. Structured descriptions of complex objects. In Third International Joint Conference on Artificial Intelligence. Advance papers of the conference. Menlo Park, Ca.: Stanford Research Institute Publications Department, 1973. D. A. Norman 8t D. E. Rumelhart (Eds.), Explorations in cognition. San Francisco: Freeman, 1975. PAL’TSEV, YE. I., & EL’NER, A. M. Preparatory and compensatory period during voluntary movement in patients with involvement of the brain of different localization. Biophysics, 1967, 12, 161-168. PAUL, R. C. Modelling, trajectory calculation and servoing of a computer controlled arm. (AIM-l 77) Artificial Intelligence Laboratory (STAN-CS-72-31 I), Computer Science Department, Stanford University, 1972. (NTIS No. AD-785 071) PEG,, R. W. Human perceptual-motor performance. In B. H. Kantowitz (Ed.), Human information processing: Tutorials in performance and cognition. Hillsdale, N. J. Erlbaum, 1974. PIAGET, J. The origins of intelligence in children. New York: Intern. Univ. Press, 1952. PICK, H. L., JR., YONAS, A., & RIESER, J. Spatial reference systems in perceptual development. In M. Bornstein 81 W. Kessen (Eds.), Psychological development from infancy: image to irrtention. Hillsdale, N. J.: Erlbaum, 1978. POLANTI, M. Personal knowledge. Chicago: Univ. of Chicago Press, 1958. POIAT, A., & BIZZI, E. Characteristics of motor programs underlying arm movements in monkeys. Jozrtmzl of Neurophysiology, 1979, 42, 183-194. POSNER, M. I., & KLEIN, R. On the functions of consciousness. In S. Kornblum (Ed.), Attention and performance I Vol. 4). New York: Academic Press, 1973. RACIC, P. hl. H., & WESTBURY, D. R. The effects of length and stimulus rate on tension in the isometric cat soleus muscle. Journal of Physiology, 1969, 204, 443-460. RAIBERT, M. A state space model for sensorimotor control and learning (AIM-351). Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1976. (NTIS No. AD-A026 960) RAIBERT, hl. H. A model for sensorimotor control and learning. Biologica Cybernetics, 1978, 29, 29-36. RIESER, J. J., &PICK, H. L., JR. Reference systems and the perception of tactual and haptic orientation. Perception and Psychophysics, 1976, 19, 117-l 2 I. ROCK, I. Orientation and form. New York: Academic Press, 1973. SCHANK, R. C. Identification of conceptualizations underlying natural language. In R. C. Schank & K. iFI. Colby (Eds.), Computer models of thozrght and language. San Francisco: Freeman, 1973. SCHMIDT, R. A. A schema theory of discrete motor skill learning. Psychological Review, 1975, 82, 225-260. SHAXV, R. E., S; MCINTYRE, M. Algoristic foundations to cognitive psychology. In W. Weimer & D. Palermo (Eds.), Cognition and the symbolic processes. Hillsdale, N. J.: Erlbaum, 1974. SHETH, P. N., & UICKER, J. J., JR. A generalized symbolic notation for mechanisms. ASME Jo~mal of Engineering for Industry, 1971, 93, 102-l 12. SMITH, K. U., & SMITH, W. M. Perception and motion. Philadelphia: Saunders, 1962. STEIX, R. B. Peripheral control of movement. Physiological Reviews, 1974, 54, 215-243. SY.MON, K. R. Mechanics, third edition. Reading, Mass.: Addison-Wesley, 1971. TAL~IY, L. Semantics and syntax of motion. In J. P. Kimball (Ed.), Syntax and semantics (Vol. 4). New York: Academic Press, 1975. TIR~OSHENKO, S., & YOUNG, D. H. Advanced dynumics. New York: McGraw-Hill, 1948. TOL~IAN, E. C. Purposive behavior in animals and men. New York: The Century Company, 1932. TuRfA, J. J. Statics. New York: Quantum Publishers, 1974. MEIROVITCH,

MERIAM, MILLER,

162

ELLIOT

SALTZMAN

TURVEY, M. T. Preliminaries to a theory of action with reference to vision. In R. Shaw & J. Bransford (Eds.), Perceiving, acting and Knot&g. Hillsdale, N. J.: Erlbaum, 1977. TURVEY, M. T., & FOWLER, C. Skill acquisition: an event approach with special reference to searching for the optimum of a function of many variables. In G. E. Stelmach (Ed.), Informntion processing in motor control and learning. New York: Academic Press, 1978. TURVEY, M. T., SHAW, R. E., & MACE, W. Issues in the theory of action: degrees of freedom, coordinative structures and coalitions. In J. Requin (Ed.), Attention and performance VII. Hillsdale, N. J.: Erlbaum, 1978. WATERLAND, J. C., & MUNSON, N. Reflex association of head and shoulder girdle in nonstressful movements of man. American Journal Physical Medicine, 1964, 43, 98-108. WELLS, D. A. Lugrangian dynamics. New York: McGraw-Hill, 1967. WELLS, K. F. Kinesiology, fifth edition. Philadelphia: Saunders, 1971. WHITNEY, D. E. Resolved motion rate control of manipulators and human prostheses. IEEE Transactions on Man-Machine Systems, 1969, MMS-IO, 47-53. WHITNEY, D. E. The mathematics of coordinated control of prosthetic arms and manipulators. ASME Journal of Dynamic Systems, Measurement and Control, 1972,94, 303-309. WICKSTROM, R. L. Fundamental motor patterns. Philadelphia: Lea & Febiger, 1977. WINOGRAD, T. A procedural model of language understanding. In R. C. Shank & K. M. Colby (Eds.), Computer models of thought and language. San Francisco: Freeman, 1973. WYLIE, C. R. Adwanced engineering mathematics, fourth edition. New York: McGraw-Hill, 1975.

REFERENCE

NOTES

1. GREENE, P. H. Strategies for helerarchical control-an essay. I. A style of controlling complex systems. Department of Computer Science, Illinois Institute of Technology, Chicago, December, 1975. 2. GREENE, P. H. Strategies for heterarchical control-an essay. II. Theoretical exploration of a style of control. Department of Computer Science, Illinois Institute of Technology, Chicago, December, 1975. 3. MARR, D. Representing visual information. AIM-415, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1977. 4. m, D., & NISHIHAFU, H. K. Representation and recognition of the spatial organization of three dimensional shapes. AIM-41 6, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1977. 5. GREENE, P. H. Coordination of effecters and transfer of adaptation. Memo No. 15, Institute for Task Analysis, Chicago, Illinois, 1973. 6. WATERS, R. C. Mechanical arm control. Vision Flash 42, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1973. 7. PICK, H. L., JR. Visual coding of non-visual spatial information. Paper presented at S.S.R.C. Modes of perception workshop, Minneapolis, Minnesota, June, 1974. 8. JACOBSEN, S. C., & MANN, R. W. A graphical representation of the functional musculo-skeletal anatomy of the shoulder and arm. Paper presented at 27th Annual Conference on Engineering in Medicine and Biology, Philadelphia, Pennsylvania, October, 1974. 9. HORN, B. K. P. Kinematics, statics, and dynamics of two-D manipulators. AI-WP-99, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1975. 10. HORN, B. K. P., HIROWWA, K., & VAZIRANI, V. V. Dynamics of a three degree of freedom kinematic chain. AI-WP-155, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1977. 11. HORN, B. K. P., & RAIBERT, M. H. Conjiguration space control. AIM-458, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1978.

SENSORIMOTOR

REPRESENTATION

12. RAIBERT, M. H. Control and learning by the state Artificial Intelligence Laboratory, Massachusetts 13. RAIBERT, M. H. Motor control ond Zearning by Intelligence Laboratory, Massachusetts Institute 14. JACOBSEN, S. C., & MANN, R. W. Control systems Conference on Systems, Man and Cybernetics, 15. HORN, B. K. P. Personal communication, 1978. RECEIVED:

February

4, 1979

163

space model: Experimental jindings. AIM-412, Institute of Technology, 1977. the state space model. AI-TR-439, Artificial of Technology, 1977. for artificial arms. Paper presented at IEEE Boston, Massachusetts, November, 1973.