Control and communications for multiple, cooperating robots

Robotics & Computer-Integrated Manufacturino, Vol. 6, No. 1, pp. 37-53, 1989 Printed in Great Britain

•

0736-5845/89 $3.00 + 0.00 © 1989 Maxwell Pergamon Macmillan plc

Paper CONTROL

AND

COMMUNICATIONS FOR COOPERATING ROBOTS

MULTIPLE,

F. B. MAY, A. R. K A Y E * a n d S. A. M A H M O U D Department of Systems and Computer Engineering, Carleton University, Ottawa, Ontario, Canada, K1S 5B6 There is an increasing need for the flexible integration and control of multiple robots of different types and manufacturers in a single workstation. These robots must coordinate their actions with each other in order to improve throughput, to simplify the process of programming them and to accommodate variations in the working environment. We propose a flexible architecture for the control system of multiple, cooperating robots in an integrated, multiple robot system. Our new architecture has major advantages in that it provides for asynchronous operations in and between control modules at the higher levels, retaining synchronous control only at the lowest level where each robot is servoed at its own pose-clock rate. We have outlined a communications and task management architecture which calls for only a simple run-time operating system at the lower levels of the hierarchy. We have shown that the inter-task communication load at the low levels, which is where the worst case occurs, could be handled, at the speeds necessary for robot control, by at least one current message-passing operating system.

1. I N T R O D U C T I O N As manufacturing processes of increasing scope and complexity are automated using robots, there is an increasing need for the flexible integration and control of multiple robots in a single workstation. These robots must coordinate their actions with each other in order to manipulate workpieces effectively, to sequence operations and to share their working space without collisions. In order to facilitate the process of programming them, robots in a shared workspace should be able to converse and cooperate with each other in what Shin e t aL 24 have termed an integrated multiple robot system (IMRS). This ability relieves the programmer of the necessity to plan and program every move completely in advance. It also provides the flexibility necessary to accommodate variations in the working environment and workpieces as they occur and becomes vitally necessary as the process complexity increases. In addition, as the deployment of robot technology develops a longer history and becomes more widespread, it is inevitable that user organizations will need to protect their existing investment in robots while adding new facilities, which may not be of the same manufacturer or type. This will

parallel what has happened in the fields of data processing and office automation, where the development of standards for the open interconnection of heterogeneous machines is now well advanced and where the incorporation of a wide variety of computers and communication devices is now becoming the norm. Existing robot controllers do not allow for such flexibility. In this paper we propose a hierarchical model for the distributed control of an IMRS and for communications between modules within the model. Our model provides a framework within which such flexible systems can be developed and we present examples to illustrate how they would work and to demonstrate their feasibility. A variety of processes are performed in an IMRS, and the robots work either independently or in cooperation with other robots as they switch from one type of process to another. Different types of robots may be used, for instance elbow and gantry robots, and the type of robot used to perform a certain operation might be changed from time to time. Both centralized and decentralized control are necessary to maximize the parallel activity in an IMRS. 24 Current robot control systems are not suitable for an IMRS. Industrial robot controllers are designed to control only one robot at a time 23'2a and only provide simple communication capabilities. Control software

Acknowledoement--This work was supported by the National

Scienceand EngineeringResearch Council of Canada. 37

38

Robotics & Computer-IntegratedManufacturing • Volume 6, Number 1, 1989

cannot currently be ported easily from one make of robot to another and control systems for multiple robot arms are still in the laboratory. 6'8'21 Efforts are being made by standards organizations to develop standardized hierarchical robot control systems~ 2,~ 3 despite the difficulty of defining interfaces to the modules, caused by the vastly different kinematic configurations and capabilities of various robots. Work at the American National Bureau of Standards (NBS) 3 has proposed that the structure of a control system for a single robot workstation should be hierarchical, flexible, modular, and expandable. A hierarchy allows the control of the processes to be examined at different levels of detail, and flexibility can be incorporated to allow a variety of processes to be performed. Modularity permits the building of the control system in blocks by independent groups, as long as the interfaces to the blocks are well defined and respected. These requirements for a single robot workstation are also valid for an IMRS. However, since many actions proceed in parallel in an IMRS, straightforward control and coordination of the parallel activity of dissimilar robots must be incorporated as well. The proposed control architecture of NBS 1'3'4 does not provide for this. The hierarchical model we shall present for an IMRS is a substantial extension of the NBS architecture which resolves many of the above-mentioned problems. The communication primitives required to transmit commands and information between the modules of our architecture are based on the work of Shin and Epstein 25 and Gentleman 16 with modifications to achieve greater simplicity and efficiency. In particular, all inter-module communications in our model are based on exchanging simple message-passing primitives, thus reducing the complexity proposed in the previous works. We explicitly consider only the positional control of robots in this paper, however, our concepts are equally applicable to robots operating under force control. Interfaces to the modules are described only in general terms; it is beyond the scope of this paper to specify all the interfaces to the modules of the hierarchy in detail. After presenting our proposals, we analyze the computational considerations of implementing the lower levels of our hierarchy using a hypothetical, but typical, IMRS as an example. We state the requirements of the operating system for these levels and discuss the performance of some current messagepassing operating systems which might be considered for this application. 5'~°'~*-17 Finally, we show how a variety of processes may be performed within the constraints of our hierarchy.

2. RELATED RESEARCH

The NBS hierarchical control model for a sensorbased robot working in an automated workstation 1'3'26 is a structure of modules, which allow the actions of a robot to be seen in various levels of detail. A complex task command enters the hierarchy at the highest level and results in the generation of a sequence of simple commands to the next lower level. A module at a level may control multiple subordinates at the next lower level, but a module may only accept commands from one module above. External systems may be interfaced at different levels. Figure 1 shows the lowest five levels of the model with a vision system interfaced at the Elemental Move (E-Move) level and a coordinated joint and gripper system at the lowest level. At each level, there are modules for command decomposition, a world model and feedback processing. The work at each level is done in the format best suited to the level and each level uses information from the command module above and feedback from the level below to decide what commands to send to its subordinate(s). The allowable response time of a level increases at progressively higher levels of the hierarchy, hence decisions that must be made quickly must be made at low levels. The Real-time Control System (RCS) 4'18 developed by the NBS is an implementation of their control model and is designed to control an automatic workstation where a robot tends one or more machine tools. It is a synchronous system, in which every control level is executed each pose-clock cycle time. Control levels are implemented as state tables. Each cycle time the world input state is examined, and this must match with one of the defined states in the table. The actions that correspond to this input state are then executed. The NBS system for a single robot workstation is a good starting point for an IMRS architecture, but more features and flexibility are required. Horizontal communication between control modules is necessary, as we will show, and a synchronous system of state tables is unattractive since the parallel actions of multiple robots involve a large number of states which makes programming difficult to implement and validate. Specification of standard interfaces between modules at different levels is not trivial, as many variables must be taken into account. For example, a standardization of the Primitive/Coordinated Joint interface, where the former sends new poses to the latter in Cartesian coordinates, must resolve three problems. 13 First, a robot may achieve the pose in more than one configuration, for instance, an elbow robot can work with its elbow up or down and only

Control and communications for multiple, cooperating robots • F. B. MAY et al.

Work statk~n

39

1 Transfer (Obj from A to B end arC) Acquire (Obj at A) ReLease(ObJ) Move (Obj fromAtoB)

Status = busy,done, failed reason for failure

Task (Obj) Pickup (Obj) Move-to (Location) Move-to obj (Location) Pause locate

Status = busy, done, failed reason for faiture

Vision system

-~

E-move

#

Goto (Point) Gothru (Point) Approach position finger (Obj) Depart position finger (Obj) rmmediate grasp (Obj) Pause

Line flesh (Obj Surface NormaL) -returns distance to object FLood flash (Obj,Dist~lce) -returns distance to centroid of object

Primitive

I

~atus= busy, done,failed reason for faiLu~/~f/Goto (Pose)

Co- ordinated j o int

Position (Opening) ~

Grasp

Gri pper system

I Robot servoes Fig. 1. The NBS hierarchical control model for a single robot. A vision system is interfaced at the E-Move level and a gripper at the Primitive level.

one configuration may be acceptable. Second, different robots have different singular points. Third, different robots have different limits on the travel of their joints and the speed at which they may travel. However, work is proceeding to define interfaces between modules at different levels. 13 Shin e t al., 24 the originators of the IMRS concept, have developed a useful set of four basic classes of processes that occur in an IMRS: independent, loosely coupled, tightly coupled and serialized motion. In independent processes, each process has independent goals, the actions taken by each robot are independent, and each robot may be controlled independently. In loosely coupled processes, each robot performs independent work, but the actions taken by each robot depend on the actions of the other robots. Each robot may be controlled independently, but

there must be co-operation between their control tasks to co-ordinate their actions. In tightly coupled processes, the work-related actions of the robots depend on each other, and the robots cannot be controlled independently. An example is that of two robots carrying a large workpiece. In the last basic class, serialized motion processes, the work done by each robot is dependent upon the completion of work done by another, but the actions taken by each robot are independent. Independent control of each robot is possible, but there must be communication between the control tasks to indicate completion of work. Shin e t al. hold that all robot processes can be expressed as combinations of these four basic types. Also, processes may be work coupled, that is, one robot's control tasks can monitor another, and attempt to compensate in the event of failure.

40

Robotics & Computer-IntegratedManufacturing • Volume 6, Number 1, 1989

Shin and Epstein 25 have also studied the communication necessary between the robots' control tasks. Ports 27 are recommended as communication channels between tasks rather than direct task naming, so that fault tolerance is improved. An extensive set of communication primitives is recommended: for message passing, blocking sends and receives, with non-blocking replies '6 and port filters may be used to provide non-blocking sends as well. The Query, Response and Order statements are suggested to permit a task to interrupt another to exchange status information. Query allows a task to request information from another task via a port, which results in the invocation of the Response handler in the queried task. A problem with the Query-Response method of exchanging status data between tasks is that the queried task may be in the middle of an urgent operation that should not be interrupted. To circumvent this, sections of control tasks would have different priorities that would be set by the Order directive. A task would only honour a Query if its current priority was low enough. To synchronize more than two tasks and allow them to exchange data, the Waitfor system call is recommended. This bypasses the port communication subsystem, synchronizes all the tasks named in the parameter list, and allows them to exchange data. The classification of process types of Shin et al., cited above, is most useful and is adopted in this paper. However, they do not structure the tasks controlling each of the robots into a hierarchical form. We will show that, if this is done, the number of intertask communication primitives can be reduced from the set suggested by Shin and Epstein. Some interesting work has been done in the realm of two robots handling one workpiece. 2'11 In this application, both robots are centrally controlled by one task, and the robots work with one arm as the master and the other as the slave. With regard to programming systems for workstations with multiple robots, Cassinis a has proposed an interesting method whereby the available resources (tools, etc.) and jobs are assigned automatically at execution time. This is in contrast to conventional programming systems where the robots' actions must be specified explicitly in the application program. This removes from the programmer the responsibility of having to specify complete execution sequences, but a complex run time scheduler is required to make the system efficient. The AL language from Stanford 21 allows for the control of up to four arms simultaneously, under position of force servoing. AL is a sequential programming language, with all arms controlled by one control task. The only provision for simultaneous motion

of the arms is the C O B E G I N - C O E N D construct, and no collision checking is done by AL at compile time or run time. Advances in process control, such as those discussed in the preceding three paragraphs, constitute further reasons why a flexible, easily adaptable, modular control architecture is required for multiple robot stations. Although much has been achieved in the control of multiple robots, little progress has been reported with regard to such a flexible, multi-purpose control architeture. 3. T H E P R O P O S E D H I E R A R C H I C A L M O D E L Our proposed hierarchical model for a multiplerobot workstation is represented conceptually by a number of levels. A level is an abstract concept which can be defined as a collection of modules, one for each robot in the multiple-robot system, except for the highest, Workstation level, at which there is a single module for each workstation. In the lower levels, each module carries out similar functions, determined by the level, for its particular robot. This paper deals with the structure at the Workstation and lower levels. McLean 2° has proposed multiple workstations grouped together to form virtual cells. Our hierarchy could be extended to cope with such a case by superimposing additional levels on top of it, but that topic is beyond the scope of this paper. A block diagram of our model (Fig. 2) shows the six levels: Workstation, Process, Step, Elemental Move (E-Move), Primitive, and Coordinated Joint, with N modules per level for N robots. The module at each level receives a command from the module at the level above, and generates a series of commands to the module(s) below and to any external systems interfaced to it. Each module also monitors the modules

I WorkstatiIon

F Fig. 2. The proposed hierarchicalcontrol model for an integrated multiple robot system(IMRS).

Control and communicationsfor multiple, cooperating robots • F. B. MAYet al. below it, and external systems connected to it, and reacts according to the feedback received. Each module is defined to consist of a set of computing tasks which will be described in more detail in Section 4.3. The tasks communicate with each other by means of messages. A task belonging to a module at level i in the hierarchy can communicate only with the following subset of tasks: • tasks of modules at level i; • tasks of modules at level (i + 1), i.e. the next higher level; • tasks at level (i - 1), i.e. the next lower level. 4. F U N C T I O N S , MESSAGES, M O D U L E S AND TASKS 4.1. Distribution of functions by level The top level module for each robot is at the Process level, and it receives commands from the Workstation module to perform processs. It receives, as parameters to a command, the names of tools and objects required to do the job. An example of a command to the Process level module of a robot in a machining station is "Deburr (Part)", which results in the robot carrying out a deburring operation on a part after moving it to a table and getting the grinder to do the work. We have introduced the Process level into our model in order to provide for the efficient control of multiple robots and to allow for parallelism in the computer architecture of the Workstation. Many processes occur simultaneously in an IMRS and the Workstation module should only be expected to supervise the overall operation and not to direct the individual steps of each process. The potential amount of parallel activity would drop if the Process level did not exist and the Workstation sent commands directly to the Step level because the Workstation module could easily be delayed in passing new commands to one robot while other robots required some attention from the Workstation. The Step level fulfills the same function as the Task level in the NBS hierarchy. 4 We have chosen to call it the Step level to avoid confusion with the computing "tasks" of which our modules are made up. The Process level divides the operation into steps, and passes commands to the Step level to have them carried out. The E-Move, Primitive, and Coordinated Joint levels fulfill the same functions as in the NBS hierarchy. 4 The E-Move level receives commands to "Move-To" named locations, and generates the motion segments necessary. It sends " G o t o (Point)" commands to the Primitive level, where "Point" is the transform describing the goal location of the next

41

motion segment. When a robot is to perform a series of motion segments, the Primitive level is commanded to "Gothru" each point, that is to move smoothly past each point, and "Goto" the last. The Primitive level calculates the trajectory to the goal point specified by E-Move, outputs the poses to the Coordinated Joint level, and sends commands to the gripper to open and close. (The gripper is not shown in Fig. 2 but would be interfaced to a module at the Primitive level, in parallel with the Coordinated Joint, as in the single robot case of Fig. 1.) The Coordinated Joint module outputs sets of joint angles to the servo-computers of the joint. Other systems, for instance sensors, can be interfaced to the control system at various levels, as is done with the NBS system. 4.2. Message semantics The semantic content of messages sent between the layers below the Process level is the same in our proposed hierarchy as in the NBS hierarchy, with the exception of the E-Move and Primitive levels. Instead of E-Move sending a new goal position and orientation of the end-effector, it sends the name of a position equation which the Primitive module uses to determine the goal position to which the robot should be sent. The format of the position equation is similar to that of current robot languages, such as VAL-II. 23 An example of such an equation is: "Move Table: Pallet:Grasp;", where "Table" is a transform representing the location of a table in world coordinates. Pallet is a transform of the location of a pallet relative to the table and Grasp is a transform that expresses the grasp position relative to the pallet. Internally, the Primitive module represents the equation as: "Z:T6:Tool = Table:Pallet:Grasp;", where Z is a transform of the location of the base of the robot in world coordinates, T6 is the end-effector face-plate transform relative to the base, and the Tool is the transform of the tool in use relative to the faceplate. This type of motion specification keeps the calculation and knowledge of actual physical locations to the lowest level possible, and allows for functionally defined motions and tracking with no further involvement from the E-Move module above. Performing these types of motion in the RCS of NBS would be more difficult because the E-Move level would have to solve the position equation in each pose-clock cycle and send a new pose to the Primitive level every cycle time. This would nullify the functionality of the Primitive level and the E-Move level would have to deal with actual locations of objects, rather than referring to them by name.

42

Robotics & Computer-Integrated Manufacturing • Volume 6, N u m b e r 1, 1989

Transforms in a position equation are one of the following types: • constant; • parameterized by an E-Mode task; • functionally defined; • sensor defined. The values of constant transforms are retrieved from a database while those parameterized by E-Move are evaluated at the start of the motion corresponding to the position equation. Functionally defined transforms are evaluated in each pose-clock cycle. Sensor defined transforms are created, in a homogeneous format suitable for use by the Primitive module, on the basis of the data supplied.

4.3. Modules and tasks Figure 3 shows the hierarchy of modules for one of the robots in a workstation. Each of the higher level modules, from Process to E-Move, includes a supervisor task and worker tasks, each of which has a port for receiving messages. The modules at these levels execute their functions asynchronously in response to commands from above and feedback from subordinate modules below. Each task can be represented by a two-state finite state machine as shown in Fig. 4. In the wait state the task receives messages from other tasks in the form of parameterized commands or status reports. Once a message is received, a transition to the processing state

To

From

Supervisor o-~

Lr

//

robot worker

/

supervisor

/

robot worker

/

rclx~t worker

t.(

Robot servces

o-~

-

L ( supervisor

/

o~-

o_ /

o-~

~

o-~

Gripper

Fig. 3. Modules and tasks in the logical architecture.

Sensor

Control and communications for multiple, cooperating robots • F. B. MAY et

Receive From other tasks

exceed the time constraint of the Primitive module. In order to avoid this situation, prioritized worker tasks are used in a structure such as that shown in Fig. 6. The E-Move supervisor task passes robot-related commands to the Robot Worker task, and passes vision commands to the Vision Worker task. The Robot Worker has higher priority than the Vision Worker, so that the time constraint of the Primitive module can be met. The relative priority of the tasks increases at progressively lower levels of the hierarchy. All the tasks that run at the Primitive level and below are of equal priority. The Primitive level module executes synchronously, driven by pose-clock interrupts, but each Primitive module runs at a the pose-clock rate of its specific robot which may be different from that of other robots. At each clock cycle, a Primitive module checks for a new c o m m a n d from the E-Move module above, checks the status reports received from its subordinates from the previous cycle, sends them new commands if they have finished, then returns its status to E-Move.

~

command ( p o r o m e t ~ ) ( _ . ~ _'.'*_'_". _'_.. ~

Wait state

1 /

Comm::g )

Sendcommand (parameter) !

to other tasks

Fig. 4. Two-state representation of a control task.

occurs. In this state, the message is interpreted and a series of one or more messages, again containing commands or status reports, is sent to other tasks. The sending task then returns to its wait state. The messages may be sent to or received from external entities, such as a vision system, as well as to other tasks within our defined modules. The set of tasks in each module, and at each of these higher levels, execute their transitions asynchronously in response to the messages received from other entities. The use of supervisor and worker tasks at the higher levels allows for new subsystems to be interfaced and for the prioritization of the actions performed at a level. If just one task was used at each level to interact with both the level below and external systems, it would be difficult for the task to respond to them in an order other than first come, first served. The interaction required between the E-Move and Primitive level modules illustrates this point. When a robot is finishing a motion segment and must make a transition to a new segment without stopping, the Primitive module sends a "Done" to E-Move, and must receive a new position equation before the next cycle time. If, for example, a vision system was interfaced to the E-Move module, and just one task was used to monitor both the vision and the Primitive module, the response time constraint between the E-Move and Primitive modules might not be met. This is shown in Fig. 5, where the E-Move task is still busy preparing another c o m m a n d for the vision system when the Primitive level requires a new position equation. It is possible that the time between points (2) and (4) in the timing diagram of Fig. 5b might

43

al.

4.4. Discussion of key features of our model In summary, key features of the model we have

E-move

/

( Primitive

The E-move Levelmodule implementeda single task

Primitive

I

Time

Vision

m

3! E-move

I

( I )Vision sends a ' ~ e " t o E-move, F-move prepares another vision command (2) Primitive send a"Done"to E-move end requires o new position equoUon before the next. cycle time. (3) E-move sends a new vision command,, and receives the the "Dane"from primltNe (4) E-move sends a new position equationto primitive. Fig. 5. An E-Move module using a single task to control a Primitive module and a vision system. (a) An E-Move module implemented by a single task. (b) Timing diagram showing that the time from (2) to (4) may be too long for the Primitive pose-clock cycle.

44

Robotics & Computer-Integrated Manufacturing • Volume 6, Number 1, 1989

/

,

"~ce~~[".-

step

From

'

Primitive

Fig. 6. An E - M o v e module implemented by multiple prioritized

tasks.

presented are: • it is implemented by modules and tasks running asynchronously in parallel rather than as a synchronous monolithic system as in the NBS RCS system; • the robots are not all required to have the same pose-clock rate; • the multiple tasks of a module are structured as a supervisor task and one or more worker tasks with execution priorities. The asynchronous execution of the higher level modules in response to commands from above and feedback from below is more efficient than having each module at each level execute synchronously at every cycle time. This is because commands given at the higher levels typically take a large number of cycles to complete, so that it is inefficient to have all tasks at all levels execute each cycle. Our architecture has the potential to provide a better response time between levels than a synchronous system. In a synchronous system, feedback

and commands take one cycle to pass between adjacent modules. If, for example, the Primitive module in a synchronous system encounters a problem that requires a decision by the Process level, the earliest possible response from the Process level is returned to the Primitive module in six cycle times, since the feedback and commands would have to pass through three intermediate levels. An asynchronous system, with equivalent computational power, has the potential to provide a better response time in this situation provided the inter-task communications are well designed. The parallel nature makes it suitable for use with multiple processors and distributed systems. It also facilitates the achievement of greater fault tolerance. The freedom to use robots having different poseclocks in a single workstation provides for the integration of robots of different vendors and generations. The use of multiple worker tasks in modules, where required, allows for the use of priorities in executing tasks associated with different subordinate tasks and external systems. This is necessary in order to cope with the problems of meeting real-time deadlines for communicating with the various entities dealt with by each module. 5. H A R D W A R E ARCHITECTURE The nature of the computing required to control an IMRS is well suited to a distributed network of computing nodes. Figure 7 shows a possible network of processor nodes. All the nodes are physically connected to a LAN (not shown) but logical connections between them are restricted to those shown by the solid lines and occur either vertically, in a control hierarchy, or horizontally, between nodes at the same level. Since many actions occur simultaneously in an IMRS, it is best to have the control tasks executing truly in parallel rather than sequentially on a single

Workst.otion

Process I

i

Primitive N Co-ordinoted Joint N Gripper N Sensor N

E

t

Process 2 Step 2 E-move 2

f Primitive 2 Co-ordinoted Joint 2 Gripper 2 Sensor 2

Co-ordirmted Joint I Gripper I Sensor I

Fig. 7. A possible network of processor nodes with permitted logical connections.

Control and communicationsfor multiple, cooperating robots • F. B. MAYet al. processor, so that deadlines for response to external events can be met. The context switching overhead would be very large if one computer was used, and fault tolerance would be less than that of a distributed system. Partitioning the control functions into separate modules allows processors suited to particular functions to be used; for instance, fast numeric processors for trajectory generation at the Primitive level and symbolic reasoning machines at the Process and Step levels. Also, since each robot is allowed to have its own internal pose-clock, each robot's synchronous control tasks should execute on separate processors. A single processor would have difficulty responding to pose-clock interrupts of all the different robots within the required response times. 6. INTER-TASK C O M M U N I C A T I O N S , TASK M A N A G E M E N T AND O P E R A T I N G SYSTEM IMPLICATIONS In this section, we present our proposal for intertask communications and for task management to support the distribution of functions within the model as presented in Section 4. In this way, we establish the requirements for services to be offered by the operating system. In doing this, we shall be concerned to keep the complexity of the operating system requirements to a minimum so as to lay the basis for a relatively simple and efficient operating system. Efficiency is a major requirement for an operating system supporting message passing in an IMRS because of the stringent conditions imposed by the real-time control of the robots. We include guidelines for implementation of an operating system. Later, in Section 8, we discuss performance issues with respect to the operating system. 6.1. Ports, filters and communication primitives We propose the use of ports, as suggested by Shin and Epstein, z5 as logical communication channels. Since ports are part of the services provided by the operating system, discussed below, they may normally be expected to be less error prone than the application tasks of the system. We also adopt several of the port restrictions suggested by Shin and Epstein. All messages are restricted to flow through a port from the user's end to the owner's end. The users of a port are restricted in order to allow the port subsystem to trap illegal messages addressed to a task. The lists of users must be adjusted dynamically to allow certain types of processes to be performed, as will be shown later. Fixed message sizes are used to help minimize the message passing times, and port timers are required to allow tasks to control the time they spend waiting to receive or send a message. Unlike Shin and Epstein,

45

we prioritize the users of a port so that tasks of high priority are serviced first. For example, in Fig. 6, the robot's actions are considered a higher priority than those of the vision system. Our inter-task communications require the use of only three primitives, previously described by Gentleman: 16 blocking sends, and blocking receives used with non-blocking replies. In certain situations, described below, the use of non-blocking sends is also appropriate. However, we propose that this facility be simulated by using a port filter, rather than requiring an additional primitive in the operating system. A port filter, of a type we call a Collector, simulates a non-blocking send by queuing incoming messages for its owner and may also perform some processing of messages from specified users. A filter task is the highest priority task in a module so that the filter users do not have to wait. Since we require filters to be used for other purposes, in association with the ports, they must be provided in any case and we can avoid additional complexity in the operating system by using them for this purpose. Both blocking and non-blocking sends are used for transmitting commands from one task to another. In a situation where new commands to a subordinate task should not be queued, blocking sends are used and a new command should not be sent until a "command finished" status is received from the subordinate. An example is the transmission of commands between higher level robot control tasks because a catastrophe could occur if a Step level task failed after sending a queue of commands to the E-Move level. Using a blocking send for issuing a new command forces an ADA-like rendezvous 22 to occur between the superior and subordinate tasks, but the former never needs to wait because the latter must be ready for a new command. Also, information useful to the superior can be enclosed in the subordinate's reply. Where commands to a subordinate task may be queued, or where it is undesirable for the task sending a command to wait, non-blocking sends are used. An example of the first ease is the vision system, interfaced at the E-Move level in Fig. 6, in which the Vision Worker task may serve several robots. An example of the second case is the sending of commands from the E-Move Robot Worker to the Primitive task. If the commands were sent to the Primitive task using a blocking send, rather than via the Collector filter, the Robot Worker would be forced to wait until the next pose-clock interrupt for the Primitive task to receive the command. If the E-Move Supervisor wished to send to the Robot Worker during this time, it would be held up until the Robot Worker was released by the Primitive task. A non-blocking send, on the other

46

Robotics & Computer-Integrated Manufacturing • Volume 6, Number 1, 1989

hand, allows the Robot Worker to return to wait at its port for an acknowledgement of receipt of the command, or commands from above. To prevent the Robot Worker from sending a second c o m m a n d before an acknowledgement of receipt of the first, checks can be installed in both the Worker and Primitive tasks, as follows. After sending a command, the Worker sets a flag "Able-to-Send" false; the reception of an acknowledgement of the c o m m a n d from the Primitive task sets the flag true again. In the Primitive task, if a second c o m m a n d is received before the first is acknowledged, it returns an error code to the Worker. Tasks at the Primitive level and higher use nonblocking sends to send feedback to their superiors. In this way, a task that is executing a c o m m a n d that takes a relatively long time to perform can send status reports during execution. For example, if a Step task commands a robot to move in a way that requires a series of actions, the E-Move task reports at intermediate times during the motion. If blocking sends were used, a task could only report feedback upon completion of a command, as in Gentleman's "Administrator" concept. 16 Intelligent filter tasks may be used to provide the most timely status reports from subordinates to a superior. If a subordinate sends more than one report before the superior receives from his port, the filter groups all the reports from a subordinate (or group of them) into one message, with the reports listed in lastin-first-out order. Thus, the superior sees the most recent status first, and examines older reports if it chooses. Gentleman's "Administrator" concept 16 is useful at the Primitive level and below. U p o n completion of a command, a subordinate of the Primitive task sends the result to the Primitive task, which holds it at its port until a new c o m m a n d is returned as a reply. These can be blocking sends since a control task at this low level needs to report only upon completion of its c o m m a n d because the time to complete a command is only a few pose-clock cycles. An exception to this rule is the Coordinated Joint task, which reports at each cycle whether the robot has reached the set of joint angles sent to it during the previous clock cycle. Blocking sends also reduce the number of messages that need to be passed at the Primitive level, and the number of ports required. By partitioning the control into a hierarchy of modules, and by incorporating status report as an integral part of the system using the reply primitive, we eliminate the requirement for the other primitives suggested by Shin and Epstein. 25 Also, in our control architecture, high priority functions are performed by high priority tasks so that the Order statement is not

required. Similarly, a task in one robot's set of control modules can send to another task in another robot's control modules to request the latter's status, and this is returned to the former as soon as possible as a reply; hence no Query primitive is necessary. The Response mechanism, which would be used with a Query, allows a task to invoke its dedicated Response handler only when it is queried. The use of the Send, Receive and Reply primitives in our model allows the interrogated task more flexibility in its response.

6.2. Task manaoement A mechanism is needed to allow more than two control tasks to synchronize and exchange data with each other. An example is several robots performing a loosely coupled process in which they are required to stop periodically and inform each other of the work that has been completed. The Waitfor system call suggested by Shin and Epstein 25 is unsuitable because it adds unnecessarily to the complexity of the underlying operating system. To synchronize a group of tasks running on distributed processors where there is no pre-defined master task, a temporary coordinator must be elected to control the synchronization and message exchange. Solutions to this problem already exist, 9'15 and these may be implemented with Send, Receive and Reply primitives. In addition to message passing, the operating system for the network of processors must provide efficient network communication, task scheduling, and device interrupt handling. Fortunately, only basic support is required for these. For example, the task scheduler can be quite simple since the number of tasks running on a processor is fixed, along with their priorities. The ports to which a task may send should be defined before run-time so that run-time checks on qualified users of a port can be eliminated. Qualified users can be verified at compile time by checking the port(s) used by a task against the port user list. Data copying in message passing should be minimized. Tasks should not be allowed to share memory, which means that local message passing will entail data copying, but this cost is not too high for messages of less than 100 bytes, x° To minimize data copying in remote message passing, the kernel at each processor node should be responsible for writing messages over the network, as is done in the V System, 1° rather than employing a network server for this purpose. This eliminates a local copy at both the sending and receiving nodes and a context switch at the sending node. Output to a device should be done via a blocking system call, which outputs directly to the device, if the

Control and communications for multiple, cooperating robots • F. B. MAY et al.

puts a new joint vector to the servo system, outputs other commands to the gripper and the proximity sensor, then sends a message to the Receive port of the Primitive task (via its Collector) stating that another pose-clock cycle has started. For each device, robot or sensor, a device handier reads the data from the device into a buffer in the memory block of the associated task and resets the device, then makes a system call to signal the task, including a pointer to the data as a parameter. The system call passes on the pointer, and calls the task scheduler. Points identified in Fig. 8 correspond to the following events: 1. The pose-clock interrupt routine sends new joint angles to the Collector of the robot. 2. The Primitive task receives a clock signal and generates a new robot pose and commands for the gripper and sensors. 3. The Primitive replies with new commands to its subordinates and sends a request to E-Move for a new position. 4. The Coordinated Joint generates a new set of joint angles and stores it in a buffer to await the start of the next clock cycle. 5. The Gripper generates a new c o m m a n d to be output at the start of the next cycle. 6. The Force/Torque task sends out a new comm a n d immediately. 7. The Proximity task sends out a new c o m m a n d immediately. 8. The Force/Torque task receives its data, begins processing it and is interrupted by the Collector filter.

action can start any time during the pose-clock cycle. If the action can start only at the start of the next poseclock cycle, the call should store the data and request that it be output to the device by the pose-clock handler at that time. An example of the first case is a proximity sensor, and of the second case, the gripper. In either event, a system call for output causes the calling task to block and rescheduling to occur. Parameters passed to the system call are a pointer to the data to be output, and the device identification. The parameter returned to the calling task is a pointer to the data obtained by the device, or the result of the c o m m a n d sent out.

6.3. Interrupt handling There are three classes of device interrupts that need to be handled: • pose-clock interrupts; • robot servo-system interrupts from the joints to signal that they are within range of their goal angles and from the gripper and sensors to signal either completion of a c o m m a n d or data ready; and • network interrupts for incoming messages. Pose-clock interrupts have highest priority to ensure that new joint vectors are sent smoothly to the robot. Servo-system and sensor interrupts have the next lower priority level so that the latest state of the robot environment is always on hand. Figure 8 illustrates a possible sequence of events during a period between successive pose-clock interrupts. During this period the pose-clock handler out-

Task or Routine

2

z~3 e~

-3

Primitive

I--1

14 r-n

Co-ordinated Joint

,

e5 Gripper

iol AI Force/tor~e sensor

i t

e7 Proximity

i

seNSOr

Cotlector Pose crock interrupt rouli ne

i i

; I

II

nIi

I I

t6

,,

N i

14

i

i i

n

H

il

Pose crockinterrupt

Legend ,, Message Exchange • Context switch Fig. 8. Interrupt sequencing-timing diagram.

I~1

i i

I, 121

-1

k

r-3

I I I I

47

/X

i '

-- Time

48

Robotics & Computer-Integrated Manufacturing • Volume 6, Number 1, 1989

9. The Collector filter receives the new position equation from E-Move. 10. The F o r c e / T o r q u e task finishes its processing and sends the result to the Primitive. 11. The Primitive receives the new position equation and the force/torque data. 12. The Gripper receives the status of the c o m m a n d sent out at the start of the cycle and sends it to the Primitive. 13. The Primitive receives from the Gripper. 14. The Proximity task receives a reading, processes it and sends to the Primitive. 15. The Primitive receives from the Proximity. 16. The C o o r d i n a t e d Joint receives a signal from the servo system that the r o b o t is within range of the goal pose and sends a " D o n e " to the Primitive. 17. The Primitive receives from the Proximity. 18. The Primitive receives another clock signal and calculates the trajectory for the new m o t i o n segment. To deal with incoming messages, the network handler makes a system call to invoke a routine to read the message from the network interface directly into the destination task's m e m o r y block, and calls the scheduler. This m e t h o d of device handling entails no local copying of data; they are always copied directly to or from the m e m o r y block of the associated task. Also, data are returned to the task associated with a device as soon as they become available, rather than at the next pose-clock cycle time. This allows a superior task to pass a processed result to the Primitive task for use during the next cycle of calculations. 7. S U M M A R Y O F R E Q U I R E D O P E R A T I N G SYSTEM FEATURES In the previous section, we discussed a n u m b e r of requirements that our architecture places on the operating system and suggested features for meeting them. These features can be summarized as follows: • Support of ports with associated port filters capable of queuing messages received. • Ports have prioritized lists of users, defined before run-time. • Fixed message sizes. • C o m m u n i c a t i o n primitives: blocking sends, and blocking receives used with non-blocking replies. • Messages to be sent on a network are written by the kernel directly o n t o the network. • Prioritized interrupts. 8. P E R F O R M A N C E OF CURRENT OPERATING SYSTEMS In this section, we predict the performance that would be obtained if several previously developed operating systems were used to implement the archi-

tecture we have described. We do this by determining the n u m b e r and size of the messages that would have to be passed during a pose-clock cycle in the lower levels of our architecture (Primitive and lower) in a typical I M R S and by using previously published performance information for the operating systems. As our example we shall use the pose-clock cycle, which was described in detail in Section 6 and Fig. 8, for a robot, equipped with a gripper, a wrist forcetorque sensor, and a proximity sensor. The clock cycle is one in which one m o t i o n segment finishes and another is to start without the robot stopping. Table 1 shows that the Primitive module needs to exchange a m a x i m u m of nine messages during this clock cycle. Seven of the messages are local, between the Primitive module and its subordinates, and two are remote, between the Primitive module and the R o b o t W o r k e r in the E - M o v e module above (see Figs 3 and 7).

Table 1. Messages exchanged at the Primitive level Message number 1

2

3 4 5 6 7 8 9 1. 2. 3. 4. 5. 6. 7.

Activity Pose-clock interrupt routine sends to collector Collector relays the message on Primitive replies to Coordinated Joint Primitive replies to gripper Primitive replies to force/torque sensor Primitive sends to E-Move Coordinated Joint sends to Primitive Gripper sends to Primitive Force/torque sends to Primitive Proximity sends to Primitive Collector receives Primitive receives from

Message content Start of cycle time

Message size Note 4

1

4

1

New pose, servoparameters

100

2

New gripper opening Command to read force/torque wrist Status,request new position equation Status

4

3

4

3

4

4

4

4

4

4

Force/torque array, status

100

5

Range transform

100

6

New position equation name New position equation name

100

7

100

7

Status

Number of pose-clock ticks since start of process. An array (a transform) of nine double precision real numbers. Commands to devices are assumed to be four bytes. Status reports of four bytes are assumed. An array of four vectors and four status bytes. A transform and four status bytes. Name and parameters of new position equation.

Control and communications for multiple, cooperating robots • F. B. MAY et al.

It can be seen that a total of 16 context switches occur, 12 of which are associated with message exchanges. During this cycle time, the Primitive module needs to generate the next pose based on the current goal and the sensor data. The Coordinated Joint module must translate this pose to a joint vector, and the sensor tasks must prepare new commands. During the subsequent clock cycle, the Primitive module will have the additional burden of calculating a transition segment between the current motion segment and the next. However, there will be two less message exchanges during this cycle since E-Move will not have to send a new goal position. F r o m Table 1 it can be seen that the message system should allow messages of two sizes: four bytes for short status reports, and 100 bytes for new commands or complex sensor data. The operating systems we have examined are: Meglos, ~4 the V system, 1° Cedar 5 and Harmony. 17 The first three are for multiprocessors distributed over a local area network, and the last is for multiprocessors connected by a c o m m o n bus. Meglos provides message passing by tasks reading and writing "channels". The read and write operations are blocking, but the only reply that is sent back to a writing task is an acknowledgment of message receipt from the kernel at the receiving end. A channel may be used by a task and its children on one processor, and another task and its children on another processor. The V system provides explicit message passing between tasks, and it is transparent to a task whether the task is local or remote. Blocking sends, receives and replies, as described in this paper, are used. Cedar uses remote procedure calls for communication, which allow a task to send parameters to another, and receive others back. H a r m o n y uses the same style of message passing as the V system.

49

The characteristics of the various operating systems are summarized in Table 2, together with the times required for each of them to send messages of 10 and 100 bytes and to receive replies of the same size. In the case of Meglos, the time that would be required for the receiving task to write back a reply has been included in the given time. 14 It can be seen that Meglos is the fastest of all the systems. It requires about 0.9 and 1.2 ms respectively for 10 and 100 byte message exchanges between remote processors. H a r m o n y is the slowest, requiring about 1.1 ms for a message exchange in which no data are transferred. Unfortunately, there is no published information indicating why H a r m o n y is so slow. Meglos is fast because of efficient message passing, not because of its higher network data transmission rate. Cheriton 1° found that the ratio of processing time to network time for a remote data transfer was about seven in the V system, and estimated that increasing the transmission rate to 100 MBPS from 10 MBPS would only reduce the data transfer time by about 15 operations directly in the kernel instead of through a task server. This reduced the remote message transaction time in the V system by a factor of four. For our example, assuming a message exchange time of 1.2 ms as in Meglos, about 10.8 ms would be required to exchange all the messages to and from the Primitive module. Based on the 0.4 ms context switching time of Meglos, found by Gaglianello, 14 the total communication overhead woud be 12-13 ms for the Primitive level. If a current industrial robot pose-clock cycle time of about 33 ms is assumed, this would leave about 19 ms for trajectory calculation and sensor data processing. Hayward ~9 has shown that less than 7 ms are required by a VAX 11/780 for all the calculations of the Primitive and Coordinated Joint tasks for a typical industrial robot with a force/torque sensor.

Table 2. Times required for intertask communications in several operating systems

Operating system and language Meglos, C Cedar, Mesa V system, C Harmony, C

Processor and interconnection

Type of inter-task communication

10 MHz MC68000s on 80 MBPS S/NET Dorados on 3 MBPS Ethernet, 5-10% load 10 MHz MC68000s on 10 MBPS Ethernet, no other load 10 MHz MC68000s on Multibus I

Blocking writes and reads to/from channels Remote procedure calls

Times required for communication (ms) Tasks on one Tasks on separate processor processors Message size Message size 10 bytes 100 bytes 10 bytes 100 bytes -

-

Send-receive-reply

0.9

Send-receive-reply

1.2

A Dorado is roughly equivalent to an IBM 370/178 computer. These times are for a null message.

w

1.0

0.9

1.2

1.1

1.9

2.2

3.0

1.1

50

Robotics & Computer-Integrated Manufacturing • Volume 6, Number 1, 1989

Assuming such a processor as a reference machine, which is much more powerful than the Motorola 68000 microcomputers used by Meglos, and assuming, conservatively, that it requires the same time to process messages as Meglos, the total processing time for the low level modules would be about 19-20 ms. Consequently, we conclude that our proposed control system could be implemented, using available systems, in situations where the Primitive module controls the actions of a single robot. 9. P E R F O R M I N G PROCESSES O F THE F O U R CLASSES In Section 2 we referred to the four classes of process which may have to be performed by an I M R S according to Shin et al. 24 In this section we discuss how this may be performed within the constraints of the our control hierarchy. Combinations of centralized and decentralized control, and horizontal and vertical communication are required. 9.1. Independent processes In independent processes, each robot works independently under the control of its own set of modules and the only interaction between its modules and those of the other robots is to change the value of shared state variables. The Workstation passes a c o m m a n d to each robot's Process module, which reports back to the Workstation when the robot has finished performing the job, or when a problem occurs. We use a "Controller" module for this purpose, installed at a level appropriate to the particular process. Modules at the same level of the individual robot controllers access the Controller to request changes in the value of the shared variable.

ecv -

0----,- //Primitive / / / robot i /

•

/

•

/

An example of a shared state variable is the speed of a conveyor in an application in which multiple robots unload parts from the conveyor; the belt speed is a parameter in trajectory calculation for each of the robots. This situation is shown in Fig. 9. The Controller is at the Primitive level because trajectories are calculated at this level. A separate Controller module is desirable because tasks only need to correspond with it when changes to the variable(s) occur. Each corresponding robot module keeps a local copy of the variable(s), and only modifies them when informed by the Controller of changes. A module that wishes to change the value of a shared variable sends the new value to the Controller using a non-blocking send. The latter sends it to all the affected modules, along with the time that the change occurred. The Controller arrangement is also appropriate in situations where a change in a variable value requires a consensus a m o n g the robots. 9.2. Loosely coupled processes Loosely coupled processes involve the use of shared resources in sequences that cannot be pre-planned. Each robot works under the control of its own set of modules and the shared resource that couples the robots should be controlled by an "Administrator" module between them. The Administrator is interfaced at the level that best suits the level of the resource. For example, the use of a tool shared between several robots is probably best controlled at the Step level, since the robots would need the tool for some steps of a process, but not for all of them. Thus, it would waste time to have the Administrator at the Process level. Each robot has to make inquiries of only one

o--*,-

,o ot 3 /

Fig. 9. Controlling the value of a shared variable with a single module.

Control and communications for multiple, cooperating robots • F. B. MAY et

module to use the shared resource, rather than check with all the other robots involved, as suggested by Shin et al. 24 This reduces the number of messages required to gain control of the shared resource. The addition of more robots to the system is straightforward, since all that is necessary is that the Administrator and the control modules of each additional robot should know each other's address. If a robot fails to return a shared resource before a specified period of time, the Administrator can query it to determine whether a problem has occurred. If it has, it can request that one of the other robots, perhaps one that is waiting for the resource, notify the Workstation of the problem. Management of shared resources by an administrator retains all the advantages of our distributed, asynchronous control hierarchy while ensuring the coordination between robots that would be provided in a mutual master level (the Workstation) in the NBS hierarchy. In addition, there may be more than just the loosely coupled process going on at one time and the coordination of the actions of the robots in the loosely coupled process, together with the control of the other process(es), would place a heavy computation and communication load on the Workstation in a monolithic hierarchy, possibly resulting in delays of all the processes. 9.3.

Tightly coupled processes

Tightly coupled processes should be controlled in a centralized fashion, with a single Primitive module

Process I

al.

controlling the Coordinated Joint modules, sensors, and any other modules at the same level, of all the robots as shown in Fig. 10. Figure 10 depicts the parallel hierarchies for two robots in which Robots 1 and 2 are controlled by the Primitive module of Robot 1 for the purpose of accomplishing the tightly coupled task. This form of control is similar to that suggested by others for material handling processes using multiple robots. 2'It In this case, it is necessary that the Coordinated Joint modules of all the robots should have a common pose-clock. The use of a common Primitive to servo all the robots synchronously provides good control of the process because all the trajectories of the robots are calculated by one module. Also, the common master receives feedback from all the robots each cycle time and generates a response by the next cycle time. The use of a common cycle time ensures that all the robots receive their poses at a constant rate. If a common cycle time were not used, the Primitive module could not respond consistently to all of the modules below it within the required response times, which could damage the robots or the workpiece. 9.4.

Serialized-motion processes

In the case of serialized-motion processes, direct horizontal communication between the Process level modules of robots should be used. In this way, a robot that has finished a job signals a second robot directly that its job is done, signalling the Workstation at the same time. This is more efficient than using only

Process 2

I Step I

Step 2

I

I

E-move I

I

E-move2

Primitive l / /

Co-o d noted Joint I

1

I

/

Primitive 2

Gripper I

51

Co-ord noted Joint 2

Gripper 2

Fig. lO. Multiple robots use a single Primitive module in tightly coupled processes.

52

Robotics & Computer-Integrated Manufacturing • Volume 6, Number I, 1989

vertical communication, in which case the first robot would signal only the Workstation which, in turn, would signal the second robot. 10. C O N C L U S I O N S In this paper we have proposed a flexible architecture for the control system of multiple, cooperating robots in an integrated, multiple robot system. Its hierarchical form is an extension and modification of the NBS control model for a single robot workstation. 4 Like the NBS model, it groups the various control functions into well-defined levels. Its modular form allows control modules to be developed separately, then assembled into an entire system. Our model places no restrictions on the type or manufacturer of robots that may be used or on the mixture of these within a single workstation. Our new architecture has major advantages in that it provides for asynchronous operations in and between control modules at the higher levels, retaining synchronous control, at the pose-clock rate of the individual robot, only at the lowest possible level. Thus, it is possible to incorporate multiple robots of different manufacture, and having different pose-clock rates, in the same workstation. In addition, it provides for asynchronous horizontal communication between control modules of the various robots and also between them and peripheral devices, such as vision systems. O u r architecture is particularly well-suited to implementations involving a high degree of parallelism in multiple processor and distributed systems. In the proposed hierarchy, each robot is controlled by its own set of control modules: Process, Step, Elemental Move, Primitive, Coordinated Joint, Gripper and sensor modules. The Low-level modules (Primitive and below) of each robot are driven synchronously by the robot's pose-clock. The higher-level modules of each robot execute only in response to commands from above or feedback from below. This is in contrast to the NBS Real-Time Control System 4 for a single robot workstation in which all control modules execute each pose clock cycle. We have outlined an approach to inter-task communications and to task management which calls for only a simple run-time operating system at the lower levels of the hierarchy. For inter-task communication, only simple Send, Receive and Reply primitives are required. The operating system must also provide efficient network communications, task scheduling and device handling. We emphasized simplicity in the operating system in order to permit the attainment of the efficiency necessary to accomplish all the intertask communications at the speeds necessary for effec, tive control.

We have shown that the inter-task communication load at the low levels, which is where the worst case occurs, could be handled at the speeds necessary for robot control, by at least one current message-passing operating system (Meglos 1.) designed for distributed computers running on currently available processors. It therefore follows that it could be handled more efficiently by devising a new operating system having only those features necessary for our architecture. This opens up opportunities for interesting further work in the areas of faster implementations of message passing and the design of economical computer systems designed specifically for robot control.

REFERENCES 1. Albus, J. S. et al.: Hierarchical control for robots in an automated factory. Proceedings of the 13th ISIR/ Robots 7 Conference, Chicago, 17-21 April 1983, Soc. Manuf. Engrs., Detroit, Michigan, pp. 13.29-13.43. 2. Alford, C. O., Belyeu, S. M.: Coordinated control of two robot arms Proc. IEEE Conf. on Robotics and Automation, New York, NY, 1984, Cat. No. 84CH2008, pp. 468-473. 3. Barbera, A. J.: An architecture for a robot hierarchical control system. National Bureau of Standards Report No. NBS-SP-500-23, Washington, 1977. 4. Barbera, A. J. et al.: RCS: the NBS real-time control system. Proc. Robots 8 Conference, Chicago, 1985, pp. 19.1-19.33. Soc. Manuf. Engrs., Detroit, Michigan. 5. Birrell, A. D., Nelson, B. J.: Implementing remote procedure calls. ACM Trans. Computer Systems 2(1): 39-59, February 1984. 6. Bonner, S., Shin, K. G.: A comparative study of robot languages. IEEE Computer 15(12): 82-86, December 1982. 7. Bowen, B. A., Buhr, R. J. A.: The Logical Design of Multiple-Microprocessor Systems. Englewood Cliffs, Prentice-Hall, 1980. 8. Cassinis, R., Automatic Resource Allocation in Industrial Multi-Robot Systems, Proceedings of the 1986 IEEE International Conference on Robotics and Automation, April 1986, San Francisco, pp. 2012-2017. 9. Chang, E., Roberts, R.: An improved algorithm for decentralized extrema-finding in circular configuraions of processes. Commun. ACM 22(5): 281-283, May 1979. 10. Cheriton, D. R.: The V kernel: a software base for distributed systems. IEEE Software 1(2): 19-42, April 1984. 11. Clark, C. J., Stark, L.: A comparison of control laws for a cooperative robot system. Proc. 1986 International Conference on Robotics and Automation, 7-10 April 1986 San Fransisco, pp. 390-394. 12. D'Souza, C., Blume, C.: Aspects to achieve standardized programming interfaces for industrial robots, Proc. 13th ISIR/Robots 7 Conference, Chicago, 1983, pp. 7.110-7A21, Soc. Manuf. Engrs., Detroit, Michigan. 13. Fitzgerald, M. L., Barbera, A. J.: A low-level control interface for robot manipulators. Robotics Computerlntegr. Mfg. 2(34): pp. 201-213, 1986. 14. Gaglianello, R. D., Katseff, H. P.: Meglos: an operating system for a multiprocessor environment. Proceedings of the 5th International Conference on Distributed

Control and communicationsfor multiple, cooperating robots • F. B. MAYet al. Computing Systems, Denver, May 1985, pp. 35-42, 15. Garcia-Molina, H.: Elections in a distributed computing system. IEEE Trans. Computers C-31(1): 48-59,

January 1982. 16. Gentleman, W. M.: Message passing between sequential processes: the Reply primitive and the Administrator concept. Software Practice and Experience 11: 435-466, 1981. 17. Gentleman, W. M.: Using the Harmony operating system, National Research Council Canada Report No. 24685, December 1983 18. Haynes, L. S. et al., An application example of the NBS robot control system. Robotics Computer-lnteor. Mfg, 1(1): 81-95, 1984. 19. Hayward, V., Paul, R. P.: Robot control under unix. Proceedings of the 13th ISIR/Robots 7 Conference, Chicago, 1983, pp. 20.32-20.44, Soc. Manuf. Engrs., Detroit, Michigan. 20. McLean, C. et al.: A computer architecture for small batch manufacturing. IEEE Spectrum 20(5): 59-64, May 1983. 21, Mujtuba, M. S. et al.: Stanford's AL robot programming language. Computers mech. Engno 1(1): 50-57, August 1982,

53

22. Pyle, I. C.: The ADA Programmin9 Language , p. 92ff. Englewood Cliffs, Prentice-Hall, 1981. 23. Shimano, B. E. et aL: A robot programming system incorporating real-time and supervisory control: VAL-II. Proc. Robots 8 Conference, Detroit, 1984, pp. 20,103-20.119, Soc. Manuf. Engrs. Dearborn, Michigan. 24. Shin, K. G., Epstein, M. E., ¥olz, R. A.: A module architecture for an intergrated multi-robot system. Proc. Eighteenth Annual Hawaii International Conference on System Sciences, January 1985, pp. 120-129. 25. Shin, K. G., Epstein, M. E.: Communication primitives for a distributed multi-robot system, Proceedings of the 1985 IEEE International Conference on Robotics and Automation, St. Louis, 25-28 March 1985, pp. 910 917. 26. Shneier, M.: Robot sensing for a hierarchical control system. Proc. 13th ISIR/Robots 7 Conference, Chicago, 1983, pp. 14.50--14.66, Soc. Manuf. Engrs., Detroit, Michigan. 27. Silberschatz, A.: Port directed communication. Computer J. 24(1): 78-82, 1981. 28. Ward, M., Stoddard, K.: Karei: a programming language for the factory floor. Robotics Aye 7(9): 10-13, September 1985.