Process migration and its influence on interprocess communication

Computer Communications 21 (1998) 781–792

Process migration and its influence on interprocess communication Larry Hughes* Department of Mathematics and Computing Science, Saint Mary’s University, Halifax, Nova Scotia, Canada B3H 3C3 Received 4 August 1997; received in revised form 2 March 1998; accepted 2 March 1998

Abstract Many process migration algorithms aim for transparency where neither the behaviour of the migrated process nor its appearance to the rest of the system is affected by the migration. However, there can be overheads associated with achieving transparency; maintaining communications after a migration can result in additional network traffic, increased system complexity, or both. This paper describes a taxonomy based upon the three basic techniques employed to support communications after a migration within a single network domain, compares the approaches taken in light of the number of peers, the life-span of the migrated process, the number of messages sent after a process migrates, communication delays, and whether the process undergoes a subsequent migration. Examples of existing migration algorithms are used to illustrate the taxonomy. q 1998 Elsevier Science B.V. Keywords: Distributed systems; Interprocess communication; Process migration; Mobile computing

1. Introduction Process migration is the transfer of a process’ image from its current execution host to another one. The reasons for migrating a specific process are varied; Smith [1] includes, among others, the continuation of processing in the event of execution host shutdown and the distribution of system load uniformly across the available hosts. Completion of a migration should result in the process resuming its activities on a new execution host. An important goal of a process migration algorithm is transparency. In [2], Douglis and Ousterhout describe transparency in terms of the process: neither the behaviour of the process nor its appearance to the rest of the world should be affected by the migration. However, an algorithm that is deemed process transparent may impose overheads on the supporting communication facilities as a result of the need to maintain communications between the migrated process and its peers. Systems subject to these overheads exhibit one or both of the following: 1. Additional network traffic. In addition to the expected traffic between the process and its peers, messages to the migrated process may be forwarded by one or more intervening objects. In some cases, control messages may be required to specify the location of the migrated process, while in others, messages may be delivered to hosts not involved in the migration. * Corresponding author.

0140-3664/98/$19.00 q 1998 Elsevier Science B.V. All rights reserved PII S 01 40 - 36 6 4( 9 8) 0 01 5 6- X

2. Increased complexity and state. The migration of a process can imply a name change; if the migration is to be process transparent, the supporting layers require an internal name mapping mechanism. These can be invoked once, as part of the migration, or on a demand basis, whenever a communication fails. Such overheads can be a deterrent to implementing migration in a distributed system or can even preclude the employment of an existing migration tool. Additional network traffic and increased complexity may not be acceptable in situations such as the migration of short-lived processes, or when using mechanisms that update process state information on a demand basis. Similarly, if communications after a migration cause additional network traffic, migrating long-lived processes can result in network degradation. Both problems can be compounded if a process is migrated more than once. Understanding migration induced communications overheads can influence the design and implementation of facilities for communications in distributed real-time systems, fault-tolerant systems, and mobile computing systems.

2. Background: process migration techniques A process migration is the transfer of sufficient state information, from one host to another, to ensure that the process can resume execution on its new host. Process

782

L. Hughes/Computer Communications 21 (1998) 781–792

migration is analogous to using a cellular telephone; communications are maintained as the telephone moves from one cell to another. When a process is first created, it resides on its home host; in some migration algorithms, the home host remains involved with the migrated process for the duration of process’ existence. The decision about where and when a process is migrated depends upon the system’s migration policy. These vary, but the reasons for selecting a particular policy include load distribution, continued process execution, and reducing network traffic [1]. Not all processes can be migrated since they may be required on a given host; for example, window managers are usually bound to the host upon which they reside. Once a process is selected for migration, a migration request is passed to the migration manager on its host; each host in the distributed system able to participate in the migration runs a migration manager. The local or emigrant migration manager sends a migration request message to other, potential immigrant migration managers on the network. Part of the request message may include general requirements, such as the type of host (it may not be possible to migrate processes to machines with different instruction sets or operating systems; a limitation overcome with the application of portable byte code), and process specific information, such as memory and CPU requirements. The potential immigrant migration managers that meet the requirements respond to the request. The emigrant migration manager filters the responses and selects one; the simplest approach is to accept the first response. The process’ state information (i.e. registers, memory, and other process-specific information) is then transferred to the selected immigrant host. If the transfer is successful, the immigrant host becomes the new execution host; if the migration fails (for example, the immigrant host terminates or the network partitions), process execution should resume on the emigrant host. In many migrations, the migrated process may keep references to a previous execution host (typically its home host). Consider the migration of a user’s shell that has left an opened window on the user’s machine; since the process, rather than the user, has migrated, all window output should be directed back to the original window. References to a previous host are known as residual dependencies; these are most often found on the process’ home host. Residual dependencies can require ‘garbage collection’ to handle those situations whereby the migrated process terminates abnormally but still appears to exist (the process is said to be an evanescence), or a host on which a residual dependency exists, terminates abnormally, and the migrated process cannot be located (becoming an exile) [3]. The success of a migration algorithm is influenced by a number of issues [4], notably transparency (hiding the migration operation from the migrated process and its peers), reliability (continuing execution as if the migration had not occurred), and speed (minimizing the period of

process inactivity). Of these, speed is often considered to be of paramount importance since the time the process is inactive should be minimized. The most significant area where speed gains can be made is in the migration of process state information [5]; approaches to migrating state information include: • •

•

•

Halting the process’ execution until the migration has completed (also known as freezing). Allowing the process to continue running during migration, then freezing the process while any modified state information is transferred (for example, data pages that have been modified by the process during the migration must be transferred again). Migrating the process’ working set only, keeping unwanted pages on the emigrant host as residual dependencies; the pages are transferred to the immigrant host on a demand basis. Migrating the process’ working set, but returning all pages to backing store (eliminating residual dependencies); the pages are transferred to the immigrant host on a demand basis.

Despite the recent growth in popularity of distributed systems and the potential of migration for applications such as load balancing, migration is often described as a ‘heavyweight’ activity due to the amount of processing required to migrate a process; further, migrating a shortlived process can result in few, if any, gains [6]. Another factor that can discourage migration is the requirement that both the emigrant and immigrant hosts support the same architecture and operating system; however, the availability of languages such as Java, portable byte code, and welldefined virtual machines [7,8] has been shown to overcome this problem [9].

3. Migration and communications A key objective of process migration algorithms is transparency. In every algorithm examined for this paper, transparency for the processes involved in a migration (i.e. the migrated process and its peers) is of paramount importance. Typically, the only concession to this transparency is the availability of a signal (such as SIGMIGRATE, found in Locus [10]) that informs the migrated process of its migration. Successful completion of a migration implies that all necessary process state information has been transferred to the new execution host; this includes, not surprisingly, state information involving communications with the migrated process’ peers. Maintaining communications after a migration depends upon the process naming technique and how this is supported by the communication system. Many naming schemes bind processes (either directly or indirectly) to a particular host; process identifiers that appear globally unique at the process level can require

L. Hughes/Computer Communications 21 (1998) 781–792

mapping into physical addresses within the communication system. In these situations, some form of message redirection will be needed after a migration since the identifier no longer refers to the migrated process’ execution host. If, as a result of the migration, the peers’ communication systems must change the internal mapping of an identifier to its new physical address, the migration algorithm is said to require intervention. Alternatively, if the peers’ communication systems are not to be informed of the migration, then messages for a migrated process must be forwarded to the process’ new execution host. If the naming scheme adopted does not bind the process to a particular host, the migration algorithm is said to be concealed from the communication system. These three categories, intervention, forwarding and concealed, form the nucleus of a classification scheme for describing the maintenance of interprocess communications after a migration. 3.1. Intervention Intervention is the active participation of the migrated process’ peers (or their communication systems) in maintaining communications after migration. Since intervention at the process level defeats the objective of migration transparency, it will not be considered in this paper. For migration to be process transparent, some systems have opted for intervention by the communication system. Frequently this is found in systems that maintain state information referring to the physical location of each process; when a process migrates, this information must be updated. Process naming in such systems usually involves mapping a unique process identifier into a physical address. The updating technique adopted can be classified into either immediate or demand. Immediate (or eager) intervention describes the updating of all state information before the migration has completed. At one extreme, the new physical location of the migrated process can be supplied to all active communication systems; each communication system is then responsible for updating its mapping tables for the process in question. All subsequent references to the process are routed to the process on its new execution host. This approach to immediate intervention has at least one limitation: all communication systems, whether or not they support peers of the migrated process, are requested to perform the update. The drawback can be mitigated by restricting the update message to gnly those hosts that support peers of the migrated process. This technique requires that state information be maintained by the communication system with respect to each process and the peers with which it communicates. The number of communication system update messages sent in an immediate intervention depends upon the number of remote peers and the organization of the name mapping tables. If each communication system maintains a single

783

mapping table for all of its processes, then H (the number of hosts on which the peers reside) messages are required; however, if each process is associated with its own mapping table, then up to P (the number of remote peers) messages may be needed. These messages, sent as network packets, must be acknowledged to ensure that the updates have taken place, meaning that at least H 3 2 packets will be sent on the network. The host responsible for the transmission of the update messages is a potential single point of failure; if it fails before completing the update cycle, one or more peers may be unable to maintain communications with the migrated process. Demand (or lazy) intervention requires the updating of a communication system’s state information when a migration is detected. This usually appears as a communication fault when a peer attempts to send a message to the migrated process. This approach has the advantage that updates are handled as needed rather than when the migration occurs. The problem with demand intervention is determining the location of the migrated process; two solutions to this are: 1. The physical address of the migrated process can be maintained by a ‘well-known’ object. The well-known object is typically either the migrated process’ home host or some form of name server. Whenever a migration occurs, the well-known object must be informed of the process’ new execution host. Additional message types are necessary. If the process migrates without returning to its home host, an update message must be sent to the well-known object from the new execution host. An intervention message, requesting the new execution host address from the well-known object, is also required. A concern with this type of demand intervention is that the well-known object is a potential single point of failure; typical solutions employ fault-tolerant techniques such as using a backup object that maintains copies of the addresses. The minimum number of packets generated is K 3 2, where K is the number of well-known objects. If the well-known objects share a common group identifier, the number of packets can decrease to at least K þ 1; one update message and K acknowledgements. 2. The physical address of the migrated process is known only to the host on which it resides. In this approach, the peer’s communication system must search for the migrated process by sending an intervention request message to all hosts that potentially support the migrated process. Upon receipt of the message, each communication system inspects its list of processes; if the process in question resides on the host, the communication system must respond with its physical address. This approach to demand intervention can reduce the network traffic needed to determine the location of the migrated process since intervention messages are sent only when required. However, transmitting the message to all hosts (i.e. broadcasting it) requires every communication

784

L. Hughes/Computer Communications 21 (1998) 781–792

system to inspect its process tables to determine whether it is the current execution host. Demand intervention can be divided into three distinct steps. Step 1, to determine that the process has migrated. If the message is not acknowledged after a system-defined number of transmissions to the last known host, the process is deemed either to have migrated or to have terminated. By requiring the last known host to respond with an indication that the process cannot be found, the time required to determine that the process is no longer on the last known host can be reduced; in the interest of maintaining communications, the address is assumed to be outdated and a migration is deemed to have occurred. Step 2, to send the intervention message to the well-known object or to all hosts; the response to the intervention message must include the location of the new execution host. Step 3, to retransmit the original message to the new address. Migrating a process during a demand intervention can mean that the address supplied is not valid. In this situation, the peer’s commuunication system has no alternative but to use the address, determine that it is incorrect, and to perform another demand intervention. Both immediate and demand intervention generate additional network traffic, either in the form of update messages, location request messages, or both. Further, increased complexity (usually in the communication system) is required to handle state information updating. The additional traffic and increased complexity can result in delays, either in the resumption of execution (immediate intervention or demand intervention with a ‘well-known’ object’) or when a peer’s communication fails (demand intervention and searching for the process). 3.2. Forwarding In an intervention, the communication system of each peer must modify the internal address of the migrated process to that of its new host. The forwarding technique avoids intervention by leaving the identifiers used by the peers unchanged; instead, messages continue to be transmitted as if the migration has not occurred. This means that any messages sent by the peers still arrive at the host on which the migrated process was executing when communications were first established. When a message arrives at the emigrant host, facilities must be in place to forward the message to the immigrant (i.e. new execution) host. The minimum forwarding information required is the migrated process’ host address. Messages can be forwarded by the emigrant host in one of two ways: 1. Updating the communication system’s mapping table; any references to the identifier(s) associated with the migrated process can be changed to the address of the execution host. Whenever a packet arrives containing a message with an identifier naming a migrated process,

the communication system obtains the address of process’ execution host from the mapping table and retransmits the packet with this address. 2. Replacing the migrated process with a second process (a forwarding agent) that is responsible for forwarding messages to the migrated process’ execution host. The forwarding agent assumes the name of the migrated process; messages destined for the migrated process are delivered by the communication system to the forwarding agent. When a message arrives, the forwarding agent retransmits it with the migrated process’ new destination address. Regardless of the forwarding mechanism adopted (either modifying the communication system’s mapping table or employing a forwarding agent), the need to include former execution hosts in the communication is an example of a residual dependency. For the remainder of this paper, the term ‘forwarding agent’ refers to either a forwarding process or a communication system that supports forwarding. If a process migrates more than once, a forwarding chain can be created, consisting of the former execution hosts. In these situations, the amount of network traffic grows in a linear fashion as the number of hosts in the forwarding chain increases. If the chain consists of F hosts, each message is processed and retransmitted F times (potentially more if a process visits a host more than once). If acknowledgements are required to ensure that the message has reached the next forwarding agent or communication system, processing and network traffic increases to at least F 3 2. Forwarding does not scale well: the longer the forwarding chain, the greater the delay. Maintaining a forwarding chain is costly in terms of system resources (residual dependencies must remain on each host visited) and fault-intolerant (the failure of an agent, the communication system, or the process’ host is a potential single point of failure). Techniques employed to overcome these problems include: 1. Short-cutting, an optimization technique intended to reduce the number of forwarded messages by avoiding hosts in the chain. One approach to short-cutting has the immigrant communication system informing the peer’s communication system of the new execution host address whenever a forwarded message is received. The number of packets required is at least F 3 2 þ 1 with the additional packet containing the address sent to the peer’s communication system. Subsequent messages from the peer refer to the new address. A variation on this approach requires each forwarding agent to inform the sender of the message (i.e. the peer’s communication system or a previous forwarding agent) of the ‘next’ address in the chain; subsequent messages bypass the forwarding agent’s host. Short-cutting may result in forwarding agents remaining active although they are no longer part of a chain; distributed garbage collection techniques are needed to

L. Hughes/Computer Communications 21 (1998) 781–792

terminate the ‘orphan’ agents. Forwarding with shortcutting is essentially a form of delayed immediate intervention since the peer’s communication system receives an update message after the first message has been received by the execution host. Associating each migrated process with a single forwarding agent (usually on the migrated process’ home host) and supplying that agent with the address of the new execution host each time a migration occurs, thereby ensuring that messages are forwarded only once. However, should the forwarding agent or its host cease to function, communications with the migrated process will become impossible, and result in a fault-intolerant system. Further, forwarding is not applicable to those cases where a process is associated with a host that is about to terminate. 2. Adopting a ‘belt-and-braces’ approach to ensure additional fault-tolerance by using a combination of forwarding and demand intervention; the new address of the migrated process is used by the forwarding agent and maintained by a name server. Should the forwarding agent or its host fail, the peer’s kernel can request the address from the name server. 3.3. Concealed A process migration is said to be concealed if, as a result of the migration, no control traffic is generated and increased state information is not required to maintain communications. Concealment can be achieved in one of two ways, both of which depend upon the process naming technique adopted by the system. The first method requires indirect process naming; whereby messages are not sent to the process directly, but via a fixed intermediary, such as a mailbox. Messages destined for the migrated process are sent to the mailbox and can be read when required (a mailbox is essentially a quiescent forwarding agent, that forwards messages when requested by the migrated process). Process migration is concealed from communications, regardless of the number of migrations. Indirect naming has several limitations and can be considered only a partial solution to the problem. First, without some form of fault-tolerance, all communications with the migrated process cease if the mailbox fails. Second, special consideration must be given to handling signals; since a signal cannot be passed to the process via the mailbox, a forwarding object or some form of intervention is required. The second method uses direct process naming and global, location independent, identifiers that are not bound to a specific host. Adopting a single identifier structure that is common to the entire system (i.e. processes, communication systems, and network) means that the communication system is not required to map or to modify the identifiers. Interprocess communications within a host require the communication system to supply the message to the destination process(es); while interprocess communications between hosts can be achieved by either [11]:

785

1. Encapsulating the message and identifier in a broadcast packet. The packet is sent to all hosts on the network; each communication system examines the identifier and supplies the message to any processes associated with the identifier. Transmitting broadcast packets is a less than ideal solution since part of the packet must be devoted to the identifier and all hosts are required to inspect the packet. 2. Encapsulating the message and using the identifier supplied with the message as the destination address. Systems that support this type of naming require network hardware that can be programmed to filter a packet by inspecting its destination address: if the address is acceptable to the host, a copy of the packet is taken, otherwise it is ignored. Having a communication system configure its network hardware with the identifiers of its processes ensures that only packets destined for processes on that host will be accepted. If a process migrates or terminates, the communication system can remove the identifiers from its host’s network hardware. Using either of the above techniques means that identifiers are unchanged after the process migrates and the peers’ communication systems are not informed of the migration. Messages sent after a migration arrive at the new execution host without any intervention on the part of the peer’s communication system (if signals are mapped into messages and the same communication system is used, then signals will also follow the process [12]). If a migration occurs during the transmission of a message, it may be necessary to retransmit the message. This method is concealed since additional processing and network traffic are not required when the process migrates, nor must any additional state information be maintained by the communication system. The only source of delay is the possible retransmission of a message sent during the migration of the process.

4. Implementations The taxonomy developed in the previous section classifies the effect of process migration on interprocess communications into one of three categories. To demonstrate the application of the classification scheme, a number of process migration implementations are examined and classified. 4.1. Charlotte Charlotte [13] is an example of immediate intervention; informing all the peers’ communication systems is an integral part of the migration. Charlotte interprocess communications are achieved using a duplex communication channel known as a ‘link’. A pair of processes share a link maintained by the host’s communication system (or communication systems, as the processes need not reside on the same machine). Messages

786

L. Hughes/Computer Communications 21 (1998) 781–792

sent to a peer are queued by the peer’s communication system until it receives a request to read the message. Internally, the link includes the physical address of the process on the other half of the link. The ‘transfer phase’ of the Charlotte migration algorithm involves transferring the migrating processy state and sending ‘link update’ messages to all peer communication systems. This message contains the execution host address of the migrated process; the peer communication systems are responsible for updating the specified links. All link update messages are acknowledged by the peer communication systems. Messages sent by peer processes after the links have been updated will arrive at the execution host which queues these messages until the migrated process reads its links. If a link has not been updated when a peer sends a message, the message arrives at the emigration host and is held by the emigration communication system. The final phase of the migration involves transmitting a message that includes a list of all messages arriving at the emigration host during the transfer phase; this information is used to update the migrated process’ links. Subsequent migrations cannot occur until all links have been updated. 4.2. Locus Locus [10] is an example of demand intervention using a well-known object; whenever a transmission fails, the home-host is queried for the new address. Locus supports Unix processes; a parent process can create child processes on its own or on a remote host [via a remote fork ()]. Processes are named with globally unique identifiers, each consisting of two parts: a ‘site-identifier’ and a ‘local process identifier’. The ‘site-identifier’ specifies the process’ home host. In keeping with other Unix implementations, the identifier remains with the process throughout its existence, regardless of its location. Parent processes can communicate with their children via pipes and signals. Since a process can migrate, each parent is associated with a ‘remote child list’ which specifies the location of each child not on the parent’s host. When communicating with a process, a message and the identifier of the process are supplied to the local communication system. If it determines that the process is executing locally, the message is delivered; otherwise, the message is sent to the home host (the and ‘origin site’) using the site-identifier taken from the process identifier. The home host communication system attempts to deliver the message to the process; if the process has migrated, the address of the execution host is returned. The local communication system then sends the message to the process on its execution host; to reduce network traffic, the new address is cached by the peer’s communication system [2]. Whenever a process migrates, it must inform the home host of its new execution host. If the destination process

emigrates to a new execution host during the detection cycle, the resulting communication will fail, meaning that it will be necessary to contact the home host again to determine the migrated process’ new location. Each home host is twinned with a ‘surrogate’ site that keeps a copy of the location list maintained by the home host. If the home host fails, intervention requests are sent instead to the surrogate site which responds with the new address of the migrated process. 4.3. Mosix Mosix [14], like Locus, is an example of demand intervention through a well-known object. Mosix uses process migration as part of a larger project to study load balancing. When a Mosix process is migrated, the new execution host informs the process’ home host of its new address; other hosts can be informed should additional fault-tolerlance be required. When a communication with a migrated process fails, the home host is contacted and the new address is returned. 4.4. V The V-System [15], like Locus and Mosix, is an example of a demand intervention implementation; however, unlike them, the address of the migrated process is maintained on its execution host. V process identifiers are globally unique and consist of a ‘logical host identifier’ and a ‘local index’. When a process sends a message, the kernel is responsible for mapping the logical host identifier into a physical host address; a cache of these mappings is maintained by the kernel. If the address indicates a host other than the local one, the message is transmitted to the host with the physical host address. If no response is obtained within a ‘small number of retransmissions’ it is assumed that the process has migrated. The peer’s kernel then invalidates the cache entry for the logical host and broadcasts (i.e. transmits to all hosts on the network) a request for the physical address of the host currently supporting the logical host in question. Receipt of the message containing the physical address of the logical host means that its cache entry is updated. The peer’s kernel also updates the cache when messages from the migrated process arrive. Updating the cache with addresses of incoming messages requires message transmissions from the migrated process; since this cannot be guaranteed, demand intervention must also be used. Demand intervention is required whenever communications fail. The kernel must update its mapping cache by examining incoming messages or by detecting that a process has migrated. The steps involved in detection require ad ditional complexity and the generation of additional network traffic. Furthermore, broadcasting the update request requires all hosts on the network to examine their cache tables for the specified logical host identifier.

L. Hughes/Computer Communications 21 (1998) 781–792

Immediate intervention is proposed as an alternative; the new physical host address of the migrated logical host would be broadcast to all hosts. Since a broadcast packet can be lost and there is no suggestion that this packet would be acknowledged, demand intervention would still be required by those hosts that did not have the new address of the logical host. 4.5. Sprite Sprite [2] is one of a number of systems that uses the home host to forward messages to the migrated process. When a process migrates in Sprite, the home host maintains forwarding information; messages are redirected to the migrated process on its execution host. Peers are not informed of the migration. A process does not migrate from its execution host (it can be ‘evicted’); instead, it returns to its hdme host. A subsequent migration causes the process to move to another execution host. Messages that arrive for the process during its migration are queued until the migration has completed. 4.6. DAWGS DAWGS [16], like Sprite, uses the home host to forward messages to the migrated process. Subsequent migrations of the same process pass through the home host, thereby ensuring that there is at most one forwarding object. 4.7. DEMOS/MP DEMOS/MP [17] supports forwarding chains and shortcutting. A DEMOS/MP process address consists of two parts: a globally unique process identifier and the address of the ‘last known machine’ The process address is a constituent part of a process’ ‘link table’. Each element in the link table refers to an object (i.e. a process or kernel) with which the process can communicate. When a process migrates, it leaves behind a forwarding address (a process address); the physical address of the new execution host is entered as the address of the ‘last known machine’. The arrival of a message for a migrated process requires the local kernel to extract the ‘last known machine’ address from the forwarding address and to forward the message to this address. In an attempt to reduce the number of times a message is forwarded, DEMOS/MP employs a short-cutting technique to ‘update links’. The updating procedure is invoked whenever there is a message to be forwarded. The forwarding kernel both transmits the message and returns the process address (including the ‘last known machine’ address) to the kernel that sent the message. Receiving the process address, the sending kernel scans the link table of the process that sent the message, and updates the ‘last known machine’ address of each link that refers to the migrated process.

787

Subsequent transmissions use the new machine address, bypassing the forwarding kernel. Since any number of processes can have copies of an address and it is not possible to determine every instance when they might be required for use, forwarding information can never be safely removed. The forwarding scheme developed for DEMOS/MP creates additional network traffic (in the form of update link messages), additional state information (the forwarding addresses), and additional complexity (updating link tables). 4.8. W3Objects Dealing with object, rather than process, migration, Ingham et al. [18] describe how a combination of forwarding (with short-cutting), demand intervention (to a name server), and immediate intervention (to name servers and peers) are used to maintain contact with migrated objects. The first message (sent through a chain of forwarding agents) following a migration results in two actions. First, as the message passes between the hosts, short-cutting takes place, with the removal of all forwarding agents between the host that received the message and the object itself. Second, the acknowledgement from the migrated object to the peer that transmitted the message includes the new address of the object. Subsequent messages from the peer use the new address. Should a reference to an object fail because of the migration, the peer can contact a name server with an intervention message and request the new address of the migrated object. The address may refer to the actual object or to a forwarding agent. Finally, both peers and name servers can request that immediate intervention be invoked whenever an object migrates. This ensures that the address in use kefers to the current location of the object. 4.9. Mach Mach [19] is an example of a system that can be classified into either the intervention or the forwarding categories; which category depends upon the type of ‘network server’ used. Mach processes communicate by passing messages through ‘ports’. A process with write privileges can place messages in the port, while a process with read privileges can extract messages. Processes on separate hosts can exchange messages using a pair of network servers (typically one per host) which exchange messages across the network interface. A message from a process writing to a port is read by its local network server; the message is then sent to the remote network server. It writes the message to the remote port which can be read by the remote process. If a process migrates, responsibility for maintaining communications is left to the network servers. How this is

788

L. Hughes/Computer Communications 21 (1998) 781–792

achieved depends upon the network server implementation; either intervention or forwarding can be used. 4.10. Lego The Lego system [20] is an example of concealed process migration. In Lego, all identifiers are location independent and sharable. No identifier mapping is performed when a message is sent to another process; the naming scheme is based upon a single 48-bit identifier structure that is used throughout the system. Structurally, a Lego identifier is identical to an 802.3 (Ethernet) multicast address (the multicast bit is always set). Each Lego kernel is responsible for programming its network hardware to recognize those identifiers currently in use by resident processes. When a process associates with an identifier, the kernel records the identifier in its local identifier list and programs the network hardware to recognize it. The hardware filters packets by examining a packet’s destination address; packets with destination addresses recognized by the hardware are delivered to the kernel while all others are ignored. When sending a message, the transmitting process supplies a destination identifier and specifies a routing indicator which instructs the kernel on the disposal of the message. There are four indicators: RMT-ONLY, the message is always placed onto the network; LCL-ONLY, the message is supplied only to local processes (if they exist); LCLAND-RMT, the message is supplied to any local processes and is placed onto the network; and RMT-IF-NO-LCL, the message is placed onto the network if the process does not exist on the local machine. When a Lego process migrates, it keeps the identifiers with which it is associated. If the migration is successful, the emigration kernel removes any unshared identifiers used by the process and clears its network hardware of them. The immigration kernel enables its hardware to receive the messages with the migrated process’ identifiers. Messages sent with an identifier associated with the migrated process are ignored by all hosts except the one on which it resides; neither the peers nor any kernels are affected by the migration. All communications are the responsibility of the processes involved; the kernel gives no indication of the success of a transmission. If a communication takes place during a migration and the message is lost, it is the responsibility of the peer to retransmit the message, thereby avoiding ordering problems. The number of migrations that occur between message transmissions is also concealed from the peers and their kernels. Lego process migration is concealed because its identifier structure is globally unique and programmable network hardware is employed to filter packets. No additional network traffic is generated because of the migration, nor is the kernel required to maintain extra state information.

4.11. Other systems A number of other systems make passing reference to their process migration capabilities; however, little detail about implementation is given. Eden [21] allows object migration; however, since an object can be an active participant in the migration, transparency is lost. In [22], Fowler gives a brief description of the use of forwarding in Eden. Migration is supported in Amoeba [23,24] and Chorus [25–27]; both systems appear to use some form of intervention.

5. Comparison of categories The effect of a migration on interprocess communications is influenced by a number of factors relating to scaling, including [11]: • • • •

the number of peers; the lifespan of the process (i.e. whether the migrated process is short- or long-lived); the number of messages sent by a process’ peers after a migration (notably, 0, 1, or more); a second or subsequent migration.

These factors are now used to compare and contrast the various categories of the taxonomy, using the implementations discussed in the previous section as illustrations. 5.1. Intervention The immediate intervention category ensures that after completion of a migration each peer has the correct address of the migrated process. As the number of peers increases, there will be a corresponding growth in network traffic and a comparable delay in resuming execution of the migrated process. In Charlotte, the emigrant communication system must send link update messages to the peers’ communication systems; additional network traffic is generated since information about any messages that arrived during the migration must be forwarded to the immigration communication system. Migrating a short-lived process, or one that receives very few messages after the migration, can mean that much of the time associated with updating the peers’ communication systems has been wasted. Similarly, each time a process migrates without receiving messages from its peers, the previous updates have been for naught. The addresses maintained by peers involved with migration in a system supporting demand intervention are out-ofsync until communications with the migrated process are necessary. Demand intervention ensures that only those peers required to communicate with the migrated process actually cause additional traffic; however, before the new address can be obtained, a message must be sent that is

L. Hughes/Computer Communications 21 (1998) 781–792

unacknowledged by the process or negatively acknowledged by an emigrant communication system. The number of packets required for this step is between 2 and R; 2, if the emigrant host responds with a negative acknowledgement, and R, if R transmissions of the packet are needed to determine that the address is incorrect. Further, it takes at least two packets to obtain the new address of the process (the intervention request and the response) before the message can be sent to the migrated process. In total, between 4 and R þ 2 additional packets will be generated in determining the location of the migrated process (or more, if retransmissions are required). Processes that receive few messages, are short-lived, or are subject to more than one migration lend themselves well to demand intervention, since intervention messages are sent only when required. If all peers are to communicate with the migrated process, each peer must detect that a migration has occurred and must generate an intervention message. If these requests occur simultaneously, the communication system maintaining the address can be flooded with messages. Certain optimizations can be made to demand intervention; in V, each network packet received causes the kernel’s mapping table to be updated if the execution host’s physical address is out of date. Systems such as Locus, in which the intervention message is sent to the home host, require one or possibly two message types: one indicating the address of the new execution host and the other, the intervention message. Additional complexity is required to handle the failure of the home host; in Locus, a surrogate home host is employed. If, as in V, the address of the migrated process is maintained only on the execution host, it is necessary to send the intervention message to all hosts. If N kernels attempt to locate the migrated process simultaneously, each kernel will receive at least N ¹ 1 intervention packets, potentially flooding the kernel. Although these requests may, be periodic, they introduce an unnecessary additional load on each kernel. In demand intervention, should the process migrate after the address has been found, the cycle of determining that the process has migrated and then obtaining its new address must be repeated. 5.2. Forwarding Three ‘pure’ forwarding implementations have been classified by the taxonomy: single forwarding agent (Sprite and DAWGS) and forwarding chain (DEMOS/MP). The number of forwarding agents involved is not affected by increasing the number of peers, other than the potential need for a larger message queue at each host. A single peer sending a message will result in F (the number of hosts involved in the forwarding) additional copies of the message on the network; if each message is acknowledged by its receiver, then the volume of network traffic will increase to at least F 3 2 packets (or more, if retransmissions are

789

required). To overcome the problems associated with multiple forwarding agents, short-cutting techniques for the bypassing of hosts may be attempted (as in DEMOS/MP and W3Objects). When a migrated process terminates (be it short- or longlived), some form of garbage collection may be required to remove residual forwarding agents or mapping table information. The availability of distributed garbage collection techniques means that any ‘orphaned’ forwarding agents will be removed from the system. It can be argued that if garbage collection exists, the removal of forwarding agents is done as part of the system’s responsibilities; however, if forwarding agents are not used, there is no requirement to collect them. A restriction on all types of forwarding is the failure of a forwarding agent in the chain between the migrated process and its peers. Without fault-tolerant alternatives, the failure of an agent will cause communications to cease. 5.3. Concealed Systems in which migration is concealed from interprocess communications exhibit many of the positive characteristics of demand intervention. There is little to distinguish between the two when consideration is given to an increase in the number of peers, the lifespan of the migrated process, or multiple migrations. It is only when the peers attempt to communicate that major differences become apparent. Since the migration is concealed, each message sent results in one packet (an exception to this is if the process migrates during a transmission when the peer may be required to retransiffit the message). However, in demand intervention, between 4 and R þ 2 packets are required; in the worst case, where each host supports a peer, intervention packets are sent to all hosts on the network. 5.4. Summary Table 1 summarizes the additional network traffic for each category in the taxonomy; first, in terms of the number of packets required at migration time, and then with the transmission of the first message. The number of packets are assumed to be the minimum required; if faults occur, additional transmissions may be necessary. The column labelled ‘migration-time’ lists the number of packets that must be sent during the migration phase to ensure that communications can continue after the migration. Only immediate intervention and demand intervention (using well-known objects) generate network traffic at this stage. Immediate intervention does not scale well, since the larger the number of peers, the greater the delay in re-establishing communications. The next two columns refer to the first message sent by a peer to the migrated process. ‘First message’ refers to the number of additional messages required by the specific technique when the peer process attempts to send its first

790

L. Hughes/Computer Communications 21 (1998) 781–792

Table 1 Summary of additional network traffic (in packets) for each category Migration-time

First message

Hosts involved

Nth message

Immediate intervention

((H to P) þ Q) 3 2

0

0

0

Demand intervention: Well-known object Search for process

K 3 2 or K þ 1 0

4 to R þ 2 4 to R þ 2

K All

0 0

Forwarding: Without short-cutting With short-cutting Concealed

0 0 0

F32 F32þ1 0

F 1 0

F32 0 0

Where: F—number of forwarding objects or hosts (whichever is greater); H—number of hosts supporting peers; K—number of well—known objects; P— number of peers; Q—number of messages that arrived during migration and must be forwarded (or 1 if only an indication is sent); R—number of retransmissions without acknowledgement.

message after a migration (demand intervention and forwarding only); delays can occur in both demand intervention and forwarding. ‘Hosts involved’ is the number of hosts that are required to determine the location of the process (demand intervention) or forwarding the message (forwarding). In demand intervention, delay is not a function of the number of hosts involved in determining the location of the migrated process; however, in forwarding, delay is a function of the number of hosts. As a result, forwarding without shortcutting does not scale well. The last column lists the additional network traffic for the Nth message, assuming that the process has not undergone a subsequent migration. Only in a forwarding environment where short-cutting is not employed is the effect of the migration still felt, since each message must pass through all members of the forwarding chain.

6. Related issues 6.1. Network media Not surprisingly, the network media can influence the possible choice of migration technique adopted by a system. All of the techniques have been implemented with broadcast media such as the Ethernet. However, the requirements for non-broadcast media are such that not all techniques will work. Immediate intervention, demand intervention (using a well-known object), and forwarding can all function in a non-broadcast media, since the destination is always a specific host. Demand intervention (using broadcast) cannot operate in a non-broadcast media since the intervention message will not be delivered to all possible destinations. This could be overcome, at some expense, by implementing broadcast as repeated point-to-point transmissions. Concealed exhibits similar problems, since the destination address contains no routing information.

6.2. Internetwork migration Internetwork migration is the migration of a process between networks. As with a migration on a single network, both the emigrant and immigrant hosts must support a common architecture, process naming scheme, and migration algorithm. Most issues associated with maintaining communications within a network after migration also hold true for communications between networks; differences and limitations are discussed in this section. An argument for supporting internetwork migration is that it allows a process to be brought closer to its peers, while a limitations is the speed at which the migration can take place. Security can also be an issue such as when a process emigrates to an untrusted network (or host) or when a host accepts an untrusted process. Regardless of the migration category adopted, communications with peers on remote networks will take place through some form of bridge or gateway. Peers’ packets sent from remote networks must be recognized by the gateway and routed so that they reach the migrated process’ network. The time taken to update the peers’ kernels can be a limiting factor in the immediate intervention category if ppers reside on remote networks; the availability of any intervening networks may also be an issue. On the other hand, migrating a process closer to its peers can potentially reduce the updating time. Demand intervention using a well-known object requires one of the hosts involved in the migration to inform the well-known object of the process’ new address. If peers reside on networks other than that of the well-known object (or a surrogate), the time needed to communicate with the well-known object may be an issue, as may the availability of any intervening networks. Sending an intervention message in demand intervention using a broadcast search makes it is necessary for the message to be transmitted on all potential networks in an attempt to reach the new execution host. Every host on the internetwork is therefore open to interruption. If many

L. Hughes/Computer Communications 21 (1998) 781–792

migrations occur simultaneously, the entire internetwork could be subject to broadcast storms. Communications maintained by forwarding can be influenced by the following factors. First, a forwarding object will be left on the migrated process’ emigrant network; if a process migrates to its peer’s network, the peer’s messages will be routed to the remote forwarding object and then back to the peer’s network. To reduce this unnecessary network traffic, short-cutting is essential [17]. Second, any outdated forwarding objects left on remote network(s) must be removed through the application of internetwork garbage collection, thereby creating additional network traffic. The failure of intervening routers may result in forwarding objects remaining active for long periods. Concealed migration between networks raises two issues that potentially limit the number of networks that may be interconnected. First, all networks must share a common hardware addressing scheme. Second, ‘intelligent’ bridges must be employed to accept packets for those processes (migrated or otherwise) that can be reached through a given bridge. To obtain the shortest path to a process through a network, it is necessary for the bridges to exchange information on the location of each process. When a migration occurs, the emigrant migration manager must inform the bridges to which it is connected.

7. Concluding remarks Transparency is a common objective met by many systems supporting process migration; one of its most apparent manifestations is in interprocess communications. Communication transparency after a migration is often achieved at a cost, notably additional network traffic, and increased state information and complexity. These costs can be influenced by the number of peers, the lifespan of the migrated process, the number of messages sent after a migration, and the effect of multiple migrations of the same process. The taxonomy developed in this paper classifies migration algorithms, in terms of their support for interprocess commnunications, into one of: 1. Intervention, whereby the peers’ communication systems are involved in the migration. This can be immediate, in which the emigration communication system informs the peers’ communication systems of the new execution address, or demand, when a peer’s communication system requests the location of the migrated process after a communication with the process has failed. These approaches will generate additional network traffic and may require additional communication system state information, complexity, or both. 2. Forwarding employs one or more intervening structures that are responsible for maintaining the communication. Messages are transmitted by peers to the process’ home host; from there, they are passed to the next forwarding

791

agent in the chain, eventually arriving at the migrated process. Various optimizations are possible, such as having a single forwarding agent maintained on the home host, or having the forwarding agents send address update messages back to the communication systems that have sent the messages. Depending upon the implementation, additional network traffic will always be generated; increased state information, complexity, or both may be required as well. 3. Concealed, in which communications are maintained without the generation of additional network traffic or the need increased state information. The choice of migration algorithm also determines when and for how long communication delays will occur. In immediate intervention, the delay occurs as part of the migration, in that communications will not resume until all peers have been informed; after this, the peers can communicate directly with the migrated process. However, in demand intervention, communications are not affected by the migration until a peer is unable to communicate with the migrated process; at this point, special protocols are required to determine the location of the process. Without the use of shortcutting, forwarding will always introduce delays; the greater the number of forwarding agents, the longer the delay. Concealed is not subject to delays, since the same identifier is used throughout the communication and the peers are not informed of the migration. Finally, the overheads associated with each category can be attributed, in part, to process naming and to the technique by which the communication system supports it. Naming schemes that bind a process to its current execution host require some form of message redirection should the process migrate since the process’ identifier is no longer valid. In intervention, message redirection takes place within the peers’ communication systems, requiring the peers to maintain up-to-date addresses; whereas in forwarding, the peers can continue to send messages to the process’ home host, since message redirection is performed by forwarding agents. However, by employing direct process naming using a combination of programmable network hardware and global, location independent, identifiers not bound to a specific host permits the development of process migration algorithms that are concealed from communications. Since concealed algorithms cause no additional network traffic and do not require increased state information, this approach offers benefits to both and real-time systems that support migration.

Acknowledgements The author wishes to thank the two anonymous reviewers who made a number of useful suggestions; Professor Brian Randell and the Department of Computing Science at the University of Newcastle-upon-Tyne for allowing him

792

L. Hughes/Computer Communications 21 (1998) 781–792

access to research facilities during his 1995–96 sabbatical leave; Santosh Shrivastiva and Steve Caughey of the Department for several interesting conversations on object migration; and Sandy Cook for many discussions on Lego in general and the taxonomy in particular.This research is funded by a research grant from the Natural Sciences and Engineering Research Council of Canada.

References [1] J. Smith, A survey of process migration mechanisms, ACM Operating Systems Review 22 (3) (1988) 28–40. [2] F. Douglis, J. Ousterhout, Transparent process migration: design alternatives and the sprite implementation, Software Practice and Experience 21 (8) (1991) 757–785. [3] L. Hughes, G. Stoddart, Fault-tolerant process tracking in the Lego kernel, Microprocessors and Microsystems 19 (8) (1995) 379–475. [4] F. Douglis, J. Ousterhout, Process migration in sprite: a status report, IEEE Technical Committee on Operating Systems Newsletter 3 (1) (1989) 8–10. [5] E. Zayas, Attacking the process migration bottleneck, in: Proceedings of the 11th ACM Symposium on Operating Systems Principles, 1987, pp. 13–22. [6] D. Eager, E. Lazowska, J. Zahorjan. The limited performance benefits of migrating active processes for load sharing, in: ACM Sigmetrics 1988. [7] M. Campione, K. Walrath. The Java Tutorial (Object-oriented Programming for the Internet). Addison-Wesley, Reading, MA, 1997. [8] E. Harold. The Java Developer’s Resource. Prentice Hall, Hemel Hemstead, 1996. [9] H. Peine, T. Stolpmann. The architecture of the ara platform for mobile agents, in: LNCS 1219, Springer, Berlin, 1997, pp. 50–61. [10] D. Butterfield, R. Matthews. Remote tasking, in: G. Popek, B. Walker (Eds.), The LOCUS Distributed System Architecture, chap. MIT Press, Michigan, 1985, pp. 73–89. [11] L. Hughes, Identifying migrated objects using multicast addresses, Computer Communications 14 (7) (1991) 423–431.

[12] L. Hughes. Event handling in the Lego system, Software Practice and Experience 27 (3) March 1997. [13] Y. Artsy, R. Finkel, Designing a process migration facility: the Charlotte experience, IEEE Computer 22 (9) (1989) 47–56. [14] A. Barak, S. Guday, R.G. Wheeler. The MOSIX Distibuted Operating System, Springer, Berlin, 1993. [15] D.R. Cheriton, The V distributed system, Communications of the ACM 31 (3) (1988) 314–333. [16] H. Clark, B. McMillin, DAWGS — a distributed compute server utilizing idle workstations, Journal of Parallel and Distributed Computing 14 (1992) 175–186. [17] M. Shaprio, P. Dickman, D. Plainfosse, Robust Distributed References and Acyclic Garbage Collection, in: Proceedings of the 11th ACM Symposium on Principles of Distributed Computing, 1992, pp. 135–146. [18] D. Ingham, S. Caughey, M. Little. Fixing the ‘broken link’ problem: the W3Objects approach, in: Proceedings of the Fifth nternational World Wide Web Conference, Paris, 1996. [19] R. Draves et al. Microkernel operating system architecture and Mach, in: USENIX Workshop Proceedings: Micro-kernels and Other Kernel Architectures, 1992. [20] L. Hughes, Object identification in the Lego kernel, Software Practice and Experience 23 (4) (1993) 405–418. [21] E. Lazowska et al. The architecture of the Eden system, in: Proceedings of the 8th ACM Symposium on Operating Systems Principles, 1981, pp. 148–159. [22] R. Fowler. The complexity of using forwarding addresses for decentralized object finding, in: Proceedings of the 5th ACM Symposium on Principles of Distributed Computing, 1986, 108–120. [23] A. Tanenbaum et al., Experiences with the Amoeba distributed operating system, Communications of the ACM 33 (12) (1990) 46–63. [24] M. Kaashoek et al, FLIP: an internetwork protocol for supporting distributed systems, A CM Transactions on Computer Systems 11 (1) (1993) 73–106. [25] M. Rozier, V. Abrossimov, F. Armand, et al. Overview of the Chorus distributed operating system, in: USENIX Workshop Proceedings: Micro-kernels and Other Kernel Architectures, 1992, 39–69. [26] M. Shapiro et al, SOS: an object-oriented operating system — assessment and perspectives, Computing Systems 2 (4) (1989) 287–337. [27] M. O’Conner, B. Tangney et al., Micro-kernel support for migration, Distributed Systems Engineering 1 (1994) 212–223.