- Email: [email protected]
Global conditions in debugging distributed programs
JOURNAL
OF PARALLEL
AND
DISTRIBUTED
COMPUTING
Global Conditions
15,
Software
Laboratories,
(1992)
in Debugging
YOSHIFUMI NTT
62-69
MANABE 3-9-11
AND MAKOTO
Midori-cho,
0 l!w
Academic PW,
IX.
1. INTRODUCTION Debugging distributed programs is considerably more difficult than debugging sequential programs. This is be-
cause the execution behavior in response to a fixed input may be indeterminate, with the results depending on a particular resolution of race conditions existing among processes. The nondeterminism of the distributed system prevents some distributed program debugging packages from recreating the event sequences that precede errors. If a distributed program contains neither nondeterministic statements such as asynchronous interrupts nor time-dependent statements such as reading a clock, its execution can be replayed according to a small log kept during execution. The replay technique discussed in [l, 9, 121 is based on the above premise. In the monitoring phase, the communication order-that is, the order of input events such as receiving messages and referencing shared memory-is recorded for each process. In the replay phase, this mechanism uses the communication order log to ensure that each process reads the inputs in the same order as in the monitoring phase. This paper discusses various functions for a debugger based on ‘this replay technique. One of the fundamental facilities for debuggers is setting breakpoints and halting [61. To set breakpoints, many sequential program debuggers introduce logical 62 0743-73 15192 $3.00
Copyright All rights
0 1992 by Academic Press. Inc. of reproduction in any form reserved.
Musashino-shi,
Programs
IMASE Tokyo
180, Japan
predicates written at the source-code level to describe relationships among components of the current program state (e.g., J < 10 and A(Z) = M). When the specified predicate becomes true, execution is stopped and control is transferred to the programmer’s terminal. Most distributed program debuggers restrict the predicate to an expression for one process [4, 91. Distributed processes may, however, generate many kinds of errors in which the overall system is in error but each individual process is not. To cope with such errors, global predicates, which are expressions related to multiple processes, should be introduced. Two kinds of global predicates, Conjunctive Predicates and Disjunctive Predicates, are introduced for non-replay debuggers in [7, 1 I]. This paper considers the problem of halting processes by a replay-based debugger for Conjunctive Predicate and Disjunctive Predicate breakpoint conditions. The algorithm in [Ill restricts the global predicate to a kind of predicate called Linked Predicates. The algorithm in [7] has the problem that there is a delay in the detection of predicate satisfaction. When a process detecting some breakpoint condition is satisfied, conventional debuggers halt the other processes: either the other processes halt when they receive a broadcast message signaling the detection [4, 7, 1 I], or execution continues until the halted process blocks them [9]. These methods allow the other processes some extra execution after the breakpoint condition is satisfied. Since processes other than the detected process may also have caused the error, this extra execution might hide the real cause. Using the replay technique, the same result is produced even if execution of processes is delayed in the replay. We can thus control the replay and get the exact state when the condition is satisfied. This paper defines the “first” global state that satisfies a given global condition and considers the problem of halting at this global state. For a Conjunctive Predicate, an algorithm to halt at the first global state satisfying the predicate is given. For a Disjunctive Predicate, it is shown that it is impossible to halt at the first global state, but that it is possible to halt at some global state satisfying the predicate. Another fundamental facility is selective tracing. We introduce the global predicates, Conjunctive Predicate
This paper describes algorithms for a distributed program debugger based on a replay technique. Halting at breakpoints and selective tracing are its fundamental features. In distributed systems, a given breakpoint or trace condition does not uniquely define the global state at which to halt or trace, because of the asynchrony of processesand communications. This paper therefore proposes the “firsf” global state Znf(P) to be the best global state at which to halt or trace, for a given condition P. Two kinds of global conditions related to plural processes,Conjunctive Predicates and Disjunctive Predicates, are considered. The authors present an algorithm that halts processes at Znf(P) for a given Conjunctive Predicate P. It is also shown that, for a Disjunctive Predicate P, it is impossible to halt at Znf(P), but possible to halt at some state which satisfiesP. An algorithm is also provided for selective tracing when a Conjunctive or Disjunctive Predicate se-
lection condition is &WI.
Distributed
GLOBAL
CONDITIONS
IN DEBUGGING
and Disjunctive Predicate, for the selection condition, and we show an algorithm for each selective trace. Section 2, after presenting models for a distributed system, explains a model of a debugger based on a replay technique that uses the communication order log. Section 3 discusses halting by breakpoints. Section 4 presents the selective trace algorithm. Section 5 presents conclusions and proposes further studies.
2.
2.1. Distributed
MODEL
DEFINITION
System Model
This paper assumes that values exchanged between processes depend only on the initial values in each process and the order in which processes communicate. That is, processes are assumed to be deterministic. Stated somewhat differently, there are no nondeterministic statements, such as asynchronous interrupts, and there are no time-dependent statements, such as reading a clock or timing out from a semaphore. The distributed system execution model, based on message-passing communication, is the same as that proposed by Lamport [8]. A distributed system consists of a finite set of processes and a finite set of channels. Channels are assumed to have infinite buffers, to be error-free, and to be FIFO. The delay experienced by a message in a channel is arbitrary but finite. Sending a message and receiving a message are considered special events and are called a send event and a receive event. The other events are called internal events. The initial state of each process is also considered to be an internal event. We can then define the “happened before” relation [8], denoted by “<,” as follows (In [S], -+ is used rather than <.) DEFINITION 1. [8] The relation “<” on the set of events of a system is the minimum relation satisfying the following three conditions: (1) If a and b are events in the same process, and a comes before b, then a < b. (2) If a is the sending of a message by one process and b is the receipt of the same message by another process, then a < b. (3) If a < b and b < c then a < c. For two events a and b, a 5 b if a < b or a = 6.
In some distributed systems, processes communicate via shared memory [9]. Such systems can be simulated as follows by a message-passing system. Each memory object is considered to be a process. Writing to a memory object is converted into a message from the writer process to the memory-object process. Reading from the shared memory is converted into two messages, a read request from the reader process to the memory-object process and a reply from the memory-object process to the reader process. This paper refers only to a message-
DISTRIBUTED
63
PROGRAMS
passing system, but the results are also applicable to systems communicating through shared memory. Next, for the proof of the later lemmas, Chandy and Lamport’s “meaningful global state” [2] of the distributed system is formally defined using the “happened before” relation. DEFINITION 2. Let Ej be the set of events in process i. An N-tuple of events of processes s = (e, , ez, . . . . eN) (ei E E;) is said to be a global state if and only if for all ei E Ei, ei > ei implies el & for any j(1 5 j 5 N). Let U be the set of all the global states. l?j
(In [2], the global state is defined by a set of event sequences. Our definition is equivalent to that one.) Global state s = (ei , e2, . . . . ehi) is intuitively considered as a set of concurrently occurring events for some timing occurrence, and we consider it as the state when each process i has just finished the execution of e;. The “happened before” relation for global states is defined as follows. DEFINITION 3. For two global states s = (ei , e2, . . . . eN) and s’ = (e;, e;, . . . . eh), s I, s’ if and only if ei 5 e,! for every i (1 5 i 5 N), and s < s’ if and only ifs 5 s’ and < ei for some j(1 I j 5 N). ej
2.2. Debugger
Model
This section gives a general model for a replay-based debugger. A replay-based debugger D consists of one main debugger MD and local debuggers LD; corresponding to the processes (Fig. 1). Every pair of MD and LD~ has communication channels in both directions. LD; halts process i, lets it execute events, and tests its status. MD sends commands to LD~, receives it replies, and displays the results to the user. D executes the programs iteratively. The first execution is called a monitoring phase and the others are called replay phases. In the monitoring phase, each LDi independently stores the communication history of the process locally. The
v user FIG.
1. A replay-based
debugger.
MANABE
64
communication history is a sequence of items corresponding to receive events. The item is process number or a Null symbol. When the process receives a message from process j in the c&h receive event, the ath item of the history isj. Null means that no message was received in this receive event. After the monitor phase, debugger commands such as “stop if P” are given to h4D. In the replay phase, message receiving is controlled by LDi so that the order of message arrival at the process is the same as in the original execution. The processes are deterministic, so the execution behavior of this replay is the same as that of the original execution. In the replay phase, LD~ controls process i with the following operations that sequential program debuggers offer: DEFINITION
4.
LD~ can do the following
AND IMASE Process 1
2
3
1 Time
4:
message-passing
si
: initial
e:i
event
FIG. 2.
between
processes
state
An execution in a 3-process distributed system
operations to
process i in the replay phase. 1. See the type of event to be executed next, where the type may be “receive event, ” “send event,” or “internal event.” 2. Buffer the messages sent to i and see the messages sent from i. 3. Execute the next event and half the process (and also specify which message should be read if the executed event is a receive event). 4. Examine the current state of the process. 3. HALTING
AT BREAKPOINTS
This section discusses how to set breakpoints and halt the system in the replay phase. First, predicates are defined to specify breakpoints. 3.1. Global Predicates Sequential program debuggers typically use simple predicates. These can be expressed in terms of the Boolean values of variables in the program and the program counter. The simple predicate is instantaneous in the sense that its validity does not depend on the execution history, but rather on the state at an instant. The restriction is that a simple predicate is based on the state local to a single process. Let the simple predicate for process i be SP;. The value of SPi right after the execution of event e is denoted by SP;(e). Distributed processes may generate errors in which the overall system is in error but each individual process is not. Consider the following example (Fig. 2). Process i (i = 1, 2) has a variable vi. The value of vi just after the execution of each event is also shown in Fig. 2. Suppose that Vi = 1 means i has the right to a write operation for a database item. In order to keep the database correct, two processes cannot have the right to the write operation at
the same time. That is, u1 = 1 and v2 = 1 at the same time is an error. To detect this kind of error, global predicates-that is, predicates related to plural processesmust be considered. Two kinds of global predicates are introduced in [ 11I. One is the Disjunctive Predicate (DP), which consists of simple predicates joined by disjunctive operators “U.” The other is the Conjunctive Predicate (CP) consisting of simple predicates and conjunctive operators “0.” Note that global predicates are functions mapping from I/ to {true, false}. Miller and Choi [l 11 proposed an algorithm to halt the processes for a non-replay debugger. For a given global predicate, their algorithm halts the processes when a “Linked Predicate” is satisfied. A Linked Predicate is of the form SP; < SPj < SPk < . . . . where “<” is the “happened before” relation. To halt at SP, fl SP2 17 1.. n SP, , it is necessary to produce n! Linked Predicates for different orders of predicates; otherwise the debugger can fail to halt. Haban and Weigel [7] proposed another algorithm to halt the processes for a non-replay debugger. Their algorithm considers general Disjunctive and Conjunctive predicates, and is not restricted to the Linked Predicates. However, it is impossible for their algorithm to halt the processes just when the predicate is satisfied. If the process execution is suspended and some process gathers the status of all processes, this kind of predicate satisfaction can be detected exactly when it is satisfied. If a nonreplay debugger delays the process execution, however, its behavior might be different from that without the debugger. Since a replay debugger produces the same behavior even if some process executions are delayed, we can control the execution and detect this kind of predicate satisfaction. We consider the problem of halting for a given CP and DP by controlling the process execution.
65
GLOBALCONDITIONSINDEBUGGINGDISTRIBUTEDPROGRAMS
Another aspect of distributed program debuggers is that the global state for a given global condition is not unique. When a global predicate P is given to specify a breakpoint as the command “stop if P,” there are many global states satisfying P. Let G(P) be {s E U 1 P(s) = true}. At which global state in G(P) should the debugger halt the system? Consider again the example in Fig. 2 and the condition P = (ur = 1 fl u2 = 1). This predicate has no condition concerning process 3. Where should we halt process 3? When a condition is satisfied, conventional debuggers broadcast a message to the other processes to halt [4, 7, 111 or wait until the halted process blocks the execution of the other processes [9]. The cause for P becoming true, however, is contained not only in process 1 or 2, but also in process 3. The message from process 3 might cause P to be true. In that case, if process 3 executes beyond the send event e:, its contribution to the cause might be destroyed and the real reason for the error might be hidden (consider, e.g., the case that the event e: clears all the variables used for the decision to send a message to process 1). It would therefore be best to stop the processes at the “first” global state in G(P). That is, the processes should halt at the state in which each process has executed the minimum requirements for making P true. We define the “first” global state for P and consider the problem of halting the processes at this global state. The “first” global state in G(P) can be defined as follows. DEFINITION 5. For a predicate P, Znf(P) = {s 1 s E G(P) and s’ G G(P) for any s’ E U
such that s’ < s}. For the example in Fig. 2 and CP P, = (u, = 1 fl u? = I), Znf(Pl) = {(e:, e:, e:)}. for a DP P2 = (u, = 1 U u2 = l), hf(P~) = {e:, s?, e:), (s’, e:, s3)}. Note that Znf(P) may include plural global states. Sections 3.2 and 3.3 show that it is possible to stop the processes at s E Znf(P) when P is a CP and at s E G(P) when P is a DP.
have arrived because its sender S is passive or because the execution of S is delayed. If M has not arrived, the receiver R sends a control message to ask S to send M. Then S becomes active executes the program to send M. Process S remains active and continues execution until the predicate becomes true after sending M. In other words, a process is active when its predicate is false or when it must send a message. In a control message, it is necessary to specify which message S must send. Thus the control message contains an identifying message number, which is a sequence number of the messages going through the channel from S to R. LD; counts the messages sent to or received from every channel. Note that it is unnecessary to send the message number attached to the message. We should be able to detect when every process is passive and will not become active. This is one variation of the distributed termination detection problem proposed by Dijkstra and Scholten [5], and many algorithms have been proposed for different assumptions regarding the system. The part that detects the termination is called a termination detector. Status changes are reported to the termination detector when they occur. To simplify termination detection, we assume that control messages go through the termination detector. The termination detector can be implemented either in a distributed fashion (in LDi) or in a centralized fashion (in MD). The distributed algorithm shown in [3] can be used for this termination detector. The outline of the halt algorithm [IO] is shown in Fig. 3. Proof of the correctness of this algorithm is beyond the scope of this paper. It is shown only that the above algorithm stops the processes at some s E inf(P). In preparation, a definition and two lemmas are presented. DEFINITION 6. For two events e and e’ in one process, let min(e, e’) be e if e 5 e’, and e’ otherwise. For twoglobalstatess = (er,ez, . . . . eN)ands’ = (e;,e;, . . . . eh), min(s, s’) is defined as min(s, s’) = (min(e,,
3.2. Halting for Conjunctive
e;), min(ez, e;), . . . . min(eN, eh)).
Predicate
This section proposes an algorithm which stops the processes at Znf(P) when P is a CP. For each process, we introduce “active” and “passive” states. If the process is being executed, it is called active. A passive process becomes active only when another active process make the process to be active. System halting means that all processes are passive. Suppose processes without a simple predicate have a special predicate which is always true. Initially, processes whose simple predicate is false are active and the other processes are passive. Suppose an active process tries to receive a message M. The message M may not
LEMMA 1. Zf s, s’ E U, min(s, s’) E U. Proof. Assume that min(s, s’) = (e;‘, e;‘, . . . . e;) 6 U. From the definition of U, there is a pair i and j such that 3e::” E Ei, ei” > ei’ and e:l’ < ey. Since el’ = min(ei, e,!), e:’ = e; or e:’ = e,!. Without loss of generality, it can be assumed that e:’ = ei. Then
ey > ey = ei and which contradicts LEMMA 2.
ey < er = min(ej, e,‘) 5 ej,
s E U.
n
ZfP is a CP IZnf(P)I 5 1.
66
MANABE
FIG. 3.
Halting
algorithm
for a Conjunctive
IMASE
Next we show the latter proposition. Process i executes the next event if and only if one of the following conditions is satisfied:
program HaltAtBreakpoint /* Program for LD~. */ function TermTest; if s[;] > ms[;] for all i and SP = true /* If SP does not exist for this process, SP = true */ then return (passive) else return (active) end; function TryEzec; /* let e be the next event */ if e is receive from i and there is no program message from i in local queue then begin Send control message ‘%[;I + 1” to i; return(i) end else begin Execute e; return(NzLII) end end; begin /* MAIN */ for i := 1 to N do begiu s[i] := 0; /* The number of messages sent to i */ r[i] := 0; /* The number of messages received from i */ ms[;] := 0 /* i requested to send up to ms[;] */ end; cstat := TermTest; Send cstat to TerminationDetector; if cstat =active then waitp := TryErec; When execution of event e is finished: begin if e = send to i then s[i] := s[i] + 1; if e = receive from i then r[i] := r[;] + 1; cstat = TermTest; if cstat =active then waitp := TryExec else Send cstat to TerminationDetector; end; When program message m is arrived from i: begin Insert 772 to local queue; if waitp = i then waitp := TryEzec end; When control message “k” is arrived from i : begin ms[;] := k; bstat := cstat; cstat := TermTest; if bstat =passive and cstat =active then begin Send cstat to TerminationDetector; waitp := TryEzec end; When Termination is detected: Halt; end.
AND
i has an SPi and SPi = false. i received a control message requesting and has not yet sent the message. l l
a message
Suppose that the system halts at s such that some process i executes beyond t;. Let s’ be a global state during the replay such that s’ = (s,, s?, . . . . sN) 5 inf, and for some process i which executes beyond ti in s, si = tj in s’. Let Si,f be the set of processes which satisfy si = tj. No process in Sinf executes the next event by the first of above conditions, because si = ti. Thus, they do not execute the next events unless some processj fZ S’infsends a control message to a process k E Sia, before it executes tj, to ask for a message which has not been sent at th . Let the send and receive events of the requested message be ek and ej. Thus, eh > tk and eh < ej < tj holds, which contradicts inf E U. Therefore, no process executes beyond t;. 3.3. Halting for Disjunctive
Predicate
When the given predicate is a DP, it is impossible for D to stop the processes at s E Znf(P) during the first execution in the replay phase. end
Predicate.
Proof. Let the predicate that P be SP;, f! SP;, f’ . . . n SP;, . Assume two global states s and s’ (s # s’) are both members of Znf(P). If s < s’, then s’ 6? Znf(P), which implies s $ s’. Thus, min(s, s’) < s. P(min(s, s’)) = SPi,(WZin(ei,, e(J) n SPj,(WZin(ei,, eh)) n .. . n SP,(WIin(ej,, el)) = true. min(s, s’) E U from the above lemma. Thus, min(s, s’) E G(P). That min(s, s’) < s and min(s, s’) E G(P) contradicts s E Zrzf(P). Let inf= (t, , t2, . . . . t,) be the global state in Znf(P). To show that the system halts at inf, the following properties must be demonstrated. The processes do not terminate at any s E U such that s < inf. Process i does not execute beyond ti. l
l
Assume that the system halts at some s E U. Every SPi is true at s. Thus s is contained in G(P). Ifs < inf, the definition of Znf(P) is contradicted. The former proposition is therefore true.
THEOREM 1. There is no debugger D that detects s E Znf(P) without executing beyond s when P is a DP.
Proof. Let us consider the following example. The number of processes n is 2, and P = SP, U SP2. In the initial state, processes 1 and 2 do not satisfy SP, and SP2. Processes 1 and 2 never communicate with each other. Assume that D lets process 1 move one step from the initial state. In this case, it may occur that process 1 never satisfies SP, and process 2 satisfies SP2 after executing some events. Thus, D cannot stop at Znf(P). A similar situation may occur if D lets process 2 move one step. Therefore, the proposition is valid. It is, however, possible to halt at some s E G(P) and a halt algorithm is shown in [I 11. Note that if replay is executed twice, D can halt at a global state s E Znf(P) as follows. During the first replay, apply the algorithm in [ll] for P. Thus, the processes halt at some so E G(P). Select a simple predicate SPi in P which is satisfied at sO. During the second replay, apply the halt algorithm in Fig. 3 to Conjunctive Predicate SPi. Thus, the processes halt at s’ E Znf(SP;) c Znf(P). 3.4. Channel States in the Global Predicates The former subsections considered only the process states. In this subsection, we introduce channel state predicates in the Global Predicates. Channel state predicates can be converted into process predicates.
GLOBAL
CONDITIONS
IN DEBUGGING
Let the FIFO channel from process i to process j be Ci,j. We consider two types of channel state predicates for Ci.j Message content predicate: for a message m, m E Ci,j or m G Ci,j* Message number predicate: for a constant a, ICi,jI = (>, 2, <, 5, etc.)a. l
l
A message content predicate is converted into a process predicate as follows. Let the Conjunctive Predicate with the channel predicate P be SPi, II SPi, n . . . n SP, fl CSPi{,j$ n CSPi$,j$ n ... n CSPi/,j/ . The conversion is similar when the predicate is a Disjunctive Predicate. Let the predicate CSPi, = (m E Ci,j). Initially, check whether m is in Ci,j or not.
1. While m G C;,j Change the predicates for processes i and j to “false” and SPj, respectively (predicates for the other processes are unchanged) and execute the algorithm in Fig. 3. In addition, go to case 2 if i executes “send m to j”. 2. While m E Ci,j Let m be the xth message from i toj. The value x can be gotten from the variable s[j] in the algorithm in Fig. 3. Change the predicates for processes i and j to SP; and SPj n “r[i] < X,” respectively, and execute the algorithm in Fig. 3. In addition, check ifj executes “receive m from i” or i executes “send m to j” (sending m again). In the former case, go to case 1; in the later case, change the term “r[i] < x” for processj to “r[i] < y” (if m is the yth message from i to j). The case that CSPi,j = (m 65 Ci,j) is similar to this case. Next consider a message number predicate. Let the predicate CSPi,j = (IC;,jJ 2 a). Initially, let s[j] in process i be x and r[i] in process j be y. 1Ci,j 12 a if and only if x y 2 a. 1. While ICi,jI < u Change the predicates for processes i and j to “false” and SPj, respectively (predicates for the other processes are unchanged) and execute the algorithm in Fig. 3. In addition, check if i executes “send toj” or j executes “receive from i”. In the former case, increment x and if x - y 2 a, go to case 2. In the latter case, increment y and continue. 2. While ]Ci,j] 2 a Change the predicates for processes i and j to SPi, and SPj n “r[i] I n - a,” respectively, and execute the algorithm in Fig. 3. In addition, check if i executes “send toj” orj executes “receive from i.” In the former case, increment x and change the term “r[i] 5 x - a” for processj to “r[i] I x’ - a” (x’ is the new value of x) and continue. In the latter case, increment y and if y > x - a,
DISTRIBUTED
67
PROGRAMS
go to case 1. The case that ICSPi.j( = (>, 5 etc.)a is similar to this case. 4. SELECTIVE
Many sequential tive trace facility:
TRACE
debuggers have the following
truce [condition]
selec-
print [expression].
This command means that the debugger displays the value of [expression] whenever [condition] is satisfied. The expression consists of some variables, the program counter, and arithmetic and logical operators. In the distributed debugger, selective trace is considered to display the value of the global expression when the global condition is satisfied. The global condition is given by the global predicate, and the global expression consists of plural process variables and program counters. When the predicate P is a CP, selective trace can be implemented with the halt algorithm. When the processes halt at Znf(P), the debugger calculates the expression value at the global state and prints it out. When the printing is finished, the debugger returns each process state to its initial value, “active” or “passive,” and executes the programs again (under the control of halt algorithm). As shown in Section 3.3, it is impossible to halt at Znf(P) when P is a DP. To print out the expression value, however, it is only necessary to save the values of the variables used in the expression and afterwards calculate the expression value when Znf(P) is detected. When the calculation is finished, the saved values become useless and can be discarded. The trace algorithm is outlined below, focusing on the data saving and discarding mechanisms. The trace algorithm in LDi is implemented as follows. Let P be SPj, U SP,;! U . . . U SPjk. Let ei be the current event of process i at some instant. The first global state for ei, denoted by Znf[ei], is defined as Z$(si = eJ; that is, the first global state s = (si , ~2, . . . . SN) such that si = ei. LDi prepares virtual program counters in&[ 1 . . . N]. LD; i&2], . .. . maintains inJ[l . . . N] such that (i&[l], infi[N]) = Znf[ei] when i has just finished event ei. Note that these virtual program counters do not have to simulate real program counters but need only count the number of events that occur. The maintenance mechanism is shown later. When SPi becomes true in process i, (inJ[ 11, infi[2], . ..) inJ[N]) = ZfZf(SPi). Let process i’s variables in the [expression] of the trace command be xf , xi, . . . , xi,. When process i is in the initial state or has just finished a send event, i stores the current values of xi, xi, . . . , xi, together with the current value of infi[i] locally. Note that value storing is unnecessary after receive or internal events, because such an
68
MANABEANDIMASE
program const
SelectiveTrace; /* Program for I,DI. */ VaTlist = . . /* list of variables for selective trace */ I = . . . /* number of this process */ function SPExist; /* true if simple predicate exists for this process */ end; function TermTest; /* same as in RaltAtBreakPoint */ function TTyExec; begin/* let e be the next event *J if e is receive from i then if there is no program message from i in local queue then begin Send control message ‘%[i] + 1” to i; return(i) end else begin Execute e; return(Nzlll) end else if e is se,ld m to i then begin ;nf[l] := ;nf[l] + 1; Send [m, inf] to i; return(Null) end else /* internal event */ begin Execute e; return(Nhrllll) end end; begin /* MAIN */ for i := 1 to N do begin s[i] := 0; r[i] := 0; ms[;] := 0; ;nf[;] := 0 end; Get Values of Varlist and store [Values, inf[I]]; cstat := TermTest; if cstat =passive and (SPEzist) then Send output initialize message [inf, V&ties] to MD; if cstat =active or (SPExist) then waitp := TryExec;
When execution of event e is finished: begin if e = receive from i then begin /* let the piggybacked inf message be rinf */ for j := 1 to N do if r;nf[j] > inf[j] then begin inf[j] := rinf[j]; if (SPExist) then Send remove request “inflj]” to j end; ~[i] := r[i] + 1 end if e = dend to i then begin s[i] := s[;] + 1; Get Values of Varlist and store [Valaes, ;~~f[l]] end; cstat := TermTest; if cstat =passive and (SPExist) thenbegin Get Values of Varlist; Send output initialize message [;nf, ValzLes] to MD end; if &at =active or (SPExist) then waitp = TryExec end; When program message [m, ~inf] is arrived from i : begin lnsert [m, rinf to local queue; if waitp = i then waitp := TryExec end; When control message “I;” is arrived from i: begin if not (SPExist) then begin ms[;] := k; .bstat := cstat; cstat := TermTest; if bstat =passive and cstat =active then waitp := TryExec end end; When output request message “opt” is arrived from MD : begin Send Values of the record [Values, opt] to MD end; When remove request “TIC” is arrived from i: begin Mark record [Values, k] by i if k < rpc; Remove records which was marked by all i end; end. FIG.
4.
Trace
algorithm
for a Disjunctive
Predicate.
event e; in process i cannot be an element Of ZfcSPj) (j f i). If SPi becomes true, LD; reports to MD that SPi has become true, together with the current value of inA[ 1 . . . N] and xi, xi, . . . . XL,. When MD receives this report, it asks each LD~ to send the values of 9;) Y!, . . . . 2;, at
infi[j]. From the replies, MD calculates the expression value at Z&V;). To maintain inA[ 1 . . . N], at each send event LD~ sends the current value of inJ;[l . . . N] with the message. At a receive event, the receiver (let the receiver be j) receives the value inh[l . . . N] with the message and calculates injJk] = max(inJ;[k], i&[k]) fork = 1,2, . . . . N. It is easily checked that ir$[l], inJ[2], . . . . inJ[N]) = Znf[e;]. If the current value of inJ;[j] is a, the values of x’], x’;, . . . , ,$ when infj [j] < a are no longer used for calculating the expression value for SPi. Thus, whenever the values of inx[j] change, it is reported to LDj. If a value is unnecessary for all predicates, that is, for some U, infJj] 2 a for every i, then the values when inh[j] < u are removed from local storage. The algorithm is outlined in Fig. 4. 5. CONCLUSION
This paper introduced global conditions for some commands of distributed program debuggers. For setting breakpoints and halting, an algorithm is shown to halt at If(P) for a Conjunctive Predicate P. It is also shown to be impossible to halt at Znf(P) for a given Disjunctive Predicate P. A selective trace algorithm is shown for a given (Conjunctive or Disjunctive) trace condition P. For further study, global conditions defined by temporal logic might also be needed. ACKNOWLEDGMENTS The authors thank Drs. Masaru Takesue and Rikio Onai for their guidance and encouragement. They also thank Dr. Naohisa Takahashi for useful discussions, and Drs. Robert Cooper and Terunao Soneoka for their comments on an earlier version of this manuscript.
REFERENCES 1. Carver, R. H., and Tai, K. Reproducible grams based on shared variable. Proc. ence on Distributed Computing Systems. 2. Chandy, K. M., and Lamport, ing global states of distributed tems. 3, l(Feb. 1985), 63-75.
testing of concurrent pro6th International ConferMay 1986, pp. 428-433.
L. Distributed systems, ACM
3. Cohen, S., and Lehmann, D. Dynamic uted termination. Proc. of 2nd ACM Computing. 1982, pp. 29-33.
snapshots: DeterminTrans. Comput. Sys-
systems Symposium
and their distribon Distributed
4. Cooper, R. Pilgrim. A debugger for distributed systems. 7th International Conference on Distributed Computing Sept. 1987, pp. 458-465. 5. Dijkstra, diffusing
E. W., and computations.
6. Fairley, R. E. Softwure York, pp. 288-289.
Scholten, Inform. Engineering
Proc. of Systems.
C. S. Termination detection for Process. Lett. 11, 1(1980), l-4. Concepts.
McGraw-Hill,
New
7. Haban, D. and Weigel, W. Global events and global breakpoints in distributed systems. 21st Hawaii International Conference on System Science. Jan. 1988, pp. 166-175. 8. Lamport, L. Time, clocks, uted system. Comm. ACM.
and the ordering
of events
21, 7(July 1978), 558-565.
in a distrib-
GLOBAL
CONDITIONS
IN DEBUGGING
9. LeBlanc, T. J., and Mellor-Crummey, J. M. Debugging parallel programs with instant replay. IEEE Trans. Comput. C-36, 4tApril 19871, 471-480. IO. Manabe, Y., and Imase, M. Global condition in debugging distributed programs. DEICE Technical Report COMP89-99, Dec. 1989. I I. Miller, B. P., and Choi, J.-D. Breakpoints and halting in distributed programs. 8th International Conference on Distributed Computing Sysrems. June 1988, pp. 316-323. 12. Takahashi, N. Partial replay of parallel programs based on shared objects. ZEICE Technical Report COMP89-98, Dec. 1989. [In Japanese] YOSHIFUMI MANABE received B.E. and M.E. degrees in information engineering from Osaka University, Japan, in 1983 and 1985, Received December 19, 1990; accepted June 25, 1991
DISTRIBUTED
PROGRAMS
69
respectively. In 1985, he joined Nippon Telegraph and Telephone Corporation (NTT) Electrical Communications Laboratories. He is currently a research engineer in NTT Software Laboratories. His research interests are distributed algorithms, communications networks, and graph theory. He is a member of the ACM. MAKOTO IMASE received B.E. and M.E. degrees in information engineering from Osaka University, Japan, in 1975 and 1977, respectively. He received the D.E. degree form Osaka University in 1986. In 1977 he joined Nippon Telegraph and Telephone Corporation (NTT) Electrical Communications Laboratories. He is now a senior research engineer, supervisor in NTT Software Laboratories. He has been engaged in research on distributed switching systems and communications networks. He is a member of the IEEE Computer Society and received the IEICE Young Engineer Award in 1984.
OF PARALLEL
AND
DISTRIBUTED
COMPUTING
Global Conditions
15,
Software
Laboratories,
(1992)
in Debugging
YOSHIFUMI NTT
62-69
MANABE 3-9-11
AND MAKOTO
Midori-cho,
0 l!w
Academic PW,
IX.
1. INTRODUCTION Debugging distributed programs is considerably more difficult than debugging sequential programs. This is be-
cause the execution behavior in response to a fixed input may be indeterminate, with the results depending on a particular resolution of race conditions existing among processes. The nondeterminism of the distributed system prevents some distributed program debugging packages from recreating the event sequences that precede errors. If a distributed program contains neither nondeterministic statements such as asynchronous interrupts nor time-dependent statements such as reading a clock, its execution can be replayed according to a small log kept during execution. The replay technique discussed in [l, 9, 121 is based on the above premise. In the monitoring phase, the communication order-that is, the order of input events such as receiving messages and referencing shared memory-is recorded for each process. In the replay phase, this mechanism uses the communication order log to ensure that each process reads the inputs in the same order as in the monitoring phase. This paper discusses various functions for a debugger based on ‘this replay technique. One of the fundamental facilities for debuggers is setting breakpoints and halting [61. To set breakpoints, many sequential program debuggers introduce logical 62 0743-73 15192 $3.00
Copyright All rights
0 1992 by Academic Press. Inc. of reproduction in any form reserved.
Musashino-shi,
Programs
IMASE Tokyo
180, Japan
predicates written at the source-code level to describe relationships among components of the current program state (e.g., J < 10 and A(Z) = M). When the specified predicate becomes true, execution is stopped and control is transferred to the programmer’s terminal. Most distributed program debuggers restrict the predicate to an expression for one process [4, 91. Distributed processes may, however, generate many kinds of errors in which the overall system is in error but each individual process is not. To cope with such errors, global predicates, which are expressions related to multiple processes, should be introduced. Two kinds of global predicates, Conjunctive Predicates and Disjunctive Predicates, are introduced for non-replay debuggers in [7, 1 I]. This paper considers the problem of halting processes by a replay-based debugger for Conjunctive Predicate and Disjunctive Predicate breakpoint conditions. The algorithm in [Ill restricts the global predicate to a kind of predicate called Linked Predicates. The algorithm in [7] has the problem that there is a delay in the detection of predicate satisfaction. When a process detecting some breakpoint condition is satisfied, conventional debuggers halt the other processes: either the other processes halt when they receive a broadcast message signaling the detection [4, 7, 1 I], or execution continues until the halted process blocks them [9]. These methods allow the other processes some extra execution after the breakpoint condition is satisfied. Since processes other than the detected process may also have caused the error, this extra execution might hide the real cause. Using the replay technique, the same result is produced even if execution of processes is delayed in the replay. We can thus control the replay and get the exact state when the condition is satisfied. This paper defines the “first” global state that satisfies a given global condition and considers the problem of halting at this global state. For a Conjunctive Predicate, an algorithm to halt at the first global state satisfying the predicate is given. For a Disjunctive Predicate, it is shown that it is impossible to halt at the first global state, but that it is possible to halt at some global state satisfying the predicate. Another fundamental facility is selective tracing. We introduce the global predicates, Conjunctive Predicate
This paper describes algorithms for a distributed program debugger based on a replay technique. Halting at breakpoints and selective tracing are its fundamental features. In distributed systems, a given breakpoint or trace condition does not uniquely define the global state at which to halt or trace, because of the asynchrony of processesand communications. This paper therefore proposes the “firsf” global state Znf(P) to be the best global state at which to halt or trace, for a given condition P. Two kinds of global conditions related to plural processes,Conjunctive Predicates and Disjunctive Predicates, are considered. The authors present an algorithm that halts processes at Znf(P) for a given Conjunctive Predicate P. It is also shown that, for a Disjunctive Predicate P, it is impossible to halt at Znf(P), but possible to halt at some state which satisfiesP. An algorithm is also provided for selective tracing when a Conjunctive or Disjunctive Predicate se-
lection condition is &WI.
Distributed
GLOBAL
CONDITIONS
IN DEBUGGING
and Disjunctive Predicate, for the selection condition, and we show an algorithm for each selective trace. Section 2, after presenting models for a distributed system, explains a model of a debugger based on a replay technique that uses the communication order log. Section 3 discusses halting by breakpoints. Section 4 presents the selective trace algorithm. Section 5 presents conclusions and proposes further studies.
2.
2.1. Distributed
MODEL
DEFINITION
System Model
This paper assumes that values exchanged between processes depend only on the initial values in each process and the order in which processes communicate. That is, processes are assumed to be deterministic. Stated somewhat differently, there are no nondeterministic statements, such as asynchronous interrupts, and there are no time-dependent statements, such as reading a clock or timing out from a semaphore. The distributed system execution model, based on message-passing communication, is the same as that proposed by Lamport [8]. A distributed system consists of a finite set of processes and a finite set of channels. Channels are assumed to have infinite buffers, to be error-free, and to be FIFO. The delay experienced by a message in a channel is arbitrary but finite. Sending a message and receiving a message are considered special events and are called a send event and a receive event. The other events are called internal events. The initial state of each process is also considered to be an internal event. We can then define the “happened before” relation [8], denoted by “<,” as follows (In [S], -+ is used rather than <.) DEFINITION 1. [8] The relation “<” on the set of events of a system is the minimum relation satisfying the following three conditions: (1) If a and b are events in the same process, and a comes before b, then a < b. (2) If a is the sending of a message by one process and b is the receipt of the same message by another process, then a < b. (3) If a < b and b < c then a < c. For two events a and b, a 5 b if a < b or a = 6.
In some distributed systems, processes communicate via shared memory [9]. Such systems can be simulated as follows by a message-passing system. Each memory object is considered to be a process. Writing to a memory object is converted into a message from the writer process to the memory-object process. Reading from the shared memory is converted into two messages, a read request from the reader process to the memory-object process and a reply from the memory-object process to the reader process. This paper refers only to a message-
DISTRIBUTED
63
PROGRAMS
passing system, but the results are also applicable to systems communicating through shared memory. Next, for the proof of the later lemmas, Chandy and Lamport’s “meaningful global state” [2] of the distributed system is formally defined using the “happened before” relation. DEFINITION 2. Let Ej be the set of events in process i. An N-tuple of events of processes s = (e, , ez, . . . . eN) (ei E E;) is said to be a global state if and only if for all ei E Ei, ei > ei implies el & for any j(1 5 j 5 N). Let U be the set of all the global states. l?j
(In [2], the global state is defined by a set of event sequences. Our definition is equivalent to that one.) Global state s = (ei , e2, . . . . ehi) is intuitively considered as a set of concurrently occurring events for some timing occurrence, and we consider it as the state when each process i has just finished the execution of e;. The “happened before” relation for global states is defined as follows. DEFINITION 3. For two global states s = (ei , e2, . . . . eN) and s’ = (e;, e;, . . . . eh), s I, s’ if and only if ei 5 e,! for every i (1 5 i 5 N), and s < s’ if and only ifs 5 s’ and < ei for some j(1 I j 5 N). ej
2.2. Debugger
Model
This section gives a general model for a replay-based debugger. A replay-based debugger D consists of one main debugger MD and local debuggers LD; corresponding to the processes (Fig. 1). Every pair of MD and LD~ has communication channels in both directions. LD; halts process i, lets it execute events, and tests its status. MD sends commands to LD~, receives it replies, and displays the results to the user. D executes the programs iteratively. The first execution is called a monitoring phase and the others are called replay phases. In the monitoring phase, each LDi independently stores the communication history of the process locally. The
v user FIG.
1. A replay-based
debugger.
MANABE
64
communication history is a sequence of items corresponding to receive events. The item is process number or a Null symbol. When the process receives a message from process j in the c&h receive event, the ath item of the history isj. Null means that no message was received in this receive event. After the monitor phase, debugger commands such as “stop if P” are given to h4D. In the replay phase, message receiving is controlled by LDi so that the order of message arrival at the process is the same as in the original execution. The processes are deterministic, so the execution behavior of this replay is the same as that of the original execution. In the replay phase, LD~ controls process i with the following operations that sequential program debuggers offer: DEFINITION
4.
LD~ can do the following
AND IMASE Process 1
2
3
1 Time
4:
message-passing
si
: initial
e:i
event
FIG. 2.
between
processes
state
An execution in a 3-process distributed system
operations to
process i in the replay phase. 1. See the type of event to be executed next, where the type may be “receive event, ” “send event,” or “internal event.” 2. Buffer the messages sent to i and see the messages sent from i. 3. Execute the next event and half the process (and also specify which message should be read if the executed event is a receive event). 4. Examine the current state of the process. 3. HALTING
AT BREAKPOINTS
This section discusses how to set breakpoints and halt the system in the replay phase. First, predicates are defined to specify breakpoints. 3.1. Global Predicates Sequential program debuggers typically use simple predicates. These can be expressed in terms of the Boolean values of variables in the program and the program counter. The simple predicate is instantaneous in the sense that its validity does not depend on the execution history, but rather on the state at an instant. The restriction is that a simple predicate is based on the state local to a single process. Let the simple predicate for process i be SP;. The value of SPi right after the execution of event e is denoted by SP;(e). Distributed processes may generate errors in which the overall system is in error but each individual process is not. Consider the following example (Fig. 2). Process i (i = 1, 2) has a variable vi. The value of vi just after the execution of each event is also shown in Fig. 2. Suppose that Vi = 1 means i has the right to a write operation for a database item. In order to keep the database correct, two processes cannot have the right to the write operation at
the same time. That is, u1 = 1 and v2 = 1 at the same time is an error. To detect this kind of error, global predicates-that is, predicates related to plural processesmust be considered. Two kinds of global predicates are introduced in [ 11I. One is the Disjunctive Predicate (DP), which consists of simple predicates joined by disjunctive operators “U.” The other is the Conjunctive Predicate (CP) consisting of simple predicates and conjunctive operators “0.” Note that global predicates are functions mapping from I/ to {true, false}. Miller and Choi [l 11 proposed an algorithm to halt the processes for a non-replay debugger. For a given global predicate, their algorithm halts the processes when a “Linked Predicate” is satisfied. A Linked Predicate is of the form SP; < SPj < SPk < . . . . where “<” is the “happened before” relation. To halt at SP, fl SP2 17 1.. n SP, , it is necessary to produce n! Linked Predicates for different orders of predicates; otherwise the debugger can fail to halt. Haban and Weigel [7] proposed another algorithm to halt the processes for a non-replay debugger. Their algorithm considers general Disjunctive and Conjunctive predicates, and is not restricted to the Linked Predicates. However, it is impossible for their algorithm to halt the processes just when the predicate is satisfied. If the process execution is suspended and some process gathers the status of all processes, this kind of predicate satisfaction can be detected exactly when it is satisfied. If a nonreplay debugger delays the process execution, however, its behavior might be different from that without the debugger. Since a replay debugger produces the same behavior even if some process executions are delayed, we can control the execution and detect this kind of predicate satisfaction. We consider the problem of halting for a given CP and DP by controlling the process execution.
65
GLOBALCONDITIONSINDEBUGGINGDISTRIBUTEDPROGRAMS
Another aspect of distributed program debuggers is that the global state for a given global condition is not unique. When a global predicate P is given to specify a breakpoint as the command “stop if P,” there are many global states satisfying P. Let G(P) be {s E U 1 P(s) = true}. At which global state in G(P) should the debugger halt the system? Consider again the example in Fig. 2 and the condition P = (ur = 1 fl u2 = 1). This predicate has no condition concerning process 3. Where should we halt process 3? When a condition is satisfied, conventional debuggers broadcast a message to the other processes to halt [4, 7, 111 or wait until the halted process blocks the execution of the other processes [9]. The cause for P becoming true, however, is contained not only in process 1 or 2, but also in process 3. The message from process 3 might cause P to be true. In that case, if process 3 executes beyond the send event e:, its contribution to the cause might be destroyed and the real reason for the error might be hidden (consider, e.g., the case that the event e: clears all the variables used for the decision to send a message to process 1). It would therefore be best to stop the processes at the “first” global state in G(P). That is, the processes should halt at the state in which each process has executed the minimum requirements for making P true. We define the “first” global state for P and consider the problem of halting the processes at this global state. The “first” global state in G(P) can be defined as follows. DEFINITION 5. For a predicate P, Znf(P) = {s 1 s E G(P) and s’ G G(P) for any s’ E U
such that s’ < s}. For the example in Fig. 2 and CP P, = (u, = 1 fl u? = I), Znf(Pl) = {(e:, e:, e:)}. for a DP P2 = (u, = 1 U u2 = l), hf(P~) = {e:, s?, e:), (s’, e:, s3)}. Note that Znf(P) may include plural global states. Sections 3.2 and 3.3 show that it is possible to stop the processes at s E Znf(P) when P is a CP and at s E G(P) when P is a DP.
have arrived because its sender S is passive or because the execution of S is delayed. If M has not arrived, the receiver R sends a control message to ask S to send M. Then S becomes active executes the program to send M. Process S remains active and continues execution until the predicate becomes true after sending M. In other words, a process is active when its predicate is false or when it must send a message. In a control message, it is necessary to specify which message S must send. Thus the control message contains an identifying message number, which is a sequence number of the messages going through the channel from S to R. LD; counts the messages sent to or received from every channel. Note that it is unnecessary to send the message number attached to the message. We should be able to detect when every process is passive and will not become active. This is one variation of the distributed termination detection problem proposed by Dijkstra and Scholten [5], and many algorithms have been proposed for different assumptions regarding the system. The part that detects the termination is called a termination detector. Status changes are reported to the termination detector when they occur. To simplify termination detection, we assume that control messages go through the termination detector. The termination detector can be implemented either in a distributed fashion (in LDi) or in a centralized fashion (in MD). The distributed algorithm shown in [3] can be used for this termination detector. The outline of the halt algorithm [IO] is shown in Fig. 3. Proof of the correctness of this algorithm is beyond the scope of this paper. It is shown only that the above algorithm stops the processes at some s E inf(P). In preparation, a definition and two lemmas are presented. DEFINITION 6. For two events e and e’ in one process, let min(e, e’) be e if e 5 e’, and e’ otherwise. For twoglobalstatess = (er,ez, . . . . eN)ands’ = (e;,e;, . . . . eh), min(s, s’) is defined as min(s, s’) = (min(e,,
3.2. Halting for Conjunctive
e;), min(ez, e;), . . . . min(eN, eh)).
Predicate
This section proposes an algorithm which stops the processes at Znf(P) when P is a CP. For each process, we introduce “active” and “passive” states. If the process is being executed, it is called active. A passive process becomes active only when another active process make the process to be active. System halting means that all processes are passive. Suppose processes without a simple predicate have a special predicate which is always true. Initially, processes whose simple predicate is false are active and the other processes are passive. Suppose an active process tries to receive a message M. The message M may not
LEMMA 1. Zf s, s’ E U, min(s, s’) E U. Proof. Assume that min(s, s’) = (e;‘, e;‘, . . . . e;) 6 U. From the definition of U, there is a pair i and j such that 3e::” E Ei, ei” > ei’ and e:l’ < ey. Since el’ = min(ei, e,!), e:’ = e; or e:’ = e,!. Without loss of generality, it can be assumed that e:’ = ei. Then
ey > ey = ei and which contradicts LEMMA 2.
ey < er = min(ej, e,‘) 5 ej,
s E U.
n
ZfP is a CP IZnf(P)I 5 1.
66
MANABE
FIG. 3.
Halting
algorithm
for a Conjunctive
IMASE
Next we show the latter proposition. Process i executes the next event if and only if one of the following conditions is satisfied:
program HaltAtBreakpoint /* Program for LD~. */ function TermTest; if s[;] > ms[;] for all i and SP = true /* If SP does not exist for this process, SP = true */ then return (passive) else return (active) end; function TryEzec; /* let e be the next event */ if e is receive from i and there is no program message from i in local queue then begin Send control message ‘%[;I + 1” to i; return(i) end else begin Execute e; return(NzLII) end end; begin /* MAIN */ for i := 1 to N do begiu s[i] := 0; /* The number of messages sent to i */ r[i] := 0; /* The number of messages received from i */ ms[;] := 0 /* i requested to send up to ms[;] */ end; cstat := TermTest; Send cstat to TerminationDetector; if cstat =active then waitp := TryErec; When execution of event e is finished: begin if e = send to i then s[i] := s[i] + 1; if e = receive from i then r[i] := r[;] + 1; cstat = TermTest; if cstat =active then waitp := TryExec else Send cstat to TerminationDetector; end; When program message m is arrived from i: begin Insert 772 to local queue; if waitp = i then waitp := TryEzec end; When control message “k” is arrived from i : begin ms[;] := k; bstat := cstat; cstat := TermTest; if bstat =passive and cstat =active then begin Send cstat to TerminationDetector; waitp := TryEzec end; When Termination is detected: Halt; end.
AND
i has an SPi and SPi = false. i received a control message requesting and has not yet sent the message. l l
a message
Suppose that the system halts at s such that some process i executes beyond t;. Let s’ be a global state during the replay such that s’ = (s,, s?, . . . . sN) 5 inf, and for some process i which executes beyond ti in s, si = tj in s’. Let Si,f be the set of processes which satisfy si = tj. No process in Sinf executes the next event by the first of above conditions, because si = ti. Thus, they do not execute the next events unless some processj fZ S’infsends a control message to a process k E Sia, before it executes tj, to ask for a message which has not been sent at th . Let the send and receive events of the requested message be ek and ej. Thus, eh > tk and eh < ej < tj holds, which contradicts inf E U. Therefore, no process executes beyond t;. 3.3. Halting for Disjunctive
Predicate
When the given predicate is a DP, it is impossible for D to stop the processes at s E Znf(P) during the first execution in the replay phase. end
Predicate.
Proof. Let the predicate that P be SP;, f! SP;, f’ . . . n SP;, . Assume two global states s and s’ (s # s’) are both members of Znf(P). If s < s’, then s’ 6? Znf(P), which implies s $ s’. Thus, min(s, s’) < s. P(min(s, s’)) = SPi,(WZin(ei,, e(J) n SPj,(WZin(ei,, eh)) n .. . n SP,(WIin(ej,, el)) = true. min(s, s’) E U from the above lemma. Thus, min(s, s’) E G(P). That min(s, s’) < s and min(s, s’) E G(P) contradicts s E Zrzf(P). Let inf= (t, , t2, . . . . t,) be the global state in Znf(P). To show that the system halts at inf, the following properties must be demonstrated. The processes do not terminate at any s E U such that s < inf. Process i does not execute beyond ti. l
l
Assume that the system halts at some s E U. Every SPi is true at s. Thus s is contained in G(P). Ifs < inf, the definition of Znf(P) is contradicted. The former proposition is therefore true.
THEOREM 1. There is no debugger D that detects s E Znf(P) without executing beyond s when P is a DP.
Proof. Let us consider the following example. The number of processes n is 2, and P = SP, U SP2. In the initial state, processes 1 and 2 do not satisfy SP, and SP2. Processes 1 and 2 never communicate with each other. Assume that D lets process 1 move one step from the initial state. In this case, it may occur that process 1 never satisfies SP, and process 2 satisfies SP2 after executing some events. Thus, D cannot stop at Znf(P). A similar situation may occur if D lets process 2 move one step. Therefore, the proposition is valid. It is, however, possible to halt at some s E G(P) and a halt algorithm is shown in [I 11. Note that if replay is executed twice, D can halt at a global state s E Znf(P) as follows. During the first replay, apply the algorithm in [ll] for P. Thus, the processes halt at some so E G(P). Select a simple predicate SPi in P which is satisfied at sO. During the second replay, apply the halt algorithm in Fig. 3 to Conjunctive Predicate SPi. Thus, the processes halt at s’ E Znf(SP;) c Znf(P). 3.4. Channel States in the Global Predicates The former subsections considered only the process states. In this subsection, we introduce channel state predicates in the Global Predicates. Channel state predicates can be converted into process predicates.
GLOBAL
CONDITIONS
IN DEBUGGING
Let the FIFO channel from process i to process j be Ci,j. We consider two types of channel state predicates for Ci.j Message content predicate: for a message m, m E Ci,j or m G Ci,j* Message number predicate: for a constant a, ICi,jI = (>, 2, <, 5, etc.)a. l
l
A message content predicate is converted into a process predicate as follows. Let the Conjunctive Predicate with the channel predicate P be SPi, II SPi, n . . . n SP, fl CSPi{,j$ n CSPi$,j$ n ... n CSPi/,j/ . The conversion is similar when the predicate is a Disjunctive Predicate. Let the predicate CSPi, = (m E Ci,j). Initially, check whether m is in Ci,j or not.
1. While m G C;,j Change the predicates for processes i and j to “false” and SPj, respectively (predicates for the other processes are unchanged) and execute the algorithm in Fig. 3. In addition, go to case 2 if i executes “send m to j”. 2. While m E Ci,j Let m be the xth message from i toj. The value x can be gotten from the variable s[j] in the algorithm in Fig. 3. Change the predicates for processes i and j to SP; and SPj n “r[i] < X,” respectively, and execute the algorithm in Fig. 3. In addition, check ifj executes “receive m from i” or i executes “send m to j” (sending m again). In the former case, go to case 1; in the later case, change the term “r[i] < x” for processj to “r[i] < y” (if m is the yth message from i to j). The case that CSPi,j = (m 65 Ci,j) is similar to this case. Next consider a message number predicate. Let the predicate CSPi,j = (IC;,jJ 2 a). Initially, let s[j] in process i be x and r[i] in process j be y. 1Ci,j 12 a if and only if x y 2 a. 1. While ICi,jI < u Change the predicates for processes i and j to “false” and SPj, respectively (predicates for the other processes are unchanged) and execute the algorithm in Fig. 3. In addition, check if i executes “send toj” or j executes “receive from i”. In the former case, increment x and if x - y 2 a, go to case 2. In the latter case, increment y and continue. 2. While ]Ci,j] 2 a Change the predicates for processes i and j to SPi, and SPj n “r[i] I n - a,” respectively, and execute the algorithm in Fig. 3. In addition, check if i executes “send toj” orj executes “receive from i.” In the former case, increment x and change the term “r[i] 5 x - a” for processj to “r[i] I x’ - a” (x’ is the new value of x) and continue. In the latter case, increment y and if y > x - a,
DISTRIBUTED
67
PROGRAMS
go to case 1. The case that ICSPi.j( = (>, 5 etc.)a is similar to this case. 4. SELECTIVE
Many sequential tive trace facility:
TRACE
debuggers have the following
truce [condition]
selec-
print [expression].
This command means that the debugger displays the value of [expression] whenever [condition] is satisfied. The expression consists of some variables, the program counter, and arithmetic and logical operators. In the distributed debugger, selective trace is considered to display the value of the global expression when the global condition is satisfied. The global condition is given by the global predicate, and the global expression consists of plural process variables and program counters. When the predicate P is a CP, selective trace can be implemented with the halt algorithm. When the processes halt at Znf(P), the debugger calculates the expression value at the global state and prints it out. When the printing is finished, the debugger returns each process state to its initial value, “active” or “passive,” and executes the programs again (under the control of halt algorithm). As shown in Section 3.3, it is impossible to halt at Znf(P) when P is a DP. To print out the expression value, however, it is only necessary to save the values of the variables used in the expression and afterwards calculate the expression value when Znf(P) is detected. When the calculation is finished, the saved values become useless and can be discarded. The trace algorithm is outlined below, focusing on the data saving and discarding mechanisms. The trace algorithm in LDi is implemented as follows. Let P be SPj, U SP,;! U . . . U SPjk. Let ei be the current event of process i at some instant. The first global state for ei, denoted by Znf[ei], is defined as Z$(si = eJ; that is, the first global state s = (si , ~2, . . . . SN) such that si = ei. LDi prepares virtual program counters in&[ 1 . . . N]. LD; i&2], . .. . maintains inJ[l . . . N] such that (i&[l], infi[N]) = Znf[ei] when i has just finished event ei. Note that these virtual program counters do not have to simulate real program counters but need only count the number of events that occur. The maintenance mechanism is shown later. When SPi becomes true in process i, (inJ[ 11, infi[2], . ..) inJ[N]) = ZfZf(SPi). Let process i’s variables in the [expression] of the trace command be xf , xi, . . . , xi,. When process i is in the initial state or has just finished a send event, i stores the current values of xi, xi, . . . , xi, together with the current value of infi[i] locally. Note that value storing is unnecessary after receive or internal events, because such an
68
MANABEANDIMASE
program const
SelectiveTrace; /* Program for I,DI. */ VaTlist = . . /* list of variables for selective trace */ I = . . . /* number of this process */ function SPExist; /* true if simple predicate exists for this process */ end; function TermTest; /* same as in RaltAtBreakPoint */ function TTyExec; begin/* let e be the next event *J if e is receive from i then if there is no program message from i in local queue then begin Send control message ‘%[i] + 1” to i; return(i) end else begin Execute e; return(Nzlll) end else if e is se,ld m to i then begin ;nf[l] := ;nf[l] + 1; Send [m, inf] to i; return(Null) end else /* internal event */ begin Execute e; return(Nhrllll) end end; begin /* MAIN */ for i := 1 to N do begin s[i] := 0; r[i] := 0; ms[;] := 0; ;nf[;] := 0 end; Get Values of Varlist and store [Values, inf[I]]; cstat := TermTest; if cstat =passive and (SPEzist) then Send output initialize message [inf, V&ties] to MD; if cstat =active or (SPExist) then waitp := TryExec;
When execution of event e is finished: begin if e = receive from i then begin /* let the piggybacked inf message be rinf */ for j := 1 to N do if r;nf[j] > inf[j] then begin inf[j] := rinf[j]; if (SPExist) then Send remove request “inflj]” to j end; ~[i] := r[i] + 1 end if e = dend to i then begin s[i] := s[;] + 1; Get Values of Varlist and store [Valaes, ;~~f[l]] end; cstat := TermTest; if cstat =passive and (SPExist) thenbegin Get Values of Varlist; Send output initialize message [;nf, ValzLes] to MD end; if &at =active or (SPExist) then waitp = TryExec end; When program message [m, ~inf] is arrived from i : begin lnsert [m, rinf to local queue; if waitp = i then waitp := TryExec end; When control message “I;” is arrived from i: begin if not (SPExist) then begin ms[;] := k; .bstat := cstat; cstat := TermTest; if bstat =passive and cstat =active then waitp := TryExec end end; When output request message “opt” is arrived from MD : begin Send Values of the record [Values, opt] to MD end; When remove request “TIC” is arrived from i: begin Mark record [Values, k] by i if k < rpc; Remove records which was marked by all i end; end. FIG.
4.
Trace
algorithm
for a Disjunctive
Predicate.
event e; in process i cannot be an element Of ZfcSPj) (j f i). If SPi becomes true, LD; reports to MD that SPi has become true, together with the current value of inA[ 1 . . . N] and xi, xi, . . . . XL,. When MD receives this report, it asks each LD~ to send the values of 9;) Y!, . . . . 2;, at
infi[j]. From the replies, MD calculates the expression value at Z&V;). To maintain inA[ 1 . . . N], at each send event LD~ sends the current value of inJ;[l . . . N] with the message. At a receive event, the receiver (let the receiver be j) receives the value inh[l . . . N] with the message and calculates injJk] = max(inJ;[k], i&[k]) fork = 1,2, . . . . N. It is easily checked that ir$[l], inJ[2], . . . . inJ[N]) = Znf[e;]. If the current value of inJ;[j] is a, the values of x’], x’;, . . . , ,$ when infj [j] < a are no longer used for calculating the expression value for SPi. Thus, whenever the values of inx[j] change, it is reported to LDj. If a value is unnecessary for all predicates, that is, for some U, infJj] 2 a for every i, then the values when inh[j] < u are removed from local storage. The algorithm is outlined in Fig. 4. 5. CONCLUSION
This paper introduced global conditions for some commands of distributed program debuggers. For setting breakpoints and halting, an algorithm is shown to halt at If(P) for a Conjunctive Predicate P. It is also shown to be impossible to halt at Znf(P) for a given Disjunctive Predicate P. A selective trace algorithm is shown for a given (Conjunctive or Disjunctive) trace condition P. For further study, global conditions defined by temporal logic might also be needed. ACKNOWLEDGMENTS The authors thank Drs. Masaru Takesue and Rikio Onai for their guidance and encouragement. They also thank Dr. Naohisa Takahashi for useful discussions, and Drs. Robert Cooper and Terunao Soneoka for their comments on an earlier version of this manuscript.
REFERENCES 1. Carver, R. H., and Tai, K. Reproducible grams based on shared variable. Proc. ence on Distributed Computing Systems. 2. Chandy, K. M., and Lamport, ing global states of distributed tems. 3, l(Feb. 1985), 63-75.
testing of concurrent pro6th International ConferMay 1986, pp. 428-433.
L. Distributed systems, ACM
3. Cohen, S., and Lehmann, D. Dynamic uted termination. Proc. of 2nd ACM Computing. 1982, pp. 29-33.
snapshots: DeterminTrans. Comput. Sys-
systems Symposium
and their distribon Distributed
4. Cooper, R. Pilgrim. A debugger for distributed systems. 7th International Conference on Distributed Computing Sept. 1987, pp. 458-465. 5. Dijkstra, diffusing
E. W., and computations.
6. Fairley, R. E. Softwure York, pp. 288-289.
Scholten, Inform. Engineering
Proc. of Systems.
C. S. Termination detection for Process. Lett. 11, 1(1980), l-4. Concepts.
McGraw-Hill,
New
7. Haban, D. and Weigel, W. Global events and global breakpoints in distributed systems. 21st Hawaii International Conference on System Science. Jan. 1988, pp. 166-175. 8. Lamport, L. Time, clocks, uted system. Comm. ACM.
and the ordering
of events
21, 7(July 1978), 558-565.
in a distrib-
GLOBAL
CONDITIONS
IN DEBUGGING
9. LeBlanc, T. J., and Mellor-Crummey, J. M. Debugging parallel programs with instant replay. IEEE Trans. Comput. C-36, 4tApril 19871, 471-480. IO. Manabe, Y., and Imase, M. Global condition in debugging distributed programs. DEICE Technical Report COMP89-99, Dec. 1989. I I. Miller, B. P., and Choi, J.-D. Breakpoints and halting in distributed programs. 8th International Conference on Distributed Computing Sysrems. June 1988, pp. 316-323. 12. Takahashi, N. Partial replay of parallel programs based on shared objects. ZEICE Technical Report COMP89-98, Dec. 1989. [In Japanese] YOSHIFUMI MANABE received B.E. and M.E. degrees in information engineering from Osaka University, Japan, in 1983 and 1985, Received December 19, 1990; accepted June 25, 1991
DISTRIBUTED
PROGRAMS
69
respectively. In 1985, he joined Nippon Telegraph and Telephone Corporation (NTT) Electrical Communications Laboratories. He is currently a research engineer in NTT Software Laboratories. His research interests are distributed algorithms, communications networks, and graph theory. He is a member of the ACM. MAKOTO IMASE received B.E. and M.E. degrees in information engineering from Osaka University, Japan, in 1975 and 1977, respectively. He received the D.E. degree form Osaka University in 1986. In 1977 he joined Nippon Telegraph and Telephone Corporation (NTT) Electrical Communications Laboratories. He is now a senior research engineer, supervisor in NTT Software Laboratories. He has been engaged in research on distributed switching systems and communications networks. He is a member of the IEEE Computer Society and received the IEICE Young Engineer Award in 1984.