- Email: [email protected]
Test generation with respect to distributed interfaces
CmHSHml RINIERHCES ELSEVIER
Computer Standards & Interfaces 16 (1994) 119- ! 32
Test generation with respect to distributed interfaces * Gang Luo *, Rachida Dssouli, Gregorv. Bochmann, Pallapa Venkataram *, Abderrazak Ghedamsi "~ Departement d'lRO, UniL~ersit~de Montreal, C.P. 6128, Succ. A, Montreal, P.O., H3C 3J7, Canada
Abstract
In the area of testing communication systems, the interfaces between systems to be tested and their testers have great impact on test generation and fault detectability. Several types of such interfaces have been standardized by the International Standardization Organization (ISO). A general distributed test architecture, containing distributed interfaces, has been presented in the literature for testing distributed systems based on the Open Distributing Processing (ODP) Basic Reference Model (BRM), which is a generalized version of ISO distributed test architecture. We study in this paper the issue of test selection with respect to such a test architecture. In particular, we consider communication systems that can be modeled by finite state machines with several distributed interfaces, called ports. A test generation method is developed for generating test sequences for such finite state machines, which is based on the idea of synchronizable test sequences. We then apply this method to generate test sequences for a so-called quorum protocol. Starting from the initial effort by Sarikaya, a certain amount of work has been done for generating test sequences for finite state machines with respect to the ISO distributed test architecture, all based on the idea of modifying existing test generation methods to generate synchronizable test sequences. However, none studies the fault coverage provided by their methods. We investigate the issue of fault coverage and point out a fact that the methods given in the literature for the distributed test architecture cannot ensure the same fault coverage as the corresponding original testing methods. We also study the limitation of fault detectability in the distributed test architecture. Key words." Distributed interfaces; ISO standard test architectures; Finite state machines; Protocol conformance testing; Protocol engineering; Software testing; Synchronizable test sequences
* Corresponding author. Now with Dept. of Computer Science, University of Ottawa, 150 Louis Pasteur Priv., Ottawa, Ont KN 6N5, Canada. Email: [email protected] * This work was supported by the IDACOM-NSERC-CWARC Industrial Research Chair on Communication Protocols at the University of Montreal (Canada).
* He is now with Dept. of Electrical Communication Engineering, Indian Institute of Science, Bangalore-560 012, India; email: [email protected] He is now with Dept. of Computer Science, Conconlia University, Montreal, Canada.
0920-5489/94/$07.00 © 1994 Elsevier Science B.V. All rights reserved SSDI 0920-5489(93)E0072-A
120
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
1. Introduction
A general distributed test architecture where the IUT (implementation under test) contains several distributed ports has been studied in [7] for testing distributed systems. It is based on the Open Distributing Processing (ODP) Basic Reference Model (BRM) (see Fig. 4). In this architecture, the I U T contains several distributed interfaces, called ports (i.e. points of control and observation); the testers cannot communicate and synchronize with one another unless they communicate through IUT, and no global clock is available in the system. This could be a test architecture of a communication network with n accessing nodes, where the testers reside in these nodes. When n = 2, this general distributed test architecture is reduced to the ISO distributed test architecture for communication protocol testing. We study in this paper test selection methods with respect to a general distributed test architecture. Usually, in the so-called local test architecture developed by ISO, the specifications of communication protocols are first abstracted into state machines [15], then test cases are generated from the resulting machines. A number of methods have been developed to generate test cases for finite state machines (FSMs) [9,18,6,19,17] or for nondeterministic FSMs [16]; however, they are not directly applicable to the distributed test architecture, because of the synchronization problem between distributed testers. In distributed test architectures, testing is relatively difficult because certain problems of synchronization between the testers may arise during the application of test sequences. To solve this problem, with respect to the ISO distributed test architecture where there are only two ports, an approach for test generation has been developed in [20] by modifying the existing test generation methods for FSMs such as the transition tour [17], the DS-method [14], and the W-method [6] such that the resulting sequences are so-called synchronizable test sequences. We develop in Section 3 an approach to generating test sequences for FSMs with n distributed ports (n _> 2), with respect a general distributed
test architecture (see Fig. 4), after defining and explaining several concepts related to distributed test architecture and FSMs. Our approach is a generalized version of the approach given in [20]. As an application example, we generate test sequences using our method for a so-called quorum-based protocol in the appendix. Starting from the initial effort by Sarikaya, a certain amount of work has been done for generating test sequences for finite state machines for ISO distributed test architecture, all based on the idea of modifying existing test generation methods to generate synchronizable test sequences. Chen [5] and Boyd [3] have studied the computation complexity issue of the approach of [20], and they have pointed out that there is no efficient (i.e. polynomial time) algorithm for obtaining a minimum length synchronizable test sequences from a given FSM. A heuristic approach to obtaining synchronizable test sequences is given in [5]. Another approach to obtaining synchronizable test sequences, presented in [12], is to insert additional synchronization statements (i.e. software probes) into the IUT. This approach is not applicable when the conformance testing is conducted by a second party other than the implementor as the party does not have access to the internal information of the IUT. Although so much work has been done on the different aspects of synchronizable test sequence generation, yet none studies the fault coverage provided by their methods. We explore in Section 4 the issue of fault coverage and discuss the observability and fault detectability in the distributed test architecture. We come out with a fact that these modified methods in the literature cannot ensure the same fault coverage as the corresponding original testing methods. For instance the modified transition tour cannot detect all output faults for FSMs with two distributed ports, although transition tour can detect all output faults for the FSMs with only one port. A class of output faults may remain undetected if the modified transition tour is applied to an FSM with two ports. Unfortunately, none of the former articles on generating synchronizable test sequences [20,12,5,3] had realized such weakened fault coverage. We also pre-
121
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
sent an improved approach that has a better fault detection power. We finally show a limitation of fault coverage in the distributed test architecture that not all faults of the I U T are detectable, showing the impossibility of complete fault coverage. Therefore, the testability of IUTs under the distributed test architecture need to be studied. We conclude in Section 5 by discussing some future research issues.
UpperTester LowerTester ImplementatiOnIuT]
L PCO 2'
:'ql
[ communicationservice Abstract testing framework for a distributed architecture
Fig. 2. Distributed test architecture.
We discuss in this section different test architectures used for testing communication systems, and finite protocol machine (i.e. finite state machine) model related to these architectures. We also present a fault model for finite state machines.
located in the same place. In the distributed test architecture, the test system is divided into socalled upper and lower testers which access the PCOs 1 and 2 of the IUT, respectively, as shown in Fig. 2. The lower interface PCO 2 is accessed over distance and indirectly through the underlying communication service. The coordinated test architecture is similar to the distributed test architecture, and the only difference between the two is that the former has some kind of coordination between u p p e r and lower testers, established using a so-called test coordination protocol through a (possibly separate) communication channel between upper and lower testers. The remote test architecture (see Fig. 3) corresponds to the distributed test architecture where only a lower tester is used; instead of an upper tester, the I U T may include a stack of several protocol layers above the layer being tested.
2.
2.1. Distributed test architectures for communication systems 2.1.1. ISO standard test architectures for protocol conformance testing ISO has defined four types of test architectures for protocol conformance testing, called local, distributed, coordinated and remote test architectures, respectively. Local test architecture corresponds to traditional software testing, as shown in Fig. 1, where PCO and I U T refer to point of control and observation, and the implementation under test, respectively. In this architecture, the two PCOs of the I U T can be viewed as a single port since the I U T and the tester are
2.1.2. A general distributed test architecture As mentioned before, the ISO test architectures only deal with an individual single protocol entity. However, to test the overall properties of
I Lower Tester
Tester
ImplementationRiT ]
L
Implementation IUT
,-II Fig. 1. Local test architecture.
Fig. 3. Remote test architecture.
G. Luo et aL / Computer Standards & Interfaces 16 (1994) 119-132
122
nize with one another unless they communicate through the IUT, and (iii) no global clock is available. In testing a communication network, this general architecture can model a communication network with n accessing nodes; the IUT models the communication network, and the testers are located in these nodes. When n = 2, the above model is reduced to the ISO standard distributed test architecture, which is shown in Fig. 2. 2.2. Finite state machines with n ports
A network with distributed terminals sometime can be modeled as a finite state machine with several ports. We define in the following the concept of multi-port finite state machines, which is a generalization of finite state machines with two ports given in [20].
Fig. 4. A general distributed test architecture.
distributed database systems and communication networks, we may face a general distributed test architecture, as shown in Fig. 4. In this general distributed test architecture, (i) the IUT contains several ports, (ii) the testers cannot communicate and synchro-
Definition. A multi-port finite state machine with n ports (np-FSM) is defined as a 6-tuple (St, Z, F, T, O, s o) where n > 1 and (1) St is a finite set of labels, called states. (2) ~ is a n-tuple (Lil, Li 2. . . . , L i , ) where Li~
L/t={ I },
L/2={2},~ L
o
z
~r =
t
c
2/ ~ . . / ~ . , ) }
,
~
~
] Li3=13}, ~
Lo3={d},
The initial state is A
Fig. 5. An example of 3p-FSM under distributed test architecture.
G. Luo et aL / Computer Standards & Interfaces 16 (1994) 119-132
is a set of inputs for port k , and L i k C~L i j = (~ denotes the empty set) when k 4:j and k , j = l , 2 . . . . . n. Furthermore, we assume L i = Li 1 U Li 2 u . . . ULi,,. (3) F is a n-tuple ( L o l , L o 2 , . . . , L o n) where L o k is a set of outputs for port k, and L o k c~ Loj = fJ when k 4=j, with k, j = 1, 2 . . . . . n. Furthermore, we assume L o = {(al, a 2. . . . . a m) ]3LOkl, Lok2 . . . . . LOkm such that a i c L o k I, a 2 E L O k 2 , . . . ,
a m
E LOkm,
l _ < k l < k 2 < ... < k m < n , and 1 _< m < n} where ( a l , a 2 . . . . . a m) is said to be an output tuple. (4) T is a transition function: . ~ S t where ~ G St × Li, (5) O is an output function. _~ ~ L o U {e} for the .~ in (4) where e stands for the empty output. (Note that the FSM is completely specified if .~ = St x Li). (6) s o ~ St is the initial state. In other words, a np-FSM has n ports, say P~, P 2 , . . . , P , , ; each port Pk is associated with a set of inputs L i k and a set of outputs L o k. For any two different ports, there are no common inputs (outputs) in the two associated input sets (output sets). A np-FSM can also be represented by a directed graph in which the nodes are the states and the directed edges are transitions linking the states. Fig. 5 shows an example of 3p-FSM under the distributed test architecture. From this definition, a np-FSM is deterministic since a state and an input uniquely determines a next state and an output tuple; but it is not necessarily completely specified in that not every s t a t e / i n p u t pair has a defined next state and an output tuple. Let a ~ Li, b ~ L o , and P, Q ~ St; we write P - a / b -~ Q to represent that T ( P , a) = Q and O ( P , a) = b. P - a / b ~ Q is called a transition f r o m P to Q with the label a / b . For the example of 3p-FSM shown in Fig. 5, we have A 1 / ( a , c ) -~ B. The np-FSMs can model a communication network with n terminals where each terminal is
123
viewed as a port. This model could be useful for network testing and verification. When n = 2, 2p-FSMs are finite state machines defined in [20,12,3], which are viewed as IUTs under the distributed test architecture. 2.3. Fault model
We present a fault model for np-FSMs. Like other state machine based test generation, fault models serve as a guide to test generation and as a basis for test coverage analysis, as described in [2]. Let SP and I U T be two np-FSMs, representing a specification and its implementation, respectively. Assume that they have the same X and F. Then the fault types are defined as follows: (1) Output fault: We say that IUT has output faults if SP can be obtained from I U T by modifying the outputs of one or more transitions in IUT. (2) Transfer fault: We say that IUT has transfer faults if SP can be obtained from I U T by modifying the end states of one or more transitions in IUT. (3) Hybrid fault: We say that I U T has hybrid faults if SP can be obtained from I U T by changing the outputs a n d / o r the end states of one or more transitions, in IUT.
3. Synchronizable test sequence generation Although a number of test generation methods have been developed based on FSMs, they cannot be applied to the FSMs with two or more distributed ports directly, as pointed out by [20], since a so-called synchronization problem exists in the distributed test architecture. We generalize in this section the concept of so-called synchronizable test sequence given in [20] for 2p-FSMs to the case of np-FSMs, dealing with the synchronization problem. Based on the generalized concept, we present an approach to modifying the existing test generation methods for FSMs, such as the transition tour [17], the W-method [6], the DSmethod [14] and so on, to obtain synchronizable test sequences for np-FSMs. For certain protocol
124
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
specifications, it is impossible to avoid synchronization problems; we use techniques for testing nondeterministic FSMs to handle such specifications.
3.1. Synchronization problem and synchronizable test sequences Under the architecture given in Fig. 4, the testers are distributed over several sites. They are only synchronized through the interactions with the implementation. In this situation, considering two consecutive transitions t l and t2 of a given np-FSM I (n > 2), one of the testers is said to face a synchronization problem if this tester did not take part in the first transition and if the second transition requires that it sends a message to the machine I. For example, consider the 3p-FSM shown in Fig. 5, tester3 faces a synchronization problem when considering the consecutive transitions t l and t7. tester3 is supposed to send the input 3 to the machine after the t l has been executed;however, since t 1 neither receives any input from, nor sends any output to the tester3 (i.e. tester3 did not take part in the transition tl), tester3 does not know whether t l has been executed, i.e. a synchronization problem. We now define the concept of synchronizable test sequences for np-FSMs, n > 2, for handling the synchronization problem of general np-FSMs. Given a np-FSM I with the ports 1, 2 . . . . . n, we require the following concepts for the ease of presentation. Definition Interaction ports (IP) of a given transition [pi, PO]. Let pi ~ {1, 2 , . . . , n} and PO c_ {1, 2 . . . . . n} (n is the number of ports). [pi, PO] is said to be an interaction port (IP) of a given transition t if t receives an input from the port pi and sends to each port in PO an output (if PO = ¢, t does not send any output). Definition Synchronizable test sequences. Given an ordered pair of transitions t l and t2 of I, let [pi, PO] and [pi', PO] be their IPs, respectively. t l and t2 are said to be synchronizable if (1) pi = pi', or (2) pi' ~ PO.
A given test sequence is said to be synchronizable if any two consecutive transitions of the sequence are synchronizable. Formerly, people define the synchronizable test sequences for 2p-FSMs by means of the so-called basic transition list [20]. However, for np-FSMs, a generalized version of this list could be too large to handle. The concept of interaction ports provides a nice means to define the synchronizable test sequences, having avoided a cumbersome presentation. As an example, for the 3p-FSM shown in Fig. 5, the IPs of the transitions t l and t2 are [1, {1, 2}] and [2, {3}], respectively; the transitions t l and t2 are synchronizable. It is easy to see that testers will not face any synchronization problem if synchronizable test sequences are used. Therefore, it is desirable to generate synchronizable test sequences for testing np-FSMs.
3.2. Generating synchronizable test sequences Guided by the idea of synchronizable test sequences, we present a modified transition tour such that the resulting test sequences are all synchronizable, using an approach similar to the one given in [20]. The other existing test generation methods for FSMs can be modified to obtain synchronizable test sequences for np-FSMs, in a similar manner. Any graph traversal algorithm such as the one given in [21] can be modified to obtain a transition tour. Assume that an algorithm TT produces a transition tour; we present in the following a procedure for generating synchronizable transition tours, by modifying the algorithm TT.
3.2.1. Generating synchronizable transition tour Each transition t to be added to the sequence by the algorithm TT is first checked whether it forms a synchronizable pair together with the last transition tt of the sequence; that is, assuming the IPs of the transitions tt and t to be [pi, PO] and [pi', PO'], respectively, check whether pi = pi' or (2) p i ' ~ PO are true. If the transitions tt and t are not synchronizable, a different transition from the present state is considered. If no
125
G. Luo et al. / Computer Standards& htterfaces 16 (1994) 119-132
suitable transition exists from the present state, the algorithm TT backtracks to the previous state, continuing the tour from there in a different way. This process continues until all the transitions of the machine are covered. For example, Fig. 6(b) shows a synchronizable transition tour for the 2p-FSM shown in Fig. 6(a). Furthermore, for the example of 3p-FSM shown in Fig. 5, a synchronizable transition tour is ' t l , t2, t3, t4, t5, t6, t7, t8, t9'. As an application example, we apply this approach to a socalled quorum protocol in the appendix. Using an approach similar to the above, one can obtain the test generation methods for npFSMs by modifying the DS-method [19], the Wmethod [6], the U I O - m e t h o d [19] the W p - m e t h o d [9] and the Petrenko method [18] such that the resulting test sequences are all synchronizable. We note that for certain np-FSMs, given a transition, there may not exist any synchronizable test sequence that can force the machine to traverse this transition. Such an example was given in [20] for a 2p-FSM. In this case, one cannot obtain any test suite using the modified methods.
4. Fault coverage In the distributed test architecture, since there is no other synchronization channel instead of 1UT, the observability and fault detectability is weaker than that in the local test architecture. Therefore, it is difficult to achieve good fault coverage. We find that the modified methods presented in [20] do not ensure the same fault
0/
~
>
Lil = 0 }, Li2 = { 1 } Lo! = b } , l.o2 = { a , c } (a)
coverage as the corresponding original testing methods; and we also present an improved approach that provides better fault coverage. We finally show that no method can ensure full fault coverage for finite state machines in the distributed architecture.
4.1. Observability and fault detectability For a system (software or communication system) that receives inputs and produces outputs, observability refers to the ease of determining if specified inputs affect the outputs; fault detectability refers to the ease of detecting specified fault. Observability and fault detectability of I U T vary in different test architectures. Consider the ISO test architectures presented in the earlier section. An I U T has the highest observability and fault detectability in the local test architecture, and has lower ones in the coordinated, distributed architectures, and the lowest one in the remote test architecture. If one associates each test architecture (local, coordinated, distributed and remote test architecture) with a set of detectable faults (S 1, S c, Sj, St), as shown in [8], the fault detectability in these architectures is reflected by the following:
S,c_Sac_S, c_S l. In the distributed testing architecture, the observability and fault detectability of I U T are limited [1]; therefore, it is difficult to achieve good fault coverage in the distributed test architecture, which is further explored in the following subsections.
Input sequence:
0, 1, 1, 0
Resulting global Port1: sequence: Port 2: PIs:
0[
[ 1
11
I
l
I
c
, <2,{ 2 }>,<2,{ 1 }>, tester 1 -- port 1 tester2 -- port 2 The initial state is A (b)
Fig. 6. An example of a 2p-FSM.
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
126
4.2. A class of undetectable output faults by synchronizable test sequences For FSMs, as we know, the transition tour [17] can detect all output faults if no transfer faults occur. However, for 2p-FSMs, a transition tour which is a synchronizable test sequence does not necessarily detect all output faults. For example, the 2p-FSM shown in Fig. 7(a) is a faulty implementation of the 2p-FSM shown in Fig. 6(a); and it has only output faults. The input sequence '0, 1, 1, 0' is a synchronizable transition tour, as described in Fig. 6(b). When the input sequence '0, 1, 1, 0' is applied to the faulty implementation, we obtain the global sequence shown in Fig. 7(b). Although the global sequence for the original 2p-FSM is different from the global sequence for the faulty one, the difference cannot be seen from the two local ports U and L since no global clock is assumed in the distributed test architecture. Therefore, a modified transition tour does not necessarily detect all output faults for 2pFSMs. Using similar arguments, it is easy to prove that the modified W-method and DS-method [20] do not necessarily detect all output and transfer faults for 2p-FSMs. [12] presented a method to solve the synchronization problem by instrumenting probes into the IUT. However, since the method only guarantees that input sequence is synchronized, the above problem still exists. Furthermore, the instrumentation of the IUT is not permissible in the conformance testing when the testing is con-
t 1: O/
~
b>
ducted by a second party other than the producer. We formalize the class of output faults which may remain undetected by the modified methods as follows: For two consecutive transitions, say t l and t2, in a given specification S, the faulty implementation I can be obtained from S by removing an output from one of the two transitions and adding the output to the other transition. We call this class of output faults outputshifting faults. We present in the following an outline of an approach which can detect this class of output faults. The outline of the approach: (1) Generate a set of synchronizable test sequences, say H, by using the approach of [20], with respect to one of the test generation methods for FSMs such as the transition tour [17], the W-method [6], and so on. (2) Find a set of all transition pairs, say/2, along the paths caused by applying the sequences of H, each pair of which may have an outputshifting fault which is undetectable using /7. (3) If /2 is empty, stop. Otherwise, add a set of additional synchronizable test sequences to /7, or concatenate some synchronizable subsequences to the sequences in H, such t h a t / 7 can ensure the absence of output-shifting faults in the transition pairs o f / 2 . Go to Step 2. It is not very difficult to give a detailed procedure for Steps 2 and 3, but under what condition an efficient procedure exists and what is efficient procedure, is the work of future research. The
Input sequence:
O, 1, 1,0
Resulting global sequence:
lOB [ ' ~ l b l
I
lalllclll
(b) (a) Fig. 7. An example of output-shiftingfault in a 2p-FSM.
ib[ Otbl
I
i I
127
G. Luo et aL / Computer Standards & Interfaces 16 (1994) 119-132
added by Step 3
Resulting global sequence:
0
bI I Ibm 0 1 b l l a 1 IIc 1 I I I i
tl, t3, f~ = {, }
t2,
t4
OIbl
I
I
IblOIblO,
laltlclll t 1,
t3,
Fig. 8. The example for Step 2.
I t2,
blO, b l O , bl
I I t4,
l I tl,
t4,
, I
lal]lcl t 1,
t3
Fig. 9. The example lor Step 3.
worst case computation complexity of the above procedure is not polynomial since the procedure contains the generation of synchronizable test sequences; and no polynomial algorithm exists for the generation [5,3]. We now explain the approach using the example of Fig. 6(a). Suppose that the transition tour is of our interest. Fig. 6(b) shows the resulting test sequence after Step 1. Fig. 8 shows the result of Step 2; the two arrows in the figure indicate the two possible outputshifting faults which cannot be detected by the test sequence. Fig. 9 shows the result of Step 3; the test sequence can ensure the absence of any output-shifting faults shown in Fig. 8. The transition string 'tl,t4' ensures the absence of outputshifting fault in ( t l , t3), and the rightmost tran-
sition t3 of the test sequence ensures the absence of output-shifting fault in (t3, t2). Although the improved approach detects more faults than the original, it does not necessarily guarantee the total absence of faults. Therefore, the approach is still heuristic "in nature since no precise fault coverage is easily given under the given fault model. We also note that it is impossible to guarantee the total absence of faults in the distributed test architecture, as described in the following section. 4.3. Impossibility o f complete test cot:erage in a distributed test architecture
We show in the following that, in the distributed test architecture, no test suite (i.e. a set
hi
The initial state is A Liu ={ a,b }, LiL ={c } Lov ={ d,e }, LoL = { f,g}
Fig. 10. An example of 2p-FSM.
128
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
of test sequences) can ensure the total absence of the faults specified in the fault model. In other words, for a np-FSM, not all faults specified in the fault model are detectable. We first note: no method that is based on the concept of synchronizable test sequences can ensure full fault coverage for all np-FSMs. The reason is that, for certain np-FSMs, given a transition, there may not exist any synchronizable test sequence that can force the machine to traverse this transition. Such an example was given in [20] for a 2p-FSM. One may argue that, for the np-FSMs where some transitions cannot be forced to be traversed by any synchronizable test sequences, to traverse these transitions, certain test sequences can be applied to the machines repeatedly up to a sufficient number of time. Then the probability that not all these transitions are exercised at least once, can usually be reduced to close to zero (note: this is a common assumption for testing nondeterministic finite state machines [16]). We argue that, even using such an approach and the assumption, still no test suite can ensure full fault coverage. We prove this using the 2p-FSM shown in Fig. 10. For the sake of convenience, we introduce in the following several terms for 2p-FSMs. A test case is denoted as two I / O sequences [o-u, O-L] such that o"U consists of the inputs and outputs of the port U, and O-L of the port L, respectively. We say that a 2p-FSM I passes a test case [o-u, o-L] if there is a path p in I such that when p is traversed, o-u and O-L will be observed at the ports U and L, respectively. A fault in the implementation I is detectable against its specification S if and only if there exists a test case [o-u, °'L] such that S passes [o-u, o-L] and I does not. Consider the example of Fig. 10, let the 2pFSM be S (where plain line transitions make the machine complete). Assume that I is a corresponding faulty implementation which results from changing the output ' f ' of the transition t3 to 'g' (see the dashed transition). The transition t3 in I has an output fault. We argue that the fault is not detectable. We note that all the paths which start from the initial state A and traverse the faulty transition t3, begin with the transition
string 'tl, t2, t3'; and t3 does not appear in the rest of the paths any more. The input sequence pair for the ports U and L, corresponding to the path 'tl, t2, t3', is [a.b, c]. Since no synchronization exists between the ports U and L, the [a.b, c] may cause the machine to be executed along one of the three paths: 'tl, t2, t3', 'tl, t5, t7', 't4, t6, t7'. The transitions of these paths are indicated by bold lines shown in Fig. 10. The total of two test cases, [a/d.b/e, c/f] and [a/d.b/e, c/g] are derived from the three paths. If we assume on the contrary that the fault in t3 is detectable, it is not difficult to see that the fault must be detected using the two test cases [a/ d.b/e, c/f] and [a/d.b/e, c/g]. However, from Fig. 10 both S and I pass the test cases [a/d.b./e, c/f] and [a/d.b/e, c/g]. Therefore, the fault is not detectable. Nevertheless, all faults are detectable if synchronization exists between the distributed ports or there is a global clock - in this case, the test generation methods for FSMs can be applied directly. Therefore, the testability of IUTs under the distributed test architecture should be studied.
5. Conclusion
Test architectures represent interfaces between systems to be tested and their testers; and they are important topics in the International Standardization Organization (ISO). A general distributed test architecture, which is a generalized version of the ISO distributed architecture, has been presented in the literature for testing distributed systems based on the Open Distributing Processing (ODP) Basic Reference Model (BRM). Based on this test architecture, for the systems modeled by finite state machines, we developed a test generation method, and applied this method to generate test sequences for a so-called quorum protocol. We investigated the issue of fault coverage. We pointed out that the methods given in the literature for the distributed test architecture, modified from certain existing methods, cannot ensure the same fault coverage as the corre-
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
120
each node uses a quorum-based protocol
no(~
node 3 Fig. 11. The architecture of the distributed system using quorum-based protocol.
Fig. 12. The test architecture for testing a quorum-based protocol.
sponding original testing methods. We also studied the limitation of fault detectability in the distributed test architecture. The distributed test architecture also gives rise to new problems of FSM based diagnosis, and the diagnosis method of FSMs of one port [10] may be modified for np-FSMs by applying the concept of synchronizable test sequences.
(1) Each port, i, has associated with it a number of votes, ui, and ~'i is a positive integer. (2) Let V indicate the sum of the votes of all ports of the distributed system. (3) A transaction must collect a commit quorum Vc before committing. (4) A transaction must collect an abort quorum Va before aborting. (5) Va + Vc > V.
Appendix: Test generation for a quorum-based protocol To illustrate the method we have taken a quorum-based protocol which is used for nonblocking of resources and increasing the distributed system's availability [11,4]. We generate test sequences for this protocol. Fig. 11 gives the architecture of a distributed system that use a quorum-based protocol. The test architecture for testing this protocol is shown in Fig. 12. The basic idea of the protocol is that a majority of nodes in the distributed system must agree on the commit or abort of a transaction before the transaction is committed or aborted. The weights which are assigned to the ports are usually called votes, since they are used when a port 'votes' on the commit or abort of a transaction. The basic rules of a quorum-based protocol are:
bf
/
, mi
h nl
Fig. 13. A quorum-based prott)cc~l.
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
130
In p r a c t i c e Va + Vc = V + 1 is t h e m o s t conven i e n t one. E a c h d i r e c t e d e d g e in Fig. 13 r e p r e s e n t s a set o f transitions. F o r e x a m p l e , c o n s i d e r a i a n d dij, w h e r e 0 < i , j < N a n d i 4:j, r e p r e s e n t s N a n d N(N1) t r a n s i t i o n s , respectively. N is a n u m b e r of p o r t s in a d i s t r i b u t e d system. F o r clarity we c h o o s e N = 3 a n d d e s c r i b e the l e g e n d of Fig. 13 in T a b l e 1. Fig. 14 explains t h e set o f t r a n s i t i o n s for a i a n d dij for N = 3. T h e i n p u t a n d o u t p u t sets u s e d at e a c h p o r t a r e t h e following: Li I = {No_com_reqp
No_vote
req~2,
N o _ v o t e _ r e q ~ 3 , C o m _ r e q 1, V o t e _ r e q ~ 2 , V o t e _ r e q 13, V o t e _ cast 12, V o t e _ cast 13, Vc 1, V a 1, T a s k _ execute 1, A b o r t _ req 1, R e l e a s e _ req l, T i m e _ out 1, R e s t a r t _ req 1} L i 2 = {No _ c o m _ r e q 2, N o _ vote _ r e q 21, N o _ v o t e _ req 23, C o m _ r e q 2, V o t e _ req 21, V o t e _ r e q 2 3 , Vote_cast2~, Vote_cast23, Vc 2, V a 2, T a s k _ e x e c u t e 2, A b o r t _ req 2, R e l e a s e _ req 2, T i m e _ o u t 2, R e s t a r t _ req 2} L i 3 = {No _ c o m _ r e q 3, N o _ r e q 31, N o _ v o t e _ r e q 32, C o m _ req 3, V o t e _ r e q 31, V o t e req32, V o t e _ c a s t 3 t , V o t e _ c a s t 3 2 , Vc3, Va3, T a s k _ e x e c u t e 3, A b o r t _ r e q 3, R e l e a s e _ r e q 3, T i m e _ o u t 3, R e s t a r t _ r e q 3}
Table 1 Legend for the protocol in Fig. 13
ai bij cI ce c3
dij eij fl f2 f3 gi hi k1 k2 k3
li mI m2 rn 3
ni
In the following, , j = 1, 2, 3, and i 4:j, and abbreviated I / 0 actions are described in parenthesis. No_com_reqi/e (No commit request from the ith port/e) N o v o t e reqij/e (No vote request from ith port to the jth port/e) Com reql/Com_waitl, Com vot%2, Com votel3 Com_req2/Com wait 2, Com_vote21, Com_votee3 Com_reqa/Com wait> Corn vote31, Com_vote32 (Commit request from ith port/{Commit wait indication to the ith port, Commit vote request broadcast from ith port to the rest}) Vote-reqiJAckji (Vote request from ith port to the jth port/{ack from jth to ith port}) Vote_castij/Ackji (Vote send in favour or against commit from ith port to the jth port/{ack from jth to ith port}) V c l / C ° m m i t l , C°m-indl2, C°m-ind13 Vc2/Commit2, Com ind21, Com_ind23 Vc3/Commit3, Com_ind31 , C o m ind32 (Vc count received in favour of ith port to commit/{commit-execute to ith port, commit indication to the rest}) Vai/Prepare-ab°rti (Va count received against ith port commit/{prepare abort message to ith port}) Task_ execute//e (Task execute by ith port/e) Abort_reql/Abot-donel, Abort_ind12, Abort_ind13 Abort_reqJAbot-done2, Abort ind21, Abort ind23 Abort_req3/Abot-done3, Abort ind32, Abort_ind31 (abort request from the ith port/{abort done to the ith port, abort indication to the rest}) Release_ req i/Release _ donei (Release request from ith port/Release done to the ith port) Time_outl/Abot-donel, Abort_ind12, Abort_ind13 Time out2/Abot-done 2, Abort_ind21, Abort_ind23 Time out3/Abot-done 3, Abort_ind31, Abort ind32 (Time out indication from ith port/{abort done to the ith port, abort indication the rest}) Restart_reqi/e (Restart request from ith port/e)
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
131
References
a2
1
Fig. 14. Description of transition set of Fig. 13. L o t = {Corn ._w a i t ~, C o m _ v o t e z ~ , C o m vote31, Ack2~ , Ack31, C o m m i t s , C o m _ i n d 2 t , C o m _ i n d 3 ~ , P r e p a r e - a b o r t ~, A b o r t - d o n e i, A b o r t _ ind 2~, A b o r t _ ind 3 t, R e l e a s e d o n e 1} Lo2 = { C o r n _ w a i t 2 , C o r n voter2, C o r n _ v o t e 3 2 , Ack~2, Ack32, C o m m i t 2 , C o m _ i n d t 2, C o r n _ ind 32, P r e p a r e - a b o r t 2, A b o r t - d o n e 2, A b o r t ind12, A b o r t _ i n d 3 2 , R e l e a s e d o n e 2} L o 3 = { C o r n _ w a i t 2, C o m _ v o t e ~ 3 , C o r n _ v o t e 2 3 , A c k ~3, A c k 23, C o m m i t 3, Corn _ ind ~3, Corn ind23, P r e p a r e - a b o r t 3, A b o r t - d o n e 3, A b o r t _ i n d , 3 , A b o r t ind23, R e l e a s e _ d o n e 3} A p p l y i n g t h e test g e n e r a t i o n a l g o r i t h m g i v e n in S e c t i o n 3.2, w e g e n e r a t e t h e f o l l o w i n g s y n c h r o n i z a b l e t e s t s e q u e n c e , for t h e q u o r u m - b a s e d p r o tocol, w h i c h is a t r a n s i t i o n tour. f l , h i , ll, a l , b12, b13, e l , f 2 , h2, 12, a2, b21, b2.~, Q , f 3 , h3, 13, a3, b3J, b32, c3, el2, d12, e13, all3, c I , e21, d21, e23, d23, c 2,
e3j, d3~, e32,
d32, c3, g l , k l , Ill, e l , g2, k 2 , /72, c2, g3, k3, 113, c3, m l , F/I, Cl~ m 2 , //2, c2, m 3 , /13 If w e r a n d o m l y c h o o s e a t e s t s e q u e n c e for q u o r u m - b a s e d p r o t o c o l , w e m a y f i n d a test s e q u e n c e w h i c h is not s y n c h r o n i z a b l e . F o r i n s t a n c e , t h e test s e q u e n c e 'a~, b32 . . . . ' is n o t s y n c h r o n i z a b l e , since t h e p a i r o f t r a n s i t i o n s 'a~, b32' is n o t s y n c h r o n i z able.
[1] G.v. Bochmann, R. Dssouli and J. Zhao, Trace analysis for conformance and arbitration testing, IEEE Trans. Software Eng. SE-15 (11) (1989) 1347-1356. [2] G.v. Bochmanm A. Das, R. Dssouli, M. Dubuc, A. Ghedamsi and G. Luo, fault models in testing, IFIP Transactions, Protocol Testing Systems IV (Proc. IkTP 7"('6 Fourth Int. Workshop on Protocol Test Systems ), ed. by J. Kroon, R.J. Heijink and E. Brinksma (North-Holland Amsterdam, 1992) 17-30. [3] S. Boyd and H. Ural, The syncronization problem in protocol testing and its complexity, hrJbrmat. Process. Lett. 40 (8)(Nov. 1991) 131-136. [4] S. Ceri adn G. Pelagatti, Distributed Batabases: Principh's and Systems (McGraw-Hill, New York, 1984). [5] W.-H. Chen, C.-S. Lu, J.-T. Wang and R.-J. Lee, Contrained Chinese Postman Problem with its application on synchronizable protocol test generation, J. lnformat. Eng. 6 (1990) 149-157. [6] T.S. Chow, Testing software design modeled by finitestate machines, IEEE Trans. S~(tware Eng., SE-4 (3l (1978). [7] J. de Meer, V, Heymer, J. Burmeister, R. Hirr and A. Rennoch, Distributed testing, Participants Proc. Int. Workshop on Protocol Testing Systems. The Netherlands (Oct. 15-17, 1991)IV43 51. [8] R. Dssouli and G. v. Bochmanm Conformance testing with multiple observers, Proc. IFIP Workshop on Prot. Specification, Testing and Validation (North-Holland, Amsterdam, 1986) 217-229. [9] S. Fujiwara, G. yon Bochmann, F. Khendek, M. Amalou and A. Ghedamsi, Test selection based on finite state models, IEEE Trans. Sofiware Eng. SE- 17 (6) (June 1901 ) 591-603. [10] A. Ghedamsi and G. v. Bochmann, Diagnositic tests for finite state machines, Proc. 12th bzt. Conf. on Distributed Computing Systems, Japan (1992) (IEEE Computer Society Press) 244-251. [11] M. Herlihy, A quotum-concensus replication method for abstract data types, ACM Trans. Comput. Syst. 4 (1) (Feb. 1986) 32-53. [12] D. Hubbard, Determinisitc-Execution-Testing of FSMbased protocols, A T&T Technical J. ( 1) (1990) 119-28. [13] 1SO/TC97/SC21, OSI conformance Testing Methodology and Framework - Parts 1-5, ISO, 1991. [14] Z. Kohavi, Switching and Finite Automata Theory (McGraw-Hill, New York, 1978). [t5] D.Y. Lee and J.Y. Lee, A well-efined Estelle specification for the automatic test generation, 1EEE Trans. Cornput. 40, (4) (April 1991) 526-542. [16] G. Luo. G. v. Bochmann and A. Petrenko, Test selection based on communicating nondeterministic finite state machines using a generalized Wp-method, to appear in IEEE Trans. Software Eng. 20 (2) (1994) 149 162. [17] S. Naito and M. Tsunoyama, Fault detection for sequential machines by transition tours, in Proc. IEEE Fault Tolerant Comput. Co~(/': ( 1981 ).
132
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
[18] A. Peternko, Checking experiments with protocol machines, IFIP Trans. Protocol Testing Systems I V (Proc. IFIP TC6 Fourth Int. Workshop on Protocol Test Systems, 1991), ed. by J. Kroon, R.J. Heijink and E. Brinksma (North-Holland, Amsterdam, 1992) 83-94. [19] K. Sabnani and A.T. Dahbura, A protocol test generation procedure, Computer Networks and 1SDN Syst. 15 (4) (1988) 285-297. [20] B. Sarikaya and G. v. Bochmann, Synchronization and specification issues in protocol testing, IEEE Trans. Commun. COM-32 (4) (April 1984) 389-395. [21] R. Tarjan, Depth-first search and linear graph algorithms, SIAM J. Comput. (2) (1972).
Gang Luo received the B.E. degree in computer and electrical engineering from Chongqing University, Chongqing, China, in 1982, the M.E. degree in computer science and the Ph.D. degree in computer science and telecommunication from Beijing University of Posts & Telecommunications, Beijing, China, in 1987 and in 1989, respectively. He has been a postdoctoral fellow with ' I D A C O M - N S E R C - C W A R C Industrial Research Chair on Communication Protocols' at UniversitE de MontrEal, MontrEal, Canada, since 1990. He was an assistant engineer at No. 6 research institute of China Ministry of Electronics Industry, from 1982 to 1984. He served as a lecturer and a leader of the telecommunication software research group at Beijing University of Posts & Telecommunications from 1989 to 1990. His current research interests include telecommunication software engineering, software and protocol testing, formal specification techniques, software testability, software verification, telecommunication network protocols and multimedia application. Rachida Dssouli received the Doctorat d'UniversitE degree in computer science from the UniversitE PaulSabatier of Toulouse, France, in 1981, and the Ph.D. degree in computer science in 1987, from the University of MontrEal, Canada. From 1981 to 1989 she worked as professor at the UniversitE Mohamed ler, Oujda, Morroco. She is currently an Assistant professor at the University of MontrEal. Her research area is software engineering and her research interests include software specification, software testability, protocol testing and observation.
Gregor v. Bochmann received the Diploma degree in physics from the University of Munich, Munich, West Germany, in 1968 and the Ph.D. degree from McGill University, Montreal, P.Q., Canada, in 1971. He has worked in the areas of programming languages, compiler design, communication protocols, and software engineering and has published many papers in these areas. He holds the I D A C O M - N S E R C - C W A R C chair of industrial research on communication protocols in the DEpartement d'informatique et de recherche opErationnelle, UniversitE de MontrEal, MontrEal. His present work is aimed at design methods for communication protocols and distributed systems. He has been actively involved in the standardization of formal description techniques for OSI. From 1977 to 1978 he was a Visiting Professor at the Ecole Polytechnique FEdErale, Lausanne, Switzerland. From 1979 to 1980 he was a Visiting Professor in the Computer Systems Laboratory, Stanford University, Stanford, CA. From 1986 to 1987 he was a Visiting Researcher at Siemens, Munich. Abderrazak Ghedamsi received the B. Sc. and M. Sc. degrees in computer science from the University of Ottawa, Canada, in 1985 and 1987, respectively, and the Ph.D. degree in computer science in 1993, from the University of MontrEal, Canada. From 1987 to 1988 he worked as a representative and research contributor for Mitel Corp. at the national and international standard committees in the area of computer networks, Ottawa, Canada. He is currently an Assistant professor at Concordia University. His research areas are the data communication and software engineering. His main research interests include the test, analysis and diagnostic of communication protocols and general software, and protocol synthesis.
Pallapa Venkataram received Ph.D. degree, Information Science, from the University of Sheffield, United Kingdom, in 1986. He is currently an Assistant Professor in Dept. of Electrical Communication Engineering, Indian Institute of Science, Bangalore, India. From 1980 to 1982 he was a Deputy Manager (computers) with Hindustan Aeronautics Ltd., Bangalore. His research fields include protocol engineeing, network management, distributed systems and knowledge based systems.
Computer Standards & Interfaces 16 (1994) 119- ! 32
Test generation with respect to distributed interfaces * Gang Luo *, Rachida Dssouli, Gregorv. Bochmann, Pallapa Venkataram *, Abderrazak Ghedamsi "~ Departement d'lRO, UniL~ersit~de Montreal, C.P. 6128, Succ. A, Montreal, P.O., H3C 3J7, Canada
Abstract
In the area of testing communication systems, the interfaces between systems to be tested and their testers have great impact on test generation and fault detectability. Several types of such interfaces have been standardized by the International Standardization Organization (ISO). A general distributed test architecture, containing distributed interfaces, has been presented in the literature for testing distributed systems based on the Open Distributing Processing (ODP) Basic Reference Model (BRM), which is a generalized version of ISO distributed test architecture. We study in this paper the issue of test selection with respect to such a test architecture. In particular, we consider communication systems that can be modeled by finite state machines with several distributed interfaces, called ports. A test generation method is developed for generating test sequences for such finite state machines, which is based on the idea of synchronizable test sequences. We then apply this method to generate test sequences for a so-called quorum protocol. Starting from the initial effort by Sarikaya, a certain amount of work has been done for generating test sequences for finite state machines with respect to the ISO distributed test architecture, all based on the idea of modifying existing test generation methods to generate synchronizable test sequences. However, none studies the fault coverage provided by their methods. We investigate the issue of fault coverage and point out a fact that the methods given in the literature for the distributed test architecture cannot ensure the same fault coverage as the corresponding original testing methods. We also study the limitation of fault detectability in the distributed test architecture. Key words." Distributed interfaces; ISO standard test architectures; Finite state machines; Protocol conformance testing; Protocol engineering; Software testing; Synchronizable test sequences
* Corresponding author. Now with Dept. of Computer Science, University of Ottawa, 150 Louis Pasteur Priv., Ottawa, Ont KN 6N5, Canada. Email: [email protected] * This work was supported by the IDACOM-NSERC-CWARC Industrial Research Chair on Communication Protocols at the University of Montreal (Canada).
* He is now with Dept. of Electrical Communication Engineering, Indian Institute of Science, Bangalore-560 012, India; email: [email protected] He is now with Dept. of Computer Science, Conconlia University, Montreal, Canada.
0920-5489/94/$07.00 © 1994 Elsevier Science B.V. All rights reserved SSDI 0920-5489(93)E0072-A
120
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
1. Introduction
A general distributed test architecture where the IUT (implementation under test) contains several distributed ports has been studied in [7] for testing distributed systems. It is based on the Open Distributing Processing (ODP) Basic Reference Model (BRM) (see Fig. 4). In this architecture, the I U T contains several distributed interfaces, called ports (i.e. points of control and observation); the testers cannot communicate and synchronize with one another unless they communicate through IUT, and no global clock is available in the system. This could be a test architecture of a communication network with n accessing nodes, where the testers reside in these nodes. When n = 2, this general distributed test architecture is reduced to the ISO distributed test architecture for communication protocol testing. We study in this paper test selection methods with respect to a general distributed test architecture. Usually, in the so-called local test architecture developed by ISO, the specifications of communication protocols are first abstracted into state machines [15], then test cases are generated from the resulting machines. A number of methods have been developed to generate test cases for finite state machines (FSMs) [9,18,6,19,17] or for nondeterministic FSMs [16]; however, they are not directly applicable to the distributed test architecture, because of the synchronization problem between distributed testers. In distributed test architectures, testing is relatively difficult because certain problems of synchronization between the testers may arise during the application of test sequences. To solve this problem, with respect to the ISO distributed test architecture where there are only two ports, an approach for test generation has been developed in [20] by modifying the existing test generation methods for FSMs such as the transition tour [17], the DS-method [14], and the W-method [6] such that the resulting sequences are so-called synchronizable test sequences. We develop in Section 3 an approach to generating test sequences for FSMs with n distributed ports (n _> 2), with respect a general distributed
test architecture (see Fig. 4), after defining and explaining several concepts related to distributed test architecture and FSMs. Our approach is a generalized version of the approach given in [20]. As an application example, we generate test sequences using our method for a so-called quorum-based protocol in the appendix. Starting from the initial effort by Sarikaya, a certain amount of work has been done for generating test sequences for finite state machines for ISO distributed test architecture, all based on the idea of modifying existing test generation methods to generate synchronizable test sequences. Chen [5] and Boyd [3] have studied the computation complexity issue of the approach of [20], and they have pointed out that there is no efficient (i.e. polynomial time) algorithm for obtaining a minimum length synchronizable test sequences from a given FSM. A heuristic approach to obtaining synchronizable test sequences is given in [5]. Another approach to obtaining synchronizable test sequences, presented in [12], is to insert additional synchronization statements (i.e. software probes) into the IUT. This approach is not applicable when the conformance testing is conducted by a second party other than the implementor as the party does not have access to the internal information of the IUT. Although so much work has been done on the different aspects of synchronizable test sequence generation, yet none studies the fault coverage provided by their methods. We explore in Section 4 the issue of fault coverage and discuss the observability and fault detectability in the distributed test architecture. We come out with a fact that these modified methods in the literature cannot ensure the same fault coverage as the corresponding original testing methods. For instance the modified transition tour cannot detect all output faults for FSMs with two distributed ports, although transition tour can detect all output faults for the FSMs with only one port. A class of output faults may remain undetected if the modified transition tour is applied to an FSM with two ports. Unfortunately, none of the former articles on generating synchronizable test sequences [20,12,5,3] had realized such weakened fault coverage. We also pre-
121
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
sent an improved approach that has a better fault detection power. We finally show a limitation of fault coverage in the distributed test architecture that not all faults of the I U T are detectable, showing the impossibility of complete fault coverage. Therefore, the testability of IUTs under the distributed test architecture need to be studied. We conclude in Section 5 by discussing some future research issues.
UpperTester LowerTester ImplementatiOnIuT]
L PCO 2'
:'ql
[ communicationservice Abstract testing framework for a distributed architecture
Fig. 2. Distributed test architecture.
We discuss in this section different test architectures used for testing communication systems, and finite protocol machine (i.e. finite state machine) model related to these architectures. We also present a fault model for finite state machines.
located in the same place. In the distributed test architecture, the test system is divided into socalled upper and lower testers which access the PCOs 1 and 2 of the IUT, respectively, as shown in Fig. 2. The lower interface PCO 2 is accessed over distance and indirectly through the underlying communication service. The coordinated test architecture is similar to the distributed test architecture, and the only difference between the two is that the former has some kind of coordination between u p p e r and lower testers, established using a so-called test coordination protocol through a (possibly separate) communication channel between upper and lower testers. The remote test architecture (see Fig. 3) corresponds to the distributed test architecture where only a lower tester is used; instead of an upper tester, the I U T may include a stack of several protocol layers above the layer being tested.
2.
2.1. Distributed test architectures for communication systems 2.1.1. ISO standard test architectures for protocol conformance testing ISO has defined four types of test architectures for protocol conformance testing, called local, distributed, coordinated and remote test architectures, respectively. Local test architecture corresponds to traditional software testing, as shown in Fig. 1, where PCO and I U T refer to point of control and observation, and the implementation under test, respectively. In this architecture, the two PCOs of the I U T can be viewed as a single port since the I U T and the tester are
2.1.2. A general distributed test architecture As mentioned before, the ISO test architectures only deal with an individual single protocol entity. However, to test the overall properties of
I Lower Tester
Tester
ImplementationRiT ]
L
Implementation IUT
,-II Fig. 1. Local test architecture.
Fig. 3. Remote test architecture.
G. Luo et aL / Computer Standards & Interfaces 16 (1994) 119-132
122
nize with one another unless they communicate through the IUT, and (iii) no global clock is available. In testing a communication network, this general architecture can model a communication network with n accessing nodes; the IUT models the communication network, and the testers are located in these nodes. When n = 2, the above model is reduced to the ISO standard distributed test architecture, which is shown in Fig. 2. 2.2. Finite state machines with n ports
A network with distributed terminals sometime can be modeled as a finite state machine with several ports. We define in the following the concept of multi-port finite state machines, which is a generalization of finite state machines with two ports given in [20].
Fig. 4. A general distributed test architecture.
distributed database systems and communication networks, we may face a general distributed test architecture, as shown in Fig. 4. In this general distributed test architecture, (i) the IUT contains several ports, (ii) the testers cannot communicate and synchro-
Definition. A multi-port finite state machine with n ports (np-FSM) is defined as a 6-tuple (St, Z, F, T, O, s o) where n > 1 and (1) St is a finite set of labels, called states. (2) ~ is a n-tuple (Lil, Li 2. . . . , L i , ) where Li~
L/t={ I },
L/2={2},~ L
o
z
~r =
t
c
2/
,
~
~
] Li3=13}, ~
Lo3={d},
The initial state is A
Fig. 5. An example of 3p-FSM under distributed test architecture.
G. Luo et aL / Computer Standards & Interfaces 16 (1994) 119-132
is a set of inputs for port k , and L i k C~L i j = (~ denotes the empty set) when k 4:j and k , j = l , 2 . . . . . n. Furthermore, we assume L i = Li 1 U Li 2 u . . . ULi,,. (3) F is a n-tuple ( L o l , L o 2 , . . . , L o n) where L o k is a set of outputs for port k, and L o k c~ Loj = fJ when k 4=j, with k, j = 1, 2 . . . . . n. Furthermore, we assume L o = {(al, a 2. . . . . a m) ]3LOkl, Lok2 . . . . . LOkm such that a i c L o k I, a 2 E L O k 2 , . . . ,
a m
E LOkm,
l _ < k l < k 2 < ... < k m < n , and 1 _< m < n} where ( a l , a 2 . . . . . a m) is said to be an output tuple. (4) T is a transition function: . ~ S t where ~ G St × Li, (5) O is an output function. _~ ~ L o U {e} for the .~ in (4) where e stands for the empty output. (Note that the FSM is completely specified if .~ = St x Li). (6) s o ~ St is the initial state. In other words, a np-FSM has n ports, say P~, P 2 , . . . , P , , ; each port Pk is associated with a set of inputs L i k and a set of outputs L o k. For any two different ports, there are no common inputs (outputs) in the two associated input sets (output sets). A np-FSM can also be represented by a directed graph in which the nodes are the states and the directed edges are transitions linking the states. Fig. 5 shows an example of 3p-FSM under the distributed test architecture. From this definition, a np-FSM is deterministic since a state and an input uniquely determines a next state and an output tuple; but it is not necessarily completely specified in that not every s t a t e / i n p u t pair has a defined next state and an output tuple. Let a ~ Li, b ~ L o , and P, Q ~ St; we write P - a / b -~ Q to represent that T ( P , a) = Q and O ( P , a) = b. P - a / b ~ Q is called a transition f r o m P to Q with the label a / b . For the example of 3p-FSM shown in Fig. 5, we have A 1 / ( a , c ) -~ B. The np-FSMs can model a communication network with n terminals where each terminal is
123
viewed as a port. This model could be useful for network testing and verification. When n = 2, 2p-FSMs are finite state machines defined in [20,12,3], which are viewed as IUTs under the distributed test architecture. 2.3. Fault model
We present a fault model for np-FSMs. Like other state machine based test generation, fault models serve as a guide to test generation and as a basis for test coverage analysis, as described in [2]. Let SP and I U T be two np-FSMs, representing a specification and its implementation, respectively. Assume that they have the same X and F. Then the fault types are defined as follows: (1) Output fault: We say that IUT has output faults if SP can be obtained from I U T by modifying the outputs of one or more transitions in IUT. (2) Transfer fault: We say that IUT has transfer faults if SP can be obtained from I U T by modifying the end states of one or more transitions in IUT. (3) Hybrid fault: We say that I U T has hybrid faults if SP can be obtained from I U T by changing the outputs a n d / o r the end states of one or more transitions, in IUT.
3. Synchronizable test sequence generation Although a number of test generation methods have been developed based on FSMs, they cannot be applied to the FSMs with two or more distributed ports directly, as pointed out by [20], since a so-called synchronization problem exists in the distributed test architecture. We generalize in this section the concept of so-called synchronizable test sequence given in [20] for 2p-FSMs to the case of np-FSMs, dealing with the synchronization problem. Based on the generalized concept, we present an approach to modifying the existing test generation methods for FSMs, such as the transition tour [17], the W-method [6], the DSmethod [14] and so on, to obtain synchronizable test sequences for np-FSMs. For certain protocol
124
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
specifications, it is impossible to avoid synchronization problems; we use techniques for testing nondeterministic FSMs to handle such specifications.
3.1. Synchronization problem and synchronizable test sequences Under the architecture given in Fig. 4, the testers are distributed over several sites. They are only synchronized through the interactions with the implementation. In this situation, considering two consecutive transitions t l and t2 of a given np-FSM I (n > 2), one of the testers is said to face a synchronization problem if this tester did not take part in the first transition and if the second transition requires that it sends a message to the machine I. For example, consider the 3p-FSM shown in Fig. 5, tester3 faces a synchronization problem when considering the consecutive transitions t l and t7. tester3 is supposed to send the input 3 to the machine after the t l has been executed;however, since t 1 neither receives any input from, nor sends any output to the tester3 (i.e. tester3 did not take part in the transition tl), tester3 does not know whether t l has been executed, i.e. a synchronization problem. We now define the concept of synchronizable test sequences for np-FSMs, n > 2, for handling the synchronization problem of general np-FSMs. Given a np-FSM I with the ports 1, 2 . . . . . n, we require the following concepts for the ease of presentation. Definition Interaction ports (IP) of a given transition [pi, PO]. Let pi ~ {1, 2 , . . . , n} and PO c_ {1, 2 . . . . . n} (n is the number of ports). [pi, PO] is said to be an interaction port (IP) of a given transition t if t receives an input from the port pi and sends to each port in PO an output (if PO = ¢, t does not send any output). Definition Synchronizable test sequences. Given an ordered pair of transitions t l and t2 of I, let [pi, PO] and [pi', PO] be their IPs, respectively. t l and t2 are said to be synchronizable if (1) pi = pi', or (2) pi' ~ PO.
A given test sequence is said to be synchronizable if any two consecutive transitions of the sequence are synchronizable. Formerly, people define the synchronizable test sequences for 2p-FSMs by means of the so-called basic transition list [20]. However, for np-FSMs, a generalized version of this list could be too large to handle. The concept of interaction ports provides a nice means to define the synchronizable test sequences, having avoided a cumbersome presentation. As an example, for the 3p-FSM shown in Fig. 5, the IPs of the transitions t l and t2 are [1, {1, 2}] and [2, {3}], respectively; the transitions t l and t2 are synchronizable. It is easy to see that testers will not face any synchronization problem if synchronizable test sequences are used. Therefore, it is desirable to generate synchronizable test sequences for testing np-FSMs.
3.2. Generating synchronizable test sequences Guided by the idea of synchronizable test sequences, we present a modified transition tour such that the resulting test sequences are all synchronizable, using an approach similar to the one given in [20]. The other existing test generation methods for FSMs can be modified to obtain synchronizable test sequences for np-FSMs, in a similar manner. Any graph traversal algorithm such as the one given in [21] can be modified to obtain a transition tour. Assume that an algorithm TT produces a transition tour; we present in the following a procedure for generating synchronizable transition tours, by modifying the algorithm TT.
3.2.1. Generating synchronizable transition tour Each transition t to be added to the sequence by the algorithm TT is first checked whether it forms a synchronizable pair together with the last transition tt of the sequence; that is, assuming the IPs of the transitions tt and t to be [pi, PO] and [pi', PO'], respectively, check whether pi = pi' or (2) p i ' ~ PO are true. If the transitions tt and t are not synchronizable, a different transition from the present state is considered. If no
125
G. Luo et al. / Computer Standards& htterfaces 16 (1994) 119-132
suitable transition exists from the present state, the algorithm TT backtracks to the previous state, continuing the tour from there in a different way. This process continues until all the transitions of the machine are covered. For example, Fig. 6(b) shows a synchronizable transition tour for the 2p-FSM shown in Fig. 6(a). Furthermore, for the example of 3p-FSM shown in Fig. 5, a synchronizable transition tour is ' t l , t2, t3, t4, t5, t6, t7, t8, t9'. As an application example, we apply this approach to a socalled quorum protocol in the appendix. Using an approach similar to the above, one can obtain the test generation methods for npFSMs by modifying the DS-method [19], the Wmethod [6], the U I O - m e t h o d [19] the W p - m e t h o d [9] and the Petrenko method [18] such that the resulting test sequences are all synchronizable. We note that for certain np-FSMs, given a transition, there may not exist any synchronizable test sequence that can force the machine to traverse this transition. Such an example was given in [20] for a 2p-FSM. In this case, one cannot obtain any test suite using the modified methods.
4. Fault coverage In the distributed test architecture, since there is no other synchronization channel instead of 1UT, the observability and fault detectability is weaker than that in the local test architecture. Therefore, it is difficult to achieve good fault coverage. We find that the modified methods presented in [20] do not ensure the same fault
0/
~
>
Lil = 0 }, Li2 = { 1 } Lo! = b } , l.o2 = { a , c } (a)
coverage as the corresponding original testing methods; and we also present an improved approach that provides better fault coverage. We finally show that no method can ensure full fault coverage for finite state machines in the distributed architecture.
4.1. Observability and fault detectability For a system (software or communication system) that receives inputs and produces outputs, observability refers to the ease of determining if specified inputs affect the outputs; fault detectability refers to the ease of detecting specified fault. Observability and fault detectability of I U T vary in different test architectures. Consider the ISO test architectures presented in the earlier section. An I U T has the highest observability and fault detectability in the local test architecture, and has lower ones in the coordinated, distributed architectures, and the lowest one in the remote test architecture. If one associates each test architecture (local, coordinated, distributed and remote test architecture) with a set of detectable faults (S 1, S c, Sj, St), as shown in [8], the fault detectability in these architectures is reflected by the following:
S,c_Sac_S, c_S l. In the distributed testing architecture, the observability and fault detectability of I U T are limited [1]; therefore, it is difficult to achieve good fault coverage in the distributed test architecture, which is further explored in the following subsections.
Input sequence:
0, 1, 1, 0
Resulting global Port1: sequence: Port 2: PIs:
0[
[ 1
11
I
l
I
c
Fig. 6. An example of a 2p-FSM.
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
126
4.2. A class of undetectable output faults by synchronizable test sequences For FSMs, as we know, the transition tour [17] can detect all output faults if no transfer faults occur. However, for 2p-FSMs, a transition tour which is a synchronizable test sequence does not necessarily detect all output faults. For example, the 2p-FSM shown in Fig. 7(a) is a faulty implementation of the 2p-FSM shown in Fig. 6(a); and it has only output faults. The input sequence '0, 1, 1, 0' is a synchronizable transition tour, as described in Fig. 6(b). When the input sequence '0, 1, 1, 0' is applied to the faulty implementation, we obtain the global sequence shown in Fig. 7(b). Although the global sequence for the original 2p-FSM is different from the global sequence for the faulty one, the difference cannot be seen from the two local ports U and L since no global clock is assumed in the distributed test architecture. Therefore, a modified transition tour does not necessarily detect all output faults for 2pFSMs. Using similar arguments, it is easy to prove that the modified W-method and DS-method [20] do not necessarily detect all output and transfer faults for 2p-FSMs. [12] presented a method to solve the synchronization problem by instrumenting probes into the IUT. However, since the method only guarantees that input sequence is synchronized, the above problem still exists. Furthermore, the instrumentation of the IUT is not permissible in the conformance testing when the testing is con-
t 1: O/
~
b>
ducted by a second party other than the producer. We formalize the class of output faults which may remain undetected by the modified methods as follows: For two consecutive transitions, say t l and t2, in a given specification S, the faulty implementation I can be obtained from S by removing an output from one of the two transitions and adding the output to the other transition. We call this class of output faults outputshifting faults. We present in the following an outline of an approach which can detect this class of output faults. The outline of the approach: (1) Generate a set of synchronizable test sequences, say H, by using the approach of [20], with respect to one of the test generation methods for FSMs such as the transition tour [17], the W-method [6], and so on. (2) Find a set of all transition pairs, say/2, along the paths caused by applying the sequences of H, each pair of which may have an outputshifting fault which is undetectable using /7. (3) If /2 is empty, stop. Otherwise, add a set of additional synchronizable test sequences to /7, or concatenate some synchronizable subsequences to the sequences in H, such t h a t / 7 can ensure the absence of output-shifting faults in the transition pairs o f / 2 . Go to Step 2. It is not very difficult to give a detailed procedure for Steps 2 and 3, but under what condition an efficient procedure exists and what is efficient procedure, is the work of future research. The
Input sequence:
O, 1, 1,0
Resulting global sequence:
lOB [ ' ~ l b l
I
lalllclll
(b) (a) Fig. 7. An example of output-shiftingfault in a 2p-FSM.
ib[ Otbl
I
i I
127
G. Luo et aL / Computer Standards & Interfaces 16 (1994) 119-132
added by Step 3
Resulting global sequence:
0
bI I Ibm 0 1 b l l a 1 IIc 1 I I I i
tl, t3, f~ = {
t2,
t4
OIbl
I
I
IblOIblO,
laltlclll t 1,
t3,
Fig. 8. The example for Step 2.
I t2,
blO, b l O , bl
I I t4,
l I tl,
t4,
, I
lal]lcl t 1,
t3
Fig. 9. The example lor Step 3.
worst case computation complexity of the above procedure is not polynomial since the procedure contains the generation of synchronizable test sequences; and no polynomial algorithm exists for the generation [5,3]. We now explain the approach using the example of Fig. 6(a). Suppose that the transition tour is of our interest. Fig. 6(b) shows the resulting test sequence after Step 1. Fig. 8 shows the result of Step 2; the two arrows in the figure indicate the two possible outputshifting faults which cannot be detected by the test sequence. Fig. 9 shows the result of Step 3; the test sequence can ensure the absence of any output-shifting faults shown in Fig. 8. The transition string 'tl,t4' ensures the absence of outputshifting fault in ( t l , t3), and the rightmost tran-
sition t3 of the test sequence ensures the absence of output-shifting fault in (t3, t2). Although the improved approach detects more faults than the original, it does not necessarily guarantee the total absence of faults. Therefore, the approach is still heuristic "in nature since no precise fault coverage is easily given under the given fault model. We also note that it is impossible to guarantee the total absence of faults in the distributed test architecture, as described in the following section. 4.3. Impossibility o f complete test cot:erage in a distributed test architecture
We show in the following that, in the distributed test architecture, no test suite (i.e. a set
hi
The initial state is A Liu ={ a,b }, LiL ={c } Lov ={ d,e }, LoL = { f,g}
Fig. 10. An example of 2p-FSM.
128
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
of test sequences) can ensure the total absence of the faults specified in the fault model. In other words, for a np-FSM, not all faults specified in the fault model are detectable. We first note: no method that is based on the concept of synchronizable test sequences can ensure full fault coverage for all np-FSMs. The reason is that, for certain np-FSMs, given a transition, there may not exist any synchronizable test sequence that can force the machine to traverse this transition. Such an example was given in [20] for a 2p-FSM. One may argue that, for the np-FSMs where some transitions cannot be forced to be traversed by any synchronizable test sequences, to traverse these transitions, certain test sequences can be applied to the machines repeatedly up to a sufficient number of time. Then the probability that not all these transitions are exercised at least once, can usually be reduced to close to zero (note: this is a common assumption for testing nondeterministic finite state machines [16]). We argue that, even using such an approach and the assumption, still no test suite can ensure full fault coverage. We prove this using the 2p-FSM shown in Fig. 10. For the sake of convenience, we introduce in the following several terms for 2p-FSMs. A test case is denoted as two I / O sequences [o-u, O-L] such that o"U consists of the inputs and outputs of the port U, and O-L of the port L, respectively. We say that a 2p-FSM I passes a test case [o-u, o-L] if there is a path p in I such that when p is traversed, o-u and O-L will be observed at the ports U and L, respectively. A fault in the implementation I is detectable against its specification S if and only if there exists a test case [o-u, °'L] such that S passes [o-u, o-L] and I does not. Consider the example of Fig. 10, let the 2pFSM be S (where plain line transitions make the machine complete). Assume that I is a corresponding faulty implementation which results from changing the output ' f ' of the transition t3 to 'g' (see the dashed transition). The transition t3 in I has an output fault. We argue that the fault is not detectable. We note that all the paths which start from the initial state A and traverse the faulty transition t3, begin with the transition
string 'tl, t2, t3'; and t3 does not appear in the rest of the paths any more. The input sequence pair for the ports U and L, corresponding to the path 'tl, t2, t3', is [a.b, c]. Since no synchronization exists between the ports U and L, the [a.b, c] may cause the machine to be executed along one of the three paths: 'tl, t2, t3', 'tl, t5, t7', 't4, t6, t7'. The transitions of these paths are indicated by bold lines shown in Fig. 10. The total of two test cases, [a/d.b/e, c/f] and [a/d.b/e, c/g] are derived from the three paths. If we assume on the contrary that the fault in t3 is detectable, it is not difficult to see that the fault must be detected using the two test cases [a/ d.b/e, c/f] and [a/d.b/e, c/g]. However, from Fig. 10 both S and I pass the test cases [a/d.b./e, c/f] and [a/d.b/e, c/g]. Therefore, the fault is not detectable. Nevertheless, all faults are detectable if synchronization exists between the distributed ports or there is a global clock - in this case, the test generation methods for FSMs can be applied directly. Therefore, the testability of IUTs under the distributed test architecture should be studied.
5. Conclusion
Test architectures represent interfaces between systems to be tested and their testers; and they are important topics in the International Standardization Organization (ISO). A general distributed test architecture, which is a generalized version of the ISO distributed architecture, has been presented in the literature for testing distributed systems based on the Open Distributing Processing (ODP) Basic Reference Model (BRM). Based on this test architecture, for the systems modeled by finite state machines, we developed a test generation method, and applied this method to generate test sequences for a so-called quorum protocol. We investigated the issue of fault coverage. We pointed out that the methods given in the literature for the distributed test architecture, modified from certain existing methods, cannot ensure the same fault coverage as the corre-
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
120
each node uses a quorum-based protocol
no(~
node 3 Fig. 11. The architecture of the distributed system using quorum-based protocol.
Fig. 12. The test architecture for testing a quorum-based protocol.
sponding original testing methods. We also studied the limitation of fault detectability in the distributed test architecture. The distributed test architecture also gives rise to new problems of FSM based diagnosis, and the diagnosis method of FSMs of one port [10] may be modified for np-FSMs by applying the concept of synchronizable test sequences.
(1) Each port, i, has associated with it a number of votes, ui, and ~'i is a positive integer. (2) Let V indicate the sum of the votes of all ports of the distributed system. (3) A transaction must collect a commit quorum Vc before committing. (4) A transaction must collect an abort quorum Va before aborting. (5) Va + Vc > V.
Appendix: Test generation for a quorum-based protocol To illustrate the method we have taken a quorum-based protocol which is used for nonblocking of resources and increasing the distributed system's availability [11,4]. We generate test sequences for this protocol. Fig. 11 gives the architecture of a distributed system that use a quorum-based protocol. The test architecture for testing this protocol is shown in Fig. 12. The basic idea of the protocol is that a majority of nodes in the distributed system must agree on the commit or abort of a transaction before the transaction is committed or aborted. The weights which are assigned to the ports are usually called votes, since they are used when a port 'votes' on the commit or abort of a transaction. The basic rules of a quorum-based protocol are:
bf
/
, mi
h nl
Fig. 13. A quorum-based prott)cc~l.
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
130
In p r a c t i c e Va + Vc = V + 1 is t h e m o s t conven i e n t one. E a c h d i r e c t e d e d g e in Fig. 13 r e p r e s e n t s a set o f transitions. F o r e x a m p l e , c o n s i d e r a i a n d dij, w h e r e 0 < i , j < N a n d i 4:j, r e p r e s e n t s N a n d N(N1) t r a n s i t i o n s , respectively. N is a n u m b e r of p o r t s in a d i s t r i b u t e d system. F o r clarity we c h o o s e N = 3 a n d d e s c r i b e the l e g e n d of Fig. 13 in T a b l e 1. Fig. 14 explains t h e set o f t r a n s i t i o n s for a i a n d dij for N = 3. T h e i n p u t a n d o u t p u t sets u s e d at e a c h p o r t a r e t h e following: Li I = {No_com_reqp
No_vote
req~2,
N o _ v o t e _ r e q ~ 3 , C o m _ r e q 1, V o t e _ r e q ~ 2 , V o t e _ r e q 13, V o t e _ cast 12, V o t e _ cast 13, Vc 1, V a 1, T a s k _ execute 1, A b o r t _ req 1, R e l e a s e _ req l, T i m e _ out 1, R e s t a r t _ req 1} L i 2 = {No _ c o m _ r e q 2, N o _ vote _ r e q 21, N o _ v o t e _ req 23, C o m _ r e q 2, V o t e _ req 21, V o t e _ r e q 2 3 , Vote_cast2~, Vote_cast23, Vc 2, V a 2, T a s k _ e x e c u t e 2, A b o r t _ req 2, R e l e a s e _ req 2, T i m e _ o u t 2, R e s t a r t _ req 2} L i 3 = {No _ c o m _ r e q 3, N o _ r e q 31, N o _ v o t e _ r e q 32, C o m _ req 3, V o t e _ r e q 31, V o t e req32, V o t e _ c a s t 3 t , V o t e _ c a s t 3 2 , Vc3, Va3, T a s k _ e x e c u t e 3, A b o r t _ r e q 3, R e l e a s e _ r e q 3, T i m e _ o u t 3, R e s t a r t _ r e q 3}
Table 1 Legend for the protocol in Fig. 13
ai bij cI ce c3
dij eij fl f2 f3 gi hi k1 k2 k3
li mI m2 rn 3
ni
In the following, , j = 1, 2, 3, and i 4:j, and abbreviated I / 0 actions are described in parenthesis. No_com_reqi/e (No commit request from the ith port/e) N o v o t e reqij/e (No vote request from ith port to the jth port/e) Com reql/Com_waitl, Com vot%2, Com votel3 Com_req2/Com wait 2, Com_vote21, Com_votee3 Com_reqa/Com wait> Corn vote31, Com_vote32 (Commit request from ith port/{Commit wait indication to the ith port, Commit vote request broadcast from ith port to the rest}) Vote-reqiJAckji (Vote request from ith port to the jth port/{ack from jth to ith port}) Vote_castij/Ackji (Vote send in favour or against commit from ith port to the jth port/{ack from jth to ith port}) V c l / C ° m m i t l , C°m-indl2, C°m-ind13 Vc2/Commit2, Com ind21, Com_ind23 Vc3/Commit3, Com_ind31 , C o m ind32 (Vc count received in favour of ith port to commit/{commit-execute to ith port, commit indication to the rest}) Vai/Prepare-ab°rti (Va count received against ith port commit/{prepare abort message to ith port}) Task_ execute//e (Task execute by ith port/e) Abort_reql/Abot-donel, Abort_ind12, Abort_ind13 Abort_reqJAbot-done2, Abort ind21, Abort ind23 Abort_req3/Abot-done3, Abort ind32, Abort_ind31 (abort request from the ith port/{abort done to the ith port, abort indication to the rest}) Release_ req i/Release _ donei (Release request from ith port/Release done to the ith port) Time_outl/Abot-donel, Abort_ind12, Abort_ind13 Time out2/Abot-done 2, Abort_ind21, Abort_ind23 Time out3/Abot-done 3, Abort_ind31, Abort ind32 (Time out indication from ith port/{abort done to the ith port, abort indication the rest}) Restart_reqi/e (Restart request from ith port/e)
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
131
References
a2
1
Fig. 14. Description of transition set of Fig. 13. L o t = {Corn ._w a i t ~, C o m _ v o t e z ~ , C o m vote31, Ack2~ , Ack31, C o m m i t s , C o m _ i n d 2 t , C o m _ i n d 3 ~ , P r e p a r e - a b o r t ~, A b o r t - d o n e i, A b o r t _ ind 2~, A b o r t _ ind 3 t, R e l e a s e d o n e 1} Lo2 = { C o r n _ w a i t 2 , C o r n voter2, C o r n _ v o t e 3 2 , Ack~2, Ack32, C o m m i t 2 , C o m _ i n d t 2, C o r n _ ind 32, P r e p a r e - a b o r t 2, A b o r t - d o n e 2, A b o r t ind12, A b o r t _ i n d 3 2 , R e l e a s e d o n e 2} L o 3 = { C o r n _ w a i t 2, C o m _ v o t e ~ 3 , C o r n _ v o t e 2 3 , A c k ~3, A c k 23, C o m m i t 3, Corn _ ind ~3, Corn ind23, P r e p a r e - a b o r t 3, A b o r t - d o n e 3, A b o r t _ i n d , 3 , A b o r t ind23, R e l e a s e _ d o n e 3} A p p l y i n g t h e test g e n e r a t i o n a l g o r i t h m g i v e n in S e c t i o n 3.2, w e g e n e r a t e t h e f o l l o w i n g s y n c h r o n i z a b l e t e s t s e q u e n c e , for t h e q u o r u m - b a s e d p r o tocol, w h i c h is a t r a n s i t i o n tour. f l , h i , ll, a l , b12, b13, e l , f 2 , h2, 12, a2, b21, b2.~, Q , f 3 , h3, 13, a3, b3J, b32, c3, el2, d12, e13, all3, c I , e21, d21, e23, d23, c 2,
e3j, d3~, e32,
d32, c3, g l , k l , Ill, e l , g2, k 2 , /72, c2, g3, k3, 113, c3, m l , F/I, Cl~ m 2 , //2, c2, m 3 , /13 If w e r a n d o m l y c h o o s e a t e s t s e q u e n c e for q u o r u m - b a s e d p r o t o c o l , w e m a y f i n d a test s e q u e n c e w h i c h is not s y n c h r o n i z a b l e . F o r i n s t a n c e , t h e test s e q u e n c e 'a~, b32 . . . . ' is n o t s y n c h r o n i z a b l e , since t h e p a i r o f t r a n s i t i o n s 'a~, b32' is n o t s y n c h r o n i z able.
[1] G.v. Bochmann, R. Dssouli and J. Zhao, Trace analysis for conformance and arbitration testing, IEEE Trans. Software Eng. SE-15 (11) (1989) 1347-1356. [2] G.v. Bochmanm A. Das, R. Dssouli, M. Dubuc, A. Ghedamsi and G. Luo, fault models in testing, IFIP Transactions, Protocol Testing Systems IV (Proc. IkTP 7"('6 Fourth Int. Workshop on Protocol Test Systems ), ed. by J. Kroon, R.J. Heijink and E. Brinksma (North-Holland Amsterdam, 1992) 17-30. [3] S. Boyd and H. Ural, The syncronization problem in protocol testing and its complexity, hrJbrmat. Process. Lett. 40 (8)(Nov. 1991) 131-136. [4] S. Ceri adn G. Pelagatti, Distributed Batabases: Principh's and Systems (McGraw-Hill, New York, 1984). [5] W.-H. Chen, C.-S. Lu, J.-T. Wang and R.-J. Lee, Contrained Chinese Postman Problem with its application on synchronizable protocol test generation, J. lnformat. Eng. 6 (1990) 149-157. [6] T.S. Chow, Testing software design modeled by finitestate machines, IEEE Trans. S~(tware Eng., SE-4 (3l (1978). [7] J. de Meer, V, Heymer, J. Burmeister, R. Hirr and A. Rennoch, Distributed testing, Participants Proc. Int. Workshop on Protocol Testing Systems. The Netherlands (Oct. 15-17, 1991)IV43 51. [8] R. Dssouli and G. v. Bochmanm Conformance testing with multiple observers, Proc. IFIP Workshop on Prot. Specification, Testing and Validation (North-Holland, Amsterdam, 1986) 217-229. [9] S. Fujiwara, G. yon Bochmann, F. Khendek, M. Amalou and A. Ghedamsi, Test selection based on finite state models, IEEE Trans. Sofiware Eng. SE- 17 (6) (June 1901 ) 591-603. [10] A. Ghedamsi and G. v. Bochmann, Diagnositic tests for finite state machines, Proc. 12th bzt. Conf. on Distributed Computing Systems, Japan (1992) (IEEE Computer Society Press) 244-251. [11] M. Herlihy, A quotum-concensus replication method for abstract data types, ACM Trans. Comput. Syst. 4 (1) (Feb. 1986) 32-53. [12] D. Hubbard, Determinisitc-Execution-Testing of FSMbased protocols, A T&T Technical J. ( 1) (1990) 119-28. [13] 1SO/TC97/SC21, OSI conformance Testing Methodology and Framework - Parts 1-5, ISO, 1991. [14] Z. Kohavi, Switching and Finite Automata Theory (McGraw-Hill, New York, 1978). [t5] D.Y. Lee and J.Y. Lee, A well-efined Estelle specification for the automatic test generation, 1EEE Trans. Cornput. 40, (4) (April 1991) 526-542. [16] G. Luo. G. v. Bochmann and A. Petrenko, Test selection based on communicating nondeterministic finite state machines using a generalized Wp-method, to appear in IEEE Trans. Software Eng. 20 (2) (1994) 149 162. [17] S. Naito and M. Tsunoyama, Fault detection for sequential machines by transition tours, in Proc. IEEE Fault Tolerant Comput. Co~(/': ( 1981 ).
132
G. Luo et al. / Computer Standards & Interfaces 16 (1994) 119-132
[18] A. Peternko, Checking experiments with protocol machines, IFIP Trans. Protocol Testing Systems I V (Proc. IFIP TC6 Fourth Int. Workshop on Protocol Test Systems, 1991), ed. by J. Kroon, R.J. Heijink and E. Brinksma (North-Holland, Amsterdam, 1992) 83-94. [19] K. Sabnani and A.T. Dahbura, A protocol test generation procedure, Computer Networks and 1SDN Syst. 15 (4) (1988) 285-297. [20] B. Sarikaya and G. v. Bochmann, Synchronization and specification issues in protocol testing, IEEE Trans. Commun. COM-32 (4) (April 1984) 389-395. [21] R. Tarjan, Depth-first search and linear graph algorithms, SIAM J. Comput. (2) (1972).
Gang Luo received the B.E. degree in computer and electrical engineering from Chongqing University, Chongqing, China, in 1982, the M.E. degree in computer science and the Ph.D. degree in computer science and telecommunication from Beijing University of Posts & Telecommunications, Beijing, China, in 1987 and in 1989, respectively. He has been a postdoctoral fellow with ' I D A C O M - N S E R C - C W A R C Industrial Research Chair on Communication Protocols' at UniversitE de MontrEal, MontrEal, Canada, since 1990. He was an assistant engineer at No. 6 research institute of China Ministry of Electronics Industry, from 1982 to 1984. He served as a lecturer and a leader of the telecommunication software research group at Beijing University of Posts & Telecommunications from 1989 to 1990. His current research interests include telecommunication software engineering, software and protocol testing, formal specification techniques, software testability, software verification, telecommunication network protocols and multimedia application. Rachida Dssouli received the Doctorat d'UniversitE degree in computer science from the UniversitE PaulSabatier of Toulouse, France, in 1981, and the Ph.D. degree in computer science in 1987, from the University of MontrEal, Canada. From 1981 to 1989 she worked as professor at the UniversitE Mohamed ler, Oujda, Morroco. She is currently an Assistant professor at the University of MontrEal. Her research area is software engineering and her research interests include software specification, software testability, protocol testing and observation.
Gregor v. Bochmann received the Diploma degree in physics from the University of Munich, Munich, West Germany, in 1968 and the Ph.D. degree from McGill University, Montreal, P.Q., Canada, in 1971. He has worked in the areas of programming languages, compiler design, communication protocols, and software engineering and has published many papers in these areas. He holds the I D A C O M - N S E R C - C W A R C chair of industrial research on communication protocols in the DEpartement d'informatique et de recherche opErationnelle, UniversitE de MontrEal, MontrEal. His present work is aimed at design methods for communication protocols and distributed systems. He has been actively involved in the standardization of formal description techniques for OSI. From 1977 to 1978 he was a Visiting Professor at the Ecole Polytechnique FEdErale, Lausanne, Switzerland. From 1979 to 1980 he was a Visiting Professor in the Computer Systems Laboratory, Stanford University, Stanford, CA. From 1986 to 1987 he was a Visiting Researcher at Siemens, Munich. Abderrazak Ghedamsi received the B. Sc. and M. Sc. degrees in computer science from the University of Ottawa, Canada, in 1985 and 1987, respectively, and the Ph.D. degree in computer science in 1993, from the University of MontrEal, Canada. From 1987 to 1988 he worked as a representative and research contributor for Mitel Corp. at the national and international standard committees in the area of computer networks, Ottawa, Canada. He is currently an Assistant professor at Concordia University. His research areas are the data communication and software engineering. His main research interests include the test, analysis and diagnostic of communication protocols and general software, and protocol synthesis.
Pallapa Venkataram received Ph.D. degree, Information Science, from the University of Sheffield, United Kingdom, in 1986. He is currently an Assistant Professor in Dept. of Electrical Communication Engineering, Indian Institute of Science, Bangalore, India. From 1980 to 1982 he was a Deputy Manager (computers) with Hindustan Aeronautics Ltd., Bangalore. His research fields include protocol engineeing, network management, distributed systems and knowledge based systems.