TTCN: Towards a formal semantics and validation of test suites

EJSEVIER

ComputerNetworks and ISDN Systems 29 ( 1996) 15-47

TTCN: Towards a formal semantics and validation of test suites Finn Kristoffersen a,*, Thomas Walter b~l a Tele Danmark Research, Lyngs@ All.6 2. 2970 H@rsholm, Denmark b Computer Engineering and Networks Laboratory, ETH Ziirich, 8092 Ziirich, Switzerland

Abstract

TICN (Tree andTabularCombinedNotation) is the standardizedtest notationfor the descriptionof OS1conformance tests.Sinceapplicability of TTCN is restricted,work on the definitionof concurrentTEN, an extendedversionof TEN for the specificationof test casesfor multi party testing,hasbeenstarteda few yearsago.We discussdifferent approaches for the definition of an operationalsemantics of TTCN andconcurrentTTCN andwe discussissuesrelatedto the validation of (concurrent) TI’CN test cases.Becausefor the validation of test casesa proper semanticsdefinition is a prerequisite we have developeda semanticsdefinition which utilizes labelledtransitionsystemsas its basicmodel.The applicability of the modelis demonstrated: First, we showhow identifiedincompletenesses and ambiguitiesof TTCN can be solved. Second,we developa validationframeworkthat definesthe necessarymachineryfor the validationof functionalproperties of test casesagainsta formal specification.We illustratehow validationof properterminationand verdict assignments can be performedandhow a testcasecan be validatedagainstan SDL specification. Keywords:

Confomrance testing;Formaldescription techniques; Communication protocols

1. Introduction TTCN and concurrent TTCN [ 14,151 are notations for specification of test suites of OS1 protocol standards. Standardized test suites are produced to ensure that different implementations of the same protocol standard are checked for the same set of requirements [ 12,13,7]. The production of test suites in TTCN and concurrent TTCN has until recently been a manual process * and quite often errors has therefore occurred in test suite specifications. As errors in test suites have a significant impact on the quality achieved by conformance testing, validation of test suites is necessary. Performing test suite validation requires the test suite to be given in a well-defined notation, i.e. a notation which has a formally defined syntax and semantics. A formally defined syntax is necessary to utilize tools to validate the syntactical correctnessof a test suite. A formal semanticsis necessaryfor the validation of a test * Corresponding author. Email: [email protected]. t Email: [email protected]. 2 The authors’ own experiences in BTSI and EWOS projects are that test suite production is still mainly performed manually. Tool support is provided for the editing of test suites and for checking syntax and static semantics. Tools supporting test case generation has been applied only very recently. 0169-7552/96/$15.00 Copyright @ 1996 Elsevier Science B.V. All rights reserved. PUSO169-7552(96)00016-5

16

I? Kristoffrsen.

7: Walter/Computer

Nefworks

and ISDN Systems 29 (1996)

15-47

suite against functional properties; e.g. proper termination (every test run yields a verdict after a finite number of test steps) or whether valid behaviour as specified in a test case is indeed valid behaviour with respect to the protocol specification. Besides for the validation of test suites, a test notation with a formal syntax and semantics is also of importance in order to implement tools for the generation and the implementation of test suites. The latter issues are, however, out of the scope of this paper. In this paper we discuss two approaches for a semantics definition of TTCN. We give a survey of the official semantics definition standardized in Annex B of [ 14,151. This semantics defines an abstract machine that executes test cases. As it has certain weaknesses and does not support formal validation of functional properties we look for an alternative approach to define a semantics for ‘ITCN and concurrent TEN. This alternative semantics has been influenced by a semantics definition for SDL [ 81. The defined semantics is an operational semantics that allow reasoning about functional properties of test cases at different levels of abstraction. The semantics is not meant to replace the standardized semantics of TEN, but rather as an alternative model that supports validation of functional properties of test cases against requirements specified in a formal protocol specification. For the latter kind of validation we describe a “validation framework”. The essential components are a semantical model for SDL [33] and TTCN and concurrent TEN, and an appropriate behaviour relation. The model, called common semantics representation (CSR) [ 30,29,3 1 ] defines an operational semantics for both languages, providing an abstract execution model which allows to “execute” a test case and a specification and to derive their respective input-output behaviour. Using the defined behaviour relation we demonstrate how the CSR may be used to validate the input-output behaviour of a test case with respect to the behaviour of an SDL specification. The outline of the paper is as follows. In the next section we present some related work which has influenced the definition of the validation framework. In Section 3 an introduction to TEN and concurrent TTCN is given. We present different approaches for the definition of a semantical model of TTCN (Section 4). The one selected for the validation framework is described in more detail. Different aspects of test case validation are identified and in Section 6 we present the validation framework for TTCN test case behaviour against behaviour of an SDL specification. Finally the conclusions are presented. A list of abbreviations used is provided at the end of the paper. Throughout the paper we illustrate concepts with concrete examples. For the introduction to TTCN and concurrent TTCN we use as an example a test case from a test suite for the TP protocol. 2. Related work

It is common understanding that defining semantical models for ‘ITCN and concurrent TTCN is necessary in order to enable test generation, test validation and test implementation. In this section we discuss some approaches. 2.1. TEN

semantics

A definition of a ‘ITCN semantics can, in general, be given using two different approaches: direct mapping approach [ 23,561 and semantical model approach [ 3,10,25,14,15]. 2.1. I. Direct mapping approach The basic idea of this approach is to define a mapping of TTCN and concurrent TTCN constructs to constructs of a chosen target language that is “semantics preserving”. Since the target language is assumed to have a formally defined semantics, the semantics of the target language becomes the semantics of TTCN and concurrent TEN, too. In [ 23,5] a direct mapping from TEN to LOTOS is defined and in [ 6,261 a mapping

E Kristoffersen,

T Walter/Computer

Networks

and ISDN Systems 29 (1996)

15-47

17

from TTCN to SDL. Both mappings are based on the identification of conceptual equivalences between the languages. For instance, a TTCN test suite is mapped to a LOTOS specification or SDL specification, a TTCN behaviour tree is mapped to a LOTOS process or SDL system, TTCN PCOs are mapped to SDL channels in the TTCN to SDL mapping, or constraints in TTCN and signal instances in SDL are identified as conceptually equivalent. Although a direct mapping seems to be intuitive there do exist some problems. First, in the TTCN to LOTOS direct mapping PCOs are mapped to gates. But communication at PCOs is asynchronous whereas communication in LOTOS [ 11 J is synchronous, i.e. processes synchronize over gates. So, the mapping of PCOs to gates is not semantics preserving. Secondly, in SDL the following temporal relation between sending a signal instance, receiving a signal instance in an input queue and processing a signal instance by the receiving process instance can be derived: tsenlh&?G t receiving

6 tprvcessing .

Since in ‘ITCN before a set of alternatives is evaluated a snapshot is taken, i.e. all PC0 queues and timer lists are updated, the following temporal relations hold: tstvzdin~ < treceiving

A treceiving

6 tpr*cessing*

but

The above timing relations of events are interpreted as follows: In SDL, sending, receiving and processing of a signal instance are three events which may be performed simultaneously. In TTCN, on the other hand, only two events may be performed simultaneously: sending and receiving or receiving and processing. Thus, whenever PCOs and SDL channels are considered conceptually equivalent the described subtle difference in temporal ordering of events has to be taken into account. The above discussion illustrates that even if the direct mapping approach seems intuitively easy to perform the requirement to make it semantics preserving can be difficult or even impossible to fulfill. As this requirement is basic to validation we believe that another approach shall be taken if the mapping is to serve also for validation purposes. 2.1,2. Semantical model approach Over the past years a number of semantical models for concurrent programming languages have been developed. In the following we consider only a few that we have found suitable for the definition of an operational semantics of ‘TTCN and concurrent TTCN. The selection of semantical models has been performed with the following requirements in mind: First, the semantical model should improve the understanding of problems related to computer aided test generation and test suite validation; secondly, the chosen semantical model should not imply any restrictions on TTCN and concurrent TTCN; and thirdly, the semantical model should support the analysis of semantical properties (e.g. functional properties) of test cases at different levels of abstraction (i.e. with respect to the observable behaviour of entities within a system). Pseudo code (see [ 14,151). The intention is to define an operational semantics in terms of an abstract notation. The pseudo code describes the anticipated behaviour of an abstract machine which executes a test case (see also Section 4.2.1) . Streams and stream process function (see [ 31). This semantical model has been used to define an semantics for SDL. The model is based on functional descriptions of systems using the concept of streams and stream processing function. Streams can be understood as representations of the history of communications between

18

l? Kristoffersen,

T Walter/Computer

Networks

and ISDN Systems 29 (1996)

15-47

components of a system. Stream processing functions are state machines with input and output. Since the approach does not support the reasoning about timed behaviour, e.g. the specification of timers, this approach is not appropriate for the definition of a semantical model of TTCN and concurrent TTCN. Asynchronous communication trees (ACT) (see [ IO] >. ACTS can be regarded as labelled transition systems [20]. States of an ACT are comprised of local states of each process and all pending communications sent out by a process but not yet received by another process. The following events are distinguished: reception of a pending communication by the environment, reception of a communication from the environment, and reception of a pending communication by a process within the system. Dependent on the level of abstraction, not all events mentioned are observable. Particularly, the class of events determined by the reception of a communication by a process within the system is regarded unobservable. As we want a semantical model which allows reasoning on system behaviour at different levels of event abstraction, the ACT model is less appropriate for this purpose. Common semantics (see [25,22]). The SPECS approach is most closely related to our work (Section 4.2.2) although the SPECS objectives are different from those we have in mind. Within the SPECS project a common semantics for LOTOS and SDL have been developed. The common semantics consists of a mathematical representation and a common representation language. The mathematical representation is a specification language with the expressive power equal to LtYTOS and SDL. Its semantics is given in terms of labelled transition systems. Therefore, it is quite similar to our approach. The essential difference is that we interpret TTCN and concurrent ‘ITCN directly in terms of labelled transition systems whereas if we have had followed the SPECS approach, TTCN and concurrent TTCN where mapped to the common representation language whose semantics is defined by its mapping to the mathematical representation.

2.2. Validation of KTN

test cases

Several aspects are to be covered by ‘ITCN test case validation. A first one is checking a TTCN or concurrent ‘ITCN test case for syntactic and static semantics correctness. This kind of validation is well understood. Tools like ITEXr” [ 161, support this kind of validation. Syntax and static semantics checking is only an initial step in TTCN test case validation. In [ 11, the use of TTCN is validated against a set of heuristic rules. The rules aim at assessing the correctness of a TTCN test case with respect to the protocol specification in a formal description technique. A simulator is used for both, TTCN and the formal description technique in order to relate the behaviours possessed by a TTCN test case and a protocol specification. In Section 6 we propose an alternative approach of doing TTCN behaviour validation against SDL specifications. In [4], validation of verdict assignment in TTCN test cases is discussed. The emphasis is on the validation of the inconclusive test verdict, i.e., with respect to the defined test purpose, the test case execution outcome is such that neither Pass verdict nor Fail verdict can be given. The authors argument that assignment of the inconclusive test verdict is mainly due to the fact that the criteria which define the assignmentsof the Pass and Fail test verdicts are not mutually exclusive. To solve this problem they propose procedures for resolving TTCN test caseswith inconclusive test verdicts. We extend the work of [4] in the sensethat we do a validation of Pass and Fail verdict assignmentswith respect to the behaviour of a protocol specification (see Section 6). According to TTCN semanticsa test case execution should always (after a finite number of steps) result in the assignmentof a test verdict [ 141. This property cannot be checked statically. A feasible approach validating this kind of property is to “execute” a TICN test case and to identify unguarded infinite loops or unguarded recursion. In casewe are able to “execute” a test casethen we can also try to validate other properties, e.g., that the test caseis correct with respect to the protocol specification. Our approach is to define a common semantical

E Kristoffersen,

T. Walter/Computer

Networks

and ISDN Systems 29 (1996) Tesl System

Fig. 1. Conceptual

testing architecture.

19

15-47

Fig. 2. Distributed

1

SUT

1

test method

model for TTCN and concurrent TTCN and the formal description technique. This approach is followed in Section 6 (see also [ 18 ] ) .

3. TTCN

and concurrent

TTCN

For the purpose of this paper we restrict our attention to ‘ITCN and concurrent TTCN concepts related to the description of test case dynamic behaviour. For further details on other aspects of TEN and concurrent ‘ITCN we refer the reader to [ 14,151 or some tutorial papers [ 17,24,21]. We start with a brief description of testing architectures and testing functions as standardized by IS0 and ITU-T [ 12,131. This discussion is essential for the following sections, particularly because it partly motivates our approach to define an operational semantics for TTCN and concurrent TTCN (Section 4.2.2). 3.1. Testing architectures and testing functions Test cases are defined in terms of input events received from an implementation under test (TUT) and output events sent to the IUT. Since an IUT is generally part of a real system, a variety of possibilities exists on how the behaviour ,of an IUT can be controlled and observed during execution of a test case. The OS1 conformance testing methodology and framework [12,13], therefore, has prescribed four different testing architectures or abstract test methods: local, distributed, coordinated and remote test methods. A conceptual testing architecture is depicted in Fig. 1. It can be seen that an IUT is controlled and observed in terms of protocol data units (PDU) and abstract service primitives (ASP) that are exchanged with an IUT. The interfaces between IUT and Tester are pairs of opposite directed FIFO-queues, called points of control and observation (PCO). Since it cannot be assumed that the interfaces below and above an IUT are directly accessible for testing the conceptual testing architecture may be refined as shown in Fig. 2 which is the distributed test method. Access to the lower interface of an IUT is provided via a service provider. If the interface above the IUT is not possible to observe and control, then this defines the remote test method. The testing architecture shown in Fig. 2 is appropriate if an IUT communicates with only one peer entity. The testing architecture is referred to as single party testing architecture. Performing control and observation of an IUT is distributed over components, called lower (LT) and upper tester (UT). The LT interacts with an underlying service provider and is said to control and to observe the IUT from below. On the other hand, the UT controls and observes the IUT from above. Conceptually, LT and UT are assumed to run on different systems. The UT forms part of the system under test (SUT) and the LT is in the test system. In case an IUT has to communicate with multiple open systems, the shown testing architecture has to be extended with several concurrently running LTs and UTs which exchange PDUs or ASPS with the IUT (Fig. 3), This testing architecture is called multi party testing unzhitecture.

20

F: Kristolfrsen,

T. Walter/Computer

Conml

Networks

and ISDN Systems 29 (1996)

1.5-47

Funaicm

Fig. 3. Multi-party testing model [ 131.

From Figs. 2 and 3, we can identify four (abstract) testing functions [ 121: Lower Tester (LT). An entity that controls and observesan IUT from below via an underlying service provider Lower Tester Control Function (LTCF). An entity that coordinatesLTs and UTs, and that is responsiblefor the assignmentof a final test verdict (see also Section 3.3). Upper Tester (UT). An entity that controls and observesan IUT from above. Test Coordination Procedure (TCP). A procedure that defines, either explicitly or implicitly, how LTs, UTs and LTCF cooperate during testing. In the single party testing architecture the following testing functions are required: an LT which behavesas a peer entity to the IUT and which assignsa test verdict, optionally a UT which behaves as a “user” of the IUT, and optionally a TCP between LT and UT. Note that the LT assignsthe test verdict and thus also performs tasks that are, in other testing architectures, done by the LTCF. In the single party testing architecture LT and LTCF functions are merged into a single function. In the multi party testing architecture the following testing functions are required: a number of LTs which all behave as peer entities to the IUT, optionally a number of UTs which behave as “users” of the IUT, a LTCF which coordinates the behaviour of LTs and UTs (if any), and optionally a TCP between LTs, UTs and LTCF.

3.2. 7TCN and concurrent TTCN testing architectures In [ 131 it is recommendedto use a standardized test notation, i.e. either TEN [ 141 or concurrent TTCN [ 151, to specify test cases.TEN is applicable in the single party testing architecture and concurrent TTCN is applicable in the multi party testing architecture. Generally speaking, concurrent TTCN is a supersetof TTCN. Whereas in TTCN a test caseis executed by a single test component which implements the LT and optionally UT functions, concurrent TTCN supports execution of a test case by several test components(K) running in parallel. LTs, UTs and LTCF in the multi party testing architectures are realized as TCs. Communication between TCs is asynchronous.TCs are interconnected by coordination points (CPs) . A CP is a concept similar to a PCO, i.e. a CP is realized by two FIFO-queues one for each direction of communication,

E Kristoffersen,

7: Walter/Computer

Networks

Fig. 4. Test component

Fig. 5. Coordination

: lM CamOrnnt

and ISDN Systems 29 (1996)

21

IS-47

declaration.

point declaration

used me PTC.

Fig. 6. Test configuration.

however a CP does not connect an LT or a UT with an IUT. An additional restriction applies [ 151: CPs shall not be used between pairs of LTs and UTs (e.g. LTI and UT1 in Fig. 3). In order to coordinate their joint behaviour TCs exchange information, called coordination messages (CM). The exchange of CMs between TCs and the processingof received CMs is defined in a TCl?

Example

1. ‘Concurrent ‘ITCN defines a meansto set up complex test configurations. This is done in several steps.First, all TCs usedin a test configuration are declared (Fig. 4) 3. Note that a distinction is made between main test component (MTC) and parullel test components (PTC). Generally, the MTC plays the role of the LTCF (therefore the name LTCF is assignedto the MTC in Fig. 4) and LTs are simply PTCs. Second, the CPs used are declared (Fig. 5). Third, a test componentsconfiguration is given (Fig. 6) providing the information 3 Throughout the paper we use the graphical (or tabular) form (TTCN.GR) of the TTCN syntax. TTCN.GR human readability” [ 141. All lTCN.GR tables shown have been produced using ITEXm Version 2.2 [ 161. ITEX”’ environment that suppotts editing and analyzing of ‘ITCN and concurrent ‘ITCN test suites.

is the form is a TEN

“suitable for development

22

E Kristoffersen,

____,____.____-_ ‘“I _-_..----.---4

I: Walter/Computer

Networks

CREATE&T-T, T-SDL0300401) -_- _________----. -._. CREAE(LT0, SDL03CQ401~0) I -_---_-_-_ ---. ----.-.-... CP-T? Start-TS-CM

and ISDN Sysiems 29 (1996)

I.-

._.-._-.-.--

I .w.-.ht-atat

I I ___..____.____________ -__ __,_.__._._....___.cm_- ! __ start __ __ __ -z-r-TS CM_ _. _. --. -. -. CPO ? Cmp-TS-CM

15-47

_-_ ti--

--

_-.-

Fig. 7. Test case behaviour description.

which test components, PCOs and CPs are used. Note that MTC LTCF does not use PCOs, that LTCF and LT-T communicate through CP CP-T, and that LTCF and LTO communicate through CP CPO. Note further, that no UT is used in this configuration. Thus, test configuration CONFIGl (Fig. 6) is an instance of the multi party testing architecture shown in Fig. 3. A test component configuration can be used in several test cases. For test suites that are defined using concurrent TTCN every test case should make a reference to a test component configuration. 3.3. Test case dynamic behaviour descriptions A TTCN test case describes the dynamic behaviour of TCs during test execution. A dynamic behaviour description consists of statements and verdicts. A verdict is a statement concerning the conformance of an IUT with respect to the sequence of test case events that was performed. Possible verdicts are: Pass, Fail and inconclusive. Every execution of a test case results in a verdict assignment. Statements can be grouped into statement sequences and sets of alternatives. In the graphical form of TTCN, sequences of statements are identified by different levels of indentation, e.g. lines 1-7 of Fig. 7. Statements on the same level of indentation are alternatives, as line 8 and 9 of Fig. 7. TTCN distinguishes statements that are test events, constructs and so-called pseudo events. l Test events define basic input and output actions. An input action or RECEIVE event is the reception of an abstract service primitive (ASP) or a protocol data unit (PDU) from a PCO. In case of concurrent TKN, the input action includes also reception of a coordination message (CM) from a coordination point (CP). An output action or SEND event is sending an ASP or PDU to a PC0 or sending a CM to a CR Test events may be qualified by Boolean expressions and may be followed by a combination of assignments and timer operations. Example 2. Examples of RECEIVE and SEND events are given on lines 4 and 5 of Fig. 7. The RECEIVE

event in line 4 is to be interpreted as follows: The TC accepts from CP CP-T a CM of type Start -TSCM

E Kristoffersen,

‘I Walter/Computer

Networks

and ISDN Systems 29 (1996)

.-_-_-[aw=FALSE) _-------_-.-.-.-.-.LO ? P-DAlAhd

Fig. 8. Test component

15-47

23

tBDLGIJidnn(corr

behaviour

description.

with value as specified by constraint reference cMstart. If the RECEIVE event is executed, test case execution continues with the SEND event that transmits a CM of type Camp-TSCM with value CM-start (see column “Constraints Ref”) along CP CPO to TC LTO. is a test event that can be used by the MTC to check whether a specific PTC or all PTCs have terminated. The DONE event is successful when all of the indicated PTCs have terminated. For example, in Fig. 7 line 8 and 12 it is checked for PTC LT-T and the PTCs LT-T and LTO respectively. Constructs are used to guide the flow of control in test cases and to give test cases a modular structure. For the latter TTCN provides the ATTACH mechanism that allows combinations of statements, called test steps, to be “attached” to a test case. For determining the flow of control, GOT0 and REPEAT constructs, which are in one or another form known from many other programming languages, are introduced into TTCN. Additionally, CREATE is a construct for controlling the concurrent behaviour of TCs. It is the responsibility of the MTC to set up all PTCs of a test configuration. The CREATE operation associates a PTC with a behaviour tree. If the behaviour tree is parameterized the CREATE operation passes also actual parameters for formal parameters, For instance, the CREATE statement on line 3 in Fig. 7 instantiates TC LTO with behaviour description BDL0300401-0 (Fig. 8). The newly created PTC runs in parallel with the MTC. If the PTC to be created is in the course of execution of a behaviour tree then execution of a CREATE by the MTC results in a test case error indication. Pseudo-events are qualifiers (i.e. Boolean expressions), assignments and timer operations. DONE

l

l

3.4. Test case execution A behaviour tree [ 14,151 defines the relative ordering of test events that a TC 4 (either the MTC or a PTC) executes. Execution of a test case starts with the execution of the MTC which is responsible for setting up the test configuration defined for the test case. As stated above, MTC and PTCs run in parallel. 4 We assume that LT and UT of a single party testing architecture for TTCN.

are modeled

by one TC. Thus, the following

statements

are also valid

24

E Kristoffersen,

T. Walter/Computer

Networks

and ISDN

Systems 29 (1996)

15-47

The statements given in a behaviour tree are executed in sequence only if a statement is successful. A statement is evaluated successfully under the following conditions: A REPEAT and a GOT0 is always successful. A SEND event, an assignment and a timer operation are always successful provided that the optional accompanying qualifier holds. A RECEIVE event is successful if an ASP, a PDU or a CM can be received from the specified PC0 or CP and the ASP, PDU or CM received matches the constraint defined in column headed “Constraints Ref” and provided that the optional qualifier holds. A CREATE operation and a DONE event are successful under the above mentioned conditions. A qualifier evaluates successfully if it evaluates to true. A TIMEOUT event (Fig. 7 line 9) is successful if the specified timer (i.e. timer CASE-TIME) has expired. In TTCN and concurrent TTCN a discrete time model is assumed where time is counted in time units of arbitrary granularity (from picoseconds to minutes). Alternatives of a set of alternatives are evaluated one after another starting with the first alternative until one alternative is found successful. If all alternatives have been evaluated unsuccessfully, the evaluation is repeated starting with the first alternative. Note that before a set of alternatives is evaluated the current state of the test system is frozen or, as stated in [ 141, a snapshot is taken. That is all PC0 and CP queues and expired timer lists are updated and during the evaluation no update takes place. In that sense evaluation of a set of alternatives is an atomic action. Test case execution terminates when a final verdict is assigned or a leaf in the behaviour tree is reached. With respect to concurrent TTCN, termination of a test case is quite more complex. As a general rule, whenever the MTC terminates either by assigning a final verdict or reaching a leaf in the behaviour tree, test case execution terminates independently of any PTC still running. If a PTC terminates either by assigning a final verdict or reaching a leaf in the behaviour tree then this does not imply that test case execution terminates. If a test verdict is assigned in a PTC then this is a local matter only. The test verdict assigned by a PTC may only be communicated to the MTC by exchanging an appropriate CM. Every TC manages its own set of variables and timers. In the context of concurrent TTCN, variables local to a TC are denoted test component variables. Access to globally defined variables, e.g. test suite variables, is restricted to the MTC. Test suite parameters and constants are shared between TCs.

4. TTCN and concurrent

TTCN

semantics

In the OS1 conformance testing standard [ 141 it is stated: “ ... this part of ISO/IEC 9646 defines an informal test notation, called the Tree and Tabular Combined Notation (TTCN), for OS1 conformance testing, . ..” However, the test case development process requires a precisely defined semantics as certain tasks, e.g. test case validation, test case compilation, test case execution, and test result analysis, can only be supported by tool based methods if a formal semantics is available. The main focus of this section is to present an approach to formally define a semantics of TTCN that enables test case validation. 4.1. Preliminaries The operational semantics presented in the following two sections is defined for TTCN test cases that are in a specific form which we call canonical form. Whenever during execution of a TTCN test case a set of alternatives is reached, then the set of alternatives is transformed into its respective canonical form as follows (see Annex B of [ 14,151): First, all REPEAT constructs are substituted by a combination of a Boolean expression and a GOT0 construct. Second, all default behaviour trees 5 are “attached”. Third, all attached trees are expanded, i.e. the referenced test step behaviour description is substituted for the tree attachment construct. 5 A default

behaviour

description

contains

all statements

which may occur at any level of a behaviour

tree.

E Kristoffersen,

T. Walter/Computer

Networks

and ISDN Systems 29 (1996)

15-47

25

Example 3. The initial set of alternatives in the behaviour tree shown in Fig. 8 is in canonical form since it does not make use of the REPEAT construct and no tree attachment is used. After the initial set of alternatives has been evaluated, tree attachment +PRS3nInpO has to be expanded into canonical form before evaluation of this set of alternatives can be done. 4.2. Operatiorlal

semantics of 7TCN and concurrent

TTCN

Although the standard itself [ 14,151 does not claim to be a formal definition of an operational semantics, it is a starting point for the definition of a formal semantics of TTCN and concurrent TTCN. In Section 4.2.2 we present an alte:rnative approach and subsequently we compare the two approaches. 4.2.1. The ISO operational

semantics

The official IS0 ‘ITCN semantics [ 14,151 is an operational semantics defined in terms of a TTCN machine. A TTCN machine executes a programme that simulates the execution of a test case in canonical form. The programme is defined in terms of a set of functions that includes a function for every test event, construct and pseudo-event not eliminated during transformation of a test case into its canonical form. The programme is defined using a pseudo-code notation. In the following we elaborate on some of the functions. Receive event, A RECEIVE event checks whether a specific ASP, PDU or CM (specific with respect to types and values) has been received; i.e. whether the first element of the PC0 satisfies the requirements of the “Constraints Ref” (Fig. 8). Recall that ;I test event can be qualified by a Boolean expression and associated with assignments and timer operations. The function that defines the semantics of receiving an ASP or PDU and execution of assignments and timer operations is as follows: function RECEIVE(PCOidentifier or CPidentifier, ASPidentifier or PDUidentifier or CMidentifier, Qualifier, Assignment, TimerOperation, ConstraintsReference, Verdict, Level) : BOOLEAN begin if RECEIVE-EVENT(PCOidentifier or CPidentifier) then begin if (RECEIVED-OBJECT(ASPidentifier or PDUidentifier or CMidentifier, ConstraintsReference) A:ND EVALUATE-BOOLEAN(Qualifier>> begin EXECUTE-ASSIGNMENT(Assignment) ; TIM:%-OP(TimerOperation); REMOVE-OBJECT(PCOidentifier or CPidentifier); VERDICTWerdict) ; Level := NEXT-LEVEL; LOG(PCOidentifier or CPidentifier, ReceivedObject); RETURN(TRUE); end

26

F: Kristofersen,

T. Walter/Computer

Networks

and ISDN Systems 29 (1996)

15-47

else RETURN(FALSE) else RETURN(FALSE) end where function RECEIVE_EVENT() returns true if the incoming queue of PC0 PCOidentif ier or CP CPidentifier is not empty. In this case a copy of the first element is created. In the next step the value of the received ASP, PDU or CM is compared with the value defined in constraint ConstraintsRef erence. If function RECEIVED-OBJECT returns true and if qualifier Qualifier is true, i.e. function EVALUATEBOOLEAN () returns true, then the assignments are executed (function EXECUTE-ASSIGNMENT0 ), the timer operations are performed (function TIMERaP )and the received object is removed from the incoming queue (function REMOVE-OBJECT()). The current value of the verdict is calculated by function VERDICT. If the verdict is a final verdict then test execution is stopped; otherwise, the next level of alternatives is determined (function NEXT-LEVEL (> ) . As part of the execution of the RECEIVE event, a log file is updated before processing of the next set of alternatives is started. Since function RECEIVE0 returns only after all assignments and all timer operations accompanying a RECEIVE event, we conclude that execution of a RECEIVE event is an atomic action. This holds true for all other TKN statements as well (see below and [ 14,151). CREATE. A PTC should only be created by the MTC if the PTC is not already executing; otherwise a Test Case Error should be generated. If creation of a PTC is successful then the PTC runs in parallel with the MTC and all other PTCs previously created. The pseudo-code for the CREATE construct is as follows: function CREATE(TCIdentifier, TreeReference(ActualParList), Qualifier, . . . > : BOOLEAN begin if EVALUATE-BOOLEAN(Qualifier) begin ... START-EVALUATION(TCIdentifier, TreeReference(ActualParList));

then

RETURN (TRUE) ; end else RETURN (FALSE) end Function STARTEVALUATION() instantiates the PTC TCIdentif ier with behaviour description TreeRef erence and actual parameters ActualParList. PTC TCIdentif ier starts to execute which means that a TTCN machine is created simulating the execution of behaviour tree TreeReference. Function START-EVALUATION (> then returns immediately. DONE. The MTC can use the DONE construct to check the status of PTCs specified as actual parameters. function DONE(TCompList, Qualifier, . . . > : BOOLEAN

E Kristoflersen, T. Walter/Computer Networks and ISDN Systems29 (1996) 15-47

27

begin if EVALUATE,BClOLEAN(Qualifier) AND ALL-TEFtMINATED(TCompList) then begin l&URN (TRUE) ; end else RETURNCFALSE) end The status of PTCs TCompList is checked by calling function ALL3ERMINATED() which returns true if all PTCs have temlinated, or which returns false otherwise. Note that an optional qualifier is evaluated first. EVALUATE-TEST-CASE.Up to now we have discussedexecution of test events, constructs and pseudo-events. We continue with a brief discussionof the execution of a behaviour tree: procedure EVALUATE-TEST-CASEcTestCaseId, BehaviourTree) begin Level := FIRST-LEVELcBehaviourTree); EVALUATE:-LEVEL(TestCaseId, BehaviourTree, end

Level,

TestCaseId,

DefRefList)

procedure EVALUATE-LEVEL(TestCaseOrStep, Level, Def aulta.) begin EXPAND-L,EVEL(TestCaseOrStep, Tree, Level, Defaults); repeat TAKE-SNAPSHOT; if EVALUATE-EVENT-LINE(A1, Level, Defaults) THEN EVALUATE-LEVEL(TestCaseOrStep, Tree, Level, Defaults); ... if EVALUATE-EVENT-LINEcAm, Level, Defaults) THEN EVALUATE-LEVEL(TestCaseOrStep, Tree, Level, Defaults); UNTIL SNAPSHOT-FIXED(Leve1); STOP end Execution of a behaviour tree starts with the first level of indentation, i.e. the first set of alternatives. If necessarya set of alternatives is expanded (EXPANDLEVEL) in preparation for evaluation of alternatives. The evaluation (EVALUATEEVENTLINE) is done one by one starting with the first alternative and ending with the last one. But before evaluation of alternatives is performed a snapshot (TAKE-SNAPSHOT)is taken which meansthat incoming PC0 and CP queues and timer list are updated and the termination status of any other test component is determined. The first alternative which is evaluated successfully is taken and processed. Afterwards, i.e. upon return from EVALUATE-EVENT-LINEthe set of alternatives at the next level of indentation that follows the event taken is evaluated. This process continuous until test case execution terminates (by assigning a final verdict) or behaviour tree execution deadlocks, i.e. all PC0 and CP queues have some

28

E Kristoffersen, 1: Walter/Computer Networks and ISDNSystems 29(1996)15-47

B&c PfOCC.SS

Input Process

Timer PKIWSS

Fig. 9. Common semantics representation for concurrent TTCN.

messagespending and all relevant timers have expired but theseare not the messagesor timers expected to be received or to be expired. At first sight the definition of an operational semanticsin a procedural form seemsto be adequate.One can think of implementing a programme that, as described by the different functions, executes a test case step by step. However, some details of the informal (but normative) definition of ‘ITCN and concurrent TTCN are not properly defined: l [ 14, Section 14.9.2 (c)] states that “in cases where a qualifier is specified on the event line, the quali$er shall evaluate to TRUE; the qualijer may contain references to ASP parameter and/or PDU jields”. Given function RECEIVE0 (see above), evaluation of a qualifier is done by calling function EVALUATEBOOLEANO with parameter Qualifier. However, parameter Qualifier is the same as the formal parameter Qualifier of function RECEIVE() . Thus, possiblereferencesto ASP parameters,PDU fields or, in the case of concurrent T’KN [ 15, Sections 14.9.1 and 14.9.21, CM fields are not updated by the referenced values of an ASP, a PDU or a CM received. Evaluation of Qualifier may yield an incorrect or even undefined result. We conclude that the definition of function RECEIVE0 is incomplete and ambiguous. l A PTC should be created only if the PTC is not already running [ 151: “The execution of a CREATE construct on a Test Component which has already been created shall be a Test Case Error”. In function CREATE0 no check is performed whether PTC TCidentif ier is running or not. It is neither obvious when to generate the Test Case Error nor is it foreseen in the definition of function CREATE0 that creation of a PTC may fail. As indicated, some flaws can still be found even in the definition of TTCN, although TTCN is already an international standard and concurrent TTCN is to become international standard. 4.2.2. A common semanticsrepresentation approach The common semanticsrepresentation (CSR) has originally been developed as a semantical model for both SDL and ‘ITCN [30]. The CSR is a model for the representation of the dynamic behaviour of syntactically well-formed and static semantically correct TTCN and concurrent TTCN test cases.It is defined as a hierarchical model with a structure as shown in Fig. 9. The structure of the CSR has been motivated by the following observations: l The test system defines the top level in the semantical model. Communication between a test system and an IUT takes place via PCOs. All communication along a PC0 can be seen by an observer in the environment of a test system. l In the CSR each PC0 is modeled by two unidirectional channels. PCOs introduce a finite delay on the messagesthey convey between source and destination.

E Kristoffersen,

T. Walter/Computer

Networks

and

ISDN

Systems

29 (1996)

IS-47

29

The test module is the model for test components and CPs of a test configuration. Since communication between test components through CPs is inside a test system, this communication is not observable outside the test system. A test module encapsulates the internal structure of a test system and abstracts also from internal communication. l Lower and upper testers (Figs. 2 and 3) are mapped to test conzponents in the CSR. In case of concurrent TTCN, KS may be connected by CPs (see the restrictions discussed in Section 3.2). Like PCOs, each CP is modeled by a pair of unidirectional channels in the CSR. l Every test component manages its own set of variables that includes test suite constants, test suite parameters and test component variables, and its own set of timers. If the test component is the MTC it manages also test suite variables and test case variables, For this purpose a test component in the CSR has associated an environment and storage for the mapping of variables to (memory) locations and the assignment of locations to values, and a timer process that maintains a list of running timers. A basic process is the entity that controls the actions that may be performed next by a test component. To be more precise, the basic process is a representation of the behaviour description that has been assigned to the test component. As PCGs and CPs introduce a finite delay on messages, it is necessary to control the reception of ASPS, PDUs and CMs by a test component. An input process stores all messages that a test component has received but not yet processed. Note that in the model no distinction is made between lower and upper tester nor does the model distinguish the lower test control function; all are test components. We define an operational semantics for TTCN and concurrent TTCN in terms of labelled transition systems (LTS) 1201. I?articularly, we associate with each CSR entity an LTS that describe the behaviour of that entity. l

Definition

4. A labelled transition system (LTS) is a triple

which consists of a set S of states, a set E of atomic events and a transition relation C S x F x S. From a formal point of view the operational semantics of an entity is given by a family of LTSs, i.e. an LTS for every initial state. An elemeni in --+ is termed transition and is written s A s’ for s, s’ E S, e E E, and (s, e, s’) E . In this paper we do not give the complete definition of the LTSs in the CSR. More complete descriptions of the CSR can be found in [ 3 1,30,18]. In the following paragraphs we give a brief description of the principles of the approach. Basic processes. The basic process is the control part of a test component. A behaviour description of a basic process is co,nstructed from a set of operators and a set of atomic events. The set of operators include event pre@ing, denoted “;“, and priority choice, denoted @. The set of atomic events includes events for input and output of messages, control of timers, manipulation of variables and data terms, managing of test components, etc. We refer to the set of atomic actions and the set of operators as basic process algebra (BPA) . Example 5. Given the TTCN Test Case Dynamic Behaviour of the MTC shown in Fig. 7. In the BPA, this behaviour description is represented as shown in Table 1. A ‘ITCN statement is transformed into a sequence of atomic events. Since in our approach parallelism is modeled by arbitrary interleaving of events (see Fig. IO), whenever an atomic event has been executed the same or another CSR entity may be scheduled for execution. With respect to the evaluation of a set of alternatives this does not cause problems as taking a snapshot and checking which alternative is successful

30

E Kristoffersen,

7: Walter/Computer

Nehvorks

and ISDN Systems 29 (1996)

15-47

Table 1 Basic processof test casedynamic behaviourdescription (Fig.

7)

P ::=

start( CASE-TIME,

now + TIMEOUT~VALUE);

TBDLO300401);

createtc(LT-T,

createtc(LT0,BDL0300401D); input

(chfstart)

output{

from

cMsfart)

via

input(cMxomp) output

CP-T[truej; CPO;

from

(ckfxomp)

(done(LT-T)

CfO[rrue];

CP-T;

via ; stop;

nil)

@ (input

(casehne)

[true]

;

output(cMfimeout)

via

CPA?

output(cMhneout)

via

CPO;

done(LT-T,LTO); ~1 := Fail;

stop;

nil)

coincide. Furthermore, all test components have their own copies of variables and timers so that the update of a variable or expiration of a timer by one process does not interfere with events performed by another process. Timeouts are modeled as messages, i.e. whenever a timer expires, a timeout messageis created and is put into the input process. The timeout message can be retrieved using a simple input event. For more details on the transformation of TTCN into BPA expressions and the evaluation of set of alternatives the reader is referred to [30-j. Given a basic process all information that determines the future behaviour of that process is included in the process definition itself. A basic process evolves by performing atomic events thus performing state-transitions. The start state and the next state of a state-transition of a basic process are given by a term in the BPA. The set of states S of an LTS consists, thus, of all terms over the BPA. The transition set -+ of an LTS is not given explicitly but it can be determined by applying a set of inference rules [ 20,9] : Given a state s, an event e and a next state s’ then S$ 3’ is a transition if this transition can be proven from the set of inference rules. Note that indices are used to distinguish between inference rules for different entities. Example 6. Let e be an event of a basic process, and let P, P’, Q, Q’, e; P and P @Q be basic processes. The

following inference rules are defined:

(1) (2)

e;P --f--+8 P P&BP’ P@Q

-q-‘~ P’

and

Q~-‘BQ’ PcBQ~BQ’

i.e. a basic process e; P can always perform any event e and evolve into P (inference rule ( 1)) and a basic process P @ Q can perform an event e if one of its constituent basic processes can perform event e (inference rule (2)).

E Kristoffersen,

T. Walter/Computer

Networks

and ISDN Systems 29 11996) IS-47

31

Test components. Similarly an operational semantics is defined for all other basic entities in terms of an LTS. Basic entities of the CSR are input processes, timer processes, CPs and PCOs. For the other entities of the CSR (test components, test modules and test system), the operational semantics is defined relative to the lower level entities. Fror each such entity the operational semantics is deduced from the operational semantics of its constituent parts. Example 7.

In order to define an LTS of a test component we have to consider its lower level constituents (Fig. 9): basic process, input process and timer process. A state of a test component is given by the state of the basic process, the state of the input process and the state of the timer process. Additionally, the local storage of the test component, i.e., the mapping of variables to values forms part of a state for a test component. The compositional structure of the CSR is not only reflected in the definition of states but also in the definition of inference rules. Let P, P’ be states of a basic process, let cr, u’ be states of an input process, that is sequences of messages, and let TC, TC’ be states of a test component. The inference rule that defines the behaviour of a test component processing a received message is defined by: P

input --B

via I Ibrl

TC --&‘

P’

deliver(sig,l) U------Id

Tc’

c

A test component in state TC can perform an internal event 7 evolving into state TC’ provided the basic process in state P can perform event input via I [ bt] evolving into state P’ and provided the input process in state (T can perform event deliver(sig, I) evolving into state (+‘. These two transitions form part of the premise of the inference rule. To verify the premises of the inference rule it suffices to check whether both transitions can be proven from the inference rules for basic processes and for input processes (see [ 181) . Recall that a test component performs a RECEIVE event only if received values and expected values (as given by a constraint) match. In our approach this requirement is expressed as part of a side-condition to the inference rule; in the above example C denotes the following side-condition: C=

sig = l14Lp Ilbt’Ijc,P = true

i nowe, p = time The constraint is specified by data term sf that is evaluated in environment E and storage p, this means that the evaluation is performed with a specific mapping of variables to values. For the condition to be satisfied the received value sig must match the expected value IIstlle,p.Since processing of the messages should be performed only if Boolean expression bt holds, this requirement is expressed in side-condition C as Ilbt’lla,+,= true where bt’ is derived from bt by substituting (received) values for references in bt. The third line of the side-condition is to express that before the transition can be performed the local clock must hold the current global time. This condition is specified for every inference rule of a test component. This is in order to deal with timing constraints as imposed by timers. In the CSR an discrete model of time is assumed. Time is counted in ticks but no specific duration is associated with a tick. An inference rule is applicable only if the side-condition is true. For the CREATE operation and the DONE event the inference rules are as follows: Example 8. In order to determine the status of PTCs we introduce a Boolean array b[ 0. . . n] where n is the

number of test components. Before test case execution starts, all elements in array b[ ] are set to false. After test case execution has started, element b[O] becomes true indicating that the MTC has been created and is running. Whenever a PTC is created by the MTC and starts execution, the corresponding element in array b[ ] becomes true. If a PTC stops execution then the corresponding element in array b[ ] is again set to f&e.

E Kristoffersen,

32

7: Walter/Computer

Networks

and ISDN Systems 29 (1996)

15-47

Assuming the MTC creates a PTC X (the test component X must be a valid test component of the currently used test configuration) and associates to X behaviour description (Q, (11,. . . , Zp), ( yt , . . . , y,) , ( zn, t,, ) ) ) where Q is a basic process, II, . . . , Z, are formal PC0 and CP identifiers, yt , . . . , ym ((Zl,fl),..., are formal value parameters and (~1, tl ), . . . , (z,,, tn) are local variables and their initial value, then the following inference rule applies: P TC

createtc(X,Q(l; createtc(X,Q(l;

,...,$,)(f~,.Jnt)) ,..., $)(al,...

‘B

p’

‘TC

TC’

,a,,,))

c

where I{, . . . , IL are actual PC0 and CP identifiers and ai = I]tille,p Side condition C has the following form:

for i = 1,. . . , m are actual value parameters.

b[X] =fulse ’ = i Iln~wll~,~ = time i.e. a test component should only be created if it is not already being used, condition b[ X] = false. For the DONE event the inference rule applies: done(XI ,..., X,,)

P -g

done(Xl,..., -TC

n:

P’

X,,)

C

T%’

where C=

(

Vi = I . . .K b[Xi]

=false

~~Izow(~~,~ = time

i.e. a done event is successful only if all PTCs XI,. . . , X,, have terminated (condition

Vi =

1 . . . n: b[ Xi] =

false).

The following inference rule models the evaluation of a set of alternatives. Let priority choice expression. Let U, (+’ be states of an input process and let 0,O’ be For the following inference rule let TC = (P, U, 0) be a state of a test component, states of a basic process, an input process and a timer process. The inference rule in a set of alternatives is as follows: { T(Qir a. 0) /~TC (Qjv (~90) %C

(e fl,@) &C

P = Ql @I . . . @ Qn, be a

states of a timer process. i.e. a tuple consisting of the CSR for evaluation of

1i < j}

(Q;, a', 0') (Q,!,CT',0')

c

where C

= jlnOwlle,p

= time.

The premise states that if the jth alternative is evaluated successfully and performs an event ej then, see the conclusion, the test component performs the same event and evolves into a state given by the successor state of the jth alternative, i.e. Qj, with possibly input and timer process updated. It is requires, however, that none of the alternatives that precede the jth alternative can perform an event, i.e. none is evaluated successfully. In the CSR approach taking a snapshot is implicitly done in the sense that evaluation of alternatives and taking a snapshot coincide. Note that if the inference rule is applied input process and timer process and termination status of other test components do not change.

E Kristoffersen, T, Walter/Computer Networks and ISDN Systems29 (1996) 15-47 h ITC

T O PTCLT-T

(

33

r Ticks

Timerqeration

1 I

CIUW

createI

Communicalion

Reception in input procrss 1 Pmcessing “f ,nessage I

I

Communication

Reception in input pmcess

Fig. 10. Schedule of the executionof a test module. and hiding. The compositional structure of the CSR model allows in addition to derive the behaviour of the higher level entities from the basic entities, also to describe the observable behaviour from different levels of abstraction. This property is important when it comes to validation of test case behaviour. So to support validation of functional properties, we introduce abstraction or hiding over temporal behaviour

Abstraction

[19,21. For each CSR entity we decide which of the events that the entity can perform are observable outside the entity. Then oblservability of the behaviour of a test system can change according to where it is assumed that an observer is placed. An observer in the environment of a test system can observe only messages received from or sent to the test system. An observer of a test component can observe the reception of a message by this test component, but cannot observe the processing of the received message. On the other hand, if the MTC creates a new IFTC this can be observed in the test module and, therefore, creation of PTCs is an observable event (Example 8). Execution model. In the CSR concurrency is modeled by arbitrary interleaving of atomic events. A possible schedule of events of a test module and test components (Figs. ‘7 and 8) is depicted in Fig. 10. Note that we do not make any assumptions concerning the execution time of events neither do we make assumptions on the time between successive events. If two entities are communicating, e.g. the reception of a CM by a TC, then both entities have to perform corresponding events simultaneously. 4.3. Evaluatioll

of the approaches

We conclude this section with a brief evaluation of the two approaches. l The Boolean expression coded with a RECEIVE event may refer to received values. In the CSR this is dealt wit!h as the Boolean expression is updated with received values before it is evaluated.

34

E Kristoffersen.

7: Walter/Computer

Networks

and ISDN Systems 29 (1996)

15-47

Execution of a CREATE operation implies a check whether the PTC is running (b[] = true) or is not running (b[ ] =fulse) . If the latter condition is fulfilled then the CREATE operation is executed successfully. Since 6[] becomes false again if a PTC terminates we allow the MTC to create a PTC several times during execution of a test case. Whether this is allowed by operational semantics of TTCN [ 1.51 is not evident from its current definition. l While a test case is executed a test system might run into a situation where an event might be possible but, according to TTCN semantics definition, should not be executed. If the actions is executed nevertheless, this should result in a Test Case Error (see [ 15, p. 181 for the CREATE operation). In the CSR such a situation is dealt with that one or all inference rules are not applicable (thus the system is in a “deadlock”). We do not indicate a Test Case Error. Our approach is consistent in the sense that applications of inference rules only give allowed behaviour whereas in TTCN a system is allowed to exhibit incorrect behaviour (although this might produce a Test Case Error). We could have worked around the problem simply by adding inference rules with side conditions inverted and indicating Test Case Error. Then when these inference rules are applied a Test Case Error action would be performed. We conclude that in the definition of a formal operational semantics we have resolved the identified problems. Furthermore, the model support the definition of methods for formally based validation of test case behaviour. l

5. TTCN and concurrent

TTCN validation

Before a TTCN test suite is made into an international standard it has to be validated. Test suite validation can be separated into two (more or less) succeeding phases. First, a test suite is validated with respect to syntax and static semantics and, in the second phase, with respect to functional properties. In this section we concentrate on syntax and static semantics and on functional properties that can be validated by looking at a test suite or test case only. 5.1. Syntax and static semantics Syntax and static semantics of ‘ITCN and concurrent TTCN are defined in [ 14,151, Annex A. It is the syntax of TTCN.MP, the machine processable syntax of TTCN and concurrent TTCN, that is standardized. The forma1 syntax of ‘ITCN is given in BNF (Backus-Naur Form). The static semantics is given along with the BNF syntax rules. Static semantics rules are used to define the context-dependent syntax, i.e. those constraints on the derivation of terminal symbols from non-terminal symbols that cannot be expressed by a context free grammar (like BNF). A typical example follows where non-terminals start with a capital letter and terminals are enclosed in ’ ’ and ’ ‘. Example 9. The following grammar rule (number 291) is cited from [ 151. The rule defines the syntax of a RECEIVE event:

Receive ::= [ PCO-Identifier I CP-Identifier ( FormalParIdentifier 1 "?" ( ASP-Identifier I PDU-Identifier I CM-Identifier > /* STATIC SEMANTICS - PCO-Identifier or FormalParIdentifier shall be present if the test suite uses more than one PC0 */ /* STATIC SEMANTICS - Only CMs may be exchanged on CPs and only ASPS and PDUs may be exchanged on PCOs */ The first static semantics requirement states that if the testing architecture used is, for instance, the distributed testing architecture (Fig. 2) then a RECEIVE event should start with the indication of a PC0 or, if the statement

E Kristoffersen,

I: Wafter/Computer

Networks

and ISDN Systems 29 (1996)

IS-47

3.5

is part of a tesl step, it should start with a formal parameter in which case an actual PC0 identifier is supplied for the formal parameter when the test step is called. The second static semantics requirement restricts the possible combinations of PC0 and CP identifiers with ASP, PDU or CM identifiers. If the non-terminal that precedes the question mark generates a CP identifier then the non-terminal that follows the question mark must reduce to a CM identifier. Similarly, if the non-terminal that precedes the question mark generates a PC0 identifier then the non-terminal that follows the question mark must reduce to an ASP or a PDU identifier. The combination of a CP identifier and an ASP identifier does respect the syntax rule but is static semantically incorrect. For both TTCN and concurrent TTCN tools have been implemented such as lexical analyzers and parsers that are capable of checking a test suite for syntactical correctness. Furthermore, the tools can also perform a static semantics check although the extent of how accurate the static semantics requirements are validated varies from tool to tool. 5.2. Dynamic ,properties

In the following paragraphs we define the basic principles necessary to perform validation of dynamic properties, i.e. functional properties, in the CSR. We do not deal with all possible dynamic properties but give an example that illustrates the application of the CSR to prove a specific dynamic property. Fig. 10 gives a graphical representation of how an execution of a test case is modeled by an LTS (Definition 4). A test component (e.g. TC MTC in Fig. 10) evolves by executing atomic events one after another, i.e. we have a sequence of transitions for which the next state of one transition is the start state of the following or next transition. This observation is formalized in the following definition: Definition

10.

Let LTS be a family of labelled transition system (S,E, -

), A trace of LTS is a finite

sequence so -2

s1 S] as,

s2 -2

s3

.. .

sn -

c,,

Sn+l

of transitions of LTS where the last state of one transition is the first state of the next one. We will write traces as en-I Since every CSR entity is modeled by an LTS the schedule of Fig. 10 gives the following trace of internal events: tmj --&M

tm2

MTC Timer operation

tm2 --&M

tm3

MTC Create

tm3 --&+j

tm4

MTC Update local clock

tm4 -&M

trns

MTC Create

tms -I‘+TM tm6

PTC LT-T Update local clock

tm6 -&M

tml

PTC LT-T and CP CP-T Communication

trnl --&M

tmg

MTC Update local clock

tmg ---&M

tm9

MTC and CP CP-T Reception

tm9

ATM

m0

MTC Processing

36

E Kristofersen. tml0

-%M

tml I LTM

T. Walter/Computer

Networks

tml1

MTC and CP CPO Communication

tm12

PTC LTO Update local clock

tw2

-GM

tm13

PTC LTO and CP CPO Reception

tmt3

--%M

tw4

PTC LTO Processing

and ISDN Systems 29 (1996)

15-47

where tmi denotes a state of the test module, i.e. the combination of states of all CPs and states of all test components. We have internal events only since no messages are sent to an IUT through a PCO, i.e. a test module abstracts from internal behaviour of test components and CPs. An external (relative to the test module) observer cannot “observe” a progress of the system in terms of messages sent or received. But for validation we are free to position an observer wherever we like as the following example shows: Example

11. For the validation of certain dynamic properties of a test case it might be quite convenient to have a closer look into the internal behaviour of a test system. A CREATE operation is performed by the MTC within a test module. As explained above, an observer outside the test module notes that some internal progress has been made, However, if the observer is located inside the test module then the observer is capable to observe the creation of a PTC, and if the observer is even further moved into a test component, then the observer can observe that the basic process has performed a createtc( LT-T, TBDLO300401) event. Table 2 summarizes this discussion. The example given is for the CREATE event. From top to bottom each line counts for the events performed by those entities that are in the currently considered test configuration. If the entity is involved in the execution of a CREATE event then the event executed is displayed or “-” for “not involved” otherwise.

In the following examples we give an outline of how to prove that a test case has terminated after having performed a finite number of test steps. Example 12. Test case execution terminates with an assignment of a final verdict. Following the approach

outlined above, we move an observer into a test module to determine whether an internal event at the test module level can be related to the execution of an assignment of the form “v := Fad” at the test component level. However, at the test component level an assignment is also an internal event. If we move the observer further into a test component then the verdict assignment becomes observable (Table 3) : basic process MTC performs assignment v := Fail and then execution event stop indicating that it has terminated. We deduce that the MTC first performs a verdict assignment and then executes the stop event. Section 6 will extend the theory developed so far. We will show how to validate a test case against a formal protocol specification.

6. Concurrent TTCN test case validation

The validation framework we define here is concerned with the consistency between the dynamic behaviours of a test case and an SDL specification. The framework uti!izes the CSR for concurrent TTCN and SDL for the validation. Using formal methods for the validation of test cases also allows for analysis of test suite coverage as defined in [ 281.

E: Krisfoffersen,

5: Walter/Computer

Networks

and ISDN Systems 29 (1996)

15-47

31

Table 2 Different levels of observation: CREATEoperation Name Event CSR Entity Test Module CP Test Component Basic Process Input Process Timer Process

TM

7

MTC BP MTC -

createtc(U’-7’. T-BDLO300401) createtc(LTJ’, T-BDLO300401)

Table 3 Different levels of observation: Termination of a test component

6.1. Preliminaries

CSR Entity

Name

Event

Event

Test Module CP Test Component Basic Process Input Process Timer Process

TM

7

7

MTC BP MTC

7

7

v := Fail

stop

and limitations

For the sort of TTCN test case validation we aim at, the following assumptions are made. For TTCN test cases to be validated the following restriction applies. Only bounded recursion is allowed in order to ensure a finite validation process as every sequence of test events will be finite. The CSR itself also implies limitations on the validation framework. Since in the CSR concurrency is modelled by interleaving of events, the validation of real concurrent behaviour is not covered. Real time properties are not covered either as the CSR assumes a discrete time model. Furthermore we restrict the validation of timing aspects such that sequences of events are considered equal if they differ only with respect to time events. The interface between the SDL system and the test system to be validated is assumed to be empty which means that the two systems exchange events synchronously. This assumption is similar to the “ideal test architecture” [‘7] where no test context, e.g. an underlying service provider, is assumed. The TTCN test cases and SDL specifications are supposed to be syntactically and static semantically correct. Validation of a TTCN test suite is based on the validation of individual test cases because there is no higher level entity for which a dynamic semantics is defined in the CSR. The framework covers test case validation of valid behaviour and test cases with inopportune events when these messages. are declared. The framework does not cover test cases that test for invalid behaviour using syntactically or semantically invalid test events since these events cannot be related to any SDL event. 6.2. Validation framework The validation framework is defined assuming the mentioned restrictions to be satisfied. The semantical model described in the previous sections forms the basis for the framework for validation of the behaviour of test cases against the behaviour of an SDL specification. Let S denote the SDL specification against which the test case r is to be validated. The mapping functions from an SDL specification to the CSR and from a TTCN test case to the CSR are denoted 73s and & respectively. For a test case t and an SDL specification S we denote the CSR representations of mappings Br( t) and f3s( S) by B, and Bs.

38

E Kristoffersen,

Z Walter/Computer

pi-J TTCN

-

Nehvorks

and ISDN Systems 29 (1996)

15-47

CSRsm

6 -cl

Fig. II. A model of the validation process for dynamic behaviour.

The term correcme.rS criterion denotes the requirements to be satisfied by a test case in order to be termed a valid test case. We define a correctness criterion as a binary relation Rc over the set of test cases Sp.~c~~., and SDL specifications SpecsDL: RC

s SpecnCN

x sPeCSDL

(1)

where index C may range over different correctness criteria, e.g. syntactic and static semantics correctness. However, because our emphasis is on dynamic bebaviour validation and because the CSR is an appropriate model to represent the behaviour of systems, we seek for a refinement of correctness criterion Rc in terms of the CSR:
(2)

where Qc denotes a refined definition of criterion Rc. This general definition of a correctness criterion is elaborated to provide an instance of the validation framework. Fig. 11 illustrates the validation process for dynamic behaviour. Rc defines a correctness criterion between ‘ITCN test cases and SDL specifications. The correctness of a test case is not determined directly in terms of the TTCN test case and the SDL specification. Instead, the test case and specification are transformed to entities in the CSR. If the corresponding entities are in relation
suite except of those test cases testing for syntactically and semantically invalid behaviour. Let S denote an SDL specification. A test suite TS is valid with respect to SDL specification S, denoted Valid( TS, S), if correctness criterion Rc is satisfied by every test case of the test suite. Valid(TS, S)

iff

Vt E TS: (t, S) E Rc.

As Vt E TS: (B,,Bs) E
Valid(TS,S).

Validation of the dynamic behaviour of a test case can be mapped to the problem of validating that the assignment of test verdicts, i.e. Pass, Fail, and Inconclusive, is consistent with the behaviour expressed in the specification. To check if the assignment of verdicts is consistent with a specification it suffices to analyze the sequences of test events assigned a final verdict. Let To”(t) be the set of test event sequences with a final

E Kristoffersen,

T Walter/Computer

Networks

and ISDN Systems 29 (19%)

IS-47

39

verdict assignmtent. TrFV( t) can be partitioned into disjoint sets dependent on the verdict assigned: TrPass(t), Trl,,,,,( t), and T&ii(t) . Intuitively, validation of a test event sequence u E Trpa,,( t) has to confirm that the sequence of events is possible according to the specification. For an instantiation of the validation framework also a mapping of test case events to events of the SDL specification and vice versa has to be defined. To define the mapping, events of the domain and range must be selected. Example 14. An intuitive choice of CSR events to include in the validation process is the events which are exchanged at the system level. In terms of the CSR that is: ES = {input (sig) , output(sig), T}. One reason to select this set of events is that they constitute the events of interest when conformance testing is performed. Hiowever, an SDL system may always receive a signal if there is a channel that can convey the signal. Then to perform a more detailed analysis of the consistency of a test case, the validation process must cover also internal behaviour of the SDL system. So, the behaviour validation should not be restricted to system level events.

For the definition of a relation between TTCN test case events and SDL specification events it should be noted that the behaviour of an SDL specification and a test case reflect two different views on the system requirements. An SDL specification expresses the dynamic requirements that an implementation must satisfy, while a test case specifies sequences of test events to be applied to determine the conformance of an implementation. We shall use the notation x to denote complementary behaviour of X, where X denotes behaviour of either a test case or ;an SDL specification: If x denotes valid behaviour of a test case t, then the complementary behaviour of the SDL specification is denoted R. One way to define the static relation between specification and test case events is by mapping tables as described in [ 271. Note that we shall overload the overline operator ?Z to indicate complementary descriptions for behaviours, traces, events, and data parameters. 6.3. An instance of the validation framework The previous discussion suggests to define correctness criterion <,C in terms of the dynamic behaviour of test cases and SDL systems. The CSR provides a model that allows to reason about the dynamic behaviour of test cases and SDL systems. 6.3. I. Trace validation As the instantiation of the framework we aim at is concerned with the correctness of verdict assignments in a test case, only behaviour is considered for which a verdict is assigned. Thus, if the validation process is based on traces of system level events only a subset of the traces is used during the validation process. We denote the set of traces of a test case t by Tr(B,) and the trace set of the specification S by Tr(Bs). A trace for which a verdict is assigned is termed a complete trace and the set of complete traces is Tr,,(B,), where Tr,,(Br) G Tr( B,). Similar to the partitioning of the behaviour sets described previously, we introduce three sets of traces, one for each type of verdict assigned: Tr,,,

(

&

1

q :

TrPass(

&

>

U

hconc

(BI)

U

%il(Bt).

(3)

A minimal requirement that a test case must satisfy in order to be considered valid with respect to an SDL specification is that all behaviours indicating non-conformance, i.e. result in a Fail verdict assignment, should have no corresponding behaviour in the SDL specification. In terms of the set of complete traces this criterion is formalized in the following definition. Definition 15. The behaviour B, of a TTCN test case t is valid with respect to the behaviour of an SDL

system Bs if

40

E Kristoffersen,

Br &I where &i

Bs

iff

V’a E Trrait(

Z Walter/Computer Bt):

Networks

and ISDN Systems 29 (1996)

i7: $ Tr( Bs),

denotes the relation expressing the correctness criterion and F the trace that corresponds

Additionally, for complete traces assigned a Pass or Inconclusive complementary behaviour of the SDL system. Definition 16. The behaviour of a TEN Bs if & 6

15-47

BS

iff

Bt

6

Bs

Vu

E TOP,,,

to u.

verdict, it may be required to have a

test caseB, is valid with respect to the behaviour of an SDL system

A

u 7”1nconc(Bt):

5 E WBs)

where &2 denotes the relation that expressesthe correctnesscriterion. Definitions 15 and 16 are based on comparison of complementary traces of input and output events of a TTCN test case and an SDL specification. A prerequisite in order to perform a comparison of traces is that a correspondencebetween inputs and outputs has been defined. Since such a correspondencecannot usually be defined on a pure syntactical basiswe assumethat a static mapping between TTCN and SDL events is defined, e.g. Corresponding event of CSR event of TIYCN test case SDL system input(x) output(x)

output(Y) input(X)

where the overline operator used for parameter x is to be resolved using a mapping between TTCN and SDL parameter names. Validation of invalid traces is not as straightforward as indicated by Definition 15. In TTCN, the Otherwise statementdefines a controlled way to handle unforeseenevents. So traceswhich include an Otherwise statement lead to a Fail verdict assignmentaccording to [ 141. An Otherwise construct will not only match unforeseen events but any event that is not mentioned in the set of alternatives. This means that also input actions that are correct with respect to the specification may be handled by an Otherwise construct. We therefore propose another approach for the validation of such invalid traces. Intuitively, an Otherwise statementis valid if the traces specified in the SDL specification are all covered by complementary traces in the TTCN test casesuch that these traces are all membersof the Pass or Inconclusive trace sets. In order to formalize this definition we introduce the following notation. The set of othenvise truces, denoted Tr, (B,) , is defined as = {crlcr. x E

Tr,(B,)

Tr(B,)

and input x is consumedby an Otherwise}.

This set consistsof traces that may be extended by an event such that this event is handled by an Otherwise statement. Similarly, we define the set of traces of a specification S suchthat these traces are possibleextensions of complementary traces p E Tro( B,) : Tr(p,

Bs)

= {A

] A E Tr( Bs)

A x E ES -

{T}

A j3. x = A}.

Next we define predicate VulidOTr that is satisfied if for all valid extensions I+ of trace p E Tr,(Bt) with (B,) such that F is a prefix of u’, denoted ZF5 u’. respect to specification S, there is a u’ E Trpass( Bt) U Trh,,,,

E Kristoffersen,

‘I: Walter/Computer

Networks

and ISDN Systems 29 (1996)

15-47

41

Fig. 12. Behaviourmodelsof a test case& and two SDL systemsB(U) and B(S2) The intended use of this predicate is to check that for all possible traces with respect to the specification these traces are mapped to traces in the test case that result in a Pass or Inconclusive verdict assignment. Definition 17. Let S be an SDL specification. Let p be a trace in the set of otherwise traces Tr,(B,). ValidOTt- is de:fined as follows: ValidOTr(

iff

p)

Q’a E Tr( p, Bs) : 3~’ E Trpass(B,)

U

Redicate

Trl,,,,,( B,): 5 j v’.

Then we define a correctness criterion that integrates the validation of the Otherwise statement. To support the validation process we define a reduced set of traces associated with a Fail verdict. The trace set TreFail2 Tqail is the set of Fail traces such that no event is handled by an Otherwise statement. Definition 18. The behaviour of a ‘ITCN Bs if Bt

G3

iff

Bs

test case B, is valid with respect to the behaviour of an SDL system

QU E TreFail(B,) I is: $?!Tr( Bs) A Qu

E Tr~ass(Brl

U Tnnconc(B,):

F

E NBs)A

Qa E Tr,( B,) : ValidOTr(a)

Summarizing, &3 is a refined definition of
shown, For the validation process based on Definition TAP&

&I

=

{ (acd)

) I

hconc

(&I

= 0,

Trot&)

test case t and two SDL systems Sl and S2 are 18 we derive from the test case behaviour: = {(a)),

TY,Fail(Bt)

= {lab)}.

To validate the TTCN behaviour against the SDL behaviour Bsl we must check that the SDL specification cannot perform the Fail trace (i$). This is seen to be the case. Furthermore, we observe that the SDL system can perform a trace (acd) that corresponds to the Pass trace. Finally, as BSI after sequence (a) cannot perform any other action than Z that is a prefix of a Pass trace, then also the requirement on Otherwise statements is satisfied. Hence test case t is valid with respect to SDL specification Sl. For the SDL specification S2 it can be shown that the requirements on Fail and Pass traces are satisfied. However, for the Otherwise trace (a) it is seen that the SDL specification may perform (@ and (Zi?). The former is not a prefix of any Pass trace and so it is handled by the Otherwise statement. From this analysis we conclude that test case t does not satisfy correctness criterion &3 with respect to SDL specification S2.

42

E Kristoffersen,

iY Walter/Computer

Networks

and ISDN Systems 29 (1996)

15-47

6.3.2. Event validation To perform the trace validation defined we must also define how the events of these traces are validated in terms of the CSR model. This is necessary as the validation of an input event of a TI’CN test case is done by checking that the SDL system is able to perform the corresponding output event. If a test case sends a message it is checked that the SDL system is able to receive a corresponding signal. Since an SDL system may always receive a signal if there is a channel that can convey the signal, a stronger requirement should be used for validation of TTCN test case output events. A test case output event is valid, if it can be proven that the corresponding signal is received and consumed by a process instance within the system. To perform event validation the set of complete traces needs to be derived first. The jnal verdict predicate Fv identifies traces of system level events which are assigned a final verdict. Definition 20. Let t be a test case and B’ be the corresponding CSR entity. Let the behaviour of test case t be

given by Ss, = (S,, at, -t , so). Let s’, s” E St where s’ = ( (+I,. . . , u+k,(~k+i, . . . ,~l,Pl,...rP,...,Pn)), and s” = s’[p’/p]. 6 In the state description gi, 1 < i 6 k, denote PC0 queues, (pi, k + 1 < i ,< I, denote CP queues, and pi denotes process instances of test components. For (T E .s: the final verdict predicate W(a) is defined as Fv( (T)

iff

SO5

s’ A s’ +

p --%I

p’

s” A

A P = (Pc,p,

1, T) A p’

= (f’&,r,

I, T)

A

P ::=R := v;P’ A p’=p[Rw] where v E {Pass, Fail, Inconc}, I denotes the state of the input process, and T denotes the state of the timer process. Notation -% denotes a sequence of observable events. Since execution of the test case stops in an assignment of a final verdict (IV), a complete trace is a trace that cannot be extended. TrCp(B’) = {a E 7’r(B’) ) W(a)} is the set of complete traces of test case t. For the validation of output events of a test case it must be validated that the SDL system is able to receive and to consume the complementary input event. We define predicates R and C related to the reception and consumption of a message. Definition 21. Let S denote an SDL specification and Bs the corresponding CSR entity. The CSR system level

description of the behaviour is denoted Ss, = (Ss, as, -s current state s= ((+I ,...,

Uk,rnl,...,

(..., &id;,...),j

, so). Let s, s’, s”, s”’ E Ss where s denotes the

,..., m/n>,

S’ = S[ al/ail and S” = s’[&,/~+,id’, ay/ai] . In the state description (+ir 1 < i < k, denote queues of links (channels) and mi, 1 < i ‘< m, denote modules (blocks). Then the receive predicate R( e, s) where e = input(msg) E es and msg E Sig is satisfied when

R(e,s)

iff

s&s’A(~~=msg*~iA S’ -S

ppid; fly

7

d’

A &,;d,

= (&,

(a,

receive( -PI

g’),

msg) p;;d,

T) A p;jd’

A

= (pe+m

(msg

. u, (+‘> 9 T) A

= (J-i

where (u, (T’) denotes the state of the input queue and T the state of the timer process. Note that (Ti must be empty to have R( e, s) be fulfilled. “The

notation

s’ =

s[t’/t]

denotes

that

S’ is equal

to s except

for that

t’

has been

substituted

for t.

E Kriaoffersen,

7: Walter/Compurer

Networks

and ISDN

Sysrems 29 (1996)

43

15-47

ProcessP v State1

PCO?x

I a

I b

X

Y

J, State1

Fig. 13. A test case I and process part of a corresponding

SDL specification.

The consume predicate C( e, s) may now be defined based on the receive predicate. Let state s”’ = s”[p$/&,,]. Then the consume predicate C(e,s> where e = input(msg) E ES and msg E Sig is satisfied when C(e,S)

iff

R(e,s) A

pLidi= (Pe,p,(a *msg,~‘1, T) A &, = (f’&, tu, ~‘1~7’)A P

::=

input(x);

P’ A p’ = p[x H msg]

of Pass, Inconclusive, and Otherwise tracescan be defined using the requirements on events. The following predicate sketch how these can be validated. Let Bs denote the CSR entity of an SDL system against which a trace 0 of system level events is to be validated. Let the system level of BS be S~~ = (Ss, es, --+s , SO)and s, s’ E Ss where s denotes the current state. Finally the CSR representation in state s’ is denoted B& The predicate CheckTruce(a, Bs) then defines the requirement. The

validation

CheckTrace(cr, Bs)

iff

tg = 0) V (((7 = e. d) ts ss V (((+=e.a’) ts -S

A (e = input(x))

A

s’) A ( CheckTrace(d, Bk) ) ) A(e=output(x))

A

s’) A C(Z,s) A (CheckTrace(d, B&)))

We can use this predicate also as a basis for the validation of Fail traces. In this case it must be shown that the predicate CheckTruce is not satisfied. We ihustrate the use of this validation criterion in the following example. The example does not use constructs from concurrent ‘ITCN as it would expand the description of the example significantly. However, the principles of the validation procedure remain the same.

Example 22.

An outline of a test case t and a process of an SDL specification S is shown in Fig. 13. The following process may be performed to validate the correctness of test case t according to the proposed instantiation of the validation framework. First, from the CSR representation of the test casewe derive the set of complete traces to be validated. The basic processexpression of the test case behaviour in the CSR is

44

E Kristoffersen, Pf

7: Walter/Computer

Networks

and ISDN Systems 29 (1996)

15-47

::= output(a) to PCO; (input(x) from PC0 ; R := Pass ; stop ; nil) @ (otherwise PC0 [true] ; R := Fail ; stop; nil)

The final verdict predicate FV is used to derive the complete traces. From the CSR representation we get the non-empty sets of system level traces: Trpass(Br) = {(output(a) . input(x))} and Tr,( B,) = {(output(a))}. The internal behaviour of the SDL specification is needed also to validate the test case traces. The CSR representation of the process P is P ::= (input(a)

; output(x)

; P) + (input(b)

; output(y)

; stop;

nil),

where the occurrence of P in the expression represents the infinite behaviour of process P. Validation of the Pass trace is performed according to the CheckTracepredicate. The initial event of the trace is output (a), then it must be derived from the CSR representation of the SDL specification that a complementary input action is possible and that this signal is received and consumed. If si denote system states of the CSR representation Bs with so the initial state, then to validate the output event we must derive: SO 3s

~1 A C(input(a),so)

i.e. the SDL system can perform an input event and this input is received and consumed by a process instance (predicate C). As the links may always receive a signal that can be conveyed on the channel, the transition from SOto st is always possible. From state st, a process instance in the system must be able to receive signal a, that is receive(a)

PpidApI

p;jd

A

sl &S

‘2

where ppid and &, denote states of a process instance. As signal a is in the input set of SDL process P, the process instance that represents the behaviour of the process always can receive the signal in its input process. Then from the derived system state s2 it must be shown that signal a can be consumed. In the initial system state SO the state of the basic process is equal to the basic process expression P. As the previously described transitions has not changed the state of the basic process we can derive that from system state 32 the process instance can perform an internal event, consuming signal a from the input port.

Hence we can conclude that the test case output event is valid. Then from system state s3 we must show that the system can perform output event x as the next external system level event. This corresponds to the next action of the Pass trace, i.e. input(x). When this event is derived, the trace is empty and predicate CheckTrace satisfied. As there is no other Pass trace to be checked, what needs to be done is to show that the requirement on the Otherwise statement is satisfied. The only system level event that the SDL system can perform after input event a is output event x, so the requirement is satisfied. Finally, we can conclude that test case t is valid with respect to SDL specification S for correctness criterion <3. From the example it is seen how correctness of a test case can be determined. But from the example it should be clear also that for real test cases the validation process must be supported by tools.

F: Krisioffersen,

i? Walter/Computer

Networks

and ISDN Systems 29 (I 996) 15-47

45

7. Conclusions

In this paper we have given an introduction to TTCN and concurrent ‘ITCN. Our focus has been on the semantics of TTCN and concurrent ‘ITCN and on the validation of test cases. We have started out from an informal presentation of TTCN and concurrent TTCN. Whereas in TTCN only one lower tester and optionally one upper tester are active, concurrent TTCN supports the definition of complex test configurations which, in general, consists of several concurrently running test components. Since several test components are involved in the execution of a test case, an instrumentality is needed supporting communication between test components, This instrumentality is given in terms of coordination points capable to convey coordination messages between test components. Although concurrent TTCN has introduced a further level of complexity in the specification of test cases, its official semantics definition [ 151 Annex B has been kept simple. The behaviour of a concurrent TTCN test case is defined using a ‘ITCN machine that interprets a TTCN behaviour tree in canonical form. Test components are executed independently of each other except that they can communicate by exchanging coordination messages. We have looked into the semantics definition of TTCN and concurrent TICN and identified a few shortcomings. In the alternative approach for the definition of an operational semantics for TTCN and concurrent TTCN we have demonstrated how these problems may be solved. We have defined a specific model of a test system which has a hierarchical structure. Every entity in the model has an operatmnal semantics defined in terms of labelled transition systems. We have defined our semantical model as a hie.rarchical and compositional model so that the semantics of higher level entities is deduced from the semantics of its constituent parts. In order to demonstrate the suitability of our definition of an operational semantics we have discussed an approach for the validation of TTCN test cases. We have distinguished two phases. First, general properties of a test case are validated; for instance, syntax and static semantics and proper termination. In a second phase we have used the protocol specification to enable validation of test case behaviour against the dynamic properties of the protocol specification. For example, we have demonstrated principles for validation of behaviour associated with a Fail verdict with respect to the behaviour defined in the protocol. For the validation of test cases we have also utilized the abstraction properties of the semantical model. For example, we have defined the validation process for the test case output message to include also that the protocol specification must be able to consume the message. This means that we shall identify a process instance in the specification that from its current state is able to perform an input action and consume the message, To perform this we need a model which allows to identify the behaviour at different levels of abstraction, which is exactly what the model does. We see our definition of an operational semantics for TTCN and concurrent ‘ITCN not as a replacement but as a supplement to the officially released semantics definition. The official semantics definition provides an intuitive clear definition of the semantics of ‘ITCN and concurrent ‘ITCN. For formal reasoning the described operational semantics provides a better basis. Also for validation and implementation of tools the operational semantics complements the standardized semantics definition.

Appendix

ASP BPA CM CP CSR

A. List of abbreviations

Abstract Service Primitive Basi.c Process Algebra Coordination Message Coordination Point Common Semantics Representation

46

E Kristoffersen.

ETSI EWOS FIFO IAP IS0 ITU IUT LT LTCF LTS MTC PC0 PDU PTC TC TCP TTCN UT

T. Walter/Computer

Networks

and ISLW Systems 29 (1996)

15-47

European Telecommunications Standards Institute European Workshop for Open Systems First-In-First-Out Implementation Access Point International Organization for Standardization International Telecommunication Union Implementation Under Test Lower Tester Lower Tester Control Function Labelled Transition System Main Test Component Point of Control and Observation Protocol Data Unit Parallel Test Component Test Component Test Coordination Procedure Tree and Tabular Combined Notation Upper Tester

Acknowledgment

We would like to thank Jean-Philippe Favreau, System and Network Architecture Division, National Institute of Standards and Technology, who provided us with an example of a concurrent TTCN test case. We are also indebted to Bernd Baumgarten, GMD Darmstadt, Germany, and OS Monkewich, DSI, Canada, who provided further information on TTCN and particularly concurrent TTCN. The authors would like to gratefully acknowledge the anonymous referees for fruitful comments to the previous version of this paper. References [ 11 U. Bar and J. Schneider, Automated validation of TTCN test suites, in: Prac. PSTV XII (North-Holland, Amsterdam, 1992). [ 2 J J. Bergstra and J. Klop, Algebra of communicating processes with abstraction, Theoret. Camput. Sci. 37 ( 1985). ] 3 ] M. Broy, Towards a formal foundation of the specification and description language SDL, Forma/ Aspects Compur. ( 199 1). ]41 S. Chanson and Q. Li, On inconclusive verdict in conformance testing, in: Proc. PTS V (North-Holland, Amsterdam, 1992). [S] M. Dubuc, G. Bochmann, 0. Bellal and E Saba, Translation from ITCN to LOTOS and the validation of test cases, in: Proc. FORTE ‘90 (North-Holland, Amsterdam, I99 1) [ 61 ETSI STC MTS, ETSI project semantical relationship between SDL and ‘ITCN: Feasibility study of the direct mapping and common semantics representation approach, ETSI Technical Committee Methods for Testing and Specification, 1991. [ 7] ISO/ITU-T, Formal methods in conformance testing, ITU-T TS SGlO 48 and IS0 SC21 WGI P54 Southampton output, 1994. ( 8] J. Godskesen, An operational semantic model for basic SDL, TFL Tech. Rept., TFL RR 1991-2, 1991. [ 9] J. Groote, Process algebra and structured operational semantics, Ph.D. Thesis, University of Amsterdam, 199 1. [ 101 D. Hogrefe, Observable behaviour of SDL specifications for the purpose of testing, CCITT X/3, 1989. [ 1 I ] ISO, Information processing systems - Open Systems Interconnection - LOTOS - A formal description technique based on the temporal ordering of observational behaviour, IS0 8807, 1989. ] 121 ISO/IEC, Information technology - Open Systems Interconnection - Conformance testing methodology and framework - Part I: General concepts, ISO/IEC 9646-1, 1994. [ 13 ] ISO/IEC, Information technology - Open Systems Interconnection - Conformance testing methodology and framework - Part 2: Abstract Test Suite specification, ISO/IEC 9646-2, 1994. [ I4 ] ISO/IEC, Information technology - Open SystemsInterconnection - Conformance testing methodology and framework - Part 3: The Tree and Tabular Combined Notation (‘ITCN), ISO/IEC 9646-3, 1992.

E Kristoffersen,

7I Walter/Computer

Networks

and ISDN Systems 29 (1996)

I.547

4-l

[ 151 ISO/IEC, Information technology - Open Systems Interconnection - Conformance testing methodology and framework - Part 3: The Tree and Tabular Combined Notation (‘ITCN): Amendment 1: TTCN extensions. ISO/IEC 9646-3 DAM 1, 1993. [ 161 Telelogic, Using the ITEX-DE TTCN editor Version 2.2, Telelogic AB, 1994. [ 17 1 R. Linn, Conformance evaluation methodology and protocol testing, /EEE J. Selected Areas Commun. 7 (7) ( 1989). [ 181 E Kristoffersen and T. Walter, Test case validation TTCN test case validation against SDL specifications. TIK-Rept. Nn. 8; also available as Tele Danmark Research Rept. TDR RR I994- I, 1994. [ 191 R. Mimer, Communication and Concurrency (Prentice-Hall, Englewood Cliffs, NJ, 1989). 1201 G. Plotkin, A structural approach to operational semantics, Computer Science Dept., Aarhus University, 1981. [ 2 I 1 R. Probert and 0. Monkewich, TTCN: The international notation for specifying tests of communications systems, Computer Nehvnrks ISDN Systemr 23 ( 1992). [22] R. Reed, W. IBouma, J. Evans, M. Dauphin and M. Michel, Specijication and Programming Environmentfor Communication Sojhwre (North-Holland, Amsterdam, 1993). 123 1 B. Sarikaya .md Q. Gao, Translation of test specifications in TTCN to LOTOS, in: hoc. PSTV VIII (North-Holland, Amsterdam, 1988). 124 1 B. Sarikaya, Conformance testing: Architectures and test sequences, Computer Networks LSDN Systems 17 ( 1989). 1251 SPECS RACE 1046, Definition of MR and CRL Version 2.1, SPECS-Semantics and Analysis, SPECS consortium, 1990. 1261 M. Torii and K. Tamay, Principles for validation of abstract test suites specified in concurrent TTCN. in: P. Dembinski and M. Sredniawa, eds., Proc. Protocol Specification, Testing and Ver$cation ( 1995). [ 27 1 R.J. Velthuys, Conformance testing based on formal system models, Ph.D. Thesis, University of Beme, 1992. [ 28 I S. Vuong and J. Alilovic-Curgus, On test coverage metrics for communication protocols, in: .I. Kroon, R. Heijink and E. Brinksma. eds., lWPTS IV, IFIP (North-Holland, Amsterdam, 199 I ) 3 l-45. I29 I T. Walter, J.. Ellsberger, F. Kristoffersen, and Pv.d. Merkhof. A common semantics representation for SDL and TTCN, in: Proc. PSTV XII (North-Holland, Amsterdam, 1992) [ 301 T. Walter, J. Ellsberger, E Kristoffersen and P.v.d. Merkhof, Methods for testing and specification (MTS) semantical relationship between SDL and TTCN a common semantics representation, European Telecommunications Standards Institute, ETR 07 I, 1993. [3 I 1 T. Walter and B. Plattner, An operational semantics for concurrent TTCN, in: Proc. PTS V (North-Holland, Amsterdam, 1992). [ 321 ITU/CCITT. Data Communication Networks: Open SystemsInterconnection (OSI); Model and Notation, Service Definition, X.200, 1989.

[33 I IUT, Functional Specification and Description Language SDL. ITU-T Recommendation, 1992. Finn Kristoffersen is M.Sc. EE from the Technical University of Denmark in 1986, from the Department of Computer Science. Since 1987 he has been working at Tele Danmark Research in the Software Engineering Department. From 1987-1990 he worked with SDL tool development. Since 1991 he has worked with test methods based on formal descriptions of protocol systems. He has been involved in standardisation within ITU and participated in an ETSI (European Telecommunications Standards Institute) project on conformance testing based on SDL specifications. His current research interests are in the areas of formal methods in conformance resting and software quality measurements.

Thomas Walter performed his studies at the University of Hamburg from where he received the Diploma in Informatics in 1987. He then moved to Zthich working towards a Doctorate degree which he received in 1993 from the Swiss Federal Institute of Technology in Ztirich (ETHZ). From 1980 to 1986 he was appointed as software engineer. Since 1986 he is at the Computer Engineering and Networks Laboratory (TIK) in Zurich where he holds a position as senior research assistent (Oberassistent). In the years 1991 and 1992 he led a project at the ETSI (European Telecommunications Standards Institute) and in 1994 he participated in an EWOS (European Workshop for Open Systems) project. Both projects were in conformance testing of communication protocols. His current research interests are in formal methods and conformance testing. He recently started to work on the application of formal methods for the validation of real-time properties of high speed protocols, particularly quality-of-service testing. Besides this he is also active in setting up a infrastructure for teleteaching at ETHZ. Dr. Walter is member of the Gesellschaft Rir Informatik (GI) and IEEE.