Protocol quality engineering: addressing industry concerns about formal methods

computer communicatii

ELSEVIER

Computer Communications 19 (1996) 1258-1267

Protocol quality engineering: addressing industry concerns about formal methods Robert L. Probert, Ning Lew Telecommunications Software Engineering Research Group, Department of Computer Sciencellnformatique, University of Ottawa, Ottawa, Ontario KIN 6N5, Canada

Abstract In this paper, we describe a quality-directed perspective on the lifecycle process of designing and assembling communications systems and services. We claim this perspective addresses some of the industrial concerns of quality and productivity for the protocol engineering process,

while allowing for some of the best formal techniques known for protocol synthesis, verification, conformance testing and performance assessment. We hope that this perspective will assist in the development of a generic conceptual framework which enables the evolution, integration and practical application of protocol engineering models, methods, languages and tools. Keywords:

Protocol engineering;

1. Introduction

Communication

probcols;

Quality engineering

and motivation

One of the more promising areas of research and development in terms of its potential for enhancing the quality of industrial products and the productivity of the development processes is that of Protocol Engineering [l] (more generally, Communications Systems Engineering). By protocol engineering (PE) is meant ‘the efficient use of trusted components, methods and tools to construct an integrated architecture of devices and processes which collaborate to provide desired communications services while satisfying constraints such as cost, time, quality, reliability and safety’ (an integration and enhancement of definitions from Turner)

PI* Protocol engineering is a relatively new field, but has reached the stage where several experimental case studies are in progress to relate techniques and tools to real industrial concerns. Unfortunately, even in the area of automated test generation, the most advanced technology in protocol engineering, there is only one reported industrial use of a formal method (the Unique Input Output method). In general, the impact of formal methods on industry is not yet significant [3]. However, we are aware of a number of large corporations who are using other formal techniques intemally but have not discussed their experience in the public literature. The term protocol quality engineering (PQE) is a new 0140-3664/96/$15.00 0 1996 Elsevier Science B.V. All rights reserved PII: SO140-3664(96)01159-O

term intended to denote ‘the integrated process which results by incorporating methods and tools for ensuring quality of product into the protocol engineering process itself’. A similar term, sofhvare quality engineering, is used, for example, to denote the production of high quality software systems [4,5].

1.1.

Quality factors

It is recognized that the term ‘quality’ may easily be the subject of many treatises in itself. However, most working definitions include quality factors such as reliability, safety, liveness, robustness, efficiency and completeness (in order approximately by user priority). We briefly discuss these factors below: l

l

reliability - refers to the released product and measures the degree to which the product will not fail entirely, nor fail often. More precisely, reliability is the extent over time to which a program can be expected to perform its intended function within the required performance and precision constraints. safety applies primarily to the design phase, and refers to the verified absence of deadlocks and other design errors. A design may be verified as safe with respect to specific types of errors. Informally, the design satisfies safety if it can be demonstrated that

R.L. Probert, N. LewlComputer Communications 19 (1996) 1258-1267

l

l

l

l

anything undesirable will never happen. Deadlocks, livelocks, unspecified receptions and assertion violations would all be classified as violations of safety properties

quality factor. This is essential for effective management project resources.

[6771.

1.2. The Protocol Quality Engineering (PQE) cycle

liveness - also applies to the design phase. Informally, the design satisfies liveness if it can be demonstrated that something desirable will eventually happen [6]. For example, a protocol should guarantee that data sent will always arrive eventually, or that appropriate diagnostics will be received. robustness - refers to the product or service released to the customer. However, it implies a local notion of robustness at the design phase and prototyping phases as well. At the requirements phase, robustness is the degree to which the service is able to withstand inappropriate or unexpected requests or responses from the user, and to some degree, the ability to withstand inappropriate underlying service behaviours. The system should respond with, at worst, graceful degradation of service, rather than catastrophic failure. efficiency this has several meanings at different phases. Informally, it refers to the degree of minimization of costs and overhead associated with performing a function. At the requirements phase, it involves the lack of delay and minimization of cost in delivering a service. At the design phase, it can refer to the transmission rate of data relative to control information for a specific quality of service. completeness - again, refers to several phases. At the requirements phase, this refers to the proportion of functionality and capability provided to the user. At the design phase, a protocol is said to be incomplete if at any state, one entity is offered a message for which the reaction is unspecified.

Other quality factors which should be considered include capacity and stress tolerance, fault tolerance and recoverability, ease of repair, ease of adaptibility to change, testability, ease of use, interoperability, and so on. These are of varying degrees of importance to users or clients and to designers, system integrators and field support personnel. Part of the PQE process involves the identification and ranking of these quality factors, as well as the derivation of related appropriate metrics, and an assignment of measurement and assessment activities to specific phases. Some of this is indicated in the appendix, but space limitations do not allow a comprehensive treatment here. All of these quality factors can be formal or informal methods (in principle), and all contribute to the overall quality of the resulting protocols. However, industry is not satisfied with what can be achieved by methods inprinciple. PQE researchers must now focus attention to practical industrial concerns [3]. For example, industrial strength metrics and tools must be developed to allow the assessment of present quality level and estimated distance (time, cost) to achieve the appropriate target quality objective for each

1259

of

There already exist several approaches to generalpurpose software development and software engineering. Amongst the better-known of these are the waterfall model, the spiral model, the cleanroom process [8], and their variants. The PQE model builds on the strengths of each for protocol engineering; PQE emphasizes the significance of quality and risk assessment throughout the process. The Protocol Quality Engineering cycle (Fig. 1) is represented as a ferris wheel (a wheel of wheels) consisting of an outer wheel of progress towards the finished service or product and an inner wheel of rework from which any previous phase may be re-iterated to respond to quality or risk assessment concerns. Typical quality concerns were listed above. Risk assessment considers additional related factors such as consequence of error, potential for loss or damage, the estimated probabilities of each, and the costs associated with trying to avoid undesirable outcomes [9-111. At each local stage or phase in the PQE ferris wheel, two basic types of activities are carried out. The first type of activity (PE activity) involves essentially synthesis or building of a PE artifact, and the second type of activity (PQE activity) involves assessing the quality of the artifact and the degree of risk associated with continuing to the next stage of development. The quality and risk assessment activities conducted at stage i are denoted Q&R:i. For example, at the requirements phase, requirements capture involves assembling sequences of service primitives to satisfy customer objectives, a PE activity. In Q&R:l, each usecase (sequence of service primitives (SPs)) is verified for compatibility with the known ordering constraints. When a practical degree of completeness of requirements is attained, the entire service definition can be verified for potential loss of synchronization due to concurrency, and potential unspecified receptions. All types of verification or validation, whether incremental or monolithic, are PQE activities. A working definition and illustration of the terms verification and validation is given in Section 3. Both quality and risk assessment (Q&R:i) contribute to a ‘go’ or ‘no go’ decision at the end of phase i. A ‘go’ decision implies movement on the wheel of progress in a clockwise direction to phase i + 1. A ‘no go’ decision implies an exit to the wheel of rework, traversal in a counter-clockwise direction, and exit to the appropriate previous phase to conduct the rework activity. Any previous phase is accessible from the wheel of rework; thus the PQE ferris wheel is capable of representing most other cycles and models including waterfall, object-oriented, iterative, spiral, fast prototyping, and re-engineering. Finally, with suitable mapping of phase labels, and appropriate mapping of quality and risk assessment activities, the PQE model can also represent the cleanroom model for system development [8].

1260

R.L. Probert, N. LewlComputer Communications I9 (1996) 1258-1267

Legend Q&R:iQualityassurance

& risk assessment activities at phase i (see Appendix) means ‘proceed to next phase’

I-&,

a

means ‘rework is required

Fig. 1. The protocol quality (re-)engineering

Part of the responsibility of a protocol quality engineer is to recommend the process (progress and rework) activities which have proved to be most cost-effective for his or her organization, together with recommended synthesis techniques, quality and risk assessment methods, and tools for each branch in these workflows. The outline of this paper follows exactly the PQE cycle depicted in Fig. 1. Sections 2-6 discuss the issues that arise at each of the five phases: (1) service requirements capture and revision; (2) design (synthesis and re-engineering); (3) component prototyping; (4) system integration; and (5) product release engineering. We conclude with a discussion relating the PQE framework to the work of others.

2. Phase 1: Service requirements

capture/revision

The starting point for any well-engineered product is a thorough understanding of its domain of use and the real requirements of users. Real requirements are often not explicitly captured in a formalized statement of requirements. This step in the PQE cycle is denoted Service Requirements Capture/Revision in Fig. 1. Requirements may be revised to reflect changing user requirements or domain constraints as a result of clarifications obtained during the iterative development cycle. Service requirements (SRs) may be captured using a variety of models and formal description techniques [2,12-161.

ferris wheel.

The fundamental components of SRs are: l

l

l

the syntax of Service Primitives (SPs), the messages which provide the service interface to users, the ordering relationships among SPs (service constraints), and the semantics associated with a service capability or function.

Any of the three standardized Formal Description Techniques (FDTs) namely ESTELLE, Lotos and SDL, is adequate for the first component. None of the FDTs appears to have been widely accepted for the third component. Two approaches seem to have coalesced for the second component, representing ordering constraints. 2.1. Representing service constraints (ordering) One approach described by Turner is the scenario-based approach in which legal end-to-end service scenarios are described explicitly. Time-sequence diagrams (TSDs) are proposed for representing individual scenarios [2]. These scenarios may be parameterized to represent service capabilities, and then represented in Lotos. Another approach under development is the use of amrotated finite state machine (FSM) models or Extended FSM models to represent the SPs (including parameters), which may be exchanged between users and the service provider at a number of Service Access Points (SAPS). In the current

R.L. Probert, N. LewlComputer Communications 19 (1996) 1258-1267

”

E2

A Al

s2

82

s3

Cl

Dl

s4

Fig. 2. Service state machine model.

version of this approach, the SRs are represented simply by service FSMs (no parameters). A simple example of such a service specification is shown in Fig. 2 [121.This service has five SPs (A, B, C, D, E) and two SAPS (1 and 2). Thus, for example, in service provider state S2, a C message may occur at SAP1 or a B message may occur at SAP2. This reflects typical concurrency in a distributed service provider. Concurrency can be handled at the requirements stage in a number of ways depending on the user’s preference; for example, the user may require full synchronization (no collisions allowed), priority assignment, or recovery by reset or rollback. This is described and illustrated in the paper by Saleh and Probert [12]. Both of these approaches have their benefits. The Lotos (scenario-driven) approach appears to be better suited to SR capture by enumerating scenarios. This is similar to the popular Object-Oriented approach to system analysis and design which also emphasizes requirements capture by use-cases. (A ‘use-case’ is a sequence of normal interactions and related exceptional interactions between user and system in order to achieve a user function [17].) Use-cases (and message sequence charts (MSCs)) are extremely useful for constructing service-based system and sub-system tests. Part of the PQE process at this phase is the planning, construction and certification of use-cases and related service test cases. On the other hand, the Service FSM approach is more compact, easier to verify for consistency, and currently (in the case of SDL as a representation for Service FSMs) better supported by commercial grade tools. Both approaches can be utilized for automated or semi-automated protocol synthesis, especially if the FDT used for protocol specification is the same as that for the service requirements. In an ideal world, wherein cross-checking of distinct views for consistency is recommended, a declarative method should be used to capture the customer view, (for example, Lotos scenarios), and cross-validated against the network view captured by a more process-directed method (for example, communicating extended FSMs). However, the cost of such a complementary-view based approach is still prohibitive. Indeed, both approaches have their disadvantages. Industrial-strength tools for Lotos are minimal, and methodologies for both verification and validation of requirements

1261

are not well supported by tools in either approach. Again, SDL tools such as TeleLogic’s SDL/SDT and Verilog’s GEODE kit of tools have some verification and validation capabilities, basically reachability analysis and interpretive execution of scenarios (using MSCs in SDL). However, these tools are really based on a forward engineering paradigm, either interpretive simulation or compiled prototyping. Neither is ideal for the type of interactive development which occurs when requirements are continually changing. Research is needed into whether these approaches can be extended appropriately or whether entirely new approaches are required. Finally, neither approach is strongly connected to practical industrial quality assurance activities, such as formal inspections and certification of consistency and completeness, regression analysis (new or modified requirements have not corrupted retained requirements), cause-effect analysis, and risk analysis. Typically, but unfortunately, tools which support one approach are not compatible with tools supporting the other approach.

3. Phase 2: Design (synthesis, re-engineering) Formal design for protocols still follows two basic design approaches: analytic and synthetic [7]. In the former approach, the designer starts with a preliminary version of a protocol by defining messages and the effects of their exchange. Standard design verification and analysis methods are applied to detect inconsistencies which may then be diagnosed and removed. The (PQE) cycle of analysis and correction is repeated until the design is free of errors. Certain aspects of this process have been automated. One weakness of the analytic approach is that it is often difficult to correctly modify an existing design. By comparison, the synthetic approach takes a partially specified design and completes it such that the interactions between its protocol entities proceed without manifesting errors and provide the set of specified services. The synthesis may be performed interactively, step-by-step, or auto,matically, without designer intervention. The resulting protocol should be inherently ‘correct’ if the synthesis method is sound, thereby eliminating the need for further verification. Unfortunately, many synthesis methods are not suitable for industrial application because they consider only synchronous communication, only two-party protocols, or only the control portion of the protocol (without the data portion). Current efforts are directed at methods for automating synthesis, for example, from execution traces [18] or from an FDT, like Lotos. Other work seeks to handle message collisions, allow for parallel execution of primitives, or investigate component-based assembly methods; however, these approaches are not yet suitable for adoption in industry.

1262

3.1.

R.L. Probert, N. LewlComputer Communications19 (19%) 1258-1267

Design fur fault tolerance and recovery

Because of hardware failures or other real-world problems, communicating entities may lose messages or arrive at inconsistent global states. The system as a whole then needs to recognise this and re-orient itself. Recovery, one aspect of fault tolerance, refers to the ability of a system to return to a known and stable state. Such a capability needs to be included at the design stage. The conventional strategy is to create a history of ‘checkpoints’ as the system executes. When an inconsistent condition is detected, the system can be ‘rolled back’ to the previous checkpoint. For distributed systems, the problem becomes more difficult because of timing considerations and the distributed nature of (global) reference points. Saleh and others describe a recovery algorithm that requires a minimum number of rollbacks on only the affected processes [19]. The method described is nonintrusive in that it takes advantage of the overhead information that is already being exchanged by the distributed processes. 3.2. Verification and validation at the design phase. In the general realm of software engineering, verification refers to the set of activities that ensure that software is designed right; validation refers to a different set of activities that ensure that the right software has been designed [17,5]. Within protocol engineering, the purpose of verification is to demonstrate the safety and robustness of the protocol. Safety can be checked by using reachability analysis to reveal conditions under which a protocol might deadlock or not know how to interpret an input. Likewise, there should be no unreachable states or non-executable transitions. Holzmamr and others have developed reachability analysis-based language and tools such as PROMELA/SPIN for design verification. Although state-space explosion is a common problem for such tools, efficient algorithms and inexpensive cycles permit the verification of moderate-size designs. Partial-order strategies can help make reachability methods more amenable to industry. Godefroid and others show that the problem of state-space exploration can be eased by selectively restricting the search to only those global states that are necessary to verify a given property [20]. Their verification tool was applied to a collection of virtual finite state machines modelling the AT&T ‘SESS’ telephone switching system. Validation in software or in protocol engineering involves checking that actual behaviours conform to worthwhile behaviours. For protocols, the key concern is liveness. Reachability analysis can also be used in principle to check liveness; however, most tools check only safety [6]. Moreover, formal methods must be enhanced to estimate system performance in order to address real-world concerns. A major problem is that a protocol does not generally account for particular aspects of the system or environment

such as transmission speed and processing speeds. Hendaz and Budkowski [21] provide an annotated version of ESTELLE useful for identification of potential performance concerns. Heck and others showed the feasibility of enhancing a formal definition model by including implementation-dependent information thereby rendering it suitable for use with a performance evaluation tool [22]. In the same vein as Heck, a more recent effort [23] applies a hierarchical method of simulation to evaluate performance of protocol architectures directly from the formal specifications of protocols at each layer. Note that conformance tests should be derived at this stage for execution when a prototype becomes available. Deriving and strategically executing these tests in Phase 3 require designing for testability (grey-box analysis) [5] in Phase 2. Validation testing in Phase 3 will make use of ‘black-box’ (or functionality-based) methods to demonstrate that the external behaviour of a protocol implementation corresponds to the behaviour required by its specification. 3.3. Protocol re-engineering Recently, some work has been initiated on the very practical issue of protocol re-engineering. This work is intended to provide automated methods to detect places in an existing protocol where changes must be made. Initial work considered only two kinds of changes to a transition in a finite state machine model, namely a change in output and a change in target state [24]. Recently, this has been extended by Xie and the authors to handle more substantial changes, namely the addition of transitions, removal of transitions, and to handle non-determinism. However, these algorithms are still at the experimental stage of development and assessment, and are not yet ready for adoption in industry.

4. Phase 3: Component prototyping At the next stage on the PQE ‘ferris wheel’, we are interested in practical strategies and techniques to aid in (fast) prototyping of components. Potential problems in performance and interfacing should be identified here before actual integration occurs. Primary quality engineering concerns include component validation, conformance testing, and performance estimation. At this stage, since code is available, code-directed (white-box) test design techniques are applicable. White box testing in the vendor’s or developer’s lab has also been denoted diagnostic testing [37]. This type of testing is primarily concerned with debugging and exercising all logical control flow branches in the code, as well as key (high risk) data flow paths from client-set requestor response parameters (import definitions) to observable indications (export uses). The degree to which all such structural constructs have been exercised is denoted the level of code coverage.

R.L. Probert, N. LewlComputer

Communications

4.1. Techniques and tools for component prototyping For several years, FDT compilers have been developed, distributed and refined for SDL, ESTELLE, and Lotos. At present, only two commercial offerings claim that the generated code is reasonably efficient (SDWSDT, GEODE). These compilers and translators require substantial resources to maintain and support. Object-oriented methods are natural for component prototyping. The grouping of objects into class libraries facilitates re-use, which supports the basic engineering concept of constructing solutions from known components. Rumbaugh and others describe the Object Modelling Technique (OMT), an object-oriented software development methodology [25]. At least one corporation has developed a pragmatic means of translating from the use of OMT in the analysis (requirements capture) phase to the use of SDL-92 in the design phase with the intention of applying commercial code generation tools such as TeleLOGIC SDT [26,27]. Verilog’s ObjectGEODE also ‘supports the coherent integration of complementary object-oriented approaches.. . such as OMT, SD1 and MSCs. It can also generate C or C++ code. Verilog has also developed LOV/OMT, a suite of tools that includes LOV/Reverse C++, a tool that allows reverse engineering of C++ source code into OMT design models. Such tools are just starting to penetrate into the industrial software development process. 4.2. Protocol conformance testing Protocol conformance testing is a Q&R:3 activity and area of research dedicated to demonstrating that a protocol implementation conforms to its specification. The International Organisation for Standardization (ISO) has produced a seven-part standard for conformance testing methodology and framework (CTMF) [28], Knightson [29] gives an introduction to the standard by describing test architectures (remote, distributed and co-ordinated), testing terminology, certification and some real-world organizations involved. Probert and Monkewich [30] present a tutorial on the International Standard Test Language TTCN (Tree and Tabular Combined Notation) corresponding to part 3 of the CTMF standard. Baumgarten and Giessler [31] also review the standard, present detailed information on TI’CN, and list some commercially available test tools related to conformance testing and ‘ITCN. For example, Verilog’s GEODE tool package contains a program called TTCgeN that ‘automatically derives [TTCN] test cases from an SDL description and test objectives,’ discussed in Section 7. A related thrust is towards a common semantic model to bridge the testability-related differences between the FDTs (SDL, ESTELLE, Lotos) and TTCN [32]. An initial effort involves a scenario-based formal semantics for testing labelled transition systems [33]. Such work has great potential as a basis for use-case driven PQE methodologies. Protocol behaviour is often modelled by finite state

19 (1996) 1258-1267

1263

machines. There exist numerous FSM-based methods for generating appropriate test sequences, including Transition Tour, the W-method, Wp-method, unique input output (UIO) method, UIOv-method, all of which rely on using some kind of distinguishing sequence, characterizing sequences, or UIO sequences. The overall problem is basically that of verifying each of the transitions (edges) and the states (nodes). The methods are all very similar in that they all apply stimuli to the implementation under test (and observe the outputs produced) in a predictable, systematic fashion so that any unexpected output will provide conclusive evidence of non-conformance, i.e. the existence of a fault in the implementation. These techniques only handle the FSM semantics, and require careful consideration of data and parameter test requirements. Only one or two commercially adopted data-flow based protocol test generation tools have been successful as far as the authors are aware. An example is the work by Ural and others on generating TTCN test requirements to satisfy both control-flow and data-flow coverage requirements, which has been incorporated into an SDL-based development process [34]. There are many good survey papers available such as that of Sidhu [35], a comprehensive review of formal methods in protocol conformance testing. Ramalingam and others [36] extend this by comparing the fault detection and diagnosis capabilities of the various methods. Vuong and others [37] provide a detailed list of specific implementation and testing issues that can influence the testability of protocols.

5. Phase 4: System integration As we start assembling a service provider from components (protocol stacks, channels), certain checks and tests must be carried out to ensure that the system is integrated properly. We are concerned that (end-to-end) services advertised truly exist. We also need to show that the protocol components run on the designated platforms. Many interoperability problems can also be detected here. In addressing these issues, we effectively improve robustness; fault detection and recovery; and maintainability. Service testing is an underemphasized part of the integration phase. By ‘service’ we mean the abstraction of the functions of a layer 12, 381. The intent of service testing is to confirm the existence of any services claimed to be implemented and to verify that their behaviour is one of the acceptable options according to the standard(s). The National Physical Laboratory in the UK is one institution that conducts OS1 service testing. Nevertheless, service testing remains an area in considerable need of future development. Of particular practical importance are cost effective techniques for incremental integration. When components are integrated, sanity testing needs to be carried out to ensure that the components are mutually compatible. Grey-box analysis techniques [5] consider

1264

R.L. Probert, N. Lao/Computer Communications 19 (1996) 1258-1267

internally visible interfaces and the selection of tests to exercise (cover) these interaction points. Related objectoriented testing techniques are still under development. It is during this phase that the significance of hardware and operating system interfaces becomes greater. However, a complete discussion of the software/system interface is beyond the scope of this paper.

6. Phase 5: Product Release Engineering Aproduct is a package of components and systems which are perceived to provide services and capabilities to a target customer base in a self-contained manner. Thus, for example, products may come with graphical user interfaces (GUIs), installer routines, registration procedures, warranties and field support infrastructure, update services, performance benchmarks, and tuning guidelines, etc. Capabilities and services of a product are provided by embedded protocols and real-time software. This phase is part of the PQE process because it assures that the embedded protocol will inter-workand interoperate with other components in the product and, more typically, other products and platforms. The four main constituents of Q&R:5 are robustness testing, reliability testing, regression testing, and interoperability testing. 6.1. Robustness, reliability, and regression testing As discussed in the introduction, robustness testing involves ensuring that the product will detect and recover appropriately from inappropriate or unforeseen patterns of use. Appropriate black-box (customer-directed) test-design techniques include boundary-interior analysis, errorguessing and cause-effect graphing. As well, these release engineering activities include load and stress tests, performance checks and software reliability engineering (SRE). In SRE, reliability-directed measures (e.g. rate of occurrence of failures during product execution) are collected and used to assist in the release decision. For this approach to work, it is necessary to ‘soak’ the product in test traffic which faithfully emulates at least the most common customer scenarios involving the most popular features and services, augmented by known high-risk use-cases. In this way, problems which would have been encountered by the customer in the field are discovered first in the lab, leading to higher perceived reliability in the customer base. If the protocols provide new services, we must confirm that previously existing services were not affected adversely (that the product has not ‘regressed’). Specific customer operational profiles [39] are used here, corresponding to particular markets. Finally, product documentation must be verified against actual operational behaviour. 6.2. Interoperability testing This is primarily a protocol and services verification and

validation activity of key interest to commercial vendors of protocol-based products. Obviously, a high degree of interoperability is desired across environments and implementations. Although it is theoretically possible to guarantee basic interworking by extensive conformance testing, there will always be a pragmatic requirement for interoperability testing. The practical limitation of testing an exponential number of combinations for interoperability is being addressed by industrial-strength methods such as orthogonal testing [40] based on orthogonal tests of compilers [41]. This method is based on limiting testing to all pairwise combinations of protocols and services, rather than all combinations possible. For example, suppose that for an end-to-end network test, there were 16 different service providers to interconnect, and for each there were three possible kinds of network element. The potential number of network test configurations would be approximately 43 million - an impossibility to test. By considering only pairwise combinations, we would need only 18 tests. A related issue is the resolution of interoperability problems by various means, for example, protocol conversion [42]. In most cases, protocol conversion is done on an ad hoc basis; no commercial tools are available to provide lowcost specialized protocol conversion with no appreciable loss of performance. A discussion of these issues is beyond the scope of this paper. Finally, acceptance testing is part of the product release phase. Customer acceptance procedures are dependent on many factors including customer preference, local standards and certification requirements, and availability of customer equipment in the development lab. Some support for this phase and for SDL-based system (product) evolution is provided by ESPRIT’s PROTEUS project’s PCL configuration language and methodology. An industrial pilot involving SDG88 experienced significant up-front costs, but expects compensating savings to accrue from resulting automation of the process [43]. Product release is thus a key phase in the PQE process; considerable additional development of commercial tool support is still needed for this phase.

7. Discussion of related practical concerns The global PQE process is an integration of stepwise PQE activities. Construction or synthesis of some of the PQE artifacts at early stages cannot currently be produced employing compatible languages with compatible tools. The engineering challenge is to select the best languages, models and tools at each phase, so as to optimize the overall cost-effectiveness of the PQE cycle (highest quality product at lowest production cost). This task appears quite difficult at present, largely because of the relative independence of formalisms and associated tool infrastructures. Some work has attempted to compare different formal specification

R.L. Probert, N. LewlComputer Communications 19 (1996) 1258-1267

notions [44]. Those authors found that maintainability is not a strength of any of the formal languages studied, namely, Lotos, Z and SDL. No single FDT is appropriate for all phases and purposes. As well, tool manufacturers have a particular interest in providing crossover tools to interoperate to products in their preferred development environments. As an example, Verilog has produced TTCgeN, a test generation tool to work from its main GEODE family of tools. This is part of a forward engineering paradigm. However, many organizations will have extensive experience with other TTCN tools for editing and revising test suites. Because TICN has a standard machine-processable format, ‘ITCN.MP, in theory, a ‘ITCN editor tool can use the output from TTCgeN. However, practical issues still exist, for example backward traceability to designs and the provisioning of a variety of intermediate platforms to support inter-tool compatibility. Finally, the tests generated by TTCgeN are based on formulating test objectives in a third language, GOAL (GEODE Observation Automata Language). It may be difficult to attain levels of test coverage efficiently using this approach unless the original acceptable SDL design included testability enhancements. Therefore, a competent PQ engineer has to plan a practical migration strategy to implement PQE taking such issues into account. It should be noted that some activities, such as automated synthesis, combine both types of activities in one (a construction component and a verification component). Moreover, the global PQE process has to take into account a variety of starting points, which for simplicity are not indicated in Fig. 1. Obvious examples are maintenance, evolutionary development and re-engineering. In each case, requirements for modification must be captured and current service requirements must be revised where necessary. A re-engineering paradigm must then be applied in each local process up to and including the integration phase. For example, suppose that a new type of network indication message must be introduced into a component to support interoperability. Re-engineering in the design phase is required, but first the point(s) of modification of the existing protocol must be identified. These points are used to map back to the service requirements specification to guide revision of requirements, followed by validation by regression analysis to show that the modification does not compromise or contradict other user requirements. Appropriate revised service requirements and use-cases are developed and verified. The protocol is re-engineered appropriately and verified, and then the iterated prototype for the affected component is generated, enhanced and re-tested for conformance. As Miller points out, the continuity in our PQE cycle is broken up by gaps: the gaps between formal protocol specification techniques, formal protocol verification models and practical testing techniques. We look for a

1265

way ‘to develop a unified approach for formal specification, verification, and testing that meets the goals of richness of representational power for specification purposes, mathematical tractability for verification purposes, and simplicity for testing purposes’ [45]. Some progress has been made in this area (for example, with Misra and Chandy’s unity language) [46]. Unified models are critical for improving traceability throughout the PQE cycle. There also remains a gap between the methods studied by academia and those employed by industry. Vissers and others [47] indicate that the formal methods community should follow through with ‘real industrial tools’ to support their methods. Lai and Leung consider both points of view and suggest ways in which the two worlds might converge. ‘Formal methods of testing need to be greatly simplified.. . They need to be able to pinpoint the critical paths, give a good assessment of test coverage, identify the point of diminishing returns, provide an optimal test suite design and demonstrate theories using real-life non-trivial protocols as examples’ [3]. Also advocated are gradual migration strategies, user-friendly support tools, and simply improved dialogue through collaborative workshops.

8. Concluding remarks We have described a unified framework for the development and maintenance of communications protocol software. This five-phased cycle can be used to form the basis of a PQE plan for formal construction, verification and validation activities. We advocate that an emphasis be placed on integrating quality and risk assessment activities into the protocol engineering lifecycle, and that this integration be properly planned, executed, monitored and optimized as are typical successful engineering processes. Finally, we hope this recognition, definition and explication of the Protocol Quality Engineering framework will aid in the mutually beneficial convergence of both academic formal methods and cost-effective industrial practices. Certainly, there is widespread awareness of the need for a new collaborative approach to the cost-effective development of complex, ubiquitous communications software and systems.

Acknowledgements References to trademarked names are for information only, and should not be construed as endorsement by the authors or the University of Ottawa. The authors gratefully acknowledge support for this work from the Telecommunications Research Institute of Ontario (TRIO) and the Natural Sciences and Engineering Research Council of Canada (NSERC).

1266

R.L. Probert, N. LewlComputer

Appendix: Example of quality & risk assessment activities (cf. Fig. 1). In Phase 1 (Service Requirements Capture/Revision) a. verification of service definition with respect to safety and consistency; b. validation of functional capabilities against use-cases; c. validation of operational model (profile) and assessment of associated risks; d. customer certification of use-cases and related service tests. In phase 2 (Design (Synthesis, Re-engineering)) a. verification that order relationships (service constraints for Phase 1) are satisfied (unless guaranteed by synthetic algorithm); b. incorporation and verification of fault-tolerance and recovery (if added to synthesized design); c. verification of consistency of augmented design (e.g. performance modifications); d. verification of design completeness with respect to robustness; e. validation of design against use-cases (partial projection); f. identification of potential performance concerns; g. testability analysis (grey-box test points). In Phase 3 (Component Prototyping) a. testing conformance against protocol specification (design); b. identifying performance parameters (execution profiling); c. identifying interface issues and risks; d. validation of component against global use-cases (partial projection). In Phase 4 (System Integration) a. platform testing; b. collaboration testing for robustness, fault detection, reporting and recovery; c. performance measurement and risk assessment (including feature interactions). In Phase 5 (Product Release Engineering) a. robustness assurance (black-box methods, particularly collision analysis, boundary analysis, error guessing); b. reliability assessment and stress testing against operational profile; c. interoperability testing; d. acceptance testing.

References [l] M.T. Liu, Introduction to special issue on protocol engineering, IEEE Trans. Computers, 40(4) (April 1991) 373-375. [2] K.J. Turner, An engineering approach to formal methods in A.

Communications 19 (19%) 1258-1267

Danthine, G. Ledue and P. Wolper (eds.), Protocol Specification, Testing and Verification XIII, Elsevier, 1993, pp. 357-380. [3] R. Lai and W. Leung, Industrial and academic protocol testing: the gap and the means of convergence, Computer Networks & ISDN Systems, 27 (1995) 537-547. [4] S.H. Kan, Metrics and Models in Software Quality Engineering,

Addison-Wesley, Reading, 1995. [.5] R.L. Probert, Software Quality Engineering, course notes. University of Ottawa, 1994. [6] G.J. Holxmann, Design and Validation of Computer Protocols, Prentice-Hall, New Jersey, 1991. [7] R.L. Probert and K. Saleh, Synthesis of communication protocols: survey and assessment, IEEE Trans. Computers, 40(4) (April 1991) 468-476. [8] H.D. Mills, M. Dyer and R.C. Linger, Cleanroom software engineering, IEEE Sofrware (September 1987) 19-25. [9] D.W. Karolak, Software Engineering Risk Management, IEEE Press, 1996. [lo] C. Chittister and Y.Y. Haimes, Risk associated with software development: a holistic framework for assessment and management, IEEE Trans. Systems, Man, and Cybernetics, 23(3) (May/June 1993) 710-723. [ll] R. Fairley, Risk management for software projects, IEEE Software (May 1994) 57-61. [12] K. Saleh and R.L. Probert, An extended service-oriented method for the synthesis of protocols, Proc. Int. Symposium on Computer and Information Science, North-Holland, 1991, pp. 547-557. [ 131 D. Hogrefe, OS1 formal specification study: The INRES protocol and service, revised. Technical Report, Universitaet Bern, Institut fuer Informatik, 1992. [14] B. Sarikaya, Principles of Protocol Engineering and Conformance Testing. Ellis Horwood, Chichester, 1993. [15] IS0 IS8807, LOTOS - A Formal Description for the Temporal Ordering of Observational Behaviour, 1989. [16] CCI’IT, Functional Specification and Description Language (SDL), Recommendation 2.100, CCI’IT Blue Book, 1988. [17] I. Jacobson, M. Christerson, P. Jonnson and G. Overgaard, Objectoriented Software Engineering, Addison-Wesley, Reading, MA, 1992. [18] K. Koskimies and E. Makinen, Automatic synthesis of state machines from state diagrams, Software - Practice and Experience, 24(7) (July 1994) 643-658. [19] K. Saleh, I. Ahmad and K. Al-Saqabi, An efficient recovery procedure for fault tolerance in distributed systems, J. Sysrems Software, 25 (1994) 39-50. [20] P. Godefroid, D. Peled and M. Staskauskas, Using partial-order methods in the formal validation of industrial concurrent programs, Proc. Int. Symposium on Software Testing and Analysis (ISSTA), ACM Press, 1996, pp. 261-269. [21] M. Hendaz and S. Budkowski, A new approach for protocols performance evaluation using annotated Estelle specifications, Proc. 8th Int. Conf. Formal Description Techniques for Distributed Systems and Communications Protocols (FORTE), 1995, pp. 338-345. [22] E. Heck, D. Hogrefe and B. Miiller-Closterman, Hierarchical performance evaluation based on formally specified protocols, IEEE Trans. Computers, 40(4) (April 1991) 500-513. [23] A.E. Conway, D. Dimitrijevic and R. Hwang, Hierarchical simulation of layered protocol architectures using formal specifications, in H.G. Perros and Y. Viniotis (eds.), High Speed Networks and their Performance, Elsevier, Amsterdam, 1994,309-325. [24] D. Lee and K. Sabnani, Reverse-engineering of communication protocols, Proc. Int. Conf. Networks and Protocols, IEEE Press, 1993, pp. 208-216. [25] J. Rumba@, M. Blaha, W. Premerlani, F. Eddy and W. Lorensen, Object-Oriented Modeling and Design, Prentice-Hall, Englewood Cliffs, NJ, 1991. [26] F. Guo and T. MacKenzie, Translation of OMT to SDL92, Proc. SDL’95 Forum, 1995.

R.L. Probert, N. LewlComputer [27] TeleLOGIC

SDT 3.0, Malmo,

MALMO, Sweden. [28] IS0 IEC9646, OS1 Conformance work, 1994.

AB, P.O.

Box

4128,

Testing Methodology

S-203

12

and Frame-

[29] K.G. Knightson, OS1 Protocol Conformance Testing: IS 9646 Explained, McGraw-Hill, Toronto, 1993. [30] R.L. Probert and 0. Monkewich, ‘ITCN: the international notation for specifying tests of communication systems, Computer Networks, 25(5) (1992) 417-438. [31] B. Baumgarten and A. Giessler, OS1 Conformance Testing Methodology and TTCN, Elsevier, Amsterdam, 1994. [32] ISOiITU-T, Formal Methods in Conformance Testing, ITU-T TS SGlO Q8 and IS0 SC21 WGl P54, Southampton output, 1994. [33] R.L. Probert and L. Wei, Towards a ‘practical formal method’ for test derivation, in A. Cavalli and S. Budkowski (eds.), Protocol Test Systems VIII, Chapman & Hall, London, 1995. [34] H. Ural and A. Williams, Test generation by exposing control and data dependencies within system specifications in SDL, Proc. 6th Int. Conf. Formal Description Techniques (FORTE), 1993. [35] D.P. Sidhu, Protocol testing: the first ten years. The next ten years, in L. Logrippo et al. (eds.), Protocol Specification, Testing and Verification, X, Elsevier, Amsterdam, 1990, pp. 47-68. [36] T. Ramalingam, A. Das and K. Thulasiraman, Fault detection and diagnosis capabilities of test sequence selection methods based on the finite state machine model, Computer Comm., 18(2) (February 1995) 113-122. [37] S.T. Vuong, A.A.F. Loureiro and ST. Chanson, A framework for the design for testability of communication protocols, in 0. Rafiq (ed.), Protocol Test Systems, VI, Elsevier, Amsterdam, 1994, 89-108.

Communications 19 (1996) 1258-1267

1267

[38] IS0 7498/CCITT X.200, Basic Reference Model for Open Systems Interconnection. [39] J.D. Musa, Operational profiles in software-reliability engineering, IEEE Software (March 1993) 14-32. [40] A. Williams and R.L. Probert, A practical strategy for testing coverage of network interfaces, Proc. 7th Int. Symp. on Software Reliability Engineering (ISSRE), IEEE, White Plains, NY, 1996. [41] R. Mandl, Orthogonal Latin Squares: an application of experimental design to compiler testing, Comm. ACM, 28(10) (October 1985) 41-47. [42] P.E. Green, Protocol conversion, IEEE Trans. Commun., COM-34(3) (March 1986) 257-268. [43] B. Gulla and J. Gorman, Supporting evolution of SDL-based systems: industrial experience, Proc. 8th Int. Conf. Formal Description Techniques for Distributed Systems and Communications Protocols (FORTE), 1995, pp. 253-268. [44] M.A. Ardis, J.A. Chaves, L.J. Jagadeesan, P. Mataga, C. Puchol, M.G. Staskaukas and J. Von Olnhausen, A framework for evaluating specification methods for reactive systems, Proc. 17 Int. Conf. Software Engineering, ACM Press, 1996, pp. 159-168. [45] R.E. Miller, Protocol verification: the first ten years. The next ten years; some personal observations, in L. Logrippo et al. (eds.), Protocol Specification, Testing and Verification, X , Elsevier, Amsterdam, 1990,199-225. (461 M.K. Chandy and J. Misra, A Foundation of Parallel Program Design, Addison-Wesley, Reading, MA, 1988. [47] CA. Vissers, M. van Sinderen and L.F. Pires, What makes industries believe in formal methods, in A. Danthine et al. (eds.), Protocol Specification, Testing and Verification, XIII, Elsevier, Amsterdam, 1990, pp. 3-26.