- Email: [email protected]
Software Validation
297
Conference Reports
Software Validation A Symposium on Software Validation was held in Darmstadt, F.R.G. from September 25-30, 1983. It aimed at a comprehensive review of current methods and techniques of cost-effective software analysis. Four themes were addressed at the symposium (1) testing, (2) inspection, (3) verification and (4) alternative methods. For each of these problem areas, a number of lectures was delivered, discussing the advantages and shortcomings of modern techniques currently applied in that area, as well as the general problem of whether and how to integrate these different approaches to software validation into a uniform validation technique. Also discussed were the questions of whether and how final or intermediate results of one validation procedure influence effective use of complimentary techniques. In addition to these technical aspects, management issues were discussed with respect to their contribution to cost-effective software quality assurance. According to H.L. Hausen, editor of the published conference proceedings, this symposium was the first successful attempt to discuss how methods from the areas of inspection, testing and verification can be applied along or in combina-
North-Holland Performance Evaluation 4 (1984) 297-307
tion in order to assure software quality during the entire life cycle. Hausen submitted the following report on the conference:
1. Introduction
The control and assurance of software quality is an essential and critical component of the software production process. There are two reasons for the demands for improved software quality: First, the considerable cost of software usage and maintenance is due primarily to errors introduced in earlier stages and to difficulties in making changes. Second, there are an increasing number of application fields for computers where unreliability can have an impact on human life (e.g. air traffic control, flight control systems on commercial aircraft, and control systems for nuclear reactors). In other fields, unreliability can result in loss of important data or excessive unavailability of key resources. Furthermore, software quality is no longer determined by the 'Black and white' issues of correctness and efficiency. The more difficult properties such as user friendliness, modifiability, and robustness must also be considered. The weighting of the properties, which may be in conflict with each other, depends on the requirements of the particular application field. 'Constructive' methods, such as those that suggest standard viewpoints to modular decomposition and to the overall process of proceeding in a systematic way from requirements to a fully implemented system, have improved the overall quality of software. Unfortunately, these methods in the past have not been sufficiently supported by tools, nor do they adequately address validation. Software designers are dissatisfied with the pre-
0166-5316/84/$3.00 © 1984, Elsevier Science Publishers B.V. (North-Holland)
298
Conference Reports
sent status of quality assurance and control. Methods and tools are being developed that attempt to locate errors in systems or to demonstrate the absence of such errors. Although many of these tools are still experimental and difficult to use, they have been used successfully in a number of applications. Although additional research on validation is necessary, it is likely that developers of systems make good use of some of these tools provided they are robust and stable. The currently available methods and tools for software validation range - from informal procedures, such as inspection, - t h r o u g h semi-formal procedures, such as pro-
gram testing, - to formal procedures, such as verification, - and to alternative methods in which validation is carried out at a each stage in the life cycle. From the point of assuring quality throughout the entire life cycle, most of the existing methods and tools are only suitable for specific phases and error classes. By suitable combinations of methods and tools, quality assurance and control should become more effective. The required combination depends on the specific requirements, on the current situation of a project, and last but not least, on the available resources. The goals of this symposium have been to review the current status of software validation technology, to provide an in-depth look at the issues, and to project future developments, all in the light of the overall aim of achieving an integrated framework for software validation. It was the aim to bring together researchers and practitioners from the principal areas of software validation, namely inspection, testing, verification and new alternative methods. The general theme of the symposium was how these techniques may be combined into an overall and integrated software validation procedure. To make the symposium productive, it was organized as a workshop with limited attendance; so that all participants--tool developers and tool users--could present their ideas and perspectives.
Topics The symposium was broken into the following sessions: inspection methods, testing methods, verification methods, and alternative development methods for validation purposes.
Each of these sessions included presentations on available techniques and tools, successful applications, and outstanding problems. The prospects for integration of tools and techniques using each of the above mentioned methods as the focal point was also discussed. Within the session on inspection methods two central topics have been discussed. Creating one uniform method of analyzing software deisng, software specifications and programs by humans and providing appropriate tools was one of the two. The other topics was concerned with the problems of obtaining appropriate concepts for the organization of inspections, the problems of setting a precise definition of the relationship between the software development process and the inspection process, and the problems of introducing inspections. At the conclusion to the inspection session, the exercises of inspections in industrial environments have been discussed by a panel. The session on testing methods addressed static and dynamic analysis. Among others the following issues have been discussed: support from static analysis to dynamic analysis, definition of test cases with good coverage, achievement of a test, and use of assertions for test case definition. In conclusion of the testing session a panel discussed problems of integration from the viewpoint of testing. A second point of the panel was the experience of practitioners and the theoretical power of testing. The session on verification methods has been oriented to issues such as: showing the conformity of a program to a specification, extensions needed to handle the needs of real systems, verification of designs or specifications or program code of large systems, use of verification methods in transformation systems to create correct programs. As a conclusion to the verification session, a panel discussed the prospects for practical verification and transformation systems. A central point in this discussion was the question of whether and how formal verification can be made more practical. The session on alternative development methods for validation purposes was oriented toward nonclassical software engineering disciplines, such as knowledge-based software development, and software construction and analysis exploiting techniques of Computer Aided Design/Computer Aided Manufacturing ( C A D / C A M ) . Knowledge-
Conference Reports based programming was presented as an approach to automating the process of proceeding systematically from a requirements statement to implementation. This transformation is driven by rules (the knowledge base), some of which are general to the programming process and some of which are application specific. The discussion in this part of the session was focussed on the role of knowledgebased systems in the precise definition of high-level descriptions and in their incremental and correctness-preserving refinement in,to executable code. In traditional C A D / C A M systems the 'divide and conquer' principle is widely used to automate labor-intensive work. It is hoped that similar techniques will apply to software engineering. The session on C A D / C A M - b a s e d software development discussed therefore techniques of stepwise piecemeal software development, analysis, and integration, with the emphasis on those aspects that can be automated. The symposium closed with a panel on cross-benefits of validation techniques. This panel addressed the questions of how some of the shortcomings or limitations of one technique might be overcome by using available results of complementary validation techniques. Central topic of this panel were the impacts of both management techniques and procedures of construction and validation on cost-effective software quality assurance. The prime purpose of the symposium was to discuss the state of the art and current practice in industry within a group of attendees from both industry and university. Collecting and discussing position statements from all the different viewpoints was seen as a requirement for further improvement in research as well as in practice. Conference or even seminars in the past have been organised in order to discuss one of the main issues of software validation, such as testing or formal verification, in detail. In contrast to this approach, it was the aim of this symposium to bring together nearly all the different camps in the area of software validation. The experts in theory and in practice of each area were asked to discuss the good and bad points of each technique in comparison to the shortcomings or even the advantages of the other areas.
299
LECTURES PRESENTED
Software quality assurance An Introduction of Quality Assurance and Control of Software Hans-Ludwig Hausen, Monika Mfdlerburg
Validation by inspection Software Inspections and the Industrial Production of Software A Frank Ackerman, Priscilla J. Fowler, Robert G. Ebenau Application of Software Inspection Methodology in Design and Code Robert D. Buck, James A. Dobbins Integrated Software Validation in the View of Inspections/Reviews Horst E. Remus
Validation by testing Introduction to the Formal Treatment of Testing John S. Gourlay Integrating the Testing, Analysis and Debugging of Programs Leon Osterweil RXVP - Today and Tomorrow Sabina H. Saib Analysis of Concurrent Software by Cooperative Application of Static and Dynamic Techniques Richard N. Taylor
Validation by symbolic evaluation or symbolic execution Symbolic E v a l u a t i o n - An Aid to Testing and Verification Lori A. Clarke, Debra J. Richardson Symbolic Evaluation as a Basis for Integrated Validation Erhard Ploedereder
Validation by formal verification Integrated Program Development and Verification Ch. Beierle, M. Gerlach, R. GObel, W. Olthoff R. Raulefs, A. Voss
300
Conference Repor~
On the Design of Anna, A Specification Language for Ada David C. Luckharn Representation and Refinement of Visual Specifications Mark Moriconi, Amy L. Lansky
What about C A D / C A M ARGUS Concept Leon G. Stucki
Software validation management and certification
Comments on Practical Constraints of Software Validation Techniques Hans-Ludwig Hausen A List of Selected Literature Monika Mi~llerburg
Quality Management Technology: Practical Applications Edward F. Miller Conformance Testing of Graphics Software Using a Configurable Reference System GUnther E. Pfaff Validation by non-classical methods
Knowledge-Based Programming: An Overview of Data and Control Structure Refinement Allen Goldberg, Gordon Kotik
for Software? The
General topics
The Proceedings of this conference have been edited by H.L. Hausen and published by North-Holland under the title SOFTWARE VALIDATION. !984. XII + 376 pages. ISBN 0-444-87593. Price: Dfl. 150,--.
Mathematical Computer Performance & Reliability An International Workshop on "Applied Mathematics and Performance/Reliability Models of Computer/Communication Systems" was held in Pisa, Italy from September 26-30, 1983. This workshop was organized by the Computer Science Department of the University of Pisa. The purpose of the event was to stimulate an exchange of ideas and experience between scientists who, with various areas of expertise, are active in applied mathematical research of interest to the analysis of computer performance and reliability. According to G. lazeolla (University of Pisa, Italy) and P.J. Courtois (Philips Research, Brussels, Belgium), the formulation and the study of models of computer system behavior rely on distinct branches
of Applied Mathematics. Queueing Theory, Markov Chain Theory, and Numerical Analysis are only a few of the most significant ones. These disciplines complement each other in performance and reliability analysis. A multidisciplinary approach appears to be even essential to the solution of some of the more interesting open problems in this field. The emphasis of the workshop was on such interdisciplinary aspects. 27 lectures were presented and were divided into four thematic parts: (1) Queueing System Models, (2) Approximation Techniques, (3) Performance and Reliability Models, and (4) State of the Art and Future Directions. The first part dealt with queueing network models; fundamental mathematical
Conference Reports
Software Validation A Symposium on Software Validation was held in Darmstadt, F.R.G. from September 25-30, 1983. It aimed at a comprehensive review of current methods and techniques of cost-effective software analysis. Four themes were addressed at the symposium (1) testing, (2) inspection, (3) verification and (4) alternative methods. For each of these problem areas, a number of lectures was delivered, discussing the advantages and shortcomings of modern techniques currently applied in that area, as well as the general problem of whether and how to integrate these different approaches to software validation into a uniform validation technique. Also discussed were the questions of whether and how final or intermediate results of one validation procedure influence effective use of complimentary techniques. In addition to these technical aspects, management issues were discussed with respect to their contribution to cost-effective software quality assurance. According to H.L. Hausen, editor of the published conference proceedings, this symposium was the first successful attempt to discuss how methods from the areas of inspection, testing and verification can be applied along or in combina-
North-Holland Performance Evaluation 4 (1984) 297-307
tion in order to assure software quality during the entire life cycle. Hausen submitted the following report on the conference:
1. Introduction
The control and assurance of software quality is an essential and critical component of the software production process. There are two reasons for the demands for improved software quality: First, the considerable cost of software usage and maintenance is due primarily to errors introduced in earlier stages and to difficulties in making changes. Second, there are an increasing number of application fields for computers where unreliability can have an impact on human life (e.g. air traffic control, flight control systems on commercial aircraft, and control systems for nuclear reactors). In other fields, unreliability can result in loss of important data or excessive unavailability of key resources. Furthermore, software quality is no longer determined by the 'Black and white' issues of correctness and efficiency. The more difficult properties such as user friendliness, modifiability, and robustness must also be considered. The weighting of the properties, which may be in conflict with each other, depends on the requirements of the particular application field. 'Constructive' methods, such as those that suggest standard viewpoints to modular decomposition and to the overall process of proceeding in a systematic way from requirements to a fully implemented system, have improved the overall quality of software. Unfortunately, these methods in the past have not been sufficiently supported by tools, nor do they adequately address validation. Software designers are dissatisfied with the pre-
0166-5316/84/$3.00 © 1984, Elsevier Science Publishers B.V. (North-Holland)
298
Conference Reports
sent status of quality assurance and control. Methods and tools are being developed that attempt to locate errors in systems or to demonstrate the absence of such errors. Although many of these tools are still experimental and difficult to use, they have been used successfully in a number of applications. Although additional research on validation is necessary, it is likely that developers of systems make good use of some of these tools provided they are robust and stable. The currently available methods and tools for software validation range - from informal procedures, such as inspection, - t h r o u g h semi-formal procedures, such as pro-
gram testing, - to formal procedures, such as verification, - and to alternative methods in which validation is carried out at a each stage in the life cycle. From the point of assuring quality throughout the entire life cycle, most of the existing methods and tools are only suitable for specific phases and error classes. By suitable combinations of methods and tools, quality assurance and control should become more effective. The required combination depends on the specific requirements, on the current situation of a project, and last but not least, on the available resources. The goals of this symposium have been to review the current status of software validation technology, to provide an in-depth look at the issues, and to project future developments, all in the light of the overall aim of achieving an integrated framework for software validation. It was the aim to bring together researchers and practitioners from the principal areas of software validation, namely inspection, testing, verification and new alternative methods. The general theme of the symposium was how these techniques may be combined into an overall and integrated software validation procedure. To make the symposium productive, it was organized as a workshop with limited attendance; so that all participants--tool developers and tool users--could present their ideas and perspectives.
Topics The symposium was broken into the following sessions: inspection methods, testing methods, verification methods, and alternative development methods for validation purposes.
Each of these sessions included presentations on available techniques and tools, successful applications, and outstanding problems. The prospects for integration of tools and techniques using each of the above mentioned methods as the focal point was also discussed. Within the session on inspection methods two central topics have been discussed. Creating one uniform method of analyzing software deisng, software specifications and programs by humans and providing appropriate tools was one of the two. The other topics was concerned with the problems of obtaining appropriate concepts for the organization of inspections, the problems of setting a precise definition of the relationship between the software development process and the inspection process, and the problems of introducing inspections. At the conclusion to the inspection session, the exercises of inspections in industrial environments have been discussed by a panel. The session on testing methods addressed static and dynamic analysis. Among others the following issues have been discussed: support from static analysis to dynamic analysis, definition of test cases with good coverage, achievement of a test, and use of assertions for test case definition. In conclusion of the testing session a panel discussed problems of integration from the viewpoint of testing. A second point of the panel was the experience of practitioners and the theoretical power of testing. The session on verification methods has been oriented to issues such as: showing the conformity of a program to a specification, extensions needed to handle the needs of real systems, verification of designs or specifications or program code of large systems, use of verification methods in transformation systems to create correct programs. As a conclusion to the verification session, a panel discussed the prospects for practical verification and transformation systems. A central point in this discussion was the question of whether and how formal verification can be made more practical. The session on alternative development methods for validation purposes was oriented toward nonclassical software engineering disciplines, such as knowledge-based software development, and software construction and analysis exploiting techniques of Computer Aided Design/Computer Aided Manufacturing ( C A D / C A M ) . Knowledge-
Conference Reports based programming was presented as an approach to automating the process of proceeding systematically from a requirements statement to implementation. This transformation is driven by rules (the knowledge base), some of which are general to the programming process and some of which are application specific. The discussion in this part of the session was focussed on the role of knowledgebased systems in the precise definition of high-level descriptions and in their incremental and correctness-preserving refinement in,to executable code. In traditional C A D / C A M systems the 'divide and conquer' principle is widely used to automate labor-intensive work. It is hoped that similar techniques will apply to software engineering. The session on C A D / C A M - b a s e d software development discussed therefore techniques of stepwise piecemeal software development, analysis, and integration, with the emphasis on those aspects that can be automated. The symposium closed with a panel on cross-benefits of validation techniques. This panel addressed the questions of how some of the shortcomings or limitations of one technique might be overcome by using available results of complementary validation techniques. Central topic of this panel were the impacts of both management techniques and procedures of construction and validation on cost-effective software quality assurance. The prime purpose of the symposium was to discuss the state of the art and current practice in industry within a group of attendees from both industry and university. Collecting and discussing position statements from all the different viewpoints was seen as a requirement for further improvement in research as well as in practice. Conference or even seminars in the past have been organised in order to discuss one of the main issues of software validation, such as testing or formal verification, in detail. In contrast to this approach, it was the aim of this symposium to bring together nearly all the different camps in the area of software validation. The experts in theory and in practice of each area were asked to discuss the good and bad points of each technique in comparison to the shortcomings or even the advantages of the other areas.
299
LECTURES PRESENTED
Software quality assurance An Introduction of Quality Assurance and Control of Software Hans-Ludwig Hausen, Monika Mfdlerburg
Validation by inspection Software Inspections and the Industrial Production of Software A Frank Ackerman, Priscilla J. Fowler, Robert G. Ebenau Application of Software Inspection Methodology in Design and Code Robert D. Buck, James A. Dobbins Integrated Software Validation in the View of Inspections/Reviews Horst E. Remus
Validation by testing Introduction to the Formal Treatment of Testing John S. Gourlay Integrating the Testing, Analysis and Debugging of Programs Leon Osterweil RXVP - Today and Tomorrow Sabina H. Saib Analysis of Concurrent Software by Cooperative Application of Static and Dynamic Techniques Richard N. Taylor
Validation by symbolic evaluation or symbolic execution Symbolic E v a l u a t i o n - An Aid to Testing and Verification Lori A. Clarke, Debra J. Richardson Symbolic Evaluation as a Basis for Integrated Validation Erhard Ploedereder
Validation by formal verification Integrated Program Development and Verification Ch. Beierle, M. Gerlach, R. GObel, W. Olthoff R. Raulefs, A. Voss
300
Conference Repor~
On the Design of Anna, A Specification Language for Ada David C. Luckharn Representation and Refinement of Visual Specifications Mark Moriconi, Amy L. Lansky
What about C A D / C A M ARGUS Concept Leon G. Stucki
Software validation management and certification
Comments on Practical Constraints of Software Validation Techniques Hans-Ludwig Hausen A List of Selected Literature Monika Mi~llerburg
Quality Management Technology: Practical Applications Edward F. Miller Conformance Testing of Graphics Software Using a Configurable Reference System GUnther E. Pfaff Validation by non-classical methods
Knowledge-Based Programming: An Overview of Data and Control Structure Refinement Allen Goldberg, Gordon Kotik
for Software? The
General topics
The Proceedings of this conference have been edited by H.L. Hausen and published by North-Holland under the title SOFTWARE VALIDATION. !984. XII + 376 pages. ISBN 0-444-87593. Price: Dfl. 150,--.
Mathematical Computer Performance & Reliability An International Workshop on "Applied Mathematics and Performance/Reliability Models of Computer/Communication Systems" was held in Pisa, Italy from September 26-30, 1983. This workshop was organized by the Computer Science Department of the University of Pisa. The purpose of the event was to stimulate an exchange of ideas and experience between scientists who, with various areas of expertise, are active in applied mathematical research of interest to the analysis of computer performance and reliability. According to G. lazeolla (University of Pisa, Italy) and P.J. Courtois (Philips Research, Brussels, Belgium), the formulation and the study of models of computer system behavior rely on distinct branches
of Applied Mathematics. Queueing Theory, Markov Chain Theory, and Numerical Analysis are only a few of the most significant ones. These disciplines complement each other in performance and reliability analysis. A multidisciplinary approach appears to be even essential to the solution of some of the more interesting open problems in this field. The emphasis of the workshop was on such interdisciplinary aspects. 27 lectures were presented and were divided into four thematic parts: (1) Queueing System Models, (2) Approximation Techniques, (3) Performance and Reliability Models, and (4) State of the Art and Future Directions. The first part dealt with queueing network models; fundamental mathematical