- Email: [email protected]
Software quality assurance in metrology
ISA TRANSACTIONS” ELSEVIER
ISA Transactions 33 (1994)421-428
Software quality assurance in metrology Dieter Richter Physikalisch-Technische Bundesanstalt,Abbe&. 2-10, 10587 Berlin, Germany
Abstract This paper discusses the specific problem of software quality assurance in metrology. There are different levels of quality requirements of metrology and of software engineering methods. In the scope of software engineering there
is a great variety of quality assurance techniques available. On one hand they reach from non-computer aided methods over the application of standards to the use of software tools. On the other hand the quality assurance ranges over the whole software life cycle with the focuses on constructive methods and on analytic methods. Additionally, management activities are becoming a more important role in software quality assurance. Because software is not included in the physically determined traceability chain other approaches seem as necessary. Possible ways are (1) special concepts for particular problems, (2) a taxonomy of methods supported by statistics and (3) metrology software libraries. Keywords:
Software quality assurance; Metrology software
1.Introduction Computer integrated techniques have become a key role in many areas. This concerns manufacturing techniques, medicine techniques as well as measurement techniques. Whereas hardware dominated in the early years of computer technique, now the software has become the bottle-neck with respect to the costs as well as to the reliability. Having in mind possible consequences of erroneous software like - wrong data in a intensive care station of a hospital, - missiles missing the target, - measuring instruments providing unreliable values, the importance of high quality software becomes visible.
As in other areas in metrology the software is widely used. Software is integrated in a large number of instruments in order to support the data acquisition, to pre-process the measured raw data, to calculate the target information, or to control the whole process. So it has an essential effect on the accuracy and on the reliability of the final values as well as on their traceability to basic units. In the classical metrology the aspects of information processing did not have to be considered, predominantly. The accuracy and traceability of the measured values have been proved mainly by physical laws. But more and more there are demands to extend the classical measurement process by combining the physical measurement with the subsequent information processing. The reason is a growing interest to get an application
0019-0578/94/$07.00 Q 1994 Elsevier Science B.V. All rights reserved SSDZ 0019-0578(94)00044-l
D. Richter/ISA Transactions 33 (1994) 421-428
422
oriented information rather than a physically measurable value, in the case that there is a difference. ,We call this slightly changed understanding of metrology information metrology. Each area of metrology seems to be concerned more or less with aspects of the information metrology. However, the information metrology is for the coordinate measurement metrology and for medical measurements of special interest. In the computer science the software engineering has been established as a special discipline dealing with how to develop or more generally how to handle software. The quality assurance is a research area of priority. However, software quality assurance methods do not answer directly quality demands of the metrology. Generally, there is a gap between aims of metrology and software engineering methods. This gap has become clear only very recently and needs to be bridged by further research. Nevertheless, knowing the software engineering methods is a prerequisite to discuss the problems. We overview the methods of software quality assurance in Section 3 from the point of view of software engineering. Before, in Section 2 we present the different levels of quality criteria of metrology and quality assurance methods. Finally, in Section 4, we outline possible approaches of metrology to bridge the mentioned gap.
2. Quality criteria of metrology When using software compliance with certain
-1
in the metrology the quality criteria is de-
manded. The quality criteria related to functional aspects are basically retraceable to quality demands of the metrology. Other criteria are related directly to software specific aspects. As functional aspects of quality we distinguish between the following classes: - direct characteristics of physics and metrology (non-interactivity of components subject to mandatory verification, threshold values of measuring instruments, access to primarily measured values, form tolerances in coordinate metrology); - mathematical and algorithmic features (numerical stability, complexity of an algorithm, calculation of residuals); - standards of metrology, guidelines, recommendations (international standards, recommendations of national metrology institutes, guidelines of test and verification bodies, comparison to existing reference software); - task specific characteristics (authenticity of a software module, absence of failures following special “nightmare” lists). Software specific aspects which are of general importance, i.e. for metrology software too, are general characteristics of software quality (reli. ability, usability, portability, etc.); prevention of bugs (coding bugs, data bugs, etc.); software standards, guidelines, recommendations (software design guidelines, standardized programming languages, documentation rules, etc.). In software engineering a great variety of quality assurance methods is available. However, these methods are oriented to support and to prove,
Evaluation from the point of view of Metrology
I
Software Quality Measure
Fig. 1. Relation between aims of metrology and software engineering methods.
I
D. Richter/ISA Transactions 33 (1994) 421-428
423
respectively, quality features ranking among the level of software engineering and being expressible in terms of those techniques. Therefore, the use of software assurance methods prerequisites the establishment of quality demands on the software which are already expressed as a software engineering problem. Even the process of figuring out these problems is simultaneously the problem and a chance for the specific software quality assurance in metrology as it is pointed out in Section 4. Vice versa, after performing a software quality measure an evaluation of its result is only acceptable if the measure has been determined on the basis of the known relation between the quality aim of metrology and the derived software quality aim. Fig. 1 demonstrates this relation. A direct way from the quality aim of metrology to software assurance measures leads to an unsure subsequent evaluation of the results because of a not systematic determination of the measure.
Table 1 Class of analytical methods
3. Software quality assurance methods
analytical methods bases on the application of certain constructive methods or on the availability of certain documents. So far there is an interrelation between constructive and analytical methods. Table 1 gives a survey on analytical methods. The table contains a classification of methods. The manual inspections are methods relying on analyzing capabilities of humans. They have been used for a long time and can be successful if proceeding systematically. Especially the inspection by checklist is an appropriate means to discuss the documentation of software and detect shortcomings this way. Milestone checks are rather applicable to enhance the transparency of the development process from the point of view of management. The walk through is used by developers to check their programs. The program is executed by hand with simple test cases. The static analysers search for static and structural aspects of programs without executing them. The tools of this class contain at least components of compilers. Or, this methods or’ parts of them are performed during the compilation process. The application of static analysers is possi-
Meanwhile there exists a multitude of methods and tools to support the development and to prove high quality software. Three categories of methods are to distinguish: - analytical methods, - constructive methods, - management methods. The first group contains methods to test and to verify features of the software, whereas the methods of the second group support the development of software. There are interrelations between the methods of these both groups. The third category of methods is the newest one but of growing importance. Recent activities of standardisation consider already these methods [l]. 3.1. Analytical methods Analytical methods are test methods. They determine the state of a piece of software with respect to given aims. Analytical methods do not change the software but confirm a desired quality criterion or reject it, respectively. The use of
Classes of methods
Test methods belonging to
Manual inspections
Inspection by check lists Milestone check Walk through Requirement analysis Code analysis Data flow analysis Control flow analysis Interface analysis Function test (black-box-test) Performance test Statistical test Structural test (white-box-test) Equivalence class analysis Mutation test Test case coverage analysis Causal connection analysis Formal verification Symbolic execution
Static analyses
Dynamic tests
Test data selections
Formal proofs
424
D. Richter/ISA
Transactions 33 (1994) 421-428
ble in the early stages of programming as well as on ready programs. The source code of the programs is necessary in each case. The requirement analysis proves the consistency, the uniqueness, the completeness, etc. of software requirements. A prerequisite of a tool supported requirement analysis is the availability of requirements in formalized form. The code analysis searches for errors in the code and violations of programming guides. These methods are supported by tools very well. Again, the application of tools for testing programming guides supposes their availability in formalized form. The methods of data flow analysis are used to make the interrelations and dependencies of data or variables, respectively, visible. Meanwhile, they are supported by tools sufficiently. The data flow analysis is an excellent means to prove the correctness of data relations. The control flow analysis determines the sequence of execution steps and the structure of program instructions or submodules, respectively. The result is usually represented graphically in order to be able to catch the information. The interface analysis proves the consistency of interfaces between interworking modules of a system. The dynamic tests comprize methods which execute the software with certain test data. Of course, these methods can only be used if executable modules are available. This is usually not the case during the development of software. Therefore dynamic tests are of special interest for users but they are used by programmers, too. They are as an analyzing tool widespread and valuable. The function test proves the correctness of the realized function. This type of test is most commonly known as black-box test. The tester chooses an appropriate set of test data and runs the software with them. By comparing the computed result against the expected result a correctness statement can be derived. Functional tests do not need any inside into programming details. The availability of the source code is not necessary. So, not finally for that reason, functional tests are very popular. But, the test data selection is of great importance. Often it can be observed that not enough attention is paid to that point.
An unsystematic selection of test data leads to an unsure evaluation of test results. Often the relevance of a few test runs is overestimated. The performance test determines the resources bound by the software like running time, virtual memory, hard disk, peripheral devices, etc. With the help of statistical tests statements on interested features are empirically determined. Such features are for instance the average response time, the average time between two failure states, the number of pages of output, etc. Test data are generated randomly in the range of valid input data and the software is executed with them to measure the desired values. Statistical tests are often demanded for final inspections or quality checks. If test data are generated by exploiting the inner structure of the software, then the tests are called structural test. This type of test are better known as white-box-test. When knowing the structure it is possible to establish tests running through all the instructions, all the paths, all the branches or through other desired parts of the software, respectively. Otherwise, having performed tests the percentage of coverage of certain features of the software by the test data can be determined. The advantage of this type of tests is a quality statement according to the features of the software development methods and tools. Functional aspect can be included but are not of first priority. Obviously, performing structural tests the source code of the software must be available. Moreover, structural test should be prepared by static analyses in order to identify the interested features. The fest data selection for dynamic tests plays a fundamental role for evaluation of test results. The aim of the equivalence class analysis is to divide the range of test data into such classes that each member of the class causes the same test result. Then the number of test can be restricted to one member of each class. The mutation test is a method to prove the sensitivity of the used test data. The test coverage analysis determines the degree of completeness according to given features like branch coverage, path coverage, etc. Because 100% are practically not achievable the really achieved degree is an important information about the quality of the test. The causal
D. Richter/ISA Transactions 33 (1994) 421-428 Table 2 Constructive methods software life cycle
with relation
to the phases of the
Software life cycle phase
Constructive software quality method belonging to
Overlapping methods
Documentation standards Data dictionary Structural analysis Formal specification Simulation Rapid prototyping Modular design Program design languages Petri-nets Object oriented design Programming standards Defensive programming Structural programming High level languages Program libraries Program generators Version and configuration management Automatic system generation
Requirement phase Commitment phase Design phase
Implementation
phase
Integration phase
connection analysis is a method the test cases and the results.
to systematize
3.2. Constructive methods Constructive methods of software quality assurance are those methods of software development that influence the quality of creating software. All the phases of the software life cycle are included. The overview in Table 2 relates the methods to the life cycle phases. Not in all cases is an unambiguous relation possible. In these cases a main phase has been chosen. To support the co-operative software development process, documentation standards are necessary and data dictionaries are very helpful. The documentation of requirements on the software is an important component of quality assurance. The structural analysis supports the structural understanding of the task performed by the software. There are function oriented analyses and data oriented ones. The aim of both
425
classes consists in getting a rough plan about the software which shall be developed. The formal specification supports the documentation of software requirements and allows a formal proof of requirement features. A suitable language is necessary. Even this fact appears as the bottle-neck of using formal specifications. The methods placed in the commitment phase fulfill the purpose of getting early answers about using features of the software system that has to be developed. With the help of existing modules and appropriate tools a software system similar to the target system is composed rapidly in order to study several aspects. The simulation concentrates on functional aspects and the rapid prototyping shifts the focus to the user interface. In the design phase the logical arrangement of the program is established. The modular design arranges the software as a set of functionally transparent and closed submodules. Design languages support this aim. By Petri nets software systems with parallel actions can be designed. A far-reaching and very flexible design method is the object oriented design. By principles like data encapsulation, inheritance, integration of methods and rules into data definition the maintaining and reusing of software is supported very well. In the implementation phase there are a lot of techniques contributing to high quality software. Programming standards are a set of rules especially to be followed by a group of implementers who work on the same system. As defensive programming techniques are called which utilize each possibility of proving the validity of data. By using these methods a price for the high reliability has to be paid in terms of reduced efficiency. By structural programming the programs become more transparent, maintainable and changeable. High level programming languages offer more and more quality concepts directly to the programmer. Program libraries are an approved mean to avoid unnecessary risks of new software. The already implemented know-how is reused in new systems. Program generators appear as another way to minimize human mistakes. The integration phase describes the composition of modules to an executable software fulfilling the corresponding requirements. Due to the
426
D. Richter /ISA Transactions 33 (1994) 421-428
frequent changes and updates a continuously actual knowledge on the used versions of the integrated modules is an important aspect. This control process is called version and configuration control. Again, by automatic system generators the risk of human mistakes can be decreased.
3.3. Management methocis Long ago the software quality has been not only a problem of programmers and testers. The aim of software quality concerns all people and especially the management being in relation to software. Several activities have to be coordinated. Recent research activities in the software engineering emphasize even this aspect. Typical projects are - the international standardisation of quality management guidelines including the software quality [ 11; - the formalization of actions of planning and realizing software projects in administrative authorities in Germany 121. In Table 3 an overview on management methods that contribute to software quality assurance is given. In this paper we cannot go into further details of management methods but we refer to [3-51.
Table 3 Management methods of software quality assurance Activities of management
Methods contributing to software quality assurance
Planning
Determination of quality aims Determination of quality assurance systems Software quality education Quality planning Software risk analysis Software test planning Configuration mangement Trend analysis Audits and reviews Progress reports Probbm analyses Source of error analyses Correction measures
Supervising Controlling
4. Special approaches of software quality assurance in metrology As already pointed out in Section 2 we have to consider the circumstance that a quality aim of metrology cannot be satisfied directly by using quality methods of software engineering. At first a mapping process from quality criteria of metrology to software quality criteria has to be performed. After carrying out the software quality measure an evaluation must determine how much the original quality aim of metrology has been met. We suggest three different approaches which might help to get higher quality software in metrology. Each of them bases on completely different concepts. However, even together they do not cover all the existing problems, so far. 4.1. Special mappings Sometimes it is possible to map the aim of metrology rather directly to a feature of software which is achievable by an appropriate software quality measure. This approach cannot be generalized from special cases to a reusable methodology. It is to work out each time from new. We illustrate the principle with an example. In measuring instruments equipped with software like automatic weighing machines there are parts underlying mandatory verification and parts underlying no rules. An aim of metrology consists in ensuring that data controlled by the software of the part underlying mandatory verification cannot be manipulated by software belonging not to this part. If this condition is fulfilled, then the producer of the instrument is free to design the software in parts of the instrument not underlying verification. These criteria can be translated into a software engineering form. Then it says that all the data (variables) of the software belonging to the part to be verified must not be influenced by software and data from outside unless the influence is regular with respect to the measuring process. If the software has been written in a high level language and the source code is available, then with the help of the data flow analysis as a
D. Richter/ISA Transactions 33 (1994)
static analysis method all the data changing relations are perceptible. They can be made transparent so that a measurement specialist is able to decide which of them are necessarily allowed and which not. Altogether we have in this concrete case a clear software feature based on the aim of metrology which enables to decide about corresponding quality measures and leads to an evaluation of the result of the measure from the point of view of metrology.
4.2. Statbtics on software quality measures Within the problematic field of mapping quality criteria of metrology on software quality features in order to decide about the appropriate software quality assurance techniques each time two fundamental questions are to answer: 1. Which of the techniques are to apply in order to ensure a certain quality criterion? 2. Which value has a result of an applied assurance technique? Often there is not an answer uniquely derivable. Then statistics on the occurrence of failures (bugs, errors, other features to avoid) may help to judge the usefulness of a method. There is no universally best way to categorize software quality measures. A bug can be detected sometimes better on one way and sometimes on another way. This depends on a lot of facts (chosen test cases, coverage rate of test criteria, kind of requirement specification, used programming language and environment, etc.). A taxonomy of methods is difficult to establish in general. It has to be done tailored to the specific application area. But, whatever sophisticated taxonomy may be worked out it can be only an initial one which serves as a framework for the statistic. So far a well developed initial taxonomy supports enormously the importance of the statistic. A statistic is useful if carried out in a long-term period. Then it can also help to recognize changes of the frequency of bugs. Such changes really happen because of changes of development and test technologies.
421-428
427
4.3. Metrology software libraries Another approach to get higher quality software in metrology establishes as the set up of metrology software libraries. Because software is not included in the physically determined traceability chain, an other approach seems as appropriate. Actually, the idea is not new but has been applied up to now only on general purpose software. The idea is very simple. Software which has been used successfully in metrology practice is offered to be reused in similar application cases. In this way potential users is given an access to good software and, more important, to software with practical experience. The best practice can be extended. For metrology software a lot of commonalties exist so that a reuse of software is practically relevant. The difference to conventional software libraries consists in the combined availability of software and its application field including the already existing experience. Classical software libraries offer only the software modules. The user has to decide by himself which of the modules he uses for his problem. In a BCR project the questions related to the set up of metrology software repository has been investigated [6]. The points of investigation were - what are the problematic areas of metrology where a software library is of interest; - which content has to be included in such a library; - what acceptance procedures for the inclusion of software are to be applied; - what practical aspects are to be regarded when setting up this library; - how to technically realize such a library. As a result of the study the establishment of a metrology software library appears feasible and is recommended.
References [l] International Standard IS0 9000-3; Quality Management and Quality Assurance Standard; Part 3: Guidelines for the Application of IS0 9001 to the Development, Supply and Maintenance of Software, 1992. [2] Vorgehenmodell: Planung und DurchjYihnmg von IT-
428
D. Richter/ISA Transactions 33 (1994) 421-428
Vorhaben, Koordinierungs- und Beratungsstelle der Bun&sregienuzg fiir Informationstechnik (Bonn, 1992). [3] J.H. Dobbius, Software Quality Assurance and Evaluation (Quality Press, Milwaukee, 1990). [4] E. Miller, The 5th International Sojbvare Quality Week Proceedings) (San Francisco, 1992). [5] Methoden und Verfahren der Softwarequalitiilssichenmg (Beuth-Verlag, Berlin, 1992).
[6] Feasibility study for a metrology software repository, BCR project 3~9/1/0/~/~/6-~-~~30), 1993. [7] B. Beizer, Software Testing Techniques (Van Nostrand Reinhold, New York, 1990). [8] Softwarezuverliissigkeit: Grundkzgen, konstruktive MaJ3nahmen, Nachweisverfahren (WI-Verlag, Diisseldorf, 1993).
ISA Transactions 33 (1994)421-428
Software quality assurance in metrology Dieter Richter Physikalisch-Technische Bundesanstalt,Abbe&. 2-10, 10587 Berlin, Germany
Abstract This paper discusses the specific problem of software quality assurance in metrology. There are different levels of quality requirements of metrology and of software engineering methods. In the scope of software engineering there
is a great variety of quality assurance techniques available. On one hand they reach from non-computer aided methods over the application of standards to the use of software tools. On the other hand the quality assurance ranges over the whole software life cycle with the focuses on constructive methods and on analytic methods. Additionally, management activities are becoming a more important role in software quality assurance. Because software is not included in the physically determined traceability chain other approaches seem as necessary. Possible ways are (1) special concepts for particular problems, (2) a taxonomy of methods supported by statistics and (3) metrology software libraries. Keywords:
Software quality assurance; Metrology software
1.Introduction Computer integrated techniques have become a key role in many areas. This concerns manufacturing techniques, medicine techniques as well as measurement techniques. Whereas hardware dominated in the early years of computer technique, now the software has become the bottle-neck with respect to the costs as well as to the reliability. Having in mind possible consequences of erroneous software like - wrong data in a intensive care station of a hospital, - missiles missing the target, - measuring instruments providing unreliable values, the importance of high quality software becomes visible.
As in other areas in metrology the software is widely used. Software is integrated in a large number of instruments in order to support the data acquisition, to pre-process the measured raw data, to calculate the target information, or to control the whole process. So it has an essential effect on the accuracy and on the reliability of the final values as well as on their traceability to basic units. In the classical metrology the aspects of information processing did not have to be considered, predominantly. The accuracy and traceability of the measured values have been proved mainly by physical laws. But more and more there are demands to extend the classical measurement process by combining the physical measurement with the subsequent information processing. The reason is a growing interest to get an application
0019-0578/94/$07.00 Q 1994 Elsevier Science B.V. All rights reserved SSDZ 0019-0578(94)00044-l
D. Richter/ISA Transactions 33 (1994) 421-428
422
oriented information rather than a physically measurable value, in the case that there is a difference. ,We call this slightly changed understanding of metrology information metrology. Each area of metrology seems to be concerned more or less with aspects of the information metrology. However, the information metrology is for the coordinate measurement metrology and for medical measurements of special interest. In the computer science the software engineering has been established as a special discipline dealing with how to develop or more generally how to handle software. The quality assurance is a research area of priority. However, software quality assurance methods do not answer directly quality demands of the metrology. Generally, there is a gap between aims of metrology and software engineering methods. This gap has become clear only very recently and needs to be bridged by further research. Nevertheless, knowing the software engineering methods is a prerequisite to discuss the problems. We overview the methods of software quality assurance in Section 3 from the point of view of software engineering. Before, in Section 2 we present the different levels of quality criteria of metrology and quality assurance methods. Finally, in Section 4, we outline possible approaches of metrology to bridge the mentioned gap.
2. Quality criteria of metrology When using software compliance with certain
-1
in the metrology the quality criteria is de-
manded. The quality criteria related to functional aspects are basically retraceable to quality demands of the metrology. Other criteria are related directly to software specific aspects. As functional aspects of quality we distinguish between the following classes: - direct characteristics of physics and metrology (non-interactivity of components subject to mandatory verification, threshold values of measuring instruments, access to primarily measured values, form tolerances in coordinate metrology); - mathematical and algorithmic features (numerical stability, complexity of an algorithm, calculation of residuals); - standards of metrology, guidelines, recommendations (international standards, recommendations of national metrology institutes, guidelines of test and verification bodies, comparison to existing reference software); - task specific characteristics (authenticity of a software module, absence of failures following special “nightmare” lists). Software specific aspects which are of general importance, i.e. for metrology software too, are general characteristics of software quality (reli. ability, usability, portability, etc.); prevention of bugs (coding bugs, data bugs, etc.); software standards, guidelines, recommendations (software design guidelines, standardized programming languages, documentation rules, etc.). In software engineering a great variety of quality assurance methods is available. However, these methods are oriented to support and to prove,
Evaluation from the point of view of Metrology
I
Software Quality Measure
Fig. 1. Relation between aims of metrology and software engineering methods.
I
D. Richter/ISA Transactions 33 (1994) 421-428
423
respectively, quality features ranking among the level of software engineering and being expressible in terms of those techniques. Therefore, the use of software assurance methods prerequisites the establishment of quality demands on the software which are already expressed as a software engineering problem. Even the process of figuring out these problems is simultaneously the problem and a chance for the specific software quality assurance in metrology as it is pointed out in Section 4. Vice versa, after performing a software quality measure an evaluation of its result is only acceptable if the measure has been determined on the basis of the known relation between the quality aim of metrology and the derived software quality aim. Fig. 1 demonstrates this relation. A direct way from the quality aim of metrology to software assurance measures leads to an unsure subsequent evaluation of the results because of a not systematic determination of the measure.
Table 1 Class of analytical methods
3. Software quality assurance methods
analytical methods bases on the application of certain constructive methods or on the availability of certain documents. So far there is an interrelation between constructive and analytical methods. Table 1 gives a survey on analytical methods. The table contains a classification of methods. The manual inspections are methods relying on analyzing capabilities of humans. They have been used for a long time and can be successful if proceeding systematically. Especially the inspection by checklist is an appropriate means to discuss the documentation of software and detect shortcomings this way. Milestone checks are rather applicable to enhance the transparency of the development process from the point of view of management. The walk through is used by developers to check their programs. The program is executed by hand with simple test cases. The static analysers search for static and structural aspects of programs without executing them. The tools of this class contain at least components of compilers. Or, this methods or’ parts of them are performed during the compilation process. The application of static analysers is possi-
Meanwhile there exists a multitude of methods and tools to support the development and to prove high quality software. Three categories of methods are to distinguish: - analytical methods, - constructive methods, - management methods. The first group contains methods to test and to verify features of the software, whereas the methods of the second group support the development of software. There are interrelations between the methods of these both groups. The third category of methods is the newest one but of growing importance. Recent activities of standardisation consider already these methods [l]. 3.1. Analytical methods Analytical methods are test methods. They determine the state of a piece of software with respect to given aims. Analytical methods do not change the software but confirm a desired quality criterion or reject it, respectively. The use of
Classes of methods
Test methods belonging to
Manual inspections
Inspection by check lists Milestone check Walk through Requirement analysis Code analysis Data flow analysis Control flow analysis Interface analysis Function test (black-box-test) Performance test Statistical test Structural test (white-box-test) Equivalence class analysis Mutation test Test case coverage analysis Causal connection analysis Formal verification Symbolic execution
Static analyses
Dynamic tests
Test data selections
Formal proofs
424
D. Richter/ISA
Transactions 33 (1994) 421-428
ble in the early stages of programming as well as on ready programs. The source code of the programs is necessary in each case. The requirement analysis proves the consistency, the uniqueness, the completeness, etc. of software requirements. A prerequisite of a tool supported requirement analysis is the availability of requirements in formalized form. The code analysis searches for errors in the code and violations of programming guides. These methods are supported by tools very well. Again, the application of tools for testing programming guides supposes their availability in formalized form. The methods of data flow analysis are used to make the interrelations and dependencies of data or variables, respectively, visible. Meanwhile, they are supported by tools sufficiently. The data flow analysis is an excellent means to prove the correctness of data relations. The control flow analysis determines the sequence of execution steps and the structure of program instructions or submodules, respectively. The result is usually represented graphically in order to be able to catch the information. The interface analysis proves the consistency of interfaces between interworking modules of a system. The dynamic tests comprize methods which execute the software with certain test data. Of course, these methods can only be used if executable modules are available. This is usually not the case during the development of software. Therefore dynamic tests are of special interest for users but they are used by programmers, too. They are as an analyzing tool widespread and valuable. The function test proves the correctness of the realized function. This type of test is most commonly known as black-box test. The tester chooses an appropriate set of test data and runs the software with them. By comparing the computed result against the expected result a correctness statement can be derived. Functional tests do not need any inside into programming details. The availability of the source code is not necessary. So, not finally for that reason, functional tests are very popular. But, the test data selection is of great importance. Often it can be observed that not enough attention is paid to that point.
An unsystematic selection of test data leads to an unsure evaluation of test results. Often the relevance of a few test runs is overestimated. The performance test determines the resources bound by the software like running time, virtual memory, hard disk, peripheral devices, etc. With the help of statistical tests statements on interested features are empirically determined. Such features are for instance the average response time, the average time between two failure states, the number of pages of output, etc. Test data are generated randomly in the range of valid input data and the software is executed with them to measure the desired values. Statistical tests are often demanded for final inspections or quality checks. If test data are generated by exploiting the inner structure of the software, then the tests are called structural test. This type of test are better known as white-box-test. When knowing the structure it is possible to establish tests running through all the instructions, all the paths, all the branches or through other desired parts of the software, respectively. Otherwise, having performed tests the percentage of coverage of certain features of the software by the test data can be determined. The advantage of this type of tests is a quality statement according to the features of the software development methods and tools. Functional aspect can be included but are not of first priority. Obviously, performing structural tests the source code of the software must be available. Moreover, structural test should be prepared by static analyses in order to identify the interested features. The fest data selection for dynamic tests plays a fundamental role for evaluation of test results. The aim of the equivalence class analysis is to divide the range of test data into such classes that each member of the class causes the same test result. Then the number of test can be restricted to one member of each class. The mutation test is a method to prove the sensitivity of the used test data. The test coverage analysis determines the degree of completeness according to given features like branch coverage, path coverage, etc. Because 100% are practically not achievable the really achieved degree is an important information about the quality of the test. The causal
D. Richter/ISA Transactions 33 (1994) 421-428 Table 2 Constructive methods software life cycle
with relation
to the phases of the
Software life cycle phase
Constructive software quality method belonging to
Overlapping methods
Documentation standards Data dictionary Structural analysis Formal specification Simulation Rapid prototyping Modular design Program design languages Petri-nets Object oriented design Programming standards Defensive programming Structural programming High level languages Program libraries Program generators Version and configuration management Automatic system generation
Requirement phase Commitment phase Design phase
Implementation
phase
Integration phase
connection analysis is a method the test cases and the results.
to systematize
3.2. Constructive methods Constructive methods of software quality assurance are those methods of software development that influence the quality of creating software. All the phases of the software life cycle are included. The overview in Table 2 relates the methods to the life cycle phases. Not in all cases is an unambiguous relation possible. In these cases a main phase has been chosen. To support the co-operative software development process, documentation standards are necessary and data dictionaries are very helpful. The documentation of requirements on the software is an important component of quality assurance. The structural analysis supports the structural understanding of the task performed by the software. There are function oriented analyses and data oriented ones. The aim of both
425
classes consists in getting a rough plan about the software which shall be developed. The formal specification supports the documentation of software requirements and allows a formal proof of requirement features. A suitable language is necessary. Even this fact appears as the bottle-neck of using formal specifications. The methods placed in the commitment phase fulfill the purpose of getting early answers about using features of the software system that has to be developed. With the help of existing modules and appropriate tools a software system similar to the target system is composed rapidly in order to study several aspects. The simulation concentrates on functional aspects and the rapid prototyping shifts the focus to the user interface. In the design phase the logical arrangement of the program is established. The modular design arranges the software as a set of functionally transparent and closed submodules. Design languages support this aim. By Petri nets software systems with parallel actions can be designed. A far-reaching and very flexible design method is the object oriented design. By principles like data encapsulation, inheritance, integration of methods and rules into data definition the maintaining and reusing of software is supported very well. In the implementation phase there are a lot of techniques contributing to high quality software. Programming standards are a set of rules especially to be followed by a group of implementers who work on the same system. As defensive programming techniques are called which utilize each possibility of proving the validity of data. By using these methods a price for the high reliability has to be paid in terms of reduced efficiency. By structural programming the programs become more transparent, maintainable and changeable. High level programming languages offer more and more quality concepts directly to the programmer. Program libraries are an approved mean to avoid unnecessary risks of new software. The already implemented know-how is reused in new systems. Program generators appear as another way to minimize human mistakes. The integration phase describes the composition of modules to an executable software fulfilling the corresponding requirements. Due to the
426
D. Richter /ISA Transactions 33 (1994) 421-428
frequent changes and updates a continuously actual knowledge on the used versions of the integrated modules is an important aspect. This control process is called version and configuration control. Again, by automatic system generators the risk of human mistakes can be decreased.
3.3. Management methocis Long ago the software quality has been not only a problem of programmers and testers. The aim of software quality concerns all people and especially the management being in relation to software. Several activities have to be coordinated. Recent research activities in the software engineering emphasize even this aspect. Typical projects are - the international standardisation of quality management guidelines including the software quality [ 11; - the formalization of actions of planning and realizing software projects in administrative authorities in Germany 121. In Table 3 an overview on management methods that contribute to software quality assurance is given. In this paper we cannot go into further details of management methods but we refer to [3-51.
Table 3 Management methods of software quality assurance Activities of management
Methods contributing to software quality assurance
Planning
Determination of quality aims Determination of quality assurance systems Software quality education Quality planning Software risk analysis Software test planning Configuration mangement Trend analysis Audits and reviews Progress reports Probbm analyses Source of error analyses Correction measures
Supervising Controlling
4. Special approaches of software quality assurance in metrology As already pointed out in Section 2 we have to consider the circumstance that a quality aim of metrology cannot be satisfied directly by using quality methods of software engineering. At first a mapping process from quality criteria of metrology to software quality criteria has to be performed. After carrying out the software quality measure an evaluation must determine how much the original quality aim of metrology has been met. We suggest three different approaches which might help to get higher quality software in metrology. Each of them bases on completely different concepts. However, even together they do not cover all the existing problems, so far. 4.1. Special mappings Sometimes it is possible to map the aim of metrology rather directly to a feature of software which is achievable by an appropriate software quality measure. This approach cannot be generalized from special cases to a reusable methodology. It is to work out each time from new. We illustrate the principle with an example. In measuring instruments equipped with software like automatic weighing machines there are parts underlying mandatory verification and parts underlying no rules. An aim of metrology consists in ensuring that data controlled by the software of the part underlying mandatory verification cannot be manipulated by software belonging not to this part. If this condition is fulfilled, then the producer of the instrument is free to design the software in parts of the instrument not underlying verification. These criteria can be translated into a software engineering form. Then it says that all the data (variables) of the software belonging to the part to be verified must not be influenced by software and data from outside unless the influence is regular with respect to the measuring process. If the software has been written in a high level language and the source code is available, then with the help of the data flow analysis as a
D. Richter/ISA Transactions 33 (1994)
static analysis method all the data changing relations are perceptible. They can be made transparent so that a measurement specialist is able to decide which of them are necessarily allowed and which not. Altogether we have in this concrete case a clear software feature based on the aim of metrology which enables to decide about corresponding quality measures and leads to an evaluation of the result of the measure from the point of view of metrology.
4.2. Statbtics on software quality measures Within the problematic field of mapping quality criteria of metrology on software quality features in order to decide about the appropriate software quality assurance techniques each time two fundamental questions are to answer: 1. Which of the techniques are to apply in order to ensure a certain quality criterion? 2. Which value has a result of an applied assurance technique? Often there is not an answer uniquely derivable. Then statistics on the occurrence of failures (bugs, errors, other features to avoid) may help to judge the usefulness of a method. There is no universally best way to categorize software quality measures. A bug can be detected sometimes better on one way and sometimes on another way. This depends on a lot of facts (chosen test cases, coverage rate of test criteria, kind of requirement specification, used programming language and environment, etc.). A taxonomy of methods is difficult to establish in general. It has to be done tailored to the specific application area. But, whatever sophisticated taxonomy may be worked out it can be only an initial one which serves as a framework for the statistic. So far a well developed initial taxonomy supports enormously the importance of the statistic. A statistic is useful if carried out in a long-term period. Then it can also help to recognize changes of the frequency of bugs. Such changes really happen because of changes of development and test technologies.
421-428
427
4.3. Metrology software libraries Another approach to get higher quality software in metrology establishes as the set up of metrology software libraries. Because software is not included in the physically determined traceability chain, an other approach seems as appropriate. Actually, the idea is not new but has been applied up to now only on general purpose software. The idea is very simple. Software which has been used successfully in metrology practice is offered to be reused in similar application cases. In this way potential users is given an access to good software and, more important, to software with practical experience. The best practice can be extended. For metrology software a lot of commonalties exist so that a reuse of software is practically relevant. The difference to conventional software libraries consists in the combined availability of software and its application field including the already existing experience. Classical software libraries offer only the software modules. The user has to decide by himself which of the modules he uses for his problem. In a BCR project the questions related to the set up of metrology software repository has been investigated [6]. The points of investigation were - what are the problematic areas of metrology where a software library is of interest; - which content has to be included in such a library; - what acceptance procedures for the inclusion of software are to be applied; - what practical aspects are to be regarded when setting up this library; - how to technically realize such a library. As a result of the study the establishment of a metrology software library appears feasible and is recommended.
References [l] International Standard IS0 9000-3; Quality Management and Quality Assurance Standard; Part 3: Guidelines for the Application of IS0 9001 to the Development, Supply and Maintenance of Software, 1992. [2] Vorgehenmodell: Planung und DurchjYihnmg von IT-
428
D. Richter/ISA Transactions 33 (1994) 421-428
Vorhaben, Koordinierungs- und Beratungsstelle der Bun&sregienuzg fiir Informationstechnik (Bonn, 1992). [3] J.H. Dobbius, Software Quality Assurance and Evaluation (Quality Press, Milwaukee, 1990). [4] E. Miller, The 5th International Sojbvare Quality Week Proceedings) (San Francisco, 1992). [5] Methoden und Verfahren der Softwarequalitiilssichenmg (Beuth-Verlag, Berlin, 1992).
[6] Feasibility study for a metrology software repository, BCR project 3~9/1/0/~/~/6-~-~~30), 1993. [7] B. Beizer, Software Testing Techniques (Van Nostrand Reinhold, New York, 1990). [8] Softwarezuverliissigkeit: Grundkzgen, konstruktive MaJ3nahmen, Nachweisverfahren (WI-Verlag, Diisseldorf, 1993).