- Email: [email protected]
Quality assurance in the computer age
0045.794937 $3 00 + 0.00 ,< i987 Pergamon Journals Ltd.
QUALITY
ASSURANCE
IN THE COMPUTER
AGE
LADISLAV
Civil/Structural
Department,
J. BOROS Bateman Engineering Ltd, Kiryat Bialic, Israel
(Received 21 Fehnry
1986)
Abstract--in recent years commercially avaiiabfe software has mushroomed. In fact, hardware and software development has advanced at such a pace that neither quality control, nor legal problem solving methods have been able to keep abreast with the development. Like many other products, computer software can be marketed in different qualities. A better quality product is usually more expensive, and the same can be true for computer programs. The authorities, client or user can decide which quality of software has to be used for each particular application. This article proposes three quality levels of software: regular, registered and certified, and describes the quality requirements for each category. The article also proposes an international system for registration of software, which will provide a framework and protection against copying for the program writers, and quality assurance for the users.
I~RODU~ON
After 2.5 years of extensive utilization of computer technology in engineering, it is time for us to stop for a minute and take a look backward, with pride, on our accompishments and analyse our successes and failures before we enter into the next stage. The first question we have to ask ourselves is: are our new structures safer and better designed than the old ones, which were designed by hand calculation? Unfortunately, the answer is not a clear yes. The problem is especially serious in civil and structural en~nee~ng appli~tions, because of the number of people and value of property which are involved in any disaster. Furthermore, unlike other fields, the product is not going through an actual testing in order to verify the design, except in very few cases. A standing structure does not mean that it is safe. It means only that the actual load did not exceed the critical load until now. It does not indicate that the actual safety factor fits the requirement. Of course, the actual safety factor depends also on other factors, in addition to the design, but they are not within the scope of this paper. Although the intr~uction of computers affects almost every profession, because of the abovementioned problems, civil and structural engineers have to take the lead in assuring the quality of software used in design and other applications. In recent years commercially available software has mushroomed. The process has accelerated with the introduction of more and more powerful desk-top computers which are marketed at relatively low prices, The small personal computer of arcade games improved in no time into a very serious tool. More and more people are entering into programming and many of them are developing software in fields in which their knowl~ge is very limited. The software
is
leased, without any ~s~nsibility on the side of the programmer, as a black box, which is not to be opened, corrected or improved by the user without permission from the owner, which is usually not given. For a correction to the program the user is in the hands of the vendor, who deals with it according to his priorities, if he does it at all. Because of the competitive market, software vendors have been forced to reduce prices, mostly at the expense of testing and checking. As long as these programs operate games all this sounds fine, but can we put design and public welfare related problems into the control of such a system? In fact, the hardware and software development has advanced at such a pace that neither the quality control system, nor the Iegal problem solving methods have been able to keep abreast with the development. Therefore, there is an urgent need to create a framework in which a minimum quality level of commercially available software can be secured. The aim of this paper is to propose an international system to do this task.
DE~N~ON
An acceptable quality computer program is a complex product which, with the help of proper hardware, through clear documentation will produce correct results for correctly defined problems. From the definition it is understood that it is not creating solutions for problems for which it is not capable of providing correct solutions. In other words, the result is either a correct solution or a clear error message.
Computer programs, like other products, are going 495
LADISI_AVJ BOROS
496
through many changes and improvements during their period of application. It is therefore important to identify each edition of the program in a unique manner which is readable by the user and which is difficult to manipulate by the programmer. The proposed identification number contains three parts. The first part is alphabetical, representing the name or initial of the program developing company. The second part is the quality indicator and the third part is numerical, which represents the serial number of the product. This number is the length of the program. The length of program differs for each edition, it is readable by the user and difficult to manipulate by the programmer. Therefore, it is ideal for identification purposes. The alphabetical part could have three digits, with one digit as the quality indicator which is followed by a numerical part. QUALITY LEVELS
Like any other product, computer software can be marketed in different qualities. A better quality product is usually more expensive as it is made from better materials, or a more expensive process is used, or the workmanship is more time consuming. The same is true for computer software. According to the proposed system the software in general can be marketed in three different quality levels as follows: (1) Regular-The program is made in a similar manner to most of the programs today. The user is responsible for the program’s application and its results. (2) Registered-The program has to pass a minimum quality requirement criterion. The user is responsible for the program’s application and its results. (3) Certified-The program has to pass the minimum quality requirement criterion and has to be fully tested for its application. The originating firm or vendor is responsible for its results. Government agencies, local authorities, consultants or the client can decide which quality of software has to be used for each particular application. The programmer also has the freedom to decide on which quality and price level he would like to produce his software, based on his commercial and marketing approach. The majority of the engineering software will probably be produced in the registered quality. As a result of requests for certified programs in some applications a new geld, software insurance, will develop. Insurance companies will employ specialists for computer program testing in the same way that they now employ professionals for design and value evaluations. Of course, the certified software will be the most expensive as far as quality is concerned. QUALITY REQUIREMENTS
In order to register a program it has to fuffif the
following minimum quality requirements. output (1) The program has to be able to produce printed output, (2) The output has to contain four sections: (i) Comments. This section is independent from the program. The section starts by the date of last updating, identification number and title of the related program. It includes a short description of its appIication, limits of its apphcation, list of problems, and bugs discovered but not fixed in the current edition which can produce incorrect rest&s. Warnings and other comments by the author. The comments section can be changed as many times as required and the updated version has to be stored in the register office. (ii) Front page. This is the first part of the program. It includes identification number, title, version number and release date of the program, the name of the author or vendor, and the address and phone number of the info~ation service for the user. Each of the following pages have to be headed by the problem’s title and a continuous page number. (iii) Input data. The data have to be reported in an easily readable manner. The data have to be processed by the program prior to the solution and an input summary in numerical or graphical order, or both, has to be produced. The program has to have a dry run capacity in which the program can be stopped after data check. After a dry run the user has to be able to modify, add, or remove any data of the original input without the need to input the whole data again. Repeated data without request for change has to produce a warning message. A good quahty program should also test the data for common input errors. Traps are placed against input of unlogical data and warning messages are produced to indicate possible problems. (iv) Results. The result is either a correct solution or a clear error message. The results have to be printed in an easily readable format, clearly stating to what the results are related and in which units they are given. In addition to detailed information the results shouid include a general summary section showing validity of the results, such as equilibrium check. The end of the output has to be indicated on the printout. A good quality program would also print a list of completed major steps during the execution of the problem and provide an option to store and retrieve the results for future post processing by the user’s independent software. Solution verification
The test problems have to be documented. A test disk has to be prepared by the author or checker of the program, which includes the input data for the test problems, There is a possibility that some part of the program logic can be bugged by fixing or modifying another
Quality assurance in the computer age part of the program; therefore, each edition of the program has to be tested and verified independently. for all the benchmark applications, as follows: (i) Verification of fexfbook problems. Checking against a wide range of typical applications. These are short solutions which can be verified by hand calculations. (ii) Verification of complex problems. Cheking several complex applications against the same problem solved by at least two other independent computer programs. Checking if the program is capable of solving the stated maximum size problems. (iii) Verijicafion of error checking mechanism. Checking of the error handling system and diagnostic messages against common basic and typographic input errors. The execution should stop in cases of incorrectly defined problems, or if an attempt is made to use it beyond its intended applications. (iv) Accurucy test. Checking against hand calculations, tables, and results of independent computer runs. User manual
For each program a user manual has to be provided. The manual has to be written in a simple and easily understandable language. The paragraphs have to be clearly identifiable. The user manual should include, at least, the following sections: (i) Scope of the program. Applications and limitations of the program. Hardware requirements and specifications. (ii) Theoretical background. Short description of formulae and theory which are utilized, including a list of reference material. (iii) Modeling. Interpretation of the problems into a correct computer model. (iv) Data processing. instructions for input preparation and description of various options. Listing and explanation of output options. (v) Sample problems. Explanation of several typical problems, including modeling, input and output. Data referring to computer processing time on a described hardware. (vi) Verification. Description of program verification method and its resu!ts. (vii) Error messages. List of error messages and their meaning. (viii) Responsibilities. A clear statement of liabilities, responsibilities and their limitations. (ix) User support. Name, address and phone number of at least three qualified people, who are thoroughly familiar with the programs and are able to advise users. (x) Cusromer satisfaction statement. The vendor’s promise to correct any defect in the program which prevents correct applications of the program in any of the fields in which it is intended to apply, without any additional charge during a reasonable period of time. The length of time can be decided by the vendor
491
and it can vary for each program, but it has to be stated clearly. In addition, the vendor promises to list warnings of defects on the central comment page within 48 hours after it has been discovered by himself, or he has been notified by the users. The size of this manual can vary from a couple of pages to numerous volumes, depending upon the size and application of the program. REGISTRATION PROCEDURE’
Registration of commercially available software will serve the program developer and the user’s interest as well. Probably the best fitting organization to handle the registration will be the copyright office. Already many programs are submitted to this office for copyright. With minimal extension and additional staff the copyright office can provide software registration too. The registration office would have no responsibility whatsoever for content of the software just as the copyright office is not responsible for content of books. Its only duty would be to register software for which the quality requirement form is completed and signed by its author. Based on the form, the program passes the minimum quality requirements and the user manual includes all ten chapters. The author has also to submit a disk containing the program, a ‘test disk’, and the user manual to the office. This material can be used in future arbitration if it becomes necessary. Certified software has to be submitted to the insurance agent too in the same manner. A reasonable registration fee could be changed, in order to keep the registration office self-supporting. Because there are only quality requirements for registration of a program, and not publishing requirements as in the case of copyright registration, in-house programs can be registered too. For certified software the certification can be limited for certain well-tested applications, if necessary, in the same manner that insurance companies limit their coverage in the common insurance practice. This limitation has to be clearly stated on the program’s comment page. Because of the frequent changes and additions to the comment page, the registration office will keep the comment page of each registered program on active computer file. The comment page will be updated as soon as an updated version is received by the office from the author. The comment pages will be available to the user by interface of the user’s computer through the common phone lines. ERROR REPORTING PROCEDURE
If an error in the program is detected by a user, in order to solve his problem and to serve the public welfare, he should report it to the program’s supporting personnel, If it seems that the error did not result from incorrect application of the program or
498
LADELAY
input error made by the user, the vendor has to notify, within 48 hours, the registration office about discovery of a potential error in the software. The registration office will immediately put a warning message on the comment page of the program, describing the potential error and its application, a text of which is supplied by the vendor. Within two weeks the vendor has to check the potential error. If the error is real, a program limitation message has to be sent by the vendor, and the comment page will be updated by the registration ofhce. If the vendor does not respond or he does not follow the above described procedure, the user can report the error to the registration ofhce directly and the office will automatically produce the warning message.
NEW EDITIONS
If the limitation is serious or numerous limitations accumulate during the program’s application, the vendor can decide to correct his program, remove the limitation and possibly make some other improvements. The timing for error correction has to fulfil his promises in the custumer satisfaction statement. After his work is completed and tested, he can submit it as a new edition to the registration office. If there are improvements increasing the capacity and application of the program, a new edition of the ‘test tape’ has to be prepared. The new tape will include all the test data of the old tape in addition to tests made on new applications. Also, the user manual has to be updated accordingly. Customer protection laws against purchasing damaged products in most countries will require correction or exchange of damaged products free of charge, but if other improvements are made in the program, the vendor can charge for supplying the new edition.
J. BOROS LEGAL RESPONSIBILITY AND OTHER CONSIDERATIONS
From the legal point of view the responsibility of the programmer is to provide correct information to the registration office and to fulfil the customer satisfaction statements. The registration office does not provide any tests of the submitted programs, but if it appears during arbitration that some of the statements were false (such as not ail the test problems listed on the ‘test tape’ being actually executed and evaluated), the casecan be handled as ‘provision of false statement to a government agency’ by the common law. Of course, a correct solution also depends on correct interpretation of the problem to the program’s terms, correct modeling, and input. Correct understanding and postprocessing of the output is also essential. But the most important part is the program and its documentation. Results cannot be correct if the program fails to provide the correct solution to the submitted problem. Computer software testing is a complex, time consuming and expensive procedure. In the past, it was mostly the larger and more reputable organizations which developed software for mainframe computers and they tried to keep certain quality standards. But since minicomputers have become cheaper and more powerful, more and more complex problems can be solved with them. The number of software vendors has also increased significantly. As a result software quality asssurance is growing into a very important and critical problem. The proposed software tracking method is made to provide reasonable quality software for alI the professions with minimum reorgani~tion effort, oIKce staff and expense. The system also creates a wide variety of options and a framework for programming personnel.
QUALITY
ASSURANCE
IN THE COMPUTER
AGE
LADISLAV
Civil/Structural
Department,
J. BOROS Bateman Engineering Ltd, Kiryat Bialic, Israel
(Received 21 Fehnry
1986)
Abstract--in recent years commercially avaiiabfe software has mushroomed. In fact, hardware and software development has advanced at such a pace that neither quality control, nor legal problem solving methods have been able to keep abreast with the development. Like many other products, computer software can be marketed in different qualities. A better quality product is usually more expensive, and the same can be true for computer programs. The authorities, client or user can decide which quality of software has to be used for each particular application. This article proposes three quality levels of software: regular, registered and certified, and describes the quality requirements for each category. The article also proposes an international system for registration of software, which will provide a framework and protection against copying for the program writers, and quality assurance for the users.
I~RODU~ON
After 2.5 years of extensive utilization of computer technology in engineering, it is time for us to stop for a minute and take a look backward, with pride, on our accompishments and analyse our successes and failures before we enter into the next stage. The first question we have to ask ourselves is: are our new structures safer and better designed than the old ones, which were designed by hand calculation? Unfortunately, the answer is not a clear yes. The problem is especially serious in civil and structural en~nee~ng appli~tions, because of the number of people and value of property which are involved in any disaster. Furthermore, unlike other fields, the product is not going through an actual testing in order to verify the design, except in very few cases. A standing structure does not mean that it is safe. It means only that the actual load did not exceed the critical load until now. It does not indicate that the actual safety factor fits the requirement. Of course, the actual safety factor depends also on other factors, in addition to the design, but they are not within the scope of this paper. Although the intr~uction of computers affects almost every profession, because of the abovementioned problems, civil and structural engineers have to take the lead in assuring the quality of software used in design and other applications. In recent years commercially available software has mushroomed. The process has accelerated with the introduction of more and more powerful desk-top computers which are marketed at relatively low prices, The small personal computer of arcade games improved in no time into a very serious tool. More and more people are entering into programming and many of them are developing software in fields in which their knowl~ge is very limited. The software
is
leased, without any ~s~nsibility on the side of the programmer, as a black box, which is not to be opened, corrected or improved by the user without permission from the owner, which is usually not given. For a correction to the program the user is in the hands of the vendor, who deals with it according to his priorities, if he does it at all. Because of the competitive market, software vendors have been forced to reduce prices, mostly at the expense of testing and checking. As long as these programs operate games all this sounds fine, but can we put design and public welfare related problems into the control of such a system? In fact, the hardware and software development has advanced at such a pace that neither the quality control system, nor the Iegal problem solving methods have been able to keep abreast with the development. Therefore, there is an urgent need to create a framework in which a minimum quality level of commercially available software can be secured. The aim of this paper is to propose an international system to do this task.
DE~N~ON
An acceptable quality computer program is a complex product which, with the help of proper hardware, through clear documentation will produce correct results for correctly defined problems. From the definition it is understood that it is not creating solutions for problems for which it is not capable of providing correct solutions. In other words, the result is either a correct solution or a clear error message.
Computer programs, like other products, are going 495
LADISI_AVJ BOROS
496
through many changes and improvements during their period of application. It is therefore important to identify each edition of the program in a unique manner which is readable by the user and which is difficult to manipulate by the programmer. The proposed identification number contains three parts. The first part is alphabetical, representing the name or initial of the program developing company. The second part is the quality indicator and the third part is numerical, which represents the serial number of the product. This number is the length of the program. The length of program differs for each edition, it is readable by the user and difficult to manipulate by the programmer. Therefore, it is ideal for identification purposes. The alphabetical part could have three digits, with one digit as the quality indicator which is followed by a numerical part. QUALITY LEVELS
Like any other product, computer software can be marketed in different qualities. A better quality product is usually more expensive as it is made from better materials, or a more expensive process is used, or the workmanship is more time consuming. The same is true for computer software. According to the proposed system the software in general can be marketed in three different quality levels as follows: (1) Regular-The program is made in a similar manner to most of the programs today. The user is responsible for the program’s application and its results. (2) Registered-The program has to pass a minimum quality requirement criterion. The user is responsible for the program’s application and its results. (3) Certified-The program has to pass the minimum quality requirement criterion and has to be fully tested for its application. The originating firm or vendor is responsible for its results. Government agencies, local authorities, consultants or the client can decide which quality of software has to be used for each particular application. The programmer also has the freedom to decide on which quality and price level he would like to produce his software, based on his commercial and marketing approach. The majority of the engineering software will probably be produced in the registered quality. As a result of requests for certified programs in some applications a new geld, software insurance, will develop. Insurance companies will employ specialists for computer program testing in the same way that they now employ professionals for design and value evaluations. Of course, the certified software will be the most expensive as far as quality is concerned. QUALITY REQUIREMENTS
In order to register a program it has to fuffif the
following minimum quality requirements. output (1) The program has to be able to produce printed output, (2) The output has to contain four sections: (i) Comments. This section is independent from the program. The section starts by the date of last updating, identification number and title of the related program. It includes a short description of its appIication, limits of its apphcation, list of problems, and bugs discovered but not fixed in the current edition which can produce incorrect rest&s. Warnings and other comments by the author. The comments section can be changed as many times as required and the updated version has to be stored in the register office. (ii) Front page. This is the first part of the program. It includes identification number, title, version number and release date of the program, the name of the author or vendor, and the address and phone number of the info~ation service for the user. Each of the following pages have to be headed by the problem’s title and a continuous page number. (iii) Input data. The data have to be reported in an easily readable manner. The data have to be processed by the program prior to the solution and an input summary in numerical or graphical order, or both, has to be produced. The program has to have a dry run capacity in which the program can be stopped after data check. After a dry run the user has to be able to modify, add, or remove any data of the original input without the need to input the whole data again. Repeated data without request for change has to produce a warning message. A good quahty program should also test the data for common input errors. Traps are placed against input of unlogical data and warning messages are produced to indicate possible problems. (iv) Results. The result is either a correct solution or a clear error message. The results have to be printed in an easily readable format, clearly stating to what the results are related and in which units they are given. In addition to detailed information the results shouid include a general summary section showing validity of the results, such as equilibrium check. The end of the output has to be indicated on the printout. A good quality program would also print a list of completed major steps during the execution of the problem and provide an option to store and retrieve the results for future post processing by the user’s independent software. Solution verification
The test problems have to be documented. A test disk has to be prepared by the author or checker of the program, which includes the input data for the test problems, There is a possibility that some part of the program logic can be bugged by fixing or modifying another
Quality assurance in the computer age part of the program; therefore, each edition of the program has to be tested and verified independently. for all the benchmark applications, as follows: (i) Verification of fexfbook problems. Checking against a wide range of typical applications. These are short solutions which can be verified by hand calculations. (ii) Verification of complex problems. Cheking several complex applications against the same problem solved by at least two other independent computer programs. Checking if the program is capable of solving the stated maximum size problems. (iii) Verijicafion of error checking mechanism. Checking of the error handling system and diagnostic messages against common basic and typographic input errors. The execution should stop in cases of incorrectly defined problems, or if an attempt is made to use it beyond its intended applications. (iv) Accurucy test. Checking against hand calculations, tables, and results of independent computer runs. User manual
For each program a user manual has to be provided. The manual has to be written in a simple and easily understandable language. The paragraphs have to be clearly identifiable. The user manual should include, at least, the following sections: (i) Scope of the program. Applications and limitations of the program. Hardware requirements and specifications. (ii) Theoretical background. Short description of formulae and theory which are utilized, including a list of reference material. (iii) Modeling. Interpretation of the problems into a correct computer model. (iv) Data processing. instructions for input preparation and description of various options. Listing and explanation of output options. (v) Sample problems. Explanation of several typical problems, including modeling, input and output. Data referring to computer processing time on a described hardware. (vi) Verification. Description of program verification method and its resu!ts. (vii) Error messages. List of error messages and their meaning. (viii) Responsibilities. A clear statement of liabilities, responsibilities and their limitations. (ix) User support. Name, address and phone number of at least three qualified people, who are thoroughly familiar with the programs and are able to advise users. (x) Cusromer satisfaction statement. The vendor’s promise to correct any defect in the program which prevents correct applications of the program in any of the fields in which it is intended to apply, without any additional charge during a reasonable period of time. The length of time can be decided by the vendor
491
and it can vary for each program, but it has to be stated clearly. In addition, the vendor promises to list warnings of defects on the central comment page within 48 hours after it has been discovered by himself, or he has been notified by the users. The size of this manual can vary from a couple of pages to numerous volumes, depending upon the size and application of the program. REGISTRATION PROCEDURE’
Registration of commercially available software will serve the program developer and the user’s interest as well. Probably the best fitting organization to handle the registration will be the copyright office. Already many programs are submitted to this office for copyright. With minimal extension and additional staff the copyright office can provide software registration too. The registration office would have no responsibility whatsoever for content of the software just as the copyright office is not responsible for content of books. Its only duty would be to register software for which the quality requirement form is completed and signed by its author. Based on the form, the program passes the minimum quality requirements and the user manual includes all ten chapters. The author has also to submit a disk containing the program, a ‘test disk’, and the user manual to the office. This material can be used in future arbitration if it becomes necessary. Certified software has to be submitted to the insurance agent too in the same manner. A reasonable registration fee could be changed, in order to keep the registration office self-supporting. Because there are only quality requirements for registration of a program, and not publishing requirements as in the case of copyright registration, in-house programs can be registered too. For certified software the certification can be limited for certain well-tested applications, if necessary, in the same manner that insurance companies limit their coverage in the common insurance practice. This limitation has to be clearly stated on the program’s comment page. Because of the frequent changes and additions to the comment page, the registration office will keep the comment page of each registered program on active computer file. The comment page will be updated as soon as an updated version is received by the office from the author. The comment pages will be available to the user by interface of the user’s computer through the common phone lines. ERROR REPORTING PROCEDURE
If an error in the program is detected by a user, in order to solve his problem and to serve the public welfare, he should report it to the program’s supporting personnel, If it seems that the error did not result from incorrect application of the program or
498
LADELAY
input error made by the user, the vendor has to notify, within 48 hours, the registration office about discovery of a potential error in the software. The registration office will immediately put a warning message on the comment page of the program, describing the potential error and its application, a text of which is supplied by the vendor. Within two weeks the vendor has to check the potential error. If the error is real, a program limitation message has to be sent by the vendor, and the comment page will be updated by the registration ofhce. If the vendor does not respond or he does not follow the above described procedure, the user can report the error to the registration ofhce directly and the office will automatically produce the warning message.
NEW EDITIONS
If the limitation is serious or numerous limitations accumulate during the program’s application, the vendor can decide to correct his program, remove the limitation and possibly make some other improvements. The timing for error correction has to fulfil his promises in the custumer satisfaction statement. After his work is completed and tested, he can submit it as a new edition to the registration office. If there are improvements increasing the capacity and application of the program, a new edition of the ‘test tape’ has to be prepared. The new tape will include all the test data of the old tape in addition to tests made on new applications. Also, the user manual has to be updated accordingly. Customer protection laws against purchasing damaged products in most countries will require correction or exchange of damaged products free of charge, but if other improvements are made in the program, the vendor can charge for supplying the new edition.
J. BOROS LEGAL RESPONSIBILITY AND OTHER CONSIDERATIONS
From the legal point of view the responsibility of the programmer is to provide correct information to the registration office and to fulfil the customer satisfaction statements. The registration office does not provide any tests of the submitted programs, but if it appears during arbitration that some of the statements were false (such as not ail the test problems listed on the ‘test tape’ being actually executed and evaluated), the casecan be handled as ‘provision of false statement to a government agency’ by the common law. Of course, a correct solution also depends on correct interpretation of the problem to the program’s terms, correct modeling, and input. Correct understanding and postprocessing of the output is also essential. But the most important part is the program and its documentation. Results cannot be correct if the program fails to provide the correct solution to the submitted problem. Computer software testing is a complex, time consuming and expensive procedure. In the past, it was mostly the larger and more reputable organizations which developed software for mainframe computers and they tried to keep certain quality standards. But since minicomputers have become cheaper and more powerful, more and more complex problems can be solved with them. The number of software vendors has also increased significantly. As a result software quality asssurance is growing into a very important and critical problem. The proposed software tracking method is made to provide reasonable quality software for alI the professions with minimum reorgani~tion effort, oIKce staff and expense. The system also creates a wide variety of options and a framework for programming personnel.