- Email: [email protected]
Proposed structure for decomposition software reliability prediction model
Proposed structure for decomposition software reliability prediction model N Ashrafi, R C Baker* and J P Kuilboer*
Considerable emphasis in the software community has been directed at software quality. A major difficulty in dealing with software, however, is the absence of a widely accepted definition of software quality and lack of quantitative measures of software quality products. A general discussion is presented on the issues relating to the importance, definition, and measurement of software quality. The effectiveness of current approaches to enhance software quality is analysed. An approach is suggested that allows periodic measurement of software quality at different stages of the development process in a quantitative fashion. This approach allows the prediction o] the degree to which the final product can be relied on to perform its intended function. st~ftware quality, software reliability, modelling, prediction model
Software quality - why is it so important? As computer systems have proliferated and assumed the duties of more complex tasks, there has been a corresponding increase in efforts to create and maintain high-quality software. Littlewood 1 reports two principal reasons for the emphasis on software quality: the critical nature of some tasks performed by the computer and the rising cost of software development. There has been increasing dependence on computer systems to perform in situations where failure can be catastrophic, both personally and economically. Space and defence systems, manufacturing process control, and traffic control are just some of the application areas.
Also the continuing revolution in hardware technology has decreased hardware cost. The labour-intensive aspect of software development, however, will probably never allow such reductions in software cost to be achieved, particularly in the operation and maintenance phases. Thus rising cost of software has been associated with falling cost of hardware. These two reasons have intensified the pressures to quantify software quality and to measure and control the level of quality delivered. Department of Management Sciencc, University of Massachusetts. Boston, MA 02125, USA. *University of Texas, PO Box 19437, Arlington. TX 761119,USA vol 32 no 1 january/february 1990
Musa 2 gives an additional reason for the importance of software quality. There is international competition among the software developers to produce high-quality software as a means of gaining market advantage. The initial fascination with computing has diminished and users are demanding higher quality, lower cost, and faster delivery of software products. These are often conflicting needs. Software that can be produced quickly, and at little cost, is often of poor quality, while quality software is often more expensive and takes longer to produce. As software costs and schedules are quantitative measures, they tend to dominate as the primary criteria for competitive software, often at the neglect of quality. Software quality cannot, however, be overlooked. The additional time and maintenance costs attributed to poor-quality software can have a strong negative impact on the usability of software. The question is how to define and measure software quality.
M E A N I N G OF Q U A L I T Y IN S O F T W A R E Software quality is defined as: Totality of features and characteristics of a software product that bear on its ability to satisfy given needs; for example, conform to specifications (IEEE 1983). This definition indicates that the primary quality factor to be addressed for computer software is compliance with desired performance requirements (these requirements relate to the functions to be performed by the software). Note that quality is defined by relating it to conformance to specifications, that is, if a product meets its specifications, it is deemed to be of acceptable quality. Traditionally, the features of software that relate to its quality are identified and evaluated qualitatively, which is less precise than a quantitative measure because it involves human judgement and often arbitrary determination of some measure of quality. A more descriptive definition has been proposed by Cooper and Fisher 3, who define software quality as: "the composite of all attributes which describes the degree of excellence of the computer software' (p.7) The term 'degree of excellence' implies a measurable quality that can be precisely specified by the prospective software owner.
0951)-5849/90/0100934)6© 1990 Butterworth & Co (Publishers) Ltd
93
Evans and Marciniak 4 report that to ensure that development products meet a defined quality standard when delivered, standards must be defined early, be specific to the project, and define quality attributes for individual data products. Without a means to measure conformance to standards, however, quality cannot be monitored.
NEED TO MEASURE SOFTWARE QUALITY The foregoing argument stresses the need for a quantitative measure of software quality, especially to control the level of quality delivered. Musa e reports that quantifying software quality is beneficial for developers of the software as well as for users. Software-quality measures guide developers towards better decisions. They enable the developer to determine the quality level during testing, thus providing the means of evaluating the effect of various actions on quality. The user benefits from software-quality measures because software quality is directly related to the efficient operation of a software system. Thus if software quality is measured before delivery to the user, the time and money involved in software maintenance can be reduced. Software quality, therefore, is something that should be measured. There is a significant amount of ongoing effort within industry, academia, and government towards developing the entire area of what is called software-quality metrics. Concern is primarily centred around describing the overall characteristics, or attributes, of quality software and the measurement of these characteristics. Software-quality factors that are used to assess the quality of software include: maintainability, reliability, flexibility, correctness, testability, portability, reusability, efficiency, integrity, and interoperability. An examination of these factors reveals that all, except reliability, are stated in qualitative terms. The subsequent transformation into quantitative expressions is accomplished through metrics. It is generally agreed among practitioners and researchers that reliability is the most important and most measurable factor of software quality. Furthermore, reliability is the most customer-oriented attribute of software quality. It is the measurement of how well a piece of software functions to meet its operational requirements. As, ultimately, it is the purchaser of the software who must judge the system's quality, it seems justified to express the quality of the software in terms of its reliability. A major question, therefore, is how to measure and predict software reliability. This study addresses this question and proposes an approach that allows early prediction of software reliability.
RELIABILITY: MEASURABLE FACTOR Reliability, as mentioned, is the most quantifiable attribute of software quality and has become a central issue in computer system design. Software reliability 94
can be viewed as the extent to which the software will perform without any failure within a specified period in a given environment. Goel 5 reports that software is essentially an instrument used to process a set of inputs into a set of outputs. Clearly, software development is a process involved in implementing a human intellectual endeavour. Any human endeavour commonly makes errors. Thus software produced by humans is often imperfect, in the sense that a discrepancy exists between what the software can do versus what the user wants it to do. These discrepancies are called failures. Goel 5 considers the following to be some of the causes of software failure: 'An ignorance of the rules of the user requirements, ignorance of the rules of the computing environment, and the poor communication of software requirements between the user and the programmer or poor documentation of the software by the programmer.' (p. 1411) In addition to these, a flaw in program logic may also result in failure.
CURRENT APPROACHES TO PRODUCE RELIABLE SOFTWARE Software developers have several techniques to enhance the reliability of a software unit. DeMillo et al. 6 indicate that: 'The central technical issue concerns the existence of an independent objective scale upon which to base qualitative judgment of the ability of a given program (software) to function as intended in a specified environment over a specified time interval.' (p. 185) Although several scales have been suggested, it is generally agreed that software reliability depends on the software development methodology. The best technique to produce reliable software is to use a good development methodology. Software reliability also depends on validation methods. Software will probably be more reliable if a comprehensive set of testing strategies has been applied to validate it. Another method of enhancing software reliability is program proving.
SOFTWARE D E V E L O P M E N T There is considerable confusion in software engineering as to what method is most appropriate for software development. All agree, however, that the first two essential steps to producing reliable software are requirement specification and design activities.
Requirement specification Requirements refer to the needs of the users. Specification is the description of what the system is supposed to information and software technology
do to meet its requirements. Requirement specification is becoming an important issue in software engineering because several studies show that 30% of errors found during the testing and operation phase are due to poor understanding of requirements. These errors are almost 10-/00 times more expensive to correct than other kinds of errors during the operational phase 7. To avoid these errors, ambiguities, inconsistencies, and incompleteness in requirements must be minimized. Several specification models have been developed to overcome these problems, but none ensure developmcnt of an error-free software product.
System design It has been estimated that 30% of the time spent on software development goes into design. This stage is error prone, because most decisions are made here. Many design methodologies have been discussed, the most popular ones being functional decomposition, dataflow design, and datastructure design. Even though a good development approach increases software reliability, it does not guarantee error-free software. Thus some mechanism is needed to enhance software reliability further. Program proving and program testing are two ways to promote software reliability. Program proving is formal and mathematical 8, while program testing is more practical and heuristic 9. Despite the formalism and mathematical exactness of program proving, it is still an imperfect tool for verifying program correctness. For example, Gerhart and Yelowitz I° showed several programs that were proved to be correct but still contained errors. Program testing is the symbolic or physical execution of a set of test cases, to expose program errors. Like program proving, program testing remains an imperfect tool for assuring program correctness. A given testing strategy may be good for exposing certain kinds of errors, but not for all possible kinds of errors in a program. In practice, neither proving nor testing can guarantee complete confidence in the reliability of a software system. Due to the imperfection of these approaches in assuring a reliable program, a metric is needed that reflects the degree of program reliability. A commonly used approach for measuring software reliability is via an analytical model whose parameters are generally estimated from available data on software failure. Reliability and relevant measures are then computed fr~m the fitted models. RELIABILITY
MODELS
To evaluate a system's reliability in quantitative terms, a reliability model is needed. Software reliability models refer to mathematical models developed to assess the reliability of software from parameters that vol 32 no I january/february 1990
are either known or estimated from observations or experiments on software 11. Numerous efforts have been initiated to develop and validate mathematical models to estimate the reliability of software systems. In addition, efforts have been undertaken to develop better models for determining when a software package should be released to a potential user ~2. Most of these models are unstructured in that they treat software as a black box and attempt to model reliability against time. Practitioners argue that the primary problem with existing software reliability models is that they are developed to evaluate software reliability during the system testing phase at the delivery and into operation. The problem with this approach is that if the software system has a low reliability measure due to specification errors and/or design logic flaw it is difficult, if not impossible, to go back and correct these errors. Thus the user is forced to accept a product that cannot perform the intended functions. An approach is needed that enables the employment of software reliability models throughout the development process. A framework is suggested that allows the prediction of a software system based on the reliability of its components. The primary advantage of this model is that it may be used to obtain a rough assessment of software reliability during software development. This assessment can be updated and improved as more data becomes available. For example, before any module or integration testing is performed, the model can be used to predict software reliability based on the software engineer's belief in the performance of the modules. As modules are developed, tested, debugged, and integrated, new sets of data on module failure and interfacing failure become available and may be used to update the assessed reliability of the software. This updating process continues until the estimated software reliability is acceptable. An accurate reliability estimate can only be achieved, however, once data from the system testing phase becomes available. The proposed approach is only applicable to software systems that can be decomposed into their elementary components. DECOMPOSITION SYSTEM
OF SOFTWARE
The most important step in the practical application of software reliability modelling is gaining a full understanding of the software system. A software system may be divided into a set of components, where the reliability of each component is known or can be estimated. Then the relationship between the reliabilities of the components and the overall reliability of the system can be established. There is need for a paradigm that treats software as a system that can be decomposed into subsystems where subsystems comprise the programs and programs are further divided into modules. The following are definitions of software system components in the context of software reliability. 95
Software as system
A software product is developed to perform a set of functions. The system concept relates to the examination of these functions, which the system must perform, and the needs it must meet. Depending on the size of the project, the system is divided into detailed functions, which can be assigned to individual software developers. Thus the overall system may be divided into subsystems where a subsystem is a collection of programs that accomplish some specified function. A program is defined as a set of complete machine instructions (operations with operands) that execute within a single computer and relate to the accomplishment of some major function. In this case, program is defined as a string of modules that perform a specific function. Note that modules are combined to create programs, programs are combined into subsystems, and subsystems are integrated into software systems. Thus the reliability of a software system may be calculated from the reliabilities of its components. Therefore, the first step in computing software reliability is to measure model reliability and the probability of interface failure. This is started by defining module, module failure, and module interface failure. Module
There are many definitions of the term 'module'. In general, it is used to refer to a separately compilable program component. Each module is devoted to one or more tasks related to a function and may be accessed from one or several places in a software system. In addition, American Defense Standard DOD-STD2167A defines modules as a computer software unit (CSU), which is 'an element specified in the design of a computer software component (CSC) that is separately testable.' CSCs make up computer software configuration items (CSCIs), which are the software configuration items for the system. Module failure The first step in understanding module failure is defining its external characteristics. This information is expressed in a module external specification, which contains all information needed by modules that call this module. According to Pressman 13 the module external specification should contain the following information.
• Module name. • Module function: describes what the module does when it is called. • Input: description of all input parameters of the module. It includes a definition of format, size, attributes, units, and valid domain of all input data. • Output: description of all output data structure by the module. It should include the definition of the format, size, attributes, units, and valid ranges of all outputs. This description should relate the outputs to the inputs in a cause-and-effect manner. 96
Module specifications allow flaws in the structure, such as missing functions, incomplete interfaces, and incorrect results, to be looked for. Test cases are designed and used to find discrepancies between the module's logic and interfaces and its module external specifications. Module interface
Modules interface through their use of data. The interface may be obvious and direct through data passed from one module to another or modules may interface through multiple use of common data. When a system is designed to interface modules by passing data from one module to another, these modules are serially reusable and are designed to be executed sequentially as part of one user task. Modules may also be re-entrant as opposed to reusable serially. Reentrant modules can be entered for multiple calls to execute before completion of previous calls. STRUCTURE
OF MODEL
The model suggested by this study is a decomposition model. The software system is broken down into subsystems, where each subsystem is described as a collection of programs. For this study, subsystems are defined as a collection of programs, each starting with the same module and each program as a sequence of modules that interact with one another. A breakdown of a software system with three modules is presented in Figure 1. Software system is decomposed into three subsystems, each containing all programs that have the same initial module. For example, subsystem 1 is the collection of programs that contain a sequence of modules where the starting module is module 1. The letter C indicates successful completion of each program. Note that each module can be called from many places in the software system. This structure allows the calculation of software reliability in terms of its subsystem and how frequently each subsystem is used. The reliability of each subsystem is calculated in terms of successful completion of each program and frequency of use. The reliability of
l Software system l I
i
t
i
I
Figure 1.Breakdown of software system with three modules information and software technology
each program is a function of the reliabilities of the modules they employ, the frequency of transfer of control among modules, and the probability of successful transfer of control (interfacing) among modules. Thus prediction of software reliability is based on the following components:
mated by the ratio of the successful transfers of control between modules i and j to the total number of transfers. Without such information, subjective estimates might be assigned as the starting point of an estimating procedure.
• • • •
This study considers software systems in which modules interface by passing data from one module to another. These modules are serially reusable and designed to be executed sequentially as part of one user task. This means that control tends to pass sequentially from module to module. The transfer of control probability can be estimated by observing the proportion of i to j transfers that occurs during system testing, or subjective estimates might be assigned as a starting point of an estimating procedure.
module reliability interface reliability transfer of control probability user profile, where user profile determines the control structure of the software system, frequency of use of each program, and consequently the frequency of use of each subsystem
Each component is discussed separately, then in combination with other components, to illustrate the formulation of the proposed model. Module reliability Under the assumption that the results of a software run may be described as a Bernoulli process, the binomial distribution can be used to describe the number of module failures in N runs, where N is the number of test cases used for module testing. Two points must be emphasized: • The runs are independent from each other because each run uses a different set of inputs. • The number of runs for each module will be determined by the complexity and size of the module. Note that special care must be taken to ensure that the test cases are designed to represent the operational use of the software system. Reliability of each module is defined as the probability that the module performs successfully according to module specification. Statistically, module reliability can be estimated by the ratio of the number of module outputs that meet output specifications to the total number of outputs generated by the module. To obtain this ratio the black-box testing strategy is employed to verify the end-results. The white-box approach is usually used to detect errors in the internal structure of a module 9. If all errors are detected and repaired the reliability of the module will be one. In practice, however, not all errors are detected nor all detected errors repaired due to cost and time limitations. Thus after white-box testing is performed the module still contains a specified number of errors that will manifest themselves during black-box testing as the input parameters vary.
Interface reliability In practice new sources of failure are introduced when modules are combined within a program. These interfacing failures occur because transfers of control between modules are themselves failure prone. The interfacing failure probabilities can be estimated by examining the failure log and observing the proportion of i to j module transfers that fail during integration testing. Statistically, interface reliability can be estivol 32 no 1 january/february 1990
Transfer of control probability
User profile Reliability of a software system may be defined as the probability that the system will give the correct output with a given set of input from the user environment. The sequence of the modules to be energized depends on these input sets. In operational use of a software system, however, input sets are selected according to some operational requirements or user profile, where user profile indicates how the system will be used. Thus user profile allows determination of the control structure of the programs, direction and frequency of transfer of control between modules, frequency of use of programs, and frequency of use of subsystems. The structure of each program to the sequence of modules that are activated in a particular run depends on the input data, which are determined by user profile. Reliability of the system, therefore, depends on the user profile. In most cases, however, either the user profile does not exist with certainty or it changes over time. If the user profile does not exist explicitly and the structure of the software system allows stochastic movement between modules, then transfer of control between modules is a probabilistic event and must be described according to some probabilistic model. Conversely, if the user profile is known with certainty, transfer of control from module to module is a deterministic event. Both situations can be addressed. ASSUMPTIONS
OF MODEL
In assessment of software reliability using the proposed model the following underlying assumptions are made: • The software system was developed using the modular programming technique. • The program structure must be sequential. • Module reliability remains constant for a given user profile. • Interface reliability remains constant for a given user profile. The following is a more indepth discussion of these assumptions. 97
Modular programming technique It is assumed that the modular programming technique is used to develop the software system. The necessity of this is that the model is a decomposition model that assesses the reliability of software in terms of the components of the software system. Hence a decomposed structure is required to be able to measure the reliability of the components. Sequential software structure The structure of the software system is assumed to be sequential. Thus the correct output of the program depends on the successful execution of each module that is activated. Constant module reliability It is assumed that the probability of a successful module run remains constant. Recall that the modules are tested independently and most errors detected and removed. As there may be a deliberate decision not to repair an error that causes failure, module reliability is less than perfect. Thus individual modules, when executed, produce desired output with a constant probability that differs from module to module. That is, when in module i, if no modification is made and the user profile does not change, the module reliability remains invariant and module failure can be described according to a binomial distribution. Constant interface reliability It is assumed that the probability of a successful transfer of control between module i and j remains constant over all calls between module i and j. Thus the number of interfacing failures in N module interfaces follows a binomial distribution. When estimating this interface reliability care should be exercised to ensure that the test cases used during integration testing are representative of the user profile. In addition, the number of interfacing opportunities and the number of interfacing failures between modules should be recorded. Note that interface reliability after integration testing in most cases is expected to be close to one. CONCLUSIONS The continuing problem of definition and measurement of software quality is discussed. Quality of a software system is expressed in terms of its reliability, which shows the probability that the software system will operate according to its specifications. A framework for early measurement of software reliability is proposed that is predictive in nature and oriented toward different phases of development rather than only the finished system. This may be a solution to the current problem of evaluating software quality only at the delivery and into operations, at time, when modification and improvement is almost impossible due to cost and time limitations. The model is a decomposition model in that the software system is decomposed into subsystems. Each 98
subsystem is described as a collection of programs. The reliability of each subsystem is therefore calculated in terms of the successful completion of each program. Each program consists of a sequence of modules that interact with one another. Thus the reliability of each program is a function of the reliabilities of the modules they employ. Each component is discussed separately and then in combination with other components to illustrate the formulation of a framework for the assessment prediction of software reliability and hence the quality of a software product. The next step in validation of the proposed structure for the decomposition software reliability prediction model is to formalize the mathematical model that relates software reliability to that of its components. This may be followed by an empirical study of the actual development of a software system and the prediction of its reliability. This predicted reliability may be validated later with field-operational data.
REFERENCES 1 Littlewood, B 'Software reliability model for modular program structure' IEEE Trans. Reliability Vol 28 (August 1979) pp 241-246 2 Musa, J D 'Software quality and reliability basics' in
Proc. 1987 Fall Joint Computer Conf. Exploring Technology Today and Tomorrow (25-29 October 3
4
5
6
7 8 9 10
11 12
1987) Infomart, Dallas, TX, USA (1987) pp 114-123 Cooper, J D and Fisher, M J (eds) Software quality management Petrocelli, New York, NY, USA (1979) Evans, M W and Marciuiak, J Software quality assurance and management John Wiley, New York, NY, USA (1987) Goel, A L 'Software reliability modeling and related topics: a survey' Technical report Syracuse University, Syracuse, NY, USA (October 1985) DeMilio, R A, McCracken, W M, Martin, R J and Passafiume, J F Software testing and evaluation Benjamin/Cummings, Menlo Park, CA, USA (1987) Boehm, B W Software engineering economics Prentice Hall, Englewood Cliffs, NJ, USA (1981) Manna, Z Mathematical theory of computation McGraw-Hill, New York, NY, USA (1974) Myers, G J The art of software testing John Wiley, New York, NY, USA (1979) Gerhart, S and Yelowitz, L 'Observations of fallibility in applications of modern programming methodologies' 1EEE Trans. Soft. Eng. Vol 2 No 5 (May 1976) pp 195-207 Glass, R L Software reliability guidebook Prentice Hall, Englewood Cliffs, NJ, USA (1979) Musa, J D, lannino, A and Okumoto, K Software
reliability: measurement, prediction, application McGraw-Hill, New York, NY, USA (1987) 13 Pressman, R S Software engineering a beginner's guide McGraw-Hill, New York, NY, USA (1988) information and software technology
Considerable emphasis in the software community has been directed at software quality. A major difficulty in dealing with software, however, is the absence of a widely accepted definition of software quality and lack of quantitative measures of software quality products. A general discussion is presented on the issues relating to the importance, definition, and measurement of software quality. The effectiveness of current approaches to enhance software quality is analysed. An approach is suggested that allows periodic measurement of software quality at different stages of the development process in a quantitative fashion. This approach allows the prediction o] the degree to which the final product can be relied on to perform its intended function. st~ftware quality, software reliability, modelling, prediction model
Software quality - why is it so important? As computer systems have proliferated and assumed the duties of more complex tasks, there has been a corresponding increase in efforts to create and maintain high-quality software. Littlewood 1 reports two principal reasons for the emphasis on software quality: the critical nature of some tasks performed by the computer and the rising cost of software development. There has been increasing dependence on computer systems to perform in situations where failure can be catastrophic, both personally and economically. Space and defence systems, manufacturing process control, and traffic control are just some of the application areas.
Also the continuing revolution in hardware technology has decreased hardware cost. The labour-intensive aspect of software development, however, will probably never allow such reductions in software cost to be achieved, particularly in the operation and maintenance phases. Thus rising cost of software has been associated with falling cost of hardware. These two reasons have intensified the pressures to quantify software quality and to measure and control the level of quality delivered. Department of Management Sciencc, University of Massachusetts. Boston, MA 02125, USA. *University of Texas, PO Box 19437, Arlington. TX 761119,USA vol 32 no 1 january/february 1990
Musa 2 gives an additional reason for the importance of software quality. There is international competition among the software developers to produce high-quality software as a means of gaining market advantage. The initial fascination with computing has diminished and users are demanding higher quality, lower cost, and faster delivery of software products. These are often conflicting needs. Software that can be produced quickly, and at little cost, is often of poor quality, while quality software is often more expensive and takes longer to produce. As software costs and schedules are quantitative measures, they tend to dominate as the primary criteria for competitive software, often at the neglect of quality. Software quality cannot, however, be overlooked. The additional time and maintenance costs attributed to poor-quality software can have a strong negative impact on the usability of software. The question is how to define and measure software quality.
M E A N I N G OF Q U A L I T Y IN S O F T W A R E Software quality is defined as: Totality of features and characteristics of a software product that bear on its ability to satisfy given needs; for example, conform to specifications (IEEE 1983). This definition indicates that the primary quality factor to be addressed for computer software is compliance with desired performance requirements (these requirements relate to the functions to be performed by the software). Note that quality is defined by relating it to conformance to specifications, that is, if a product meets its specifications, it is deemed to be of acceptable quality. Traditionally, the features of software that relate to its quality are identified and evaluated qualitatively, which is less precise than a quantitative measure because it involves human judgement and often arbitrary determination of some measure of quality. A more descriptive definition has been proposed by Cooper and Fisher 3, who define software quality as: "the composite of all attributes which describes the degree of excellence of the computer software' (p.7) The term 'degree of excellence' implies a measurable quality that can be precisely specified by the prospective software owner.
0951)-5849/90/0100934)6© 1990 Butterworth & Co (Publishers) Ltd
93
Evans and Marciniak 4 report that to ensure that development products meet a defined quality standard when delivered, standards must be defined early, be specific to the project, and define quality attributes for individual data products. Without a means to measure conformance to standards, however, quality cannot be monitored.
NEED TO MEASURE SOFTWARE QUALITY The foregoing argument stresses the need for a quantitative measure of software quality, especially to control the level of quality delivered. Musa e reports that quantifying software quality is beneficial for developers of the software as well as for users. Software-quality measures guide developers towards better decisions. They enable the developer to determine the quality level during testing, thus providing the means of evaluating the effect of various actions on quality. The user benefits from software-quality measures because software quality is directly related to the efficient operation of a software system. Thus if software quality is measured before delivery to the user, the time and money involved in software maintenance can be reduced. Software quality, therefore, is something that should be measured. There is a significant amount of ongoing effort within industry, academia, and government towards developing the entire area of what is called software-quality metrics. Concern is primarily centred around describing the overall characteristics, or attributes, of quality software and the measurement of these characteristics. Software-quality factors that are used to assess the quality of software include: maintainability, reliability, flexibility, correctness, testability, portability, reusability, efficiency, integrity, and interoperability. An examination of these factors reveals that all, except reliability, are stated in qualitative terms. The subsequent transformation into quantitative expressions is accomplished through metrics. It is generally agreed among practitioners and researchers that reliability is the most important and most measurable factor of software quality. Furthermore, reliability is the most customer-oriented attribute of software quality. It is the measurement of how well a piece of software functions to meet its operational requirements. As, ultimately, it is the purchaser of the software who must judge the system's quality, it seems justified to express the quality of the software in terms of its reliability. A major question, therefore, is how to measure and predict software reliability. This study addresses this question and proposes an approach that allows early prediction of software reliability.
RELIABILITY: MEASURABLE FACTOR Reliability, as mentioned, is the most quantifiable attribute of software quality and has become a central issue in computer system design. Software reliability 94
can be viewed as the extent to which the software will perform without any failure within a specified period in a given environment. Goel 5 reports that software is essentially an instrument used to process a set of inputs into a set of outputs. Clearly, software development is a process involved in implementing a human intellectual endeavour. Any human endeavour commonly makes errors. Thus software produced by humans is often imperfect, in the sense that a discrepancy exists between what the software can do versus what the user wants it to do. These discrepancies are called failures. Goel 5 considers the following to be some of the causes of software failure: 'An ignorance of the rules of the user requirements, ignorance of the rules of the computing environment, and the poor communication of software requirements between the user and the programmer or poor documentation of the software by the programmer.' (p. 1411) In addition to these, a flaw in program logic may also result in failure.
CURRENT APPROACHES TO PRODUCE RELIABLE SOFTWARE Software developers have several techniques to enhance the reliability of a software unit. DeMillo et al. 6 indicate that: 'The central technical issue concerns the existence of an independent objective scale upon which to base qualitative judgment of the ability of a given program (software) to function as intended in a specified environment over a specified time interval.' (p. 185) Although several scales have been suggested, it is generally agreed that software reliability depends on the software development methodology. The best technique to produce reliable software is to use a good development methodology. Software reliability also depends on validation methods. Software will probably be more reliable if a comprehensive set of testing strategies has been applied to validate it. Another method of enhancing software reliability is program proving.
SOFTWARE D E V E L O P M E N T There is considerable confusion in software engineering as to what method is most appropriate for software development. All agree, however, that the first two essential steps to producing reliable software are requirement specification and design activities.
Requirement specification Requirements refer to the needs of the users. Specification is the description of what the system is supposed to information and software technology
do to meet its requirements. Requirement specification is becoming an important issue in software engineering because several studies show that 30% of errors found during the testing and operation phase are due to poor understanding of requirements. These errors are almost 10-/00 times more expensive to correct than other kinds of errors during the operational phase 7. To avoid these errors, ambiguities, inconsistencies, and incompleteness in requirements must be minimized. Several specification models have been developed to overcome these problems, but none ensure developmcnt of an error-free software product.
System design It has been estimated that 30% of the time spent on software development goes into design. This stage is error prone, because most decisions are made here. Many design methodologies have been discussed, the most popular ones being functional decomposition, dataflow design, and datastructure design. Even though a good development approach increases software reliability, it does not guarantee error-free software. Thus some mechanism is needed to enhance software reliability further. Program proving and program testing are two ways to promote software reliability. Program proving is formal and mathematical 8, while program testing is more practical and heuristic 9. Despite the formalism and mathematical exactness of program proving, it is still an imperfect tool for verifying program correctness. For example, Gerhart and Yelowitz I° showed several programs that were proved to be correct but still contained errors. Program testing is the symbolic or physical execution of a set of test cases, to expose program errors. Like program proving, program testing remains an imperfect tool for assuring program correctness. A given testing strategy may be good for exposing certain kinds of errors, but not for all possible kinds of errors in a program. In practice, neither proving nor testing can guarantee complete confidence in the reliability of a software system. Due to the imperfection of these approaches in assuring a reliable program, a metric is needed that reflects the degree of program reliability. A commonly used approach for measuring software reliability is via an analytical model whose parameters are generally estimated from available data on software failure. Reliability and relevant measures are then computed fr~m the fitted models. RELIABILITY
MODELS
To evaluate a system's reliability in quantitative terms, a reliability model is needed. Software reliability models refer to mathematical models developed to assess the reliability of software from parameters that vol 32 no I january/february 1990
are either known or estimated from observations or experiments on software 11. Numerous efforts have been initiated to develop and validate mathematical models to estimate the reliability of software systems. In addition, efforts have been undertaken to develop better models for determining when a software package should be released to a potential user ~2. Most of these models are unstructured in that they treat software as a black box and attempt to model reliability against time. Practitioners argue that the primary problem with existing software reliability models is that they are developed to evaluate software reliability during the system testing phase at the delivery and into operation. The problem with this approach is that if the software system has a low reliability measure due to specification errors and/or design logic flaw it is difficult, if not impossible, to go back and correct these errors. Thus the user is forced to accept a product that cannot perform the intended functions. An approach is needed that enables the employment of software reliability models throughout the development process. A framework is suggested that allows the prediction of a software system based on the reliability of its components. The primary advantage of this model is that it may be used to obtain a rough assessment of software reliability during software development. This assessment can be updated and improved as more data becomes available. For example, before any module or integration testing is performed, the model can be used to predict software reliability based on the software engineer's belief in the performance of the modules. As modules are developed, tested, debugged, and integrated, new sets of data on module failure and interfacing failure become available and may be used to update the assessed reliability of the software. This updating process continues until the estimated software reliability is acceptable. An accurate reliability estimate can only be achieved, however, once data from the system testing phase becomes available. The proposed approach is only applicable to software systems that can be decomposed into their elementary components. DECOMPOSITION SYSTEM
OF SOFTWARE
The most important step in the practical application of software reliability modelling is gaining a full understanding of the software system. A software system may be divided into a set of components, where the reliability of each component is known or can be estimated. Then the relationship between the reliabilities of the components and the overall reliability of the system can be established. There is need for a paradigm that treats software as a system that can be decomposed into subsystems where subsystems comprise the programs and programs are further divided into modules. The following are definitions of software system components in the context of software reliability. 95
Software as system
A software product is developed to perform a set of functions. The system concept relates to the examination of these functions, which the system must perform, and the needs it must meet. Depending on the size of the project, the system is divided into detailed functions, which can be assigned to individual software developers. Thus the overall system may be divided into subsystems where a subsystem is a collection of programs that accomplish some specified function. A program is defined as a set of complete machine instructions (operations with operands) that execute within a single computer and relate to the accomplishment of some major function. In this case, program is defined as a string of modules that perform a specific function. Note that modules are combined to create programs, programs are combined into subsystems, and subsystems are integrated into software systems. Thus the reliability of a software system may be calculated from the reliabilities of its components. Therefore, the first step in computing software reliability is to measure model reliability and the probability of interface failure. This is started by defining module, module failure, and module interface failure. Module
There are many definitions of the term 'module'. In general, it is used to refer to a separately compilable program component. Each module is devoted to one or more tasks related to a function and may be accessed from one or several places in a software system. In addition, American Defense Standard DOD-STD2167A defines modules as a computer software unit (CSU), which is 'an element specified in the design of a computer software component (CSC) that is separately testable.' CSCs make up computer software configuration items (CSCIs), which are the software configuration items for the system. Module failure The first step in understanding module failure is defining its external characteristics. This information is expressed in a module external specification, which contains all information needed by modules that call this module. According to Pressman 13 the module external specification should contain the following information.
• Module name. • Module function: describes what the module does when it is called. • Input: description of all input parameters of the module. It includes a definition of format, size, attributes, units, and valid domain of all input data. • Output: description of all output data structure by the module. It should include the definition of the format, size, attributes, units, and valid ranges of all outputs. This description should relate the outputs to the inputs in a cause-and-effect manner. 96
Module specifications allow flaws in the structure, such as missing functions, incomplete interfaces, and incorrect results, to be looked for. Test cases are designed and used to find discrepancies between the module's logic and interfaces and its module external specifications. Module interface
Modules interface through their use of data. The interface may be obvious and direct through data passed from one module to another or modules may interface through multiple use of common data. When a system is designed to interface modules by passing data from one module to another, these modules are serially reusable and are designed to be executed sequentially as part of one user task. Modules may also be re-entrant as opposed to reusable serially. Reentrant modules can be entered for multiple calls to execute before completion of previous calls. STRUCTURE
OF MODEL
The model suggested by this study is a decomposition model. The software system is broken down into subsystems, where each subsystem is described as a collection of programs. For this study, subsystems are defined as a collection of programs, each starting with the same module and each program as a sequence of modules that interact with one another. A breakdown of a software system with three modules is presented in Figure 1. Software system is decomposed into three subsystems, each containing all programs that have the same initial module. For example, subsystem 1 is the collection of programs that contain a sequence of modules where the starting module is module 1. The letter C indicates successful completion of each program. Note that each module can be called from many places in the software system. This structure allows the calculation of software reliability in terms of its subsystem and how frequently each subsystem is used. The reliability of each subsystem is calculated in terms of successful completion of each program and frequency of use. The reliability of
l Software system l I
i
t
i
I
Figure 1.Breakdown of software system with three modules information and software technology
each program is a function of the reliabilities of the modules they employ, the frequency of transfer of control among modules, and the probability of successful transfer of control (interfacing) among modules. Thus prediction of software reliability is based on the following components:
mated by the ratio of the successful transfers of control between modules i and j to the total number of transfers. Without such information, subjective estimates might be assigned as the starting point of an estimating procedure.
• • • •
This study considers software systems in which modules interface by passing data from one module to another. These modules are serially reusable and designed to be executed sequentially as part of one user task. This means that control tends to pass sequentially from module to module. The transfer of control probability can be estimated by observing the proportion of i to j transfers that occurs during system testing, or subjective estimates might be assigned as a starting point of an estimating procedure.
module reliability interface reliability transfer of control probability user profile, where user profile determines the control structure of the software system, frequency of use of each program, and consequently the frequency of use of each subsystem
Each component is discussed separately, then in combination with other components, to illustrate the formulation of the proposed model. Module reliability Under the assumption that the results of a software run may be described as a Bernoulli process, the binomial distribution can be used to describe the number of module failures in N runs, where N is the number of test cases used for module testing. Two points must be emphasized: • The runs are independent from each other because each run uses a different set of inputs. • The number of runs for each module will be determined by the complexity and size of the module. Note that special care must be taken to ensure that the test cases are designed to represent the operational use of the software system. Reliability of each module is defined as the probability that the module performs successfully according to module specification. Statistically, module reliability can be estimated by the ratio of the number of module outputs that meet output specifications to the total number of outputs generated by the module. To obtain this ratio the black-box testing strategy is employed to verify the end-results. The white-box approach is usually used to detect errors in the internal structure of a module 9. If all errors are detected and repaired the reliability of the module will be one. In practice, however, not all errors are detected nor all detected errors repaired due to cost and time limitations. Thus after white-box testing is performed the module still contains a specified number of errors that will manifest themselves during black-box testing as the input parameters vary.
Interface reliability In practice new sources of failure are introduced when modules are combined within a program. These interfacing failures occur because transfers of control between modules are themselves failure prone. The interfacing failure probabilities can be estimated by examining the failure log and observing the proportion of i to j module transfers that fail during integration testing. Statistically, interface reliability can be estivol 32 no 1 january/february 1990
Transfer of control probability
User profile Reliability of a software system may be defined as the probability that the system will give the correct output with a given set of input from the user environment. The sequence of the modules to be energized depends on these input sets. In operational use of a software system, however, input sets are selected according to some operational requirements or user profile, where user profile indicates how the system will be used. Thus user profile allows determination of the control structure of the programs, direction and frequency of transfer of control between modules, frequency of use of programs, and frequency of use of subsystems. The structure of each program to the sequence of modules that are activated in a particular run depends on the input data, which are determined by user profile. Reliability of the system, therefore, depends on the user profile. In most cases, however, either the user profile does not exist with certainty or it changes over time. If the user profile does not exist explicitly and the structure of the software system allows stochastic movement between modules, then transfer of control between modules is a probabilistic event and must be described according to some probabilistic model. Conversely, if the user profile is known with certainty, transfer of control from module to module is a deterministic event. Both situations can be addressed. ASSUMPTIONS
OF MODEL
In assessment of software reliability using the proposed model the following underlying assumptions are made: • The software system was developed using the modular programming technique. • The program structure must be sequential. • Module reliability remains constant for a given user profile. • Interface reliability remains constant for a given user profile. The following is a more indepth discussion of these assumptions. 97
Modular programming technique It is assumed that the modular programming technique is used to develop the software system. The necessity of this is that the model is a decomposition model that assesses the reliability of software in terms of the components of the software system. Hence a decomposed structure is required to be able to measure the reliability of the components. Sequential software structure The structure of the software system is assumed to be sequential. Thus the correct output of the program depends on the successful execution of each module that is activated. Constant module reliability It is assumed that the probability of a successful module run remains constant. Recall that the modules are tested independently and most errors detected and removed. As there may be a deliberate decision not to repair an error that causes failure, module reliability is less than perfect. Thus individual modules, when executed, produce desired output with a constant probability that differs from module to module. That is, when in module i, if no modification is made and the user profile does not change, the module reliability remains invariant and module failure can be described according to a binomial distribution. Constant interface reliability It is assumed that the probability of a successful transfer of control between module i and j remains constant over all calls between module i and j. Thus the number of interfacing failures in N module interfaces follows a binomial distribution. When estimating this interface reliability care should be exercised to ensure that the test cases used during integration testing are representative of the user profile. In addition, the number of interfacing opportunities and the number of interfacing failures between modules should be recorded. Note that interface reliability after integration testing in most cases is expected to be close to one. CONCLUSIONS The continuing problem of definition and measurement of software quality is discussed. Quality of a software system is expressed in terms of its reliability, which shows the probability that the software system will operate according to its specifications. A framework for early measurement of software reliability is proposed that is predictive in nature and oriented toward different phases of development rather than only the finished system. This may be a solution to the current problem of evaluating software quality only at the delivery and into operations, at time, when modification and improvement is almost impossible due to cost and time limitations. The model is a decomposition model in that the software system is decomposed into subsystems. Each 98
subsystem is described as a collection of programs. The reliability of each subsystem is therefore calculated in terms of the successful completion of each program. Each program consists of a sequence of modules that interact with one another. Thus the reliability of each program is a function of the reliabilities of the modules they employ. Each component is discussed separately and then in combination with other components to illustrate the formulation of a framework for the assessment prediction of software reliability and hence the quality of a software product. The next step in validation of the proposed structure for the decomposition software reliability prediction model is to formalize the mathematical model that relates software reliability to that of its components. This may be followed by an empirical study of the actual development of a software system and the prediction of its reliability. This predicted reliability may be validated later with field-operational data.
REFERENCES 1 Littlewood, B 'Software reliability model for modular program structure' IEEE Trans. Reliability Vol 28 (August 1979) pp 241-246 2 Musa, J D 'Software quality and reliability basics' in
Proc. 1987 Fall Joint Computer Conf. Exploring Technology Today and Tomorrow (25-29 October 3
4
5
6
7 8 9 10
11 12
1987) Infomart, Dallas, TX, USA (1987) pp 114-123 Cooper, J D and Fisher, M J (eds) Software quality management Petrocelli, New York, NY, USA (1979) Evans, M W and Marciuiak, J Software quality assurance and management John Wiley, New York, NY, USA (1987) Goel, A L 'Software reliability modeling and related topics: a survey' Technical report Syracuse University, Syracuse, NY, USA (October 1985) DeMilio, R A, McCracken, W M, Martin, R J and Passafiume, J F Software testing and evaluation Benjamin/Cummings, Menlo Park, CA, USA (1987) Boehm, B W Software engineering economics Prentice Hall, Englewood Cliffs, NJ, USA (1981) Manna, Z Mathematical theory of computation McGraw-Hill, New York, NY, USA (1974) Myers, G J The art of software testing John Wiley, New York, NY, USA (1979) Gerhart, S and Yelowitz, L 'Observations of fallibility in applications of modern programming methodologies' 1EEE Trans. Soft. Eng. Vol 2 No 5 (May 1976) pp 195-207 Glass, R L Software reliability guidebook Prentice Hall, Englewood Cliffs, NJ, USA (1979) Musa, J D, lannino, A and Okumoto, K Software
reliability: measurement, prediction, application McGraw-Hill, New York, NY, USA (1987) 13 Pressman, R S Software engineering a beginner's guide McGraw-Hill, New York, NY, USA (1988) information and software technology