Improving risk management: moving from risk elimination to risk avoidance

Information and Software Technology 41 (1999) 29–34

Improving risk management: moving from risk elimination to risk avoidance Terry R. Adler*, John G. Leonard, Ric K. Nordgren Department of Graduate Acquisition Management, Air Force Institute of Technology, Wright-Patterson Air Force Base, OH 45433-7765, USA Received 14 January 1998; received in revised form 11 May 1998; accepted 9 August 1998

Abstract This study has examined software reliability using a risk management framework to analyze popular risk management models with regard to their emphasis on risk avoidance. Too often, risk reduction occurs late in the software development life cycle when changes are costly and less effective. We suggest that early risk avoidance techniques, like the cleanroom software development process and its sub-process of software inspections, leads to superior products. Our approach is different from the current mainstream approach of testing to eliminate errors because we propose using risk models that emphasize preventive risk management early in development. We argue that the use of risk avoidance leads to benefits that far outweigh the cost of implementation. Published by Elsevier Science B.V. Keywords: Risk avoidance; Cleanroom development; Life-cycle cost

1. Introduction

2. Risk management theory and issues

Software development is currently one of the most riskprone areas of management in this decade [1]. Software development projects contain risk events that can negatively impact the development process and, if neglected, lead to massive product changes [2] and program failure [3]. The purpose of this paper is to analyze software reliability, using a risk management framework, to address specifically how risk assessment models incorporate risk avoidance techniques. We suggest that early risk avoidance techniques, like the cleanroom software development process with software inspections, leads to superior processes and products [4]. Our approach is different from the current mainstream approach of testing to eliminate errors, which investigates and corrects errors as they occur [5]. We propose that preventive risk management, with an emphasis on risk avoidance, is the better approach because errors are easier to fix, is cheaper in the long term, and is less complex than reactive approaches that occur later in the more expensive phases and activities of the software development life cycle. Without this rudimentary software development philosophy, software development practices will continue to be haphazard, reactionary and idiosyncratic.

The Department of the Air Force’s Software Technology Support Center defines risk as ‘‘the probability of an undesirable event occurring and the impact of that event occurring’’ ([1], p. 6-1). Basically anything that can affect a program, project or process implies a certain level of risk. Siropolis [6] identifies three types of risk: pure, fundamental and speculative. Risks are pure if they result in either a loss or no possibility of gain. Pure risks, such as fire, theft or the death of a key person, are beyond the control of the organization. Fundamental risk differs from pure risk in that risk events typically come from the environment. Natural disasters such as floods, tornadoes and snowstorms are fundamental risk. The saying ‘‘a rising tide affects all ships equally’’ applies to fundamental risk. However, there is little that can be done to avoid pure and fundamental risk [6]. Speculative risk involves gain or loss by an organization. An example would be the development of a new software product since the organization has the potential to reap great reward if the software enhances productivity. Alternatively, the new software product could cause a loss (i.e., loss of investment). This type of risk is classified as speculative because it is the organization, not fate, which primarily determines the outcome of the activity. Thus, speculative risk is largely deterministic in that organizational resources,

* Corresponding author. Tel.: ⫹ 1 937 255 6565 ext. 3313; fax: ⫹ 1 937 656 7988. 0950-5849/99/$ - see front matter Published by Elsevier Science B.V. PII: S0950-584 9(98)00095-0

30

T.R. Adler et al. / Information and Software Technology 41 (1999) 29–34

Fig. 1. Software Development Risk Taxonomy.

processes and core technologies affect development success or failure. Risk management, then, is the process by which an organization attempts to limit the effects of speculative risk. Software managers are limited, however, because software development processes are typically immature, chaotic and unpredictable. Compounded with inadequate estimates of cost and schedule, decision-makers often react to software problems to gain ‘‘control’’ of software developments far too late in the software development life cycle [7]. Ignoring risk only provides an opportunity to recognize the risk ex post facto, which is the software risk management practice called risk elimination. We suggest that proper risk management should consider risk avoidance. In this study, we compare and contrast four software risk management models. Although there are many models to choose from, these models represent a broad spectrum of risk management tools. The models include Software Risk Evaluation (SRE), Boehm’s Software Risk Management, Best Practices Initiative Risk Management, and Team Risk Management. Risk management, in general, should be disciplined, systematic, repeatable, and based on solid principles [1, 2]. An adequate risk management plan should balance corrective and preventive approaches to risk management. Our analysis investigates each model’s emphasis on risk avoidance, a preventive development approach, since corrective risk management is more common.

3. An analysis of software risk models and issues 3.1. Software Risk Evaluation (SRE) The SRE model was developed by the Software Engineering Institute (SEI) at Carnegie Melon University to identify, analyze, communicate and mitigate the technical risks associated with the acquisition of software-intensive systems. The SRE model includes the Risk Management Paradigm, Software Development Risk Taxonomy and Taxonomy-Based Questionnaire methods of analysis. The Risk Management Paradigm has four primary functions: detection, specification, assessment and consolida-

tion. Detection finds the software technical risks, specification records salient aspects of the identified technical risks, assessment determines the magnitude of each risk by using the formula, and consolidation is merging, combining and abstracting risk data into concise format. The Software Development Risk Taxonomy (SDRT), depicted in Fig. 1, provides a basis for organizing and studying the breadth of software development issues. It serves as a systematic way of eliciting and organizing risks and provides a consistent framework for the risk management method and technique development. The Taxonomy-Based Questionnaire (TBQ) is a tool specifically used for identifying software development risks [1]. Most software development failures could have been avoided or greatly reduced had there been an explicit early concern with identifying and resolving the high-risk elements of the project [8]. The TBQ ensures all potential risk areas are covered and contains specific cues and followup questions that allow the person administering the questionnaire to probe for risks. The SRE Method consists of the four primary functions mentioned previously and three support functions. The support SRE Risk Management Paradigm functions are planning and coordination, verification and validation, and training and communication. The SRE Method utilizes the SDRT and TBQ for early detection to find software technical risk areas. Each risk is recorded along with the risk’s condition, consequence and source. The risk is then classified as belonging to a taxonomy class, element or attribute. The magnitude of each technical risk is determined by multiplying the severity of the impact with the probability of the occurrence. The Risk Magnitude Level Matrix, shown as Fig. 2, is a quick and easy chart to determine the risk magnitude. Finally, based on expert opinion, the risk data are merged, combined and abstracted into a concise decision-making form. 3.2. Boehm’s Software Risk Management model Dr Barry Boehm developed a set of principles and practices for managing the risk of developing software called the risk-analysis paradigm. Boehm’s Software Risk Management model focuses on the concept of ‘‘risk exposure’’ as

T.R. Adler et al. / Information and Software Technology 41 (1999) 29–34

31

Table 1 Top-Ten Risk Identification Checklist Risk item

Risk management techniques

Personnel shortfalls

Staffing with top talent; job matching; team building; morale building; cross-training; prescheduling key people Detailed, multi-source cost and schedule estimation; design-tocost; incremental development; software reuse; requirements scrubbing Organizational analysis; mission analysis; operations concept formulation; user surveys; prototyping; early user’s manual Task analysis; prototyping; scenarios; user characterization (functionality, style, workload) Requirements scrubbing; prototyping; cost/benefit analysis; design-to-cost High change threshold; information hiding; incremental development (defer changes to later increments) Benchmarking; inspections; reference checking; compatibility analysis

Unrealistic schedules and budgets

Fig. 2. Risk Management Level Matrix.

defined by the relationship where the probability of an unsatisfactory outcome and the loss due to the unsatisfactory outcome determine the valence of the risk event [8]. This model uses a decision tree method to identify the software risk items and a top-ten risk identification checklist. The goal of risk management is to reduce the ‘‘risk exposure’’ associated with developing software. In addition to providing unique solutions, the decision tree method provides a framework for analyzing the sensitivity of preferred solutions to the risk-exposure parameters [8]. The Top-Ten Risk Identification Checklist, provided as Table 1, can identify high-priority risk items that affect cost, schedule and quality. The checklist also provides risk management techniques that have proved to be successful in avoiding or resolving each particular source of risk. Boehm explains that a major source of risk is the uncertainty of subjectively estimating the probability or loss associated with an unsatisfactory outcome. One means of reducing this source of risk is buying information early that can give insight into the final product, such as prototyping [9]. The risk exposure factors can be ranked in descending order to draw attention to the most critical areas. The techniques used in the Software Risk Management model are risk assessment and control. Part of risk control is risk management planning which includes buying information and risk avoidance. The method can be used in all phases of software development, but its emphasis, as indicated by items in the Top-Ten Risk Identification Checklist, is early in the development life cycle. 3.3. Best Practices Initiative’s (BPI) risk management emphasis The Program Manager’s Guide to Software Acquisition Best Practices [10] listed formal risk management as the number one activity of nine recommended software acquisition best practices. BPI provides a questionnaire to assist managers in understanding the significant risk issues and the degree to which the best risk management practices are being used on the program. Among the recommendations is the use of cleanroom processes to reduce software development risks [10].

Developing the wrong software functions

Developing the wrong user interface Goldplating

Continuing stream of requirements changes

Shortfalls in externally furnished components Shortfalls in externally performed tasks Real-time performance shortfalls Straining computer science capabilities

Reference checking; pre-award audits; award-fee contracts; competitive design or prototyping; team building Simulation; benchmarking; modeling; prototyping; instrumentation; tuning Technical analysis; cost–benefit analysis; prototyping; reference checking

The BPI Risk Management Method can be employed during all phases of software development; however, it is recommended that it start as early as possible. A questionnaire was developed to help managers understand key issues and the extent to which risk management practices are being employed. BPI emphasizes the use of questions to identify key risk issues. Identifying the risks early is the key to minimizing the effects or avoiding the effects of the risks altogether.

3.4. Team Risk Management (TRM) The TRM model was also developed by SEI and is a means of creating effective communication about software risk so that the risk can be abated or mitigated. The TRM method creates a non-threatening environment where risk is addressed at higher levels of control and actions can be taken to mitigate the risks. The goal is to put into place

32

T.R. Adler et al. / Information and Software Technology 41 (1999) 29–34

Table 2 Team Risk Management principles

Table 3 Software risk management models

Principle

Effective risk management requires

Model

Shared product vision

Sharing a product vision based upon common purpose, shared ownership and collective commitment Focusing on results Working cooperatively to achieve a common goal Pooling talent, skills, and knowledge Viewing software development within the context of the larger systems-level definition, design and development Recognizing both the potential value of opportunity and the potential impact of adverse effects Thinking toward tomorrow, identifying uncertainties, anticipating potential outcomes Managing program resources and activities while anticipating uncertainties Encouraging free-flowing information at and between all program levels Enabling formal, informal and impromptu communication Using consensus-based processes that value the individual voice (bringing unique knowledge and insight to identifying and managing risk) Making risk management an integral and vital part of program management Adapting risk management methods and tools to a program’s infrastructure and culture Sustaining constant vigilance Identifying and managing risks routinely throughout all phases of the program’s life cycle

Technique(s) used

SRE

Teamwork

Global perspective

Forward-looking view

Open communication

Integrated management

Continuous process

effective plans to manage risk before evolving into catastrophic failure. The TRM model has seven self-explanatory principles: shared product vision, teamwork, global perspective, forward-looking view, open communication, integration and continuous process (see Table 2). The TRM Method is a continuous process of improving communication between the customer and the supplier. By doing so, there are multiple perspectives that allow software managers to formulate strategies to mitigate risk based on expert opinion. The team environment brings together risks

Avoidance, control Questionnaires and expert opinion Boehm’s Avoidance, control Questionnaires and expert opinion Best Practices Avoidance, control Questionnaires and expert opinion Team Avoidance, control Questionnaires and expert opinion Cleanroom Avoidance

Life-cycle phase(s) Weaknesses All

All

All

All

All

Upfront costs, time consuming, learning curve

identified in each organization, giving decision-makers a more global perspective. 4. The cleanroom approach The four software risk management models are summarized in Table 3. All of the methods emphasize risk avoidance and risk control and can also be utilized at any phase of the development process, although they all recommend early risk identification. However, a common limitation persists in these risk management models — they require assumptions by qualified experts — assumptions regarding the probability of occurrence and assumptions of the impact of the loss. Assumptions are based on the subjective evaluations of experts since software development is primarily a human endeavour [1, 2, 12]. The subjectivity of experts, however, is itself a source of risk which tends to put risk assessment off into later stages of software development when software is more definable or knowable. One reason for this may be the inherent limitations of human cognition [11, 13]. Another may be the human predisposition to avoid decisions that entail early commitment to a course of action [14]. Yet another reason may be resource constraints that limit the amount of work that can be done, thereby limiting the amount of risk analysis that can be conducted. However, given the importance and central role of software to most new product developments [1, 15], not dedicating resources for risk avoidance early is fraught with danger and relies on the assumption that defects can be discovered and fixed in an efficient manner. One type of risk avoidance strategy is the prototyping method. When early life-cycle issues are of concern, Cerpa and Verner [16] explain that prototyping is the preferred development method because it allows developers and customers to communicate and discover defects.

T.R. Adler et al. / Information and Software Technology 41 (1999) 29–34

33

Fig. 3. Software error propagation cost.

However, by itself, the prototyping method does not guarantee an adequate risk management plan. Researchers have documented possible risks associated with using the prototype method that may even increase the risks of new developments [17, 18]. What is required is a more scientific approach to mitigating software risks. The cleanroom process provides the rigor required to be considered scientific and will reduce the risk of software defects later in the life cycle [4, 10]. Cleanroom processes provide a mathematical proof-of-correctness verification method, as well as employing inspections, reviews, audits and independent verification and validation. Each of these processes can be applied to a well-defined work product such as requirements and design documents, test plans, hardware logic and code to improve the reliability and quality of the software products [19]. By applying these techniques, software developers reduce the risk of latent software defects, lower costs associated with defect removal, and improve the overall reliability and quality of their products. Risk avoidance begins early in the software life cycle with the detection of faults as requirements are translated into specifications. By randomly inspecting specifications, ambiguities and misunderstandings can be identified and managed. According to Gilb ([20], pp. 218–221), 62% of defects occur during the design process and 38% are created during coding. Cleanroom processes, which include inspection, are proactive approaches to ensuring that defects are not allowed to reside in a software program and that they are removed prior to coding or testing. The use of cleanroom processes results in a software product that is less prone to defects, thus increasing the software’s quality and reliability [21]. While Lyu [22] claims that it is possible through exhaustive testing or mathematical proof-of-correctness to remove all defects in the code, it is usually impractical to do so in systems of significant size and complexity. Cleanroom inspections offer a more practical approach than exhaustive testing to eliminate software defects and are conducted earlier in the development life cycle when cost savings are the greatest. Boehm ([23], p. 40) explains that fixing software problems after delivery is 100 times more expensive

than fixing it in the requirements phase. Unfortunately, the traditional approach to software development is one-sided where software risk areas may be identified early in the life cycle but are not truly managed until very late in the life cycle. A common complaint among software developers is that they do not have the time to plan but they have the time to do it over again.

5. The efficacy of using cleanroom processes Roughly 44% of the cost of software development is spent towards reworking code that was developed in an ad hoc or chaotic way ([1], pp. 1–21). The 44% reduction occurs because of the propagation of errors throughout the development process that were created early in the life cycle. Defects that result in rework are one of the most significant sources of risk in terms of cost, schedule delays and poor performance [1] (see Fig. 3). Reducing software development costs and improving the quality of software are two key items that can simultaneously be addressed by utilizing cleanroom with inspections. Utilizing cleanroom processes, with inspections, comes with a cost, but the ratio of savings to costs clearly indicates some potency. The following examples indicate that cleanroom processes do work to reduce costs and increase productivity. IBM’s COBOL Structuring Facility used cleanroom processes to develop an eighty-five-thousand lines of code (KSLOC) product which automatically structures unstructured COBOL programs. In this effort the software developers were able to reduce the average number of errors to 3.4 errors/KSLOC in all testing. Furthermore, the product experienced only seven minor defects in three years of use ([1], p. 15-54; [24]). This is a testimony to the claims of reduced maintenance costs associated with products developed with the cleanroom process. Another example was the Software Engineering Laboratory (SEL) at NASA Goddard. They have completed their second and third cleanroom programs: a twenty KSLOC attitude determination subsystem and a one-hundred-andfifty KSLOC flight dynamics system that have experienced

34

T.R. Adler et al. / Information and Software Technology 41 (1999) 29–34

a combined defect rate of 4.2 defects/KSLOC in all testing ([1], p. 15-55). Dyer ([4], p. 23) reports that the SEL realized a reduction of average error rates from 6 to 3.3 errors/ KSLOC for one of the projects. In addition, Ericsson AB Telecommunications Operating Systems developed a three-hundred-and-fifty KSLOC system for switching computers that resulted in 1.0 defects/KSLOC in all testing. Productivity improvements were reported as 70% and 100% for development and testing respectively ([1], p. 15-54). Ulmer [25] observed that the Unisys corporation realized a fourfold improvement after implementing defect prevention techniques like cleanroom and inspections in software development. When considering maintenance in addition to development, the savings-to-costs ratio will surely increase. The cleanroom process approaches zero defects which results in less testing and subsequently less rework. Researchers have found that cleanroom techniques remove over 90% of software errors before testing even begins in the development process [26]. Gilb [27] also found that defect prevention has a return on investment of 13:1, which allows organizations to fund cleanroom capabilities from savings on the first project.

[2] [3] [4] [5]

[6] [7] [8] [9]

[10] [11] [12] [13] [14]

6. Summary

[15]

Although each of the risk management models and techniques presented have their own merits, the time has come to stop expending large amounts of resources on risk control. Organizations trying to determine the probability, or risk, of software defects typically ignore the processes available to reduce the risks early in development. Risk avoidance techniques, like the cleanroom approach, alleviate many of the constraints of traditional software development. However, cleanroom processes are no panacea for software development. Cleanroom processes provide rigor required for a valid development process. A key point of this paper is that risk avoidance should precede risk elimination. The cleanroom approach is one management alternative to facilitating an early life-cycle focus that improves the software development process.

[16]

References [1] United States Department of the Air Force, Software technology support center’s guidelines for successful acquisition and management of software-intensive systems: weapon systems, command and

[17]

[18] [19] [20] [21] [22] [23] [24]

[25]

[26] [27]

control systems, version 2.0, Management Information Systems, vol. 1, USAF, Washington, DC, 1996, Ch. 1, 6 and 15. W. Humphrey, Managing the Software Process, Addison-Wesley, Reading, MA, 1990. S. Pfleeger, Software Engineering, Macmillan, New York, 1991. M. Dyer, The Cleanroom Approach to Quality Software Development, John Wiley, New York, 1992, pp. 1–25. R.A. DeMillo, W.M. McCracken, R.J. Martin, J.F. Passafiume, Software Testing and Evaluation, Benjamin/Cummings, Menlo Park, CA, 1987. N.C. Siropolis, Small Business Management, 3rd edn, Houghton Mifflin, Boston, MA, 1986. B.W. Boehm, A spiral model of software development and enhancement, IEEE Computer (May 1988) 26–37. B.W. Boehm, Industrial software metrics top 10 list, Quality Time, IEEE Software, 1987, pp. 26–37. G. Tate, J. Verner, Case study of risk management, incremental development and evolutionary prototyping, Information and Software Technology 32 (1990) 207–214. United States Department of Defense, Program Manager’s Guide to Software Acquisition Best Practices, version 2.0, 1997. H. Simon, On the concept of organizational goal, Administrative Science Quarterly 8 (2) (1964) 1–22. L.L. Constantine, Constantine on Peopleware, Yourdon Press, Englewood Cliffs, NJ, 1995. J.G. March and J.P. Olsen, Ambiguity and Choice in Organizations, Universiteforlaget, Bergen, Norway, 1976. J.G. Quinn, Strategies for Change: Logical Incrementalism, Irwin, Homewood, IL, 1980, pp. 97–152. M. Aoyama, Beyond software factories: concurrent-development process and an evolution of software process technology in Japan, Information and Software Technology 38 (1996) 133–143. N. Cerpa, J. Verner, Prototyping: some new results, Information and Software Technology 38 (1996) 743–755. D.R. Graham, Incremental development: review of monolithic lifecycle development models, Information and Software Technology 31 (1991) 7–20. B.H. Boar, Application Prototyping, John Wiley, New York, 1984. G.W. Russell, Experience with inspection in ultra-large-scale developments, IEEE Software (January 1991) 25–31. T. Gilb, Principles of Software Engineering Management, Bath Press, Avon, 1988. D.W. Karolak, D. Walter, Software Engineering Risk Management, IEEE Computer Society Press, Los Alamitos, CA, 1996. M.R. Lyu, Handbook of Software Reliability Engineering, McGraw Hill, New York, 1996. B.W. Boehm, Software Engineering Economics, Prentice-Hall, Englewood Cliffs, NJ, 1981. R.C. Linger, H.D. Miles, Case study in cleanroom software engineering, in: COMPSAC ’88 Proceedings, Computer Society Press of the IEEE, Washington D.C., 1988. D. Ulmer, Life after testing: a Unisys experience, in: Proceedings 8th International Conference on Testing Computer Software, U.S. Professional Development Institute, Washington D.C., 1991. O. Oskarsson, R.L. Glass, An ISO 9000 Approach to Building Quality Software, Prentice Hall, Uppersaddle River, NJ, 1996. T. Gilb, Software Inspection, Addison-Wesley, Reading, MA, 1993.