- Email: [email protected]
Creating a software process improvement program
Creating a software process improvement program B Curtis and M Paulk
The reasons that underlie the emergence o f a software process movement in the mid-1980s are discussed. A brief overview o f the Capability Maturity Model for Software developed at the Software Engineering Institute is provided. The article then describes how this model can be used to guide software process improvement programs. Some components o f such programs are describe software process improvement, process assessment, capability maturity model
THE SOFTWARE
PROCESS
MOVEMENT
The software engineering community is learning, as has virtually every other area of engineering, that advances in productivity and quality do not materialize just because technology was thrown at a problem. Rather, improving the results of software development or maintenance operations requires attention to the processes by which these operations are performed. The growing functionality required from many systems has resulted in an exponential growth in the software required to run them 1. Reports of system growth from many organizations indicate that the amount of software embedded in many products may be growing by an order of magnitude each decade. The processes used to build and maintain small systems do not scale up for systems that are an order of magnitude larger. The management processes that work with a small team are insufficient for coordinating multiple teams, divisions, or companies. Thus, the growing size of projects, and a cultural tradition rooted in the prowess of the individual programmer, have stressed our ability to effectively manage software development and maintenance. A software process movement emerged in the mid1980s when shortcomings in managing development and maintenance processes were recognized as prime inhibitors of growth in software productivity and quality. Software organizations are finding that they should first focus on defining the processes used in their software business, and only then select tools and methods to support these processes. Thus, when used to integrate Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA 15213-3890, USA Vol 35 No 6/7 June/July 1993
people, tasks, tools, and methods, a well-defined and documented software process can provide an underlying foundation for long-term productivity and quality growth. If the maturity of an organization's software process influences its ability to meet cost, quality, and schedule targets 2, then it is wise to understand what distinguishes mature from immature organizations. In an immature software organization, software processes are generally improvised by practitioners and their management during the course of the project. Even when a software process such as peer reviews or quality assurance has been specified, it is not followed or enforced. One of the most damaging characteristics of immature organizations is that unrealistic commitments are made by managers without consulting with the software staff on whether these commitments can be met. Schedules and budgets are then routinely exceeded, because they are not based on realistic estimates that a sound engineering process can support. When hard deadlines are imposed, product functionality is often compromised, and engineering disciplines such as reviews and testing are often curtailed or eliminated in order to meet the schedule. The immature software organization is reactionary and managers are usually focused on fighting the fires that a more mature process might have prevented. When success is obtained--and an occasional striking success can emerge from an immature organization--it is achieved through the heroic efforts of talented individuals. Repeating this success is solely dependent on getting the same individuals appointed to the next project. Thus, in immature organizations it is individuals rather than the organization that carry the capability for success. The organization has not created a mechanism for learning from previous project experience, and does not have the ability to exercise effective project control. In such environments it is not surprising that we find individual talent being the best predictor of productivity and quality. The products of immature organizations are often unmistakable. They are riddled with errors that the project team did not have time to eliminate before shipping the product. The documentation is incomplete and often has not been updated to the current state of the released configuration. There is no historical trail of how the product was developed and why design and implementation decisions were made as they were. Trying to find the right person to ask a question of is
0950-5849/93/060381-06© 1993 Butterworth-Heinemann Ltd
381
Creating a software process improvement program difficult because there is no trace of who did what, and any trace that might exist is wrong because no one kept track of changes in assignments. These are exactly the conditions that make software maintenance so hard. If one wants to read the stuff of which software maintenance processes are defined, then one should read a detective novel. The lack of a defined development process leaves maintenance engineers to hunt for shards of evidence of what occurred long after the original witnesses have fled the scene. In an immature organization the maintenance engineer suffers every bit as much as the customer. On the other hand, a mature software organization possesses an institutionalized capability for managing software development and maintenance processes. A defined, organization-wide software process is accurately communicated to both existing staff and new employees, and work activities are carried out and monitored according to a plan. The defined processes are trained and are consistent with the way the work actually gets done. These defined processes are updated when necessary, and improvements are developed through controlled pilot-tests and/or cost benefit analyses. Roles and responsibilities within the defined process are clear throughout the project and to others in the organization. In a mature organization, managers monitor the quality of the software product quantitatively and ensure the customer is satisfied. There is an objective--usually quantitative--basis for judging quality and analysing problems with both the product and process. Schedules and budgets are based on historical performance and are realistic; the predicted results for product cost, schedule, functionality, and quality are routinely achieved. Based on the predictability of project results, reliable management decisions can be made about tradeoffs among these outcomes. When a software product is unprecedented in a mature organization, project management and staff know how to alter their standard software process in an orderly way to establish a basis for effective management planning, control, and decision-making. A mature process is designed to support processes that occur long after the software is initially delivered. Maintenance is treated as a natural continuation of the initial development process, and maintenance engineers are treated as customers of the processes that were performed earlier in the life-cycle. Thus, much of the return on investment for mature development processes is realized in reduced costs for product maintenance and evolution. The maintenance activity itself is subject to the same disciplined processes that have received primary attention in the development world. A disciplined process has the ability to blend development and maintenance indistinguishably into a cycle of product evolution. Capitalizing on these observations about differences between immature and mature software organizations requires construction of a process maturity framework. This framework must describe an evolutionary path from ad hoc, chaotic processes to a mature, disciplined software process. This framework is important because 382
improvement goals must be set in stages that represent a sustainable growth path from an undisciplined, immature state to a mature, institutionalized software process. THE
CAPABILITY
MATURITY
MODEL
One model of process maturity that has received extensive recent attention, and has been adopted by a large segment of the software engineering community, is the Capability Maturity Model (CMM) for software3:. This model provides software organizations with guidance on how to gain control of their process for developing and maintaining software and how to evolve their environment toward a culture of software engineering excellence. The CMM was designed to guide software organizations in selecting process improvement strategies based on the current maturity of their organization's software process. The CMM's benefit is in narrowing the scope of improvement activities to those software processes that provide the next foundational layer for the organization's growth. By focusing on a limited set of activities and working aggressively to achieve them, organizations can steadily improve their software process to enable continuous and lasting gains in software project performance. The staged structure of the CMM is based on product quality principles that have been evolved by W. Edwards Deming 5, Joseph Juran 6, and others over the last 60 years. The maturity framework into which the principles underlying the CMM have been adapted was first elaborated by Philip Crosby 7 in his book Quality is free. Crosby's quality management maturity grid describes five evolutionary stages in adopting quality practices. This framework was adapted to the software process by Ron Radice and his colleagues8 working under the direction of Watts Humphrey at IBM. Humphrey brought this framework to the Software Engineering Institute in 1986 and developed the foundation for its current use throughout the software industry. The CMM provides a framework for identifying five levels of maturity that lay successive foundations for continuous process improvement. These five maturity levels define an ordinal scale for measuring the maturity of an organization's software development capability. Each maturity level is a well-defined evolutionary plateau on the path toward becoming an exceptional software organization. Each level comprises a set of process goals that, when satisfied, make a significant addition to the sophistication and capability of an organization's software process. The five levels of the CMM and the key process areas constituting them are shown in Figure 1. The Initial maturity level (level 1) is aptly characterized by the description of an immature software development organization provided in the previous section. Maturity levels 2 through 5 can be characterized through the activities performed by the organization to establish or improve their software process. These levels will be described in a way that makes it clear how the key Information and Software Technology
B CURTIS AND M PAULK
PI OCOS.~ A r e a ~.
Continuous process Improvement
Optimizing
Managed
Technology innovation Process change management
Product and process quality
Process measurement and analysis Quality management
Engineering process
Organization process focus Organization process defn. Peer reviews Training program Intergroup coordination Software product engineering Integrated software mgt. Software project planning Software project tracking Software subcontract mgt. Software quality assurance Software configuration mgt. Requirements mgt.
Defined Project management
Repeatable 1
Defect prevention
Heros
Initial Figure I. Capability maturity model for software practices at each level apply to maintenance processes as readily as they apply to development. Level 2: the Repeatable level--The primary objective in achieving level 2 is to stabilize project commitment and management control processes. Project management policies and procedures must be established by the organization, and an audit function independent of project management (i.e., a quality assurance function) must determine that they are being practised. Planning and management of new projects or releases is based on experience with similar projects or previous releases. Project managers track software costs, schedules, and functionality. Problems in meeting commitments are identified when they arise. Software requirements and the artefacts developed or modified to satisfy them are baselined, and their integrity is controlled. Project teams work with their customers, and their subcontractors if any, to establish a stable, managed working environment. Realistic resource and schedule commitments are established from results observed on previous projects that allow project staff to repeat successful practices developed during earlier projects or releases. The six key process areas of level 2 constitute a basic software management discipline that is as relevant to maintenance activities as it is to new development. Thus, even if an organization has no new development efforts underway, it should embrace level 2 practices as a way to get its software business under basic management control. It is this level of control that begins to give customers confidence in the predicted date for a scheduled release. Level 3: the Defined level--At the defined level an organization-wide standard software process is defined and documented, including both software engineering and software project management processes. Projects use the organization's software process as a basis for designing a process tailored to each project, or release's unique characteristics. A defined process is enabled because the management discipline established at level 2 ensures that sound engineering principles will not be abandoned even under stress. Vol 35 No 6/7 June/July 1993
Having a defined process allows management and engineering activities to be coherently integrated on each project. A software engineering process group is established to facilitate process definition and improvement activities. Each project institutes a peer review process to enhance product quality. An organization-wide training program is implemented to ensure that all engineers and managers have the knowledge and skills required to carry out their tasks using the tools and methods they have selected. The capability established at level 3 is based on a common understanding of processes, roles, and responsibilities in a defined process that allows all project personnel to coordinate their activities more effectively and work from a common understanding of their mutual dependencies. Sound maintenance engineering should be guided by the same level of rigorously defined processes that characterize development. That is, the organization should have a defined process for going about the selection of enhancements, the definition of modifications to the existing product, the implementation of these modifications, the testing of the product for correctness, the updating and preparation of all artefacts supporting the product whether released with it or not, and the actual process of release and installation with the customer. This process communicates to both customers and the staff how the organization conducts business. Level 4: the Managed level--At the managed level the organization sets quantitative quality goals for software products. In addition, productivity and quality are measured for important software process activities across all projects in the organization. These fine-grained measures would be impossible to interpret effectively without establishing the carefully defined software process built at level 3. These measures establish the quantitative foundation for evaluating project processes (e.g., defect removal efficiency of reviews at different life-cycle stages) and products (e.g., defects per thousand delivered source instructions during the first year following release). Projects achieve control over their products and processes by narrowing the variation in their performance to within acceptable quantitative boundaries. Thus, it is at level 4 that the organization begins to apply the techniques of statistical quality control to upgrading the quality of the software delivered to customers. When building software in unfamiliar application domains, the risks involved in moving up the learning curve during development are known and carefully managed. Not surprisingly, the organization is better able to achieve desired results on subsequent releases as it moves off the learning curve for the new application. The organization is then better able to predict process and product quality trends within established quantitative bounds. When these bounds are violated, management action is taken to correct the situation. Maintenance organizations have the advantage of observing the data on a project over multiple releases. As the organization learns much more about the application it is fielding, it has an opportunity to begin stabilizing the defect rates for a product. That is, as the organization 383
Creating a software process improvement program
moves up the learning curve for a product, it has an opportunity to remove many sources of variation that contributed noise into the product's quality. This provides an opportunity to move the product under statistical quality control. Thus the key practices constituting level 4 may have their most effective implementation in maintenance operations. Level 5: the Optimizing level--At the optimizing level the organization focuses on continuous process improvement. Based on the quantitative capabilities established at level 4, it can identify weak process elements and strengthen them, with the goal of enhancing quality. Empirical evidence is available on process effectiveness and is used to perform cost/benefit analyses on new technologies. Technologies with the greatest payoff for the specific processes used by the organization (as per their defined process) are selected for a managed introduction into their organization. Although the software process is mature, it is a target for continuous enhancement. Project teams analyse process performance to determine the causes of defects. Software processes are evaluated to prevent known types of defects from recurring, and the lessons learned are disseminated to other projects and future releases. An environment of software engineering excellence has been achieved. Maturity levels 4 and 5 are unknown territory for the software industry in 19929. There are a few examples of projects with level 4 and 5 characteristics, but no level 4 and 5 software organizations have been observed. Characteristics of level 2 and 3 organizations have been established through extensive assessment of existing software development organizations. The characteristics of organizations at levels 4 and 5, however, have been defined by analogy with other industries and by anticipating the next most likely steps to be taken by the most advanced level 3 organizations. Additional characteristics may be added to describe level 4 and 5 organizations as we study the few level 3 organizations currently on the verge of becoming level 4. SOFTWARE PROGRAMS
PROCESS
IMPROVEMENT
Because many of the most severe problems facing the software industry involve management of the software development and maintenance process, process improvement programs have been designed to improve software productivity and quality through attacking these problems. There are several principles that must be observed in initiating these programs. (1) Commitment to improve must start at the top. Without visible executive commitment, the improvement effort is destined to fail. This commitment takes numerous forms. First, it requires adequate funding for the resources to support the program. Second, it requires a willingness to reinforce managers and staff for process improvements, and not just for delivered products. Middle management will not support the program if they believe that the executive office will
384
not assess their performance at least in part based on accomplishing improvement objectives. Finally, executives need to provide visible leadership to feed the motivation to improve. Executive support must be solid over several years, and must be robust even through the inevitable changes in managers. (2) Focus on the process, not the people. After a disastrous project there is usually a search for the guilty, resulting in the summary execution of the responsible manager. The staff are often afraid that software measurements will be used for performance appraisals. These problems create an organization that works on the basis of fear. In such organizations, management and staff are unwilling to be truthful about problems, and are constantly assigning blame to someone else. No process improvement program has a chance of succeeding in this type of environment. A hallmark of successful process improvement programs is that people were not afraid of dealing with facts because the emphasis was on improving the organization rather than on finding guilt. This openness is crucial in achieving the kind of consensus needed to tackle systemic problems like immature processes. (3) First understand the current process. The root causes of problems cannot be found unless the process from which they resulted is understood. Further, an organization cannot agree on which problems to solve first unless they reach consensus on what the fundamental problems are. Thus, a process improvement program must begin with an assessment of the current state of an organization's software development processes. The organization must understand both its process strengths and weaknesses, because it must build upon the former to improve the latter. Analysing the process has the advantage of putting the attention on the process and not the people as the root cause of problems. (4) Change must become a way of life. The improvement process is not like getting over an illness. Once over an illness you go back to things as normal. Software process improvement is a never-ending activity where each improvement is used as the basis for making the next improvement. There is a danger in believing that once an organization has achieved a particular level in its maturation that it can rest and go on to other issues. Unfortunately a highly competitive software environment will not allow software developers the luxury of resting long on its accomplishments. Constant improvement is the only way to maintain a competitive advantage in the marketplace. (5) Improvement requires investment. Improvement programs fail when people are told to perform improvement activities on top of other responsibilities from which there is no relief. Process improvement requires dedicated resources and guidance that can maintain the organization's focus and vision on improvement. Someone must have primary responsibility for executing the improvement effort. They must be supported by people who can dedicate some Information and Software Technology
B CURTIS AND M PAULK
portion of their time to contributing to the change process. The investment in improvement must be multi-year if the program is to have a chance to institutionalize its improvements and show their benefits on the bottom line. A high level overview of a method for conducting process improvement programs is presented in Figure 2. Not surprisingly, the first step is to build executive support. Until this support is solid, no large-scale improvement program should be undertaken. Executives musl be briefed on the need for improvement, preferably using data from the organization's own development experience. For instance, executives are much more likely to commit funds for improvement if they can see how improved software quality will reduce their field service costs, achieve required reliability goals set by customers, shorten time to market, improve the predictability of project costs, or accomplish some other goal that is clearly related to their business objectives or the bottom line. They must also be able to envision the organization they are trying to create, so they can link process improvements to desired organizational changes. Once executive commitment has been established, then an organizational infrastructure must be set up to manage the improvement program and support the change process. This infrastructure starts with the creation of a management steering committee. This commitment is drawn from both senior organizational management and project management. The responsibilities of this group are to set the business objectives to which the improvement program must respond. This committee also charters the improvement group (the SEI calls these Software Process Improvement Groups, SEPGs). The SEPG consists of a small number (1% to 3% of an organization's technical staff) of respected software professionals who are dedicated full time to executing the improvement program. They will charter the assessment and manage the resulting improvement activities. The SEPG will also convene technical working groups to work on specific changes. These working groups consist of project personnel who are allowed to dedicate a portion of their time to defining new software processes and practices and to installing these back on their own projects. These working groups bring a sense of realism on the amount of change an organization
Build executive II support II
I I
Implement action plan
Build Improvement infm~h'uctum
Am the organization's
Figure 2. Process improvement cycle Vol 35 No 6/7 June/July 1993
Develop Improvement
Figure 3. Process improvement infrastructure can reasonably absorb at one time. A model of this infrastructure is presented in Figure 3. When the infrastructure has been set in place, the SEPG charters a software process assessment. The assessment has three purposes. First, it gathers data about the process issues facing the organization. This is the raw input from which the improvement program will be designed. Second, it builds consensus among the software staff and management concerning the issues and the priority with which they should be addressed. Finally, it peaks the motivation for launching an improvement program. Thus, the timing of an assessment is important, in that the organization must be prepared to move quickly into action planning once the recommendations have been made. The SEI software process assessment method has been designed to achieve these three objectives through an extensive preparation period followed by five intense days on site where data are gathered, reviewed through the organization, and finally presented to senior management. The assessment findings and recommendations become the raw input into the improvement planning process. Improvement planning must be executed with the same level of thoroughness and rigour that the improvement team is trying to install in the software development process. The SEPG develops a strategic plan that responds to the business objectives of the management steering committee. Once this strategic plan has been approved, a technical working group is convened for each area of improvement and they further develop the tactical plans by which each major process will be improved across the organization. These tactical plans are then further decomposed into action plans for how each project will implement a change. The action plans must match improvement objectives with a realistic assessment of how much change a project can absorb. The organization may choose to run a pilot implementation before going to an organization-wide rollout of a new process. These decisions are why the planning process is so important in improvement programs. The implementation phase is aided by two techniques that serve process improvement, process definition and measurement. Technical working groups must define the processes they want to install, and these descriptions
385
Creating a software process improvement program
must be understandable by those who execute the software process. A well-defined process provides everyone in the organization with an understanding of their responsibilities, of their mutual dependencies, and of how the organization intends to conduct its business. Once improved processes have been defined, they should be measured. Only through measurement can the feedback loops so necessary for learning be installed. Measurements will also be needed to present the case for continued funding after the initial excitement has worn off and the organization faces contention for the funding that is in such short supply. The SEI is currently piloting methods for accomplishing both of these objectives. Throughout the improvement program, the organization is managing not only technical change, but also behavioural change. SEPG members can benefit from training in how to be internal consultants and manage technological change. Managing the social aspects of the change process is often the most difficult, because established patterns may need to be modified. Typically we find organizations chartering a reassessment every two years; but there is no magic in this number. The crucial issue is that the improvement process must be considered a repeating cycle.
CONCLUSION When managed properly, improvement programs can yield decided benefits to the sponsoring organization. Using the Capability Maturity Model assists programs in deciding what to do next among the myriad opportunities for improvement. Data from Hughes Aircraft%
386
Raytheon 1°, and Tinker Air Force Base H suggest that the expected paybacks from well-run improvement programs tend to be in the range of 5 : 1 to 9:1 for the return on investment. This paper provides a very brief overview, and those wanting to launch an improvement program should seek the assistance of those with experience in running them successfully.
REFERENCES 1 Boehm, B 'Increasing software productivity' IEEE Computer Vol 29 No 9 (1987) pp 43-57 2 Humphrey, W S, Snyder, T R, and Willis, R R 'Software process improvement at Hughes Aircraft' IEEE Software Vol 8 No 4 (1991) pp 11-23 3 Humphrey, W S Managing the software process AddisonWesley (1989) 4 Panik, M C, Curtis, B, Chrissis, M B, Averill, E L, Bamberger, J, Kasse, T C, Konrad, M, Perdue, J R, Weber, C V and Withey, J V The capability maturity model for software (CMU/SEI-91-TR-24) Software Engineering Institute, Carnegie Mellon University (1991) 5 Deming, W E Out of crisis MIT Center for Advanced Engineering Study (1982) 6 Juran, J M Juran on leadershipfor quafity Free Press (1989) 7 Crosby, P Quafity is free McGraw-Hill (1979) 8 Radice, R A, Roth, N K, O'Hara, A C and Ciarfeila, W A 'A programming process architecture' IBM Systems J. Vol 24 No 2 (1985) pp 79-90 9 Kitson, D H and Masters, S An analysis of SEI software process assessment results: 1987-1991 (CMU/SEI-92TR-24) Software Engineering Institute, Carnegie Mellon University (1992) 10 Dion, R IEEE Software Vol 9 No 3 (1992) 11 Lipke, W H and Butler, K L 'Software process improvement: a success story' Crosstalk: J. Defense Soft. Eng. No 38 (1992) pp 29-31
Information and Software Technology
The reasons that underlie the emergence o f a software process movement in the mid-1980s are discussed. A brief overview o f the Capability Maturity Model for Software developed at the Software Engineering Institute is provided. The article then describes how this model can be used to guide software process improvement programs. Some components o f such programs are describe software process improvement, process assessment, capability maturity model
THE SOFTWARE
PROCESS
MOVEMENT
The software engineering community is learning, as has virtually every other area of engineering, that advances in productivity and quality do not materialize just because technology was thrown at a problem. Rather, improving the results of software development or maintenance operations requires attention to the processes by which these operations are performed. The growing functionality required from many systems has resulted in an exponential growth in the software required to run them 1. Reports of system growth from many organizations indicate that the amount of software embedded in many products may be growing by an order of magnitude each decade. The processes used to build and maintain small systems do not scale up for systems that are an order of magnitude larger. The management processes that work with a small team are insufficient for coordinating multiple teams, divisions, or companies. Thus, the growing size of projects, and a cultural tradition rooted in the prowess of the individual programmer, have stressed our ability to effectively manage software development and maintenance. A software process movement emerged in the mid1980s when shortcomings in managing development and maintenance processes were recognized as prime inhibitors of growth in software productivity and quality. Software organizations are finding that they should first focus on defining the processes used in their software business, and only then select tools and methods to support these processes. Thus, when used to integrate Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA 15213-3890, USA Vol 35 No 6/7 June/July 1993
people, tasks, tools, and methods, a well-defined and documented software process can provide an underlying foundation for long-term productivity and quality growth. If the maturity of an organization's software process influences its ability to meet cost, quality, and schedule targets 2, then it is wise to understand what distinguishes mature from immature organizations. In an immature software organization, software processes are generally improvised by practitioners and their management during the course of the project. Even when a software process such as peer reviews or quality assurance has been specified, it is not followed or enforced. One of the most damaging characteristics of immature organizations is that unrealistic commitments are made by managers without consulting with the software staff on whether these commitments can be met. Schedules and budgets are then routinely exceeded, because they are not based on realistic estimates that a sound engineering process can support. When hard deadlines are imposed, product functionality is often compromised, and engineering disciplines such as reviews and testing are often curtailed or eliminated in order to meet the schedule. The immature software organization is reactionary and managers are usually focused on fighting the fires that a more mature process might have prevented. When success is obtained--and an occasional striking success can emerge from an immature organization--it is achieved through the heroic efforts of talented individuals. Repeating this success is solely dependent on getting the same individuals appointed to the next project. Thus, in immature organizations it is individuals rather than the organization that carry the capability for success. The organization has not created a mechanism for learning from previous project experience, and does not have the ability to exercise effective project control. In such environments it is not surprising that we find individual talent being the best predictor of productivity and quality. The products of immature organizations are often unmistakable. They are riddled with errors that the project team did not have time to eliminate before shipping the product. The documentation is incomplete and often has not been updated to the current state of the released configuration. There is no historical trail of how the product was developed and why design and implementation decisions were made as they were. Trying to find the right person to ask a question of is
0950-5849/93/060381-06© 1993 Butterworth-Heinemann Ltd
381
Creating a software process improvement program difficult because there is no trace of who did what, and any trace that might exist is wrong because no one kept track of changes in assignments. These are exactly the conditions that make software maintenance so hard. If one wants to read the stuff of which software maintenance processes are defined, then one should read a detective novel. The lack of a defined development process leaves maintenance engineers to hunt for shards of evidence of what occurred long after the original witnesses have fled the scene. In an immature organization the maintenance engineer suffers every bit as much as the customer. On the other hand, a mature software organization possesses an institutionalized capability for managing software development and maintenance processes. A defined, organization-wide software process is accurately communicated to both existing staff and new employees, and work activities are carried out and monitored according to a plan. The defined processes are trained and are consistent with the way the work actually gets done. These defined processes are updated when necessary, and improvements are developed through controlled pilot-tests and/or cost benefit analyses. Roles and responsibilities within the defined process are clear throughout the project and to others in the organization. In a mature organization, managers monitor the quality of the software product quantitatively and ensure the customer is satisfied. There is an objective--usually quantitative--basis for judging quality and analysing problems with both the product and process. Schedules and budgets are based on historical performance and are realistic; the predicted results for product cost, schedule, functionality, and quality are routinely achieved. Based on the predictability of project results, reliable management decisions can be made about tradeoffs among these outcomes. When a software product is unprecedented in a mature organization, project management and staff know how to alter their standard software process in an orderly way to establish a basis for effective management planning, control, and decision-making. A mature process is designed to support processes that occur long after the software is initially delivered. Maintenance is treated as a natural continuation of the initial development process, and maintenance engineers are treated as customers of the processes that were performed earlier in the life-cycle. Thus, much of the return on investment for mature development processes is realized in reduced costs for product maintenance and evolution. The maintenance activity itself is subject to the same disciplined processes that have received primary attention in the development world. A disciplined process has the ability to blend development and maintenance indistinguishably into a cycle of product evolution. Capitalizing on these observations about differences between immature and mature software organizations requires construction of a process maturity framework. This framework must describe an evolutionary path from ad hoc, chaotic processes to a mature, disciplined software process. This framework is important because 382
improvement goals must be set in stages that represent a sustainable growth path from an undisciplined, immature state to a mature, institutionalized software process. THE
CAPABILITY
MATURITY
MODEL
One model of process maturity that has received extensive recent attention, and has been adopted by a large segment of the software engineering community, is the Capability Maturity Model (CMM) for software3:. This model provides software organizations with guidance on how to gain control of their process for developing and maintaining software and how to evolve their environment toward a culture of software engineering excellence. The CMM was designed to guide software organizations in selecting process improvement strategies based on the current maturity of their organization's software process. The CMM's benefit is in narrowing the scope of improvement activities to those software processes that provide the next foundational layer for the organization's growth. By focusing on a limited set of activities and working aggressively to achieve them, organizations can steadily improve their software process to enable continuous and lasting gains in software project performance. The staged structure of the CMM is based on product quality principles that have been evolved by W. Edwards Deming 5, Joseph Juran 6, and others over the last 60 years. The maturity framework into which the principles underlying the CMM have been adapted was first elaborated by Philip Crosby 7 in his book Quality is free. Crosby's quality management maturity grid describes five evolutionary stages in adopting quality practices. This framework was adapted to the software process by Ron Radice and his colleagues8 working under the direction of Watts Humphrey at IBM. Humphrey brought this framework to the Software Engineering Institute in 1986 and developed the foundation for its current use throughout the software industry. The CMM provides a framework for identifying five levels of maturity that lay successive foundations for continuous process improvement. These five maturity levels define an ordinal scale for measuring the maturity of an organization's software development capability. Each maturity level is a well-defined evolutionary plateau on the path toward becoming an exceptional software organization. Each level comprises a set of process goals that, when satisfied, make a significant addition to the sophistication and capability of an organization's software process. The five levels of the CMM and the key process areas constituting them are shown in Figure 1. The Initial maturity level (level 1) is aptly characterized by the description of an immature software development organization provided in the previous section. Maturity levels 2 through 5 can be characterized through the activities performed by the organization to establish or improve their software process. These levels will be described in a way that makes it clear how the key Information and Software Technology
B CURTIS AND M PAULK
PI OCOS.~ A r e a ~.
Continuous process Improvement
Optimizing
Managed
Technology innovation Process change management
Product and process quality
Process measurement and analysis Quality management
Engineering process
Organization process focus Organization process defn. Peer reviews Training program Intergroup coordination Software product engineering Integrated software mgt. Software project planning Software project tracking Software subcontract mgt. Software quality assurance Software configuration mgt. Requirements mgt.
Defined Project management
Repeatable 1
Defect prevention
Heros
Initial Figure I. Capability maturity model for software practices at each level apply to maintenance processes as readily as they apply to development. Level 2: the Repeatable level--The primary objective in achieving level 2 is to stabilize project commitment and management control processes. Project management policies and procedures must be established by the organization, and an audit function independent of project management (i.e., a quality assurance function) must determine that they are being practised. Planning and management of new projects or releases is based on experience with similar projects or previous releases. Project managers track software costs, schedules, and functionality. Problems in meeting commitments are identified when they arise. Software requirements and the artefacts developed or modified to satisfy them are baselined, and their integrity is controlled. Project teams work with their customers, and their subcontractors if any, to establish a stable, managed working environment. Realistic resource and schedule commitments are established from results observed on previous projects that allow project staff to repeat successful practices developed during earlier projects or releases. The six key process areas of level 2 constitute a basic software management discipline that is as relevant to maintenance activities as it is to new development. Thus, even if an organization has no new development efforts underway, it should embrace level 2 practices as a way to get its software business under basic management control. It is this level of control that begins to give customers confidence in the predicted date for a scheduled release. Level 3: the Defined level--At the defined level an organization-wide standard software process is defined and documented, including both software engineering and software project management processes. Projects use the organization's software process as a basis for designing a process tailored to each project, or release's unique characteristics. A defined process is enabled because the management discipline established at level 2 ensures that sound engineering principles will not be abandoned even under stress. Vol 35 No 6/7 June/July 1993
Having a defined process allows management and engineering activities to be coherently integrated on each project. A software engineering process group is established to facilitate process definition and improvement activities. Each project institutes a peer review process to enhance product quality. An organization-wide training program is implemented to ensure that all engineers and managers have the knowledge and skills required to carry out their tasks using the tools and methods they have selected. The capability established at level 3 is based on a common understanding of processes, roles, and responsibilities in a defined process that allows all project personnel to coordinate their activities more effectively and work from a common understanding of their mutual dependencies. Sound maintenance engineering should be guided by the same level of rigorously defined processes that characterize development. That is, the organization should have a defined process for going about the selection of enhancements, the definition of modifications to the existing product, the implementation of these modifications, the testing of the product for correctness, the updating and preparation of all artefacts supporting the product whether released with it or not, and the actual process of release and installation with the customer. This process communicates to both customers and the staff how the organization conducts business. Level 4: the Managed level--At the managed level the organization sets quantitative quality goals for software products. In addition, productivity and quality are measured for important software process activities across all projects in the organization. These fine-grained measures would be impossible to interpret effectively without establishing the carefully defined software process built at level 3. These measures establish the quantitative foundation for evaluating project processes (e.g., defect removal efficiency of reviews at different life-cycle stages) and products (e.g., defects per thousand delivered source instructions during the first year following release). Projects achieve control over their products and processes by narrowing the variation in their performance to within acceptable quantitative boundaries. Thus, it is at level 4 that the organization begins to apply the techniques of statistical quality control to upgrading the quality of the software delivered to customers. When building software in unfamiliar application domains, the risks involved in moving up the learning curve during development are known and carefully managed. Not surprisingly, the organization is better able to achieve desired results on subsequent releases as it moves off the learning curve for the new application. The organization is then better able to predict process and product quality trends within established quantitative bounds. When these bounds are violated, management action is taken to correct the situation. Maintenance organizations have the advantage of observing the data on a project over multiple releases. As the organization learns much more about the application it is fielding, it has an opportunity to begin stabilizing the defect rates for a product. That is, as the organization 383
Creating a software process improvement program
moves up the learning curve for a product, it has an opportunity to remove many sources of variation that contributed noise into the product's quality. This provides an opportunity to move the product under statistical quality control. Thus the key practices constituting level 4 may have their most effective implementation in maintenance operations. Level 5: the Optimizing level--At the optimizing level the organization focuses on continuous process improvement. Based on the quantitative capabilities established at level 4, it can identify weak process elements and strengthen them, with the goal of enhancing quality. Empirical evidence is available on process effectiveness and is used to perform cost/benefit analyses on new technologies. Technologies with the greatest payoff for the specific processes used by the organization (as per their defined process) are selected for a managed introduction into their organization. Although the software process is mature, it is a target for continuous enhancement. Project teams analyse process performance to determine the causes of defects. Software processes are evaluated to prevent known types of defects from recurring, and the lessons learned are disseminated to other projects and future releases. An environment of software engineering excellence has been achieved. Maturity levels 4 and 5 are unknown territory for the software industry in 19929. There are a few examples of projects with level 4 and 5 characteristics, but no level 4 and 5 software organizations have been observed. Characteristics of level 2 and 3 organizations have been established through extensive assessment of existing software development organizations. The characteristics of organizations at levels 4 and 5, however, have been defined by analogy with other industries and by anticipating the next most likely steps to be taken by the most advanced level 3 organizations. Additional characteristics may be added to describe level 4 and 5 organizations as we study the few level 3 organizations currently on the verge of becoming level 4. SOFTWARE PROGRAMS
PROCESS
IMPROVEMENT
Because many of the most severe problems facing the software industry involve management of the software development and maintenance process, process improvement programs have been designed to improve software productivity and quality through attacking these problems. There are several principles that must be observed in initiating these programs. (1) Commitment to improve must start at the top. Without visible executive commitment, the improvement effort is destined to fail. This commitment takes numerous forms. First, it requires adequate funding for the resources to support the program. Second, it requires a willingness to reinforce managers and staff for process improvements, and not just for delivered products. Middle management will not support the program if they believe that the executive office will
384
not assess their performance at least in part based on accomplishing improvement objectives. Finally, executives need to provide visible leadership to feed the motivation to improve. Executive support must be solid over several years, and must be robust even through the inevitable changes in managers. (2) Focus on the process, not the people. After a disastrous project there is usually a search for the guilty, resulting in the summary execution of the responsible manager. The staff are often afraid that software measurements will be used for performance appraisals. These problems create an organization that works on the basis of fear. In such organizations, management and staff are unwilling to be truthful about problems, and are constantly assigning blame to someone else. No process improvement program has a chance of succeeding in this type of environment. A hallmark of successful process improvement programs is that people were not afraid of dealing with facts because the emphasis was on improving the organization rather than on finding guilt. This openness is crucial in achieving the kind of consensus needed to tackle systemic problems like immature processes. (3) First understand the current process. The root causes of problems cannot be found unless the process from which they resulted is understood. Further, an organization cannot agree on which problems to solve first unless they reach consensus on what the fundamental problems are. Thus, a process improvement program must begin with an assessment of the current state of an organization's software development processes. The organization must understand both its process strengths and weaknesses, because it must build upon the former to improve the latter. Analysing the process has the advantage of putting the attention on the process and not the people as the root cause of problems. (4) Change must become a way of life. The improvement process is not like getting over an illness. Once over an illness you go back to things as normal. Software process improvement is a never-ending activity where each improvement is used as the basis for making the next improvement. There is a danger in believing that once an organization has achieved a particular level in its maturation that it can rest and go on to other issues. Unfortunately a highly competitive software environment will not allow software developers the luxury of resting long on its accomplishments. Constant improvement is the only way to maintain a competitive advantage in the marketplace. (5) Improvement requires investment. Improvement programs fail when people are told to perform improvement activities on top of other responsibilities from which there is no relief. Process improvement requires dedicated resources and guidance that can maintain the organization's focus and vision on improvement. Someone must have primary responsibility for executing the improvement effort. They must be supported by people who can dedicate some Information and Software Technology
B CURTIS AND M PAULK
portion of their time to contributing to the change process. The investment in improvement must be multi-year if the program is to have a chance to institutionalize its improvements and show their benefits on the bottom line. A high level overview of a method for conducting process improvement programs is presented in Figure 2. Not surprisingly, the first step is to build executive support. Until this support is solid, no large-scale improvement program should be undertaken. Executives musl be briefed on the need for improvement, preferably using data from the organization's own development experience. For instance, executives are much more likely to commit funds for improvement if they can see how improved software quality will reduce their field service costs, achieve required reliability goals set by customers, shorten time to market, improve the predictability of project costs, or accomplish some other goal that is clearly related to their business objectives or the bottom line. They must also be able to envision the organization they are trying to create, so they can link process improvements to desired organizational changes. Once executive commitment has been established, then an organizational infrastructure must be set up to manage the improvement program and support the change process. This infrastructure starts with the creation of a management steering committee. This commitment is drawn from both senior organizational management and project management. The responsibilities of this group are to set the business objectives to which the improvement program must respond. This committee also charters the improvement group (the SEI calls these Software Process Improvement Groups, SEPGs). The SEPG consists of a small number (1% to 3% of an organization's technical staff) of respected software professionals who are dedicated full time to executing the improvement program. They will charter the assessment and manage the resulting improvement activities. The SEPG will also convene technical working groups to work on specific changes. These working groups consist of project personnel who are allowed to dedicate a portion of their time to defining new software processes and practices and to installing these back on their own projects. These working groups bring a sense of realism on the amount of change an organization
Build executive II support II
I I
Implement action plan
Build Improvement infm~h'uctum
Am the organization's
Figure 2. Process improvement cycle Vol 35 No 6/7 June/July 1993
Develop Improvement
Figure 3. Process improvement infrastructure can reasonably absorb at one time. A model of this infrastructure is presented in Figure 3. When the infrastructure has been set in place, the SEPG charters a software process assessment. The assessment has three purposes. First, it gathers data about the process issues facing the organization. This is the raw input from which the improvement program will be designed. Second, it builds consensus among the software staff and management concerning the issues and the priority with which they should be addressed. Finally, it peaks the motivation for launching an improvement program. Thus, the timing of an assessment is important, in that the organization must be prepared to move quickly into action planning once the recommendations have been made. The SEI software process assessment method has been designed to achieve these three objectives through an extensive preparation period followed by five intense days on site where data are gathered, reviewed through the organization, and finally presented to senior management. The assessment findings and recommendations become the raw input into the improvement planning process. Improvement planning must be executed with the same level of thoroughness and rigour that the improvement team is trying to install in the software development process. The SEPG develops a strategic plan that responds to the business objectives of the management steering committee. Once this strategic plan has been approved, a technical working group is convened for each area of improvement and they further develop the tactical plans by which each major process will be improved across the organization. These tactical plans are then further decomposed into action plans for how each project will implement a change. The action plans must match improvement objectives with a realistic assessment of how much change a project can absorb. The organization may choose to run a pilot implementation before going to an organization-wide rollout of a new process. These decisions are why the planning process is so important in improvement programs. The implementation phase is aided by two techniques that serve process improvement, process definition and measurement. Technical working groups must define the processes they want to install, and these descriptions
385
Creating a software process improvement program
must be understandable by those who execute the software process. A well-defined process provides everyone in the organization with an understanding of their responsibilities, of their mutual dependencies, and of how the organization intends to conduct its business. Once improved processes have been defined, they should be measured. Only through measurement can the feedback loops so necessary for learning be installed. Measurements will also be needed to present the case for continued funding after the initial excitement has worn off and the organization faces contention for the funding that is in such short supply. The SEI is currently piloting methods for accomplishing both of these objectives. Throughout the improvement program, the organization is managing not only technical change, but also behavioural change. SEPG members can benefit from training in how to be internal consultants and manage technological change. Managing the social aspects of the change process is often the most difficult, because established patterns may need to be modified. Typically we find organizations chartering a reassessment every two years; but there is no magic in this number. The crucial issue is that the improvement process must be considered a repeating cycle.
CONCLUSION When managed properly, improvement programs can yield decided benefits to the sponsoring organization. Using the Capability Maturity Model assists programs in deciding what to do next among the myriad opportunities for improvement. Data from Hughes Aircraft%
386
Raytheon 1°, and Tinker Air Force Base H suggest that the expected paybacks from well-run improvement programs tend to be in the range of 5 : 1 to 9:1 for the return on investment. This paper provides a very brief overview, and those wanting to launch an improvement program should seek the assistance of those with experience in running them successfully.
REFERENCES 1 Boehm, B 'Increasing software productivity' IEEE Computer Vol 29 No 9 (1987) pp 43-57 2 Humphrey, W S, Snyder, T R, and Willis, R R 'Software process improvement at Hughes Aircraft' IEEE Software Vol 8 No 4 (1991) pp 11-23 3 Humphrey, W S Managing the software process AddisonWesley (1989) 4 Panik, M C, Curtis, B, Chrissis, M B, Averill, E L, Bamberger, J, Kasse, T C, Konrad, M, Perdue, J R, Weber, C V and Withey, J V The capability maturity model for software (CMU/SEI-91-TR-24) Software Engineering Institute, Carnegie Mellon University (1991) 5 Deming, W E Out of crisis MIT Center for Advanced Engineering Study (1982) 6 Juran, J M Juran on leadershipfor quafity Free Press (1989) 7 Crosby, P Quafity is free McGraw-Hill (1979) 8 Radice, R A, Roth, N K, O'Hara, A C and Ciarfeila, W A 'A programming process architecture' IBM Systems J. Vol 24 No 2 (1985) pp 79-90 9 Kitson, D H and Masters, S An analysis of SEI software process assessment results: 1987-1991 (CMU/SEI-92TR-24) Software Engineering Institute, Carnegie Mellon University (1992) 10 Dion, R IEEE Software Vol 9 No 3 (1992) 11 Lipke, W H and Butler, K L 'Software process improvement: a success story' Crosstalk: J. Defense Soft. Eng. No 38 (1992) pp 29-31
Information and Software Technology