- Email: [email protected]
Software quality management
Microprocessing and M~.roprogramming32 (1991) 609-616 Norlh-Holiand
SOFTWARE Q U A L r r Y M A N A G E b ' ~ r
Ernest W
~
A T A G Informatik A G / Ernst & Young Switzerland, Kanalstr. 31, CH-8152 Glattbrugg, Switzerland Goals and principles of software quality management for application development are discussed. A well-defined methodology is the key for all quality activitiesand has to support a variety of components for quality management. Some components prevent bad quality, others assess and measure quality.W e show these components in the context of our methodology called N A V I G A T O R Systems Series©. 1.
INTRODUCTION
The Software Industry has a reputation for producing software that does not meet the needs of the purchaser, is late and over budget. In Unite,,~ Kingdom a Public Accounts Committee report on Ministry of Defence (MOD) project spending has revealed unforeseen (by both M O D and contractor Project Management) cost overspends which equalled 3 to 4 billion£ in a 9 billion £ annual budget, i.e. over 30% of the annual spending is taken up in redefining programmes which are "firm fixed price contracts", with clearly defined and Lgreed requirements. Price Waterhouse [1] have recently produced a review for the Departement of Trede and Industry covering the cost and benefits of the use of soRware quality standards. The review focused on the m a r k e t relating to sol,ware produced by UK suppliers and sold to UK users. They estimated the cost resulting from poor quality software in the market to be at least 500 million £ a year, one h a l f of the m a r k e t value. These costs can be categorised in terms of: * * * *
low productivity project overruns error correction unnecessary maintenance
The cost of low productivity was estimated to be 50 million £ a year. British Aerospace, Warton have been measuring productivity and the effect of new techniques and methods for several years. They have shown substantial improvement in productivity over three generations of aircraft. A~ much as 70 % of total life cycle costs can be attributed to maintenance activities, much of this represents the correction of errors which appear when the system is in use, and changes to meet requirements which were adequately specified.
Reductions of up to 67 % in maintenance costs can be achieved b y i n t r o d u d n g more rigorous approaches to soi~ware development [2]. 2.
FUNDAMENTAL PRINCIPLES OF QUAL]TY MANAGEMENT
Providing quality products and services, focused on a n entorp~se's needs, should be one of t h e foremost goals of a n y application development effort. 2.1. Definitions
Quality denotes a degree of excellence achieved when the business needs are fully met. Defined this way, quality is ~ clear goal t h a t guides the application development process. To fully meet the enterprise's needs, the project team m u s t first determine w h a t exactly those needs are; then design an application t h a t meets tlmm; a n d finally, build a n application t h a t meets agreed-upon design specifications a n d the user's expectations.
Quality management is a prevention-based m a n a g e n i e n t a p p r o a c h for i d e n t i f y i n g a n d meeting t h e business's requirements a t every level of operation through continuous improvement in the processes through which products are produced and services are delivered. 2.2. Keys to Quality Quality m a n a g e m e n t demands h a r d work a n d dedication. No shortcuts to quality exist, and it cannot be achieved without t h e support and commitment of the entire organization. The basic principles of quality management were developed in the manufacturing arena over thirty years ago. Refined, a n d in some cases expanded, t h e s e principles apply j u s t as well to other industries.
610
E. Wa/ImOIler
spent doing unnecessary work, and time not worked due to poor planning or indecision are also included in this category.
The guidelines which follow - most true for any industry, some specificto application development - are keys to successful quality management: * Quality improvement as an intentional business strategy ® Focus on the user's expectations ® D o the right task, the right way, the firsttime e Follow a systems development methodology ® Use information engineering and soRware enginee~-ingtechniques and tools ® Shiftfrom appraisal to prevention ® Use proven projectplanning and tracking techniques ® E m p o w e r all employees in the improvement process ® Provide for training ® Develop a method to measure quality ® Use a formal and rigorous testingprocess ® Use walk-throughs to eliminate errors early 2.3. The Case for Quality Management
• E x t e r n a l F a i l u r e Costs - costs incurred by an organization because of a defective product or service. These costs are potentially the most devastating because they include not only the work involved in fixing the defect, but also any damage to the organization's reputation or any loss of future sales. Providing high-quality service or building a highquality product m a y prevent incurring all of these costs. Some of them m a y remain unmeasured, but only prevention and appraisal costs are inevitable.As a by-product of a quality management program that focuses on meeting business needs and doing work correctly the first time, internal and external failure costs should be minimized, as shown in Fig. I.
Good software m a n a g e m e n t stresses quality because of the developer's obligationto determine and satisfy the user's needs. But quality also makes good business sense. A n effective quality management program leads to user satisfaction. Related benefits often include increased sales, a good reputation, and superior products and services at less cost.
[ ] External Failur~
This last benefit deserves amplification.The cost of quality is the s u m of four component costs:
Internal Failure
Poor quality Effective quality Management
® P r e v e n t l o n C o s t s - the cost of actions taken to prevent defects from occurring in the first place, including the expense of careful planning for qaality, education about quality, CASE tools, t r a i n i n g , s u p p o r t systems, s u p p o r t m a t e r i a l s , methodologies, s t a n d a r d s , procedures and job aids.
[ ] Appraisal
® Appraisal Costs - the cost associated with measuring, evaluating, and auditing products or services for conformance to specificationsor standards. Appraisal costs cover reviews, inspections, waik-threughs, testing, and the time spent to prepare for these activities, b u t not the time spent to fix defects. Correcting defects uncovered by these processes falls under the cost of internal failure.
Spending time and money on prevention is the surest, most cost-effective way of meeting the user's needs. Preventive measures are aimed a t eliminating the source of errors. These measures continue to be effective over time, allowing to catch - before t h e y have a chance to occur countless errors t h a t would have been costly to fix. Over t h e long run, preventive m e a s u r e s should reduce the cost not only of failure b u t of inspection. I t takes less time to inspect a product with fewer defects.
• I n t e r n ~ F a i l u r e C o s t s - the cost of fixing defective products or services (i.e., those which fail to meet the user's needs) before they are delivered. These include the expense of all of the redesign, rework, reappraisal, redocumentation, and retesting that must be done w h e n inspection or testing uncovers product defects.The expenses associated with cancelled projects,lostbenefits due to project delay, time
r"l Prevention
I
[
Figure 1. The Cost of Quality Management
3.
QUALITY MANAGEMENT IN APPLICATION DEVELOPMENT
This section examines the more specific ways in which quality management works in an application development pro:iect.
Sof~vare quality management W e show our concepts of quality management on the basis of the Ernst & Young N A V I G A T O R Systems Series©J3]. N A V I G A T O R Systems Series © is an infi)rmation engineer{ng oriented approach to the ~,lanning, analysis, design, construction, implementation and evolution of information systems. This comprehensive approach provides a framework for using state-of-the-art Computer Aided Software Engineering (CASE) tools and techniques to help ensure that information systems save strategic business resources. 3.1. The Goal of Quality Management Quality management refers to all those actions that lead to consistently providing high-quality service and developing high-qualityproducts [4]. B y seeking to ensure high-quality products and services, a quality management program should eliminate m u c h of the basis for such complaints. It should, over the long run, reduce the cost in time and money of offering a Idgb,quality product or serviceby making sure the right tasks are done the right way, the first time. This is always a cheaper and more efficientpolicy.The benefits of an effectiveapproach to quality management are numerous. They include: ®
• • • • o ® • •
Higher profits through reduced quality costs High quality staff attracted and retained Improved customer/client retention Competitive advantage through shorter development cycle Stronger user relationship Decreased development costs Dim~nlshed false startsand misdirection Increased job satisfaction Meeting user expectations
the project. The group evaluates the project, not just deliverables, at various points to ensure that good quality management practices are being followed. Quality control is the testing of produ¢~ for quality at any iteratien of the p r ~ . In way, quality control, applied to iterations of the work product, is preventive as well. Although quality control is not the same as quality management, it is one of quality m a n ~ o ment's most visible and frequently deployed processes. The most important reach for q u e r y control is that its data, or the feedback it provides, constantly steers any quality management program; it indicates h o w cl~ely user needs are being met and in what areas efforts should be redoubled to result in high quaUty.
The distinction between quality management and quality assurance is less clear. In a quality oriented corporate culture, quality assurance is the organizational -function d e d i ~ d ~ quality and entitled "Quality Assurance Group". (Personnel in this organization are sometimes referred as Quality Advisors.) Quality management is the set of functions and processes that improves the quality of a particular pro~ect and identifies potential improvements in the overall development process for the entire organization within the context of a particular project. 3.3. Quality Management Components Effective quality management depends on consistent actions that lead to quality. The NAVIGATOR Systems Series© approach is divided into twelve major components: Prevention Quality Components
The ultimate goal of a quality approach is to provide the users with the highest-quality application focused on their needs, and to do so as productively as possible.
® * ® ®
3.2. Quality Management versus Quality Assurance versus Quvlity Control
•
o
Quality management should not be confused with quality assurance or quality control.In an up-tedate methodology, the entire set of activities associated with quality is defined as quality management. Quality assurance is a functien performed by an independent part of the organization which is dedicated to improve quality for the organization. This function is responsible for overseeing the quality assurance program for an enterprise or business ~cgment. Its primary goal is to assist the project manager and project team to meet the quality objectivesof
611
® * * *
Role of the Project Manager Role of the Project Team Pro~sa Model Quality Management Approach and Development and Project Management Techniques CASE Tools based on a Knowledge Base and a Knowledge Base C c o r d ~ t o r Training Proje~spec~e Procedures Phase Review and Assessment Quality Assurance Program
Components for Q u e r y ® *
Reviews Tes~n~
Figure 2 shows the way these components interact to help ensure quality. Not all of them
612
E. Wal/miHler
are unique; in fact,most organizations have some of them in place already. The question is to which extent all components are integrated and support a cohesive qua~ty management program.
assumes responsibility for quality in his work. Only the team members have the ability to actually build quality in, not inspect it in. Only the team members know where the weaknesses are in the deliverable and the severity of the weaknesses. Only the team members can produce a quality product. Process Model A well-defined process model is the cornerstone of quality management in application development [5, 6]. It embodies a standard approach to building information systems, implementing software packages, and effectively managing both of these efforts. With quality as the goal, the project team wants to do the right things, the right way, the first time.
Figure 2. Quality Management Components P r e v e n t i o n Quality Components ® Role of the Project Manager The project manager, not an external board of reviewers, is accountable for overall project quality. It is imperative that the project rnRnAger assumes responsibility for quality. The project manager is in the best position to build in quality by informing the team about quality expectations early on by holding regular discussions with the team, and by initiating team reviews, and formal reviews. Too often, the project manager incorporates only sufficient quality to obtain approval from the user of the application. Unfortunately, the result is an application which does not satisfy the real business needs. Therefore, the project manager is the key component of quality management. The manager must assume the responsibility of producing a product that not only obtains the formal approval of the project sponsor, but also meets the business needs. In addition, the rnsn~ger must make decisions when they need to be made, based on the available information. ® Role of the Project Team The project team nmmhers are responsible for the quality of their particular portion of the project deliverables. Their desire to achieve a quality product is the key to a high-quality result. It is imperative that every team member
By furnishing a standard application development life cycle, with objectives and user deliverables identified for each segment of work, NAVIGATOR Systems Series©provides the means for building quality into the project from the start. It helps the project manager determine what tasks should be done and in what order, what skills are needed to perform them, and what techniques are most appropriate to use in completing them. As a framework for project management, a process model aids not only the planning but also the monitoring and control of work done throughout the project. Controlling this work depends, in part, on using the inspection elements of a methodology: reviews, testing, and quality assurance evaluations. Guidelines help maximize the contribution these appraisal components make toward reducing the total cost of quality, while providing checkpoints to verify whether the user's needs and expectations are being met. Quality Management Approach and Plan As a part of the project charter, an overall approach for managing quality is developed. It is the basis for a formal quality management plan that is monitored and controlled throughout the project. The quality management plan is a component of the overall project plan. Effective project planning helps ensure that all work necessary to the completion of the project gets done. A good project plan clearly defines the work to be done, how long it will take, who will do it, and when it will be completed. The project manager creates the plan by first determining the high-level structure of the project and then developing a detailed plan. In the high-level structure process, the project manager tailors the objectives and definitions
Software qual/ty management
of user delivereblcsthat appear in the standard applicationdevelopment lifecycleso that they will meet the requirements of the specific project.At the same time, the manager defines the quality management approach for the project.The qualitymanagement approach is a part of the project charter. The high-level structuring drives detailedplanning activities that includeadding or deletingtasks from work planning templates, adjusting the sequence of tasks, assigning staff, estimating the time required to complete each task,and developing a schedule. During the development of this detailed plan, a formal plan is also developed for quality.All inspection points are defined and responsibilitiesare finalized.This plan is calledthe qualitymanagement plan. • Development and Project Management Techniques An information engineering oriented methodology includes several development techniques that help to increase a work product's overall quality. For example, a technique exists for checking the business area information model's completeness and consistency. In addition, the joint session techniques and prototyping techniques improve the overall quality by increasing the participation of the user. These information engineering techniques are essential to the quality of the work product.
There ~ also projectmanagement techniques that can improve the quality of the processes the project manager performs. In addition, every technique used in projects should include a set of quality considerations. These considerations assist the project team in completing its work as well as assist reviewers in knowing what to look for. • C A S E Tools based on a Knowledge Base and a Knowledge Base Coordinator A fundamental tenet of information engineering is that a C A S E tool improves the quality of the work product. In particular, many of the modeling checks which used to he perfornted manually, are now performed automatically by the CASE tool. A good architecture of such a'tool has a control layer called knowledge base coordinator. Every access to the knowledge base is handled by this layer performing a lot of formal con~tency and completeness checks. The use of these tools can dramatically improve the quality of the work product. • Training
A critical part of quality management is
613
assigning professionalsappropriately skilled for the work to be performed. The component of quality management goes handin-hand with having the rightpeople to the project. Without properly skilled and trained team members, it will take longer and cost mere build the same high-quali~ product. Therefore, the costs associated with training are considered part of preventio~
Project,specificProcedures No system development methodology can include all the policies or procedures that could apply to a project. Therefore, p r ~ e ~ s p e c ~ c procedures must augment the me~edo]c~.
The followingis a list of types ofpreject-speci~ procedures oRen neede~ - Additional standards for use o f a u ~ t e d tools -Formats for all word processing documents - Status reporting guidelines - Examples Ofthe corroctuse Of a ~ - Examples of the incorrect use of a technique - Design tips created by the project team Project-specific procedures are especially critical to the success Oflarge projects. Instead of each team member learning on his own about the various tools, methods, and procedures involved in a project, he can benefit from the knowledge and experience of others. These procedures are another way of preventing lost time on a p ~ Phase Review and Assessment The concluding stage of any phase is structured to provide a review and assessment of the work and deliverables of that phase. While the focus here would seem to be on inspection, there is in fact a strong aspect of prevention to this stage. In performing the activities of this stage, the project manager and others will assess the quality of deliverables and how closely the deliverables meet the project's objectives. However, they will also review the processes performed to produce these deliverables. Any problems uncovered will be exam;ned to see ff the problems were in the execution of the process or in the process itself. I f the process used - the execution of these steps in the methodology - proves to be flawed, corrections can be made before the process is executed again. In tiffs way, any problems or
614
E. Wal/mfil/er
failures in this project can be used as a means of refining the process for the future. This corrective feedback is a prevention component in the overall quality management program. * QuaUty Assurance Program A quaUty assurance program is established to help ensure quality in work products for an entire organization. Typically the quality assurance program is administered by a permanently established unit or in some instances, a group m a y be identifiedfor a single project.In beth cases, the orgauization/group is ~ud~pendent of the project team and is called Quality Assurance Group. Quality assurance programs are established to communicate about quality, to help identify and implement practices that improve quality,and to assistthe individual project manager and project team to meet the quality objectivesof a project.The Q A program should ~Lclude evaluations of a project at various points to ensure that good quality management practices are being followed. In addition, m e m b e r s of a Q A group should participate in the development of the project's quality management approach and plan and in various other activitiesof the project. The quality assurance group helps to ensure that the project team delivers the highestquality products and service to the user. Their role is a proactive one, they are assigned to the project before it begins and remain involved through the post-projectreview.
These reviews a r e called M a n a g e m e n t Reviews and the participants are usuaUy members of the project Steering Committee. In certain cases, a task to obtain management approval will also appear in mid stage. This is done if a criticaldecision point occurs at a point other than a stage break, and the project work should not continue until management has approved the preceding work. - In addition to these reviews, quality assurance evaluations occur at least one time per stage and at phase end. This fourth type of review, while preventative in nature, has an inspection agenda. These reviews are part of the prevention process "Quality Assurance Program". During the analysis and design stages, reviews are a m o n g the most powerful techniques available to system developers to determine whether their products will fulfillthe user's needs and meet the stated requirements. / Testing
Reviews
Testing is the predominant quality control technique for finding logic errors, technical language errors,database integrityerrors, and other defects that m a y cause software to be unreliable. The testing process begins with the creation of test strategies and plans and culminates with execution of these plans and any rework necessary to help ensure error-free code.
A well-defined process model should be defined with following quality m a n a g e m e n t review cbeckpeints:
Eight major testing events m a y occur during the application development life cycle. These are:
- Most tasks end with the work.step Conduct T e a m Review. This workstep is an informal review of a work product by one or more members of the project team. Its objective is to obtain feedback on the task'swork product and it will often be executed multiple times, once for each iteration of work product. These reviews are calledT e a m Reviews.
Requirements Mapping T e s t i n g to v e r i f y t h a t each s y s t e m requirement supports a t least one business objective, and t h a t each business objective is s u p p o r t e d b y a t l e a s t one s y s t e m requirement. This testing helps ensure t h a t the business problem is addressed and t h a t no extraneous requirements (those without a firm business basis) a r e included. I t also helps ensure that requirements are complete, consistent, feasible, and measurable. This is the first major testing event in the life cycle.
C o m l m n e n t s for Quality appraisal *
Every stage ends with the activity Review and Apprwre.
- Every technical activity ends with the task Conduct Formal Review. This task includes assembling the dra~ deliverable component created by the tasks within that activity, reviewing it for quality,and recording issues on the Open Issues Log. These reviews are called Formal Reviews and the participants are called the Work Product Review Group.
Design Mapping Testing of the conceptual system design of the ne~ system to see that it meets the stated requirements. This event maps each
Software qua/ily management
design feature to one or more requirements in order to help ensure that all requirements are covered by the chosen design. - Design Quality Testing Testing the business system and technical system design to ensure that it is of high quality and that it supports the general design of the system. It also includes aspects of usability,controls and procedures testing. Base Sot~ware Package Validation Testing Initial testing of the base software package to determine whether it performs as specified. Unit testing Testing primarily concerned with uncovering bugs or logic errors in a unit of code (typically a module or program).
615
prone points of development slmuld s i c i l y reduce appraisal and failure costs. This data enables system professiona]s to determine whether the quality of their ~ is increasing over the cour~ of a project and, over a longer period, whether it is increasing with e ~ h new p~ect. Analysis of this data slmuld also reveal any specific changes in app~cafion development processes and methods, which would include training, that are needed to e"hmina~ the causes of errors. For instance, analysis of this data might reveal that additional training in a particular skill is needed, or that additional workstops need to be added to the n m ~ address a new technology. A description of each source of data follows Fig. 3, and shows how the data is used to improve the application development process.
Integration Testing Testing that focuses on the integration of units of code into larger parts of the system. Integration testing uncovers errors in the interactions and interfaces between units that have been brought together. System Testing Testing of the overall operation of the system, including functionality, performance, and fit. System testing demonstrates how the system meets the requirements and how it works within the defined constraints. Meny separate testing events may be performed: usability testing, final requirements testing, performance testing, volume and s ~ s s testing, security and controls testing~ recovery testing, documentation and procedures testing, and multi-site testing. -
Acceptance Testing Testing that focuses on ;~beuser's acceptance of the system and the validation that all requirements have been satisfied. This final quality-control check should affirm that the project team has achieved its goal of quality - meeting the needs of the user.
3.4. Quality Metrics Measurement of quality focuses on two sources of data: defects (recorded from reviews and testing) and variances (the result of comparing project plan actuals to project plan estimates). This data is invaluable to projects, because it draws attention to steps in application development that are particularly prone to errors. The goal, of course, is not just find and fix errors but eliminate the sources of these errors as well. The extra time and money spent on error-prevention at error-
SystemDevelopmenttNethocl~o~
1t
Figure 3. Improve the Development Process with Data * Analysis of Dofecte Data about defects can be gathered from m a n y sources, but quality review results and testing logs generally yield the best information. A technique of structured walk-througlm is used during peer reviews of work products. During team reviews, or walk-throughs, the members discover and record errors that are later corre~d by the person who produced the work. Defects m y also be uncovered during reviews performed by specialists who are not nmmbors of the project team. These defects are also officially recorded, then corrected. The testing process provides additional opportunity to gather data about defects. For every problem found during the testing pmeess, three events should occur: the error should be recorded, the application code should be fixed, and the new code should again be tested. It m a y also be necessary to add new test data to the test databases.
616
E. Wallm~ller
Do not confuse this rework time with iteration. Iteration is successively refining a work product. It involves, for example, progressively more detail or expanded breadth. In the instance of data modeling, the first iteration m a y focus on identifying the major entities and relationships. The second iteration will begin to attribute these entities and identify most entities and relationships. The final iteration completes the da~L model. Each iteration can be evaluated in ~ r m s of "quality". However, ~utermediate iterations are not expected to be perfect or have the same rigor as zhe final iteration. Iteration is not a strategy for taking another pass at getting the product right, each iteration is expected to produce a quality product. ® Analysis of Project Plan Executing and tracking the project plan provides a wealth of data that can be used to steer quality management procedures. This data can be classifiedas either hard or soft. Hard data usually consists of figures, such as the discrepancy between estimated and actual time to complete a task. These figures can be analyzed using statisticalmethods to improve the est~matien heuristics. This data should then er~ble the project manager to make the necessary adjustments to the project and to hone his esthnating skills. Soft data is more subjective. It will often consist of notes or short narratives accompanying project status reports. Soft data should include the project manager's reflections on his project and his reason for settling on a given plan and method of execution. Examples of se~ data would be an explanation of why a certain workstep took longer than expected, or a discussion of the benefits of overlapping several worksteps. This data m a y later be used tx~develop case studies for project management classes, or it m a y provide~ other project managers with valuable insights into the pract~.cal aspects of project management,
4.
CONCLUSION
The role of quality management is a proactive one. It offers a standard approach to building information systems and implementing software packages, not to mention managing these efforts effectively. In so doing, it helps ensure that quality is planned into a project, that it is engineered into products, and that it is verified through reviews and testing. No matter how good a methodology is, its effectiveness depends on the professionals who use it. A well-structured methodology provides a framework for training; it offers guidelines for determining who should be trained, in what techniques, and when. But it does more than simply make clear what professionals must know in order to do their jobs effectively. A well-defined methodology should have a rich instrumentarium to handle all aspects of quality. This instrumentarium is employed in the software development process as well as in the management process. The rigorous employment of i~lformation engineering and software engineering practice can achieve significant software quality improvement over current practice without a corresponding increase in cost. By concentrating on engineering practices to achieve software and information system quality, concomitant improvements in productivity will also be attained. REFERENCES [1] Software Quality Standards, the Costs and Benefits. Report for DTI by Price Waterhouse (April 1988) [2] Rathbone, M.P. and Vale, J.M., Software Quality Management, Quality Assurance, Vol,14, No.3 (1988) [3] Ernst & Young NAVIGATOR Systems Series ©, Methodology Overview Monograph (1990) [4] Wallmi~er, E., Software-Quallt~tssicherung in der Praxls (I-Ianser,1990) [5] Humphrey, W., Managing the Software Process (Addison Wesley, 1989) [6] Chreust, G., Goldmann, H. and Gschwandtnet, O., The Role of Workmanagement in Application Development, I B M Systems Journal, Voi.29, No.2 (1990)
SOFTWARE Q U A L r r Y M A N A G E b ' ~ r
Ernest W
~
A T A G Informatik A G / Ernst & Young Switzerland, Kanalstr. 31, CH-8152 Glattbrugg, Switzerland Goals and principles of software quality management for application development are discussed. A well-defined methodology is the key for all quality activitiesand has to support a variety of components for quality management. Some components prevent bad quality, others assess and measure quality.W e show these components in the context of our methodology called N A V I G A T O R Systems Series©. 1.
INTRODUCTION
The Software Industry has a reputation for producing software that does not meet the needs of the purchaser, is late and over budget. In Unite,,~ Kingdom a Public Accounts Committee report on Ministry of Defence (MOD) project spending has revealed unforeseen (by both M O D and contractor Project Management) cost overspends which equalled 3 to 4 billion£ in a 9 billion £ annual budget, i.e. over 30% of the annual spending is taken up in redefining programmes which are "firm fixed price contracts", with clearly defined and Lgreed requirements. Price Waterhouse [1] have recently produced a review for the Departement of Trede and Industry covering the cost and benefits of the use of soRware quality standards. The review focused on the m a r k e t relating to sol,ware produced by UK suppliers and sold to UK users. They estimated the cost resulting from poor quality software in the market to be at least 500 million £ a year, one h a l f of the m a r k e t value. These costs can be categorised in terms of: * * * *
low productivity project overruns error correction unnecessary maintenance
The cost of low productivity was estimated to be 50 million £ a year. British Aerospace, Warton have been measuring productivity and the effect of new techniques and methods for several years. They have shown substantial improvement in productivity over three generations of aircraft. A~ much as 70 % of total life cycle costs can be attributed to maintenance activities, much of this represents the correction of errors which appear when the system is in use, and changes to meet requirements which were adequately specified.
Reductions of up to 67 % in maintenance costs can be achieved b y i n t r o d u d n g more rigorous approaches to soi~ware development [2]. 2.
FUNDAMENTAL PRINCIPLES OF QUAL]TY MANAGEMENT
Providing quality products and services, focused on a n entorp~se's needs, should be one of t h e foremost goals of a n y application development effort. 2.1. Definitions
Quality denotes a degree of excellence achieved when the business needs are fully met. Defined this way, quality is ~ clear goal t h a t guides the application development process. To fully meet the enterprise's needs, the project team m u s t first determine w h a t exactly those needs are; then design an application t h a t meets tlmm; a n d finally, build a n application t h a t meets agreed-upon design specifications a n d the user's expectations.
Quality management is a prevention-based m a n a g e n i e n t a p p r o a c h for i d e n t i f y i n g a n d meeting t h e business's requirements a t every level of operation through continuous improvement in the processes through which products are produced and services are delivered. 2.2. Keys to Quality Quality m a n a g e m e n t demands h a r d work a n d dedication. No shortcuts to quality exist, and it cannot be achieved without t h e support and commitment of the entire organization. The basic principles of quality management were developed in the manufacturing arena over thirty years ago. Refined, a n d in some cases expanded, t h e s e principles apply j u s t as well to other industries.
610
E. Wa/ImOIler
spent doing unnecessary work, and time not worked due to poor planning or indecision are also included in this category.
The guidelines which follow - most true for any industry, some specificto application development - are keys to successful quality management: * Quality improvement as an intentional business strategy ® Focus on the user's expectations ® D o the right task, the right way, the firsttime e Follow a systems development methodology ® Use information engineering and soRware enginee~-ingtechniques and tools ® Shiftfrom appraisal to prevention ® Use proven projectplanning and tracking techniques ® E m p o w e r all employees in the improvement process ® Provide for training ® Develop a method to measure quality ® Use a formal and rigorous testingprocess ® Use walk-throughs to eliminate errors early 2.3. The Case for Quality Management
• E x t e r n a l F a i l u r e Costs - costs incurred by an organization because of a defective product or service. These costs are potentially the most devastating because they include not only the work involved in fixing the defect, but also any damage to the organization's reputation or any loss of future sales. Providing high-quality service or building a highquality product m a y prevent incurring all of these costs. Some of them m a y remain unmeasured, but only prevention and appraisal costs are inevitable.As a by-product of a quality management program that focuses on meeting business needs and doing work correctly the first time, internal and external failure costs should be minimized, as shown in Fig. I.
Good software m a n a g e m e n t stresses quality because of the developer's obligationto determine and satisfy the user's needs. But quality also makes good business sense. A n effective quality management program leads to user satisfaction. Related benefits often include increased sales, a good reputation, and superior products and services at less cost.
[ ] External Failur~
This last benefit deserves amplification.The cost of quality is the s u m of four component costs:
Internal Failure
Poor quality Effective quality Management
® P r e v e n t l o n C o s t s - the cost of actions taken to prevent defects from occurring in the first place, including the expense of careful planning for qaality, education about quality, CASE tools, t r a i n i n g , s u p p o r t systems, s u p p o r t m a t e r i a l s , methodologies, s t a n d a r d s , procedures and job aids.
[ ] Appraisal
® Appraisal Costs - the cost associated with measuring, evaluating, and auditing products or services for conformance to specificationsor standards. Appraisal costs cover reviews, inspections, waik-threughs, testing, and the time spent to prepare for these activities, b u t not the time spent to fix defects. Correcting defects uncovered by these processes falls under the cost of internal failure.
Spending time and money on prevention is the surest, most cost-effective way of meeting the user's needs. Preventive measures are aimed a t eliminating the source of errors. These measures continue to be effective over time, allowing to catch - before t h e y have a chance to occur countless errors t h a t would have been costly to fix. Over t h e long run, preventive m e a s u r e s should reduce the cost not only of failure b u t of inspection. I t takes less time to inspect a product with fewer defects.
• I n t e r n ~ F a i l u r e C o s t s - the cost of fixing defective products or services (i.e., those which fail to meet the user's needs) before they are delivered. These include the expense of all of the redesign, rework, reappraisal, redocumentation, and retesting that must be done w h e n inspection or testing uncovers product defects.The expenses associated with cancelled projects,lostbenefits due to project delay, time
r"l Prevention
I
[
Figure 1. The Cost of Quality Management
3.
QUALITY MANAGEMENT IN APPLICATION DEVELOPMENT
This section examines the more specific ways in which quality management works in an application development pro:iect.
Sof~vare quality management W e show our concepts of quality management on the basis of the Ernst & Young N A V I G A T O R Systems Series©J3]. N A V I G A T O R Systems Series © is an infi)rmation engineer{ng oriented approach to the ~,lanning, analysis, design, construction, implementation and evolution of information systems. This comprehensive approach provides a framework for using state-of-the-art Computer Aided Software Engineering (CASE) tools and techniques to help ensure that information systems save strategic business resources. 3.1. The Goal of Quality Management Quality management refers to all those actions that lead to consistently providing high-quality service and developing high-qualityproducts [4]. B y seeking to ensure high-quality products and services, a quality management program should eliminate m u c h of the basis for such complaints. It should, over the long run, reduce the cost in time and money of offering a Idgb,quality product or serviceby making sure the right tasks are done the right way, the first time. This is always a cheaper and more efficientpolicy.The benefits of an effectiveapproach to quality management are numerous. They include: ®
• • • • o ® • •
Higher profits through reduced quality costs High quality staff attracted and retained Improved customer/client retention Competitive advantage through shorter development cycle Stronger user relationship Decreased development costs Dim~nlshed false startsand misdirection Increased job satisfaction Meeting user expectations
the project. The group evaluates the project, not just deliverables, at various points to ensure that good quality management practices are being followed. Quality control is the testing of produ¢~ for quality at any iteratien of the p r ~ . In way, quality control, applied to iterations of the work product, is preventive as well. Although quality control is not the same as quality management, it is one of quality m a n ~ o ment's most visible and frequently deployed processes. The most important reach for q u e r y control is that its data, or the feedback it provides, constantly steers any quality management program; it indicates h o w cl~ely user needs are being met and in what areas efforts should be redoubled to result in high quaUty.
The distinction between quality management and quality assurance is less clear. In a quality oriented corporate culture, quality assurance is the organizational -function d e d i ~ d ~ quality and entitled "Quality Assurance Group". (Personnel in this organization are sometimes referred as Quality Advisors.) Quality management is the set of functions and processes that improves the quality of a particular pro~ect and identifies potential improvements in the overall development process for the entire organization within the context of a particular project. 3.3. Quality Management Components Effective quality management depends on consistent actions that lead to quality. The NAVIGATOR Systems Series© approach is divided into twelve major components: Prevention Quality Components
The ultimate goal of a quality approach is to provide the users with the highest-quality application focused on their needs, and to do so as productively as possible.
® * ® ®
3.2. Quality Management versus Quality Assurance versus Quvlity Control
•
o
Quality management should not be confused with quality assurance or quality control.In an up-tedate methodology, the entire set of activities associated with quality is defined as quality management. Quality assurance is a functien performed by an independent part of the organization which is dedicated to improve quality for the organization. This function is responsible for overseeing the quality assurance program for an enterprise or business ~cgment. Its primary goal is to assist the project manager and project team to meet the quality objectivesof
611
® * * *
Role of the Project Manager Role of the Project Team Pro~sa Model Quality Management Approach and Development and Project Management Techniques CASE Tools based on a Knowledge Base and a Knowledge Base C c o r d ~ t o r Training Proje~spec~e Procedures Phase Review and Assessment Quality Assurance Program
Components for Q u e r y ® *
Reviews Tes~n~
Figure 2 shows the way these components interact to help ensure quality. Not all of them
612
E. Wal/miHler
are unique; in fact,most organizations have some of them in place already. The question is to which extent all components are integrated and support a cohesive qua~ty management program.
assumes responsibility for quality in his work. Only the team members have the ability to actually build quality in, not inspect it in. Only the team members know where the weaknesses are in the deliverable and the severity of the weaknesses. Only the team members can produce a quality product. Process Model A well-defined process model is the cornerstone of quality management in application development [5, 6]. It embodies a standard approach to building information systems, implementing software packages, and effectively managing both of these efforts. With quality as the goal, the project team wants to do the right things, the right way, the first time.
Figure 2. Quality Management Components P r e v e n t i o n Quality Components ® Role of the Project Manager The project manager, not an external board of reviewers, is accountable for overall project quality. It is imperative that the project rnRnAger assumes responsibility for quality. The project manager is in the best position to build in quality by informing the team about quality expectations early on by holding regular discussions with the team, and by initiating team reviews, and formal reviews. Too often, the project manager incorporates only sufficient quality to obtain approval from the user of the application. Unfortunately, the result is an application which does not satisfy the real business needs. Therefore, the project manager is the key component of quality management. The manager must assume the responsibility of producing a product that not only obtains the formal approval of the project sponsor, but also meets the business needs. In addition, the rnsn~ger must make decisions when they need to be made, based on the available information. ® Role of the Project Team The project team nmmhers are responsible for the quality of their particular portion of the project deliverables. Their desire to achieve a quality product is the key to a high-quality result. It is imperative that every team member
By furnishing a standard application development life cycle, with objectives and user deliverables identified for each segment of work, NAVIGATOR Systems Series©provides the means for building quality into the project from the start. It helps the project manager determine what tasks should be done and in what order, what skills are needed to perform them, and what techniques are most appropriate to use in completing them. As a framework for project management, a process model aids not only the planning but also the monitoring and control of work done throughout the project. Controlling this work depends, in part, on using the inspection elements of a methodology: reviews, testing, and quality assurance evaluations. Guidelines help maximize the contribution these appraisal components make toward reducing the total cost of quality, while providing checkpoints to verify whether the user's needs and expectations are being met. Quality Management Approach and Plan As a part of the project charter, an overall approach for managing quality is developed. It is the basis for a formal quality management plan that is monitored and controlled throughout the project. The quality management plan is a component of the overall project plan. Effective project planning helps ensure that all work necessary to the completion of the project gets done. A good project plan clearly defines the work to be done, how long it will take, who will do it, and when it will be completed. The project manager creates the plan by first determining the high-level structure of the project and then developing a detailed plan. In the high-level structure process, the project manager tailors the objectives and definitions
Software qual/ty management
of user delivereblcsthat appear in the standard applicationdevelopment lifecycleso that they will meet the requirements of the specific project.At the same time, the manager defines the quality management approach for the project.The qualitymanagement approach is a part of the project charter. The high-level structuring drives detailedplanning activities that includeadding or deletingtasks from work planning templates, adjusting the sequence of tasks, assigning staff, estimating the time required to complete each task,and developing a schedule. During the development of this detailed plan, a formal plan is also developed for quality.All inspection points are defined and responsibilitiesare finalized.This plan is calledthe qualitymanagement plan. • Development and Project Management Techniques An information engineering oriented methodology includes several development techniques that help to increase a work product's overall quality. For example, a technique exists for checking the business area information model's completeness and consistency. In addition, the joint session techniques and prototyping techniques improve the overall quality by increasing the participation of the user. These information engineering techniques are essential to the quality of the work product.
There ~ also projectmanagement techniques that can improve the quality of the processes the project manager performs. In addition, every technique used in projects should include a set of quality considerations. These considerations assist the project team in completing its work as well as assist reviewers in knowing what to look for. • C A S E Tools based on a Knowledge Base and a Knowledge Base Coordinator A fundamental tenet of information engineering is that a C A S E tool improves the quality of the work product. In particular, many of the modeling checks which used to he perfornted manually, are now performed automatically by the CASE tool. A good architecture of such a'tool has a control layer called knowledge base coordinator. Every access to the knowledge base is handled by this layer performing a lot of formal con~tency and completeness checks. The use of these tools can dramatically improve the quality of the work product. • Training
A critical part of quality management is
613
assigning professionalsappropriately skilled for the work to be performed. The component of quality management goes handin-hand with having the rightpeople to the project. Without properly skilled and trained team members, it will take longer and cost mere build the same high-quali~ product. Therefore, the costs associated with training are considered part of preventio~
Project,specificProcedures No system development methodology can include all the policies or procedures that could apply to a project. Therefore, p r ~ e ~ s p e c ~ c procedures must augment the me~edo]c~.
The followingis a list of types ofpreject-speci~ procedures oRen neede~ - Additional standards for use o f a u ~ t e d tools -Formats for all word processing documents - Status reporting guidelines - Examples Ofthe corroctuse Of a ~ - Examples of the incorrect use of a technique - Design tips created by the project team Project-specific procedures are especially critical to the success Oflarge projects. Instead of each team member learning on his own about the various tools, methods, and procedures involved in a project, he can benefit from the knowledge and experience of others. These procedures are another way of preventing lost time on a p ~ Phase Review and Assessment The concluding stage of any phase is structured to provide a review and assessment of the work and deliverables of that phase. While the focus here would seem to be on inspection, there is in fact a strong aspect of prevention to this stage. In performing the activities of this stage, the project manager and others will assess the quality of deliverables and how closely the deliverables meet the project's objectives. However, they will also review the processes performed to produce these deliverables. Any problems uncovered will be exam;ned to see ff the problems were in the execution of the process or in the process itself. I f the process used - the execution of these steps in the methodology - proves to be flawed, corrections can be made before the process is executed again. In tiffs way, any problems or
614
E. Wal/mfil/er
failures in this project can be used as a means of refining the process for the future. This corrective feedback is a prevention component in the overall quality management program. * QuaUty Assurance Program A quaUty assurance program is established to help ensure quality in work products for an entire organization. Typically the quality assurance program is administered by a permanently established unit or in some instances, a group m a y be identifiedfor a single project.In beth cases, the orgauization/group is ~ud~pendent of the project team and is called Quality Assurance Group. Quality assurance programs are established to communicate about quality, to help identify and implement practices that improve quality,and to assistthe individual project manager and project team to meet the quality objectivesof a project.The Q A program should ~Lclude evaluations of a project at various points to ensure that good quality management practices are being followed. In addition, m e m b e r s of a Q A group should participate in the development of the project's quality management approach and plan and in various other activitiesof the project. The quality assurance group helps to ensure that the project team delivers the highestquality products and service to the user. Their role is a proactive one, they are assigned to the project before it begins and remain involved through the post-projectreview.
These reviews a r e called M a n a g e m e n t Reviews and the participants are usuaUy members of the project Steering Committee. In certain cases, a task to obtain management approval will also appear in mid stage. This is done if a criticaldecision point occurs at a point other than a stage break, and the project work should not continue until management has approved the preceding work. - In addition to these reviews, quality assurance evaluations occur at least one time per stage and at phase end. This fourth type of review, while preventative in nature, has an inspection agenda. These reviews are part of the prevention process "Quality Assurance Program". During the analysis and design stages, reviews are a m o n g the most powerful techniques available to system developers to determine whether their products will fulfillthe user's needs and meet the stated requirements. / Testing
Reviews
Testing is the predominant quality control technique for finding logic errors, technical language errors,database integrityerrors, and other defects that m a y cause software to be unreliable. The testing process begins with the creation of test strategies and plans and culminates with execution of these plans and any rework necessary to help ensure error-free code.
A well-defined process model should be defined with following quality m a n a g e m e n t review cbeckpeints:
Eight major testing events m a y occur during the application development life cycle. These are:
- Most tasks end with the work.step Conduct T e a m Review. This workstep is an informal review of a work product by one or more members of the project team. Its objective is to obtain feedback on the task'swork product and it will often be executed multiple times, once for each iteration of work product. These reviews are calledT e a m Reviews.
Requirements Mapping T e s t i n g to v e r i f y t h a t each s y s t e m requirement supports a t least one business objective, and t h a t each business objective is s u p p o r t e d b y a t l e a s t one s y s t e m requirement. This testing helps ensure t h a t the business problem is addressed and t h a t no extraneous requirements (those without a firm business basis) a r e included. I t also helps ensure that requirements are complete, consistent, feasible, and measurable. This is the first major testing event in the life cycle.
C o m l m n e n t s for Quality appraisal *
Every stage ends with the activity Review and Apprwre.
- Every technical activity ends with the task Conduct Formal Review. This task includes assembling the dra~ deliverable component created by the tasks within that activity, reviewing it for quality,and recording issues on the Open Issues Log. These reviews are called Formal Reviews and the participants are called the Work Product Review Group.
Design Mapping Testing of the conceptual system design of the ne~ system to see that it meets the stated requirements. This event maps each
Software qua/ily management
design feature to one or more requirements in order to help ensure that all requirements are covered by the chosen design. - Design Quality Testing Testing the business system and technical system design to ensure that it is of high quality and that it supports the general design of the system. It also includes aspects of usability,controls and procedures testing. Base Sot~ware Package Validation Testing Initial testing of the base software package to determine whether it performs as specified. Unit testing Testing primarily concerned with uncovering bugs or logic errors in a unit of code (typically a module or program).
615
prone points of development slmuld s i c i l y reduce appraisal and failure costs. This data enables system professiona]s to determine whether the quality of their ~ is increasing over the cour~ of a project and, over a longer period, whether it is increasing with e ~ h new p~ect. Analysis of this data slmuld also reveal any specific changes in app~cafion development processes and methods, which would include training, that are needed to e"hmina~ the causes of errors. For instance, analysis of this data might reveal that additional training in a particular skill is needed, or that additional workstops need to be added to the n m ~ address a new technology. A description of each source of data follows Fig. 3, and shows how the data is used to improve the application development process.
Integration Testing Testing that focuses on the integration of units of code into larger parts of the system. Integration testing uncovers errors in the interactions and interfaces between units that have been brought together. System Testing Testing of the overall operation of the system, including functionality, performance, and fit. System testing demonstrates how the system meets the requirements and how it works within the defined constraints. Meny separate testing events may be performed: usability testing, final requirements testing, performance testing, volume and s ~ s s testing, security and controls testing~ recovery testing, documentation and procedures testing, and multi-site testing. -
Acceptance Testing Testing that focuses on ;~beuser's acceptance of the system and the validation that all requirements have been satisfied. This final quality-control check should affirm that the project team has achieved its goal of quality - meeting the needs of the user.
3.4. Quality Metrics Measurement of quality focuses on two sources of data: defects (recorded from reviews and testing) and variances (the result of comparing project plan actuals to project plan estimates). This data is invaluable to projects, because it draws attention to steps in application development that are particularly prone to errors. The goal, of course, is not just find and fix errors but eliminate the sources of these errors as well. The extra time and money spent on error-prevention at error-
SystemDevelopmenttNethocl~o~
1t
Figure 3. Improve the Development Process with Data * Analysis of Dofecte Data about defects can be gathered from m a n y sources, but quality review results and testing logs generally yield the best information. A technique of structured walk-througlm is used during peer reviews of work products. During team reviews, or walk-throughs, the members discover and record errors that are later corre~d by the person who produced the work. Defects m y also be uncovered during reviews performed by specialists who are not nmmbors of the project team. These defects are also officially recorded, then corrected. The testing process provides additional opportunity to gather data about defects. For every problem found during the testing pmeess, three events should occur: the error should be recorded, the application code should be fixed, and the new code should again be tested. It m a y also be necessary to add new test data to the test databases.
616
E. Wallm~ller
Do not confuse this rework time with iteration. Iteration is successively refining a work product. It involves, for example, progressively more detail or expanded breadth. In the instance of data modeling, the first iteration m a y focus on identifying the major entities and relationships. The second iteration will begin to attribute these entities and identify most entities and relationships. The final iteration completes the da~L model. Each iteration can be evaluated in ~ r m s of "quality". However, ~utermediate iterations are not expected to be perfect or have the same rigor as zhe final iteration. Iteration is not a strategy for taking another pass at getting the product right, each iteration is expected to produce a quality product. ® Analysis of Project Plan Executing and tracking the project plan provides a wealth of data that can be used to steer quality management procedures. This data can be classifiedas either hard or soft. Hard data usually consists of figures, such as the discrepancy between estimated and actual time to complete a task. These figures can be analyzed using statisticalmethods to improve the est~matien heuristics. This data should then er~ble the project manager to make the necessary adjustments to the project and to hone his esthnating skills. Soft data is more subjective. It will often consist of notes or short narratives accompanying project status reports. Soft data should include the project manager's reflections on his project and his reason for settling on a given plan and method of execution. Examples of se~ data would be an explanation of why a certain workstep took longer than expected, or a discussion of the benefits of overlapping several worksteps. This data m a y later be used tx~develop case studies for project management classes, or it m a y provide~ other project managers with valuable insights into the pract~.cal aspects of project management,
4.
CONCLUSION
The role of quality management is a proactive one. It offers a standard approach to building information systems and implementing software packages, not to mention managing these efforts effectively. In so doing, it helps ensure that quality is planned into a project, that it is engineered into products, and that it is verified through reviews and testing. No matter how good a methodology is, its effectiveness depends on the professionals who use it. A well-structured methodology provides a framework for training; it offers guidelines for determining who should be trained, in what techniques, and when. But it does more than simply make clear what professionals must know in order to do their jobs effectively. A well-defined methodology should have a rich instrumentarium to handle all aspects of quality. This instrumentarium is employed in the software development process as well as in the management process. The rigorous employment of i~lformation engineering and software engineering practice can achieve significant software quality improvement over current practice without a corresponding increase in cost. By concentrating on engineering practices to achieve software and information system quality, concomitant improvements in productivity will also be attained. REFERENCES [1] Software Quality Standards, the Costs and Benefits. Report for DTI by Price Waterhouse (April 1988) [2] Rathbone, M.P. and Vale, J.M., Software Quality Management, Quality Assurance, Vol,14, No.3 (1988) [3] Ernst & Young NAVIGATOR Systems Series ©, Methodology Overview Monograph (1990) [4] Wallmi~er, E., Software-Quallt~tssicherung in der Praxls (I-Ianser,1990) [5] Humphrey, W., Managing the Software Process (Addison Wesley, 1989) [6] Chreust, G., Goldmann, H. and Gschwandtnet, O., The Role of Workmanagement in Application Development, I B M Systems Journal, Voi.29, No.2 (1990)