- Email: [email protected]
A statistical method for controlling software defect detection process
i ~ l l l
imginll~rlg PERGAMON
Computers & Industrial Engineering 37 (1999) 137-140
A STATISTICALMETHOD FOR CONTROLLINGSOFTWARE DEFECT DETECTION PROCESS by
G. Y. Hong
M. Xie
P. Shanmugan
Singapore SoftwareCentre Industrial and Systems Engineering SingaporeSoftwareCentre Motorola ElectronicsPte. Ltd. Department Motorola ElectronicsPte. Ltd. Email: [email protected] National Universityof Singapore. ABSTRACT - The quest for solutions to stay competitive in the marketplace brings software industries to take steps to control their products and processes by introducing metrics as part of their quality system and setting triggers for action when their capability limits are exceeded. Institutionalized software inspection and testing, together with previews and postmortems within each phase of the software development life cycle have been widely recognized as a solid foundation for defect prevention. However, there still lacks a systematic control mechanism for these defect detection activities. In this paper, a statistical method is proposed to control the software defect detection process together with further defect prevention analysis. This method is currently being piloted in the Motorola Singapore Software Centre. © 1999 Elsevier Science Ltd. All rights reserved. Keywords - Statistical Process Control (SPC), Software Quality, Software Inspection, Software Testing, Causal Analysis. 1. INTRODUCTION Software defects occur throughout the software development life cycle. Defects in the context of software can cause the deviation of a piece of software from customer expectation. The quest for solutions to stay competitive in the marketplace brings software industries to take steps to control their products and processes by introducing metrics as part of their quality system and setting triggers for action when their capability limits are exceeded. Many companies have mechanisms in place to capture the quantitative snapshot of a project's performance and use these data to understand when the projects are going off-track. Institutionalized software inspection and testing, together with previews and postmortems within each phase of the software development life cycle have been widely recognized as a solid foundation for defect prevention. There are two ways of modeling software defects. One is the statistical models that involve counting techniques, comparison of historical data and detecting special causes. These models are quantitative and can be automated and costless, but abstract to the programmer. The other is the cause analysis model that investigates the details of a few defects. These models have the programmers' perspective and are applicable to a wide range of domains, but are human-insentive and lack of integrity. The purpose of this paper is to build the missing link between these two extreme ways and obtain an optimised control and modeling mechanism. 2. BASIC CONCEPTS 0360-8352/99 - see front matter © 1999 Elsevier Science Ltd. All rights reserved. PII: S0360-8352(99)00040-6
138
Proceedings of the 24th International Conference on Computers and Industrial Engineering
It is well known that statistical process controls (SPC) have been well established in hardware manufacturing industries for a long time, see e.g., Shewhart (1931). The theory and application of control charts and acceptance sampling procedures for setting action triggers to mass production manufacturing are well understood and practised by quality professionals. However, only lately have SPC techniques been attempted for software production, see example Gardiner (1986). This is because the software tasks are not repetitive with the same short periodicity, the process variations are much larger, and the quality of inputs cannot be made uniform. In addition, the situation becomes more difficult due to our lack of ability to foresee and define the numerous and complex requirements. For the use of SPC in a software development environment, certain assumptions must be made and verified. Possible applications of U chart for monitoring software inspection and testing process are presented in this paper. U charts are designed for detecting defects per unit inspected or tested, see Montgomery (1991). Assuming that introducing defects into code is a random and rare event, a Poisson distribution can be used to give the probability of finding a certain number of defects in the code and to derive control charts for the code inspection process. The upper control limits (UCL) and lower control limits (LCL) are set up based on the average number of defects per inspection unit: UCL = fo +3" fo~f~ LCL = fo - 3 " ~ o In Here, n equals to the lines of code (LOC) of the module inspected. From a data analysis point of view, special causes of variation within the software processes should be systematically identified and eliminated. Events and situations which are found 'to cause behavior outside the bounds of normal statistical variation will be isolated and, wherever possible, prevented from reoccurring. Such special cause variation is different from those which fall within the normal range of expected behavior, since these are common causes of variation which should not be a matter of concern nor trigger nugatory corrective action in a stable system. Following up the statistical analysis of the defects, causal analysis is suggested to yield improvements for preventing classes of defects. Actions are taken to minimize the root causes. For example, checklist can be generated to include specific items that should be compliant in order to avoid certain defects from occurring. It is also useful to analyze defect trend at an organizational level to determine tools, technology and training that are required to bring down the maximum defect categories. 3. STATISTICAL PROCESS CONTROL FOR SOFTWARE DEFECT DETECTION Among the many methods and techniques introduced to minimize loss caused by software failures, testing is widely recognized as an effective way of detecting the malfunction of a piece of executable software. However, the earlier a defect is unsurfaced, the less costly it takes to fix. When the customer reports a problem with a product, it might be related to the failure of the product, unclear user documentation, poor user interfaces, etc.. Quantitative measurements can be utilized throughout all kinds of defect prevention processes, such as review, inspection and testing. Due to the space limitation, here we only use software inspection as an example to illustrate the method. This is because the human inspection of the software documents and code are still a uniquely important method for software quality improvement. Unlike formal testing, reviews do
Proceedings o f the 24th International Conference on Computers and Industrial Engineering
139
not require an executable. Inspections are effective for discovering certain soft but costly defects such as codes that are poorly structured even though they are logically correct. The inspection process can be enhanced by using control charts to track the metrics, indicative of the effectiveness of code inspections, and performing causal analysis to eliminate the root cause. For illustration, data from 32 code inspection units are collected and presented here to describe how to use U-charts for software inspection process control. In Table 1, for each inspection, the code size and the raw defect data are shown together with their control limits. Defect density is measured instead of number of defects and code size is documented instead of sample size.
Table 1 Defect data from software code inspections
,,,.,)
No.
,
LOC Errors
No.
LOC Errors
No.
LOC Errors
No.
LOC Errors
1
490
12
9
540
10
17
180
8
25
240
8
2
409
8
10
450
14
18
250
23
26
516
19
3
55
I
11
550
15
19
139
8
27
354
14
4
338
13
12
167
22
20
372
2
28
250
13
5
245
27
13
164
12
21
484
15
29
264
20
6
160
12
14
70
6
22
361
2
30
276
7
7
290
9
15
330
7
23
356
4
31
300
5
8
300
10
16
112
7
24
320
11
32
150
5
iiii
According to the defect data, U chart is plotted against the defect density according to the Poisson distribution in Figure 1. The vertical axis is the defect density (say, defects/LOC), and the horizontal axis is the individual inspection number. We notice that for variable code size inspected, their upper and lower control limits are no longer constant. The lower control limit stops at 0 because there are no negative defect densities. Special cautions need to be paid when we are trying to interpret the chart. 0.1400
A
OA2OO --
;': ! !
~' O. 1000
1~
,: ~,Upper control Limit
-/\~ -! , .~,"~"~"~..< .,
_
i-:-' , .: f'~-.~: ~
.
,.-..:..,~ - f-....~..
i
i
.,,, o.04oo 0.(~00
,,,/_-_ .%,"t..... "";
0.0000
r..
,".
3
..... " ""
", ~.
5
, :. . . . .
7
9
'.
,.. ",
11
,
13
!
f-"~,c-*.Lp_,,~.t,o,_q~!
. :, '_ L , ¢ Y
15
17
, T
19
~'.ot
21
,
23
--7",",-,"
25
27
29
btlptctlon Number
Fig I U Chart of defect data analysis
From the control chart, we can see that inspections 5, 12, 16 fall above the upper solid line. This pattern shows an unexpectedly large variation in the quality of the process output which needs attention. Regardless of the underlying cause for this otit of control situation, it implies that more defects were detected than planned. That means either worse quality than anticipated had been obtained or better inspection had been performed. Here, management action is required to bring the process under control. The quality engineer will investigate the nature of the problem, isolate the causes and make decision on whether rework will be done on the out of control inspections.
140
Proceedings of the 24th International Conference on Computers and Industrial Engineering
In addition to the above situation, inspections 18 & 20 were found falling below the lower solid line. Initially, we may think this is more desirable because it indicates fewer problems. This interpretation is very risky and may lead to serious problems. The only interpretation is that the process is producing a large unexpected variation in the process quality which means that fewer defects were detected than planned. This may indicate either better quality or ineffective inspections were taking place. Management judgement is required again to identify the improvements or to do the reinspection. The out of control points should be removed from the recalculation of the control limits. 4. QUANTITATIVE ANALYSIS FOR DEFECT DATA For the purpose of analysis and continuous improvement, all the defects detected in Table 1 were classified into proper categories. Pareto chart is used to display the common error categories. We found that 55% of the defects found are documentation errors and standards violations and near 10% are misimplemmented design in coding. From the analysis we discovered that templates and coding standard should be suggested to the development team. A causal analysis was done to discover the root causes of the mistakes for the purpose of future defect prevention. Mapping the causes to the defect types, we found that 33% of the defects are due to oversight and 22% are due to 'lack of domain knowledge' and 16% are due to 'design not detailed enough'. This analysis was discussed in the coding phase postmortem and special actions are taken to minimize the causes from happening again. For example, domain training was identified for inexperienced programmers and the assumptions were documented separately and reviewed. 5. SUMMARY As a summary, the statistical method proposed in this paper has been implemented in the Motorola Singapore Software Center. Feedback tells us that this method can help us to make better processes and decisions by collecting relevant data, deriving information from data, and learning from past mistakes. A software project may be unique, but quite similar to subsequent releases. Each software project may have its own idiosyncrasies, but where and how projects fail or succeed may be quite similar from one project to another. Thus, with many software projects over a long period of time, many similar problems happen again and again, and statistical analysis could be quite meaningful and useful. Even within a single software production cycle, many events are both numerous and repetitive, such as the bug-fix change control process, the coding and inspection processes, and the build and release processes. These types of processes provide opportunities for process improvement with immediate benefits for the current development cycle. Above all, the keys to effective use of any quantitative process management techniques in the software environment are commitment and persistence.
REFERENCES Gardiner J. S. (1986), Using Statistical Control Charts For Software Quality Control, Quality and Reliability Engineering International, Vol. 3, pp. 15-20. Montgomery D.C.(1991), Introduction to Statistical Quality Control, John Wiley & Sons, Inc. Shewhart W. A. (1931), Economic Control of Quality of Manufactured Products, Van Nostrand, New York.
imginll~rlg PERGAMON
Computers & Industrial Engineering 37 (1999) 137-140
A STATISTICALMETHOD FOR CONTROLLINGSOFTWARE DEFECT DETECTION PROCESS by
G. Y. Hong
M. Xie
P. Shanmugan
Singapore SoftwareCentre Industrial and Systems Engineering SingaporeSoftwareCentre Motorola ElectronicsPte. Ltd. Department Motorola ElectronicsPte. Ltd. Email: [email protected] National Universityof Singapore. ABSTRACT - The quest for solutions to stay competitive in the marketplace brings software industries to take steps to control their products and processes by introducing metrics as part of their quality system and setting triggers for action when their capability limits are exceeded. Institutionalized software inspection and testing, together with previews and postmortems within each phase of the software development life cycle have been widely recognized as a solid foundation for defect prevention. However, there still lacks a systematic control mechanism for these defect detection activities. In this paper, a statistical method is proposed to control the software defect detection process together with further defect prevention analysis. This method is currently being piloted in the Motorola Singapore Software Centre. © 1999 Elsevier Science Ltd. All rights reserved. Keywords - Statistical Process Control (SPC), Software Quality, Software Inspection, Software Testing, Causal Analysis. 1. INTRODUCTION Software defects occur throughout the software development life cycle. Defects in the context of software can cause the deviation of a piece of software from customer expectation. The quest for solutions to stay competitive in the marketplace brings software industries to take steps to control their products and processes by introducing metrics as part of their quality system and setting triggers for action when their capability limits are exceeded. Many companies have mechanisms in place to capture the quantitative snapshot of a project's performance and use these data to understand when the projects are going off-track. Institutionalized software inspection and testing, together with previews and postmortems within each phase of the software development life cycle have been widely recognized as a solid foundation for defect prevention. There are two ways of modeling software defects. One is the statistical models that involve counting techniques, comparison of historical data and detecting special causes. These models are quantitative and can be automated and costless, but abstract to the programmer. The other is the cause analysis model that investigates the details of a few defects. These models have the programmers' perspective and are applicable to a wide range of domains, but are human-insentive and lack of integrity. The purpose of this paper is to build the missing link between these two extreme ways and obtain an optimised control and modeling mechanism. 2. BASIC CONCEPTS 0360-8352/99 - see front matter © 1999 Elsevier Science Ltd. All rights reserved. PII: S0360-8352(99)00040-6
138
Proceedings of the 24th International Conference on Computers and Industrial Engineering
It is well known that statistical process controls (SPC) have been well established in hardware manufacturing industries for a long time, see e.g., Shewhart (1931). The theory and application of control charts and acceptance sampling procedures for setting action triggers to mass production manufacturing are well understood and practised by quality professionals. However, only lately have SPC techniques been attempted for software production, see example Gardiner (1986). This is because the software tasks are not repetitive with the same short periodicity, the process variations are much larger, and the quality of inputs cannot be made uniform. In addition, the situation becomes more difficult due to our lack of ability to foresee and define the numerous and complex requirements. For the use of SPC in a software development environment, certain assumptions must be made and verified. Possible applications of U chart for monitoring software inspection and testing process are presented in this paper. U charts are designed for detecting defects per unit inspected or tested, see Montgomery (1991). Assuming that introducing defects into code is a random and rare event, a Poisson distribution can be used to give the probability of finding a certain number of defects in the code and to derive control charts for the code inspection process. The upper control limits (UCL) and lower control limits (LCL) are set up based on the average number of defects per inspection unit: UCL = fo +3" fo~f~ LCL = fo - 3 " ~ o In Here, n equals to the lines of code (LOC) of the module inspected. From a data analysis point of view, special causes of variation within the software processes should be systematically identified and eliminated. Events and situations which are found 'to cause behavior outside the bounds of normal statistical variation will be isolated and, wherever possible, prevented from reoccurring. Such special cause variation is different from those which fall within the normal range of expected behavior, since these are common causes of variation which should not be a matter of concern nor trigger nugatory corrective action in a stable system. Following up the statistical analysis of the defects, causal analysis is suggested to yield improvements for preventing classes of defects. Actions are taken to minimize the root causes. For example, checklist can be generated to include specific items that should be compliant in order to avoid certain defects from occurring. It is also useful to analyze defect trend at an organizational level to determine tools, technology and training that are required to bring down the maximum defect categories. 3. STATISTICAL PROCESS CONTROL FOR SOFTWARE DEFECT DETECTION Among the many methods and techniques introduced to minimize loss caused by software failures, testing is widely recognized as an effective way of detecting the malfunction of a piece of executable software. However, the earlier a defect is unsurfaced, the less costly it takes to fix. When the customer reports a problem with a product, it might be related to the failure of the product, unclear user documentation, poor user interfaces, etc.. Quantitative measurements can be utilized throughout all kinds of defect prevention processes, such as review, inspection and testing. Due to the space limitation, here we only use software inspection as an example to illustrate the method. This is because the human inspection of the software documents and code are still a uniquely important method for software quality improvement. Unlike formal testing, reviews do
Proceedings o f the 24th International Conference on Computers and Industrial Engineering
139
not require an executable. Inspections are effective for discovering certain soft but costly defects such as codes that are poorly structured even though they are logically correct. The inspection process can be enhanced by using control charts to track the metrics, indicative of the effectiveness of code inspections, and performing causal analysis to eliminate the root cause. For illustration, data from 32 code inspection units are collected and presented here to describe how to use U-charts for software inspection process control. In Table 1, for each inspection, the code size and the raw defect data are shown together with their control limits. Defect density is measured instead of number of defects and code size is documented instead of sample size.
Table 1 Defect data from software code inspections
,,,.,)
No.
,
LOC Errors
No.
LOC Errors
No.
LOC Errors
No.
LOC Errors
1
490
12
9
540
10
17
180
8
25
240
8
2
409
8
10
450
14
18
250
23
26
516
19
3
55
I
11
550
15
19
139
8
27
354
14
4
338
13
12
167
22
20
372
2
28
250
13
5
245
27
13
164
12
21
484
15
29
264
20
6
160
12
14
70
6
22
361
2
30
276
7
7
290
9
15
330
7
23
356
4
31
300
5
8
300
10
16
112
7
24
320
11
32
150
5
iiii
According to the defect data, U chart is plotted against the defect density according to the Poisson distribution in Figure 1. The vertical axis is the defect density (say, defects/LOC), and the horizontal axis is the individual inspection number. We notice that for variable code size inspected, their upper and lower control limits are no longer constant. The lower control limit stops at 0 because there are no negative defect densities. Special cautions need to be paid when we are trying to interpret the chart. 0.1400
A
OA2OO --
;': ! !
~' O. 1000
1~
,: ~,Upper control Limit
-/\~ -! , .~,"~"~"~..< .,
_
i-:-' , .: f'~-.~: ~
.
,.-..:..,~ - f-....~..
i
i
.,,, o.04oo 0.(~00
,,,/_-_ .%,"t..... "";
0.0000
r..
,".
3
..... " ""
", ~.
5
, :. . . . .
7
9
'.
,.. ",
11
,
13
!
f-"~,c-*.Lp_,,~.t,o,_q~!
. :, '_ L , ¢ Y
15
17
, T
19
~'.ot
21
,
23
--7",",-,"
25
27
29
btlptctlon Number
Fig I U Chart of defect data analysis
From the control chart, we can see that inspections 5, 12, 16 fall above the upper solid line. This pattern shows an unexpectedly large variation in the quality of the process output which needs attention. Regardless of the underlying cause for this otit of control situation, it implies that more defects were detected than planned. That means either worse quality than anticipated had been obtained or better inspection had been performed. Here, management action is required to bring the process under control. The quality engineer will investigate the nature of the problem, isolate the causes and make decision on whether rework will be done on the out of control inspections.
140
Proceedings of the 24th International Conference on Computers and Industrial Engineering
In addition to the above situation, inspections 18 & 20 were found falling below the lower solid line. Initially, we may think this is more desirable because it indicates fewer problems. This interpretation is very risky and may lead to serious problems. The only interpretation is that the process is producing a large unexpected variation in the process quality which means that fewer defects were detected than planned. This may indicate either better quality or ineffective inspections were taking place. Management judgement is required again to identify the improvements or to do the reinspection. The out of control points should be removed from the recalculation of the control limits. 4. QUANTITATIVE ANALYSIS FOR DEFECT DATA For the purpose of analysis and continuous improvement, all the defects detected in Table 1 were classified into proper categories. Pareto chart is used to display the common error categories. We found that 55% of the defects found are documentation errors and standards violations and near 10% are misimplemmented design in coding. From the analysis we discovered that templates and coding standard should be suggested to the development team. A causal analysis was done to discover the root causes of the mistakes for the purpose of future defect prevention. Mapping the causes to the defect types, we found that 33% of the defects are due to oversight and 22% are due to 'lack of domain knowledge' and 16% are due to 'design not detailed enough'. This analysis was discussed in the coding phase postmortem and special actions are taken to minimize the causes from happening again. For example, domain training was identified for inexperienced programmers and the assumptions were documented separately and reviewed. 5. SUMMARY As a summary, the statistical method proposed in this paper has been implemented in the Motorola Singapore Software Center. Feedback tells us that this method can help us to make better processes and decisions by collecting relevant data, deriving information from data, and learning from past mistakes. A software project may be unique, but quite similar to subsequent releases. Each software project may have its own idiosyncrasies, but where and how projects fail or succeed may be quite similar from one project to another. Thus, with many software projects over a long period of time, many similar problems happen again and again, and statistical analysis could be quite meaningful and useful. Even within a single software production cycle, many events are both numerous and repetitive, such as the bug-fix change control process, the coding and inspection processes, and the build and release processes. These types of processes provide opportunities for process improvement with immediate benefits for the current development cycle. Above all, the keys to effective use of any quantitative process management techniques in the software environment are commitment and persistence.
REFERENCES Gardiner J. S. (1986), Using Statistical Control Charts For Software Quality Control, Quality and Reliability Engineering International, Vol. 3, pp. 15-20. Montgomery D.C.(1991), Introduction to Statistical Quality Control, John Wiley & Sons, Inc. Shewhart W. A. (1931), Economic Control of Quality of Manufactured Products, Van Nostrand, New York.