- Email: [email protected]
Advantages of CUSUM techniques for quality control in clinical chemistry
393
Clinica Chimica Acta, 108 (1980) 393-397 @ Elsevier/North-Holland Biomedical Press
CGA 1577
ADVANTAGES OF CUSUM TECHNIQUES IN CLINICAL CHEMISTRY
FOR QUALITY
CONTROL
R.J. ROWLANDS a, D.W. WILSON b**, A.B.J. NIX a, K.W. KEMP a and K. GRIFFITHS
b
a Department of Mathematical Statistics and Operational Research, University College, Cardiff, CFI 1XL (U.K.) and b Tenovus Institute for Cancer Research, Welsh National School of Medicine, Heath Park, Cardiff, CF4 4XX (U.K.) (Received May lOth, 1980)
Summary The performance of routine analytical laboratories is assessed, inter alia, by the use of appropriate internal quality control techniques. Despite evidence that the cumulative sum technique is generally superior to Shewhart-type control charts and many others, its use has been limited by the popular misconception that it is inferior to Shewhart’s in detecting large variations and/or outliers in quality control data. The application of computer simulation methods has enabled us to answer this criticism and has provided the basis for further improvements in the design of the appropriate control scheme for general use in clinical chemistry.
Introduction Although the cumulative sum (CUSUM) quality control technique is well established in fields such as chemicals production engineering [l] its advantages have yet to be fully appreciated in clinical chemistry. In order to promote the CUSUM technique, Westgard et al. [2] have advocated the use of the “decision limit” procedure in preference to the equivalent “V-mask” CUSUM technique, because the former employs the control limit concept familiar to users of Shewhart or LevyJennings charts. The present paper rectifies an important misconception, which exists concerning the use of CUSUM schemes of any kind, namely that, although they may be more efficient at detecting small systematic changes, Shewhart charts are preferable for larger ones.
* To whom correspondence should be addressed.
394
Comparison of Shewhart and CUSUM schemes Definitions
and notation
(a) Average run length The average run length (ARL) of any continuous inspection scheme is defined as the average number of quality control (QC) samples taken before lack of control is indicated. Its value depends on the parameters of the scheme and on 8, the deviation of the current mean of the control statistic from its target value. The ARL of any sensible scheme must be large when the assay is in control (0 = 0), but small when it is not (6 # 0). The possibility that the assay will be falsely rejected cannot be completely eliminated when the standard deviation u of the determinations is non-zero. The probability that false rejection will occur during an analytical run containing N control samples is given approximately in terms of the ARL by the formula
l-(l-&)N For example, if the in-control ARL of a continuous inspection scheme is 250, the probability that it will falsely reject an assay which is in control during an analytical run containing 30 control samples is approximately 0.11. The relative merits of rival schemes can be assessed by comparing their respective rejection probabilities or run length distribution functions * at selected values of 8, but it is more convenient to examine graphs of their average run lengths plotted against 6’. Consider the ARL functions shown in Fig. la. The in-control ARL’s of the two schemes A and B are the same, but scheme B is preferable to A because its ARL is always less than the ARL of A when 8 f 0, as discussed by Roberts [3]. However, the choice is not always so easy, as shown in Fig. lb. The ARL curves cross at 8 = e0 so that neither scheme is strictly preferable to the other over the whole range of possible values of 0. In this case more information is needed before a proper choice can be made. For example, if the main concern is the detection of departures from target which exceed e0 then C is preferable to D. On the other hand, if the crossover point is so far from the origin that both ARL functions are effectively equal to unity beyond e0 then D should be used in preference to C. (b) Control schemes One-sided and symmetrical two-sided decision limit or decision interval schemes are specified by two parameters called the decision interval and references value, respectively. These schemes have been described extensively in the literature [ 1,2,4]. The two-sided decision limit CUSUM scheme with decision interval h and reference value 12will be denoted by C (h, k).
* The run length distribution function is the probability that the scheme will take, at most. N samples to reject the assay plotted against N. Westgard et al. present various run length distribution functions for a number of selected schemes.
395
(b)
(0)
,t_-_--------------------
l______--_--__-_:_-____
0
e
0
e
00
Fig. 1. Average run length functions. (a) ARL functions of two schemes A and B: B preferable to A: (b) ARL functions of two schemes C and D: D preferable to C if 0 < 00. C preferable to D if 0 > 80.
The Shewhart scheme * with action limits located at +K will be denoted by S(K). CS(h, Iz IK) will denote the scheme which is a combination of the schemes C(h, h.) and S(K), whereas the scheme which is a combination of the two CUSUM schemes C(hl, k,) and C(hz, h,) will be denoted by CC(hl, kl Ihz, h,). Results When individual QC determinations in an assay system are normally distributed and the control statistic is the arithmetic mean f (if each sample consists of a single control measurement X, then x = a) of the QC scheme, the schemes S(2.5) and C(3.3, 0.5) have the same in-control ARL of 81, but Table I shows that the CUSUM will detect a shift of 20 or less more quickly on average than the Shewhart scheme. On the other hand, S(2.5) responds sooner to shifts in excess of 20. Which scheme is preferable depends on the application of interest and if it is only large shifts from target which have clinical significance, then S(2.5) is preferable to C(3.5, 0.5). Among CUSUM schemes whose in-control ARL is 81, C(3.5, 0.5) is most efficient when 19= u, but if rapid detection is not of interest at such small values of 0 then C(3.5, 0.5) is not the CUSUM scheme to use. The in-control ARL of C(1.17, 1.4) is also 81, but this scheme is most efficient when 0 = 2.8~ and so is better suited to detecting large shifts. In fact, inspection of Table I shows that it is better than S(2.5) for all values of 8 up to 40. Beyond this point there is nothing to choose between the two schemes since their average run lengths are both effectively equal to unity. Therefore, of the schemes considered, it is the CUSUM scheme C(1.17,1.4) which is preferable when the only concern is the detection of systematic changes of size 2u or more. The above example is typical. Given any Shewhart scheme and any value of eo, a CUSUM scheme with the same in-control ARL can be found, which is * Control charts with action and warning limits are not considered in this paper. Warning limits are used to increase the Shewhart chart’s sensitivity to small systematic changes. but if it is important to detect such changes quickly then a CUSUM scheme should be used 141.
396 TABLE I AVERAGE RUN LENGTHS OF VARIOUS SCHEMES FOR CONTROLLING A NORMAL FUNCTION OF 0. THE DEVIATION OF THE MEAN OF X FROM THE TARGET VALUE
MEAN AS A
(D = 01 / Jn where n is the number of control measurements per sample and 01 is the standard deviation of each determination. ARL values of C(3.3. 0.5) were obtained from the contour nomogram’ of Goel and Wu [51 and ARL values of C(1.17.1.4) from the table of Chiu [Sl.) Control scheme
S(2.5) (x3.3.0.5) C(1.17.1.4) CC(3.7,0.5/1.17,1.6) CS(3.7,0.5/2.74)
eio 0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
81 81 81 81 81
41.5 20.0 35.0 21.4 21.7
14.9 1.0 11.3 7.3 7.4
6.3 4.0 5.0 4.0 4.1
3.24 2.88 2.64 2.71 2.14
2.00 2.28 1.82 1.93 2.01
1.45 1.92 1.40 1.50 1.53
1.23 1.64 1.19 1.25 1.25
1.11 1.42 1.08 1.11 1.11
better at detecting every size of shift up to and including one of size 8,,, by taking h large enough. Beyond e0 the Shewhart scheme will be more efficient, but when 19~ is large this advantage is only of academic interest since both schemes will have average run lengths practically equal to unity. Consequently, when only large changes are of clinical significance, it is still advantageous to use the appropriate CUSUM scheme rather than a Shewhart chart. This is a fundamental conclusion which is rarely considered in choosing a suitable control scheme by practising clinical chemists. Returning to the example, what scheme should be used when it is important to detect small systematic changes as well as large ones? The scheme C( 1.17,1.4) is still better than S(2.5) in this case, but should C(3.3, 0.5) now be preferred? The latter is considerably more efficient in the neighbourhood of 13= u, but at the price of being slightly less responsive to large shifts. In fact, this question can be answered only by taking into account the economic and other costs associated with the use of these schemes in the laboratory. Such costs can be difficult to assess, and an alternative is to compromise by using a combined scheme in which two control schemes are run simultaneously; one to detect small changes and the other to detect the larger ones.
Combined schemes The ARL of a combined scheme is defined as the average number of QC samples taken before one or other of the component schemes indicates a lack of control. The combined CUSUM scheme CC(3.3,0.5 I 1.17,1.4), not included in Table I, is certainly efficient at detecting both large and small changes, but its ARL at 8 = 0 is considerably less than 81, the in-control average run length of its component schemes. In order to find a combined CUSUM scheme whose incontrol ARL is 81, a simulation study was undertaken and the scheme CC(3.7, 0.511.17, 1.6) was found to have this property. The results are presented in Table I. It can be seen that it is almost as efficient as C(3.3, 0.5) at detecting small deviations, and responds nearly as quickly as C(1.17,1.4) when the shift is large. An alternative procedure, which has received some attention in the literature [ 21, is to combine a single CUSUM scheme with a suitably chosen Shewhart scheme, though some loss of efficiency can be expected. For
397
example, it was found that the scheme CS(3.7, 0.51 2.74) has an in-control ARL-of-81, but it can be seen from Table I that CC(3.7, 0.51 1.17, 1.6) performs slightly better throughout the working range. Discussion If one is interested in detecting changes in 8 over a limited range, then the procedure to adopt is to select the appropriate optimal CUSUM scheme [7,8]. If, however, one is interested in detecting shifts of any size, then combined schemes should be considered as these are more efficient overall at detecting both small and large changes in the control statistic being monitored than either of the component schemes alone. Consequently the incorporation of the CUSUM technique into the quality control program would improve laboratory performance and contribute further to health care assurance. Acknowledgements The authors are grateful financial support.
to the Tenovus
Organisation
for their generous
References Woodward,
R.H. and Goldsmith,
P.L. (1964)
Cumulative
Sum Techniques,
ICI Monograph
No. 3, pp.
l-65, Oliver and Boyd. Edinburgh Westgard, J.O., Groth. T.. Aronsson. T. and De Verdier, C.-H. (1977) Combined Shewhart-CUSUM control chart for improved quality control in clinical chemistry. Chn. Chem. 23. 1881-1887 Roberts, S.W. (1966) A comparison of some control chart procedures. Technometrics 8, 411-430 Nix. A.B.J., Rowlands. R.J.. Wilson, D.W.. Kemp, K.W. and Griffiths. K. (1980) Application of CUSUM techniques to monitor error in hormone assays. In: Quality Control in Clinical Endocrinology, (Wilson, D.W.. Gaskell. S.J. and Kemp, K.W.. eds.), Alpha Omega Publishing Ltd., Cardiff. in press Gael. A.L. and Wu, S.M. (1971) Determination of ARL and a contour nomogram for CUSUM charts to control normal mean. Technometrics 13.221-230 Chiu, W.K. (1974) The economic design of CUSUMcharts for controlling norfnal means. Appl. Statistics 23.420433 Kemp, K.W.. Nix. A.B.J., Wilson, D.W. and Griffiths, K. (1978) Internal quality control of radioimmunoassays. J. Endocrinol. 76.203-210 Wilson, D.W.. Griffith% K.. Kemp, K.W.. Nix, A.B.J. and Rowlands, R.J. (1979) Internal quality control of radioimmunoassays: monitoring of error. J. Endocrinol. 80. 365-372
Clinica Chimica Acta, 108 (1980) 393-397 @ Elsevier/North-Holland Biomedical Press
CGA 1577
ADVANTAGES OF CUSUM TECHNIQUES IN CLINICAL CHEMISTRY
FOR QUALITY
CONTROL
R.J. ROWLANDS a, D.W. WILSON b**, A.B.J. NIX a, K.W. KEMP a and K. GRIFFITHS
b
a Department of Mathematical Statistics and Operational Research, University College, Cardiff, CFI 1XL (U.K.) and b Tenovus Institute for Cancer Research, Welsh National School of Medicine, Heath Park, Cardiff, CF4 4XX (U.K.) (Received May lOth, 1980)
Summary The performance of routine analytical laboratories is assessed, inter alia, by the use of appropriate internal quality control techniques. Despite evidence that the cumulative sum technique is generally superior to Shewhart-type control charts and many others, its use has been limited by the popular misconception that it is inferior to Shewhart’s in detecting large variations and/or outliers in quality control data. The application of computer simulation methods has enabled us to answer this criticism and has provided the basis for further improvements in the design of the appropriate control scheme for general use in clinical chemistry.
Introduction Although the cumulative sum (CUSUM) quality control technique is well established in fields such as chemicals production engineering [l] its advantages have yet to be fully appreciated in clinical chemistry. In order to promote the CUSUM technique, Westgard et al. [2] have advocated the use of the “decision limit” procedure in preference to the equivalent “V-mask” CUSUM technique, because the former employs the control limit concept familiar to users of Shewhart or LevyJennings charts. The present paper rectifies an important misconception, which exists concerning the use of CUSUM schemes of any kind, namely that, although they may be more efficient at detecting small systematic changes, Shewhart charts are preferable for larger ones.
* To whom correspondence should be addressed.
394
Comparison of Shewhart and CUSUM schemes Definitions
and notation
(a) Average run length The average run length (ARL) of any continuous inspection scheme is defined as the average number of quality control (QC) samples taken before lack of control is indicated. Its value depends on the parameters of the scheme and on 8, the deviation of the current mean of the control statistic from its target value. The ARL of any sensible scheme must be large when the assay is in control (0 = 0), but small when it is not (6 # 0). The possibility that the assay will be falsely rejected cannot be completely eliminated when the standard deviation u of the determinations is non-zero. The probability that false rejection will occur during an analytical run containing N control samples is given approximately in terms of the ARL by the formula
l-(l-&)N For example, if the in-control ARL of a continuous inspection scheme is 250, the probability that it will falsely reject an assay which is in control during an analytical run containing 30 control samples is approximately 0.11. The relative merits of rival schemes can be assessed by comparing their respective rejection probabilities or run length distribution functions * at selected values of 8, but it is more convenient to examine graphs of their average run lengths plotted against 6’. Consider the ARL functions shown in Fig. la. The in-control ARL’s of the two schemes A and B are the same, but scheme B is preferable to A because its ARL is always less than the ARL of A when 8 f 0, as discussed by Roberts [3]. However, the choice is not always so easy, as shown in Fig. lb. The ARL curves cross at 8 = e0 so that neither scheme is strictly preferable to the other over the whole range of possible values of 0. In this case more information is needed before a proper choice can be made. For example, if the main concern is the detection of departures from target which exceed e0 then C is preferable to D. On the other hand, if the crossover point is so far from the origin that both ARL functions are effectively equal to unity beyond e0 then D should be used in preference to C. (b) Control schemes One-sided and symmetrical two-sided decision limit or decision interval schemes are specified by two parameters called the decision interval and references value, respectively. These schemes have been described extensively in the literature [ 1,2,4]. The two-sided decision limit CUSUM scheme with decision interval h and reference value 12will be denoted by C (h, k).
* The run length distribution function is the probability that the scheme will take, at most. N samples to reject the assay plotted against N. Westgard et al. present various run length distribution functions for a number of selected schemes.
395
(b)
(0)
,t_-_--------------------
l______--_--__-_:_-____
0
e
0
e
00
Fig. 1. Average run length functions. (a) ARL functions of two schemes A and B: B preferable to A: (b) ARL functions of two schemes C and D: D preferable to C if 0 < 00. C preferable to D if 0 > 80.
The Shewhart scheme * with action limits located at +K will be denoted by S(K). CS(h, Iz IK) will denote the scheme which is a combination of the schemes C(h, h.) and S(K), whereas the scheme which is a combination of the two CUSUM schemes C(hl, k,) and C(hz, h,) will be denoted by CC(hl, kl Ihz, h,). Results When individual QC determinations in an assay system are normally distributed and the control statistic is the arithmetic mean f (if each sample consists of a single control measurement X, then x = a) of the QC scheme, the schemes S(2.5) and C(3.3, 0.5) have the same in-control ARL of 81, but Table I shows that the CUSUM will detect a shift of 20 or less more quickly on average than the Shewhart scheme. On the other hand, S(2.5) responds sooner to shifts in excess of 20. Which scheme is preferable depends on the application of interest and if it is only large shifts from target which have clinical significance, then S(2.5) is preferable to C(3.5, 0.5). Among CUSUM schemes whose in-control ARL is 81, C(3.5, 0.5) is most efficient when 19= u, but if rapid detection is not of interest at such small values of 0 then C(3.5, 0.5) is not the CUSUM scheme to use. The in-control ARL of C(1.17, 1.4) is also 81, but this scheme is most efficient when 0 = 2.8~ and so is better suited to detecting large shifts. In fact, inspection of Table I shows that it is better than S(2.5) for all values of 8 up to 40. Beyond this point there is nothing to choose between the two schemes since their average run lengths are both effectively equal to unity. Therefore, of the schemes considered, it is the CUSUM scheme C(1.17,1.4) which is preferable when the only concern is the detection of systematic changes of size 2u or more. The above example is typical. Given any Shewhart scheme and any value of eo, a CUSUM scheme with the same in-control ARL can be found, which is * Control charts with action and warning limits are not considered in this paper. Warning limits are used to increase the Shewhart chart’s sensitivity to small systematic changes. but if it is important to detect such changes quickly then a CUSUM scheme should be used 141.
396 TABLE I AVERAGE RUN LENGTHS OF VARIOUS SCHEMES FOR CONTROLLING A NORMAL FUNCTION OF 0. THE DEVIATION OF THE MEAN OF X FROM THE TARGET VALUE
MEAN AS A
(D = 01 / Jn where n is the number of control measurements per sample and 01 is the standard deviation of each determination. ARL values of C(3.3. 0.5) were obtained from the contour nomogram’ of Goel and Wu [51 and ARL values of C(1.17.1.4) from the table of Chiu [Sl.) Control scheme
S(2.5) (x3.3.0.5) C(1.17.1.4) CC(3.7,0.5/1.17,1.6) CS(3.7,0.5/2.74)
eio 0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
81 81 81 81 81
41.5 20.0 35.0 21.4 21.7
14.9 1.0 11.3 7.3 7.4
6.3 4.0 5.0 4.0 4.1
3.24 2.88 2.64 2.71 2.14
2.00 2.28 1.82 1.93 2.01
1.45 1.92 1.40 1.50 1.53
1.23 1.64 1.19 1.25 1.25
1.11 1.42 1.08 1.11 1.11
better at detecting every size of shift up to and including one of size 8,,, by taking h large enough. Beyond e0 the Shewhart scheme will be more efficient, but when 19~ is large this advantage is only of academic interest since both schemes will have average run lengths practically equal to unity. Consequently, when only large changes are of clinical significance, it is still advantageous to use the appropriate CUSUM scheme rather than a Shewhart chart. This is a fundamental conclusion which is rarely considered in choosing a suitable control scheme by practising clinical chemists. Returning to the example, what scheme should be used when it is important to detect small systematic changes as well as large ones? The scheme C( 1.17,1.4) is still better than S(2.5) in this case, but should C(3.3, 0.5) now be preferred? The latter is considerably more efficient in the neighbourhood of 13= u, but at the price of being slightly less responsive to large shifts. In fact, this question can be answered only by taking into account the economic and other costs associated with the use of these schemes in the laboratory. Such costs can be difficult to assess, and an alternative is to compromise by using a combined scheme in which two control schemes are run simultaneously; one to detect small changes and the other to detect the larger ones.
Combined schemes The ARL of a combined scheme is defined as the average number of QC samples taken before one or other of the component schemes indicates a lack of control. The combined CUSUM scheme CC(3.3,0.5 I 1.17,1.4), not included in Table I, is certainly efficient at detecting both large and small changes, but its ARL at 8 = 0 is considerably less than 81, the in-control average run length of its component schemes. In order to find a combined CUSUM scheme whose incontrol ARL is 81, a simulation study was undertaken and the scheme CC(3.7, 0.511.17, 1.6) was found to have this property. The results are presented in Table I. It can be seen that it is almost as efficient as C(3.3, 0.5) at detecting small deviations, and responds nearly as quickly as C(1.17,1.4) when the shift is large. An alternative procedure, which has received some attention in the literature [ 21, is to combine a single CUSUM scheme with a suitably chosen Shewhart scheme, though some loss of efficiency can be expected. For
397
example, it was found that the scheme CS(3.7, 0.51 2.74) has an in-control ARL-of-81, but it can be seen from Table I that CC(3.7, 0.51 1.17, 1.6) performs slightly better throughout the working range. Discussion If one is interested in detecting changes in 8 over a limited range, then the procedure to adopt is to select the appropriate optimal CUSUM scheme [7,8]. If, however, one is interested in detecting shifts of any size, then combined schemes should be considered as these are more efficient overall at detecting both small and large changes in the control statistic being monitored than either of the component schemes alone. Consequently the incorporation of the CUSUM technique into the quality control program would improve laboratory performance and contribute further to health care assurance. Acknowledgements The authors are grateful financial support.
to the Tenovus
Organisation
for their generous
References Woodward,
R.H. and Goldsmith,
P.L. (1964)
Cumulative
Sum Techniques,
ICI Monograph
No. 3, pp.
l-65, Oliver and Boyd. Edinburgh Westgard, J.O., Groth. T.. Aronsson. T. and De Verdier, C.-H. (1977) Combined Shewhart-CUSUM control chart for improved quality control in clinical chemistry. Chn. Chem. 23. 1881-1887 Roberts, S.W. (1966) A comparison of some control chart procedures. Technometrics 8, 411-430 Nix. A.B.J., Rowlands. R.J.. Wilson, D.W.. Kemp, K.W. and Griffiths. K. (1980) Application of CUSUM techniques to monitor error in hormone assays. In: Quality Control in Clinical Endocrinology, (Wilson, D.W.. Gaskell. S.J. and Kemp, K.W.. eds.), Alpha Omega Publishing Ltd., Cardiff. in press Gael. A.L. and Wu, S.M. (1971) Determination of ARL and a contour nomogram for CUSUM charts to control normal mean. Technometrics 13.221-230 Chiu, W.K. (1974) The economic design of CUSUMcharts for controlling norfnal means. Appl. Statistics 23.420433 Kemp, K.W.. Nix. A.B.J., Wilson, D.W. and Griffiths, K. (1978) Internal quality control of radioimmunoassays. J. Endocrinol. 76.203-210 Wilson, D.W.. Griffith% K.. Kemp, K.W.. Nix, A.B.J. and Rowlands, R.J. (1979) Internal quality control of radioimmunoassays: monitoring of error. J. Endocrinol. 80. 365-372