- Email: [email protected]
A computerized approach to statistical quality control for radioimmunoassays in the clinical chemistry laboratory
ComputerPrograms in Biomedicine 7 (1977) 179-190 © Elsevier/North-Holland Biomedical Press
A COMPUTERIZED
179
APPROACH TO STATISTICAL QUALITY CONTROL
FOR RADIOIMMUNOASSAYS
IN T H E C L I N I C A L C H E M I S T R Y L A B O R A T O R Y
Bernard F. MCDONAGH bbod and Drug Administration, Rockville, Maryland and Peter J. MUNSON and David RODBARD Reproduction Research Branch, National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, Maryland 20014, USA
Methods for statistical quality control for the clinical laboratory in general, and radioimmunoassay in particular, have been proposed for many years. Unfortunately, only a very small number of laboratories have adapted these procedures. By use of teletypes and other remote terminals, it is possible for all laboratories to access centralized computers where a general purpose quality control program can be stored. This relieves each laboratory of the costly task of developing software, provides some degree of inter-laboratory standardization and facilitates comparison of precision and accuracy between laboratories. A prototype program for this purpose is described. This program evaluates within-assay and between-assay variability, by means of an analysis of variance for a one-way classification random-effects model, and can monitor any assay parameter by use of control chart techniques. In addition, several tests are provided to evaluate the temporal stability of the assay system, and appropriate tests for outliers are included. Also, methods are described for combination of information from several quality control samples. This provides a valid basis for adjustment of assay results or for outright rejection of an assay. For convenience, this program is designed for output on a teletype or similar terminal located in the laboratory. Simplified versions of this program can be readily adapted to desk-top calculators. The original purpose for developing this system was to provide the clinical laboratory with a simple, general, and flexible method for assessingthe performance of radioimmunoassays, but its usefulness should extend to virtually all assay methods. Radioimmunoassays
Clinical Chemistry
1. Introduction The need for quality control in the chemical laboratory has long been appreciated. One of the primary methods for monitoring performance has been the use of control charts. The application of control chart techniques and the rationale behind them has not changed much since first introduced by Levey and Jennings [12] and Henry and Segalove [11]. At about the same time, Bliss [3] and Finney [6] discussed the usefulness of control charts in the area of bioassay and Bennett and Franklin [2] in the area of chemistry. The techniques and analyses presented in these references are straight-forward applications of those described by Grant [7] and later in Duncan [5]. Radioimmunoassay (RIA) methods now constitute one of the most poPUlar and important class of meth-
Quality Control
ods in the clinical chemistry laboratory. RIAs need careful monitoring, or quality control, perhaps more than most other procedures. RIAs are notoriously unstable, with problems of blanks, large inter-assay and inter-laboratory variation, and fluctuating specificity. In part, this is due to the need for frequent preparation of 131 I- or 125I-labeled reagents and the instability of many reagents at 10 - 1 ° M concentrations. The first application of quality control methods to RIA was by Rodbard et al. [15], who discussed several assay parameters which should be monitored, together with a strategy for analyzing radioimmunoassay results to asses~ whether or not the assay is in control. Recently, Challand and Chard [4] have commented on the usefulness of some of these assay parameters for monitoring a particular assay. Regarding the analysis of result~from the clinical
180
B.F. McDonagh et al., Computerized approach to statistical quafity control
laboratory, a systematic design model for assessing analytic variation was developed by Harris [9] and applied by Williams et al. [17] and Harris et al. [10]. Adaptations of this model were recently described by Rodbard [14] for RIA and by Russell et al. [16] for the routine clinical laboratory. Russell provides an excellent introduction to readers with limited statistical background. Also, Barnett [ 1] and Youden and Steinet [18] have presented a variety of techniques - especially analysis of variance (ANOVA) - for use in evaluating variation, within and between assays, variation between reagents and technicians, and between laboratories. This paper serves to describe a prototype computer programs package which provides the essentials of an automated quality control package. It should be applicable to all types of radioimmunoassay, or for that matter, to any other type of biochemical analysis ranging from blood sugars to vitamin B12. The reason for developing this package was to overcome the following problems: 1. It appears that most laboratories have no formal type of quality control procedures. Indeed, many laboratories do not even analyze a single sample repetitively in several assays in order to evaluate the severity of between-assay variability. Many laboratories appear to be unaware of the fact that between-assay variability is often many times larger than within-assay variability. Further, there have been relatively few studies to assess the problems of between-laboratory variability. 2. The laboratory director may be unfamiliar with the mathematical, statistical or computational details of statistical quality control procedures. 3. Computations and graphics are tedious and burdensome to busy laboratory personnel. 4. The expense of developing and setting up quality control procedures. 5. To date, such programs have not been generally available. The use of a prepackaged computer program for analysis and graphics written in FORTRAN or PL/1
has several advantages. First, no special expertise is required by the laboratory director to institute a quality control system. Second, computers are now almost universally accessible via telephone and most laboratories have a small or medium-sized computer. Teletype machines and several of the larger desk-top calculators can serve as input/output devices for larger computers. Third, use of these programs requires only a few minutes of personnel time. Further, the cost of materials and analyses of quality control samples is minimal. Indeed, institution of proper quality control procedures can significantly reduce overall laboratory costs. Finally, acquisition of quality control data is necessary if a laboratory is interested in maintaining its accreditation.
2. Procedures For our purposes the two most important factors relating to radioimmunoassays (and other automated techniques) are: 1. The ability to analyze several specimens in replicate within each assay. 2. The ability to obtain measurements for a sample in each of several successive assays. In this regard, RIAs differ from many of the bioassays which they replaced. With regard to most assay parameters, the classical control chart techniques (to be discussed later) contained in the package will adequately serve to monitor the performance of the assay. Accordingly, we can apply a structural analysis to the quality control samples to assess assay performance as indicated by the measures of within-assay, between-assay and betweenlaboratory variability. The operating procedures for using these quality control samples is as follows: The essential data are obtained by running a few quality-control samples in duplicate (triplicate, etc.) in each of several assays. These samples may be from large pools of plasma or serum from several patients. Alternatively, a large pool may be obtained from a single patient by plasmapheresis. The quality-control sample should resemble the unknowns insofar as pos-
B.F. McDonagh et al., Computerized approach to statistical quality control
sible. The endogenous ligand (antigen) may be removed (e.g., stripping of steroids by charcoal) and a known quantity added; this permits testing of accuracy as well as precision. Alternatively, the endogenous ligand may be used. Generally, quality-control samples should be prepared for at least three widely separated dose levels or concentrations corresponding to the high, medium, and low range of the test as this is commonly employed. This is important, since the precision of the answer depends on the position it falls on the dose-response curve (Rodbard [14]). These samples should be aliquoted into multiple tubes which can be frozen, and then rethawed for assay on a daily, weekly or monthly basis as needed. These samples should be given to the technician in exactly the same fashion as any other clinical sample. The position of the sample in the assay should, ideally, be randomized, and unknown to the technician. If one has severe problems with between-assay variability, 5 or even 10 control samples may be used. The larger the assay, the larger the number of quality control samples that can be economically included.
181
Table 1 ANOVA for within- and between-assay variance, with equal number of replicates. This may also be used for within-lab and between-lab variance, n represents the number of assays; r represents the number of replicates of a sample per assay, and N is the total number of observations N = n.r. MS
Source
d.f,
SS
Total
n,-1
~ x~ nr~ 2
Between
n 1
SSB - r E (xi ~)2
Within
n(r 1)
SSw
q (~ij ~i )2
E(MS]
- SS
df
MS w :
SSw
¢~2
To estimate o~, note that: 2 = ro~. E ( M S b ) - E ( M S w ) = ( o 2 + ra 2) - ~w
(2)
Hence an unbiased estimate of 0 2 is given by = M&
- MSw
(3)
r
3. Statistical model * When the number of replicate determinations is the same for a particular quality control sample (the case of non-equal replicates is described in Appendix I), it is assumed that the jth replicate from the ith assay has the following structure: Xi/ = # + a i + b i j
i = 1 , 2 ..... n j = 1,2 ..... r ,
(1)
where i represents assays, and j represents replicates within assays. In addition we assume that ai is a random variable distributed normally with mean zero and variance o~ (written a i ~ N(0, 02)) and bij ~ N(0, o 2 ) and ai and b q are stochastically independent. The Analysis of Variance for this model is shown in table 1. The within-assay mean square is an unbiased estimate of a~. * Readers without previous familiarity with analysis of variance (ANOVA) may wish to proceed to the next section (4. Control Charts).
This model simply states that the total variance observed in the series of measurements is the sum of the contributions of within-assay and between-assay variability. One wishes to obtain the magnitude of betweenassay variability if within-assay variability could be reduced to a negligible level, e.g., by use of an infinite number of replicates/assay. This is called the betweenassay component of variance. Due to the random sampling errors in M S w and MSb, it is possible that the sample estimate of the between-assay component of variance is negative. This is a serious problem and reasons for negative estimates and how to handle them have been put forward by McHugh and Mielke [13]. All computations have been described in detail (cf. Appendix I of Rodbard [14] and Russell et al. [16]). The model is readily extended to cover the case of unequal number of replicates (Appendix I) and to consider another level or source of variation, e.g., betweenlaboratory variation (Appendix II).
4. Control Charts Quality control charts should be kept for each assay. These charts permit an at-a-glance analysis of the
182
B.F. McDonagh et aL, Computerized approach to statistical quality control
stability and reliability of RIA systems. For each assay, one can plot the values observed for each replicate, and the mean for these replicates. Upper and lower control limits indicate when the mean is out-ofcontrol. The range of the values is readily seen. This chart should be inspected for any trends, cycles, or outliers. After each assay is performed, the within-assay standard deviation and/or coefficient of variation is calculated for each of the quality control samples and for any other samples analyzed in replicate. These estimates may be compared with previous (cumulative) within-assay error. If the two estimates are compatible, they may be combined into an updated current, cumulative within-assay variance. Between-assay variance is analyzed as follows. Values obtained for several samples analyzed on the present assay are plotted, to be compared with results for the sample on previous assays and with either the cumulative or running average of values for that sample. The between-assay standard deviation is calculated by an analysis of variance (or, simply, as the standard deviation of the result for a given sample from each of several assays). The results on the present assay (today's result) may be compared with the average of results from all previous assays: this contrast may be called the local between-assay variance (Rodbard et al. [15] ). If this is compatible with the previous between-assay variance, it may be combined with the previous estimate to yield a new, updated estimate of cumulative between-assay variance. In a research model of the program, we also perform the following analyses: (1) The replicates within each assay are screened for outliers by means of the gap test (applicable when r > 3): Any outliers (at P < .02) are rejected, and replaced to avoid the problem of missing values (Bennett and Franklin [2] ). (2) The assay means are screened for outliers (at P < .02)). Any outlier is flagged, but computations continue to use all data. Thus, rejection of an outlier requires active intervention by the user, to eliminate data from the input file.
(a) The mean square successive difference (MSSD) for local assay means is calculated and compared with the overall standard deviation, and a test statistic is evaluated for significance at the P < .05 level. This test will indicate the presence of either long-term trends, or the presence of large short-term oscillations. (b) Several nonparametric for randomness tests are used. These include a count of the number of runs up, the number of runs down, and the length of the longest run up or down. (c) The median is calculated; we then count the number of runs above or below the median, and the length of the longest run (above or be-
low). These statistics only are printed out, along with an appropriate warning, when there is a strong suggestion (P < .02) of departure from randomness. In all cases, the program calculates the critical level of the test statistic, to eliminate the need for the user to consult handbooks or tables. Thus, the program provides an automatic check for outliers and assay system stability, using objectively defined (rather than subjective or intuitive) criteria which are uniform, reliable, can be adjusted to give a desirable level of false-positive and false-negative responses, and mercilessly enforced.
5. Sample run An example of the data input for the program is shown in fig. 1A. The program will accept as input quality control information from up to 100 assays. The various parameters of interest may be a unique value for each assay (e.g., slope, initial (BIT)o, miniD~TASZ'5
ASGgY I 2
6
602.000 567.900 ~%.009 ~I.090 537.000 52~.e00
7
~v~.O00
3
5
9 10
(3) Assay means are tested for randomness by several criteria (Bennett and Franklin, [2] ):
~9.
716.5~C 551.909 5~.')00
~ nEPLIC~TYS
g~.~3~ 557.000 ~2.399 u~g.900 554.39? ~B9.009 ~g2.399 ~75.0~9 526.399 31~.OOO REPLICATES/r.SS ~Y
Fig. 1A.
507.000 520.000 531.090 ~68.000 532.000 ~96.000 U9~.O00 652.00~ 559.000 ~67.000
6!3.500 562.000 526.000 ua2.00O 533.C00 ~79.039 50~.000 699.090 572.099 5eO.O00
B.F. McDonagh et al., Computerized approach to statistical quality control
.
,
,
.
,
,
.
,
¢xl
,
0 ~0
ml
or. c <
~o
m
COl
•
g
. . . . . . .
•
•
•
,
.
•
•
•
•
,
bJl
~I
141
. . . . . . . . . . . . . . . . . . .
•
•
,
"
.
•
,
•
,
•
1.4 >L
183
184
B.F. McDonagh et al., Computerized approach to statistical quality control o
o o
o
o o
o o
o
o o co
u~
~
o o
-~
2-
~
~
o
c, o
0 0
Z~2
__
,w
~ U
o o o um M ~ IX, I
~ 0
c,
o
e
'1
r
m
e
M~
e c
2• 2
~c c
S •n
c
o °
*
t
o
°
°
°
*
°
°
*
°
*
*
°
°
,
°
°
°
°
t
°
°
°
,
,
°
*
.
°
°
~,
t
UI c,
o o
~
oc ~
B.F. McDonagh et al., Computerized approach to statistical quality control o
o
°o
o
c)
o
o
185
~ ° - -
_
o
c~
c~
o
cb
cv.~ o E o o
o o
o
z, o
c
o
o°
o
r~
ul,
o o ~q
¢:) ,-1
t~ 0 0
o
%
0
m~E
L
.
186
B.F. McDonagh et aL, Computerized approach to statistical quality control
mal detectable dose), or may be replicated up to 10times within assays. Figure 1B shows an example of the output. This consists of the following: (A) Regurgitation of input data (for verification purposes).
equal to Okx = Ow/r2+ o 2. This is close to the larger of the two values U2w/rand o2: usually it will be very close to 02 .
6. Discussion
6.1. Start up (B) Tabular output showing for each assay: (1) Local mean, standard deviation and coefficient of variation (the latter based on the grand mean). (2) Cumulative mean (on all assays to date), and the cumulative estimate of the within-assay standard deviation and %CV. (3) Cumulative component of between-assay variance, expressed as a standard deviation and as a %CV. (4) Flags to indicate whether one has a 'bad point', 'bad assay', or 'unstable assay'. One has a bad point, if the local within-assay variance is significantly larger than the previous cumulative within-assay variance. One has a bad assay, if the local between-assay variance is significantly larger than the previous cumulative betweenassay variance. One has an unstable assay system if the MSo/MSw exceeds a critical value. (C) Graphical presentation (Fig. 1C, 1D) of: (1) Individual values for each assay (*). (2) Mean for each assay (/14). (3) Control limits for the mean based on between-assay variation. (4) Local (L) and cumulative (W) within-assay coefficient of variation for each assay. (5) Cumulative between-assay component of variance (B) for each assay, expressed as a %CV, i.e. percent of grand mean. (D) A table giving the within- and between-assay variability (in terms of SD or %CV) expected if a sample were analyzed at various degrees of replication. This is followed by a statement of the overall mean and its 95% control limits (fig. 1 B, bottom). The total variance for an arbitrary number of replicates may also be shown on this graph. This is simply
Any quality control system must be based on either a previous experience to indicate what may be regarded as reasonable control, or some a priori definition of what constitutes acceptable quality. Present experience indicates that with as few as five assays, one can obtain a fairly reasonable estimate of the underlying means and components of variance. Thus, by analyzing a sample in duplicate in each of five assays, i.e., an expenditure of only ten tubes, one obtains an estimate of within-assay variance with five degrees of freedom and between-assay variance with four degrees of freedom. One strategem to initiate a quality control program, would be to analyze quality control samples let us say in quintuplicate in the first three assays, providing us with a total of 12 degrees of freedom of within-assay variability and two degrees of freedom of between-assay variability. Thereafter, the number of quality control samples analyzed within each assay could be decreased. However, the use of unequal numbers of replicates results in a great increase in the complexity of calculations, and, more importantly, a more complicated format for data entry.
6.2. Combining results from several QC samples * The present program enables us to obtain reasonably good estimates of within- and between-assay variability at any one dose level. However, since we may have quality control samples at three or even more dose levels, it would be desirable to combine these re* A computer program is available in PL/1 for an IBM 370 computer in BATCH mode with interactive input, or in FORTRAN for a CDC 3600 computer for time-share operation. The PL/1 version and its documentation is included in: Faden, V.B. and Rodbard, D. (1975), Radioimmunoassay Data Processing, Third Edition. National Technical Information Service, Report PB 246222 (Computer Magnetic Tape), PB 246224 (Printed Listings), Springfield, Virginia 22151.
B.F. McDonagh et al., Computerized approach to statistical quality control 70[-
/
',,, ~
',,, CV
20!
i--
70
I
i
Tri
,i /
,, \
/ ,,/
#
I
20
10
10
0 ............
....... Dose
10
I00 . . . . .
i
1000
I I /1/~ I //
/ ,,,'
//
•
,<
"/
~
I
1
//
,,,/,o \
ol\
l
187
./ ,," u
E O
/
/
/
10
//" / /
~/
D..
A
1
10
100
M e a n Potency on Previous Assays
%ov ,o1_ti
i.tt,.>
:ot4k 2ol-I
,oIl \
Iit=
~.
/ /-420
"-°--Jl
I _.<- i,o
.
y = % BIBo B
Fig. 2A. Plot of within- and between-assay %CV versus dose level, i.e., the average value for four dose levels. Also shown is the theoretical relationship for within-assay variance, based on pooled data from several standard curves. (2B) Same, plotted vs. mean response (Y = %B/Bo)level. suits. There are several ways to do this (cf. Appendix II of Rodbard, [14] ). One is to plot the percent coefficient o f variation, 7'~£~V(either within or between assays) versus logarithm of dose (fig. 2). Alternatively, one may be able to combine the graphs o f %CV (fig. 1D) versus assay number for each of the three dose levels. If the results are homogeneous, they may be pooled with a gain of total number of degrees of freedom. If they appear heterogeneous, then one may evaluate the relationship between %CV and log(X). Finally, one may transform the original data (as by use of log, square-root, Studentizing, Ridit or percentile transforms) in order to achieve better uniformity of variance. However, results of analyses of these transformed data must be transformed back into terms of the original measurements.
Fig. 3. One approach to combining information from several QC-samples. Ordinate: Log of mean values for four samples on today's assay. Abscissa: Log of mean values for the same samples on the 10 most recent assays. If the assay were in control, these values should be distributed randomly around the line of identity. One assay (circles) is in good control.
Another (triangles) can be modified by a simple multiplicative correction factor, since the slope of the log-log plot is compatible with unity. In a third assay (diamonds), the slope is significantly different than unity;hence no simple correction factor can be used. It is desirable to look at the interaction o f samples and assays in terms of variability. Thus, we might have a large between-assay variability because one sample went up, another sample stayed the same, and a third sample went down. Or, we could have the same degree of between-assay variability if all three of the samples moved systematically upward (or downward). Obviously the latter would be more indicative o f a systematic change (e.g., the standard preparation may have deteriorated, or the dilutions o f the standard curve may be erroneous, etc.). One approach to this is as follows: One constructs a graph showing the nominal value for each of the three samples on the alSscissa. A logarithmic scale is useful, especially when the values span more than one decade, provided that none o f the values is close to zero. These nominal values may be obtained as the means of all of the replicates on, let us say, the ten most recent assays. One then draws in the line of identity. After each assay is constructed, the
188
B.F. McDonagh et aL, Computerized approach to statistical quality control
means of the replicates for each sample are plotted on the ordinate, which is also on a log scale (cf. fig. 3). Ideally, all of these points would fall on the line of identity. Due to both within- and between-assay variability, they should fall randomly around the line of identity, assuming that the assay is in-control. One can construct upper and lower confidence limits around the line of identity. Then we would have the rule that if one or more of the three samples falls outside those confidence limits, we have a warning that the assay is out of control. If all three of the samples fall outside the confidence limits and in the same direction, we have a definite indication that there has been a change in the performance of the assay, and it is probably justifiable to reject this assay. Alternatively, a regression through these points could be utilized to provide a correction factor. The procedure is as follows: (1) Fit a straight line. (2) Test linearity. (3) Test whether the slope is compatible with a true slope of unity (1.0). (4) Re-fit the line, forcing it to have a slope of unity. (5) Calculate the adjustment factor and its standard error. This type of analysis is not restricted to any arbitrary number of quality control samples. Indeed, one could utilize ten (or more) widely spaced samples in this fashion. The more the samples, the better the performance of the system. In this manner, the QC-samples become a secondary standard, which may be more stable and reliable than the standard curve. 6.3. When is an assay out-of-control?
Utilizing the above approach one can easily construct a quality control chart for (BIT)o, slope, 50% intercept, minimal detectable concentration, residual variance around the logit-log or four parameter logistic regression line, and several other assay parameters. In addition, we have quality control charts indicating the within- and between-assay variability for perhaps three or more quality control samples. How do we combine
all this information in order to determine whether a giveri assay is in-control, or out-of-control? In other words, when do we reject an assay, or when do we adjust all of the values in an assay (say, by multiplying by a factor of two), in order to save the assay? This is the central issue. Since there are nearly an astronomical number of assays in an unlimited number of laboratories, it is impossible to provide a totally general answer. Each laboratory must establish its own acceptance criteria. There will always be a degree of arbitrariness involved in making these decisions. However, it is important that each laboratory formulates an objective, quantitative set of rules for rejection of an assay, and then to enforce these rules. An example of these rules is as follows: If three quality control samples have each been run in duplicate in each of every assay, one can utilize the median coefficient of variation for these three. This provides protection against outliers. If the median of the cumulative within-assay %CV exceeds ten-percent or some other arbitrary value, we can state that the quality of the assay system is unsatisfactory. 2. If the between-assay %CV for any one of the QCsamples exceeds 20%, we should be alerted to the fact that the assay may be out of control, or unsatisfactory for its intended use. If the median of the cumulative between-assay %CV (component) exceeds 15%, then we have an indication that the assay system is out of control. This provides Considerable protection against outliers so that even if the results from a single tube were drastically in error, thus resulting in a massive %CV for one of the assays, we would still have quite a reliable measure of between-assay variability.
Acknowledgements We wish to thank the following individuals for assistance with programming: J. Gurian, S. Knisley, M. Kreeger, R. Sheats, E. Levine, and D.M. Hutt. T. Sellner prepared the manuscript on the WYLBUR textediting system.
B.F. McDonagh et al., Computerized approach to statistical quality con trol
poses may be generalized to take into account the contribution of laboratories, laboratory-sample interaction and replicate effects. The practical side of this analysis has been presented by Youden and Steiner [18] and the theoretical side has been developed in many standard texts (e.g., Bennett and Franklin, [2] ), under the title of Two-Way Classification with Interaction, Components of Variance Model. The model is developed as follows. The observation X i j k obtained from the k th replicate of the jth sample in the ith laboratory with i = 1 ..... l,j = 1 ..... s, k = 1 ..... r, is assumed to be described by the model:
Table 2 Source
d.f.
SS
Total
N-1
~ x2 N-~
Between
n-1
SSB = Z l l , I ' l 1_N-~ 2 '\ ri /
,,,,,,,,,o
N_o
W
MS
E(MS)
MSB=
O2w + ko o2b
2
ij
ij-i \
k°=
r~-~ /
ss.
02
MSw= ~
189
N(n- 1)
X i j k = Id + ~i + l)j + )kij + eij k ,
Appendix I. Unequal number of replicates
where the ~i represent the laboratory effects (assumed random); ~j represnets the sample effects (again assumed random), Xii represents any variations which may be peculiar to a particular combination of laboratory and sample with E(~kij ) = 0 for each ij; and eli k are normally distributed components (again random). The analysis of variance for this model is presented in table 3.
The model described in the text for a one-way components of variance analysis may be readily generalized to include the case where there is an unequal number of replicates (Graybill, [8] ). We assume that the ith assay has r i replicates, so i = 1 ..... n, and / = I .... , ri, and the total number of observations is N, w
N = ~ ri . i=1
References
The analysis of variance is shown in table 2.
[ 1 ] R.N. Barnett, Clinical Laboratory Statistics (Little Brown and Co. 1971). [2] C.A, Bennett and N.L. Franklin, Statistical Analysis in Chemistry and the Chemical Industry (J. Wiley and Sons, NY, 1954). [3] R. Bliss, The Statistics of Bioassay (Academic Press, NY 1952). [4] G.S. Challand and T. Chard, Clin. Chim. Acta (1973) 133.
Appendix II. Between-laboratory comparison The development of any RIA method will most likely require s samples to be replicated r-times within each of the laboratories in a collaborative study, to assess precision and accuracy of the method. The statistical model presented above for quality control purTable 3 Source
df
SS
Total
N 1
~k{Xij k .~)2
Between Laborato= ies
~t- 1
s 1
Between
Samples
MS
E(MS)
SSL = rsYi {~i. _~)2
SSL MS L _- ~1-
°~ +' °I
SSS = r £ E (~.j_~}2
SSS MSs - s-1
LaboratorySample Interaction
(£vl)(s-1)
SSLs = r ~1 (~ii-'~i. _~ j +'~)2
MSLs
Within Samples & Laboratories
N-~s
SSw = i~k (Xijk -"xij )2
MSw
+rso~ +r~o~
SSLs (~-1)(s-1) SSw N-£s
o~
190
B.F. McDonagh et al., Computerized approach to statistical quality control
[5] A.J. Duncan, Quality Control and Industrial Statistics, 3rd ed., (R.D. Irwin, Inc., Homewood, IU., 1965). [6] D.J. Finney, Statistical Methods in Biological Assay, 2nd ed. (Griffin, London, 1963). [7] E.L. Grant, Statistical Quality Control, 2nd ed. (McGraw-Hill, NY, 1952). [8] F. Graybill, An Introduction to Linear Models, Vol. I (McGraw-Hill, NY, 1961). [9] E.K. Harris, J. Chron. Dis. (1976) 23. [10] E.K. Harris, P. Kanofsky, G. Shakarji and E. Cotlove, Clin. Chem. 16 (1970) 1022. [11] R.J. Henry and M. Sagelove, J. Clin. Path. 5 (1952) 305. [12] S. Levey and E.R. Jennings, Am. J. Clin. Path. 20 (1950) 1059,
[13] R.B. McHugh and P.W. Mielke, Am. J. Stat. Ass. 63 (1968) 1000. [14] D. Rodbard, Clin. Chem. 20 (1974) 1255. [15] D. Rodbard, P.L. Rayford, J.A. Cooper and G.T. Ross, J. Clin. Endocrinol. Metab. 28 (1968) 1412. [16] C.D. Russell, H.J. De Blanc and H.N. Wagner, J. Hopkins. Med. 35 (1974) 334. [171 G.Z. Williams, D.S. Young, M.R. Stein and E. Cotlove, Clin. Chem. 16 (1970) 1016. [181 W.J. Younden and E.H. Steiner, (Stat. Man. Ass. Off. Analyt. Chem., Ass. Off. Anal. Chem., Washington, DC 1975).
A COMPUTERIZED
179
APPROACH TO STATISTICAL QUALITY CONTROL
FOR RADIOIMMUNOASSAYS
IN T H E C L I N I C A L C H E M I S T R Y L A B O R A T O R Y
Bernard F. MCDONAGH bbod and Drug Administration, Rockville, Maryland and Peter J. MUNSON and David RODBARD Reproduction Research Branch, National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, Maryland 20014, USA
Methods for statistical quality control for the clinical laboratory in general, and radioimmunoassay in particular, have been proposed for many years. Unfortunately, only a very small number of laboratories have adapted these procedures. By use of teletypes and other remote terminals, it is possible for all laboratories to access centralized computers where a general purpose quality control program can be stored. This relieves each laboratory of the costly task of developing software, provides some degree of inter-laboratory standardization and facilitates comparison of precision and accuracy between laboratories. A prototype program for this purpose is described. This program evaluates within-assay and between-assay variability, by means of an analysis of variance for a one-way classification random-effects model, and can monitor any assay parameter by use of control chart techniques. In addition, several tests are provided to evaluate the temporal stability of the assay system, and appropriate tests for outliers are included. Also, methods are described for combination of information from several quality control samples. This provides a valid basis for adjustment of assay results or for outright rejection of an assay. For convenience, this program is designed for output on a teletype or similar terminal located in the laboratory. Simplified versions of this program can be readily adapted to desk-top calculators. The original purpose for developing this system was to provide the clinical laboratory with a simple, general, and flexible method for assessingthe performance of radioimmunoassays, but its usefulness should extend to virtually all assay methods. Radioimmunoassays
Clinical Chemistry
1. Introduction The need for quality control in the chemical laboratory has long been appreciated. One of the primary methods for monitoring performance has been the use of control charts. The application of control chart techniques and the rationale behind them has not changed much since first introduced by Levey and Jennings [12] and Henry and Segalove [11]. At about the same time, Bliss [3] and Finney [6] discussed the usefulness of control charts in the area of bioassay and Bennett and Franklin [2] in the area of chemistry. The techniques and analyses presented in these references are straight-forward applications of those described by Grant [7] and later in Duncan [5]. Radioimmunoassay (RIA) methods now constitute one of the most poPUlar and important class of meth-
Quality Control
ods in the clinical chemistry laboratory. RIAs need careful monitoring, or quality control, perhaps more than most other procedures. RIAs are notoriously unstable, with problems of blanks, large inter-assay and inter-laboratory variation, and fluctuating specificity. In part, this is due to the need for frequent preparation of 131 I- or 125I-labeled reagents and the instability of many reagents at 10 - 1 ° M concentrations. The first application of quality control methods to RIA was by Rodbard et al. [15], who discussed several assay parameters which should be monitored, together with a strategy for analyzing radioimmunoassay results to asses~ whether or not the assay is in control. Recently, Challand and Chard [4] have commented on the usefulness of some of these assay parameters for monitoring a particular assay. Regarding the analysis of result~from the clinical
180
B.F. McDonagh et al., Computerized approach to statistical quafity control
laboratory, a systematic design model for assessing analytic variation was developed by Harris [9] and applied by Williams et al. [17] and Harris et al. [10]. Adaptations of this model were recently described by Rodbard [14] for RIA and by Russell et al. [16] for the routine clinical laboratory. Russell provides an excellent introduction to readers with limited statistical background. Also, Barnett [ 1] and Youden and Steinet [18] have presented a variety of techniques - especially analysis of variance (ANOVA) - for use in evaluating variation, within and between assays, variation between reagents and technicians, and between laboratories. This paper serves to describe a prototype computer programs package which provides the essentials of an automated quality control package. It should be applicable to all types of radioimmunoassay, or for that matter, to any other type of biochemical analysis ranging from blood sugars to vitamin B12. The reason for developing this package was to overcome the following problems: 1. It appears that most laboratories have no formal type of quality control procedures. Indeed, many laboratories do not even analyze a single sample repetitively in several assays in order to evaluate the severity of between-assay variability. Many laboratories appear to be unaware of the fact that between-assay variability is often many times larger than within-assay variability. Further, there have been relatively few studies to assess the problems of between-laboratory variability. 2. The laboratory director may be unfamiliar with the mathematical, statistical or computational details of statistical quality control procedures. 3. Computations and graphics are tedious and burdensome to busy laboratory personnel. 4. The expense of developing and setting up quality control procedures. 5. To date, such programs have not been generally available. The use of a prepackaged computer program for analysis and graphics written in FORTRAN or PL/1
has several advantages. First, no special expertise is required by the laboratory director to institute a quality control system. Second, computers are now almost universally accessible via telephone and most laboratories have a small or medium-sized computer. Teletype machines and several of the larger desk-top calculators can serve as input/output devices for larger computers. Third, use of these programs requires only a few minutes of personnel time. Further, the cost of materials and analyses of quality control samples is minimal. Indeed, institution of proper quality control procedures can significantly reduce overall laboratory costs. Finally, acquisition of quality control data is necessary if a laboratory is interested in maintaining its accreditation.
2. Procedures For our purposes the two most important factors relating to radioimmunoassays (and other automated techniques) are: 1. The ability to analyze several specimens in replicate within each assay. 2. The ability to obtain measurements for a sample in each of several successive assays. In this regard, RIAs differ from many of the bioassays which they replaced. With regard to most assay parameters, the classical control chart techniques (to be discussed later) contained in the package will adequately serve to monitor the performance of the assay. Accordingly, we can apply a structural analysis to the quality control samples to assess assay performance as indicated by the measures of within-assay, between-assay and betweenlaboratory variability. The operating procedures for using these quality control samples is as follows: The essential data are obtained by running a few quality-control samples in duplicate (triplicate, etc.) in each of several assays. These samples may be from large pools of plasma or serum from several patients. Alternatively, a large pool may be obtained from a single patient by plasmapheresis. The quality-control sample should resemble the unknowns insofar as pos-
B.F. McDonagh et al., Computerized approach to statistical quality control
sible. The endogenous ligand (antigen) may be removed (e.g., stripping of steroids by charcoal) and a known quantity added; this permits testing of accuracy as well as precision. Alternatively, the endogenous ligand may be used. Generally, quality-control samples should be prepared for at least three widely separated dose levels or concentrations corresponding to the high, medium, and low range of the test as this is commonly employed. This is important, since the precision of the answer depends on the position it falls on the dose-response curve (Rodbard [14]). These samples should be aliquoted into multiple tubes which can be frozen, and then rethawed for assay on a daily, weekly or monthly basis as needed. These samples should be given to the technician in exactly the same fashion as any other clinical sample. The position of the sample in the assay should, ideally, be randomized, and unknown to the technician. If one has severe problems with between-assay variability, 5 or even 10 control samples may be used. The larger the assay, the larger the number of quality control samples that can be economically included.
181
Table 1 ANOVA for within- and between-assay variance, with equal number of replicates. This may also be used for within-lab and between-lab variance, n represents the number of assays; r represents the number of replicates of a sample per assay, and N is the total number of observations N = n.r. MS
Source
d.f,
SS
Total
n,-1
~ x~ nr~ 2
Between
n 1
SSB - r E (xi ~)2
Within
n(r 1)
SSw
q (~ij ~i )2
E(MS]
- SS
df
MS w :
SSw
¢~2
To estimate o~, note that: 2 = ro~. E ( M S b ) - E ( M S w ) = ( o 2 + ra 2) - ~w
(2)
Hence an unbiased estimate of 0 2 is given by = M&
- MSw
(3)
r
3. Statistical model * When the number of replicate determinations is the same for a particular quality control sample (the case of non-equal replicates is described in Appendix I), it is assumed that the jth replicate from the ith assay has the following structure: Xi/ = # + a i + b i j
i = 1 , 2 ..... n j = 1,2 ..... r ,
(1)
where i represents assays, and j represents replicates within assays. In addition we assume that ai is a random variable distributed normally with mean zero and variance o~ (written a i ~ N(0, 02)) and bij ~ N(0, o 2 ) and ai and b q are stochastically independent. The Analysis of Variance for this model is shown in table 1. The within-assay mean square is an unbiased estimate of a~. * Readers without previous familiarity with analysis of variance (ANOVA) may wish to proceed to the next section (4. Control Charts).
This model simply states that the total variance observed in the series of measurements is the sum of the contributions of within-assay and between-assay variability. One wishes to obtain the magnitude of betweenassay variability if within-assay variability could be reduced to a negligible level, e.g., by use of an infinite number of replicates/assay. This is called the betweenassay component of variance. Due to the random sampling errors in M S w and MSb, it is possible that the sample estimate of the between-assay component of variance is negative. This is a serious problem and reasons for negative estimates and how to handle them have been put forward by McHugh and Mielke [13]. All computations have been described in detail (cf. Appendix I of Rodbard [14] and Russell et al. [16]). The model is readily extended to cover the case of unequal number of replicates (Appendix I) and to consider another level or source of variation, e.g., betweenlaboratory variation (Appendix II).
4. Control Charts Quality control charts should be kept for each assay. These charts permit an at-a-glance analysis of the
182
B.F. McDonagh et aL, Computerized approach to statistical quality control
stability and reliability of RIA systems. For each assay, one can plot the values observed for each replicate, and the mean for these replicates. Upper and lower control limits indicate when the mean is out-ofcontrol. The range of the values is readily seen. This chart should be inspected for any trends, cycles, or outliers. After each assay is performed, the within-assay standard deviation and/or coefficient of variation is calculated for each of the quality control samples and for any other samples analyzed in replicate. These estimates may be compared with previous (cumulative) within-assay error. If the two estimates are compatible, they may be combined into an updated current, cumulative within-assay variance. Between-assay variance is analyzed as follows. Values obtained for several samples analyzed on the present assay are plotted, to be compared with results for the sample on previous assays and with either the cumulative or running average of values for that sample. The between-assay standard deviation is calculated by an analysis of variance (or, simply, as the standard deviation of the result for a given sample from each of several assays). The results on the present assay (today's result) may be compared with the average of results from all previous assays: this contrast may be called the local between-assay variance (Rodbard et al. [15] ). If this is compatible with the previous between-assay variance, it may be combined with the previous estimate to yield a new, updated estimate of cumulative between-assay variance. In a research model of the program, we also perform the following analyses: (1) The replicates within each assay are screened for outliers by means of the gap test (applicable when r > 3): Any outliers (at P < .02) are rejected, and replaced to avoid the problem of missing values (Bennett and Franklin [2] ). (2) The assay means are screened for outliers (at P < .02)). Any outlier is flagged, but computations continue to use all data. Thus, rejection of an outlier requires active intervention by the user, to eliminate data from the input file.
(a) The mean square successive difference (MSSD) for local assay means is calculated and compared with the overall standard deviation, and a test statistic is evaluated for significance at the P < .05 level. This test will indicate the presence of either long-term trends, or the presence of large short-term oscillations. (b) Several nonparametric for randomness tests are used. These include a count of the number of runs up, the number of runs down, and the length of the longest run up or down. (c) The median is calculated; we then count the number of runs above or below the median, and the length of the longest run (above or be-
low). These statistics only are printed out, along with an appropriate warning, when there is a strong suggestion (P < .02) of departure from randomness. In all cases, the program calculates the critical level of the test statistic, to eliminate the need for the user to consult handbooks or tables. Thus, the program provides an automatic check for outliers and assay system stability, using objectively defined (rather than subjective or intuitive) criteria which are uniform, reliable, can be adjusted to give a desirable level of false-positive and false-negative responses, and mercilessly enforced.
5. Sample run An example of the data input for the program is shown in fig. 1A. The program will accept as input quality control information from up to 100 assays. The various parameters of interest may be a unique value for each assay (e.g., slope, initial (BIT)o, miniD~TASZ'5
ASGgY I 2
6
602.000 567.900 ~%.009 ~I.090 537.000 52~.e00
7
~v~.O00
3
5
9 10
(3) Assay means are tested for randomness by several criteria (Bennett and Franklin, [2] ):
~9.
716.5~C 551.909 5~.')00
~ nEPLIC~TYS
g~.~3~ 557.000 ~2.399 u~g.900 554.39? ~B9.009 ~g2.399 ~75.0~9 526.399 31~.OOO REPLICATES/r.SS ~Y
Fig. 1A.
507.000 520.000 531.090 ~68.000 532.000 ~96.000 U9~.O00 652.00~ 559.000 ~67.000
6!3.500 562.000 526.000 ua2.00O 533.C00 ~79.039 50~.000 699.090 572.099 5eO.O00
B.F. McDonagh et al., Computerized approach to statistical quality control
.
,
,
.
,
,
.
,
¢xl
,
0 ~0
ml
or. c <
~o
m
COl
•
g
. . . . . . .
•
•
•
,
.
•
•
•
•
,
bJl
~I
141
. . . . . . . . . . . . . . . . . . .
•
•
,
"
.
•
,
•
,
•
1.4 >L
183
184
B.F. McDonagh et al., Computerized approach to statistical quality control o
o o
o
o o
o o
o
o o co
u~
~
o o
-~
2-
~
~
o
c, o
0 0
Z~2
__
,w
~ U
o o o um M ~ IX, I
~ 0
c,
o
e
'1
r
m
e
M~
e c
2• 2
~c c
S •n
c
o °
*
t
o
°
°
°
*
°
°
*
°
*
*
°
°
,
°
°
°
°
t
°
°
°
,
,
°
*
.
°
°
~,
t
UI c,
o o
~
oc ~
B.F. McDonagh et al., Computerized approach to statistical quality control o
o
°o
o
c)
o
o
185
~ ° - -
_
o
c~
c~
o
cb
cv.~ o E o o
o o
o
z, o
c
o
o°
o
r~
ul,
o o ~q
¢:) ,-1
t~ 0 0
o
%
0
m~E
L
.
186
B.F. McDonagh et aL, Computerized approach to statistical quality control
mal detectable dose), or may be replicated up to 10times within assays. Figure 1B shows an example of the output. This consists of the following: (A) Regurgitation of input data (for verification purposes).
equal to Okx = Ow/r2+ o 2. This is close to the larger of the two values U2w/rand o2: usually it will be very close to 02 .
6. Discussion
6.1. Start up (B) Tabular output showing for each assay: (1) Local mean, standard deviation and coefficient of variation (the latter based on the grand mean). (2) Cumulative mean (on all assays to date), and the cumulative estimate of the within-assay standard deviation and %CV. (3) Cumulative component of between-assay variance, expressed as a standard deviation and as a %CV. (4) Flags to indicate whether one has a 'bad point', 'bad assay', or 'unstable assay'. One has a bad point, if the local within-assay variance is significantly larger than the previous cumulative within-assay variance. One has a bad assay, if the local between-assay variance is significantly larger than the previous cumulative betweenassay variance. One has an unstable assay system if the MSo/MSw exceeds a critical value. (C) Graphical presentation (Fig. 1C, 1D) of: (1) Individual values for each assay (*). (2) Mean for each assay (/14). (3) Control limits for the mean based on between-assay variation. (4) Local (L) and cumulative (W) within-assay coefficient of variation for each assay. (5) Cumulative between-assay component of variance (B) for each assay, expressed as a %CV, i.e. percent of grand mean. (D) A table giving the within- and between-assay variability (in terms of SD or %CV) expected if a sample were analyzed at various degrees of replication. This is followed by a statement of the overall mean and its 95% control limits (fig. 1 B, bottom). The total variance for an arbitrary number of replicates may also be shown on this graph. This is simply
Any quality control system must be based on either a previous experience to indicate what may be regarded as reasonable control, or some a priori definition of what constitutes acceptable quality. Present experience indicates that with as few as five assays, one can obtain a fairly reasonable estimate of the underlying means and components of variance. Thus, by analyzing a sample in duplicate in each of five assays, i.e., an expenditure of only ten tubes, one obtains an estimate of within-assay variance with five degrees of freedom and between-assay variance with four degrees of freedom. One strategem to initiate a quality control program, would be to analyze quality control samples let us say in quintuplicate in the first three assays, providing us with a total of 12 degrees of freedom of within-assay variability and two degrees of freedom of between-assay variability. Thereafter, the number of quality control samples analyzed within each assay could be decreased. However, the use of unequal numbers of replicates results in a great increase in the complexity of calculations, and, more importantly, a more complicated format for data entry.
6.2. Combining results from several QC samples * The present program enables us to obtain reasonably good estimates of within- and between-assay variability at any one dose level. However, since we may have quality control samples at three or even more dose levels, it would be desirable to combine these re* A computer program is available in PL/1 for an IBM 370 computer in BATCH mode with interactive input, or in FORTRAN for a CDC 3600 computer for time-share operation. The PL/1 version and its documentation is included in: Faden, V.B. and Rodbard, D. (1975), Radioimmunoassay Data Processing, Third Edition. National Technical Information Service, Report PB 246222 (Computer Magnetic Tape), PB 246224 (Printed Listings), Springfield, Virginia 22151.
B.F. McDonagh et al., Computerized approach to statistical quality control 70[-
/
',,, ~
',,, CV
20!
i--
70
I
i
Tri
,i /
,, \
/ ,,/
#
I
20
10
10
0 ............
....... Dose
10
I00 . . . . .
i
1000
I I /1/~ I //
/ ,,,'
//
•
,<
"/
~
I
1
//
,,,/,o \
ol\
l
187
./ ,," u
E O
/
/
/
10
//" / /
~/
D..
A
1
10
100
M e a n Potency on Previous Assays
%ov ,o1_ti
i.tt,.>
:ot4k 2ol-I
,oIl \
Iit=
~.
/ /-420
"-°--Jl
I _.<- i,o
.
y = % BIBo B
Fig. 2A. Plot of within- and between-assay %CV versus dose level, i.e., the average value for four dose levels. Also shown is the theoretical relationship for within-assay variance, based on pooled data from several standard curves. (2B) Same, plotted vs. mean response (Y = %B/Bo)level. suits. There are several ways to do this (cf. Appendix II of Rodbard, [14] ). One is to plot the percent coefficient o f variation, 7'~£~V(either within or between assays) versus logarithm of dose (fig. 2). Alternatively, one may be able to combine the graphs o f %CV (fig. 1D) versus assay number for each of the three dose levels. If the results are homogeneous, they may be pooled with a gain of total number of degrees of freedom. If they appear heterogeneous, then one may evaluate the relationship between %CV and log(X). Finally, one may transform the original data (as by use of log, square-root, Studentizing, Ridit or percentile transforms) in order to achieve better uniformity of variance. However, results of analyses of these transformed data must be transformed back into terms of the original measurements.
Fig. 3. One approach to combining information from several QC-samples. Ordinate: Log of mean values for four samples on today's assay. Abscissa: Log of mean values for the same samples on the 10 most recent assays. If the assay were in control, these values should be distributed randomly around the line of identity. One assay (circles) is in good control.
Another (triangles) can be modified by a simple multiplicative correction factor, since the slope of the log-log plot is compatible with unity. In a third assay (diamonds), the slope is significantly different than unity;hence no simple correction factor can be used. It is desirable to look at the interaction o f samples and assays in terms of variability. Thus, we might have a large between-assay variability because one sample went up, another sample stayed the same, and a third sample went down. Or, we could have the same degree of between-assay variability if all three of the samples moved systematically upward (or downward). Obviously the latter would be more indicative o f a systematic change (e.g., the standard preparation may have deteriorated, or the dilutions o f the standard curve may be erroneous, etc.). One approach to this is as follows: One constructs a graph showing the nominal value for each of the three samples on the alSscissa. A logarithmic scale is useful, especially when the values span more than one decade, provided that none o f the values is close to zero. These nominal values may be obtained as the means of all of the replicates on, let us say, the ten most recent assays. One then draws in the line of identity. After each assay is constructed, the
188
B.F. McDonagh et aL, Computerized approach to statistical quality control
means of the replicates for each sample are plotted on the ordinate, which is also on a log scale (cf. fig. 3). Ideally, all of these points would fall on the line of identity. Due to both within- and between-assay variability, they should fall randomly around the line of identity, assuming that the assay is in-control. One can construct upper and lower confidence limits around the line of identity. Then we would have the rule that if one or more of the three samples falls outside those confidence limits, we have a warning that the assay is out of control. If all three of the samples fall outside the confidence limits and in the same direction, we have a definite indication that there has been a change in the performance of the assay, and it is probably justifiable to reject this assay. Alternatively, a regression through these points could be utilized to provide a correction factor. The procedure is as follows: (1) Fit a straight line. (2) Test linearity. (3) Test whether the slope is compatible with a true slope of unity (1.0). (4) Re-fit the line, forcing it to have a slope of unity. (5) Calculate the adjustment factor and its standard error. This type of analysis is not restricted to any arbitrary number of quality control samples. Indeed, one could utilize ten (or more) widely spaced samples in this fashion. The more the samples, the better the performance of the system. In this manner, the QC-samples become a secondary standard, which may be more stable and reliable than the standard curve. 6.3. When is an assay out-of-control?
Utilizing the above approach one can easily construct a quality control chart for (BIT)o, slope, 50% intercept, minimal detectable concentration, residual variance around the logit-log or four parameter logistic regression line, and several other assay parameters. In addition, we have quality control charts indicating the within- and between-assay variability for perhaps three or more quality control samples. How do we combine
all this information in order to determine whether a giveri assay is in-control, or out-of-control? In other words, when do we reject an assay, or when do we adjust all of the values in an assay (say, by multiplying by a factor of two), in order to save the assay? This is the central issue. Since there are nearly an astronomical number of assays in an unlimited number of laboratories, it is impossible to provide a totally general answer. Each laboratory must establish its own acceptance criteria. There will always be a degree of arbitrariness involved in making these decisions. However, it is important that each laboratory formulates an objective, quantitative set of rules for rejection of an assay, and then to enforce these rules. An example of these rules is as follows: If three quality control samples have each been run in duplicate in each of every assay, one can utilize the median coefficient of variation for these three. This provides protection against outliers. If the median of the cumulative within-assay %CV exceeds ten-percent or some other arbitrary value, we can state that the quality of the assay system is unsatisfactory. 2. If the between-assay %CV for any one of the QCsamples exceeds 20%, we should be alerted to the fact that the assay may be out of control, or unsatisfactory for its intended use. If the median of the cumulative between-assay %CV (component) exceeds 15%, then we have an indication that the assay system is out of control. This provides Considerable protection against outliers so that even if the results from a single tube were drastically in error, thus resulting in a massive %CV for one of the assays, we would still have quite a reliable measure of between-assay variability.
Acknowledgements We wish to thank the following individuals for assistance with programming: J. Gurian, S. Knisley, M. Kreeger, R. Sheats, E. Levine, and D.M. Hutt. T. Sellner prepared the manuscript on the WYLBUR textediting system.
B.F. McDonagh et al., Computerized approach to statistical quality con trol
poses may be generalized to take into account the contribution of laboratories, laboratory-sample interaction and replicate effects. The practical side of this analysis has been presented by Youden and Steiner [18] and the theoretical side has been developed in many standard texts (e.g., Bennett and Franklin, [2] ), under the title of Two-Way Classification with Interaction, Components of Variance Model. The model is developed as follows. The observation X i j k obtained from the k th replicate of the jth sample in the ith laboratory with i = 1 ..... l,j = 1 ..... s, k = 1 ..... r, is assumed to be described by the model:
Table 2 Source
d.f.
SS
Total
N-1
~ x2 N-~
Between
n-1
SSB = Z l l , I ' l 1_N-~ 2 '\ ri /
,,,,,,,,,o
N_o
W
MS
E(MS)
MSB=
O2w + ko o2b
2
ij
ij-i \
k°=
r~-~ /
ss.
02
MSw= ~
189
N(n- 1)
X i j k = Id + ~i + l)j + )kij + eij k ,
Appendix I. Unequal number of replicates
where the ~i represent the laboratory effects (assumed random); ~j represnets the sample effects (again assumed random), Xii represents any variations which may be peculiar to a particular combination of laboratory and sample with E(~kij ) = 0 for each ij; and eli k are normally distributed components (again random). The analysis of variance for this model is presented in table 3.
The model described in the text for a one-way components of variance analysis may be readily generalized to include the case where there is an unequal number of replicates (Graybill, [8] ). We assume that the ith assay has r i replicates, so i = 1 ..... n, and / = I .... , ri, and the total number of observations is N, w
N = ~ ri . i=1
References
The analysis of variance is shown in table 2.
[ 1 ] R.N. Barnett, Clinical Laboratory Statistics (Little Brown and Co. 1971). [2] C.A, Bennett and N.L. Franklin, Statistical Analysis in Chemistry and the Chemical Industry (J. Wiley and Sons, NY, 1954). [3] R. Bliss, The Statistics of Bioassay (Academic Press, NY 1952). [4] G.S. Challand and T. Chard, Clin. Chim. Acta (1973) 133.
Appendix II. Between-laboratory comparison The development of any RIA method will most likely require s samples to be replicated r-times within each of the laboratories in a collaborative study, to assess precision and accuracy of the method. The statistical model presented above for quality control purTable 3 Source
df
SS
Total
N 1
~k{Xij k .~)2
Between Laborato= ies
~t- 1
s 1
Between
Samples
MS
E(MS)
SSL = rsYi {~i. _~)2
SSL MS L _- ~1-
°~ +' °I
SSS = r £ E (~.j_~}2
SSS MSs - s-1
LaboratorySample Interaction
(£vl)(s-1)
SSLs = r ~1 (~ii-'~i. _~ j +'~)2
MSLs
Within Samples & Laboratories
N-~s
SSw = i~k (Xijk -"xij )2
MSw
+rso~ +r~o~
SSLs (~-1)(s-1) SSw N-£s
o~
190
B.F. McDonagh et al., Computerized approach to statistical quality control
[5] A.J. Duncan, Quality Control and Industrial Statistics, 3rd ed., (R.D. Irwin, Inc., Homewood, IU., 1965). [6] D.J. Finney, Statistical Methods in Biological Assay, 2nd ed. (Griffin, London, 1963). [7] E.L. Grant, Statistical Quality Control, 2nd ed. (McGraw-Hill, NY, 1952). [8] F. Graybill, An Introduction to Linear Models, Vol. I (McGraw-Hill, NY, 1961). [9] E.K. Harris, J. Chron. Dis. (1976) 23. [10] E.K. Harris, P. Kanofsky, G. Shakarji and E. Cotlove, Clin. Chem. 16 (1970) 1022. [11] R.J. Henry and M. Sagelove, J. Clin. Path. 5 (1952) 305. [12] S. Levey and E.R. Jennings, Am. J. Clin. Path. 20 (1950) 1059,
[13] R.B. McHugh and P.W. Mielke, Am. J. Stat. Ass. 63 (1968) 1000. [14] D. Rodbard, Clin. Chem. 20 (1974) 1255. [15] D. Rodbard, P.L. Rayford, J.A. Cooper and G.T. Ross, J. Clin. Endocrinol. Metab. 28 (1968) 1412. [16] C.D. Russell, H.J. De Blanc and H.N. Wagner, J. Hopkins. Med. 35 (1974) 334. [171 G.Z. Williams, D.S. Young, M.R. Stein and E. Cotlove, Clin. Chem. 16 (1970) 1016. [181 W.J. Younden and E.H. Steiner, (Stat. Man. Ass. Off. Analyt. Chem., Ass. Off. Anal. Chem., Washington, DC 1975).