- Email: [email protected]
Statistical power for commutability testing in the presence of random, sample-related effects by use of the EP14 protocol
Clinica Chimica Acta 413 (2012) 1710–1711
Contents lists available at SciVerse ScienceDirect
Clinica Chimica Acta journal homepage: www.elsevier.com/locate/clinchim
Letter to the editor Statistical power for commutability testing in the presence of random, sample-related effects by use of the EP14 protocol
Abbreviations: OLR; ordinary linear regression CVa; analytical CV CVsr; sample-related effect CV DR; Deming regression Keywords: Lack of fit error Virtual sample pooling Matrix effects Prediction interval
Dear editor Commutability testing is of particular importance in the area of external quality assessment [1], internal quality control, and standardization [2]. While different experimental and statistical approaches for commutability testing are in use [2], power calculations have been reported only for the EP14 guideline from the Clinical and Laboratory Standards Institute [3,4]. This protocol recommends testing of materials with 2 methods, measurement of the materials in parallel with at least 20 native samples, in triplicate, preferably within 1 run [4]. Commutability is investigated by use of the prediction interval obtained from ordinary linear regression (OLR). It was shown that the statistical power was most influenced by the lack of fit and the pure random error
[3]. The publication stated that “the lack of fit error is an uncontrollable source of error, that is, no amount of retesting or replicate testing will reduce the influence of this factor” [3]. However, we recently showed that this amount of error could be reduced by virtual sample pooling when so-called random, sample-related effects are the cause of the lack of fit error [5]. Virtual sample pooling implies that means of ranked sample concentrations are calculated, i.e. for pooling with n = 4, groups with 4 consecutive samples are averaged. Here, we expand the power calculations for the EP14 protocol to situations when sample-related effects are present. We used IBM SPSS Statistics 19 to simulate 1000 data sets for each investigated situation with the reviewed error structure in y, corresponding to the test method. Analytical and sample-related effect CV were both set at 1% (CVa and CVsr). The simulated data in x, corresponding to the reference method, were uniformly distributed [2.0, 3.0]. We estimated the effects that number of samples, replicates and virtual pooling of samples have on reducing the total error for each of the investigated situations by the mean of the 1000 simulated total errors. We used this estimate of total error to calculate the power by the method described in Ref. [3] for each situation and for different levels of bias by using Microsoft Excel for Mac 2011. Fig. 1 compares power curves with (blue lines) and without (red and black lines) virtual pooling of samples. It is apparent that replication beyond the standard protocol (n = 3) does not result in considerably increased power when the sample-related effects are in the order of the analytical imprecision (compare red lines with black lines). Virtual pooling of samples, however, has a marked effect on the power (compare blue lines with red lines). Thus, sample sizes of n > 40 and virtual pooling may be beneficial for uncovering matrix effects, in contrast to the recommendations given in the original publication [3]. Such sample sizes are typically available in international standardization projects. To test the effect in a situation when analytical errors in both the x and y methods are present, we did power calculations using Deming regression (DR) (power curves not shown). The same effect was observed as in the OLR case. However, it is a fact that with DR, sample-related effects can only be observed, without the possibility to attribute them to the method x, y or both. Finally, we recommend power calculations in commutability experiments because otherwise large prediction intervals that do not allow meaningful conclusions about sample commutability may be obtained. References
Fig. 1. Power curves illustrating influence of virtual pooling of samples and replication (both CVa and CVsr = 1%) (from left to right): 100 (60) samples, 3 replicates, 5 (3) pooling [blue solid (long dashed) line]; 20 samples, 15 (9) replicates, no pooling [red solid (long dashed) line]; 100 (60, 20) samples, 3 replicates, no pooling [black solid (long, short dashed) line].
0009-8981/$ – see front matter © 2012 Elsevier B.V. All rights reserved. doi:10.1016/j.cca.2012.05.002
[1] Miller WG, Jones GR, Horowitz GL, Weykamp C. Proficiency testing/external quality assessment: current challenges and future directions. Clin Chem 2011;57:1670–80. [2] Vesper HW, Miller WG, Myers GL. Reference materials and commutability (Review). Clin Biochem Rev 2007;28:139–47. [3] Long T. Statistical power in the detection of matrix effects. Arch Pathol Lab Med 1993;117:387–92. [4] Clinical and Laboratory Standards Institute. CLSI/NCCLS approved guideline EP14–A2. Evaluation of Matrix Effects. Wayne, Pennsylvania: CLSI; 2005. [5] Stöckl D, Stepman HCM, Van Houcke SK, Thienpont LM. Importance of sample-related effects for commutability testing according to the EP14 protocol. Clin Chim Acta 2010;411:1378–9.
Letter to the editor
Thomas H. Røraas Norsk kvalitetsforbedring av laboratorievirksomhet utenfor sykehus (“NOKLUS”), Haraldsplass Diakonale Sykehus AS, Ulriksdal 8, Bergen, Norway Sofie K. Van Houcke Laboratory for Analytical Chemistry, Faculty of Pharmaceutical Sciences, Ghent University, Harelbekestraat 72, Ghent, Belgium Dietmar Stöckl STT Consulting, Horebeke, Belgium
1711
Linda M. Thienpont Laboratory for Analytical Chemistry, Faculty of Pharmaceutical Sciences, Ghent University, Harelbekestraat 72, Ghent, Belgium Corresponding author at: Harelbekestraat 72, B-9000 Ghent, Belgium. Tel.: +32 9 264 81 04; fax: +32 9 264 81 98. E-mail address: [email protected]. 1 March 2012
Contents lists available at SciVerse ScienceDirect
Clinica Chimica Acta journal homepage: www.elsevier.com/locate/clinchim
Letter to the editor Statistical power for commutability testing in the presence of random, sample-related effects by use of the EP14 protocol
Abbreviations: OLR; ordinary linear regression CVa; analytical CV CVsr; sample-related effect CV DR; Deming regression Keywords: Lack of fit error Virtual sample pooling Matrix effects Prediction interval
Dear editor Commutability testing is of particular importance in the area of external quality assessment [1], internal quality control, and standardization [2]. While different experimental and statistical approaches for commutability testing are in use [2], power calculations have been reported only for the EP14 guideline from the Clinical and Laboratory Standards Institute [3,4]. This protocol recommends testing of materials with 2 methods, measurement of the materials in parallel with at least 20 native samples, in triplicate, preferably within 1 run [4]. Commutability is investigated by use of the prediction interval obtained from ordinary linear regression (OLR). It was shown that the statistical power was most influenced by the lack of fit and the pure random error
[3]. The publication stated that “the lack of fit error is an uncontrollable source of error, that is, no amount of retesting or replicate testing will reduce the influence of this factor” [3]. However, we recently showed that this amount of error could be reduced by virtual sample pooling when so-called random, sample-related effects are the cause of the lack of fit error [5]. Virtual sample pooling implies that means of ranked sample concentrations are calculated, i.e. for pooling with n = 4, groups with 4 consecutive samples are averaged. Here, we expand the power calculations for the EP14 protocol to situations when sample-related effects are present. We used IBM SPSS Statistics 19 to simulate 1000 data sets for each investigated situation with the reviewed error structure in y, corresponding to the test method. Analytical and sample-related effect CV were both set at 1% (CVa and CVsr). The simulated data in x, corresponding to the reference method, were uniformly distributed [2.0, 3.0]. We estimated the effects that number of samples, replicates and virtual pooling of samples have on reducing the total error for each of the investigated situations by the mean of the 1000 simulated total errors. We used this estimate of total error to calculate the power by the method described in Ref. [3] for each situation and for different levels of bias by using Microsoft Excel for Mac 2011. Fig. 1 compares power curves with (blue lines) and without (red and black lines) virtual pooling of samples. It is apparent that replication beyond the standard protocol (n = 3) does not result in considerably increased power when the sample-related effects are in the order of the analytical imprecision (compare red lines with black lines). Virtual pooling of samples, however, has a marked effect on the power (compare blue lines with red lines). Thus, sample sizes of n > 40 and virtual pooling may be beneficial for uncovering matrix effects, in contrast to the recommendations given in the original publication [3]. Such sample sizes are typically available in international standardization projects. To test the effect in a situation when analytical errors in both the x and y methods are present, we did power calculations using Deming regression (DR) (power curves not shown). The same effect was observed as in the OLR case. However, it is a fact that with DR, sample-related effects can only be observed, without the possibility to attribute them to the method x, y or both. Finally, we recommend power calculations in commutability experiments because otherwise large prediction intervals that do not allow meaningful conclusions about sample commutability may be obtained. References
Fig. 1. Power curves illustrating influence of virtual pooling of samples and replication (both CVa and CVsr = 1%) (from left to right): 100 (60) samples, 3 replicates, 5 (3) pooling [blue solid (long dashed) line]; 20 samples, 15 (9) replicates, no pooling [red solid (long dashed) line]; 100 (60, 20) samples, 3 replicates, no pooling [black solid (long, short dashed) line].
0009-8981/$ – see front matter © 2012 Elsevier B.V. All rights reserved. doi:10.1016/j.cca.2012.05.002
[1] Miller WG, Jones GR, Horowitz GL, Weykamp C. Proficiency testing/external quality assessment: current challenges and future directions. Clin Chem 2011;57:1670–80. [2] Vesper HW, Miller WG, Myers GL. Reference materials and commutability (Review). Clin Biochem Rev 2007;28:139–47. [3] Long T. Statistical power in the detection of matrix effects. Arch Pathol Lab Med 1993;117:387–92. [4] Clinical and Laboratory Standards Institute. CLSI/NCCLS approved guideline EP14–A2. Evaluation of Matrix Effects. Wayne, Pennsylvania: CLSI; 2005. [5] Stöckl D, Stepman HCM, Van Houcke SK, Thienpont LM. Importance of sample-related effects for commutability testing according to the EP14 protocol. Clin Chim Acta 2010;411:1378–9.
Letter to the editor
Thomas H. Røraas Norsk kvalitetsforbedring av laboratorievirksomhet utenfor sykehus (“NOKLUS”), Haraldsplass Diakonale Sykehus AS, Ulriksdal 8, Bergen, Norway Sofie K. Van Houcke Laboratory for Analytical Chemistry, Faculty of Pharmaceutical Sciences, Ghent University, Harelbekestraat 72, Ghent, Belgium Dietmar Stöckl STT Consulting, Horebeke, Belgium
1711
Linda M. Thienpont Laboratory for Analytical Chemistry, Faculty of Pharmaceutical Sciences, Ghent University, Harelbekestraat 72, Ghent, Belgium Corresponding author at: Harelbekestraat 72, B-9000 Ghent, Belgium. Tel.: +32 9 264 81 04; fax: +32 9 264 81 98. E-mail address: [email protected]. 1 March 2012