- Email: [email protected]
Use of error grid analysis to evaluate acceptability of a point of care prothrombin time meter
Clinica Chimica Acta 411 (2010) 131–134
Contents lists available at ScienceDirect
Clinica Chimica Acta j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / c l i n c h i m
Use of error grid analysis to evaluate acceptability of a point of care prothrombin time meter John R. Petersen a,⁎, Hans M. Vonmarensdorf a, Heidi L. Weiss b, M. Tarek Elghetany a a b
University of Texas Medical Branch, Galveston, Texas, United States University of Kentucky, Lexington, KY, United States
a r t i c l e
i n f o
Article history: Received 20 May 2009 Received in revised form 1 September 2009 Accepted 9 November 2009 Available online 12 November 2009 Keywords: Coagulation monitoring Error grid analysis Point of care testing POCT PT INR
a b s t r a c t Background: Statistical methods (linear regression, correlation analysis, etc.) are frequently employed in comparing methods in the central laboratory (CL). Assessing acceptability of point of care testing (POCT) equipment, however, is more difficult because statistically significant biases may not have an impact on clinical care. We showed how error grid (EG) analysis can be used to evaluate POCT PT INR with the CL. Materials and methods: We compared results from 103 patients seen in an anti-coagulation clinic that were on Coumadin maintenance therapy using fingerstick samples for POCT (Roche CoaguChek XS and S) and citrated venous blood samples for CL (Stago STAR). To compare clinical acceptability of results we developed an EG with zones A, B, C and D. Results: Using 2nd order polynomial equation analysis, POCT results highly correlate with the CL for CoaguChek XS (R2 = 0. 955) and CoaguChek S (R2 = 0. 93), respectively but does not indicate if POCT results are clinically interchangeable with the CL. Using EG it is readily apparent which levels can be considered clinically identical to the CL despite analytical bias. Conclusion: We have demonstrated the usefulness of EG in determining acceptability of POCT PT INR testing and how it can be used to determine cut-offs where differences in POCT results may impact clinical care. © 2009 Elsevier B.V. All rights reserved.
1. Introduction Treatment of patients with oral anticoagulants, such as warfarin or Coumadin, is necessary to prevent thromboembolic events in the treatment of a number of clinical conditions [1]. However, the therapeutic effect of a dose of Coumadin given to a patient can be highly variable depending upon genetics, diet, and medications prescribed for other co-morbid conditions. According to the most recent recommendations of the American College of Chest Physicians, changes in warfarin dosage depend on 2 factors: INR values and the presence of bleeding [2]. Thus, regular monitoring of the prothrombin time (PT) international normalized ratio (INR) is necessary to minimize both the complications of bleeding and thromboembolic events if the PT INR is not kept within a narrow therapeutic range. A number of studies involving patient self monitoring and point of care (POCT) testing have shown positive patient outcomes with patients more often in the therapeutic range [3–7] while other studies have questioned the accuracy of POCT PT INR meters [8–10]. Although POCT PT INR has been shown to be useful, the results should be comparable to the central laboratory, at least in the therapeutic range
⁎ Corresponding author. University of Texas Medical Branch, Galveston, TX 775550551, United States. Tel.: +1 409 772 1350. E-mail address: [email protected] (J.R. Petersen). 0009-8981/$ – see front matter © 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.cca.2009.11.010
(PT INR b4.0). Determining the acceptability of a new method usually depends upon simple statistical tools, such as Bland Altman plots, linear regression, and correlation analysis. While use of these methods is adequate when performing method comparison in the clinical laboratory (CL); using them to determine the clinical acceptability of POCT meters is often more difficult. This is because statistically significant analytical biases may not alter the clinical impression and thus have no impact on clinical care. Change in warfarin dosage is a clinical decision, which among other factors depends on the patient's underlying medical condition. Anderson et al. attempted to deal with these differences by developing additional criteria that tried to account for the clinical impression along with analytical biases [11]. While useful, their criteria requires detailed analysis and may not be stringent enough, especially when the PT INR is b3.0. Another method that is useful in comparing POCT to the CL is error grid (EG) analysis. While developed to determine acceptability of POCT glucose meters [12,13] EG analysis has not been used routinely in the evaluation of other POCT tests. The only exception is a recent study in which EG analysis was used to evaluate the acceptability of two PT INR meters for patient self monitoring [14]. During our evaluation of the CoaguCheck XS (Roche Diagnostics, Indianapolis, IN) as a potential replacement for the CoaguCheck S (Roche) we had the opportunity to use EG analysis to compare the results of the POCT meters with the results obtained from the CL (Stago STAR, Parsippany, NJ).
132
J.R. Petersen et al. / Clinica Chimica Acta 411 (2010) 131–134
2. Materials and methods 2.1. PT INR measurements Central laboratory analysis was performed on a Stago STAR (Stago, ISI = 1.29) using 3.2% citrated platelet-poor plasma. Samples were processed and analyzed within 8 h of collection. The CoaguChek S® (Roche) device used recombinant thromboplastin (ISI = 1.0) with clot formation detected by iron particles that are moved by an alternating electrical field. The CoaguChek XS (Roche Diagnostics, Indianapolis, IN) device used recombinant thromboplastin (ISI = 1.0) and employs electrochemical current detection to measure clot formation. 2.2. Study participants The study was approved by the University of Texas Medical Branch (UTMB) Institutional Review Board and conducted in the anticoagulation and outpatient clinics at UTMB. The clinics have a high volume of patients that are routinely evaluated for long term anticoagulation therapy. Patients were included in the study (103 for Stago STAR, 99 for CoaguChek XS and 64 for CoaguChek S) as they presented to the clinic and gave permission that in addition to a finger stick they were also willing to have drawn a tube of blood (3.2% Sodium citrate) that was sent to the main Hematology laboratory for PT INR analysis. Patients that were not on Coumadin therapy (20 for Stago STAR and CoaguChek XS and 14 for CoaguChek S) were considered normal and were included in the study if they gave permission to have a finger stick and to have drawn a tube of citrated blood that was sent to the main Hematology laboratory for analysis. Blood for analysis in the central laboratory was drawn within 15 min of finger stick analysis. No patients were excluded from the study due to extremely high PT-INR. 2.3. Statistical analysis The EG was developed in conjunction with a Hematopathologist (MTE) and an Internal Medicine physician (HMV) with expertise in anti-coagulation therapy. The EG that was developed was based entirely on expert opinion and has had no performance-based validation. Using the EG that had been developed to evaluate POCT blood glucose monitors as the model [7,8] the following EG zones developed: Zone A - Results that may not be identical but the difference between the methods (defined as Clinical Laboratory PT-INR ± 15%) will have no effect on the required clinical action. In other words the results are clinically identical. Zone B - The results are substantially different, however, any altered clinical action will have little or no impact on a patient's clinical outcome (defined as Clinical Laboratory PT-INR ± 15–25% for PT INR values b4.0). For example, corrective action, while required, is in the correct direction but will be slightly greater or less than required. Zone C - The results are substantially different resulting in an altered clinical action that could have a significant medical risk. This could lead to under or over dosing the patient causing a patient to be at risk of bleeding complications (over anticoagulated) or thromboembolic events (under anticoagulation). Zone D - The results are substantially different resulting in altered clinical action possibly having dangerous and life threatening medical risk. The difference between the methods is great enough that severe over or under anti-coagulation has occurred and the patient is at a significantly increased risk of bleeding complications or thromboembolic events.
The statistical differences between the PT INR devices and the CL were assessed by the Wilcoxon rank sum test for paired samples using MedCalc (Mariakerke, Belgium). We also utilized McNemar's test and the κ statistic to measure the degree of agreement between the CL vs. CoaguChek XS® and CoaguChek S®. Specifically, we utilized the mean± SD PT INR values from CL and dichotomized both the CoaguChek XS® and CoaguChek S® PT INR values as within or beyond the Zone A boundary from the CL data. We also performed power calculations based on the κ statistic. 3. Results As part of this study, patients had capillary whole blood PT INR determinations using POCT instrumentation (CoaguChek XS and/or CoaguChek S) followed by venous blood sampling for PT INR determination in the CL (Stago STAR). Of the 103 patient samples (20 normal and 83 on Coumadin therapy) sent to the CL for PT INR determination, 99 were tested using the CoaguChek XS meter, 64 were tested using the CoaguChek S meter, and 56 were tested using both POCT meters. The CL PT INR values for the samples ranged from 0.9 to 7.0. Table 1 shows the mean (SD) for CL PT INR ranging from ≤3.0, N3.0–4.0, and N4.0. For a CL PT INR ≤3.0 the mean bias between the CoaguChek S and the CoaguChek XS relative to the CL was 0.06 (p = 0.63) and 0.09 (p = 0.003) PT INR units, respectively. The differences between the CoaguChek XS and CoaguChek S were not statistically significant (p N 0.1), however, the CoaguChek S showed increased variability as compared to the CoaguChek XS relative to the CL values (Sy/x = 0.25 vs. 0.13, respectively). The same trend is also seen for all PT INR ranges. Although both the CoaguChek XS and CoaguChek S compare reasonably well with the CL when the CL PT INR b3.0, when the CL PT INR is N3.0 overestimation of the CL values is apparent with both meters. Both the increased variability for the CoaguChek S and the overestimation for both meters with increasing PT INR are readily apparent by inspection of the respective EG (Fig. 1A–B). Since the differences between the CoaguChek XS and CoaguChek S appear to be curvilinear the relationship was expressed as a 2nd order polynomial equation: y = − 0.02 x2 + 1.40 x − 0.55; R2 = 0.955 and y = 0.15 x2 + 0.56 x + 0.23; R2 = 0.93 for CoaguChek XS and CoaguChek S, respectively. In order to better assess the clinical significance of differences between the POCT and CL PT INR results we applied our proposed
Table 1 Mean PT INR for Stago STAR, CoaguChek S, and CoaguChek XS for patients whose PT INR for Stago STAR was ≤3.0, 3.1–4.0, N 4.0.
Stago STAR PT INR (SD) N CoaguChek S PT INR (SD) Sy/x N p (Wilcoxon) (vs. Stago STAR) κ Coefficient H0: κ = 0 p (McNemar's test of equality of proportions) CoaguChek XS PT INR (SD) Sy/x N p (Wilcoxon) (vs. Stago STAR) κ Coefficient H0: κ = 0 p (McNemar's test of equality of Proportions)
Stago STAR
Stago STAR
Stago STAR
PT INR ≤3.0
PT INR N3.0–4.0
PT INR N4.0
1.79 (0.64) 57
3.6 (0.27) 22
4.8 (0.76) 24
1.85 (0.74) 0.25 41 0.47 0.66 b 0.0001 NS
4.3 (0.81) 0.52 13 0.014 0.64 0.022 NS
5.94 (1.11) 0.95 10 0.002 Not estimable
1.88 (0.70) 0.13 54 0.009 0.82 b 0.0001 NS
4.18 (0.47) 0.31 22 b0.001 0.10 0.52 0.011
5.92 (0.83) 0.54 23 b 0.001 0.12 0.22 0.0003
J.R. Petersen et al. / Clinica Chimica Acta 411 (2010) 131–134
133
Table 2 Number of patients and % of patients within Zones A, B, and C for CoaguChek S and CoaguChek XS at various PT INR levels. PT INR ≤ 3.0 Zone A
Zone B
CoaguChek S N 34 7 % 82.9% 17.1% CoaguChek XS N 53 1 % 98.1% 1.9%
PT INR N 4.0
PT INR 3.1–4.0 Zone C
Zone A
Zone B
Zone C
Zone A
Zone B
Zone C
0 0.0%
7 53.8%
3 30.8%
3 15.4%
2 20.0%
5 50.0%
3 30.0%
0 0.0%
15 68.2%
7 31.8%
0 0.0%
7 30.4%
16 72.7%
0 0.0%
results that fell outside of error zone A when the CL PT INR was ≤3.0 with 17.1% falling outside of risk Zone A (17.1% risk zone B) vs. 1.9% falling outside risk Zone A for CoaguChek XS. This trend is even more apparent for a CL PT INR N3.0 in which 60.9% (34.8% in risk Zone B and 26.1% in risk Zone C) vs. 51.1% (all in risk Zone B) that fall outside risk Zone A for CoaguChek S and CoaguChek XS, respectively. In order to determine if we had an adequate number of samples on which to base our conclusions we performed power calculations based on the κ statistic. Assuming a null hypothesis equal to 0 vs. a κ statistic equal to 0.65 (as seen Table 1), we have over 95% power in detecting this κ value with 40 samples (CL PT INR values ≤3.0 and 3–4) vs. the CoaguChek S. Similarly using a κ statistic equal to 0.80 (as seen in Table 1), we have over 95% power in detecting this κ value with 55 samples (CL PT INR values ≤3.0) vs. the CoaguChek XS. 4. Discussion
Fig. 1. Error-grid for PT INR comparison of CoaguChek S (A) and CoaguChek XS (B) vs. Stago STAR. The grid shows risk zones A to D with increasing clinical relevance of disagreement between the measurements. The dashed lines are the line of identity for the analysis. The solid lines are the 2nd order polynomial equations: y = − 0.02 x2 + 1.40 x − 0.55; R2 = 0.955 and y = 0.15 x2 + 0.56 x + 0.23; R2 = 0.93 for CoaguChek XS and CoaguChek S, respectively.
error grid (Fig. 1 A–B) and plotted the measurements for CoaguChek XS and CoaguChek S vs. the CL. Our use of CL PT INR ± 15% as clinically acceptable for zone A is supported by a very recent study that indicated a deviation of 15% at an INR of 2.5 would probably have no clinical impact in terms of increased thromboembolism or bleeding events in patients using POCT meters [15]. As shown by Fig. 1 and in Table 2 all CoaguChek XS and CoaguChek S results were within zones A and B for CL PT INR values ≤3.0. This indicates that for PT INR values b3.0 the CoaguChek XS and CoaguChek S values could replace the CL values with no or little impact on a patient's clinical outcome. This is also shown by the excellent agreement (κ coefficient = 0.82) between the CL vs. XS and good agreement between the CL vs. S (κ coefficient = 0.66 and 0.64, respectively). The corresponding p-values for the κ coefficients indicate that there is a statistically significant correlation between these methods. In addition the McNemar's test shows that there are no significant differences in proportions between the CL vs. CoaguChek XS and CoaguChek S when the CL PT INR is ≤3. Above a CL PT INRs N3.0 it is readily apparent that both the CoaguChek XS and CoaguChek S give results that are significantly higher than the CL. Closer inspection of the EG and Table 2 also reveals that the CoaguChek S had a higher proportion (both higher and lower) of
The use of POCT PT INR to monitor patient anti-coagulation has been established by a number of studies [2–6], although others have questioned if POCT meters have adequate accuracy to monitor and regulate Coumadin dosages relative to a CL [8–10]. Normally, POCT meters have been compared to the CL methods using statistical methods developed for highly accurate CL methods that may indicate statistically significant differences but may not be relevant when making the dosing decisions. Although Bland–Altman plots give the impression of potential bias and variability of the methods being compared, the clinical acceptability of the differences between methods is not readily apparent. On the other hand EG analysis, which has been used to determine the clinical significance of inaccuracies when evaluating POCT glucose meters, provides a straight forward representation of the clinical importance of analytical differences in methods by ascertaining the potential risk of bleeding or thrombotic complications when using the POCT results to make treatment decisions as compared to results from the CL. By simple visual inspection of the EG it is obvious as to whether a new POCT PT INR meter is acceptable or not. If a more quantitative measure is desired identification of the number of values in each error zone such as shown in Table 2 would be an easy solution. Using these methods it is apparent that the CoaguChek XS performs better than the CoaguChek S as compared to the CL. Similar results have been previously been identified when comparing the CoaguChek XS and CoaguChek S [16,17]. By using the EG analysis (Fig. 1 A–B) and Table 2 it is evident that for PT INR ≤3.0 the results from the CoaguChek XS compares better to the CL with only 1 (1.9%) result outside of error zone A as compared to 7 (17.1%) seen using the CoaguChek S. Above INR N3.0 the high bias inherent in both instruments means that N50% of the values are outside of zone A for both meters. Additionally 6 (21.4%) were in zone C for CoaguChek S which could put the patient at significant risk. The analysis of errors in POCT PT INR is a complicated problem because the clinical consequence of any error not only depends on the absolute difference between the POCT and the CL result but also the relative deviation. Thus we conclude that EG analysis when used in
134
J.R. Petersen et al. / Clinica Chimica Acta 411 (2010) 131–134
conjunction with other statistical tools, such as regression analysis, could be an extremely useful tool in determining the clinical acceptability of new POCT PT INR methods. The analysis of errors in POCT PT INR is a complicated problem because the clinical consequence of any error not only depends on the absolute difference between the POCT and the CL result but also the relative deviation. Thus we conclude that EG analysis when used in conjunction with other statistical tools, such as regression analysis, is an extremely useful tool in determining the clinical acceptability of new POCT PT INR methods. In addition EG analysis would also provide useful information in the comparison of new CL methods. We also raise some caution regarding whether immediate action, including the need for patients to see physicians urgently, can be reliably decided upon using POCT INR results performed at home.
References [1] Hirsh J, Dalen JE, Anderson DR, et al. Oral anticoagulants: mechanism of action, clinical effectiveness, and optimal therapeutic range. Chest 1998;114:445S–69S. [2] Ansell J, Hirsh J, Hylek E, Jacobson A, Crowther M, Palareti G. Pharmacology and management of the vitamin K antagonists: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition). Chest 2008;133:160S–98S Suppl. [3] Fitzmaurice DA, Murray ET, McCahon D, et al. Self management of oral anticoagulation: randomised trial. BMJ 2005;331:1057–60. [4] Cromheecke ME, Levi M, Colly LP, et al. Oral anticoagulation self-management and management by a specialist anticoagulation clinic: a randomized cross-over comparison. Lancet 2000;356:97–102. [5] Gardiner C, Williams K, Mackie IJ, Machin SJ, Cohen H. Can oral anticoagulation be managed using telemedicine and patient self-testing? A pilot study. Clin Lab Haematol 2006;28:122–5.
[6] Gosselin R, Owings JT, White RH, et al. A comparison of point-of-care instruments designed for monitoring oral anticoagulation with standard laboratory methods. Thromb Haemost 2000;83:698–703. [7] Gill JM, Landis MK. Benefits of a mobile, point-of-care anticoagulation therapy management program. Jt Comm J Qual Improv 2002;28:625–30. [8] Loebstein R, Kurnik D, Lubetsky A, Ezra D, Halkin H. Potential dosing errors using portable prothrombin time monitoring devices. Blood Coagul Fibrinolysis 2003;14:479–83. [9] McBane R, Felty C, Hartgers M, et al. Importance of device evaluation for point-ofcare prothrombin time international normalized ratio testing programs. Mayo Clin Proc 2005;80:181–6. [10] Moore G, Henley A, Cotton S, et al. Clinically significant differences between pointof-care analysers and a standard analyser for monitoring the international normalized ratio in oral anticoagulation therapy: a multi-instrument evaluation in a hospital outpatient setting. Blood Coagul Fibrinolysis 2007;18:287–92. [11] Anderson DR, Harrison L, Hirsh J. Evaluation of a portable prothrombin time monitor for home use by patients who require long-term oral anticoagulant therapy. Arch Intern Med 1993;153:1441–7. [12] Clarke WL, Cox D, Gonder-Frederick LA, Carter W, Pohl SL. Evaluating clinical accuracy of systems for self-monitoring of blood glucose. Diabetes Care 1987;10:622–8. [13] Parkes JL, Slatin SL, Pardo S, Ginsberg BH. A new consensus error grid to evaluate the clinical significance of inaccuracies in the measurement of blood glucose. Diabetes Care 2000;23:1143–8. [14] Hemkens LG, Hilden KM, Hartschen S, et al. A randomized trial comparing INR monitoring devices in patients with anticoagulation self-management: evaluation of a novel error-grid approach. J Thromb Thrombolysis 2008;26:22–30. [15] Christensen TD, Larsen TB, Jensen C, Maegaard M, Sørensen B. International normalised ratio (INR) measured on the CoaguChek S and XS compared with the laboratory for determination of precision and accuracy. Thromb Haemost 2009;101:563–9. [16] Williams VK, Griffiths AB. Acceptability of CoaguChek S and CoaguChek XS generated international normalized ratios against a laboratory standard in a paediatric setting. Pathology 2007;39:575–9. [17] Karon BS, McBane RD, Chaudhry R, Beyer LK, Santrach PJ. Accuracy of capillary whole blood international normalized ratio on the CoaguChek S, CoaguChek XS, and i-STAT 1 point-of-care analyzers. Am J Clin Pathol 2008;130:88–92.
Contents lists available at ScienceDirect
Clinica Chimica Acta j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / c l i n c h i m
Use of error grid analysis to evaluate acceptability of a point of care prothrombin time meter John R. Petersen a,⁎, Hans M. Vonmarensdorf a, Heidi L. Weiss b, M. Tarek Elghetany a a b
University of Texas Medical Branch, Galveston, Texas, United States University of Kentucky, Lexington, KY, United States
a r t i c l e
i n f o
Article history: Received 20 May 2009 Received in revised form 1 September 2009 Accepted 9 November 2009 Available online 12 November 2009 Keywords: Coagulation monitoring Error grid analysis Point of care testing POCT PT INR
a b s t r a c t Background: Statistical methods (linear regression, correlation analysis, etc.) are frequently employed in comparing methods in the central laboratory (CL). Assessing acceptability of point of care testing (POCT) equipment, however, is more difficult because statistically significant biases may not have an impact on clinical care. We showed how error grid (EG) analysis can be used to evaluate POCT PT INR with the CL. Materials and methods: We compared results from 103 patients seen in an anti-coagulation clinic that were on Coumadin maintenance therapy using fingerstick samples for POCT (Roche CoaguChek XS and S) and citrated venous blood samples for CL (Stago STAR). To compare clinical acceptability of results we developed an EG with zones A, B, C and D. Results: Using 2nd order polynomial equation analysis, POCT results highly correlate with the CL for CoaguChek XS (R2 = 0. 955) and CoaguChek S (R2 = 0. 93), respectively but does not indicate if POCT results are clinically interchangeable with the CL. Using EG it is readily apparent which levels can be considered clinically identical to the CL despite analytical bias. Conclusion: We have demonstrated the usefulness of EG in determining acceptability of POCT PT INR testing and how it can be used to determine cut-offs where differences in POCT results may impact clinical care. © 2009 Elsevier B.V. All rights reserved.
1. Introduction Treatment of patients with oral anticoagulants, such as warfarin or Coumadin, is necessary to prevent thromboembolic events in the treatment of a number of clinical conditions [1]. However, the therapeutic effect of a dose of Coumadin given to a patient can be highly variable depending upon genetics, diet, and medications prescribed for other co-morbid conditions. According to the most recent recommendations of the American College of Chest Physicians, changes in warfarin dosage depend on 2 factors: INR values and the presence of bleeding [2]. Thus, regular monitoring of the prothrombin time (PT) international normalized ratio (INR) is necessary to minimize both the complications of bleeding and thromboembolic events if the PT INR is not kept within a narrow therapeutic range. A number of studies involving patient self monitoring and point of care (POCT) testing have shown positive patient outcomes with patients more often in the therapeutic range [3–7] while other studies have questioned the accuracy of POCT PT INR meters [8–10]. Although POCT PT INR has been shown to be useful, the results should be comparable to the central laboratory, at least in the therapeutic range
⁎ Corresponding author. University of Texas Medical Branch, Galveston, TX 775550551, United States. Tel.: +1 409 772 1350. E-mail address: [email protected] (J.R. Petersen). 0009-8981/$ – see front matter © 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.cca.2009.11.010
(PT INR b4.0). Determining the acceptability of a new method usually depends upon simple statistical tools, such as Bland Altman plots, linear regression, and correlation analysis. While use of these methods is adequate when performing method comparison in the clinical laboratory (CL); using them to determine the clinical acceptability of POCT meters is often more difficult. This is because statistically significant analytical biases may not alter the clinical impression and thus have no impact on clinical care. Change in warfarin dosage is a clinical decision, which among other factors depends on the patient's underlying medical condition. Anderson et al. attempted to deal with these differences by developing additional criteria that tried to account for the clinical impression along with analytical biases [11]. While useful, their criteria requires detailed analysis and may not be stringent enough, especially when the PT INR is b3.0. Another method that is useful in comparing POCT to the CL is error grid (EG) analysis. While developed to determine acceptability of POCT glucose meters [12,13] EG analysis has not been used routinely in the evaluation of other POCT tests. The only exception is a recent study in which EG analysis was used to evaluate the acceptability of two PT INR meters for patient self monitoring [14]. During our evaluation of the CoaguCheck XS (Roche Diagnostics, Indianapolis, IN) as a potential replacement for the CoaguCheck S (Roche) we had the opportunity to use EG analysis to compare the results of the POCT meters with the results obtained from the CL (Stago STAR, Parsippany, NJ).
132
J.R. Petersen et al. / Clinica Chimica Acta 411 (2010) 131–134
2. Materials and methods 2.1. PT INR measurements Central laboratory analysis was performed on a Stago STAR (Stago, ISI = 1.29) using 3.2% citrated platelet-poor plasma. Samples were processed and analyzed within 8 h of collection. The CoaguChek S® (Roche) device used recombinant thromboplastin (ISI = 1.0) with clot formation detected by iron particles that are moved by an alternating electrical field. The CoaguChek XS (Roche Diagnostics, Indianapolis, IN) device used recombinant thromboplastin (ISI = 1.0) and employs electrochemical current detection to measure clot formation. 2.2. Study participants The study was approved by the University of Texas Medical Branch (UTMB) Institutional Review Board and conducted in the anticoagulation and outpatient clinics at UTMB. The clinics have a high volume of patients that are routinely evaluated for long term anticoagulation therapy. Patients were included in the study (103 for Stago STAR, 99 for CoaguChek XS and 64 for CoaguChek S) as they presented to the clinic and gave permission that in addition to a finger stick they were also willing to have drawn a tube of blood (3.2% Sodium citrate) that was sent to the main Hematology laboratory for PT INR analysis. Patients that were not on Coumadin therapy (20 for Stago STAR and CoaguChek XS and 14 for CoaguChek S) were considered normal and were included in the study if they gave permission to have a finger stick and to have drawn a tube of citrated blood that was sent to the main Hematology laboratory for analysis. Blood for analysis in the central laboratory was drawn within 15 min of finger stick analysis. No patients were excluded from the study due to extremely high PT-INR. 2.3. Statistical analysis The EG was developed in conjunction with a Hematopathologist (MTE) and an Internal Medicine physician (HMV) with expertise in anti-coagulation therapy. The EG that was developed was based entirely on expert opinion and has had no performance-based validation. Using the EG that had been developed to evaluate POCT blood glucose monitors as the model [7,8] the following EG zones developed: Zone A - Results that may not be identical but the difference between the methods (defined as Clinical Laboratory PT-INR ± 15%) will have no effect on the required clinical action. In other words the results are clinically identical. Zone B - The results are substantially different, however, any altered clinical action will have little or no impact on a patient's clinical outcome (defined as Clinical Laboratory PT-INR ± 15–25% for PT INR values b4.0). For example, corrective action, while required, is in the correct direction but will be slightly greater or less than required. Zone C - The results are substantially different resulting in an altered clinical action that could have a significant medical risk. This could lead to under or over dosing the patient causing a patient to be at risk of bleeding complications (over anticoagulated) or thromboembolic events (under anticoagulation). Zone D - The results are substantially different resulting in altered clinical action possibly having dangerous and life threatening medical risk. The difference between the methods is great enough that severe over or under anti-coagulation has occurred and the patient is at a significantly increased risk of bleeding complications or thromboembolic events.
The statistical differences between the PT INR devices and the CL were assessed by the Wilcoxon rank sum test for paired samples using MedCalc (Mariakerke, Belgium). We also utilized McNemar's test and the κ statistic to measure the degree of agreement between the CL vs. CoaguChek XS® and CoaguChek S®. Specifically, we utilized the mean± SD PT INR values from CL and dichotomized both the CoaguChek XS® and CoaguChek S® PT INR values as within or beyond the Zone A boundary from the CL data. We also performed power calculations based on the κ statistic. 3. Results As part of this study, patients had capillary whole blood PT INR determinations using POCT instrumentation (CoaguChek XS and/or CoaguChek S) followed by venous blood sampling for PT INR determination in the CL (Stago STAR). Of the 103 patient samples (20 normal and 83 on Coumadin therapy) sent to the CL for PT INR determination, 99 were tested using the CoaguChek XS meter, 64 were tested using the CoaguChek S meter, and 56 were tested using both POCT meters. The CL PT INR values for the samples ranged from 0.9 to 7.0. Table 1 shows the mean (SD) for CL PT INR ranging from ≤3.0, N3.0–4.0, and N4.0. For a CL PT INR ≤3.0 the mean bias between the CoaguChek S and the CoaguChek XS relative to the CL was 0.06 (p = 0.63) and 0.09 (p = 0.003) PT INR units, respectively. The differences between the CoaguChek XS and CoaguChek S were not statistically significant (p N 0.1), however, the CoaguChek S showed increased variability as compared to the CoaguChek XS relative to the CL values (Sy/x = 0.25 vs. 0.13, respectively). The same trend is also seen for all PT INR ranges. Although both the CoaguChek XS and CoaguChek S compare reasonably well with the CL when the CL PT INR b3.0, when the CL PT INR is N3.0 overestimation of the CL values is apparent with both meters. Both the increased variability for the CoaguChek S and the overestimation for both meters with increasing PT INR are readily apparent by inspection of the respective EG (Fig. 1A–B). Since the differences between the CoaguChek XS and CoaguChek S appear to be curvilinear the relationship was expressed as a 2nd order polynomial equation: y = − 0.02 x2 + 1.40 x − 0.55; R2 = 0.955 and y = 0.15 x2 + 0.56 x + 0.23; R2 = 0.93 for CoaguChek XS and CoaguChek S, respectively. In order to better assess the clinical significance of differences between the POCT and CL PT INR results we applied our proposed
Table 1 Mean PT INR for Stago STAR, CoaguChek S, and CoaguChek XS for patients whose PT INR for Stago STAR was ≤3.0, 3.1–4.0, N 4.0.
Stago STAR PT INR (SD) N CoaguChek S PT INR (SD) Sy/x N p (Wilcoxon) (vs. Stago STAR) κ Coefficient H0: κ = 0 p (McNemar's test of equality of proportions) CoaguChek XS PT INR (SD) Sy/x N p (Wilcoxon) (vs. Stago STAR) κ Coefficient H0: κ = 0 p (McNemar's test of equality of Proportions)
Stago STAR
Stago STAR
Stago STAR
PT INR ≤3.0
PT INR N3.0–4.0
PT INR N4.0
1.79 (0.64) 57
3.6 (0.27) 22
4.8 (0.76) 24
1.85 (0.74) 0.25 41 0.47 0.66 b 0.0001 NS
4.3 (0.81) 0.52 13 0.014 0.64 0.022 NS
5.94 (1.11) 0.95 10 0.002 Not estimable
1.88 (0.70) 0.13 54 0.009 0.82 b 0.0001 NS
4.18 (0.47) 0.31 22 b0.001 0.10 0.52 0.011
5.92 (0.83) 0.54 23 b 0.001 0.12 0.22 0.0003
J.R. Petersen et al. / Clinica Chimica Acta 411 (2010) 131–134
133
Table 2 Number of patients and % of patients within Zones A, B, and C for CoaguChek S and CoaguChek XS at various PT INR levels. PT INR ≤ 3.0 Zone A
Zone B
CoaguChek S N 34 7 % 82.9% 17.1% CoaguChek XS N 53 1 % 98.1% 1.9%
PT INR N 4.0
PT INR 3.1–4.0 Zone C
Zone A
Zone B
Zone C
Zone A
Zone B
Zone C
0 0.0%
7 53.8%
3 30.8%
3 15.4%
2 20.0%
5 50.0%
3 30.0%
0 0.0%
15 68.2%
7 31.8%
0 0.0%
7 30.4%
16 72.7%
0 0.0%
results that fell outside of error zone A when the CL PT INR was ≤3.0 with 17.1% falling outside of risk Zone A (17.1% risk zone B) vs. 1.9% falling outside risk Zone A for CoaguChek XS. This trend is even more apparent for a CL PT INR N3.0 in which 60.9% (34.8% in risk Zone B and 26.1% in risk Zone C) vs. 51.1% (all in risk Zone B) that fall outside risk Zone A for CoaguChek S and CoaguChek XS, respectively. In order to determine if we had an adequate number of samples on which to base our conclusions we performed power calculations based on the κ statistic. Assuming a null hypothesis equal to 0 vs. a κ statistic equal to 0.65 (as seen Table 1), we have over 95% power in detecting this κ value with 40 samples (CL PT INR values ≤3.0 and 3–4) vs. the CoaguChek S. Similarly using a κ statistic equal to 0.80 (as seen in Table 1), we have over 95% power in detecting this κ value with 55 samples (CL PT INR values ≤3.0) vs. the CoaguChek XS. 4. Discussion
Fig. 1. Error-grid for PT INR comparison of CoaguChek S (A) and CoaguChek XS (B) vs. Stago STAR. The grid shows risk zones A to D with increasing clinical relevance of disagreement between the measurements. The dashed lines are the line of identity for the analysis. The solid lines are the 2nd order polynomial equations: y = − 0.02 x2 + 1.40 x − 0.55; R2 = 0.955 and y = 0.15 x2 + 0.56 x + 0.23; R2 = 0.93 for CoaguChek XS and CoaguChek S, respectively.
error grid (Fig. 1 A–B) and plotted the measurements for CoaguChek XS and CoaguChek S vs. the CL. Our use of CL PT INR ± 15% as clinically acceptable for zone A is supported by a very recent study that indicated a deviation of 15% at an INR of 2.5 would probably have no clinical impact in terms of increased thromboembolism or bleeding events in patients using POCT meters [15]. As shown by Fig. 1 and in Table 2 all CoaguChek XS and CoaguChek S results were within zones A and B for CL PT INR values ≤3.0. This indicates that for PT INR values b3.0 the CoaguChek XS and CoaguChek S values could replace the CL values with no or little impact on a patient's clinical outcome. This is also shown by the excellent agreement (κ coefficient = 0.82) between the CL vs. XS and good agreement between the CL vs. S (κ coefficient = 0.66 and 0.64, respectively). The corresponding p-values for the κ coefficients indicate that there is a statistically significant correlation between these methods. In addition the McNemar's test shows that there are no significant differences in proportions between the CL vs. CoaguChek XS and CoaguChek S when the CL PT INR is ≤3. Above a CL PT INRs N3.0 it is readily apparent that both the CoaguChek XS and CoaguChek S give results that are significantly higher than the CL. Closer inspection of the EG and Table 2 also reveals that the CoaguChek S had a higher proportion (both higher and lower) of
The use of POCT PT INR to monitor patient anti-coagulation has been established by a number of studies [2–6], although others have questioned if POCT meters have adequate accuracy to monitor and regulate Coumadin dosages relative to a CL [8–10]. Normally, POCT meters have been compared to the CL methods using statistical methods developed for highly accurate CL methods that may indicate statistically significant differences but may not be relevant when making the dosing decisions. Although Bland–Altman plots give the impression of potential bias and variability of the methods being compared, the clinical acceptability of the differences between methods is not readily apparent. On the other hand EG analysis, which has been used to determine the clinical significance of inaccuracies when evaluating POCT glucose meters, provides a straight forward representation of the clinical importance of analytical differences in methods by ascertaining the potential risk of bleeding or thrombotic complications when using the POCT results to make treatment decisions as compared to results from the CL. By simple visual inspection of the EG it is obvious as to whether a new POCT PT INR meter is acceptable or not. If a more quantitative measure is desired identification of the number of values in each error zone such as shown in Table 2 would be an easy solution. Using these methods it is apparent that the CoaguChek XS performs better than the CoaguChek S as compared to the CL. Similar results have been previously been identified when comparing the CoaguChek XS and CoaguChek S [16,17]. By using the EG analysis (Fig. 1 A–B) and Table 2 it is evident that for PT INR ≤3.0 the results from the CoaguChek XS compares better to the CL with only 1 (1.9%) result outside of error zone A as compared to 7 (17.1%) seen using the CoaguChek S. Above INR N3.0 the high bias inherent in both instruments means that N50% of the values are outside of zone A for both meters. Additionally 6 (21.4%) were in zone C for CoaguChek S which could put the patient at significant risk. The analysis of errors in POCT PT INR is a complicated problem because the clinical consequence of any error not only depends on the absolute difference between the POCT and the CL result but also the relative deviation. Thus we conclude that EG analysis when used in
134
J.R. Petersen et al. / Clinica Chimica Acta 411 (2010) 131–134
conjunction with other statistical tools, such as regression analysis, could be an extremely useful tool in determining the clinical acceptability of new POCT PT INR methods. The analysis of errors in POCT PT INR is a complicated problem because the clinical consequence of any error not only depends on the absolute difference between the POCT and the CL result but also the relative deviation. Thus we conclude that EG analysis when used in conjunction with other statistical tools, such as regression analysis, is an extremely useful tool in determining the clinical acceptability of new POCT PT INR methods. In addition EG analysis would also provide useful information in the comparison of new CL methods. We also raise some caution regarding whether immediate action, including the need for patients to see physicians urgently, can be reliably decided upon using POCT INR results performed at home.
References [1] Hirsh J, Dalen JE, Anderson DR, et al. Oral anticoagulants: mechanism of action, clinical effectiveness, and optimal therapeutic range. Chest 1998;114:445S–69S. [2] Ansell J, Hirsh J, Hylek E, Jacobson A, Crowther M, Palareti G. Pharmacology and management of the vitamin K antagonists: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition). Chest 2008;133:160S–98S Suppl. [3] Fitzmaurice DA, Murray ET, McCahon D, et al. Self management of oral anticoagulation: randomised trial. BMJ 2005;331:1057–60. [4] Cromheecke ME, Levi M, Colly LP, et al. Oral anticoagulation self-management and management by a specialist anticoagulation clinic: a randomized cross-over comparison. Lancet 2000;356:97–102. [5] Gardiner C, Williams K, Mackie IJ, Machin SJ, Cohen H. Can oral anticoagulation be managed using telemedicine and patient self-testing? A pilot study. Clin Lab Haematol 2006;28:122–5.
[6] Gosselin R, Owings JT, White RH, et al. A comparison of point-of-care instruments designed for monitoring oral anticoagulation with standard laboratory methods. Thromb Haemost 2000;83:698–703. [7] Gill JM, Landis MK. Benefits of a mobile, point-of-care anticoagulation therapy management program. Jt Comm J Qual Improv 2002;28:625–30. [8] Loebstein R, Kurnik D, Lubetsky A, Ezra D, Halkin H. Potential dosing errors using portable prothrombin time monitoring devices. Blood Coagul Fibrinolysis 2003;14:479–83. [9] McBane R, Felty C, Hartgers M, et al. Importance of device evaluation for point-ofcare prothrombin time international normalized ratio testing programs. Mayo Clin Proc 2005;80:181–6. [10] Moore G, Henley A, Cotton S, et al. Clinically significant differences between pointof-care analysers and a standard analyser for monitoring the international normalized ratio in oral anticoagulation therapy: a multi-instrument evaluation in a hospital outpatient setting. Blood Coagul Fibrinolysis 2007;18:287–92. [11] Anderson DR, Harrison L, Hirsh J. Evaluation of a portable prothrombin time monitor for home use by patients who require long-term oral anticoagulant therapy. Arch Intern Med 1993;153:1441–7. [12] Clarke WL, Cox D, Gonder-Frederick LA, Carter W, Pohl SL. Evaluating clinical accuracy of systems for self-monitoring of blood glucose. Diabetes Care 1987;10:622–8. [13] Parkes JL, Slatin SL, Pardo S, Ginsberg BH. A new consensus error grid to evaluate the clinical significance of inaccuracies in the measurement of blood glucose. Diabetes Care 2000;23:1143–8. [14] Hemkens LG, Hilden KM, Hartschen S, et al. A randomized trial comparing INR monitoring devices in patients with anticoagulation self-management: evaluation of a novel error-grid approach. J Thromb Thrombolysis 2008;26:22–30. [15] Christensen TD, Larsen TB, Jensen C, Maegaard M, Sørensen B. International normalised ratio (INR) measured on the CoaguChek S and XS compared with the laboratory for determination of precision and accuracy. Thromb Haemost 2009;101:563–9. [16] Williams VK, Griffiths AB. Acceptability of CoaguChek S and CoaguChek XS generated international normalized ratios against a laboratory standard in a paediatric setting. Pathology 2007;39:575–9. [17] Karon BS, McBane RD, Chaudhry R, Beyer LK, Santrach PJ. Accuracy of capillary whole blood international normalized ratio on the CoaguChek S, CoaguChek XS, and i-STAT 1 point-of-care analyzers. Am J Clin Pathol 2008;130:88–92.