- Email: [email protected]
Statistical issues with the determination of the troponin 99th percentile – Not just a problem for troponin?
Clinical Biochemistry 49 (2016) 1105–1106
Contents lists available at ScienceDirect
Clinical Biochemistry journal homepage: www.elsevier.com/locate/clinbiochem
Editorial
Statistical issues with the determination of the troponin 99th percentile – Not just a problem for troponin? Keywords: Cardiac troponin Statistics Reference intervals 99th percentile Guidelines
Currently the upper 99th percentile limit for cardiac troponin derived from a healthy population is the formal cutpoint which should be utilized in the laboratory assessment of myocardial injury. The Universal Definition of acute myocardial infarction requires “Detection of a rise and/or fall of cardiac biomarker values [preferably cardiac troponin (cTn)] with at least one value above the 99th percentile upper reference limit (URL)” [1]. Multiple studies have demonstrated that derivation of the 99th percentile can be problematic with sample size [2], age [3], sex [4] and subclinical disease [5–7] all able to cause large changes in the derived upper limit. A study in the current issue of Clinical Biochemistry highlights another problem with the derivation of the 99th percentile [8]. Using data on 521 healthy subjects enrolled in the PIVUS study [9], using nonparametric, bootstrap and robust methods, and excluding outliers by the methods of Dixon [10], Reed [11] or Tukey [12], they found the 99th percentile may change markedly. Depending upon the method used for excluding outliers, the hs-cTnI 99th percentile value decreased by N50% for both males and females. Considering the diagnostic importance that is attached to the 99th percentile, perhaps it is time for some formal criteria relating to population selection and statistical analysis for any studies performed to define the 99th percentile reference interval. Consideration of just the physiological variables of age and sex alone would require 1200 persons and based on previous experience with excluding persons with subclinical disease it is likely that the initial enrolment would need to be 1600 persons [13]. All of these individuals would require cardiac imaging at great expense, hence the need to ensure that the studies are rigorously performed. Whilst this may be cost prohibitive for most laboratories, the uncertainty introduced by using different statistical methods to exclude possible outliers in determining the 99th percentile, may render this exercise cost frivolous if the incorrect upper estimates are determined. There are documents available for guidance in this area, with the Clinical Laboratory Standards Institute (CLSI), producing a document for defining, establishing and verifying reference intervals [14]. This document provides laboratories and in vitro diagnostic companies with guidelines for determining reference intervals and includes
specific recommendations regarding procedures that can be used to establish and verify reliable reference intervals for use in clinical laboratory medicine. This guideline states “The working group emphasizes that the most important considerations in developing reliable reference intervals are selecting appropriate reference subjects, testing an adequate number of subjects, and avoiding pre-analytical errors, not the statistical method used to estimate the reference intervals from the observed data.” This latter statement is in contrast to the study from Eggers et al. who show that defining the statistical handling of the data generated is as important as the initial population selection. This timely paper is also important in a wider context. Although CLSI provides excellent information on how reference intervals should be calculated [14], they are not proscriptive about the statistical handling of the data, nor do they provide guidance on specific assays such as troponin. Whilst they recommend removal of outliers, and discuss various methods for doing this, their guidance is conservative stating “Unless outliers are known to be aberrant observations … the emphasis should be on retaining rather than deleting them.” The CLSI document offers the opinion that the simple nonparametric method remains the recommended procedure for establishing reference intervals. However, there is a caveat that if the sample size constraints of this method prevent a laboratory from establishing reference intervals, but has access to personnel that can interpret and implement more complex procedures, then use of either bootstrap-based procedures or the robust methods is recommended. Eggers has demonstrated a difference of N60% in hscTnI 99th percentiles between the non-parametric method of analysis, compared to the robust approach, further calling into question the validity of applying general recommendations for all analytes when deriving reference intervals. When selecting the population for a reference interval study it is assumed that the population is appropriately homogeneous for the analyte(s) being validated. The question arises how to treat those results that are apparently aberrant? The CLSI guidelines offer expert opinion only. These guidelines support the use of the Dixon and Reed [10,11] method but also suggest that the method of Tukey [12] could be used to reduce the masking effect of multiple outliers. Where the data is distributed in a non-Gaussian manner, transformation is required. CLSI suggests that Box Cox transformation along with Tukey outlier detection as proposed by Horn [15] is a reasonable approach as is the appropriateness to retest the data for outliers after any outlier elimination has been undertaken. However, Eggers again demonstrates a difference between outlier elimination methods with the method of Tukey producing hs-cTnI 99th percentile concentrations N60% lower than that of the Reed and Dixon method. Furthermore, there are nonparametric datasets that cannot be transformed to yield a parametric
http://dx.doi.org/10.1016/j.clinbiochem.2016.09.013 0009-9120/© 2016 The Canadian Society of Clinical Chemists. Published by Elsevier Inc. All rights reserved.
1106
Editorial
distribution, further complicating consideration of which outlier removal method should be used. A majority of reference intervals utilized in clinical laboratory medicine are based upon the central 95% of the population and thus unlikely to be as severely affected by statistical handling as when using the 99th percentile. However, this paper raises further questions. When the three different methods for reference interval determination are combined with the two different outlier elimination methods and the 99th percentiles concentrations obtained are compared, up to a 4 fold difference can be observed. This begs the questions – should CLSI revisit the guidelines and make specific and definitive recommendations about how data is to be handled in reference interval studies for all analytes? Alternatively, should recommendations come from expert groups for the analyte in question? The clinical laboratory community already has educational material from the International Federation of Clinical Chemistry on high-sensitivity cardiac troponin assays [16]. Will the next step from this or another authoritative group in cardiac troponin testing detail a prescribed statistical approach in calculating this much discussed cutpoint? Conflict of interest Dr. Kavsak has received grants/reagents/consultant/advisor/ honoraria from Abbott Laboratories, Abbott Point of Care, Abbott Diagnostics Division Canada, Beckman Coulter, Ortho Clinical Diagnostics, Randox Laboratories, Roche Diagnostics and Siemens Healthcare Diagnostics with respect to cardiac biomarkers. McMaster University has filed patents with Dr. Kavsak listed as an inventor in the acute cardiovascular biomarker field. Dr. Saenger has received grant support and honoraria from Roche Diagnostics.
[5] P.O. Collinson, Y.M. Heung, D. Gaze, F. Boa, R. Senior, R. Christenson, F.S. Apple, Influence of population selection on the 99th percentile reference value for cardiac troponin assays, Clin. Chem. 58 (2012) 219–225. [6] G. Koerbin, W.P. Abhayaratna, J.M. Potter, F.S. Apple, A.S. Jaffe, T.H. Ravalico, P.E. Hickman, Effect of population selection on 99th percentile values for a high sensitivity cardiac troponin I and T assays, Clin. Biochem. 46 (2013) 1636–1643. [7] P.M. McKie, D.M. Heublein, C.G. Scott, M.L. Gantzer, R.A. Mehta, R.J. Rodeheffer, et al., Defining high-sensitivity cardiac troponin concentrations in the community, Clin. Chem. 1099-1107 (2013). [8] K.M. Eggers, F.S. Apple, L. Lind, B. Lindahl, The applied statistical approach highly influences the 99th percentile of cardiac troponin I, Clin. Biochem. 49 (15) (2016) 1109–1112. [9] L. Lind, N. Fors, J. Hall, K. Marttala, A. Stenborg, A comparison of three different methods to evaluate endothelium-dependent vasodilation in the elderly. The Prospective Investigation of the Vasculature in Uppsala Seniors (PIVUS) study, Arterioscler. Thromb. Vasc. Biol. 25 (2005) 2368–2375. [10] W.J. Dixon, Processing data for outliers, Biometrics 9 (1983) 74–89. [11] A.H. Reed, R.J. Henry, W.B. Mason, Influence of statistical method used on the resulting estimate of the normal range, Clin. Chem. 17 (1971) 275–284. [12] J.W. Tukey, Exploratory Data Analysis, Addison-Wesley, Reading (PA), 1977. [13] P.E. Hickman, B. Lindahl, L. Cullen, G. Koerbin, J. Tate, J.M. Potter, Decision limits and the reporting of cardiac troponin: meeting the needs of both the cardiologist and the ED physician, Crit. Rev. Clin. Lab. Sci. 52 (2015) 28–44. [14] Clinical Laboratory Standards Institute, Defining, Establishing and Verifying Reference Intervals in the Clinical Laboratory; Approved Guidelines – Third Edition. CLSI Document EP28-A3c2010, Wayne (PA), CLSI, 2008. [15] P.S. Horn, A.J. Pesce, Reference Intervals. A User's Guide, AACC Press, Washington, DC, 2005. [16] F.S. Apple, A.S. Jaffe, P. Collinson, M. Mockel, J. Ordonez-Llanos, B. Lindahl, et al., International Federation of Clinical Chemistry (IFCC) Task Force on Clinical Applications of Cardiac Bio-Markers, IFCC educational materials on selected analytical and clinical applications of high sensitivity cardiac troponin assays, Clin. Biochem. 48 (2015) 201–203.
Peter E. Hickman ACT Pathology, Garran, ACT 2605, Australia ANU Medical School, Garran, ACT 2605, Australia Corresponding author at: ACT Pathology, Garran, ACT 2605, Austrialia. E-mail address: [email protected].
Funding None to declare. References [1] K. Thygesen, J.S. Alpert, A.S. Jaffe, M.L. Simoons, B.R. Chaitman, H.D. White, et al., Third universal definition of myocardial infarction, Circulation 126 (2012) 2020–2035. [2] P.E. Hickman, T. Badrick, S.R. Wilson, D. McGill, Reporting of cardiac troponin – problems with the 99th population percentile, Clin. Chim. Acta 381 (2007) 182–183. [3] P. Venge, N. Johnston, B. Lindahl, S. James, Normal plasma levels of cardiac troponin I measured by the high-sensitivity cardiac troponin I access prototype assay and the impact on the diagnosis of myocardial ischemia, J. Am. Coll. Cardiol. 54 (2009) 1165–1172. [4] M.O. Gore, S.L. Seliger, C.R. Defilippi, V. Nambi, R.H. Christenson, I.A. Hashim, et al., Age- and sex-dependent upper reference limits for the high-sensitivity cardiac troponin T assay, J. Am. Coll. Cardiol. 63 (2014) 1441–1448.
Gus Koerbin New South Wales Health Pathology, Chatswood, NSW 2067, Australia Amy K. Saenger Dept Laboratory Medicine and Pathology, University of Minnesota, Minneapolis, MN, USA Peter A. Kavsak Department of Pathology and Molecular Medicine, McMaster University, Hamilton, Canada
Contents lists available at ScienceDirect
Clinical Biochemistry journal homepage: www.elsevier.com/locate/clinbiochem
Editorial
Statistical issues with the determination of the troponin 99th percentile – Not just a problem for troponin? Keywords: Cardiac troponin Statistics Reference intervals 99th percentile Guidelines
Currently the upper 99th percentile limit for cardiac troponin derived from a healthy population is the formal cutpoint which should be utilized in the laboratory assessment of myocardial injury. The Universal Definition of acute myocardial infarction requires “Detection of a rise and/or fall of cardiac biomarker values [preferably cardiac troponin (cTn)] with at least one value above the 99th percentile upper reference limit (URL)” [1]. Multiple studies have demonstrated that derivation of the 99th percentile can be problematic with sample size [2], age [3], sex [4] and subclinical disease [5–7] all able to cause large changes in the derived upper limit. A study in the current issue of Clinical Biochemistry highlights another problem with the derivation of the 99th percentile [8]. Using data on 521 healthy subjects enrolled in the PIVUS study [9], using nonparametric, bootstrap and robust methods, and excluding outliers by the methods of Dixon [10], Reed [11] or Tukey [12], they found the 99th percentile may change markedly. Depending upon the method used for excluding outliers, the hs-cTnI 99th percentile value decreased by N50% for both males and females. Considering the diagnostic importance that is attached to the 99th percentile, perhaps it is time for some formal criteria relating to population selection and statistical analysis for any studies performed to define the 99th percentile reference interval. Consideration of just the physiological variables of age and sex alone would require 1200 persons and based on previous experience with excluding persons with subclinical disease it is likely that the initial enrolment would need to be 1600 persons [13]. All of these individuals would require cardiac imaging at great expense, hence the need to ensure that the studies are rigorously performed. Whilst this may be cost prohibitive for most laboratories, the uncertainty introduced by using different statistical methods to exclude possible outliers in determining the 99th percentile, may render this exercise cost frivolous if the incorrect upper estimates are determined. There are documents available for guidance in this area, with the Clinical Laboratory Standards Institute (CLSI), producing a document for defining, establishing and verifying reference intervals [14]. This document provides laboratories and in vitro diagnostic companies with guidelines for determining reference intervals and includes
specific recommendations regarding procedures that can be used to establish and verify reliable reference intervals for use in clinical laboratory medicine. This guideline states “The working group emphasizes that the most important considerations in developing reliable reference intervals are selecting appropriate reference subjects, testing an adequate number of subjects, and avoiding pre-analytical errors, not the statistical method used to estimate the reference intervals from the observed data.” This latter statement is in contrast to the study from Eggers et al. who show that defining the statistical handling of the data generated is as important as the initial population selection. This timely paper is also important in a wider context. Although CLSI provides excellent information on how reference intervals should be calculated [14], they are not proscriptive about the statistical handling of the data, nor do they provide guidance on specific assays such as troponin. Whilst they recommend removal of outliers, and discuss various methods for doing this, their guidance is conservative stating “Unless outliers are known to be aberrant observations … the emphasis should be on retaining rather than deleting them.” The CLSI document offers the opinion that the simple nonparametric method remains the recommended procedure for establishing reference intervals. However, there is a caveat that if the sample size constraints of this method prevent a laboratory from establishing reference intervals, but has access to personnel that can interpret and implement more complex procedures, then use of either bootstrap-based procedures or the robust methods is recommended. Eggers has demonstrated a difference of N60% in hscTnI 99th percentiles between the non-parametric method of analysis, compared to the robust approach, further calling into question the validity of applying general recommendations for all analytes when deriving reference intervals. When selecting the population for a reference interval study it is assumed that the population is appropriately homogeneous for the analyte(s) being validated. The question arises how to treat those results that are apparently aberrant? The CLSI guidelines offer expert opinion only. These guidelines support the use of the Dixon and Reed [10,11] method but also suggest that the method of Tukey [12] could be used to reduce the masking effect of multiple outliers. Where the data is distributed in a non-Gaussian manner, transformation is required. CLSI suggests that Box Cox transformation along with Tukey outlier detection as proposed by Horn [15] is a reasonable approach as is the appropriateness to retest the data for outliers after any outlier elimination has been undertaken. However, Eggers again demonstrates a difference between outlier elimination methods with the method of Tukey producing hs-cTnI 99th percentile concentrations N60% lower than that of the Reed and Dixon method. Furthermore, there are nonparametric datasets that cannot be transformed to yield a parametric
http://dx.doi.org/10.1016/j.clinbiochem.2016.09.013 0009-9120/© 2016 The Canadian Society of Clinical Chemists. Published by Elsevier Inc. All rights reserved.
1106
Editorial
distribution, further complicating consideration of which outlier removal method should be used. A majority of reference intervals utilized in clinical laboratory medicine are based upon the central 95% of the population and thus unlikely to be as severely affected by statistical handling as when using the 99th percentile. However, this paper raises further questions. When the three different methods for reference interval determination are combined with the two different outlier elimination methods and the 99th percentiles concentrations obtained are compared, up to a 4 fold difference can be observed. This begs the questions – should CLSI revisit the guidelines and make specific and definitive recommendations about how data is to be handled in reference interval studies for all analytes? Alternatively, should recommendations come from expert groups for the analyte in question? The clinical laboratory community already has educational material from the International Federation of Clinical Chemistry on high-sensitivity cardiac troponin assays [16]. Will the next step from this or another authoritative group in cardiac troponin testing detail a prescribed statistical approach in calculating this much discussed cutpoint? Conflict of interest Dr. Kavsak has received grants/reagents/consultant/advisor/ honoraria from Abbott Laboratories, Abbott Point of Care, Abbott Diagnostics Division Canada, Beckman Coulter, Ortho Clinical Diagnostics, Randox Laboratories, Roche Diagnostics and Siemens Healthcare Diagnostics with respect to cardiac biomarkers. McMaster University has filed patents with Dr. Kavsak listed as an inventor in the acute cardiovascular biomarker field. Dr. Saenger has received grant support and honoraria from Roche Diagnostics.
[5] P.O. Collinson, Y.M. Heung, D. Gaze, F. Boa, R. Senior, R. Christenson, F.S. Apple, Influence of population selection on the 99th percentile reference value for cardiac troponin assays, Clin. Chem. 58 (2012) 219–225. [6] G. Koerbin, W.P. Abhayaratna, J.M. Potter, F.S. Apple, A.S. Jaffe, T.H. Ravalico, P.E. Hickman, Effect of population selection on 99th percentile values for a high sensitivity cardiac troponin I and T assays, Clin. Biochem. 46 (2013) 1636–1643. [7] P.M. McKie, D.M. Heublein, C.G. Scott, M.L. Gantzer, R.A. Mehta, R.J. Rodeheffer, et al., Defining high-sensitivity cardiac troponin concentrations in the community, Clin. Chem. 1099-1107 (2013). [8] K.M. Eggers, F.S. Apple, L. Lind, B. Lindahl, The applied statistical approach highly influences the 99th percentile of cardiac troponin I, Clin. Biochem. 49 (15) (2016) 1109–1112. [9] L. Lind, N. Fors, J. Hall, K. Marttala, A. Stenborg, A comparison of three different methods to evaluate endothelium-dependent vasodilation in the elderly. The Prospective Investigation of the Vasculature in Uppsala Seniors (PIVUS) study, Arterioscler. Thromb. Vasc. Biol. 25 (2005) 2368–2375. [10] W.J. Dixon, Processing data for outliers, Biometrics 9 (1983) 74–89. [11] A.H. Reed, R.J. Henry, W.B. Mason, Influence of statistical method used on the resulting estimate of the normal range, Clin. Chem. 17 (1971) 275–284. [12] J.W. Tukey, Exploratory Data Analysis, Addison-Wesley, Reading (PA), 1977. [13] P.E. Hickman, B. Lindahl, L. Cullen, G. Koerbin, J. Tate, J.M. Potter, Decision limits and the reporting of cardiac troponin: meeting the needs of both the cardiologist and the ED physician, Crit. Rev. Clin. Lab. Sci. 52 (2015) 28–44. [14] Clinical Laboratory Standards Institute, Defining, Establishing and Verifying Reference Intervals in the Clinical Laboratory; Approved Guidelines – Third Edition. CLSI Document EP28-A3c2010, Wayne (PA), CLSI, 2008. [15] P.S. Horn, A.J. Pesce, Reference Intervals. A User's Guide, AACC Press, Washington, DC, 2005. [16] F.S. Apple, A.S. Jaffe, P. Collinson, M. Mockel, J. Ordonez-Llanos, B. Lindahl, et al., International Federation of Clinical Chemistry (IFCC) Task Force on Clinical Applications of Cardiac Bio-Markers, IFCC educational materials on selected analytical and clinical applications of high sensitivity cardiac troponin assays, Clin. Biochem. 48 (2015) 201–203.
Peter E. Hickman ACT Pathology, Garran, ACT 2605, Australia ANU Medical School, Garran, ACT 2605, Australia Corresponding author at: ACT Pathology, Garran, ACT 2605, Austrialia. E-mail address: [email protected].
Funding None to declare. References [1] K. Thygesen, J.S. Alpert, A.S. Jaffe, M.L. Simoons, B.R. Chaitman, H.D. White, et al., Third universal definition of myocardial infarction, Circulation 126 (2012) 2020–2035. [2] P.E. Hickman, T. Badrick, S.R. Wilson, D. McGill, Reporting of cardiac troponin – problems with the 99th population percentile, Clin. Chim. Acta 381 (2007) 182–183. [3] P. Venge, N. Johnston, B. Lindahl, S. James, Normal plasma levels of cardiac troponin I measured by the high-sensitivity cardiac troponin I access prototype assay and the impact on the diagnosis of myocardial ischemia, J. Am. Coll. Cardiol. 54 (2009) 1165–1172. [4] M.O. Gore, S.L. Seliger, C.R. Defilippi, V. Nambi, R.H. Christenson, I.A. Hashim, et al., Age- and sex-dependent upper reference limits for the high-sensitivity cardiac troponin T assay, J. Am. Coll. Cardiol. 63 (2014) 1441–1448.
Gus Koerbin New South Wales Health Pathology, Chatswood, NSW 2067, Australia Amy K. Saenger Dept Laboratory Medicine and Pathology, University of Minnesota, Minneapolis, MN, USA Peter A. Kavsak Department of Pathology and Molecular Medicine, McMaster University, Hamilton, Canada