- Email: [email protected]
Meta-analyses of Bland–Altman-style cardiac output validation studies: good, but do they provide answers to all our questions?
296
|
Editorials
British Journal of Anaesthesia 118 (3): 296–7 (2017) doi:10.1093/bja/aew442
Meta-analyses of Bland–Altman-style cardiac output validation studies: good, but do they provide answers to all our questions? L. A. H. Critchley* Department of Anaesthesia and Intensive Care, The Chinese University of Hong Kong, Hong Kong, China *Corresponding author. E-mail: [email protected]
No one would deny the need for safe, reliable, and non-invasive cardiac output (CO) monitoring in anaesthesia and acute medicine. Measuring CO, or better still stroke volume (SV), adds a further dimension to haemodynamic monitoring as it completes the circulatory equation BP¼COPR, where BP is blood pressure, PR is peripheral resistance, CO¼SVHR, and HR is heart rate. Clinical use of the pulmonary artery thermodilution catheter has been in decline since the 1990s,1 and newer, less invasive technologies have been developed that still need clinical evaluation.2 Since its first description in 1986, the Bland–Altman method of determining accuracy and precision against a reference method, such as thermodilution, has become the gold standard in validation statistics.3 In this month’s edition of the Journal, one finds a ‘state-of-the-art’ systematic review and meta-analysis by Joosten and colleagues4 that focuses on totally non-invasive and continuous CO monitoring technology and analyses published data extracted from Bland–Altman-style CO validation studies. It is one of a number of recent reviews published by this group that uses ‘state-of-the-art’ literature searches, paper selection, and statistical analysis technics to determine the overall accuracy and precision of the monitoring methods they review.5–7 Central to these meta-analysis metrics is percentage error, a parameter described first in 1999 by the author of this editorial.8 Percentage error is derived from the spread of agreement between the reference and test methods, referred to as limits of agreement, and the average CO for all data points, a normalizing factor that facilitates comparison between outcomes from different studies. Joosten and colleagues4 identified 37 published studies of sufficient quality to be included in their review and concluded that non-invasive CO monitoring technologies, because of an overall percentage error of 47%, are not sufficiently accurate to replace the thermodilution method, provided that one accepts the 30% benchmark for exchangeability.8 Peyton and Chong9 have recently reappraised this 30% benchmark and suggested increasing it to 45%. What can be learnt from the meta-analysis by Joosten and colleagues?4 Initially, it demonstrates how the science of performing a meta-analysis has evolved, with well-described methods of searching the literature, selecting appropriate studies, and defining suitable summary measures, such as percentage error. Their paper provides a fine example. From the initial identification of 1646 potential articles, the final meta-analysis in their paper included 37 well-performed and reliable CO validation studies that used Bland–Altman analysis. Furthermore, one learns that proper reporting in the literature of the study design
and statistical outcomes is crucial to the yield and success of such reviews and meta-analyses. Thus, there is need for published guidelines on how to design a good CO validation study and what data should be presented in the results. In the November 2016 edition of the British Journal of Anaesthesia, AbuArafeh and colleagues10 published an article that addresses this need and lists the key features required for adequate presentation of Bland–Altman analysis. Surprisingly, these authors omit percentage error as one of the key items in their list, yet it is a parameter central to several recent meta-analyses involving CO validation studies and forms part of any discussion on the performance of CO monitoring techniques, as highlighted by Joosten and colleagues.4 Next, we learn that it is very difficult to find a meta-analysis based on Bland–Altman analysis and percentage error that supports the use of any of these new CO monitoring technologies.9 Suehiro and colleagues5 have recently published a metaanalysis in children that identified 20 suitable studies and did find that electrical velocimetry, a type of bioimpedance, and two-dimensional echo Doppler were interchangeable with thermodilution. However, it is noteworthy that this metaanalysis concerned publications that involved children. These new monitoring technologies do not measure CO directly, but derive it from surrogates of aortic blood flow, such as arterial pressure, bioimpedance, and pulse wave transit time. As a result, many of these monitors require external calibration based on patient characteristics, such as age, height, and weight, and derive CO in part from charts based on previously collected population data. Given that adult patients demonstrate great heterogeneity in body habitus, significant variation in CO readings between them can occur that is expressed as systematic errors that originate from this calibration process and that contributes to the size of the Bland–Altman limits of agreement and percentage error. However, the systematic error component has little to do with a monitor’s ability to measure blood flow, despite contributing to Bland–Altman outcomes. The problem with using Bland–Altman is the inability to differentiate between the random measurement and systematic calibration components. In children, who are more homogeneous in body shape than adults, the systematic error component will be smaller, and thus contribute less, which accounts for the better Bland–Altman meta-analysis results found by Suehiro and colleagues5 in their recent meta-analysis. Finally, we should note that although Bland–Altman analysis is certainly a convenient and well-used statistical method in
Editorials
evaluation of CO monitoring, it does not provide all the information we need to make a proper assessment of this newer non-invasive CO technology. Although Bland–Altman analysis provides an assessment of accuracy against a known reference standard, usually thermodilution, it does not provide an assessment of ability to trend or detect changes in CO, for instance when a therapeutic intervention, such as a fluid challenge or starting inotropes, needs to be assessed.11 There has been a recent paradigm shift in the way we use non-invasive CO monitoring clinically, with the new emphasis being on detecting changes rather than measuring absolute values, and this shift brings a need for better study designs and statistical analyses that enable one to evaluate trending ability. Therefore, however good and developed the systematic reviews and meta-analyses of CO monitoring validations studies based on Bland–Altman and percentage error have become, they do not provide all the answers to our questions regarding the performance of these new non-invasive CO technologies. Joosten and colleagues4 reported finding only 10 papers in their review that addressed the issue of assessing trending ability of CO monitors, and the data were too scant to perform any analysis. Towards this new goal, I have published several recent reviews that promote new statistical methods for evaluating the trending ability of CO monitoring technology.11 12
Declaration of interest None declared.
References 1.
2.
Wiener RS, Welch HG. Trends in the use of the pulmonary artery catheter in the United States, 1993–2004. JAMA 2007; 298: 423–9 Marik PE. Noninvasive cardiac output monitors: a stateof the-art review. J Cardiothorac Vasc Anesth 2013; 27: 121–34
3.
|
297
Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1: 307–10 4. Joosten A, Desebbe O, Suehiro K, et al. Accuracy and precision of non-invasive cardiac output monitoring devices in perioperative medicine: a systematic review and meta-analysis. Br J Anaesth 2017; 118: 298–310 5. Suehiro K, Joosten A, Murphy LS, et al. Accuracy and precision of minimally-invasive cardiac output monitoring in children: a systematic review and meta-analysis. J Clin Monit Comput 2016; 30: 603–20 6. Kim SH, Lilot M, Sidhu KS, et al. Accuracy and precision of continuous noninvasive arterial pressure monitoring compared with invasive arterial pressure: a systematic review and meta-analysis. Anesthesiology 2014; 120: 1080–97 7. Kim SH, Lilot M, Murphy LS, et al. Accuracy of continuous noninvasive hemoglobin monitoring: a systematic review and meta-analysis. Anesth Analg 2014; 119: 332–46 8. Critchley LA, Critchley JA. A meta-analysis of studies using bias and precision statistics to compare cardiac output measurement techniques. J Clin Monit Comput 1999; 15: 85–91 9. Peyton PJ, Chong SW. Minimally invasive measurement of cardiac output during surgery and critical care: a metaanalysis of accuracy and precision. Anesthesiology 2010; 113: 1220–35 10. Abu-Arafeh A, Jordan H, Drummond G. Reporting of method comparison studies: a review of advice, an assessment of current practice, and specific suggestions for future reports. Br J Anaesth 2016; 117: 569–75 11. Critchley LA, Lee A, Ho AM. A critical review of the ability of continuous cardiac output monitors to measure trends in cardiac output. Anesth Analg 2010; 111: 1180–92 12. Critchley LA, Yang XX, Lee A. Assessment of trending ability of cardiac output monitors by polar plot methodology. J Cardiothorac Vasc Anesth 2011; 25: 536–46
|
Editorials
British Journal of Anaesthesia 118 (3): 296–7 (2017) doi:10.1093/bja/aew442
Meta-analyses of Bland–Altman-style cardiac output validation studies: good, but do they provide answers to all our questions? L. A. H. Critchley* Department of Anaesthesia and Intensive Care, The Chinese University of Hong Kong, Hong Kong, China *Corresponding author. E-mail: [email protected]
No one would deny the need for safe, reliable, and non-invasive cardiac output (CO) monitoring in anaesthesia and acute medicine. Measuring CO, or better still stroke volume (SV), adds a further dimension to haemodynamic monitoring as it completes the circulatory equation BP¼COPR, where BP is blood pressure, PR is peripheral resistance, CO¼SVHR, and HR is heart rate. Clinical use of the pulmonary artery thermodilution catheter has been in decline since the 1990s,1 and newer, less invasive technologies have been developed that still need clinical evaluation.2 Since its first description in 1986, the Bland–Altman method of determining accuracy and precision against a reference method, such as thermodilution, has become the gold standard in validation statistics.3 In this month’s edition of the Journal, one finds a ‘state-of-the-art’ systematic review and meta-analysis by Joosten and colleagues4 that focuses on totally non-invasive and continuous CO monitoring technology and analyses published data extracted from Bland–Altman-style CO validation studies. It is one of a number of recent reviews published by this group that uses ‘state-of-the-art’ literature searches, paper selection, and statistical analysis technics to determine the overall accuracy and precision of the monitoring methods they review.5–7 Central to these meta-analysis metrics is percentage error, a parameter described first in 1999 by the author of this editorial.8 Percentage error is derived from the spread of agreement between the reference and test methods, referred to as limits of agreement, and the average CO for all data points, a normalizing factor that facilitates comparison between outcomes from different studies. Joosten and colleagues4 identified 37 published studies of sufficient quality to be included in their review and concluded that non-invasive CO monitoring technologies, because of an overall percentage error of 47%, are not sufficiently accurate to replace the thermodilution method, provided that one accepts the 30% benchmark for exchangeability.8 Peyton and Chong9 have recently reappraised this 30% benchmark and suggested increasing it to 45%. What can be learnt from the meta-analysis by Joosten and colleagues?4 Initially, it demonstrates how the science of performing a meta-analysis has evolved, with well-described methods of searching the literature, selecting appropriate studies, and defining suitable summary measures, such as percentage error. Their paper provides a fine example. From the initial identification of 1646 potential articles, the final meta-analysis in their paper included 37 well-performed and reliable CO validation studies that used Bland–Altman analysis. Furthermore, one learns that proper reporting in the literature of the study design
and statistical outcomes is crucial to the yield and success of such reviews and meta-analyses. Thus, there is need for published guidelines on how to design a good CO validation study and what data should be presented in the results. In the November 2016 edition of the British Journal of Anaesthesia, AbuArafeh and colleagues10 published an article that addresses this need and lists the key features required for adequate presentation of Bland–Altman analysis. Surprisingly, these authors omit percentage error as one of the key items in their list, yet it is a parameter central to several recent meta-analyses involving CO validation studies and forms part of any discussion on the performance of CO monitoring techniques, as highlighted by Joosten and colleagues.4 Next, we learn that it is very difficult to find a meta-analysis based on Bland–Altman analysis and percentage error that supports the use of any of these new CO monitoring technologies.9 Suehiro and colleagues5 have recently published a metaanalysis in children that identified 20 suitable studies and did find that electrical velocimetry, a type of bioimpedance, and two-dimensional echo Doppler were interchangeable with thermodilution. However, it is noteworthy that this metaanalysis concerned publications that involved children. These new monitoring technologies do not measure CO directly, but derive it from surrogates of aortic blood flow, such as arterial pressure, bioimpedance, and pulse wave transit time. As a result, many of these monitors require external calibration based on patient characteristics, such as age, height, and weight, and derive CO in part from charts based on previously collected population data. Given that adult patients demonstrate great heterogeneity in body habitus, significant variation in CO readings between them can occur that is expressed as systematic errors that originate from this calibration process and that contributes to the size of the Bland–Altman limits of agreement and percentage error. However, the systematic error component has little to do with a monitor’s ability to measure blood flow, despite contributing to Bland–Altman outcomes. The problem with using Bland–Altman is the inability to differentiate between the random measurement and systematic calibration components. In children, who are more homogeneous in body shape than adults, the systematic error component will be smaller, and thus contribute less, which accounts for the better Bland–Altman meta-analysis results found by Suehiro and colleagues5 in their recent meta-analysis. Finally, we should note that although Bland–Altman analysis is certainly a convenient and well-used statistical method in
Editorials
evaluation of CO monitoring, it does not provide all the information we need to make a proper assessment of this newer non-invasive CO technology. Although Bland–Altman analysis provides an assessment of accuracy against a known reference standard, usually thermodilution, it does not provide an assessment of ability to trend or detect changes in CO, for instance when a therapeutic intervention, such as a fluid challenge or starting inotropes, needs to be assessed.11 There has been a recent paradigm shift in the way we use non-invasive CO monitoring clinically, with the new emphasis being on detecting changes rather than measuring absolute values, and this shift brings a need for better study designs and statistical analyses that enable one to evaluate trending ability. Therefore, however good and developed the systematic reviews and meta-analyses of CO monitoring validations studies based on Bland–Altman and percentage error have become, they do not provide all the answers to our questions regarding the performance of these new non-invasive CO technologies. Joosten and colleagues4 reported finding only 10 papers in their review that addressed the issue of assessing trending ability of CO monitors, and the data were too scant to perform any analysis. Towards this new goal, I have published several recent reviews that promote new statistical methods for evaluating the trending ability of CO monitoring technology.11 12
Declaration of interest None declared.
References 1.
2.
Wiener RS, Welch HG. Trends in the use of the pulmonary artery catheter in the United States, 1993–2004. JAMA 2007; 298: 423–9 Marik PE. Noninvasive cardiac output monitors: a stateof the-art review. J Cardiothorac Vasc Anesth 2013; 27: 121–34
3.
|
297
Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1: 307–10 4. Joosten A, Desebbe O, Suehiro K, et al. Accuracy and precision of non-invasive cardiac output monitoring devices in perioperative medicine: a systematic review and meta-analysis. Br J Anaesth 2017; 118: 298–310 5. Suehiro K, Joosten A, Murphy LS, et al. Accuracy and precision of minimally-invasive cardiac output monitoring in children: a systematic review and meta-analysis. J Clin Monit Comput 2016; 30: 603–20 6. Kim SH, Lilot M, Sidhu KS, et al. Accuracy and precision of continuous noninvasive arterial pressure monitoring compared with invasive arterial pressure: a systematic review and meta-analysis. Anesthesiology 2014; 120: 1080–97 7. Kim SH, Lilot M, Murphy LS, et al. Accuracy of continuous noninvasive hemoglobin monitoring: a systematic review and meta-analysis. Anesth Analg 2014; 119: 332–46 8. Critchley LA, Critchley JA. A meta-analysis of studies using bias and precision statistics to compare cardiac output measurement techniques. J Clin Monit Comput 1999; 15: 85–91 9. Peyton PJ, Chong SW. Minimally invasive measurement of cardiac output during surgery and critical care: a metaanalysis of accuracy and precision. Anesthesiology 2010; 113: 1220–35 10. Abu-Arafeh A, Jordan H, Drummond G. Reporting of method comparison studies: a review of advice, an assessment of current practice, and specific suggestions for future reports. Br J Anaesth 2016; 117: 569–75 11. Critchley LA, Lee A, Ho AM. A critical review of the ability of continuous cardiac output monitors to measure trends in cardiac output. Anesth Analg 2010; 111: 1180–92 12. Critchley LA, Yang XX, Lee A. Assessment of trending ability of cardiac output monitors by polar plot methodology. J Cardiothorac Vasc Anesth 2011; 25: 536–46