Critical appraisal of left ventricular function assessment by the automated border detection method on echocardiography.

International Journal of Cardiology 65 (1998) 193–199

Critical appraisal of left ventricular function assessment by the automated border detection method on echocardiography. Is it good enough? a a, a a b Rakesh Sapra , Balbir Singh *, Deepak Thatai , D. Prabhakaran , Arun Malhotra , a Subhash C. Manchanda b

a Department of Cardiology Cardiothoracic Centre, All India Institute of Medical Sciences, New Delhi, India Department of Nuclear Cardiology, Cardiothoracic Centre, All India Institute of Medical Sciences, New Delhi, India

Received 16 December 1997; accepted 16 April 1998

Abstract Many studies have attempted to validate the echocardiographic automated border detection (ABD) method for assessing left ventricular ejection fraction (LVEF) by comparing it with various echocardiographic and non-echocardiographic standards. The main basis of assessing its accuracy has been the coefficient of correlation. The fallacy of using coefficient of correlation for assessing agreement between two methods of measurement has been well emphasized in the literature. In the present study we used the Bland and Altman test for testing the accuracy of the ABD method. We compared the ABD method for LVEF assessment with the manual edge detection technique on echocardiography and with radionuclide ventriculography in 34 patients. The majority of patients (76%) had regional wall motion abnormality. The ABD method could be adequately performed in 25 (74%) patients. LVEF was significantly underestimated by the ABD method with very wide limits of agreement when compared with radionuclide ventriculography and the manual edge detection technique (29.2621.7 and 22.7618.4 respectively, mean error62 standard deviations). Stated simply, the ABD method could overestimate LVEF by 12.5 and 15.7 or underestimate by 30.9 and 21.1 when compared with radionuclide ventriculography and manual edge detection technique, respectively. This large error is by no means acceptable for clinical purposes. It is concluded that at the present stage, the ABD method cannot replace radionuclide ventriculography and manual edge detection technique for assessing LVEF.  1998 Published by Elsevier Science Ireland Ltd. Keywords: Automated border detection; Echocardiography; Left ventricular ejection fraction

1. Introduction Conventional two-dimensional echocardiography is a powerful tool for measuring cardiac chamber dimensions, ventricular wall thickness and quantifying left ventricular function [1]. One of the important requirements of making these measurements by echocardiography is identification of endocardial bor*Corresponding author, 16 / 5, East Patel Nagar, New Delhi-110 008, India.

ders. Manual border tracing is tedious and subjective, with a variability of up to 10% in area measurement [2]. Thus, most echocardiographers do not routinely perform quantification based on endocardial border definition. In routine practice, hence, ventricular function on echocardiography is generally assessed only qualitatively and subjectively. Investigators have proposed ways to automate the process of border identification to reduce errors and analysis time, but most of the methods have required off-line computer analysis, and as a consequence,

0167-5273 / 98 / $19.00  1998 Published by Elsevier Science Ireland Ltd. All rights reserved. PII: S0167-5273( 98 )00111-9

194

R. Sapra et al. / International Journal of Cardiology 65 (1998) 193 – 199

these automated border detection techniques have received limited clinical application [3–6]. A recently developed echocardiographic imaging system permits real time automated boundary detection with on line cavity volume measurement throughout the cardiac cycle [7]. The image processing procedure that underlies automated boundary detection (ABD) is a modification of quantitative integrated backscatter imaging [8,9], performed with an integration time which is |4 times that of a normal ultrasound image. This markedly reduces the speckle noise in the image [10] and facilitates the discrimination of endocardial blood interfaces and boundaries, thus allowing automatic detection and tracing of boundaries in real time. A calculation and graphics software package computes and displays the volume of left ventricular cavity and fractional change for each beat. This technique has been reported to be reasonably accurate in assessing left ventricular ejection fraction (LVEF), when compared with various other methods [11–18] mainly on the basis of good coefficient of correlation. The inaccuracy and unpredictability of coefficient of correlation in assessing agreement between two methods of measurement has been well emphasized by Bland and Altman [19]. The objective of this study was to assess the accuracy, limitations and validity of the automated border detection method by using the recommended statistical methods for assessing agreement. For this purpose we compared this technique with the extensively validated echocardiographic manual edge detection technique and radionuclide ventriculography.

2. Materials and methods

2.1. Patient enrolment Thirty-four patients referred for left ventricular function assessment were enrolled for this study. These patients either had coronary artery disease or primary myocardial disease. Patients with valvular heart disease were excluded because of the limitations of radionuclide ventriculography in assessing left ventricular function in such patients. Informed consent was obtained from each patient.

2.2. Procedure 2.2.1. Echocardiography Echocardiographic examination was performed using a Hewlett Packard Sonos 1500 echocardiographic system and a 3.5 / 2.7 MHz transducer. Left ventricular function assessment was done by the automated boundary detection method and also by the manual edge detection method using the modified Simpson’s rule. Analysis was done in apical 4 chamber view. Patients in whom less than 75% of endocardial borders could be adequately defined in apical 4 chamber view were excluded. At least three sets of end diastolic and end systolic volumes were obtained from each method and ejection fraction was calculated from their averages. For both manual edge detection and automated border detection, particular attention was given to obtain images with optimization of both left ventricular dimensions and endocardial borders. For the manual edge detection method, end diastolic and end systolic volumes were obtained from the smallest and the largest cavities of the corresponding cardiac cycle. Automated border detection was performed as described by Perez et al. [7]. In brief, after obtaining an adequate 2-dimensional image the automated border detection system was activated. Adjustments of transmit gain, time gain compensation and lateral gain compensation were made to approximate the automated borders with the visually apparent endocardial surface. Then the region of interest was manually drawn around the left ventricle to include the endocardial surface at end diastole. Waveform display of the change in left ventricular volume with systole and diastole was then obtained. End diastolic and end systolic volumes were obtained from the maximum and the minimum volume values of the consecutive cardiac cycles. 2.2.2. Radionuclide ventriculography All patients underwent radionuclide ventriculography using a modified technique of red blood cell labeling with an initial intravenous injection of 3.4 mg of stannous pyrophosphate in subjects, followed by 20 mCi of technetium 99 m pertecnetate. Imaging was done, using an Orbiter gamma camera fitted with Icon computer (Siemens), in left anterior oblique 408 view. Data was acquired in a 64364 matrix with 32 frames / R-R interval. Data analysis was done by

R. Sapra et al. / International Journal of Cardiology 65 (1998) 193 – 199

manually plotting a region of interest on computer selected end diastolic and end systolic frames. Left ventricular ejection fraction was calculated as follows: EF5(EDC2ESC) / EDC3100 where EF is ejection fraction, EDC is background subtracted end diastolic counts and ESC is background subtracted end systolic counts.

2.3. Statistical analysis The data are expressed as mean6S.D. The magnitude of difference in end diastolic volume, end systolic volume and ejection fraction obtained by the three methods was analyzed by using a paired t-test. Linear correlation between ejection fractions obtained by different methods was assessed by Pearson’s correlation test. To assess the level of agreement between the ejection fractions obtained by different methods, the Bland and Altman method [19] was used. In this method the difference in values of LVEF obtained by two methods was plotted against the averaged LVEF from the two methods (used as an estimate of the true ejection fraction) to see any possible relationship between the error in ejection fraction and the true ejection fraction. The limits of agreement were defined as 62 standard deviations of the error. To more precisely define the error in estimating LVEF the percentage of patients in whom LVEF was overestimated or underestimated by a value of more than 5 and 10 was calculated. Also sensitivity, specificity and diagnostic accuracy of the ABD method when compared with manual edge detection technique and radionuclide ventriculography was estimated by plotting a two by two table using LVEF of 50% as the cut-off between normal and subnormal LVEF.

3. Results Thirty-four patients were enrolled in the study. They were in the age group of 36–66 years, of which 24 were males and 10 females. Twenty-nine patients had coronary artery disease and five had primary myocardial disease. Out of the patients with coronary artery disease, 76% of patients had regional wall motion abnormality. Using the automated boundary

195

detection method, endocardial borders could be adequately tracked in 25 patients (74%). Left ventricular function assessment was done by all three methods in these 25 patients. The interobserver and intraobserver variability was calculated for both ABD and manual edge detection techniques on echocardiography in 10 randomly selected patients and was found to be 9 and 10%, 10.5 and 9%, respectively. On comparison of the data (by paired t-test) end diastolic volume by ABD was found to be significantly less than that by the manual edge detection technique on echocardiography (215638.4; P, 0.0001). End systolic volume was, however, not significantly different by the two methods (26.8636.7, P50.1). The ABD method significantly underestimated LVEF when compared with radionuclide ventriculography (29.2621.7, P50.0007) and showed a trend towards underestimation when compared with the manual edge detection technique though it did not reach statistical significance (22.7618.4, P50.2). Analysis of linear correlation between ejection fraction calculated by the three methods showed that they were significantly correlated (Fig. 1 a and b). Correlation of ABD was relatively better with the manual edge detection technique on echo (r50.69, P,0.01) than with radionuclide ventriculography (r50.55, P,0.001). The difference in ejection fraction (error), when plotted as proposed by Bland and Altman (Fig. 2 a and b), did not show any relationship with the averaged ejection fraction. The limits of agreement, that is, 62 standard deviations of the error (within which 95% of the errors are expected to lie) were 115.7 (CI 111.1 to 120.3) and 221.1 (CI 216.5 to 225.7) when compared with the manual edge detection technique and 112.5 (CI 23.3 to 128.3) and 230.9 (CI 215.1 to 246.7) when compared with radionuclide ventriculography. So, the ABD method could overestimate LVEF by 15.7 and 12.5 or underestimate by 21.1 and 30.9 when compared with the manual edge detection technique and radionuclide ventriculography, respectively. On more precise analysis we found that when compared with the manual edge detection technique and radionuclide ventriculography LVEF obtained by the ABD method differed by a value of more than 5 in 72% and 86% of patients, respectively, and by a value of more than 10 in 34% and 50% of patients,

196

R. Sapra et al. / International Journal of Cardiology 65 (1998) 193 – 199

Fig. 2. (a and b) Bland-Altman analysis of the differences between values for LVEF obtained by each method plotted against averaged LVEF by two methods. Abbreviations as in Table 1. Table 1 Degree of error in LVEF estimation by ABD method Fig. 1. Comparison of left ventricular ejection fraction by automated border detection method with (a) the manual edge detection method and (b) radionuclide ventriculography, by linear regression analysis.

LVEF (%) difference

0–5

6–10

.10

ABD vs. MED ABD vs. RNV

28% 16%

36% 36%

36% 48%

ABD, automated border detection; MED, manual edge detection technique on echocardiography; RNV, radionuclide ventriculography; LVEF, left ventricular ejection fraction.

respectively (Table 1 and Table 2). The ability to differentiate normal from subnormal LVEF (using a cut-off LVEF value of 50%) by the ABD method when compared with the manual edge detection technique and radionuclide ventriculography is as presented in Table 3 and Table 4.

Table 2 Analysis of error in LVEF estimation by ABD method LVEF (%) difference

ABD vs. MED ABD vs. RNV

Underestimation

Overestimation

.5

.10

.5

.10

48% 72%

28% 44%

24% 12%

8% 4%

Abbreviations as in Table 1.

4. Discussion

Table 3 Comparison of LVEF by ABD and manual edge detection (MED) LVEF by MED

The results of this study show that the automated border detection method significantly underestimated LVEF when compared with radionuclide ventriculography and there was a definite trend towards underestimation when compared with the manual edge detection technique. This underestimation in LVEF

LVEF $50% by ABD ,50%

$50%

,50%

8% 24%

12% 56%

Abbreviations as in Table 1. Sensitivity 29%, specificity 86%, positive predictive value 50%, negative predictive value 71%, diagnostic accuracy 67%.

R. Sapra et al. / International Journal of Cardiology 65 (1998) 193 – 199 Table 4 Comparison of LVEF by ABD and radionuclide ventriculography (RNV) LVEF by RNV

LVEF $50% by ABD ,50%

$50%

,50%

8% 56%

4% 32%

Abbreviations as in Table 1. Sensitivity 14%, specificity 88%, positive predictive value 67%, negative predictive value 37%, diagnostic accuracy 41%.

was apparently due to underestimation of end diastolic volume while the end systolic volume was not significantly different. Analysis by the Bland and Altman method showed a very wide scatter of the error in LVEF estimation by ABD method when compared with the other two methods with very wide limits of agreement. Even when the best fit values were taken from the confidence intervals of the limits of agreement, they still were wide enough to be unacceptable for all clinical purposes. To be clinically acceptable, the limits of agreement should be overestimation or underestimation of LVEF by less than 5. In this study LVEF obtained by the ABD method differed by more than 5 in the majority of patients (72% and 86%). Since this error was of both types i.e. overestimation and underestimation (Table 2), a uniform constant could not be calculated to correct the LVEF estimated by the ABD method. In addition, the ABD method had a rather low diagnostic accuracy (67% and 41%) in making the simple differentiation of normal and subnormal LVEF when compared with the other two methods (Table 3 and Table 4). Previously many studies have attempted validation of the ABD method for assessing left ventricular systolic function by comparing with various other methods of left ventricular function assessment. In the majority of these studies LVEF was significantly underestimated by the ABD method [11–16]. On the other hand, the ABD method overestimated LVEF in one study [18] and LVEF was not found to be statistically different in another study [7]. Despite having such a variation in LVEF estimation, in all these studies the LVEF calculated by the ABD method correlated well with that obtained by various other methods. It has been well emphasized by Bland

197

and Altman [19] that calculation of correlation coefficient is an inappropriate method for assessing agreement between two methods of clinical measurement and is actually misleading. The presence of a high correlation coefficient does not ensure agreement between the two methods. Actually, despite having a high coefficient of correlation, individual values can vary significantly to be unacceptable for clinical purposes. Among the previous studies in which the ABD method has been compared with other methods of left ventricular function assessment, the Bland and Altman test has been used by some of the studies [14,16,18]. In all such studies the limits of agreement and the scatter of error in LVEF estimation of the ABD method has been found to be as wide as in the present study and so is unacceptable for clinical use. However enough emphasis has not been given to the wide limits of agreement and the main basis of assessing accuracy of the ABD method has been the coefficient of correlation. The majority of our patients had regional wall motion abnormality in which subgroup the ABD method has been reported to have inferior results. This could partly explain relatively poorer coefficient of correlation and wide limits of agreement in this study. But similar wide limits of agreement have been observed previously in patients without regional wall motion abnormality [18]. Moreover, LVEF estimation is more often asked for in patients with coronary artery disease and so with regional wall motion abnormality. So, assessment of its accuracy in this particular subgroup of patients with regional wall motion abnormality is rather more relevant.

4.1. Technical limitations The ABD method, though proposed to be a simplification over the manual edge detection method in quantitative left ventricular function assessment, has many technical limitations. Cavity clutter and endocardial drop out are the major problems with this method which cause false estimation of the endocardial borders and so the LVEF. These factors vary significantly with the varying gain setting [11], which makes this technique highly gain dependent. In this method, papillary muscles and false tendons are excluded from the cavity volume measurement which

198

R. Sapra et al. / International Journal of Cardiology 65 (1998) 193 – 199

causes underestimation of the left ventricular volumes. In addition, movement of the mitral valve inside the left ventricle during diastole may cause underestimation of end diastolic volume. On the other hand, movement of the atrio ventricular plane towards the apex may cause inclusion of a part of the left atrium in the region of interest and so may cause overestimation of the end systolic volume. The overestimation of end systolic volume may also be due to decrease in backscatter in systole which makes the borders less distinct so giving spuriously larger systolic area measurement [8]. And lastly, attempts to obtain images which are adequate for border detection often foreshortens the left ventricle, so sacrificing the maximal left ventricular cavity size for the sake of optimizing border visualization. Some of these errors can be minimized by certain modifications in ABD methodology. Incorporation of two orthogonal planes, like apical 4 and apical 2 chamber windows in estimation of LVEF, may improve its accuracy, more so in patients with regional wall motion abnormality. The gain setting requirements for optimization of endocardial border visualization may be different during diastole and systole. This problem may be overcome by gating the gain setting with the ECG signal, so that gain setting automatically varies during systole and diastole.

5. Conclusions The automated border detection method appears deceptively easy to use, but in reality is highly operator- and gain-setting-dependent and has significant disagreement in LVEF estimation when compared with the presently recommended methods of LVEF estimation. Moreover, this technique is not feasible in a significant proportion of patients because of poor definition of endocardial borders. So, at the present stage, the automated border detection method does not appear to be reliable enough to replace the offline manual border detection method or radionuclide ventriculography for LVEF measurement. Further refinements in its technology may improve its accuracy in left ventricular systolic function assessment.

References [1] Feigenbaum H. Echocardiography. Philadelphia: Lea and Febiger, 1986:127–87. [2] Geiser EA, Oliver LH, Gardin JM, Kerber RE, Parisi AF, Reichek N, Werner JA, Weyman AE. Clinical validation of an edge detection algorithm for two-dimensional echocardiographic short-axis images. J Am Soc Echocardiogr 1988;1:410–21. [3] Buda AJ, Delp EJ, Meyer CR, Jenkins JM, Smith DN, Bookstein FL, Pett B. Automatic computer processing of echocardiograms. Am J Cardiol 1983;52:384–9. [4] Jenkins JM, Qian G, Besozzi M, Delp EJ, Buda AJ. Computer processing of echocardiographic images of automated edge detection of left ventricular boundaries. Comput Cardiol, IEEE, 1981:391–4. [5] Zwehl W, Levy R, Garcia E, Haendchen RV, Childs W, Corday SR, Meerbaum S, Corday E. Validation of a computerized edge detection algorithm for quantitative two-dimensional echocardiography. Circulation 1983;68:1227–35. [6] Geiser EA, Conetta DA, Limacher MC, Stockton VO, Oliver LH, Jones B. A second generation computer based edge detection algorithm for short axis two-dimensional echocardiographic images: accuracy and improvement in interobserver variability. J Am Soc Echocardiogr 1990;3:79–90. [7] Perez JE, Waqgoner AD, Barzilai B, Melton Jr HE, Miller JG, Sobel BE. On line assessment of ventricular function by automatic boundary detection and ultrasonic backscatter imaging. J Am Coll Cardiol 1992;19:313–20. [8] Vered Z, Barzilai B, Mohr GA, Thomas III LJ, Sobel BE, Shoop TA, Melton HE, Miller JG, Perez JE. Quantitative ultrasonic tissue characterization with real time integrated back scatter imaging in normal human subjects and patients with dilated cardiomyopathy. Circulation 1987;79:1067–74. [9] Thomas III LJ, Barzilai B, Perez JE, Sobel BE, Wickline SA, Miller JG. Quantitative real time imaging of myocardium based on ultrasonic integrated backscatter. IEEE Trans Ultrason Ferroelectr Freq Control 1989;36:466–70. [10] Shorton DJ, Miller JG, Wickline SA, Barzilai B, Collins SM, Perez JE. Ultrasonic characterization of cardiovasc tissue. In: Marcus M, Schelbert HR, Skonton DJ, Wolff G, editors. Cardiac imaging: principles and practice. Philadelphia: WB Saunders, 1990:530–56. [11] Vandenberg BF, Rath LS, Stuhlmuller P, Melton HE, Skorton DJ. Estimation of left ventricular cavity area with an on line semiautomated echocardiographic edge detection system. Circulation 1992;86:159–66. [12] Shah PM, Crawford M, DeMaria A, et al. Recommendations for quantification of left ventricle by two-dimensional echocardiography. J Am Soc Echocardiogr 1989;2:358–67. [13] Morrissey RL, Siu SC, Guerrero JL, Newell JB, Weyman AE, Picard M. Automated assessment of ventricular volume and function by echocardiography. Validation of automated border detection. J Am Soc Echocardiogr 1994;7:107–15. [14] Yvorchuk KJ, Davies RA, Chan KL. Measurement of left ventricular ejection fraction by acoustic quantification and comparison with radionuclide angiography. J Am Cardiol 1994;74:1052–6. [15] Stewart WJ, Rodkey SM, Gunawardena S, White RD, Luvisi B, Klien Al, Sccedo E. Left ventricular volume calculation with integrated backscatter from echocardiography. J Am Soc Echocardiogr, 1993;6:553–63. [16] Marcus RH, Bednarz J, Coulden R, Shroff S, Lipton M, Lang RM.

R. Sapra et al. / International Journal of Cardiology 65 (1998) 193 – 199 Ultrasonic backscatter system for automated online endocardial boundary detection evaluation by ultrafast computed tomography. J Am Cardiol 1993;22:839–49. [17] Perez JE, Klein SC, Prater DM, Fraser CE, Cardona H, Waggner AD, Holland MR, Miller JG, Sobel BE. Automated online quantification of left ventricular dimensions and function by echocardiography with back scatter imaging and lateral gain compensation. Am J Cardiol 1992;70:1200–5.

199

[18] Chandra S, Bahl VK, Reddy SCB, Malhotra A, Wasir HS. Comparison of echocardiographic acoustic quantification system and radionuclide ventriculography for estimating left ventricular ejection fraction. Validation in patients without regional wall motion abnormalities. Am Heart J 1997;133:359–63. [19] Bland JM, Altman DG. Statistical method for assessing agreement between two methods of clinical measurement. Lancet 1986;1:307– 10.