- Email: [email protected]
Automatic myocardial segmentation in dynamic contrast enhanced perfusion MRI using Monte Carlo dropout in an encoder-decoder convolutional neural network
Journal Pre-proof
Automatic myocardial segmentation in dynamic contrast enhanced perfusion MRI using Monte Carlo dropout in an encoder-decoder convolutional neural network Yoon-Chul Kim , Khu Rai Kim , Yeon Hyeon Choe PII: DOI: Reference:
S0169-2607(19)31153-8 https://doi.org/10.1016/j.cmpb.2019.105150 COMM 105150
To appear in:
Computer Methods and Programs in Biomedicine
Received date: Revised date: Accepted date:
18 July 2019 7 October 2019 21 October 2019
Please cite this article as: Yoon-Chul Kim , Khu Rai Kim , Yeon Hyeon Choe , Automatic myocardial segmentation in dynamic contrast enhanced perfusion MRI using Monte Carlo dropout in an encoderdecoder convolutional neural network, Computer Methods and Programs in Biomedicine (2019), doi: https://doi.org/10.1016/j.cmpb.2019.105150
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 Published by Elsevier B.V.
Highlights A method for automatic segmentation of the myocardium in dynamic contrast enhanced first pass perfusion MRI is developed and validated. The method applies a U-Net model trained on cardiac cine data to dynamic perfusion data. Uncertainty estimates derived from Monte-Carlo dropout U-Net are used to select optimal frames for myocardial segmentation.
1
Automatic myocardial segmentation in dynamic contrast enhanced perfusion MRI using Monte Carlo dropout in an encoder-decoder convolutional neural network Yoon-Chul Kim1, Khu Rai Kim2, and Yeon Hyeon Choe3* 1
Clinical Research Institute, Samsung Medical Center, Sungkyunkwan University School of
Medicine, Seoul, South Korea, 2Department of Electronic Engineering, Sogang University, Seoul, South Korea, 3Department of Radiology and HVSI Imaging Center, Heart Vascular Stroke Institute, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, South Korea *
Correspondence to:
Yeon Hyeon Choe, M.D., Ph.D. Department of Radiology and HVSI Imaging Center, Heart Vascular Stroke Institute, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Ilwon-ro, Gangnam-gu, Seoul, 06351, Korea Phone: +82-2-3410-2509, Fax: +82-2-3410-2559; E-mail: [email protected] Journal: Computer Methods and Programs in Biomedicine Cover Title: Myocardial segmentation with uncertainty estimation
2
ABSTRACT Background and Objective: Cardiac perfusion magnetic resonance imaging (MRI) with first pass dynamic contrast enhancement (DCE) is a useful tool to identify perfusion defects in myocardial tissues. Automatic segmentation of the myocardium can lead to efficient quantification of perfusion defects. The purpose of this study was to investigate the usefulness of uncertainty estimation in deep convolutional neural networks for automatic myocardial segmentation. Methods: A U-Net segmentation model was trained on the cardiac cine data. Monte Carlo dropout sampling of the U-Net model was performed on the dynamic perfusion datasets frameby-frame to estimate the standard deviation (SD) maps. The uncertainty estimate based on the sum of the SD values was used to select the optimal frames for endocardial and epicardial segmentations. DCE perfusion data from 35 subjects (14 subjects with coronary artery disease, 8 subjects with hypertrophic cardiomyopathy, and 13 healthy volunteers) were evaluated. The Dice similarity scores of the proposed method were compared with those of a semi-automatic U-Net segmentation method, which involved user selection of an image frame for segmentation in the cardiac perfusion dataset. Results: The proposed method was fully automatic and did not require manual labeling of the cardiac perfusion image data for model development. The mean Dice similarity score of the proposed automatic method was 0.806 (±0.096), which was comparable to the 0.808 (±0.084) Dice score of the semi-automatic U-Net segmentation method (intraclass correlation coefficient
Reviewer#2 2.2 Statistical significance
= 0.61, P < 0.001). Conclusions: Our study demonstrated the feasibility of applying an existing model trained on cardiac cine data to dynamic cardiac perfusion data to achieve robust and automatic
3
segmentation of the myocardium. The uncertainty estimates can be used for screening purposes, which would facilitate the cases with high endocardial and epicardial uncertainty estimates to be sent for further evaluation and correction by human experts. Key words: MRI; Heart; Perfusion; Segmentation; Convolutional neural network
4
1. Introduction Cardiac perfusion magnetic resonance imaging (MRI) involves no radiation risks and is a useful tool for the non-invasive detection of ischemic lesions in the myocardium [1]. The accurate diagnosis of coronary artery disease (CAD) is important for proper treatment planning, cost saving, and improved clinical outcomes [2-4]. Cardiac perfusion MRI acquires dynamic images of the cardiac short-axis slices during the first pass of an extracellular contrast agent bolus, under an adenosine induced vasodilation or an exercise condition. Automatic myocardial segmentation in dynamic contrast enhanced (DCE) perfusion might be useful for the rapid quantification of perfusion defects in patients with suspected CAD [5-7] or coronary microvascular dysfunction [8]. Researchers have investigated various image processing and analysis techniques to improve the robustness and accuracy of the quantification of the myocardial blood flow and perfusion defects [9, 10]. Automation of myocardial segmentation is a key element in the fully automatic computer aided assessment of cardiac anatomy and function [11]. Recent deep learning technology based on convolutional neural networks (CNNs) has proved effective in segmenting the endocardial or myocardial contours [12-16] in cardiac cine MRI, which is routinely used to assess the systolic and diastolic functions of the left and right ventricles. However, it is unclear whether automatic myocardial segmentation is also robust and accurate in DCE myocardial perfusion MRI, which is a clinical imaging sequence used to assess myocardial ischemia. One of the challenging aspects in dynamic perfusion image processing is the characteristic of dynamic changes of the T1-weighted contrast agent that can influence the image contrast and potentially affect segmentation performance frame-by-frame (Figure 1). It is
5
important to systematically determine an “optimal” image frame that is suitable for myocardial segmentation. Model uncertainty in deep learning has recently gained attention in computer vision [17-19]. The investigation of model uncertainty is motivated by the fact that it is not possible, with the mere use of a CNN model, to ascertain whether the output prediction is trustworthy. Uncertainty estimates have been used for image classification in computer vision and medical image processing [20]. Recently, uncertainty estimates have been employed mainly to predict the accuracy of image segmentation [21]. In the cases where a high level of uncertainty is estimated by a machine learning model, the unseen test image can be sent to a human expert for further human evaluation of the segmentation results and for corrections if any segmentation errors are detected. As the image contrast highly depends on the time during the first pass of DCE, applying a trained segmentation CNN model to each image frame would most likely produce different segmentation results. In this study, we estimated uncertainty information in each image frame and used the uncertainty estimates to guide our selection of appropriate image frames for endocardial and epicardial segmentations. To the best of our knowledge, this is the first study that investigates the feasibility of uncertainty estimation based on Monte Carlo dropout sampling to achieve automatic myocardial segmentation in DCE first pass myocardial perfusion data. Furthermore, we investigated the feasibility of the segmentation model trained on cardiac cine data and evaluated whether the model could successfully transfer the performance of myocardial segmentation to cardiac perfusion data. The proposed method was evaluated by comparing its performance with other existing methods.
2. Methods
6
Reviewer#2 2.1
Clarify contribution s of this study
This section covers the data acquisition, deep learning model development, segmentation using uncertainty information in DCE perfusion MRI, and evaluation of the employed methods. Figure 2 illustrates the flowchart of the proposed method.
Reviewer#1 1.1
2.1. Data Acquisition Cardiac MRI scans were performed on a 1.5 T scanner (Siemens Avanto, Erlangen, Germany). All clinical MR examinations were approved by our institutional review board, and informed consent was obtained from the subjects prior to the MRI scans. Cardiac perfusion data from 35 adult subjects were evaluated. Among the 35 subjects, there were 14 patients with CAD, 8 patients with hypertrophic cardiomyopathy (HCM), and 13 healthy volunteers (VOL). A conventional electrocardiogram (ECG) gated fast gradient echo sequence was used to acquire breath-hold dynamic short-axis cardiac perfusion images during the first pass of a bolus of an extracellular contrast agent. The imaging parameters were as follows: flip angle = 15°, slice thickness = 8 mm, field-of-view (FOV) = 400 × 315 mm2, image matrix = 320 × 252, image pixel spacing = 1.25 × 1.25 mm2, and number of image frames = 80. As the acquisition of a perfusion image series required more than one minute, the subject resumed breathing after a 15-20 s of breath-hold. Respiratory motion changes the location of the heart, thus affecting the location of the myocardium. Motion correction was automatically performed frame-by-frame to align the myocardium across the frames and correct the misregistration caused by respiration [22]. 2.2. Model Development from Cardiac Cine Data A U-Net architecture [23] with dropout [24] was trained on the cardiac cine data. We used the Tensorlayer library to implement the U-Net model [25, 26]. The total number of subjects used for model development was 110 and included the publicly available York [27] data (n=33)
7
Detail about data acquisition and types of subjects
and our internal data (n=77). Cardiac cine MRI acquisition parameters for the internal data were as follows: ECG gating, number of frames = 30, slice thickness = 6 mm, echo time (TE) = 1.3 ms, FOV = 350 × 300 mm2, pixel size = 1.29 × 1.29 mm2, and spacing between the slices = 10 mm. The internal data (n=77) included 15 HCM patients, 17 healthy volunteers, 16 CAD patients, 16 AS patients, 11 patients with cardiac amyloidosis, and two subjects with chest pain symptoms. The total number of cardiac short-axis images was 17,683, ranging from the apical to basal slices in both the systolic and diastolic frames. In total, 13,535 images from 88 subjects were used for training, while 4,148 images from 22 subjects were used for validation. The dimensions of the cardiac cine images, endocardial masks, and epicardial masks were set to 256 × 216, after cropping the peripheral regions of the images. The image intensity was scaled to the 0 – 255 range, and each image was saved in the png format. The dropout ratio ranged from 0.1 to 0.3, following the implementation of the concrete dropout [28]. The batch size was set to four, and the beta value was set to 0.9. The maximum number of epochs was set to 50. Figure 3 shows the learning curves of the myocardial segmentation models with different learning rates. After inspecting the training and validation learning curves, a learning rate of 0.0001 and an epoch number of 20 were selected for the segmentation model, which was used to estimate the segmentation uncertainty in the cardiac perfusion dataset. 2.3. Segmentation with Uncertainty Information Our experiments with the uncertainty estimation revealed that the early image frames, such as the no contrast enhancement (Figure 4a) and right ventricle (RV) enhancement (Figure 4b) images, produced low uncertainty estimates even though their image contrast was not suitable for myocardial segmentation. Hence, we restricted the uncertainty estimation to the frames of
8
interest by excluding both the early image frames and late image frames, which experienced a greater misregistration of the myocardium. The automatic detection of the peak RV enhancement frame
was based on [29]. Figure 5
illustrates the overall process. The minimum intensity projection (MinIP) and maximum intensity projection (MIP) maps were obtained pixel-wise over time. (
{ where (
(
)
(
)
(
)
(1)
) (
) represents the series of DCE perfusion images. In
Reviewer#1
) defined in Eq. (1), 1.2
the subcutaneous fat region appears bright, while the LV and RV regions appear dark, as shown in Figure 5a. (
)
( (
{
) )
(2)
T is a threshold used to exclude the high-intensity subcutaneous fat signals. A morphological operation on the binary mask defined in Eq. (2) was applied to sufficiently enclose the fat regions. The MIP map was multiplied by the binary mask referred to as
(
). A pixel location ( (
)
to generate a fat-free MIP map ) that maximized
(
)
(
was determined.
)
(3)
After excluding the fat region, maximal signal intensity was observed within the RV cavity, as opposed to the LV cavity. This was because in general the RV is closer to the surface coil than the LV, thus producing a higher signal intensity in the RV. Furthermore, the contrast agent entered the RV prior to the LV, which caused a smaller dispersion of the contrast agent bolus in the RV. Finally,
was obtained by finding the time index corresponding to the peak RV signal
enhancement, as shown in Figure 5c. In this study, a seeded region growing algorithm with the
9
Write down the math
seed point (
) defined in Eq. (3) was applied to obtain a region of interest (ROI) over the RV.
The mean signal value within this ROI was computed for every frame. This helped improve the signal-to-noise ratio (SNR) in the RV signal time series. The segmentation uncertainty was estimated in a frame range of [
], which represented the candidate frames exhibiting
moderate to high myocardial contrast. We used the Monte Carlo dropout [17, 20] for uncertainty estimation with the number of repetitions = 100, where the output of the U-Net model was obtained with dropout (i.e., random selections of the units in each layer) for each realization. We obtained a standard deviation (SD) map, denoted by
(x, y), from the result of the Monte Carlo dropout sampling. In this study,
the concrete dropout library was used [28]. For each time frame of interest, the pixel-wise summation of the SD map was calculated as a metric of uncertainty, denoted by SSD. ∑
∑
(
)
(4)
where N and M are the number of image rows and columns, respectively. We computed the SSD metric for each time frame within the frame range. An “optimal” image frame for endocardial (or epicardial) segmentation, which minimized the SSD values in the time frame of interest, was selected. 2.4. Evaluation For evaluation, manual segmentation was performed by manually selecting an image frame of interest that showed adequate contrast between the myocardium and blood, and drawing the endocardial and epicardial contours. Manual segmentation served as the reference. The semiautomatic active contour [5] (Active-contour), automatic U-Net segmentation (Auto-Unet), and semi-automatic U-Net (Semiauto-Unet) segmentation methods were implemented and compared. The Active-contour method required user selection of the center point of the LV. The remaining
10
Reviewer#2 2.2 More comparison experiment needed
steps were fully automatic, which involved the selection of an image frame where the LV signal first reached the maximum value [5]. The Auto-Unet method first determined the RV enhancement frame
based on the procedure shown in Figure 5. The U-Net model was
applied to the LV enhancement frame, which was chosen to be
+7, to estimate the
endocardial and epicardial contours. The Semiauto-Unet method relied on the manual selection of an appropriate image frame, which showed adequate contrast between the myocardium and blood, followed by the U-Net segmentation to estimate the endocardial and epicardial contours. The Dice similarity score was calculated to evaluate/compare the segmentation performance between each method and manual segmentation. Computation time of the entire process of the proposed method was measured on a 64-bit Microsoft Windows 10 operating system with a 16core Intel Xeon Gold 6134 CPU @ 3.20 GHz, 64 GB RAM, and NVIDIA Quadro P5000 GPU (16 GB memory size). Statistical analysis was performed using the R software (R Foundation for Statistical Computing, Vienna, Austria). The intraclass correlation coefficient (ICC) and its 95% confidence interval (CI) were computed between two segmentation methods in terms of Dice similarity coefficients. A P value < 0.05 was considered to indicate a statistically significant correlation, given the null hypothesis of no relationship between the two measurements.
3. Results Figure 4 illustrates the uncertainty estimates for four distinct contrast enhancement patterns: no enhancement, RV enhancement, LV enhancement, and myocardial enhancement. Low uncertainty estimates were observed corresponding to the no enhancement and RV enhancement frames, which were not helpful in segmenting the myocardium. Hence, these frames were not considered when selecting the appropriate frames based on the SSD metric. The LV
11
Reviewer#2 2.2 Test statistical significance of the methods
enhancement frame shown in Figure 4c shows lower SD maps for both the endocardial and epicardial segmentations, as compared to the myocardial enhancement frame shown in Figure 4d. A visual comparison between the SD maps in Figures 4c and 4d clearly indicates that the former achieves higher myocardial segmentation accuracy as compared to the latter. Table 1 shows the Dice scores in three different subject groups and comparisons between the Dice scores of four different methods. Table 2 compares the ICC of two different segmentation methods. The proposed automatic method based on uncertainty estimation was comparable to the Semiauto-Unet method in terms of the mean Dice similarity score (0.806 vs. 0.808) and showed a statistically significant correlation (ICC = 0.61, 95% CI = [0.36 – 0.78], P value < 0.001) as compared with the Semiauto-Unet method. The CAD group showed the lowest mean Dice scores in all the evaluated methods. The HCM group showed the highest mean Dice scores in the AutoUnet, Semiauto-Unet, and proposed methods. The VOL group showed the highest mean Dice score in the Active-contour method. Figures 6a-b show the plots of the SSD values, as calculated from the Monte Carlo dropout samplings of the U-Net model, with respect to the time frame. The optimal frames with minimal SSD values within the frame range of interest are indicated in red and green, respectively. Figures 6c-d show the optimal image frames and endocardial (red) and epicardial (green) contours overlaid onto the images, respectively, which qualitatively exhibit good segmentation accuracy. Figure 7 shows the SSD values for the endocardial and epicardial uncertainties and their relationships with the Dice scores. Each sample point represents a set of endocardial and epicardial SSD results and its corresponding Dice score group for a subject’s perfusion dataset. The samples of the Dice group (0.85 <= Dice score) are in < 30 in the endocardial SSD and in <
12
60 in the epicardial SSD, and they are closer to the origin of the X-Y axes than the samples of the other Dice groups. An endocardial SSD > 40 and epicardial SSD > 70 would most likely result in poor Dice scores. Comparisons between the Semiauto-Unet and proposed methods are shown in Figures 8 and 9. The blue contours indicate the manual myocardial segmentation results, which served as the reference. The red contours indicate the endocardial segmentation results, while the green contours indicate the epicardial segmentation results. Figure 8 shows three patient cases with their Dice scores > 0.85 obtained using the proposed method. Figure 9 shows three patient cases with their Dice scores < 0.85 for obtained using the proposed method. The two methods mostly produce comparable segmentation results, but the top row of Figure 9 shows a case wherein the proposed automatic method achieved a higher segmentation accuracy as compared to the Semiauto-Unet method.
4. Discussion DCE myocardial perfusion images show a variation in the image intensity across time frames, as shown in Figure 1. The contrast enhancement timing varies depending on the subject. Hence, a manual procedure involving a frame-by-frame visual inspection of the cardiac perfusion data is helpful for selecting an image frame that shows adequate image contrast. This work focused on the automatic selection of appropriate image frames for myocardial segmentation. Encoderdecoder CNN architectures (e.g., U-Net [23], SegNet [30]) have emerged as powerful methods for automatic segmentation, and the segmentation uncertainty for these networks can be estimated using Monte-Carlo dropout sampling [17]. The automatic selection of image frames in DCE perfusion relied on minimizing the uncertainty metrics. In this study, we proposed the SSD
13
as a metric. The experimental results indicated that the uncertainty information derived from Monte Carlo dropout suggested appropriate image frames for segmentation of the DCE images. The mean Dice similarity score in the perfusion dataset was approximately 0.8. Although the perfusion data evaluated in this study admittedly had lower spatial resolution and quality than the cardiac cine dataset, the mean Dice score was not very high, requiring further improvement. It should be noted that the segmentation model was trained only on the cardiac cine data. We might expect a higher myocardial segmentation performance/accuracy if the segmentation model was trained on the cardiac perfusion data, or both the cardiac cine and perfusion data. Given that the cardiac cine image data and their associated myocardial masks have mainly been used in the development of deep learning models [12-16], our study focused on investigating the feasibility of the model, which was trained on the cardiac cine, to automatically segment the myocardial perfusion without any additional labeling of the perfusion data. The successful transfer/application of the trained model to the DCE perfusion data can be explained by the following: (1) Both the cardiac cine and cardiac perfusion datasets were acquired in the shortaxis imaging planes, and the slice anatomy of the cardiac cine images appeared similar to that of the cardiac perfusion images. (2) Certain image frames of the dynamic cardiac perfusion dataset exhibited a myocardium and blood contrast that was similar to the cardiac cine images. When fitting the optimal image frames for the epicardial and endocardial segmentations, we relied on the SD map derived from the Monte Carlo dropout results. A close observation of the SD maps suggested regional differences in the SD. For example, in a certain frame, the SD of the Reviewer#2 2.3
myocardial septal region was lower than that of the lateral region, as the contrast between the myocardium and RV was higher than the contrast between the myocardium and lung. A regional assessment of the uncertainty might help identify more suitable image frames for myocardial
14
Detailed explanation of successful transfer
segmentation, although this would require a post processing technique of automatic localization of myocardial regions based on its geometry (e.g., localization of the RV insertion points). In addition, the investigation between the SSD values and Dice coefficient (Figure 7) suggests that the set of endocardial and epicardial SSD values might help predict the segmentation performance. The selected frame range affected the segmentation results in a few cases. For example, a few HCM patients showed slow RV enhancement curves, implying that a longer delay from the peak RV enhancement frame is necessary. The healthy volunteers showed relatively fast RV enhancement curves, indicating that a shorter delay from the peak RV enhancement frame would be desirable. The adaptive adjustment of the frame range based on the enhancement time-
Reviewer#1 1.3
intensity curve was not investigated in this study, and this remains as future studies. Need comments on small approximately 86 s, which was significantly longer than the time required by the semi-automated test dataset and generalizabi U-Net method. The increased time can be attributed to the Monte Carlo simulations involving lity of the method.
The computation time for myocardial segmentation using the proposed method was
100 trials per image frame and 20 repetitions corresponding to the number of frames (20 frames in this study). The computation time can be reduced by reducing the number of trials or image frames. Alternatively, parallel processing, not used in this study, can help increase the computational efficiency. In conclusion, we have demonstrated the feasibility of automatic frame selection for endocardial (or epicardial) segmentation in dynamic cardiac perfusion MRI, based on the uncertainty information from the Monte Carlo dropout in the U-Net segmentation model. The proposed method was fully automatic and resulted in a mean Dice similarity score that was comparable to that of the semi-automatic U-Net method. The SSD metric adopted in this study
15
can be used for screening purposes, so that the cases with high endocardial and epicardial SSD values can be referred for further evaluation and correction by human experts. This was a single center and single vendor study, and a small set of patients were evaluated. Evaluating the multicenter and multi-vendor data would strengthen the robustness of the proposed method, which will be investigated in future studies. Acknowledgments The manuscript has not been published elsewhere and is not currently being considered for publication in another journal. The authors have been actively involved in the study and are responsible for the content of this manuscript. This work was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT, and Future Planning (Grant numbers: NRF-2015 R1C1A1A02036340, NRF-2018 R1D1A1B07042692). We are grateful for the comments from the reviewers. We would like to thank Editage for English language editing. Competing Interests The authors declare no competing interests.
16
TABLES Table 1. Mean and standard deviation (SD) values of the Dice similarity scores among the four evaluated methods. Manual segmentation served as the reference.
Method
Dice score (mean ± SD) Reviewer#2
CAD (n = 14)
HCM (n = 8)
VOL (n = 13)
Active-contour
0.654 ± 0.156
0.665 ± 0.092
0.728 ± 0.076
Total (n = 35) 2.2 0.684 ± 0.119
Auto-Unet
0.674 ± 0.116
0.775 ± 0.193
0.760 ± 0.137
0.729 ± 0.147 Auto-Unet
Semiauto-Unet
0.785 ± 0.109
0.835 ± 0.093
0.817 ± 0.029
0.808 ± 0.084
Proposed
0.778 ± 0.118
0.838 ± 0.108
0.815 ± 0.051
0.806 ± 0.096
Addition of
Abbreviations: SD - standard deviation; CAD - coronary artery disease; HCM - hypertrophic cardiomyopathy; VOL - healthy volunteer
17
for more comparison
Table 2. Intraclass correlation coefficient (ICC) and statistical significance between the Dice scores of the two myocardial segmentation methods applied to the test subjects (n = 35).
Dice score
ICC
95% CI
P value
Proposed vs. Semiauto-Unet
0.61
0.36 – 0.78
< 0.001
Proposed vs. Auto-Unet
0.46
0.16 – 0.69
< 0.001
Proposed vs. Active-contour
-0.06
-0.38 – 0.28
0.629
Semiauto-Unet vs. Auto-Unet
0.33
0.01 – 0.60
0.023
Semiauto-Unet vs. Active-contour
-0.11
-0.42 – 0.23
0.730
Auto-Unet vs. Active-contour
0.27
-0.06 – 0.55
0.063
Abbreviations: ICC - intraclass correlation coefficient; CI - confidence interval
18
Reviewer#2 2.2 Test statistical significance of the methods
FIGURES
Figure 1. An example of a cardiac dynamic contrast enhancement (DCE) MRI series. The goal of this study was to automatically and accurately segment the myocardium. Note that the image contrast varies over the time frames owing to contrast enhancement. The red arrow in the t = 33 frame indicates a perfusion defect, which appears as a round dark band surrounding the left ventricular blood pool.
19
Figure 2. Flowchart of the proposed method.
20
Figure 3. Learning curves for the U-Net dropout model, which was trained and validated on the cardiac cine MRI dataset. Training (dashed) and validation (solid) learning curves are shown for the learning rates of 0.0001, 0.0002, and 0.00005. After visual inspection of the learning curves, we selected the learning rate = 0.0001 and epoch number = 20 for our segmentation model, which was used for the uncertainty estimation and segmentation of the cardiac perfusion data.
21
Figure 4. Monte-Carlo dropout results of the U-Net model in four representative frames. (a) No enhancement. (b) RV enhancement. (c) LV enhancement. (d) Myocardial enhancement. (a) and (b) show no segmentation along the endocardial and epicardial borders. (c) shows good endocardial and epicardial segmentation masks with relatively low segmentation uncertainty. The non-zero values in the standard deviation (SD) maps shown in (c) are distributed near the endocardial and epicardial borders, whereas the non-zero values in the SD maps shown in (d) are more widely distributed near the endocardial and epicardial borders with higher intensity, as compared to (c).
22
Reviewer#1 1.2
Figure 5. A schematic showing the automatic detection of the peak RV enhancement frame. (a) The minimum intensity projection (MinIP) map highlights the bright fat signal. (b) The fat region is masked out and the maximum signal intensity can be observed within the RV blood pool (red cross). (c) The peak RV enhancement frame can be identified from the RV signal time series (red dot).
23
Visual explanation of peak RV enhanceme nt frame detection
Figure 6. Automatic segmentation based on the uncertainty estimates. (a-b) Sum of standard deviation (SSD) following Monte Carlo dropout is plotted with respect to the frame index. The plot shows the temporal region of interest (ROI), which ranges from 21 to 40. Frame 24 has the minimum SSD value for the endocardial case, while frame 27 has the minimum SSD value for the epicardial case. (c-d) Deep learning segmentation results corresponding to (c) the endocardial case and (d) the epicardial case. 24
Figure 7. The relationship of the endocardial sum of standard deviation (SSD) and epicardial SSD with the Dice similarity score groups.
25
Figure 8. Comparison between the semi-automatic U-Net (Semiauto-Unet) and proposed methods for the case where the Dice score of the proposed method is > 0.85. (Top) CAD patient, (endocardial SSD, epicardial SSD, Dice score) = (22.6, 37.4, 0.882). (Middle) HCM patient, (endocardial SSD, epicardial SSD, Dice score) = (21.7, 58.6, 0.886). (Bottom) a healthy volunteer, (endocardial SSD, epicardial SSD, Dice score) = (21.6, 33.9, 0.862).
26
Figure 9. Comparison between the semi-automatic U-Net (Semiauto-Unet) and proposed methods for the case where the Dice score of the proposed method is < 0.85. (Top) CAD patient, (endocardial SSD, epicardial SSD, Dice score) = (38.5, 53.9, 0.803). (Middle) HCM patient, (endocardial SSD, epicardial SSD, Dice score) = (102.9, 136.6, 0.587). (Bottom) healthy volunteer, (endocardial SSD, epicardial SSD, Dice score) = (29.7, 102.0, 0.803).
27
No conflict of interest.
References [1] O. R. Coelho-Filho, C. Rickers, R. Y. Kwong, and M. Jerosch-Herold, "MR myocardial perfusion imaging," Radiology, vol. 266, no. 3, pp. 701-15, Mar 2013, doi: 10.1148/radiol.12110918. [2] J. Schwitter et al., "MR-IMPACT II: Magnetic Resonance Imaging for Myocardial Perfusion Assessment in Coronary artery disease Trial: perfusion-cardiac magnetic resonance vs. single-photon emission computed tomography for the detection of coronary artery disease: a comparative multicentre, multivendor trial," European heart journal, vol. 34, no. 10, pp. 775-81, Mar 2013, doi: 10.1093/eurheartj/ehs022. [3] M. R. Patel et al., "Low diagnostic yield of elective coronary angiography," N Engl J Med, vol. 362, no. 10, pp. 886-95, Mar 11 2010, doi: 10.1056/NEJMoa0907272. [4] P. S. Douglas et al., "Outcomes of anatomical versus functional testing for coronary artery disease," N Engl J Med, vol. 372, no. 14, pp. 1291-300, Apr 2 2015, doi: 10.1056/NEJMoa1415516. [5] G. Tarroni et al., "Myocardial perfusion: near-automated evaluation from contrast-enhanced MR images obtained at rest and during vasodilator stress," Radiology, vol. 265, no. 2, pp. 576-83, Nov 2012, doi: 10.1148/radiol.12112475. [6] A. M. Weng, C. O. Ritter, J. Lotz, M. J. Beer, D. Hahn, and H. Kostler, "Automatic postprocessing for the assessment of quantitative human myocardial perfusion using MRI," European radiology, vol. 20, no. 6, pp. 1356-65, Jun 2010, doi: 10.1007/s00330-009-1684-z. [7] G. M. Beache, F. Khalifa, A. El-Baz, and G. Gimel'farb, "Fully automated framework for the analysis of myocardial first-pass perfusion MR images," Medical physics, vol. 41, no. 10, p. 102305, Oct 2014, doi: 10.1118/1.4893531. 28
[8] P. G. Camici and F. Crea, "Coronary microvascular dysfunction," N Engl J Med, vol. 356, no. 8, pp. 830-40, Feb 22 2007, doi: 10.1056/NEJMra061889. [9] V. Gupta et al., "Cardiac MR perfusion image processing techniques: a survey," Medical image analysis, vol. 16, no. 4, pp. 767-85, May 2012, doi: 10.1016/j.media.2011.12.005. [10] M. Jacobs, M. Benovoy, L. C. Chang, A. E. Arai, and L. Y. Hsu, "Evaluation of an automated method for arterial input function detection for first-pass myocardial perfusion cardiovascular magnetic resonance," Journal of cardiovascular magnetic resonance : official journal of the Society for Cardiovascular Magnetic Resonance, vol. 18, p. 17, Apr 8 2016, doi: 10.1186/s12968-016-0239-0. [11] P. Peng, K. Lekadir, A. Gooya, L. Shao, S. E. Petersen, and A. F. Frangi, "A review of heart chamber segmentation for structural and functional analysis using cardiac magnetic resonance imaging," Magma, vol. 29, no. 2, pp. 155-95, Apr 2016, doi: 10.1007/s10334015-0521-4. [12] L. K. Tan, Y. M. Liew, E. Lim, and R. A. McLaughlin, "Convolutional neural network regression for short-axis left ventricle segmentation in cardiac cine MR sequences," Medical image analysis, vol. 39, pp. 78-86, Jul 2017, doi: 10.1016/j.media.2017.04.002. [13] M. R. Avendi, A. Kheradvar, and H. Jafarkhani, "A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI," Medical image analysis, vol. 30, pp. 108-119, May 2016, doi: 10.1016/j.media.2016.01.005. [14] W. Xue, G. Brahm, S. Pandey, S. Leung, and S. Li, "Full left ventricle quantification via deep multitask relationships learning," Medical image analysis, vol. 43, pp. 54-65, Jan 2018, doi: 10.1016/j.media.2017.09.005.
29
[15] A. H. Curiale, F. D. Colavecchia, and G. Mato, "Automatic quantification of the LV function and mass: A deep learning approach for cardiovascular MRI," Comput Methods Programs Biomed, vol. 169, pp. 37-50, Feb 2019, doi: 10.1016/j.cmpb.2018.12.002. [16] T. A. Ngo, Z. Lu, and G. Carneiro, "Combining deep learning and level set for the automated segmentation of the left ventricle of the heart from cardiac cine magnetic resonance," Medical image analysis, vol. 35, pp. 159-171, Jan 2017, doi: 10.1016/j.media.2016.05.009. [17] Y. Gal and Z. Ghahramani, "Dropout as a bayesian approximation: Representing model uncertainty in deep learning," in international conference on machine learning, 2016, pp. 1050-1059. [18] A. Kendall, V. Badrinarayanan, and R. Cipolla, "Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding," arXiv preprint arXiv:1511.02680, 2015. [19] A. Kendall and Y. Gal, "What uncertainties do we need in bayesian deep learning for computer vision?," in Advances in Neural Information Processing Systems, 2017, pp. 55745584. [20] C. Leibig, V. Allken, M. S. Ayhan, P. Berens, and S. Wahl, "Leveraging uncertainty information from deep neural networks for disease detection," Sci Rep, vol. 7, no. 1, p. 17816, Dec 19 2017, doi: 10.1038/s41598-017-17876-z. [21] T. DeVries and G. W. Taylor, "Leveraging Uncertainty Estimates for Predicting Segmentation Quality," arXiv preprint arXiv:1807.00502, 2018.
30
[22] H. Xue et al., "Unsupervised inline analysis of cardiac perfusion MRI," Med Image Comput Comput Assist Interv, vol. 12, no. Pt 2, pp. 741-9, 2009. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/20426178. [23] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015: Springer, 2015, pp. 234-241. [24] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929-1958, 2014. [25] H. Dong et al., "Tensorlayer: A versatile library for efficient deep learning development," in Proceedings of the 25th ACM International Conference on Multimedia, 2017: ACM, pp. 1201-1204. [26] Y.-C. Kim, K. R. Kim, K. Choi, M. Kim, Y. Chung, and Y. H. Choe, "EVCMR: A tool for the quantitative evaluation and visualization of cardiac MRI data," Computers in biology and medicine, p. 103334, 2019. [27] A. Andreopoulos and J. K. Tsotsos, "Efficient and generalizable statistical models of shape and appearance for analysis of cardiac MRI," Medical image analysis, vol. 12, no. 3, pp. 335-57, Jun 2008, doi: 10.1016/j.media.2007.12.003. [28] Y. Gal, J. Hron, and A. Kendall, "Concrete dropout," in Advances in Neural Information Processing Systems, 2017, pp. 3581-3590. [29] Y. C. Kim, S. M. Kim, and Y. H. Choe, "Robust semi-automated quantification of cardiac MR perfusion using level set: Application to hypertrophic cardiomyopathy patient data,"
31
Computers in biology and medicine, vol. 71, pp. 162-73, Apr 1 2016, doi: 10.1016/j.compbiomed.2016.02.014. [30] V. Badrinarayanan, A. Kendall, and R. Cipolla, "SegNet: A Deep Convolutional EncoderDecoder Architecture for Image Segmentation," IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 12, pp. 2481-2495, Dec 2017, doi: 10.1109/TPAMI.2016.2644615.
32
Automatic myocardial segmentation in dynamic contrast enhanced perfusion MRI using Monte Carlo dropout in an encoder-decoder convolutional neural network Yoon-Chul Kim , Khu Rai Kim , Yeon Hyeon Choe PII: DOI: Reference:
S0169-2607(19)31153-8 https://doi.org/10.1016/j.cmpb.2019.105150 COMM 105150
To appear in:
Computer Methods and Programs in Biomedicine
Received date: Revised date: Accepted date:
18 July 2019 7 October 2019 21 October 2019
Please cite this article as: Yoon-Chul Kim , Khu Rai Kim , Yeon Hyeon Choe , Automatic myocardial segmentation in dynamic contrast enhanced perfusion MRI using Monte Carlo dropout in an encoderdecoder convolutional neural network, Computer Methods and Programs in Biomedicine (2019), doi: https://doi.org/10.1016/j.cmpb.2019.105150
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 Published by Elsevier B.V.
Highlights A method for automatic segmentation of the myocardium in dynamic contrast enhanced first pass perfusion MRI is developed and validated. The method applies a U-Net model trained on cardiac cine data to dynamic perfusion data. Uncertainty estimates derived from Monte-Carlo dropout U-Net are used to select optimal frames for myocardial segmentation.
1
Automatic myocardial segmentation in dynamic contrast enhanced perfusion MRI using Monte Carlo dropout in an encoder-decoder convolutional neural network Yoon-Chul Kim1, Khu Rai Kim2, and Yeon Hyeon Choe3* 1
Clinical Research Institute, Samsung Medical Center, Sungkyunkwan University School of
Medicine, Seoul, South Korea, 2Department of Electronic Engineering, Sogang University, Seoul, South Korea, 3Department of Radiology and HVSI Imaging Center, Heart Vascular Stroke Institute, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, South Korea *
Correspondence to:
Yeon Hyeon Choe, M.D., Ph.D. Department of Radiology and HVSI Imaging Center, Heart Vascular Stroke Institute, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Ilwon-ro, Gangnam-gu, Seoul, 06351, Korea Phone: +82-2-3410-2509, Fax: +82-2-3410-2559; E-mail: [email protected] Journal: Computer Methods and Programs in Biomedicine Cover Title: Myocardial segmentation with uncertainty estimation
2
ABSTRACT Background and Objective: Cardiac perfusion magnetic resonance imaging (MRI) with first pass dynamic contrast enhancement (DCE) is a useful tool to identify perfusion defects in myocardial tissues. Automatic segmentation of the myocardium can lead to efficient quantification of perfusion defects. The purpose of this study was to investigate the usefulness of uncertainty estimation in deep convolutional neural networks for automatic myocardial segmentation. Methods: A U-Net segmentation model was trained on the cardiac cine data. Monte Carlo dropout sampling of the U-Net model was performed on the dynamic perfusion datasets frameby-frame to estimate the standard deviation (SD) maps. The uncertainty estimate based on the sum of the SD values was used to select the optimal frames for endocardial and epicardial segmentations. DCE perfusion data from 35 subjects (14 subjects with coronary artery disease, 8 subjects with hypertrophic cardiomyopathy, and 13 healthy volunteers) were evaluated. The Dice similarity scores of the proposed method were compared with those of a semi-automatic U-Net segmentation method, which involved user selection of an image frame for segmentation in the cardiac perfusion dataset. Results: The proposed method was fully automatic and did not require manual labeling of the cardiac perfusion image data for model development. The mean Dice similarity score of the proposed automatic method was 0.806 (±0.096), which was comparable to the 0.808 (±0.084) Dice score of the semi-automatic U-Net segmentation method (intraclass correlation coefficient
Reviewer#2 2.2 Statistical significance
= 0.61, P < 0.001). Conclusions: Our study demonstrated the feasibility of applying an existing model trained on cardiac cine data to dynamic cardiac perfusion data to achieve robust and automatic
3
segmentation of the myocardium. The uncertainty estimates can be used for screening purposes, which would facilitate the cases with high endocardial and epicardial uncertainty estimates to be sent for further evaluation and correction by human experts. Key words: MRI; Heart; Perfusion; Segmentation; Convolutional neural network
4
1. Introduction Cardiac perfusion magnetic resonance imaging (MRI) involves no radiation risks and is a useful tool for the non-invasive detection of ischemic lesions in the myocardium [1]. The accurate diagnosis of coronary artery disease (CAD) is important for proper treatment planning, cost saving, and improved clinical outcomes [2-4]. Cardiac perfusion MRI acquires dynamic images of the cardiac short-axis slices during the first pass of an extracellular contrast agent bolus, under an adenosine induced vasodilation or an exercise condition. Automatic myocardial segmentation in dynamic contrast enhanced (DCE) perfusion might be useful for the rapid quantification of perfusion defects in patients with suspected CAD [5-7] or coronary microvascular dysfunction [8]. Researchers have investigated various image processing and analysis techniques to improve the robustness and accuracy of the quantification of the myocardial blood flow and perfusion defects [9, 10]. Automation of myocardial segmentation is a key element in the fully automatic computer aided assessment of cardiac anatomy and function [11]. Recent deep learning technology based on convolutional neural networks (CNNs) has proved effective in segmenting the endocardial or myocardial contours [12-16] in cardiac cine MRI, which is routinely used to assess the systolic and diastolic functions of the left and right ventricles. However, it is unclear whether automatic myocardial segmentation is also robust and accurate in DCE myocardial perfusion MRI, which is a clinical imaging sequence used to assess myocardial ischemia. One of the challenging aspects in dynamic perfusion image processing is the characteristic of dynamic changes of the T1-weighted contrast agent that can influence the image contrast and potentially affect segmentation performance frame-by-frame (Figure 1). It is
5
important to systematically determine an “optimal” image frame that is suitable for myocardial segmentation. Model uncertainty in deep learning has recently gained attention in computer vision [17-19]. The investigation of model uncertainty is motivated by the fact that it is not possible, with the mere use of a CNN model, to ascertain whether the output prediction is trustworthy. Uncertainty estimates have been used for image classification in computer vision and medical image processing [20]. Recently, uncertainty estimates have been employed mainly to predict the accuracy of image segmentation [21]. In the cases where a high level of uncertainty is estimated by a machine learning model, the unseen test image can be sent to a human expert for further human evaluation of the segmentation results and for corrections if any segmentation errors are detected. As the image contrast highly depends on the time during the first pass of DCE, applying a trained segmentation CNN model to each image frame would most likely produce different segmentation results. In this study, we estimated uncertainty information in each image frame and used the uncertainty estimates to guide our selection of appropriate image frames for endocardial and epicardial segmentations. To the best of our knowledge, this is the first study that investigates the feasibility of uncertainty estimation based on Monte Carlo dropout sampling to achieve automatic myocardial segmentation in DCE first pass myocardial perfusion data. Furthermore, we investigated the feasibility of the segmentation model trained on cardiac cine data and evaluated whether the model could successfully transfer the performance of myocardial segmentation to cardiac perfusion data. The proposed method was evaluated by comparing its performance with other existing methods.
2. Methods
6
Reviewer#2 2.1
Clarify contribution s of this study
This section covers the data acquisition, deep learning model development, segmentation using uncertainty information in DCE perfusion MRI, and evaluation of the employed methods. Figure 2 illustrates the flowchart of the proposed method.
Reviewer#1 1.1
2.1. Data Acquisition Cardiac MRI scans were performed on a 1.5 T scanner (Siemens Avanto, Erlangen, Germany). All clinical MR examinations were approved by our institutional review board, and informed consent was obtained from the subjects prior to the MRI scans. Cardiac perfusion data from 35 adult subjects were evaluated. Among the 35 subjects, there were 14 patients with CAD, 8 patients with hypertrophic cardiomyopathy (HCM), and 13 healthy volunteers (VOL). A conventional electrocardiogram (ECG) gated fast gradient echo sequence was used to acquire breath-hold dynamic short-axis cardiac perfusion images during the first pass of a bolus of an extracellular contrast agent. The imaging parameters were as follows: flip angle = 15°, slice thickness = 8 mm, field-of-view (FOV) = 400 × 315 mm2, image matrix = 320 × 252, image pixel spacing = 1.25 × 1.25 mm2, and number of image frames = 80. As the acquisition of a perfusion image series required more than one minute, the subject resumed breathing after a 15-20 s of breath-hold. Respiratory motion changes the location of the heart, thus affecting the location of the myocardium. Motion correction was automatically performed frame-by-frame to align the myocardium across the frames and correct the misregistration caused by respiration [22]. 2.2. Model Development from Cardiac Cine Data A U-Net architecture [23] with dropout [24] was trained on the cardiac cine data. We used the Tensorlayer library to implement the U-Net model [25, 26]. The total number of subjects used for model development was 110 and included the publicly available York [27] data (n=33)
7
Detail about data acquisition and types of subjects
and our internal data (n=77). Cardiac cine MRI acquisition parameters for the internal data were as follows: ECG gating, number of frames = 30, slice thickness = 6 mm, echo time (TE) = 1.3 ms, FOV = 350 × 300 mm2, pixel size = 1.29 × 1.29 mm2, and spacing between the slices = 10 mm. The internal data (n=77) included 15 HCM patients, 17 healthy volunteers, 16 CAD patients, 16 AS patients, 11 patients with cardiac amyloidosis, and two subjects with chest pain symptoms. The total number of cardiac short-axis images was 17,683, ranging from the apical to basal slices in both the systolic and diastolic frames. In total, 13,535 images from 88 subjects were used for training, while 4,148 images from 22 subjects were used for validation. The dimensions of the cardiac cine images, endocardial masks, and epicardial masks were set to 256 × 216, after cropping the peripheral regions of the images. The image intensity was scaled to the 0 – 255 range, and each image was saved in the png format. The dropout ratio ranged from 0.1 to 0.3, following the implementation of the concrete dropout [28]. The batch size was set to four, and the beta value was set to 0.9. The maximum number of epochs was set to 50. Figure 3 shows the learning curves of the myocardial segmentation models with different learning rates. After inspecting the training and validation learning curves, a learning rate of 0.0001 and an epoch number of 20 were selected for the segmentation model, which was used to estimate the segmentation uncertainty in the cardiac perfusion dataset. 2.3. Segmentation with Uncertainty Information Our experiments with the uncertainty estimation revealed that the early image frames, such as the no contrast enhancement (Figure 4a) and right ventricle (RV) enhancement (Figure 4b) images, produced low uncertainty estimates even though their image contrast was not suitable for myocardial segmentation. Hence, we restricted the uncertainty estimation to the frames of
8
interest by excluding both the early image frames and late image frames, which experienced a greater misregistration of the myocardium. The automatic detection of the peak RV enhancement frame
was based on [29]. Figure 5
illustrates the overall process. The minimum intensity projection (MinIP) and maximum intensity projection (MIP) maps were obtained pixel-wise over time. (
{ where (
(
)
(
)
(
)
(1)
) (
) represents the series of DCE perfusion images. In
Reviewer#1
) defined in Eq. (1), 1.2
the subcutaneous fat region appears bright, while the LV and RV regions appear dark, as shown in Figure 5a. (
)
( (
{
) )
(2)
T is a threshold used to exclude the high-intensity subcutaneous fat signals. A morphological operation on the binary mask defined in Eq. (2) was applied to sufficiently enclose the fat regions. The MIP map was multiplied by the binary mask referred to as
(
). A pixel location ( (
)
to generate a fat-free MIP map ) that maximized
(
)
(
was determined.
)
(3)
After excluding the fat region, maximal signal intensity was observed within the RV cavity, as opposed to the LV cavity. This was because in general the RV is closer to the surface coil than the LV, thus producing a higher signal intensity in the RV. Furthermore, the contrast agent entered the RV prior to the LV, which caused a smaller dispersion of the contrast agent bolus in the RV. Finally,
was obtained by finding the time index corresponding to the peak RV signal
enhancement, as shown in Figure 5c. In this study, a seeded region growing algorithm with the
9
Write down the math
seed point (
) defined in Eq. (3) was applied to obtain a region of interest (ROI) over the RV.
The mean signal value within this ROI was computed for every frame. This helped improve the signal-to-noise ratio (SNR) in the RV signal time series. The segmentation uncertainty was estimated in a frame range of [
], which represented the candidate frames exhibiting
moderate to high myocardial contrast. We used the Monte Carlo dropout [17, 20] for uncertainty estimation with the number of repetitions = 100, where the output of the U-Net model was obtained with dropout (i.e., random selections of the units in each layer) for each realization. We obtained a standard deviation (SD) map, denoted by
(x, y), from the result of the Monte Carlo dropout sampling. In this study,
the concrete dropout library was used [28]. For each time frame of interest, the pixel-wise summation of the SD map was calculated as a metric of uncertainty, denoted by SSD. ∑
∑
(
)
(4)
where N and M are the number of image rows and columns, respectively. We computed the SSD metric for each time frame within the frame range. An “optimal” image frame for endocardial (or epicardial) segmentation, which minimized the SSD values in the time frame of interest, was selected. 2.4. Evaluation For evaluation, manual segmentation was performed by manually selecting an image frame of interest that showed adequate contrast between the myocardium and blood, and drawing the endocardial and epicardial contours. Manual segmentation served as the reference. The semiautomatic active contour [5] (Active-contour), automatic U-Net segmentation (Auto-Unet), and semi-automatic U-Net (Semiauto-Unet) segmentation methods were implemented and compared. The Active-contour method required user selection of the center point of the LV. The remaining
10
Reviewer#2 2.2 More comparison experiment needed
steps were fully automatic, which involved the selection of an image frame where the LV signal first reached the maximum value [5]. The Auto-Unet method first determined the RV enhancement frame
based on the procedure shown in Figure 5. The U-Net model was
applied to the LV enhancement frame, which was chosen to be
+7, to estimate the
endocardial and epicardial contours. The Semiauto-Unet method relied on the manual selection of an appropriate image frame, which showed adequate contrast between the myocardium and blood, followed by the U-Net segmentation to estimate the endocardial and epicardial contours. The Dice similarity score was calculated to evaluate/compare the segmentation performance between each method and manual segmentation. Computation time of the entire process of the proposed method was measured on a 64-bit Microsoft Windows 10 operating system with a 16core Intel Xeon Gold 6134 CPU @ 3.20 GHz, 64 GB RAM, and NVIDIA Quadro P5000 GPU (16 GB memory size). Statistical analysis was performed using the R software (R Foundation for Statistical Computing, Vienna, Austria). The intraclass correlation coefficient (ICC) and its 95% confidence interval (CI) were computed between two segmentation methods in terms of Dice similarity coefficients. A P value < 0.05 was considered to indicate a statistically significant correlation, given the null hypothesis of no relationship between the two measurements.
3. Results Figure 4 illustrates the uncertainty estimates for four distinct contrast enhancement patterns: no enhancement, RV enhancement, LV enhancement, and myocardial enhancement. Low uncertainty estimates were observed corresponding to the no enhancement and RV enhancement frames, which were not helpful in segmenting the myocardium. Hence, these frames were not considered when selecting the appropriate frames based on the SSD metric. The LV
11
Reviewer#2 2.2 Test statistical significance of the methods
enhancement frame shown in Figure 4c shows lower SD maps for both the endocardial and epicardial segmentations, as compared to the myocardial enhancement frame shown in Figure 4d. A visual comparison between the SD maps in Figures 4c and 4d clearly indicates that the former achieves higher myocardial segmentation accuracy as compared to the latter. Table 1 shows the Dice scores in three different subject groups and comparisons between the Dice scores of four different methods. Table 2 compares the ICC of two different segmentation methods. The proposed automatic method based on uncertainty estimation was comparable to the Semiauto-Unet method in terms of the mean Dice similarity score (0.806 vs. 0.808) and showed a statistically significant correlation (ICC = 0.61, 95% CI = [0.36 – 0.78], P value < 0.001) as compared with the Semiauto-Unet method. The CAD group showed the lowest mean Dice scores in all the evaluated methods. The HCM group showed the highest mean Dice scores in the AutoUnet, Semiauto-Unet, and proposed methods. The VOL group showed the highest mean Dice score in the Active-contour method. Figures 6a-b show the plots of the SSD values, as calculated from the Monte Carlo dropout samplings of the U-Net model, with respect to the time frame. The optimal frames with minimal SSD values within the frame range of interest are indicated in red and green, respectively. Figures 6c-d show the optimal image frames and endocardial (red) and epicardial (green) contours overlaid onto the images, respectively, which qualitatively exhibit good segmentation accuracy. Figure 7 shows the SSD values for the endocardial and epicardial uncertainties and their relationships with the Dice scores. Each sample point represents a set of endocardial and epicardial SSD results and its corresponding Dice score group for a subject’s perfusion dataset. The samples of the Dice group (0.85 <= Dice score) are in < 30 in the endocardial SSD and in <
12
60 in the epicardial SSD, and they are closer to the origin of the X-Y axes than the samples of the other Dice groups. An endocardial SSD > 40 and epicardial SSD > 70 would most likely result in poor Dice scores. Comparisons between the Semiauto-Unet and proposed methods are shown in Figures 8 and 9. The blue contours indicate the manual myocardial segmentation results, which served as the reference. The red contours indicate the endocardial segmentation results, while the green contours indicate the epicardial segmentation results. Figure 8 shows three patient cases with their Dice scores > 0.85 obtained using the proposed method. Figure 9 shows three patient cases with their Dice scores < 0.85 for obtained using the proposed method. The two methods mostly produce comparable segmentation results, but the top row of Figure 9 shows a case wherein the proposed automatic method achieved a higher segmentation accuracy as compared to the Semiauto-Unet method.
4. Discussion DCE myocardial perfusion images show a variation in the image intensity across time frames, as shown in Figure 1. The contrast enhancement timing varies depending on the subject. Hence, a manual procedure involving a frame-by-frame visual inspection of the cardiac perfusion data is helpful for selecting an image frame that shows adequate image contrast. This work focused on the automatic selection of appropriate image frames for myocardial segmentation. Encoderdecoder CNN architectures (e.g., U-Net [23], SegNet [30]) have emerged as powerful methods for automatic segmentation, and the segmentation uncertainty for these networks can be estimated using Monte-Carlo dropout sampling [17]. The automatic selection of image frames in DCE perfusion relied on minimizing the uncertainty metrics. In this study, we proposed the SSD
13
as a metric. The experimental results indicated that the uncertainty information derived from Monte Carlo dropout suggested appropriate image frames for segmentation of the DCE images. The mean Dice similarity score in the perfusion dataset was approximately 0.8. Although the perfusion data evaluated in this study admittedly had lower spatial resolution and quality than the cardiac cine dataset, the mean Dice score was not very high, requiring further improvement. It should be noted that the segmentation model was trained only on the cardiac cine data. We might expect a higher myocardial segmentation performance/accuracy if the segmentation model was trained on the cardiac perfusion data, or both the cardiac cine and perfusion data. Given that the cardiac cine image data and their associated myocardial masks have mainly been used in the development of deep learning models [12-16], our study focused on investigating the feasibility of the model, which was trained on the cardiac cine, to automatically segment the myocardial perfusion without any additional labeling of the perfusion data. The successful transfer/application of the trained model to the DCE perfusion data can be explained by the following: (1) Both the cardiac cine and cardiac perfusion datasets were acquired in the shortaxis imaging planes, and the slice anatomy of the cardiac cine images appeared similar to that of the cardiac perfusion images. (2) Certain image frames of the dynamic cardiac perfusion dataset exhibited a myocardium and blood contrast that was similar to the cardiac cine images. When fitting the optimal image frames for the epicardial and endocardial segmentations, we relied on the SD map derived from the Monte Carlo dropout results. A close observation of the SD maps suggested regional differences in the SD. For example, in a certain frame, the SD of the Reviewer#2 2.3
myocardial septal region was lower than that of the lateral region, as the contrast between the myocardium and RV was higher than the contrast between the myocardium and lung. A regional assessment of the uncertainty might help identify more suitable image frames for myocardial
14
Detailed explanation of successful transfer
segmentation, although this would require a post processing technique of automatic localization of myocardial regions based on its geometry (e.g., localization of the RV insertion points). In addition, the investigation between the SSD values and Dice coefficient (Figure 7) suggests that the set of endocardial and epicardial SSD values might help predict the segmentation performance. The selected frame range affected the segmentation results in a few cases. For example, a few HCM patients showed slow RV enhancement curves, implying that a longer delay from the peak RV enhancement frame is necessary. The healthy volunteers showed relatively fast RV enhancement curves, indicating that a shorter delay from the peak RV enhancement frame would be desirable. The adaptive adjustment of the frame range based on the enhancement time-
Reviewer#1 1.3
intensity curve was not investigated in this study, and this remains as future studies. Need comments on small approximately 86 s, which was significantly longer than the time required by the semi-automated test dataset and generalizabi U-Net method. The increased time can be attributed to the Monte Carlo simulations involving lity of the method.
The computation time for myocardial segmentation using the proposed method was
100 trials per image frame and 20 repetitions corresponding to the number of frames (20 frames in this study). The computation time can be reduced by reducing the number of trials or image frames. Alternatively, parallel processing, not used in this study, can help increase the computational efficiency. In conclusion, we have demonstrated the feasibility of automatic frame selection for endocardial (or epicardial) segmentation in dynamic cardiac perfusion MRI, based on the uncertainty information from the Monte Carlo dropout in the U-Net segmentation model. The proposed method was fully automatic and resulted in a mean Dice similarity score that was comparable to that of the semi-automatic U-Net method. The SSD metric adopted in this study
15
can be used for screening purposes, so that the cases with high endocardial and epicardial SSD values can be referred for further evaluation and correction by human experts. This was a single center and single vendor study, and a small set of patients were evaluated. Evaluating the multicenter and multi-vendor data would strengthen the robustness of the proposed method, which will be investigated in future studies. Acknowledgments The manuscript has not been published elsewhere and is not currently being considered for publication in another journal. The authors have been actively involved in the study and are responsible for the content of this manuscript. This work was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT, and Future Planning (Grant numbers: NRF-2015 R1C1A1A02036340, NRF-2018 R1D1A1B07042692). We are grateful for the comments from the reviewers. We would like to thank Editage for English language editing. Competing Interests The authors declare no competing interests.
16
TABLES Table 1. Mean and standard deviation (SD) values of the Dice similarity scores among the four evaluated methods. Manual segmentation served as the reference.
Method
Dice score (mean ± SD) Reviewer#2
CAD (n = 14)
HCM (n = 8)
VOL (n = 13)
Active-contour
0.654 ± 0.156
0.665 ± 0.092
0.728 ± 0.076
Total (n = 35) 2.2 0.684 ± 0.119
Auto-Unet
0.674 ± 0.116
0.775 ± 0.193
0.760 ± 0.137
0.729 ± 0.147 Auto-Unet
Semiauto-Unet
0.785 ± 0.109
0.835 ± 0.093
0.817 ± 0.029
0.808 ± 0.084
Proposed
0.778 ± 0.118
0.838 ± 0.108
0.815 ± 0.051
0.806 ± 0.096
Addition of
Abbreviations: SD - standard deviation; CAD - coronary artery disease; HCM - hypertrophic cardiomyopathy; VOL - healthy volunteer
17
for more comparison
Table 2. Intraclass correlation coefficient (ICC) and statistical significance between the Dice scores of the two myocardial segmentation methods applied to the test subjects (n = 35).
Dice score
ICC
95% CI
P value
Proposed vs. Semiauto-Unet
0.61
0.36 – 0.78
< 0.001
Proposed vs. Auto-Unet
0.46
0.16 – 0.69
< 0.001
Proposed vs. Active-contour
-0.06
-0.38 – 0.28
0.629
Semiauto-Unet vs. Auto-Unet
0.33
0.01 – 0.60
0.023
Semiauto-Unet vs. Active-contour
-0.11
-0.42 – 0.23
0.730
Auto-Unet vs. Active-contour
0.27
-0.06 – 0.55
0.063
Abbreviations: ICC - intraclass correlation coefficient; CI - confidence interval
18
Reviewer#2 2.2 Test statistical significance of the methods
FIGURES
Figure 1. An example of a cardiac dynamic contrast enhancement (DCE) MRI series. The goal of this study was to automatically and accurately segment the myocardium. Note that the image contrast varies over the time frames owing to contrast enhancement. The red arrow in the t = 33 frame indicates a perfusion defect, which appears as a round dark band surrounding the left ventricular blood pool.
19
Figure 2. Flowchart of the proposed method.
20
Figure 3. Learning curves for the U-Net dropout model, which was trained and validated on the cardiac cine MRI dataset. Training (dashed) and validation (solid) learning curves are shown for the learning rates of 0.0001, 0.0002, and 0.00005. After visual inspection of the learning curves, we selected the learning rate = 0.0001 and epoch number = 20 for our segmentation model, which was used for the uncertainty estimation and segmentation of the cardiac perfusion data.
21
Figure 4. Monte-Carlo dropout results of the U-Net model in four representative frames. (a) No enhancement. (b) RV enhancement. (c) LV enhancement. (d) Myocardial enhancement. (a) and (b) show no segmentation along the endocardial and epicardial borders. (c) shows good endocardial and epicardial segmentation masks with relatively low segmentation uncertainty. The non-zero values in the standard deviation (SD) maps shown in (c) are distributed near the endocardial and epicardial borders, whereas the non-zero values in the SD maps shown in (d) are more widely distributed near the endocardial and epicardial borders with higher intensity, as compared to (c).
22
Reviewer#1 1.2
Figure 5. A schematic showing the automatic detection of the peak RV enhancement frame. (a) The minimum intensity projection (MinIP) map highlights the bright fat signal. (b) The fat region is masked out and the maximum signal intensity can be observed within the RV blood pool (red cross). (c) The peak RV enhancement frame can be identified from the RV signal time series (red dot).
23
Visual explanation of peak RV enhanceme nt frame detection
Figure 6. Automatic segmentation based on the uncertainty estimates. (a-b) Sum of standard deviation (SSD) following Monte Carlo dropout is plotted with respect to the frame index. The plot shows the temporal region of interest (ROI), which ranges from 21 to 40. Frame 24 has the minimum SSD value for the endocardial case, while frame 27 has the minimum SSD value for the epicardial case. (c-d) Deep learning segmentation results corresponding to (c) the endocardial case and (d) the epicardial case. 24
Figure 7. The relationship of the endocardial sum of standard deviation (SSD) and epicardial SSD with the Dice similarity score groups.
25
Figure 8. Comparison between the semi-automatic U-Net (Semiauto-Unet) and proposed methods for the case where the Dice score of the proposed method is > 0.85. (Top) CAD patient, (endocardial SSD, epicardial SSD, Dice score) = (22.6, 37.4, 0.882). (Middle) HCM patient, (endocardial SSD, epicardial SSD, Dice score) = (21.7, 58.6, 0.886). (Bottom) a healthy volunteer, (endocardial SSD, epicardial SSD, Dice score) = (21.6, 33.9, 0.862).
26
Figure 9. Comparison between the semi-automatic U-Net (Semiauto-Unet) and proposed methods for the case where the Dice score of the proposed method is < 0.85. (Top) CAD patient, (endocardial SSD, epicardial SSD, Dice score) = (38.5, 53.9, 0.803). (Middle) HCM patient, (endocardial SSD, epicardial SSD, Dice score) = (102.9, 136.6, 0.587). (Bottom) healthy volunteer, (endocardial SSD, epicardial SSD, Dice score) = (29.7, 102.0, 0.803).
27
No conflict of interest.
References [1] O. R. Coelho-Filho, C. Rickers, R. Y. Kwong, and M. Jerosch-Herold, "MR myocardial perfusion imaging," Radiology, vol. 266, no. 3, pp. 701-15, Mar 2013, doi: 10.1148/radiol.12110918. [2] J. Schwitter et al., "MR-IMPACT II: Magnetic Resonance Imaging for Myocardial Perfusion Assessment in Coronary artery disease Trial: perfusion-cardiac magnetic resonance vs. single-photon emission computed tomography for the detection of coronary artery disease: a comparative multicentre, multivendor trial," European heart journal, vol. 34, no. 10, pp. 775-81, Mar 2013, doi: 10.1093/eurheartj/ehs022. [3] M. R. Patel et al., "Low diagnostic yield of elective coronary angiography," N Engl J Med, vol. 362, no. 10, pp. 886-95, Mar 11 2010, doi: 10.1056/NEJMoa0907272. [4] P. S. Douglas et al., "Outcomes of anatomical versus functional testing for coronary artery disease," N Engl J Med, vol. 372, no. 14, pp. 1291-300, Apr 2 2015, doi: 10.1056/NEJMoa1415516. [5] G. Tarroni et al., "Myocardial perfusion: near-automated evaluation from contrast-enhanced MR images obtained at rest and during vasodilator stress," Radiology, vol. 265, no. 2, pp. 576-83, Nov 2012, doi: 10.1148/radiol.12112475. [6] A. M. Weng, C. O. Ritter, J. Lotz, M. J. Beer, D. Hahn, and H. Kostler, "Automatic postprocessing for the assessment of quantitative human myocardial perfusion using MRI," European radiology, vol. 20, no. 6, pp. 1356-65, Jun 2010, doi: 10.1007/s00330-009-1684-z. [7] G. M. Beache, F. Khalifa, A. El-Baz, and G. Gimel'farb, "Fully automated framework for the analysis of myocardial first-pass perfusion MR images," Medical physics, vol. 41, no. 10, p. 102305, Oct 2014, doi: 10.1118/1.4893531. 28
[8] P. G. Camici and F. Crea, "Coronary microvascular dysfunction," N Engl J Med, vol. 356, no. 8, pp. 830-40, Feb 22 2007, doi: 10.1056/NEJMra061889. [9] V. Gupta et al., "Cardiac MR perfusion image processing techniques: a survey," Medical image analysis, vol. 16, no. 4, pp. 767-85, May 2012, doi: 10.1016/j.media.2011.12.005. [10] M. Jacobs, M. Benovoy, L. C. Chang, A. E. Arai, and L. Y. Hsu, "Evaluation of an automated method for arterial input function detection for first-pass myocardial perfusion cardiovascular magnetic resonance," Journal of cardiovascular magnetic resonance : official journal of the Society for Cardiovascular Magnetic Resonance, vol. 18, p. 17, Apr 8 2016, doi: 10.1186/s12968-016-0239-0. [11] P. Peng, K. Lekadir, A. Gooya, L. Shao, S. E. Petersen, and A. F. Frangi, "A review of heart chamber segmentation for structural and functional analysis using cardiac magnetic resonance imaging," Magma, vol. 29, no. 2, pp. 155-95, Apr 2016, doi: 10.1007/s10334015-0521-4. [12] L. K. Tan, Y. M. Liew, E. Lim, and R. A. McLaughlin, "Convolutional neural network regression for short-axis left ventricle segmentation in cardiac cine MR sequences," Medical image analysis, vol. 39, pp. 78-86, Jul 2017, doi: 10.1016/j.media.2017.04.002. [13] M. R. Avendi, A. Kheradvar, and H. Jafarkhani, "A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI," Medical image analysis, vol. 30, pp. 108-119, May 2016, doi: 10.1016/j.media.2016.01.005. [14] W. Xue, G. Brahm, S. Pandey, S. Leung, and S. Li, "Full left ventricle quantification via deep multitask relationships learning," Medical image analysis, vol. 43, pp. 54-65, Jan 2018, doi: 10.1016/j.media.2017.09.005.
29
[15] A. H. Curiale, F. D. Colavecchia, and G. Mato, "Automatic quantification of the LV function and mass: A deep learning approach for cardiovascular MRI," Comput Methods Programs Biomed, vol. 169, pp. 37-50, Feb 2019, doi: 10.1016/j.cmpb.2018.12.002. [16] T. A. Ngo, Z. Lu, and G. Carneiro, "Combining deep learning and level set for the automated segmentation of the left ventricle of the heart from cardiac cine magnetic resonance," Medical image analysis, vol. 35, pp. 159-171, Jan 2017, doi: 10.1016/j.media.2016.05.009. [17] Y. Gal and Z. Ghahramani, "Dropout as a bayesian approximation: Representing model uncertainty in deep learning," in international conference on machine learning, 2016, pp. 1050-1059. [18] A. Kendall, V. Badrinarayanan, and R. Cipolla, "Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding," arXiv preprint arXiv:1511.02680, 2015. [19] A. Kendall and Y. Gal, "What uncertainties do we need in bayesian deep learning for computer vision?," in Advances in Neural Information Processing Systems, 2017, pp. 55745584. [20] C. Leibig, V. Allken, M. S. Ayhan, P. Berens, and S. Wahl, "Leveraging uncertainty information from deep neural networks for disease detection," Sci Rep, vol. 7, no. 1, p. 17816, Dec 19 2017, doi: 10.1038/s41598-017-17876-z. [21] T. DeVries and G. W. Taylor, "Leveraging Uncertainty Estimates for Predicting Segmentation Quality," arXiv preprint arXiv:1807.00502, 2018.
30
[22] H. Xue et al., "Unsupervised inline analysis of cardiac perfusion MRI," Med Image Comput Comput Assist Interv, vol. 12, no. Pt 2, pp. 741-9, 2009. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/20426178. [23] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015: Springer, 2015, pp. 234-241. [24] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929-1958, 2014. [25] H. Dong et al., "Tensorlayer: A versatile library for efficient deep learning development," in Proceedings of the 25th ACM International Conference on Multimedia, 2017: ACM, pp. 1201-1204. [26] Y.-C. Kim, K. R. Kim, K. Choi, M. Kim, Y. Chung, and Y. H. Choe, "EVCMR: A tool for the quantitative evaluation and visualization of cardiac MRI data," Computers in biology and medicine, p. 103334, 2019. [27] A. Andreopoulos and J. K. Tsotsos, "Efficient and generalizable statistical models of shape and appearance for analysis of cardiac MRI," Medical image analysis, vol. 12, no. 3, pp. 335-57, Jun 2008, doi: 10.1016/j.media.2007.12.003. [28] Y. Gal, J. Hron, and A. Kendall, "Concrete dropout," in Advances in Neural Information Processing Systems, 2017, pp. 3581-3590. [29] Y. C. Kim, S. M. Kim, and Y. H. Choe, "Robust semi-automated quantification of cardiac MR perfusion using level set: Application to hypertrophic cardiomyopathy patient data,"
31
Computers in biology and medicine, vol. 71, pp. 162-73, Apr 1 2016, doi: 10.1016/j.compbiomed.2016.02.014. [30] V. Badrinarayanan, A. Kendall, and R. Cipolla, "SegNet: A Deep Convolutional EncoderDecoder Architecture for Image Segmentation," IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 12, pp. 2481-2495, Dec 2017, doi: 10.1109/TPAMI.2016.2644615.
32