Hybrid method for automatic construction of 3D-ASM image intensity models for left ventricle

Journal Pre-proof

Hybrid Method for Automatic Construction of 3D-ASM Image Intensity Models for Left Ventricle Huaifei Hu , Ning Pan , Tailang Yin , Haihua Liu , Bo Du PII: DOI: Reference:

S0925-2312(19)31545-0 https://doi.org/10.1016/j.neucom.2019.10.102 NEUCOM 21497

To appear in:

Neurocomputing

Received date: Revised date: Accepted date:

26 May 2019 17 September 2019 27 October 2019

Please cite this article as: Huaifei Hu , Ning Pan , Tailang Yin , Haihua Liu , Bo Du , Hybrid Method for Automatic Construction of 3D-ASM Image Intensity Models for Left Ventricle, Neurocomputing (2019), doi: https://doi.org/10.1016/j.neucom.2019.10.102

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 Published by Elsevier B.V.

Hybrid Method for Automatic Construction of 3D-ASM Image Intensity Models for Left Ventricle☆ Huaifei Hu a, b, c, 1, Ning Pan a, b, c, 1, Tailang Yin d, *, Haihua Liu a, b, c, Bo Du e, * a. College of Biomedical Engineering, South Central University for Nationalities, Wuhan 430074, China; b. Hubei Key Laboratory of Medical Information Analysis and Tumor Diagnosis & Treatment, Wuhan 430074, China; c. Key Laboratory of Congnitive Science, State Ethnic Affairs Commission, Wuhan 430074, China; d. Reproductive Medicine Center, Renmin Hospital of Wuhan University, Wuhan, Hubei 430060, China; e. School of Computer Science, Wuhan University, Wuhan, Hubei 430072, China. ☆

This study was supported by China Scholarship Council (No. 201508420005), National Natural Science Foundation

of China (No. 61773409 and 61976227), Natural Science Foundation of Hubei Province of China (No. 2017CFB552 and 2018CFB755), Fundamental Research Funds for the Central Universities (No. CZQ16012, CZY18026, and CZY19040), and (The Applied Basic Research Programs of Wuhan (China) under Grant No.2017060201010160).

* Corresponding author. E-mail addresses: [email protected] (Bo Du), [email protected] (Tailang Yin), 1 These authors contributed equally to this work.

ABSTRACT Active Shape Models (ASMS) play an important role in model based medical image analysis. They utilize point distribution models (PDMs) in which a priori information is encoded into a template so that the objects to be detected can be represented with a fixed topology. One key element in 3D-ASM is the image intensity model (IIM), which is investigated in this work. We propose a hybrid approach to automatically construct 3D-ASM Intensity Models for the left ventricle. To train the IIM, CNN is adopted to obtain the initial shape for 3D-ASM, distance maps for endo and epicardial contours from ground truth are derived, and using PDM, training shapes can be obtained. The training shapes and cardiac images are then used to train an IIM. 1200 cardiac MRI cases from Hubei Cancer Hospital were used in this study. By comparing point-to-surface errors against a proper gold standard, it demonstrates that large-scale cardiac MRIs can be 1/18

segmented by 3D models trained under this scheme with fair accuracy. Clinical parameters are calculated using the Bland-Altman analysis, and thus we yield biases of 4.8 ml, 2.19 ml, 2.59 ml, 0.96%, 0.69 g and -2.67 g for LVEDV (LV End-diastolic Volume), LVESV (LV End-systolic Volume), LVSV (LV Stroke volume), LVEF (LV Ejection Fraction), LVM-DP (LV mass in diastolic phase) and LVM-SP (LV mass in systolic phase), respectively. Keywords: Statistic shape modelling, Cardiac MRI, image intensity modeling, CNN.

2/18

INTRODUCTION Cardiac diseases are the leading cause of global mortality, increasing concerns effective diagnosis and treatment [1]. Despite the fact that many advanced techniques have been developed for cardiac analysis, manual valuation and delineation is still time consuming clinical work for cardiologists and radiologists. However, current manual or semi-automatic analytical methods generate unmanageable workloads for clinicians due to increases in large-scale data. Therefore, algorithms with minimal manual intervention or fully automatic methods are desperately needed for cardiac data analysis. Statistic shape modelling is a powerful methodology that can been used for visualizing and quantifying geometric and functional patterns of the heart. It takes full advantage of strong prior knowledge by encoding the specific shape and appearance variability of the images. Using a priori knowledge is essential to increase the robustness and accuracy and to ensure correct topology in medical image segmentation. It is described in 3D spherical harmonics [2], 3D medial representations (m-reps) [3], mixtures of probabilistic PCA [4], and 3D statistical shape models (SSMs) [5]. SSMs have been used extensively to guide cardiac image segmentation [6, 7]. In the cardiac field, SSMs have been widely used for 3D (or 3D+t) image analysis [8]. ASMs use a shape template representing the target organ, which is typically derived from several training data sets. The shape template is constructed by distributing landmarks on the boundary, which is called a Point Distribution Model (PDM). The appearance of the image is needed when fitting a PDM to analyze novel images. Therefore, a gray-level (or image intensity) model is created as a second component of ASMs. Simulated search [9] is used by some investigators for local boundary detection. Other investigators use image intensity models based on fuzzy c-means inference [10]. Finally, other researchers create image intensity models by simulating the physics of image formation on anatomical models created from 3D PDMs derived from a different image modality. This approach has been used to automatically create image 3/18

intensity models for gated single photon emission computed tomography (gSPECT) [11], 3D US [12], and CMR [13, 14] , respectively. However, the practical application of 3D-ASM is undermined by the need to train such models on very large imaging studies. The training process is impracticable for these models using manual techniques on large-scale target image databases. To complete the training, manual delineation is needed for the target boundaries, landmarks are distributed across sample shapes, and a point distribution model (PDM) is produced by statistical shape decomposition [15] with a derived statistical intensity model. By auto-landmarking surface [16, 17] or volumetric measures [18, 19] for the target organ, PDMs can be built automatically [20]. Catalina et al. reused a prebuilt 3D PDM and successfully created intensity models by simulating image generation. Their results demonstrate that models trained under this scheme can produce acceptable results with subvoxel accuracy [21]. Consider the advantages of a model-based algorithm for large scale images, we employ preexistent 3D PDM constructed from magnetic resonance images (MRI) and develop a new scheme to build image intensity models (IIM). This paper is organized as follows: In the next section, we explain the theoretical background of ASMs, and describe the proposed procedure. Then the results and validation section is detailed, which is composed of explicit comparisons which describe the features of our method. Finally, we discuss the potential of our work in clinical application. BACKGROUND ASM comprises two elements: a shape template called PDM (point distribution model) and intensity representation called image intensity model (IIM), and the adaptation of the model to the target image in a way that shape and intensity constraints are respected [15, 22]. The key ideas are summarized below. 4/18

We assume a training set with M shapes each described by N three-dimensional points

xij  ( xij , y ij , z ij ) with i  1M and j  1 N . Let si  ( x1i , y1i , z1i ,..., xNi , yNi , zNi )T be the i th vector representing the shape of the i th LV myocardial surface (both endo- and epicardium). Finally, let  s1 ,, s M  be the set of all training shapes in matrix form. All nuisance pose parameters (e.g.

translation, rotation and scaling) have been removed from Hence, the shape class mean of

using Generalized Procrustes Analysis [15].

, s is

s

1 M

M

 s (1) i 1

i

and the shape class covariance as

C

1 M (si  s )(si  s )T (2)  M  1 i 1

The shape class covariance is represented in a low-dimensional space or Principal Component Analysis (PCA) of the shape class. This produces l eigenvectors Φl  [φ1φ2 ...φl ] , and corresponding eigenvalues Λ  diag(1 , 2 ,, l ) of the covariance matrix computed via Singular Value Decomposition (SVD). Hence, assuming the shape class follows a multi-dimensional Gaussian probability distribution, any shape in the shape class can be approximated from the following linear generative model s  s  Φl b (3)

where b are PDM parameters and restricted to bi   i fall within 99% of the shape class distribution if   3 . The parameters that reconstruct a shape s are estimated from

b  ΦTl (s  s )

(4)

The components of b are the projection coefficients of mean-centered shapes (s  s ) along the columns of Φl . A comprehensive overview of statistical shape models is beyond the scope of this paper and can be found in [15, 23, 24]. The assumption in Eq. (1) and (2) is that all shapes in the training set are described by the same set of 5/18

corresponding landmarks. Establishing 3D correspondences across a training set is, in general, non-trivial. Frangi et al. [18], proposed a method to build PDMs based on point correspondences in the heart, and Hoogendoorn et al. [25] refined the technique and constructed the cardiac PDMs we use in this paper. The key idea behind these methods is to construct a mean atlas, automatically extract landmarks from this atlas, and subsequently propagate those landmarks to each and every sample shape using volumetric non-rigid registration. For each landmark in s , we build an IIM based on the intensity information across all corresponding landmarks in all training shapes s i . The IIMs capture local intensity distribution along cardiac boundaries. We proceed by sampling 1D intensity profiles normal to the myocardial boundaries. Each profile has a length size m  15 pixels. For the i th landmark, the mean intensity profile gi , and the corresponding image intensity covariance S g i are estimated. During image segmentation, the intersections of the 3D PDM with all imaging planes defines a stack of 2D contours oriented in 3D space. The ASM algorithm proceeds by searching for the best-matching intensity profile location along the normal to the contours and over the imaging planes for each landmark. To derive the best-matching position, or candidate point, y i for each landmark, we minimize the Mahalanobis distance between profile sampled at the candidate position, viz. gi (y i ) , and the mean profile, gi , according to: y io  arg min  (g(y i )  gi )T Sgi1 (g(y i )  gi ) 

(5)

yi

The search for candidate positions occurs normally to the myocardial contours and over planes corresponding to cardiac imaging planes. However, the candidate positions themselves lay on a 3D space and the vector between the current and candidate position of each landmark can be interpreted as landmark displacement forces. These forces are distributed over neighboring nodes weighted by the geodesic distance between the search position and neighbor node as in Eq. (6) [6],

6/18

 pq w(p, q)  exp    2 2  where p  q

2

2

   

(6)

is the geodesic distance between points p and q , and  is the width of the Gaussian

kernel that reflects the degree of sparsity of the data. MATERIALS AND METHODS Data set In this study, 1200 cardiac MRI cases from Hubei Cancer Hospital were assessed. Ground truth for endo/epi cardiac boundaries drawn by experts are available for building image intensity models and comparison between automatic and manual analyses. Using retrospectively gated SSFP (Steady State Free Precession) cine imaging, the MRI datasets are derived during a breath-hold of 8 to 15 seconds duration. Overview of the proposed algorithm Our work-flow involves an automatic IIM building procedure for 3D-ASM. The entire process for the proposed approach comprises several steps, as shown in Fig. 1. The mean Point Distribution Model (PDM) is fit to the ground truth using point sets registration [26]. Secondly, distance maps are computed from the ground-truth contours of the endo and epicardial walls, which are subsequently used to drive the 3D-ASM model towards image boundaries. Thirdly, 3D-ASM is applied to refine the fit of the static shape model to the image data, while penalizing large deviations from the ground truth, and the obtained results are regarded as 3D ground truth for cardiac images. Finally, the derived 3D ground truth, combined with the image data, are adapted for automatic image intensity model construction. Step I: Initialization of 3D-ASM When the shape models are used for operation, an initial guess of the cardiac position is required. PDM performs statistical shape constraining, usually in a multi-resolution fashion in order to extend the capture 7/18

range of the algorithm. The 3D-ASM surface needs to be initialized within the capture range of the intended boundaries for a robust and accurate fit. To get the initial shape for a case, Catalina et al. adopted a traditional method using three manual selected points at the basal and apex levels [11]. However, manual initialization becomes infeasible when dealing with thousands of CMR volumes. Recently, deep learning techniques have been widely used by many researchers due to promising results [27-31]. Inspired by their success, here we employ a convolution neural networks named SegNet [32] to overcome the challenge of obtaining an initial shape for 3D-ASM. The LV are segmented by SegNet parameters which is trained using 400 randomly selected cases (See Fig. 2). We then match the 2D initial boundaries of myocardial with the myocardial contour surface of the mean PDM using non-correspondence point-set registration [26] to align the ground truth contours to the initial surface model of 3D-ASM (See Fig. 3). Step II: Distance-constrained LV refinement for 3D-ASM. Once the initial shape is ready for 3D-ASM, the 2D ground truth contours are used to build two kinds of distance maps for endo and epicardial contours. Fig. 4 shows two distance maps constructed from the initial endo and epicardial contours, respectively. For one image slice, we compute an Euclidean distance map to the nearest pixel from the endocardial contour. Pixels in the endocardial boundary are zero (Fig. 4 (c,d)). We then conduct the same computation for the epicardial contour and obtain the Euclidean distance map (see Fig. 4 (g,h)). During segmentation, the above steps are completed for all short axis cardiac MR. In the distance maps, the smaller the value of a pixel, the more likely the pixel belongs to the endo or epicardial contours. The distance maps are helpful to drive the trained active shape model to the target LV. Step III: Three-dimensional image-driven adaptation of ASM. The original 3D-ASM [22] formulation uses Eq. (5) to search for candidate points. In the current 8/18

investigation, we improve the image search step using the generated distance maps. Each candidate point is derived by reducing the value of the distance map according to Eq. (7):

y io  arg min  yi

where

(y i ) 

(7)

(y i ) denotes the value of the distance map for the candidate point y i . The complete matching

algorithm is illustrated in Algorithm 1. Algorithm 1 Matching Algorithm: 3D-ASM Require InitialShape Require PDM Require ImageStack:= Short axis images Require DmapEndoStack:= Distance map for endocardial contours Require DmapEpiStack:= Distance map for epicardial contours Ensure BestFit 1: for Iteration < MaxIteration do 2: if iteration:= 1 3: CurrentShape ← InitialShape 4: Else 5: CurrentShape ← BestShape 6: end if 7: function InterSect (ImageStack, CurrentShape) 8: for All image planes do 9: 2Dcontour ← intersection with CurrentShape 10: end for 11: CountourStack ← 2Dcontours 12: for all points in ContourStack do 13: InterPoints ← find closest landmarks in CurrentShape 14: end for 15: end function 16: function FindCandidates (InterPoints, MeanProfiles, DmapEndoStack, DmapEpiStack) 17: for All InterPoints do 18: SampledProfiles ← sample perpendicular profiles 19: if InterPoints is in Endocardial contour then 20: Sampleddis ← sample distances from DmapEndoStack 21: Else 22: Sampleddis ← sample distances from DmapEpiStack 23: end if 24: for Possible profile positions in search range do 25: Mindis ← minimal Sampleddis 26: end for 9/18

27: CanditatePoints ← store Mindis candidate positions 28: end for 29: end function 30: function Propagate (canditatePoints) 31: for all CandidatePoints do 32: Forces ← calculate propagation to neighboring nodes 33: end for 34: DefShape ← apply forces to CurrentShape 35: BestFit ← best parameters from Eq. (3) - (4) to fit DefShape 36: end function 37: end for Step IV: Image Intensity Model Training The shape model representing the left ventricles is a surface mesh comprising LV endo- and epi-cardium. To train the image intensity model, all meshes to be trained are required to have same topology. The 3D-ASM segment result from Step III is regarded as training meshes representing 3D ground truth for cardiac images. Finally, the derived training meshes combined with the image data are applied for automatic image intensity model construction. The complete training process is described in Algorithm 2. Algorithm 2 Training Algorithm: image intensity model Require Training meshes and their cardiac image cases Ensure IIM 1: Unfinished ← True 2: while Unfinished do 3: for All cases do 4: ImageStack ← Short axis images from current case 5: CurrentShape ←Current training mesh 6: function InterSect (ImageStack, CurrentShape) 7: for All image planes do 8: 2Dcontour ← intersection with CurrentShape 9: end for 10: CountourStack ← 2Dcontours 11: for all points in ContourStack do 12: InterPoints ← find closest landmarks in CurrentShape 13: end for 14: end function 15: function GetSampledProfilesStack (InterPoints, ImageStack) 16: for All InterPoints do 10/18

17: 18: 19: 20: 21: 22: 23: 24: 25:

SampledProfiles ← sample perpendicular profiles from corresponding image plane end for SampledProfilesStack ← SampledProfiles end function end for function CaluateIIM (SampledProfilesStack) for all SampledProfiles i in SampledProfilesStack do gi ← the mean intensity profile

26: 27: 28:

end for end function Unfinished ← False

S g i ← the corresponding image intensity covariance

29: end while Step V: 3D-ASM Segmentation After the image intensity model is obtained, the LV cavity can be automatically segmented by 3D-ASM. To obtain the initial shape for a new case, we adopt SegNet to obtain the coarse endo and epicardial contours. Using point clouds registration, the initial shape for 3D-ASM can be derived. VALIDATION To analyse the accuracy of 3D-ASM, we compare point-to-surface (P2S) distances, Jaccard similarity index, and cardiac functional parameters such as endocardial volume (EDV), myocardial mass (LVM), ejection fraction (EF), and stroke volume (SV).We compare P2S errors between the gold-standard and automatic contours. Results are displayed in Fig. 5 and Fig. 6. For point-to-surface (P2S) distances in Fig 5, the points are from 3D ground truth meshes obtained from 2D manual contours drawn by experts, and surfaces are from test results using 3D-ASM. As can be observed in Fig 5, the larger P2S errors seen at the basal level are acceptable since a right illustration for LV basal plane is a well-known feature in the post processing step of cardiac images with most modalities [11, 33]. Due to lower resolution and partial volume effect, P2S errors for the epicardial are larger at the ES phase when conducting the cardiac phase analysis. 11/18

For point-to-surface (P2S) distances in Fig 5 and Table 1, the points are 2D manual contours drawn by experts, and surfaces are from 3D ground truth meshes, 3D-ASM segment results, respectively. Segmentation Accuracy: Cardiac Function Indexes In table 2, the overlap and Jaccard similarity coefficient [34] are displayed to evaluate the overlap between the ground-truth and automatic results. For endocardial contours, average overlap and Jaccard indexes are 0.89  0.04 and 0.81  0.06, respectively. They are 0.88  0.05 and 0.81  0.07 for epicardial ones, respectively. Critical parameters have been employed in cardiac diagnosis, they are LVM-DP (LV mass in ED phase) and LVM-SP (LV mass in ES phase), LVEF (LV ejection fraction), LVEDV (LV end-diastolic volume in ED phase), LVESV (LV end-systolic volume in ES phase), LVSV (LV stroke volume). ED LVEDV  Vendo

(8)

ES LVESV  Vend o

(9)

ED ES LVSV  Vendo  Vend o

(10)

LVEF 

ED ES (Vend o  Vendo ) *100% ED Vendo

(11)

ED ED LVM -DP  (Vepi  Vend o )*1.05

(12)

ES ES LVM -SP  (Vepi  Vendo )*1.05

(13)

ED ED where Vend and Vepi are endo- and epi-cardial volumes in the end-diastole (ED) phase respectively; ES ES Vend and Vepi are the corresponding volumes in the end-systole (ES) phase.

In our previous work [30], the volumes for LV are calculated by the evaluation code in Radau et al. [35]. n

V  gap *  areai

(14)

i 1

12/18

where gap is the distance between two neighbor slices, and areai represents the area for the ith slice. Consider that the heart is a 3D organ, here we use a different method for volume calculation. The 3D shape obtained by 3D-ASM is clipped using the top-most slice. Then the clipped remnant is employed to obtain the volume (See Fig. 7).

For the 1200 cases, the LVEDV, LVESV, LVSV, LVEF, LVM-DP and LVM-SP are displayed in Table 3. It can been seen that the cardiac functional indexes from 3D-ASM are close to those of experts. Fig. 8 and Fig. 9 show Bland-Altman [36] plots and regression curves, respectively, for various cardiac functional indexes over the 1200 cardiac MRI cases computed both manually and automatically (proposed). For LVEDV, LVESV, LVSV, LVEF, LVM-DP and LVM-SP, the biases on the Bland-Altman plot are 4.8 ml, 2.19 ml, 2.59 ml, 0.96%, 0.69 g and -2.67 g, respectively, while the limits of agreement are (-13.57ml, 23.17ml), (-17.25 ml, 21.63 ml), (-18.86 ml, 24.04 ml), (-7.76%, 9.67%), (-12.98 g, 14.35 g) and (-29.01g, 23.67g), respectively. The correlation coefficients are 0.9689, 0.9271, 0.8491, 0.8777, 0.9585 and 0.8797 for these clinical parameters. This demonstrates that our analysis schema is in good consistent with manual analysis. DISCUSSION In this work, we proposed a hybrid method to automatically construct 3D-ASM intensity models for left ventricle segmentation of cardiac MRI. The main contributions of the paper are: 

Deep learning neural networks are adopted to obtain the initial shape for 3D-ASM;



Distance maps techniques are invented to drive the 3D-ASM model towards image boundaries;



The 3D-ASM fitting results are regarded as 3D ground truth.

Cardiac functional analysis is often performed with Cardiac MRI and CT, which yield many dynamic 3D image data sets. Due to the increasing amounts data, manual analysis of cardiac images is subjective 13/18

and time-consuming. To overcome the above difficulties, 3D-ASM is adopted. In this work, deep learning neural networks are used to obtain the coarse boundaries for endo- and epi-cardiums so as to derive the initial shape for 3D-ASM; then, using the 2D ground truth of endo and epicardial contours, two kinds of distance maps are constructed to drive the 3D-ASM model towards image boundaries. 3D-ASM, the distance maps and the images are then used to create the 3D ground truth for cardiac images. The derived 3D ground truth, combined with the image data, are subsequently adapted for automatic image intensity model construction. Finally, the accuracy of 3D-ASM is analyzed by compare point-to-surface (P2S) distances, Jaccard similarity index, and cardiac functional parameters such as endocardial volume (EDV), myocardial mass (LVM), ejection fraction (EF), and stroke volume (SV). CONCLUSIONS In the current study, a hybrid method is proposed for the automatic construction of 3D-ASM image intensity models for the left ventricle. Results indicate that large samples of cardiac MRI cases can be robustly and accurately processed by trained models using our hybrid scheme. Experimental results in LV clinical parameters are evaluated using Bland-Altman and regression analyses. For the clinical population, the current results indicate good performance with LV function parameters. Our hybrid scheme combines deep learning and statistical shape modeling is effective and robust and can compute LV functional indexes from population imaging. There are still areas for potential improvement of our method. For example, factors such as smoking, drinking, age, weight, gender, and other variables could be incorporated to enhance 3D-ASM accuracy. Conflict of interest The authors have declared that no conflict of interest exists.

14/18

15/18

REFERENCE [1] G.A. Roth, C.O. Johnson, A.A. Abajobir, F. Abdallah, S.F. Abera, G.Y. Abyu, M.B. Ahmed, B. Aksut, T. Alam, K. Alam, Global, Regional, and National Burden of Cardiovascular Diseases for 10 Causes, 1990 to 2015, Journal of the American College of Cardiology, 70 (2017) 1-25. [2] A. Kelemen, G. Székely, G. Gerig, Elastic model-based segmentation of 3-D neuroradiological data sets, IEEE Transactions on medical imaging, 18 (1999) 828-839. [3] P. Yushkevich, P.T. Fletcher, S. Joshi, A. Thall, S.M. Pizer, Continuous medial representations for geometric object modeling in 2D and 3D, Image & Vision Computing, 21 (2003) 17-27. [4] A. Gooya, K. Lekadir, I. Castro-Mateos, J.M. Pozo, A.F. Frangi, Mixture of probabilistic principal component analyzers for shapes from point sets, IEEE transactions on pattern analysis and machine intelligence, 40 (2017) 891-904. [5] I. Castro-Mateos, J.M. Pozo, M. Pereañez, K. Lekadir, A. Lazary, A.F. Frangi, Statistical Interspace Models (SIMs): Application to Robust 3D Spine Segmentation, IEEE Trans Pattern Anal Mach Intell, 34 (2015) 1663-1675. [6] V.H.H. Assen, M.M. Danilouchkine, A.A. Frangi, S.S. Ordás, J.J. Westenberg, J.J. Reiber, B.B. Lelieveldt, SPASM: a 3D-ASM for segmentation of sparse and arbitrarily oriented cardiac MRI data, Medical Image Analysis, 10 (2006) 286-303. [7] X. Alba, M. Pereanez, C. Hoogendoorn, A. Swift, J. Wild, A. Frangi, K. Lekadir, An Algorithm for the Segmentation of Highly Abnormal Hearts using a Generic Statistical Shape Model, IEEE Transactions on Medical Imaging, 35 (2016) 845-859. [8] T. Heimann, H.P. Meinzer, Statistical shape models for 3D medical image segmentation: A review, Medical Image Analysis, 13 (2009) 543-563. [9] J. Peters, O. Ecabert, C. Meyer, R. Kneser, J. Weese, Optimizing boundary detection via Simulated Search with applications to multi-modal heart segmentation, Medical Image Analysis, 14 (2010) 70-84. [10] H.C. Assen, Van, M.G. Danilouchkine, M.S. Dirksen, J.H.C. Reiber, B.P.F. Lelieveldt, A 3-D active shape model driven by fuzzy inference: application to cardiac CT and MR, IEEE Transactions on Information Technology in Biomedicine, 12 (2008) 595-605. [11] T.G. Catalina, B. Constantine, A. Santiago, S. Federico, M. Gloria, A.F. Frangi, Automatic construction of 3D-ASM intensity models by simulating image acquisition: application to myocardial gated SPECT studies, IEEE Transactions on Medical Imaging, 27 (2008) 1655-1667. [12] C. Butakoff, S. Balocco, F.M. Sukno, C. Hoogendoorn, C. Tobon-Gomez, G. Avegliano, A.F. Frangi, Left-ventricular epiand endocardium extraction from 3D ultrasound images using an automatically constructed 3D ASM, Computer Methods in Biomechanics Biomedical Engineering Imaging Visualization, 4 (2016) 265-280. [13] C. Tobongomez, F.M. Sukno, C. Butakoff, M. Huguet, A.F. Frangi, Automatic training and reliability estimation for 3D ASM applied to cardiac MRI segmentation, Physics in Medicine and Biology Biology, 57 (2012) 4155-4174. [14] R. Attar, M. Pereañez, A. Gooya, X. Albà, L. Zhang, M.H. de Vila, A.M. Lee, N. Aung, E. Lukaschuk, M.M. Sanghvi, Quantitative CMR population imaging on 20,000 subjects of the UK Biobank imaging study: LV/RV quantification pipeline and its evaluation, Medical image analysis, 56 (2019) 26-42. [15] T.F. Cootes, C.J. Taylor, D.H. Cooper, J. Graham, Active Shape Models-Their Training and Application, Computer Vision and Image Understanding, 61 (1995) 38-59. [16] G. Subsol, J.P. Thirion, N. Ayache, A scheme for automatically building three-dimensional morphometric anatomical atlases: application to a skull atlas, Medical Image Analysis, 2 (1998) 37-60. [17] A.D. Brett, C.J. Taylor, A method of automated landmark generation for automated 3D PDM construction, Image & Vision Computing, 18 (2000) 739-748. [18] A.F. Frangi, R. Daniel, J.A. Schnabel, W.J. Niessen, Automatic construction of multiple-object three-dimensional statistical shape models: application to cardiac modeling, IEEE Transactions on Medical Imaging, 21 (2002) 1151-1166. [19] M.R. Kaus, P. Vladimir, L. Christian, T. Roel, L. Steven, W. Jürgen, Automated 3-D PDM construction from segmented images using deformable models, IEEE Transactions on Medical Imaging, 22 (2003) 1005-1013. [20] R. Daniel, A.F. Frangi, J.A. Schnabel, Automatic construction of 3-D statistical deformation models of the brain using 16/18

nonrigid registration, IEEE Trans Med Imaging, 22 (2003) 1014-1025. [21] C. Tobon-Gomez, C. Butakoff, S. Aguade, F. Sukno, G. Moragas, A.F. Frangi, Automatic construction of 3D-ASM intensity models by simulating image acquisition: Application to myocardial gated SPECT studies, IEEE Transactions on Medical Imaging, 27 (2008) 1655-1667. [22] H.C. van Assen, A.F. Danilouchkine MGFrangi, S. Ordas, J.J. Westenberg, J.H. Reiber, B.P. Lelieveldt, SPASM: a 3D-ASM for segmentation of sparse and arbitrarily oriented cardiac MRI data, Medical Image Analysis, 10 (2006) 286-303. [23] T.F. Cootes, D.H. Cooper, C.J. Taylor, J. Graham, A Trainable Method of Parametric Shape Description, Image & Vision Computing, 10 (1991) 289-294. [24] T.F. Cootes, G.J. Edwards, C.J. Taylor, Active Appearance Models, IEEE Transactions on Pattern Analysis & Machine Intelligence, 23 (2001) 681-685. [25] H. Corné, D. Nicolas, S.Q. Damián, W. Tristan, F.M. Sukno, D.C. Mathieu, L. Karim, A.F. Frangi, A high-resolution atlas and statistical model of the human heart from multislice CT, IEEE Transactions on Medical Imaging, 32 (2013) 28-44. [26] H. Radu, F. Florence, Y. Manuel, D. Guillaume, Z. Jian, Rigid and articulated point registration with expectation conditional maximization, IEEE transactions on pattern analysis and

Machine Intelligence, 33 (2011) 587-602.

[27] F. Zhang, B. Du, L. Zhang, Saliency-Guided Unsupervised Feature Learning for Scene Classification, IEEE Transactions on Geoscience Remote Sensing, 53 (2015) 2175-2184. [28] W. Zhang, R. Li, H. Deng, L. Wang, W. Lin, S. Ji, D. Shen, Deep convolutional neural networks for multi-modality isointense infant brain image segmentation, NeuroImage, 108 (2015) 214-224. [29] Y. Dong, B. Du, L.J.I.J.o.S.T.i.A.E.O. Zhang, R. Sensing, Target Detection Based on Random Forest Metric Learning, IEEE Journal of Selected Topics in Applied Earth Observations Remote Sensing, 8 (2015) 1830-1838. [30] H. Huaifei, N. Pan, J. Wang, T. Yin, R. Ye, Automatic Segmentation of Left Ventricle from Cardiac MRI via Deep Learning and Region Constrained Dynamic Programming, Neurocomputing, 347 (2019) 139-148. [31] W. Bai, M. Sinclair, G. Tarroni, O. Oktay, M. Rajchl, G. Vaillant, A.M. Lee, N. Aung, E. Lukaschuk, M.M. Sanghvi, Automated cardiovascular magnetic resonance image analysis with fully convolutional networks, Journal of Cardiovascular Magnetic Resonance, 20 (2018) 65. [32] V. Badrinarayanan, A. Kendall, R. Cipolla, Segnet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE transactions on pattern analysis and

Machine Intelligence, 39 (2017) 2481-2495.

[33] E. Donal, G.Y. Lip, M. Galderisi, A. Goette, D. Shah, M. Marwan, M. Lederlin, S. Mondillo, T. Edvardsen, M. Sitges, EACVI/EHRA Expert Consensus Document on the role of multi-modality imaging for the evaluation of patients with atrial fibrillation, European Heart Journal–Cardiovascular Imaging, 17 (2016) 355-383. [34] H. Chang, A.H. Zhuang, D.J. Valentino, W. Chu, Performance measure characterization for evaluating neuroimage segmentation algorithms, NeuroImage, 47 (2009) 122-135. [35] L.Y. Radau P., Connelly K., Paul G., Dick A.J., W. G. A, Evaluation Framework for Algorithms Segmenting Short Axis Cardiac MRI, The MIDAS Journal-Cardiac MR Left Ventricle Segmentation Challenge, 49 (2009). [36] D.G. Altman, J.M. Bland, Measurement in medicine: the analysis of method comparison studies, Journal of the Royal Statistical Society: Series D (The Statistician), 32 (1983) 307-317.

17/18

Huaifei Hu received the M.S. and Ph.D in computer science and technology from Huazhong University of Science & Technology in 2007 and 2012, respectively. He is currently a Lecturer in College of Biomedical Engineering, South-Central University for Nationalities, Wuhan, Hubei, China. He was a visiting Scholar with the University Of Sheffield, UK, from 2015 to 2017. His current research interests include statistical modeling, medical image processing and deep learning.

Ning Pan is currently a Lecturer in School of Biomedical Engineering at South-Central University for Nationalities. He worked as a Post-Doctoral Fellow in School of Computer Science and Technology at Huazhong University of Science and Technology (HUST) for two years. He received his MS in Computer Software and Theory from Chongqing University (CQU) and his Ph.D in Computer Science from HUST, in 2007 and 2013 respectively. His research interests include medical image processing, data mining and statistical learning etc.

18/18

Tailang Yin is currently an Associate Professor in Reproductive Medicine Center, Renmin Hospital of Wuhan University. He received the M.S. and Ph.D in Reproductive Medicine Center from Wuhan University in 2007 and 2012, respectively. His current research interests include medical image analysis, infertility diagnosis and treatment, assisted reproductive treatment, embryo development research.

Haihua Liu received the Ph.D. degree in computer system architecture from the School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China, in 2006. He is currently a Professor with the College of Biomedical Engineering, South Central University for Nationalities, Wuhan. He has authored over 60 research articles in domestic and foreign academic journals, such as the Information Science, the Pattern Recognition, the Neurocomputing, the Magnetic Resonance Imaging, and the Science China: Information Sciences. His current research interests include computer vision, medical image analysis, pattern recognition, and cognitive computation.

Bo Du (M’10–SM’15) received the B.S. degree and the Ph.D. degree in photogrammetry and remote sensing from the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China, in 2005 and 2010, respectively. He is currently a Professor with the School of Computer, Wuhan University. He has authored over 40 research papers in the IEEE Transactions on Geoscience and Remote Sensing (TGRS), the IEEE Transactions on image Processing (TIP), the IEEE Journal of Selected Topics in Earth Observations and Applied Remote Sensing (JSTARS), and the IEEE Geoscience and Remote Sensing Letters (GRSL). Five of them are ESI hot papers or highly cited papers. His major research interests include pattern recognition, 19/18

hyperspectral image processing, and signal processing. His google scholar citation is 1796, and H-index is 22. He received the best reviewer awards from the IEEE GRSS for his service to the IEEE JSTARS in 2011 and the ACM rising star awards for his academic progress in 2015. He was the Session Chair of both the International Geoscience and Remote Sensing Symposium in 2016 and the 4th IEEE GRSS Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing. He also serves as a reviewer for 20 Science Citation Index magazines including the IEEE TGRS, TIP, JSTARS, and GRSL.

Figure Captions:

Fig 1. Work-flow for automatic image intensity model construction, CMR (cardiac magnetic resonance).

20/18

Fig 2. The pipeline of the adopted SegNet.

Fig 3. 3D-ASM initialization. (a) Endo and epi-cardial points of the mean PDM, (b) Endo and epi-cardial contours from ground truth, (c) Rigid point set registration, (d) Best approximation of PDM to endo and epi-cardial point sets from (b). 21/18

Fig 4. Distance maps for endo- and epicardial contours. (a) Original cardiac MR depicting endocardial contour, (b) Binary image of endocardial contour, (c) Distance map for endocardial contour, (d) Linear profile across distance map from endocardial contour, (e) Original cardiac MR depicting epicardial contour, (f) Binary image of epicardial contour, (f) Distance map of the epicardial contour, (d) Linear profile across distance map from epicardial contour. The solid white curves highlight endo- and epicardial contours (Cropped to enhance view).

22/18

Fig 5. Spatial distribution of point-to-surface errors.

23/18

Fig 6. Box plots of P2S errors for manual analysis and 3D-ASM. Errors computed at ED (end-diastole) and ES (end-systole) phases: ED-Endo and ES-Endo (P2S errors for endocardial surface), ED-Epi and ES-Epi (P2S errors for epicardial surface), ED-Whole and ES-Whole (P2S errors for full myocardial surface (endo and epicardium).

24/18

Fig 7. Volume calculation using 3D shapes. The left figure presents 3D segment results with the top-most slice, and the right figure depicts a clip result using the top-most slice.

25/18

Fig 8. Bland-Altman plot for LV clinical analysis.

26/18

Fig 9. Regression curve for LV clinical analysis.

27/18

Table Captions: Table 1. P2S errors for the clinical cases. Method

Diastolic phase

Systolic phase

Endo

Epi

Whole

Endo

Epi

Whole

Manual

1.78  0.15

1.59  0.11

1.50  0.07

2.06  0.32

1.65  0.19

1.63  0.09

3D-ASM

2.13  0.30

1.87  0.26

1.72  0.18

2.41  0.34

2.48  0.33

2.30  0.34

Table 2. Overlap and Jaccard similarity coefficient.

Diastolic phase Systolic phase Average

Endo Epi Endo Epi Endo Epi

Overlap

Jaccard

0.91  0.03

0.84  0.04

0.87  0.05

0.77  0.06

0.91  0.03 0.85  0.05 0.89  0.04 0.88  0.05

0.85  0.04 0.76  0.06 0.81  0.06 0.81  0.07

Table 3. Cardiac functional indexes. LVEDV (ml) LVESV (ml) LVSV (ml) LVEF (%) LVM-DP (g) LVM-SP (g)

From experts

3D-ASM

141.60  37.82 59.07  25.42 81.12  19.05 58.67  8.37 91.64  26.44 89.58  26.24

146.40  37.05 61.26  26.32 83.31  21.16 58.51  10.29 93.11  24.58 86.90  28.08

28/18