- Email: [email protected]
Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning
BBE 315 1–10 biocybernetics and biomedical engineering xxx (2018) xxx–xxx
Available online at www.sciencedirect.com
ScienceDirect journal homepage: www.elsevier.com/locate/bbe 1 2 3
Original Research Article
Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning
4 5 6
7 8 9 13 10
Q1
Beevi K. Sabeena a,*, Madhu S. Nair b, G.R. Bindu c a
Electrical & Electronics Department, T. K. M College of Engineering, Kollam, Kerala, India Department of Computer Science, Cochin University of Science and Technology, Kochi 682022, Kerala, India c Electrical Engineering Department, College of Engineering Wayanad, Wayanad 695016, Kerala, India b
11 12 14 15 16 17 16 18 17 16 19 18 17 16 20 19 18 17 16 21 20 19 18 17 16 22 21 20 19 18 17 16 23 22 21 20 19 18 17 16 24 23 22 21 20 19 18 17 16 25 24 23 22 21 20 19 18 17 16 26 25 24 23 22 21 20 19 18 17 16 27 26 25 24 23 22 21 20 19 18 17 16 28 27 26 25 24 23 22 21 20 19 18 17 16 29 28 27 26 25 24 23 22 21 20 19 18 17 16 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21
37 38 39
article info
abstract
Article history:
The exact measure of mitotic count is one of the crucial parameters in breast cancer grading
Received 15 May 2018
and prognosis. Detection of mitosis in standard H & E stained histopathology images is
Received in revised form
challenging due to diffused intensities along object boundaries and shape variation in
23 October 2018
different stages of mitosis. This paper explores the feasibility of transfer learning for mitosis
Accepted 28 October 2018
detection. A pre-trained Convolutional Neural Network is transformed by coupling random
Available online xxx
forest classifier with the initial fully connected layers to extract discriminant features from nuclei patches and to precisely prognosticate the class label of cell nuclei. The modified
Keywords:
Convolutional Neural Network accurately classify the detected cell nuclei with limited
Histopathology
training data. The designed framework accomplishes higher classification accuracy by
Mitosis
carefully fine tuning the pre-trained model and pre-processing the extracted features.
Krill herd optimization
Moreover, proposed method is evaluated on MITOS dataset provided for the MITOS-ATYPIA
Convolutional Neural Network
contest 2014 and clinical data set from Regional Cancer Centre, Thiruvananthapuram, India.
Transfer learning
Significance of Convolutional Neural Network based method is justified by comparing with
Random forest
recently reported works including a Multi Classifier System based on Deep Belief Network.
Q3 Multi Classifier System
Experiments show that the pre-trained Convolutional Neural Network model outperforms conventionally used detection systems and provides at least 15% improvement in F-score on other state-of-the-art techniques.
Q2
© 2018 Published by Elsevier B.V. on behalf of Nalecz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences.
1.
Introduction
Q4 Introducing digital slide libraries together with computer-
aided diagnosis (CAD) brought radical changes in the analysis of pathology images [1]. Among the grading factors for breast
cancer, the mitotic count is a significant characteristic of tumor proliferation [2]. Mitosis, a complex biological process, appears as hyper chromatic objects with pseudo projections around the edges. Shape of a nucleus varies through different stages [3] of mitosis such as prophase, metaphase, anaphase and telophase as shown in Fig. 1. Moreover, two divided nuclei
* Corresponding author at: Electrical & Electronics Department, T. K. M College of Engineering, Kollam, Kerala, India. E-mail addresses: [email protected] (B.K. Sabeena), [email protected] (M.S. Nair), [email protected] (G.R. Bindu). https://doi.org/10.1016/j.bbe.2018.10.007 0208-5216/© 2018 Published by Elsevier B.V. on behalf of Nalecz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences. Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
40 41 42 43 44 45
BBE 315 1–10
2
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
Fig. 1 – Cell in different phases of mitosis.
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92
in the telophase have to count as single mitosis because they are not distinct cells. The presence of lymphocytes, cells with partial nuclei, feeble edges, similar grey level of different soft tissues and shape variation among cells complicates the detection process with many false positives. In addition, a low density of nuclei undergoing mitosis makes the classification process forbiddingly challenging [4]. The already reported techniques in the literature exploit cellular features that identify morphology and intensity of mitotic nuclei. But due to shape variation and indistinguishable nuclei features, such sort of systems result in reduced detection accuracy with a large number of false positives or wrong detections. The deep learning approach uses a deep architecture such as Convolutional Neural Network (CNN) to learn proper features from the images [5]. Training of CNN for a real world problem requires excessive computations with large samples of annotated images. In [6], Ciresan et al. employed one million nuclei to train CNN which resulted an FScore of 78%. But it has to be improved further for clinical practice. Even with GPU, training time takes hours and weeks depending on the problem. Since interpretation of biomedical data requires medical expertise and is bound by diagnostic variability, the field of medical imaging lacks availability of large volumes of labelled data (Ground Truth). This paper proposes an alternate method to further improve the accuracy of mitosis detection using transfer learning technique by incorporating minimal variation in the architecture of CNN. Transfer learning is a concept where weights of a pre-trained model is transferred to another problem set on a different dataset. A pre-trained model is an already trained model created to solve a similar problem. The key idea is that information coming from the source tasks to the target task may be useful to speed-up the learning process. Based on the nature of dataset it is essential to make modifications in the pre-existing model by fine-tuning the network. In [7], Yosinski et al. enumerated the transferability of the learned representations to different domains and also established reuse of the parameters in the network for different tasks. Recently, these methods achieved acceptance in medical imaging applications [8]. This work investigates the application of such methods on mitosis dataset. The proposed approach presents an automatic mitosis detection to tackle the preceding challenges and outperforms the state-of-the-art techniques in mitosis detection. Traditional classification systems include cell segmentation, feature extraction and classification as major steps. Break up of nuclear membrane in the early stages of mitosis leads to
diffusion of nuclei and background regions. Hence, it is difficult to find a clear threshold to realize cell nuclei in images. Krill Herd Algorithm (KHA) [9] based optimal multithresholding enables quick extraction of nuclei patches from the huge pathology images by locating centroids of the nuclei regions. For good classification, the specific changes in nuclei during mitosis are to be computed as discriminant features. The intensity based and morphology based features are unable to provide variations in nuclei during mitosis. The proposed methodology looks for extracting generic low-level features and complex features from the localized cells by a very deep convolutional network Caffe model-VGGNet [10,11], which is tuned to extracts nuclei variations in images as features by successively modeling tiny information and combine them deeper in the network. Deeper networks give better results, hence the name deep transfer learning. The strength of different classifiers are exploited to deal with complex high dimensional features where the number of features are larger than the number of instances. Visualization of features further confirm reliable distinction between mitotic and non-mitotic nuclei. Hence, a strong feature representation with a strong classifier is achieved for mitosis detection. Further, the algorithm is tested on a clinical data set obtained from Regional Cancer Centre (RCC), Thiruvananthapuram, India, along with a very challenging MITOS dataset [12] to establish the effectiveness of the proposed technique. The transfer learning technique needs only moderate computational resources and less number of labelled data. Moreover, we compare the performance of the proposed technique with recently reported techniques [5,13] including a Multi Classifier System (MCS) based on Deep Belief Networks (DBN) [14] that has been used the same dataset for detecting mitosis. The discriminant features from the pre-trained CNN provide significant reduction in false positive rate and overcome the need for large labelled data. The rest of the paper is organized such that Section 2 presents relevant literature related to this work. Section 3 briefly introduces CNN model and Section 4 describes the proposed method. The Section 5 discusses experimental results and the conclusions are drawn in Section 6.
93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132
2.
133
Review of related literature
Compared to cytopathology images where nuclei are well separated, the analysis of nuclei in histopathological images is more difficult due to the complex and irregular visual
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
134 135 136
BBE 315 1–10
3
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196
appearance of the nuclei. Over two decades ago, initial methods for mitosis detection were reported with a semiautomatic algorithm in Feulgen stained breast cancer sections [15]. Probability based likelihood functions along with binary thresholding is used in [16] to identify mitotic nuclei in neuro images. Irshad et al. [17] employed thresholding of blue ratio images to detect nuclei regions and RF classifier for mitotic differentiation. Basic thresholding yields a low detection rate owing to diffusion of nuclei and background regions. Khan et al. [18] proposed a gamma-gaussian mixture model by isolating tumour regions from non-tumour areas for detection of mitotic nuclei. However, a context based tuning is required in the classification stage to reduce the number of false positives. A graph-based multi-resolution algorithm [19], extracts mitotic figures by clustering domain specific features. Anari et al. [20] applied Fuzzy C-means (FCM) clustering along with an ultra-erosion operation for detection of the mitosis index in Immuno Histo Chemical (IHC) images. For large histopathology images, clustering algorithms are very slow. In [21], many false positives were detected by hysteresis thresholding based morphological reconstruction operations in breast histology images. In [22], texture and SIFT features along with intensity, morphology, and run-length features from different color models were used for the mitosis prediction. Many redundant feature values from different colour channels result in a low classification outcome. In [23], nuclei classification was dealt with color based features. But use of chromatic filters influence performance of the Adaboost Classifier. In [24], a localized active contour model based segmentation is applied along with random features for detecting mitosis. The authors utilized Gabor features and texture features for automatic grading of breast cancer in [25]. But the similarity in shape and color attributes of other cell structures cause wrong detections. Lu and Mandal [26], applied a multi-expert system for classification of mitotic cells in multispectral images. The authors selected the spectral bands using linear discriminant analysis. Segmentation of the nuclei regions were carried out by Bayesian modeling followed by local region thresholding. A recent work [13] reported mitosis detection with an average F-score of 73% based on intensity features and morphological filtering. Ciresan et al. first utilized CNN based deep max-pooling for mitosis detection in breast histology images [6]. In [27], Wang et al. computed CNN features along with handcrafted features to detect mitosis. The authors utilized local thresholding of blue ratio image for initial segmentation. For two feature sets, independent classifiers were used and a third classifier is used for the feature combination. In [28], whose work earlier inspired us, the transfer learning facility of CNN is investigated to interpret the leukemia cell lines in cytopathology images and also in mitosis detection [29], but they require manual intervention in pre-processing and classification stages. In [30], initially pixel classification is done based on handcrafted features. Then a multi-stage CNN is employed for mitosis prediction by which the authors claim for accuracy and timing. The different kernel maps in CNN looks for local features and share the parameters along the depth volume. Training CNN requires many labelled samples and parameters that increase the computational complexity of the algorithm [5,31,32]. Since most of these features are fundamental characteristics for any
image data set, kernels can be used as a feature extractor for the other classification challenges. Wan et al. [33] extracted multi-level features by a CNN after segmentation using a hybrid active contour model. Further, multiple support vector machine (SVM) classifiers are utilized for classification. Robust identification and measurement of naturally occurring deformable structures is not always achievable using a single technique due to insufficient samples, presence of artifacts and conflicting range of inputs in biomedical domain. In this paper, we propose a hybrid technique, which maximizes entropy of nuclei regions by the Krill Herd algorithm and utilizes the effective feature extraction capability of CNN through transfer learning. The proposed CNN model provides improved accuracy over a wide range of data set, compared with other recently reported works.
197 198 199 200 201 202 203 204 205 206 207 208 209 210 211
3.
Convolutional Neural Network (CNN)
212
Consider the CNNs as a sequence of feed-forward neural network functions that extract features at multiple levels or layers [34]. At each layer, it computes features from the previous layer representations (from low-level to high-level). Internally, it comprises a group of functions organized as
213 214 215 216 217 218
gðxÞ ¼ gL ð. . .g2 ðg1 ðx; w1 Þ; w2 Þ. . .Þ; wL Þ
(1)
where x and w are the input vector and weight vector, respectively. Varying layers ‘ in CNNs perform fundamental operations, such as
Convolution with a group of filters Normalization Non-linear activation Sub-sampling or Pooling
The five main types of layers to build CNN architecture are briefly explained below: The convolutional layer carries out convolution to capture highly redundant and highly correlated local information present in the input. For a given image X, the convolution block performs the convolution of the image with a bank of K filter maps, f ðw; hÞ, with height h and width w. Each filter computes a dot product between their weights and a definite region it links to in the input image. Normalization layer regulates output from the early stages and passes to the next non linear activation layer. This process makes the weights from the network balanced without very high or low values, since it includes normalization in the gradient process. Rectified Linear Unit (ReLU) performs an element wise thresholding. It is the most commonly used activation function for CNNs. Formally, it is characterised as:
219 220 221 222 223 224 225 226 226 228 227 227 230 229 228 232 231 229 230 233 231 234 232 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249
f ðxÞ ¼ maxðx; 0Þ
(2)
CNNs with ReLUs train many times quicker than their equivalents with tanh functions. The number of complex features increase, as the network grows deeper and deeper.
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
250 251 252 253 254
BBE 315 1–10
4 255 256 257 258 259 260 261
262 263 265 264 265 267 266 266 268 267 269
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
However, the feature map size is being shrunk through the pooling operation by taking maximum activations. It is a way of down sampling the image space along the spatial dimensions. As in common Neural Networks, it connects each neuron in Fully-Connected Layer (FCL) to all neurons in the previous layer and calculates class counts.
4.
Methodology
The proposed method comprises of mainly two phases such as Phase I: Pre-processing and cell localization Phase II: Classification of cells
4.1.
Pre-processing and cell localization
270 271 272 273 274 275 276 277 278 279
Tissue samples of the MITOS data set [12] and clinical data set show significant stain variation. Hence, stain normalization is carried out by a recent color deconvolution technique [35]. KHA based optimal multi thresholding technique provides the exact centroid location of individual nuclei as explained in [36]. Localized cell patches of size 25 25 pixels, enclosing a nucleus at the center are inputs to the CNN that extracts discriminant features. The use of tiny patches adds the volume of total training data and confines the analysis to actual nuclei in images.
280
4.2.
281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297
The texture, size and shapes of the mitotic cells keep on changing in different stages of mitosis. So, this work uses the deep feature representation capability of CNN. Amid many choices, we use pre-trained CNN model VGGNet in Matlab [37], which is heavily trained on challenging images of 1000 categories of images. VGGNet also shows that the depth of the network plays an important role. It contains 9 learned layers including convolutional (C) and pooling layers (P) of particular specifications. The first layer of VGGNet architecture consists of a convolutional layer with 64 filters of 11 11. A ReLU activation function and a normalization of 5 5 are also present. Layer 2 to 5 consist of convolutional layers with 256 filters of size 3 3 and 5 5. Throughout the network pooling layers perform 3 3 maximum pooling. The R, N and FC represent ReLU, normalization and fully connected layers. The filter size of 11 11, 5 5 and 3 3 with stride of 1 and padding of 1 are
Classification of cells
carefully chosen to retain the feature map after convolution. The max pooling layer decreases the feature map size all the way to the end of the network. At the very end fully connected layers and softmax layers are added for the final classification. Table 1 shows parameters of the pre-trained CNN model in Matlab. Transfer learning is an approach where the weights of a previously developed model is transferred to another problem on a specific dataset. The network VGGNet already has trained weights on ImageNet dataset. We retrain the network on the new dataset such as, the cell patches from the MITOS and RCC datasets. Depending on the number of outputs for the new problem, the last layer should substitute with another one of our choice. Here, we have data for mitosis and non-mitosis cases. Hence, the last layer is replaced with a 2-node softmax layer. To do this, we removed the fully-connected layer at the top of the network and then added new fully connected layer with random initialization, with two output nodes. The network is retrained on the new dataset such as, the cell patches from the MITOS and RCC datasets for two classes. Since in this case size of the data is small as well as data similarity is very low, the whole process ended up with poor classification outcomes of 30% for the first two data sets. Hence as a second stage, we go for fine tuning the network to make it as a feature extractor. Each layer in a CNN successively builds up higher and stronger level of details. The earlier layers are often more generic as there are many simple patterns common among images. In addition, the last layers are very specialized on the input data supplied to the model. Since the network already learned so much about edges, curves and objects from the ImageNet dataset, it can relate them to the newer dataset. The weights of the first three layers of the pre-trained network are freezed to keep the curves and edges that are relevant to object detection task. This is done by providing low values for learning rate to the corresponding layers. To learn the specific features related to nuclei patches we again retrained the later four layers with nuclei dataset. Since the intermediate layers of CNN are not too meticulous to the originally trained dataset, they apprehend sufficient descriptions for transfer-learning studies [38]. Since the output layer of the network is too task-specific to the dataset, the network was originally trained on; it cannot be used for the final classification. Accordingly, we replace the last Softmax layer with a Random Forest classifier at the output of the network. The RF classifier is taught with the activations from the fully connected layers. The fully connected layer abstracts feature list from convolution layers by blending together features detected from the image patches for this particular
Table 1 – Architecture of pre-trained CNN model in Matlab. Layer 1 1 1 1 2 2 2
Type
Filter Dim.
Layer
Type
Filter Dim.
Layer
Type
Filter Dim.
Conv1 ReLU1 Norm1 Pool1 Conv2 ReLU2 Norm2
11 11 3 64
2 3 3 4 4 5 5
Pool2 Conv3 ReLU3 Conv4 ReLU4 Conv5 ReLU5
Max 3 3 256 256
5 6 6 7 7 8 9
Pool5 FC6(Conv) ReLU6 FC7(Conv) ReLU7 FC8(Conv) Softmax
Max 6 6 256 4096
Max 5 5 64 256
3 3 256 256 3 3 256 256
1 1 4096 4096 1 1 4096 1000
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345
BBE 315 1–10 biocybernetics and biomedical engineering xxx (2018) xxx–xxx
346 347 348 349 350 351 352 353 354 355
task. Essentially, now the network act as an automatic feature extractor. Here features are extracted from the 7th layer before the last fully connected Softmax layer. The extracted feature vector takes the same length of 4096 as that of sixth and seventh layers of VGGNet. Principal component analysis (PCA) [39] reduces the depth of these vectors from 4096 to 650. PCA is a method that can be applied for dimensionality reduction. The prime objective of PCA is to identify a compact depiction of the data, under the assumption that the inputvariables are corresponded with each other and thus redun-
5
dant. The focus is to define the initial high-dimensional data using a lower dimensional subspace. Geometrically, this can be viewed as a smooth estimate of the original data into a lower-dimensional scheme to preserve as much information as possible. The main axes of this new coordinate structure are called principal components. A classifier subset evaluator further selects a feature vector of size 61 as a subset that gives the best discriminant information by a hill climbing search [40]. The algorithm works in such a way that it trains a classifier model for each
Fig. 2 – Probability density plots and box plots for first 4 (x1–x4) nuclei features. (a) Probability distribution of mitosis (blue) and non mitosis (pink) nuclei features. (b) Boxplot of mitosis (1) and non mitosis (0) nuclei features.
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
356 357 358 359 360 361 362 363 364 365
BBE 315 1–10
6
366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
Table 2 – Description of dataset with total number of images (TNI) and total number of mitosis (TNM).
5.
Experimental results
Data set
A03
A04
A07
A10
Clinical data
5.1.
Data set
TNI TNM
96 128
86 218
64 25
80 39
53 45
new subset and a score is computed based on the errors. The subset with the maximum score is selected as the final feature subset. To assess the level of feature representation from inner layers of CNN and to understand the relationships between the extracted features of mitosis and non-mitosis nuclei, box plots and probability density plots were taken that give smooth lines for each distribution. Fig. 2(a) shows probability distribution of mitosis (blue) and non-mitosis (pink) nuclei and Fig. 2(b) displays box plot of mitosis (1) and non-mitosis (0) nuclei for first four (x1–x4) features. Both plots show significant differentiation between mitosis and non-mitosis clusters that capture adequate content for distinguishing between the two classes. Four traditional classifiers such as Support Vector Machines (SVM), Linear Discriminant Analysis (LDA), Random Forest (RF) and k-Nearest Neighbours (kNN) train the extracted features for classification. This is a suitable fusion of simple linear (LDA), composite nonlinear (SVM, RF) and nonlinear (kNN) methods. The highest accuracy is realized by RF and is used to label the mitotic and nonmitotic cell.
387 388
The algorithm is validated on the MITOS data set [12] and also on clinical data set from the RCC, Thiruvananthapuram, India. Two slide scanners such as Aperio (AP) and Hamamatsu (HM) scan tissue slides at 40 magnification with a resolution of 0.23–0.24 mm per pixel. The mitoses are recorded by expectation values of 1.0, 0.8, 0.6, 0.2 or 0. We carry out our experiments using their ‘‘in-group’’ dataset, which is the data set for which they furnish ground truth for comparison. Images from RCC are obtained by the Leica digital image acquisition system attached to the microscope. Images in the MITOS data set are of 1376 1539 3 size and RCC images are of 3264 2448 3 size. Two experienced pathologists manually assess mitotic nuclei in clinical images. The MITOS data set as well as RCC dataset comprises high power field (HPF) images of complex breast tissue stained with Hematoxylin and Eosin (H&E). Table 2 describes the different datasets, A03, A04, A07, A10 (folder names of MITOS data set with Aperio scanner) and RCC data set in terms of total number of images (TNI) and total number of mitosis (TNM). There were 5-10 mitotic cells in the telophase and each of them were taken as a single cell patch. From the four folders of ICPR 2014 dataset, mitotic regions with probability more than 0.6 were incorporated as mitotic in our training set. To treat class disparity and produce rotational invariance, nuclei patches containing
Fig. 3 – (a) Sample image with stain variation (b) Reference image (c) Stain normalized image with color deconvolution method.
Fig. 4 – Images with centroid of detected nuclei. (a) Normalized R component of original image (b) centroids located by normal thresholding, where big red circles indicate wrong centroid locations for adjacent nuclei (c) KHA optimized image, here the Q5 red circular markings represent correct centroids of the adjacent nuclei. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412
BBE 315 1–10
7
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
Table 3 – Comparison of cross validation accuracy. No.of features 4096 61
Fig. 5 – Plot of the model evaluation results.
413 414 415 416 417 418 419 420 421
mitotic nuclei were replicated with rotations. In order to cut down the skewness of data, careful sampling of non-mitotic data and random up sampling of mitotic data are further carried out. From all the data sets considered, 70% of nuclei are taken for training and remaining 30% are taken for testing. Every nucleus in the training set is marked as either mitotic or non-mitotic based on the available ground truth and test set is prepared with an equal number of mitosis and non-mitosis nuclei. The experiments were carried out by using Matlab 2016 environment.
422
5.2.
423 424 425
To assess the mitosis classification, the proposed framework use Sensitivity, Precision and F-score as given in Eqs. (3)–(5) along with Specificity and Accuracy.
Performance measures
426 Sensitivity ¼
Ntp 100 Ntp þ Nfn
(3)
428 427 429 Precision ¼
Ntp 100 Ntp þ Nfp
(4)
431 430 432 Fscore ¼ 2 433 434 435 436 437
SensitivityPrecision 100 Sensitivity þ Precision
(5)
where Ntp is the number of true positives (tp-correctly detected Mitosis), Nfp number of false positives (fp-wrongly detected Mitosis) and Nfn number of false negatives (fn-missed mitosis).
K = 10
K=5
0.86 0.94
0.84 0.91
corresponding normalized image along with a reference image. Three thresholds such as T1, T2 and T3 produced by KHA based optimal multi thresholding distinguishes cell nuclei from other cell structures. Fig. 4 displays the effect of KHA on the localization of cell nuclei. Normal thresholding (Fig. 4(b)) did not identify centroids of many adjoining nuclei, but they have been properly discriminated by the KHA based optimal thresholding (Fig. 4(c)). Hence, it helps in reducing the number of false positive and false negative nuclear centroids. The computational power of the KHA based multilevel thresholding is illustrated in [36], in which the KHA performs 15–20 times faster than the Bacterial Foraging Algorithm with 94.31% detection sensitivity.
441 442 443 444 445 446 447 448 449 450 451 452
5.4.
453
Classification
Since the sixth and seventh layers of VGGNet consist of 4096 neurons (fully connected layers) with the ReLU activation function, the extracted feature vector has the same dimension of 4096. Repeated cross validation is carried out using the linear and nonlinear classifier models to create a best classifier. We reorganize the random number seed in advance to guarantee the same data splits in each algorithm. In each experiment, training data from all category is divided into K segments, each holding the same number of cells. It also confirms that the results are comparable. The mean accuracy and kappa value is computed. The RF classifier showed the highest accuracy as shown in Fig. 5. The plot relates the spread and the mean accuracy of each model. At first, training is done with all computed features. The cross-validation accuracy, averaged over five runs performed with all features and selected features, is reported in Table 3. The RF classifier trained with the preferred feature set is employed to recognize other unknown instances of mitosis from the evaluation set. The chosen subset of features offered consistent values for all the data sets as shown in Table 4 and resulted in an average value of 88.6% and 89.66% F-score for Mitos dataset and RCC dataset respectively as shown in Table 5. The slide images from Mitotic dataset consist of tissue artefacts such as, tissue curling and many mitotic-like non mitosis regions. A07 and A10 folders contain comparatively less no. of mitotic nuclei. Due to these variations, there is reduction in detection performance in certain images of A10 and A07 slides. Moreover, it consists of many atypical nuclei. Fig. 6 shows visual results of the proposed algorithm in which, Fig. 6(a)
Table 4 – Performance measures on each MITOS dataset. Dataset Sensitivity Specificity Accuracy Precision F-Score
438 439 440
5.3.
Pre-processing and cell localization
Stain normalization increases the contrast between cell nuclei and other cell structures. Fig. 3 presents a stain varied image and
A03 A04 A07 A10
92.31 97.5 81.82 88.88
90.77 87.5 90.91 77.77
91.54 92.5 86.36 83.33
90.91 88.64 90 80
91.60 92.86 85.71 84.21
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482
BBE 315 1–10
8
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
Table 5 – Average on each dataset. Dataset Average on Mitos Data set RCC dataset
Sensitivity
Specificity
Accuracy
Precision
F-Score
90.13 86.67
86.74 93.33
88.43 90
87.39 92.86
88.60 89.66
Fig. 6 – Visual results. (a) Original image (b) nuclei located by KHA (c) Classification results (nuclei shown in Red circles: tp, Yellow circles: fp, Blue circles: fn). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Table 6 – Comparison with MCS. Classifier
Sensitivity
Precision
F-Score
DBN-MCS Proposed
93.33 90.13
75.38 87.39
83.4 88.6
Fig. 7 – Plot of the discriminant nature of CNN features compared with that of morphometric features in [14]. A clear discrimination exist between mitosis (blue) and non-mitosis (pink) features extracted by pre-trained CNN. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
BBE 315 1–10 biocybernetics and biomedical engineering xxx (2018) xxx–xxx
9
images. To realize each attribute and to understand the correlation between them, univariate and multivariate plots of the selected features are further analysed. A wide dataset (a challenging standard dataset and a clinical data set from a cancer research institute, RCC, Thiruvananthapuram, India is used to test the proposed method. The F-Score is increased from 0.86 to 0.94 after dimensionality reduction and sequential feature selection in the training data. The results prove that the proposed framework surpasses the existing techniques with high sensitivity and precision that makes it more practical in clinical diagnosis.
521 522 523 524 525 526 527 528 529 530 531 532
Fig. 8 – F-score of the proposed method with MCS and recently reported methods [5] and [13].
483 484 485 486 487
presents the original image (b) displays the nuclei locations provided by KHA based multi thresholding and (c) Classification results: nuclei in red circles (tp). Nucleus in the blue circle shows the missed mitosis (fn) and nucleus shown in a yellow circle presents false mitosis (fp).
488
5.5.
489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511
Table 6 presents the classification using the proposed method and the multi classifier system (DBN-MCS) as explained in [14], where the classification is based on cellular features extracted after accurate segmentation by Localized Active Contour Model (LACM). Fig. 7 displays the plot of discriminant nature of three sets of CNN features and morphometric features. The morphometric features of mitotic (blue) and non-mitotic nuclei (pink) are tough to differentiate compared to corresponding CNN features. A significant reduction in false positive rate is apparent from the enhanced precision values given by the proposed model with a slight reduction of sensitivity. However, it results in direct improvement of 5.4% in F-score, which is more significant. Fig. 8 compares the proposed method with two recently published techniques [5,13] who analysed the algorithm using the 2014 ICPR Mitosis training and testing dataset. Angshuman et al. [13] carried out mitosis detection using hand crafted features and Chen et al. [5] evaluated it by cascaded deep networks. The methods [5] and [13] were evaluated with test set from ICPR Mitos database while the proposed method has test set from both Mitos and RCC data set. With the proposed CNN based model, the classification becomes more robust with significant progress in precision and F-score.
512
6.
513 514 515 516 517 518 519 520
Since mitosis is a complex biological process, conventional approaches cannot accurately detect them. In this study, automated mitosis detection is carried out using deep features extracted by VGGNet, caffe model of CNN trained on the popular imaging database ImageNet. This overcomes the need for large labelled data and generates discriminative features for the correct classification. The use of localized nuclei patches by KHA confines the analysis to small nuclei in
533
Acknowledgment
534
The authors acknowledge the support rendered for this work by RCC Thiruvananthapuram, India. The authors also thank Dr. Sujathan, Dr. Jayasree, Dr. Abitha and Dr. Anju for their timely guidance and help in the evaluation process.
535 536 537 538
references
539
[1] Malon CD, Cosatto E, et al. Classification of mitotic figures with convolutional neural networks and seeded blob features. J Pathol Inform 2013;4(1):9. [2] Elston E, Ellis I. Method for grading breast cancer. J Clin Pathol 1993;46(2):189. [3] Barlow P. Changes in chromatin structure during the mitotic cycle. Protoplasma 1977;91(2):207–11. [4] Dundar MM, Badve S, Bilgin G, Raykar V, Jain R, Sertel O, et al. Computerized classification of intraductal breast lesions using histopathological images. IEEE Trans Biomed Eng 2011;58(7):1977–84. [5] Chen H, Dou Q, Wang X, Qin J, Heng PA. Mitosis detection in breast cancer histology images via deep cascaded networks. Thirtieth AAAI Conference on Artificial Intelligence; 2016. [6] Cireşan DC, Giusti A, Gambardella LM, Schmidhuber J. Mitosis detection in breast cancer histology images with deep neural networks. International Conference on Medical Image Computing and Computer-assisted Intervention. Springer; 2013. p. 411–8. [7] Yosinski J, Clune J, Bengio Y, Lipson H. How transferable are features in deep neural networks?Advances in neural information processing systems. 2014;3320–8. [8] Xu Y, Mo T, Feng Q, Zhong P, Lai M, Eric I, et al. Deep learning of feature representation with multiple instance learning for medical image analysis. 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2014. pp. 1626–30. [9] Gandomi AH, Alavi AH. Krill herd: a new bio-inspired optimization algorithm. Commun Nonlinear Sci Numer Simul 2012;17(12):4831–45. [10] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556. [11] Jia Y, et al. Caffe: an open source convolutional architecture for fast feature embedding; 2013. [12] MITOS, ICPR 2014 Contest. IPAL UMI CNRS Lab Std; 2014, http://ipal.cnrs.fr/ICPR. [13] Angshuman P, Mukherjee DP. Mitosis detection for invasive breast cancer grading in histopathological images. IEEE Trans Image Process 2015;24(11):4041–54.
540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579
Comparison
Conclusion
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
BBE 315 1–10
10 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
[14] Beevi KS, Nair MS, Bindu G. A multi-classifier system for automatic mitosis detection in breast histopathology images using deep belief networks. IEEE J Transl Eng Health Med 2017;5:1–11. [15] Belien J, Baak J, van Diest PJ, Van Ginkel A. Counting mitoses by image processing in Feulgen stained breast cancer sections: the influence of resolution. Cytometry Part A 1997;28(2):135–40. [16] Sertel HSO, Catalyurek UV, Guican MN. Computeraided prognosis of neuroblastoma: detection of mitosis and karyorrhexis cells in digitized histological images. Int. Conf. IEEE Eng. Med. Biol. Soc. (EMBC); 2009. pp. 1433–6. [17] Irshad H, Roux L, Racoceanu D. Multi-channels statistical and morphological features based mitosis detection in breast cancer histopathology. 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); 2013. pp. 6091–4. [18] Khan AM, El-Daly H, Rajpoot NM. A gamma-gaussian mixture model for detection of mitotic cells in breast cancer histopathology images. 2012 21st International Conference on Pattern Recognition (ICPR); 2012. pp. 149–52. [19] Roullier V, Lézoray O, Ta V-T, Elmoataz A. Multi-resolution graph-based analysis of histopathological whole slide images: application to mitotic cell extraction and visualization. Comput Med Imaging Graphics 2011;35 (7):603–15. [20] Anari V, Mahzouni P, Amirfattahi R. Computer-aided detection of proliferative cells and mitosis index in immunohistichemically images of meningioma. 2010 6th Iranian Conference on Machine Vision and Image Processing; 2010. pp. 1–5. [21] Beevi S, Nair MS, Bindu G. Automatic segmentation and classification of mitotic cell nuclei in histopathology images based on active contour model. 2014 International Conference on Contemporary Computing and Informatics (IC3I); 2014. pp. 740–4. [22] Irshad H, Jalali S, Roux L, Racoceanu D, Hwee LJ, Le Naour G, et al. Automated mitosis detection using texture, sift features and hmax biologically inspired approach. J Pathol Inform 2013;4(2):12. [23] Tek FB, et al. Mitosis detection using generic features and an ensemble of cascade adaboosts. J Pathol Inform 2013;4 (1):12. [24] Beevi KS, Nair MS, Bindu G. Detection of mitotic nuclei in breast histopathology images using localized acm and random kitchen sink based classifier. 2016 IEEE 38th Annual International Conference of the Engineering in Medicine and Biology Society (EMBC); 2016. pp. 2435–9. [25] Doyle S, Agner S, Madabhushi A, Feldman M, Tomaszewski J. Automated grading of breast cancer histopathology using spectral clustering with textural and architectural image features. 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro, 2008. ISBI 2008; 2008. pp. 496–9.
[26] Lu C, Mandal M. Toward automatic mitotic cell detection and segmentation in multispectral histopathological images. IEEE J Biomed Health Inform 2014;18(2):594–605. [27] Wang H, Cruz-Roa A, Basavanhally A, Gilmore H, Shih N, Feldman M, et al. Cascaded ensemble of convolutional neural networks and handcrafted features for mitosis detection. SPIE medical imaging. International Society for Optics and Photonics; 2014. p. 90410B. [28] Gopakumar G, Babu KH, Mishra D, Gorthi SS, Subrahmanyam GRS. Cytopathological image analysis using deep-learning networks in microfluidic microscopy. JOSA A 2017;34(1):111–21. [29] Albayrak A, Bilgin G. Mitosis detection using convolutional neural network based features. 2016 IEEE 17th International Symposium on Computational Intelligence and Informatics (CINTI); 2016. pp. 000335–40. [30] Li Y, Mercan E, Knezevitch S, Elmore JG, Shapiro LG. Efficient and accurate mitosis detection-a lightweight rcnn approach. ICPRAM. 2018. pp. 69–77. [31] Malon CD, Cosatto E, et al. Classification of mitotic figures with convolutional neural networks and seeded blob features. J Pathol Inform 2013;4(1):9. [32] Sirinukunwattana K, Raza SEA, Tsang Y-W, Snead DR, Cree IA, Rajpoot NM. Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans Med Imaging 2016;35 (5):1196–206. [33] Wan T, Cao J, Chen J, Qin Z. Automated grading of breast cancer histopathology using cascaded ensemble with combination of multi-level image features. Neurocomputing 2017;229:34–44. [34] Ciresan DC, Meier U, Masci J, Maria Gambardella L, Schmidhuber J. Flexible, high performance convolutional neural networks for image classification.IJCAI ProceedingsInternational Joint Conference on Artificial Intelligence; vol. 22. 2011. p. 1237. [35] Khan AM, Rajpoot N, Treanor D, Magee D. A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution. IEEE Trans Biomed Eng 2014;61(6):1729–38. [36] Beevi S, Nair MS, Bindu G. Automatic segmentation of cell nuclei using krill herd optimization based multithresholding and localized active contour model. Biocybern Biomed Eng 2016;36(4):584–96. [37] Vedaldi A, Lenc K. Matconvnet-convolutional neural networks for matlab. corr abs/1412.4564; 2014. [38] Sharif Razavian SC. Azizpour, Cnn features off-the-shelf: an astounding baseline for recognition. Computer Vision and Pattern Recognition Workshops, 2014; 2014. pp. 806–13. [39] Jolliffe I. Principal component analysis for special types of data. Principal Component Analysis. 2002;338–72. [40] Blum AL, Langley P. Selection of relevant features and examples in machine learning. Artif Intell 1997;97(1):245– 71.
634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
Available online at www.sciencedirect.com
ScienceDirect journal homepage: www.elsevier.com/locate/bbe 1 2 3
Original Research Article
Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning
4 5 6
7 8 9 13 10
Q1
Beevi K. Sabeena a,*, Madhu S. Nair b, G.R. Bindu c a
Electrical & Electronics Department, T. K. M College of Engineering, Kollam, Kerala, India Department of Computer Science, Cochin University of Science and Technology, Kochi 682022, Kerala, India c Electrical Engineering Department, College of Engineering Wayanad, Wayanad 695016, Kerala, India b
11 12 14 15 16 17 16 18 17 16 19 18 17 16 20 19 18 17 16 21 20 19 18 17 16 22 21 20 19 18 17 16 23 22 21 20 19 18 17 16 24 23 22 21 20 19 18 17 16 25 24 23 22 21 20 19 18 17 16 26 25 24 23 22 21 20 19 18 17 16 27 26 25 24 23 22 21 20 19 18 17 16 28 27 26 25 24 23 22 21 20 19 18 17 16 29 28 27 26 25 24 23 22 21 20 19 18 17 16 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21
37 38 39
article info
abstract
Article history:
The exact measure of mitotic count is one of the crucial parameters in breast cancer grading
Received 15 May 2018
and prognosis. Detection of mitosis in standard H & E stained histopathology images is
Received in revised form
challenging due to diffused intensities along object boundaries and shape variation in
23 October 2018
different stages of mitosis. This paper explores the feasibility of transfer learning for mitosis
Accepted 28 October 2018
detection. A pre-trained Convolutional Neural Network is transformed by coupling random
Available online xxx
forest classifier with the initial fully connected layers to extract discriminant features from nuclei patches and to precisely prognosticate the class label of cell nuclei. The modified
Keywords:
Convolutional Neural Network accurately classify the detected cell nuclei with limited
Histopathology
training data. The designed framework accomplishes higher classification accuracy by
Mitosis
carefully fine tuning the pre-trained model and pre-processing the extracted features.
Krill herd optimization
Moreover, proposed method is evaluated on MITOS dataset provided for the MITOS-ATYPIA
Convolutional Neural Network
contest 2014 and clinical data set from Regional Cancer Centre, Thiruvananthapuram, India.
Transfer learning
Significance of Convolutional Neural Network based method is justified by comparing with
Random forest
recently reported works including a Multi Classifier System based on Deep Belief Network.
Q3 Multi Classifier System
Experiments show that the pre-trained Convolutional Neural Network model outperforms conventionally used detection systems and provides at least 15% improvement in F-score on other state-of-the-art techniques.
Q2
© 2018 Published by Elsevier B.V. on behalf of Nalecz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences.
1.
Introduction
Q4 Introducing digital slide libraries together with computer-
aided diagnosis (CAD) brought radical changes in the analysis of pathology images [1]. Among the grading factors for breast
cancer, the mitotic count is a significant characteristic of tumor proliferation [2]. Mitosis, a complex biological process, appears as hyper chromatic objects with pseudo projections around the edges. Shape of a nucleus varies through different stages [3] of mitosis such as prophase, metaphase, anaphase and telophase as shown in Fig. 1. Moreover, two divided nuclei
* Corresponding author at: Electrical & Electronics Department, T. K. M College of Engineering, Kollam, Kerala, India. E-mail addresses: [email protected] (B.K. Sabeena), [email protected] (M.S. Nair), [email protected] (G.R. Bindu). https://doi.org/10.1016/j.bbe.2018.10.007 0208-5216/© 2018 Published by Elsevier B.V. on behalf of Nalecz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences. Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
40 41 42 43 44 45
BBE 315 1–10
2
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
Fig. 1 – Cell in different phases of mitosis.
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92
in the telophase have to count as single mitosis because they are not distinct cells. The presence of lymphocytes, cells with partial nuclei, feeble edges, similar grey level of different soft tissues and shape variation among cells complicates the detection process with many false positives. In addition, a low density of nuclei undergoing mitosis makes the classification process forbiddingly challenging [4]. The already reported techniques in the literature exploit cellular features that identify morphology and intensity of mitotic nuclei. But due to shape variation and indistinguishable nuclei features, such sort of systems result in reduced detection accuracy with a large number of false positives or wrong detections. The deep learning approach uses a deep architecture such as Convolutional Neural Network (CNN) to learn proper features from the images [5]. Training of CNN for a real world problem requires excessive computations with large samples of annotated images. In [6], Ciresan et al. employed one million nuclei to train CNN which resulted an FScore of 78%. But it has to be improved further for clinical practice. Even with GPU, training time takes hours and weeks depending on the problem. Since interpretation of biomedical data requires medical expertise and is bound by diagnostic variability, the field of medical imaging lacks availability of large volumes of labelled data (Ground Truth). This paper proposes an alternate method to further improve the accuracy of mitosis detection using transfer learning technique by incorporating minimal variation in the architecture of CNN. Transfer learning is a concept where weights of a pre-trained model is transferred to another problem set on a different dataset. A pre-trained model is an already trained model created to solve a similar problem. The key idea is that information coming from the source tasks to the target task may be useful to speed-up the learning process. Based on the nature of dataset it is essential to make modifications in the pre-existing model by fine-tuning the network. In [7], Yosinski et al. enumerated the transferability of the learned representations to different domains and also established reuse of the parameters in the network for different tasks. Recently, these methods achieved acceptance in medical imaging applications [8]. This work investigates the application of such methods on mitosis dataset. The proposed approach presents an automatic mitosis detection to tackle the preceding challenges and outperforms the state-of-the-art techniques in mitosis detection. Traditional classification systems include cell segmentation, feature extraction and classification as major steps. Break up of nuclear membrane in the early stages of mitosis leads to
diffusion of nuclei and background regions. Hence, it is difficult to find a clear threshold to realize cell nuclei in images. Krill Herd Algorithm (KHA) [9] based optimal multithresholding enables quick extraction of nuclei patches from the huge pathology images by locating centroids of the nuclei regions. For good classification, the specific changes in nuclei during mitosis are to be computed as discriminant features. The intensity based and morphology based features are unable to provide variations in nuclei during mitosis. The proposed methodology looks for extracting generic low-level features and complex features from the localized cells by a very deep convolutional network Caffe model-VGGNet [10,11], which is tuned to extracts nuclei variations in images as features by successively modeling tiny information and combine them deeper in the network. Deeper networks give better results, hence the name deep transfer learning. The strength of different classifiers are exploited to deal with complex high dimensional features where the number of features are larger than the number of instances. Visualization of features further confirm reliable distinction between mitotic and non-mitotic nuclei. Hence, a strong feature representation with a strong classifier is achieved for mitosis detection. Further, the algorithm is tested on a clinical data set obtained from Regional Cancer Centre (RCC), Thiruvananthapuram, India, along with a very challenging MITOS dataset [12] to establish the effectiveness of the proposed technique. The transfer learning technique needs only moderate computational resources and less number of labelled data. Moreover, we compare the performance of the proposed technique with recently reported techniques [5,13] including a Multi Classifier System (MCS) based on Deep Belief Networks (DBN) [14] that has been used the same dataset for detecting mitosis. The discriminant features from the pre-trained CNN provide significant reduction in false positive rate and overcome the need for large labelled data. The rest of the paper is organized such that Section 2 presents relevant literature related to this work. Section 3 briefly introduces CNN model and Section 4 describes the proposed method. The Section 5 discusses experimental results and the conclusions are drawn in Section 6.
93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132
2.
133
Review of related literature
Compared to cytopathology images where nuclei are well separated, the analysis of nuclei in histopathological images is more difficult due to the complex and irregular visual
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
134 135 136
BBE 315 1–10
3
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196
appearance of the nuclei. Over two decades ago, initial methods for mitosis detection were reported with a semiautomatic algorithm in Feulgen stained breast cancer sections [15]. Probability based likelihood functions along with binary thresholding is used in [16] to identify mitotic nuclei in neuro images. Irshad et al. [17] employed thresholding of blue ratio images to detect nuclei regions and RF classifier for mitotic differentiation. Basic thresholding yields a low detection rate owing to diffusion of nuclei and background regions. Khan et al. [18] proposed a gamma-gaussian mixture model by isolating tumour regions from non-tumour areas for detection of mitotic nuclei. However, a context based tuning is required in the classification stage to reduce the number of false positives. A graph-based multi-resolution algorithm [19], extracts mitotic figures by clustering domain specific features. Anari et al. [20] applied Fuzzy C-means (FCM) clustering along with an ultra-erosion operation for detection of the mitosis index in Immuno Histo Chemical (IHC) images. For large histopathology images, clustering algorithms are very slow. In [21], many false positives were detected by hysteresis thresholding based morphological reconstruction operations in breast histology images. In [22], texture and SIFT features along with intensity, morphology, and run-length features from different color models were used for the mitosis prediction. Many redundant feature values from different colour channels result in a low classification outcome. In [23], nuclei classification was dealt with color based features. But use of chromatic filters influence performance of the Adaboost Classifier. In [24], a localized active contour model based segmentation is applied along with random features for detecting mitosis. The authors utilized Gabor features and texture features for automatic grading of breast cancer in [25]. But the similarity in shape and color attributes of other cell structures cause wrong detections. Lu and Mandal [26], applied a multi-expert system for classification of mitotic cells in multispectral images. The authors selected the spectral bands using linear discriminant analysis. Segmentation of the nuclei regions were carried out by Bayesian modeling followed by local region thresholding. A recent work [13] reported mitosis detection with an average F-score of 73% based on intensity features and morphological filtering. Ciresan et al. first utilized CNN based deep max-pooling for mitosis detection in breast histology images [6]. In [27], Wang et al. computed CNN features along with handcrafted features to detect mitosis. The authors utilized local thresholding of blue ratio image for initial segmentation. For two feature sets, independent classifiers were used and a third classifier is used for the feature combination. In [28], whose work earlier inspired us, the transfer learning facility of CNN is investigated to interpret the leukemia cell lines in cytopathology images and also in mitosis detection [29], but they require manual intervention in pre-processing and classification stages. In [30], initially pixel classification is done based on handcrafted features. Then a multi-stage CNN is employed for mitosis prediction by which the authors claim for accuracy and timing. The different kernel maps in CNN looks for local features and share the parameters along the depth volume. Training CNN requires many labelled samples and parameters that increase the computational complexity of the algorithm [5,31,32]. Since most of these features are fundamental characteristics for any
image data set, kernels can be used as a feature extractor for the other classification challenges. Wan et al. [33] extracted multi-level features by a CNN after segmentation using a hybrid active contour model. Further, multiple support vector machine (SVM) classifiers are utilized for classification. Robust identification and measurement of naturally occurring deformable structures is not always achievable using a single technique due to insufficient samples, presence of artifacts and conflicting range of inputs in biomedical domain. In this paper, we propose a hybrid technique, which maximizes entropy of nuclei regions by the Krill Herd algorithm and utilizes the effective feature extraction capability of CNN through transfer learning. The proposed CNN model provides improved accuracy over a wide range of data set, compared with other recently reported works.
197 198 199 200 201 202 203 204 205 206 207 208 209 210 211
3.
Convolutional Neural Network (CNN)
212
Consider the CNNs as a sequence of feed-forward neural network functions that extract features at multiple levels or layers [34]. At each layer, it computes features from the previous layer representations (from low-level to high-level). Internally, it comprises a group of functions organized as
213 214 215 216 217 218
gðxÞ ¼ gL ð. . .g2 ðg1 ðx; w1 Þ; w2 Þ. . .Þ; wL Þ
(1)
where x and w are the input vector and weight vector, respectively. Varying layers ‘ in CNNs perform fundamental operations, such as
Convolution with a group of filters Normalization Non-linear activation Sub-sampling or Pooling
The five main types of layers to build CNN architecture are briefly explained below: The convolutional layer carries out convolution to capture highly redundant and highly correlated local information present in the input. For a given image X, the convolution block performs the convolution of the image with a bank of K filter maps, f ðw; hÞ, with height h and width w. Each filter computes a dot product between their weights and a definite region it links to in the input image. Normalization layer regulates output from the early stages and passes to the next non linear activation layer. This process makes the weights from the network balanced without very high or low values, since it includes normalization in the gradient process. Rectified Linear Unit (ReLU) performs an element wise thresholding. It is the most commonly used activation function for CNNs. Formally, it is characterised as:
219 220 221 222 223 224 225 226 226 228 227 227 230 229 228 232 231 229 230 233 231 234 232 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249
f ðxÞ ¼ maxðx; 0Þ
(2)
CNNs with ReLUs train many times quicker than their equivalents with tanh functions. The number of complex features increase, as the network grows deeper and deeper.
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
250 251 252 253 254
BBE 315 1–10
4 255 256 257 258 259 260 261
262 263 265 264 265 267 266 266 268 267 269
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
However, the feature map size is being shrunk through the pooling operation by taking maximum activations. It is a way of down sampling the image space along the spatial dimensions. As in common Neural Networks, it connects each neuron in Fully-Connected Layer (FCL) to all neurons in the previous layer and calculates class counts.
4.
Methodology
The proposed method comprises of mainly two phases such as Phase I: Pre-processing and cell localization Phase II: Classification of cells
4.1.
Pre-processing and cell localization
270 271 272 273 274 275 276 277 278 279
Tissue samples of the MITOS data set [12] and clinical data set show significant stain variation. Hence, stain normalization is carried out by a recent color deconvolution technique [35]. KHA based optimal multi thresholding technique provides the exact centroid location of individual nuclei as explained in [36]. Localized cell patches of size 25 25 pixels, enclosing a nucleus at the center are inputs to the CNN that extracts discriminant features. The use of tiny patches adds the volume of total training data and confines the analysis to actual nuclei in images.
280
4.2.
281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297
The texture, size and shapes of the mitotic cells keep on changing in different stages of mitosis. So, this work uses the deep feature representation capability of CNN. Amid many choices, we use pre-trained CNN model VGGNet in Matlab [37], which is heavily trained on challenging images of 1000 categories of images. VGGNet also shows that the depth of the network plays an important role. It contains 9 learned layers including convolutional (C) and pooling layers (P) of particular specifications. The first layer of VGGNet architecture consists of a convolutional layer with 64 filters of 11 11. A ReLU activation function and a normalization of 5 5 are also present. Layer 2 to 5 consist of convolutional layers with 256 filters of size 3 3 and 5 5. Throughout the network pooling layers perform 3 3 maximum pooling. The R, N and FC represent ReLU, normalization and fully connected layers. The filter size of 11 11, 5 5 and 3 3 with stride of 1 and padding of 1 are
Classification of cells
carefully chosen to retain the feature map after convolution. The max pooling layer decreases the feature map size all the way to the end of the network. At the very end fully connected layers and softmax layers are added for the final classification. Table 1 shows parameters of the pre-trained CNN model in Matlab. Transfer learning is an approach where the weights of a previously developed model is transferred to another problem on a specific dataset. The network VGGNet already has trained weights on ImageNet dataset. We retrain the network on the new dataset such as, the cell patches from the MITOS and RCC datasets. Depending on the number of outputs for the new problem, the last layer should substitute with another one of our choice. Here, we have data for mitosis and non-mitosis cases. Hence, the last layer is replaced with a 2-node softmax layer. To do this, we removed the fully-connected layer at the top of the network and then added new fully connected layer with random initialization, with two output nodes. The network is retrained on the new dataset such as, the cell patches from the MITOS and RCC datasets for two classes. Since in this case size of the data is small as well as data similarity is very low, the whole process ended up with poor classification outcomes of 30% for the first two data sets. Hence as a second stage, we go for fine tuning the network to make it as a feature extractor. Each layer in a CNN successively builds up higher and stronger level of details. The earlier layers are often more generic as there are many simple patterns common among images. In addition, the last layers are very specialized on the input data supplied to the model. Since the network already learned so much about edges, curves and objects from the ImageNet dataset, it can relate them to the newer dataset. The weights of the first three layers of the pre-trained network are freezed to keep the curves and edges that are relevant to object detection task. This is done by providing low values for learning rate to the corresponding layers. To learn the specific features related to nuclei patches we again retrained the later four layers with nuclei dataset. Since the intermediate layers of CNN are not too meticulous to the originally trained dataset, they apprehend sufficient descriptions for transfer-learning studies [38]. Since the output layer of the network is too task-specific to the dataset, the network was originally trained on; it cannot be used for the final classification. Accordingly, we replace the last Softmax layer with a Random Forest classifier at the output of the network. The RF classifier is taught with the activations from the fully connected layers. The fully connected layer abstracts feature list from convolution layers by blending together features detected from the image patches for this particular
Table 1 – Architecture of pre-trained CNN model in Matlab. Layer 1 1 1 1 2 2 2
Type
Filter Dim.
Layer
Type
Filter Dim.
Layer
Type
Filter Dim.
Conv1 ReLU1 Norm1 Pool1 Conv2 ReLU2 Norm2
11 11 3 64
2 3 3 4 4 5 5
Pool2 Conv3 ReLU3 Conv4 ReLU4 Conv5 ReLU5
Max 3 3 256 256
5 6 6 7 7 8 9
Pool5 FC6(Conv) ReLU6 FC7(Conv) ReLU7 FC8(Conv) Softmax
Max 6 6 256 4096
Max 5 5 64 256
3 3 256 256 3 3 256 256
1 1 4096 4096 1 1 4096 1000
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345
BBE 315 1–10 biocybernetics and biomedical engineering xxx (2018) xxx–xxx
346 347 348 349 350 351 352 353 354 355
task. Essentially, now the network act as an automatic feature extractor. Here features are extracted from the 7th layer before the last fully connected Softmax layer. The extracted feature vector takes the same length of 4096 as that of sixth and seventh layers of VGGNet. Principal component analysis (PCA) [39] reduces the depth of these vectors from 4096 to 650. PCA is a method that can be applied for dimensionality reduction. The prime objective of PCA is to identify a compact depiction of the data, under the assumption that the inputvariables are corresponded with each other and thus redun-
5
dant. The focus is to define the initial high-dimensional data using a lower dimensional subspace. Geometrically, this can be viewed as a smooth estimate of the original data into a lower-dimensional scheme to preserve as much information as possible. The main axes of this new coordinate structure are called principal components. A classifier subset evaluator further selects a feature vector of size 61 as a subset that gives the best discriminant information by a hill climbing search [40]. The algorithm works in such a way that it trains a classifier model for each
Fig. 2 – Probability density plots and box plots for first 4 (x1–x4) nuclei features. (a) Probability distribution of mitosis (blue) and non mitosis (pink) nuclei features. (b) Boxplot of mitosis (1) and non mitosis (0) nuclei features.
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
356 357 358 359 360 361 362 363 364 365
BBE 315 1–10
6
366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
Table 2 – Description of dataset with total number of images (TNI) and total number of mitosis (TNM).
5.
Experimental results
Data set
A03
A04
A07
A10
Clinical data
5.1.
Data set
TNI TNM
96 128
86 218
64 25
80 39
53 45
new subset and a score is computed based on the errors. The subset with the maximum score is selected as the final feature subset. To assess the level of feature representation from inner layers of CNN and to understand the relationships between the extracted features of mitosis and non-mitosis nuclei, box plots and probability density plots were taken that give smooth lines for each distribution. Fig. 2(a) shows probability distribution of mitosis (blue) and non-mitosis (pink) nuclei and Fig. 2(b) displays box plot of mitosis (1) and non-mitosis (0) nuclei for first four (x1–x4) features. Both plots show significant differentiation between mitosis and non-mitosis clusters that capture adequate content for distinguishing between the two classes. Four traditional classifiers such as Support Vector Machines (SVM), Linear Discriminant Analysis (LDA), Random Forest (RF) and k-Nearest Neighbours (kNN) train the extracted features for classification. This is a suitable fusion of simple linear (LDA), composite nonlinear (SVM, RF) and nonlinear (kNN) methods. The highest accuracy is realized by RF and is used to label the mitotic and nonmitotic cell.
387 388
The algorithm is validated on the MITOS data set [12] and also on clinical data set from the RCC, Thiruvananthapuram, India. Two slide scanners such as Aperio (AP) and Hamamatsu (HM) scan tissue slides at 40 magnification with a resolution of 0.23–0.24 mm per pixel. The mitoses are recorded by expectation values of 1.0, 0.8, 0.6, 0.2 or 0. We carry out our experiments using their ‘‘in-group’’ dataset, which is the data set for which they furnish ground truth for comparison. Images from RCC are obtained by the Leica digital image acquisition system attached to the microscope. Images in the MITOS data set are of 1376 1539 3 size and RCC images are of 3264 2448 3 size. Two experienced pathologists manually assess mitotic nuclei in clinical images. The MITOS data set as well as RCC dataset comprises high power field (HPF) images of complex breast tissue stained with Hematoxylin and Eosin (H&E). Table 2 describes the different datasets, A03, A04, A07, A10 (folder names of MITOS data set with Aperio scanner) and RCC data set in terms of total number of images (TNI) and total number of mitosis (TNM). There were 5-10 mitotic cells in the telophase and each of them were taken as a single cell patch. From the four folders of ICPR 2014 dataset, mitotic regions with probability more than 0.6 were incorporated as mitotic in our training set. To treat class disparity and produce rotational invariance, nuclei patches containing
Fig. 3 – (a) Sample image with stain variation (b) Reference image (c) Stain normalized image with color deconvolution method.
Fig. 4 – Images with centroid of detected nuclei. (a) Normalized R component of original image (b) centroids located by normal thresholding, where big red circles indicate wrong centroid locations for adjacent nuclei (c) KHA optimized image, here the Q5 red circular markings represent correct centroids of the adjacent nuclei. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412
BBE 315 1–10
7
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
Table 3 – Comparison of cross validation accuracy. No.of features 4096 61
Fig. 5 – Plot of the model evaluation results.
413 414 415 416 417 418 419 420 421
mitotic nuclei were replicated with rotations. In order to cut down the skewness of data, careful sampling of non-mitotic data and random up sampling of mitotic data are further carried out. From all the data sets considered, 70% of nuclei are taken for training and remaining 30% are taken for testing. Every nucleus in the training set is marked as either mitotic or non-mitotic based on the available ground truth and test set is prepared with an equal number of mitosis and non-mitosis nuclei. The experiments were carried out by using Matlab 2016 environment.
422
5.2.
423 424 425
To assess the mitosis classification, the proposed framework use Sensitivity, Precision and F-score as given in Eqs. (3)–(5) along with Specificity and Accuracy.
Performance measures
426 Sensitivity ¼
Ntp 100 Ntp þ Nfn
(3)
428 427 429 Precision ¼
Ntp 100 Ntp þ Nfp
(4)
431 430 432 Fscore ¼ 2 433 434 435 436 437
SensitivityPrecision 100 Sensitivity þ Precision
(5)
where Ntp is the number of true positives (tp-correctly detected Mitosis), Nfp number of false positives (fp-wrongly detected Mitosis) and Nfn number of false negatives (fn-missed mitosis).
K = 10
K=5
0.86 0.94
0.84 0.91
corresponding normalized image along with a reference image. Three thresholds such as T1, T2 and T3 produced by KHA based optimal multi thresholding distinguishes cell nuclei from other cell structures. Fig. 4 displays the effect of KHA on the localization of cell nuclei. Normal thresholding (Fig. 4(b)) did not identify centroids of many adjoining nuclei, but they have been properly discriminated by the KHA based optimal thresholding (Fig. 4(c)). Hence, it helps in reducing the number of false positive and false negative nuclear centroids. The computational power of the KHA based multilevel thresholding is illustrated in [36], in which the KHA performs 15–20 times faster than the Bacterial Foraging Algorithm with 94.31% detection sensitivity.
441 442 443 444 445 446 447 448 449 450 451 452
5.4.
453
Classification
Since the sixth and seventh layers of VGGNet consist of 4096 neurons (fully connected layers) with the ReLU activation function, the extracted feature vector has the same dimension of 4096. Repeated cross validation is carried out using the linear and nonlinear classifier models to create a best classifier. We reorganize the random number seed in advance to guarantee the same data splits in each algorithm. In each experiment, training data from all category is divided into K segments, each holding the same number of cells. It also confirms that the results are comparable. The mean accuracy and kappa value is computed. The RF classifier showed the highest accuracy as shown in Fig. 5. The plot relates the spread and the mean accuracy of each model. At first, training is done with all computed features. The cross-validation accuracy, averaged over five runs performed with all features and selected features, is reported in Table 3. The RF classifier trained with the preferred feature set is employed to recognize other unknown instances of mitosis from the evaluation set. The chosen subset of features offered consistent values for all the data sets as shown in Table 4 and resulted in an average value of 88.6% and 89.66% F-score for Mitos dataset and RCC dataset respectively as shown in Table 5. The slide images from Mitotic dataset consist of tissue artefacts such as, tissue curling and many mitotic-like non mitosis regions. A07 and A10 folders contain comparatively less no. of mitotic nuclei. Due to these variations, there is reduction in detection performance in certain images of A10 and A07 slides. Moreover, it consists of many atypical nuclei. Fig. 6 shows visual results of the proposed algorithm in which, Fig. 6(a)
Table 4 – Performance measures on each MITOS dataset. Dataset Sensitivity Specificity Accuracy Precision F-Score
438 439 440
5.3.
Pre-processing and cell localization
Stain normalization increases the contrast between cell nuclei and other cell structures. Fig. 3 presents a stain varied image and
A03 A04 A07 A10
92.31 97.5 81.82 88.88
90.77 87.5 90.91 77.77
91.54 92.5 86.36 83.33
90.91 88.64 90 80
91.60 92.86 85.71 84.21
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482
BBE 315 1–10
8
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
Table 5 – Average on each dataset. Dataset Average on Mitos Data set RCC dataset
Sensitivity
Specificity
Accuracy
Precision
F-Score
90.13 86.67
86.74 93.33
88.43 90
87.39 92.86
88.60 89.66
Fig. 6 – Visual results. (a) Original image (b) nuclei located by KHA (c) Classification results (nuclei shown in Red circles: tp, Yellow circles: fp, Blue circles: fn). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Table 6 – Comparison with MCS. Classifier
Sensitivity
Precision
F-Score
DBN-MCS Proposed
93.33 90.13
75.38 87.39
83.4 88.6
Fig. 7 – Plot of the discriminant nature of CNN features compared with that of morphometric features in [14]. A clear discrimination exist between mitosis (blue) and non-mitosis (pink) features extracted by pre-trained CNN. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
BBE 315 1–10 biocybernetics and biomedical engineering xxx (2018) xxx–xxx
9
images. To realize each attribute and to understand the correlation between them, univariate and multivariate plots of the selected features are further analysed. A wide dataset (a challenging standard dataset and a clinical data set from a cancer research institute, RCC, Thiruvananthapuram, India is used to test the proposed method. The F-Score is increased from 0.86 to 0.94 after dimensionality reduction and sequential feature selection in the training data. The results prove that the proposed framework surpasses the existing techniques with high sensitivity and precision that makes it more practical in clinical diagnosis.
521 522 523 524 525 526 527 528 529 530 531 532
Fig. 8 – F-score of the proposed method with MCS and recently reported methods [5] and [13].
483 484 485 486 487
presents the original image (b) displays the nuclei locations provided by KHA based multi thresholding and (c) Classification results: nuclei in red circles (tp). Nucleus in the blue circle shows the missed mitosis (fn) and nucleus shown in a yellow circle presents false mitosis (fp).
488
5.5.
489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511
Table 6 presents the classification using the proposed method and the multi classifier system (DBN-MCS) as explained in [14], where the classification is based on cellular features extracted after accurate segmentation by Localized Active Contour Model (LACM). Fig. 7 displays the plot of discriminant nature of three sets of CNN features and morphometric features. The morphometric features of mitotic (blue) and non-mitotic nuclei (pink) are tough to differentiate compared to corresponding CNN features. A significant reduction in false positive rate is apparent from the enhanced precision values given by the proposed model with a slight reduction of sensitivity. However, it results in direct improvement of 5.4% in F-score, which is more significant. Fig. 8 compares the proposed method with two recently published techniques [5,13] who analysed the algorithm using the 2014 ICPR Mitosis training and testing dataset. Angshuman et al. [13] carried out mitosis detection using hand crafted features and Chen et al. [5] evaluated it by cascaded deep networks. The methods [5] and [13] were evaluated with test set from ICPR Mitos database while the proposed method has test set from both Mitos and RCC data set. With the proposed CNN based model, the classification becomes more robust with significant progress in precision and F-score.
512
6.
513 514 515 516 517 518 519 520
Since mitosis is a complex biological process, conventional approaches cannot accurately detect them. In this study, automated mitosis detection is carried out using deep features extracted by VGGNet, caffe model of CNN trained on the popular imaging database ImageNet. This overcomes the need for large labelled data and generates discriminative features for the correct classification. The use of localized nuclei patches by KHA confines the analysis to small nuclei in
533
Acknowledgment
534
The authors acknowledge the support rendered for this work by RCC Thiruvananthapuram, India. The authors also thank Dr. Sujathan, Dr. Jayasree, Dr. Abitha and Dr. Anju for their timely guidance and help in the evaluation process.
535 536 537 538
references
539
[1] Malon CD, Cosatto E, et al. Classification of mitotic figures with convolutional neural networks and seeded blob features. J Pathol Inform 2013;4(1):9. [2] Elston E, Ellis I. Method for grading breast cancer. J Clin Pathol 1993;46(2):189. [3] Barlow P. Changes in chromatin structure during the mitotic cycle. Protoplasma 1977;91(2):207–11. [4] Dundar MM, Badve S, Bilgin G, Raykar V, Jain R, Sertel O, et al. Computerized classification of intraductal breast lesions using histopathological images. IEEE Trans Biomed Eng 2011;58(7):1977–84. [5] Chen H, Dou Q, Wang X, Qin J, Heng PA. Mitosis detection in breast cancer histology images via deep cascaded networks. Thirtieth AAAI Conference on Artificial Intelligence; 2016. [6] Cireşan DC, Giusti A, Gambardella LM, Schmidhuber J. Mitosis detection in breast cancer histology images with deep neural networks. International Conference on Medical Image Computing and Computer-assisted Intervention. Springer; 2013. p. 411–8. [7] Yosinski J, Clune J, Bengio Y, Lipson H. How transferable are features in deep neural networks?Advances in neural information processing systems. 2014;3320–8. [8] Xu Y, Mo T, Feng Q, Zhong P, Lai M, Eric I, et al. Deep learning of feature representation with multiple instance learning for medical image analysis. 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2014. pp. 1626–30. [9] Gandomi AH, Alavi AH. Krill herd: a new bio-inspired optimization algorithm. Commun Nonlinear Sci Numer Simul 2012;17(12):4831–45. [10] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556. [11] Jia Y, et al. Caffe: an open source convolutional architecture for fast feature embedding; 2013. [12] MITOS, ICPR 2014 Contest. IPAL UMI CNRS Lab Std; 2014, http://ipal.cnrs.fr/ICPR. [13] Angshuman P, Mukherjee DP. Mitosis detection for invasive breast cancer grading in histopathological images. IEEE Trans Image Process 2015;24(11):4041–54.
540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579
Comparison
Conclusion
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007
BBE 315 1–10
10 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633
biocybernetics and biomedical engineering xxx (2018) xxx–xxx
[14] Beevi KS, Nair MS, Bindu G. A multi-classifier system for automatic mitosis detection in breast histopathology images using deep belief networks. IEEE J Transl Eng Health Med 2017;5:1–11. [15] Belien J, Baak J, van Diest PJ, Van Ginkel A. Counting mitoses by image processing in Feulgen stained breast cancer sections: the influence of resolution. Cytometry Part A 1997;28(2):135–40. [16] Sertel HSO, Catalyurek UV, Guican MN. Computeraided prognosis of neuroblastoma: detection of mitosis and karyorrhexis cells in digitized histological images. Int. Conf. IEEE Eng. Med. Biol. Soc. (EMBC); 2009. pp. 1433–6. [17] Irshad H, Roux L, Racoceanu D. Multi-channels statistical and morphological features based mitosis detection in breast cancer histopathology. 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); 2013. pp. 6091–4. [18] Khan AM, El-Daly H, Rajpoot NM. A gamma-gaussian mixture model for detection of mitotic cells in breast cancer histopathology images. 2012 21st International Conference on Pattern Recognition (ICPR); 2012. pp. 149–52. [19] Roullier V, Lézoray O, Ta V-T, Elmoataz A. Multi-resolution graph-based analysis of histopathological whole slide images: application to mitotic cell extraction and visualization. Comput Med Imaging Graphics 2011;35 (7):603–15. [20] Anari V, Mahzouni P, Amirfattahi R. Computer-aided detection of proliferative cells and mitosis index in immunohistichemically images of meningioma. 2010 6th Iranian Conference on Machine Vision and Image Processing; 2010. pp. 1–5. [21] Beevi S, Nair MS, Bindu G. Automatic segmentation and classification of mitotic cell nuclei in histopathology images based on active contour model. 2014 International Conference on Contemporary Computing and Informatics (IC3I); 2014. pp. 740–4. [22] Irshad H, Jalali S, Roux L, Racoceanu D, Hwee LJ, Le Naour G, et al. Automated mitosis detection using texture, sift features and hmax biologically inspired approach. J Pathol Inform 2013;4(2):12. [23] Tek FB, et al. Mitosis detection using generic features and an ensemble of cascade adaboosts. J Pathol Inform 2013;4 (1):12. [24] Beevi KS, Nair MS, Bindu G. Detection of mitotic nuclei in breast histopathology images using localized acm and random kitchen sink based classifier. 2016 IEEE 38th Annual International Conference of the Engineering in Medicine and Biology Society (EMBC); 2016. pp. 2435–9. [25] Doyle S, Agner S, Madabhushi A, Feldman M, Tomaszewski J. Automated grading of breast cancer histopathology using spectral clustering with textural and architectural image features. 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro, 2008. ISBI 2008; 2008. pp. 496–9.
[26] Lu C, Mandal M. Toward automatic mitotic cell detection and segmentation in multispectral histopathological images. IEEE J Biomed Health Inform 2014;18(2):594–605. [27] Wang H, Cruz-Roa A, Basavanhally A, Gilmore H, Shih N, Feldman M, et al. Cascaded ensemble of convolutional neural networks and handcrafted features for mitosis detection. SPIE medical imaging. International Society for Optics and Photonics; 2014. p. 90410B. [28] Gopakumar G, Babu KH, Mishra D, Gorthi SS, Subrahmanyam GRS. Cytopathological image analysis using deep-learning networks in microfluidic microscopy. JOSA A 2017;34(1):111–21. [29] Albayrak A, Bilgin G. Mitosis detection using convolutional neural network based features. 2016 IEEE 17th International Symposium on Computational Intelligence and Informatics (CINTI); 2016. pp. 000335–40. [30] Li Y, Mercan E, Knezevitch S, Elmore JG, Shapiro LG. Efficient and accurate mitosis detection-a lightweight rcnn approach. ICPRAM. 2018. pp. 69–77. [31] Malon CD, Cosatto E, et al. Classification of mitotic figures with convolutional neural networks and seeded blob features. J Pathol Inform 2013;4(1):9. [32] Sirinukunwattana K, Raza SEA, Tsang Y-W, Snead DR, Cree IA, Rajpoot NM. Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans Med Imaging 2016;35 (5):1196–206. [33] Wan T, Cao J, Chen J, Qin Z. Automated grading of breast cancer histopathology using cascaded ensemble with combination of multi-level image features. Neurocomputing 2017;229:34–44. [34] Ciresan DC, Meier U, Masci J, Maria Gambardella L, Schmidhuber J. Flexible, high performance convolutional neural networks for image classification.IJCAI ProceedingsInternational Joint Conference on Artificial Intelligence; vol. 22. 2011. p. 1237. [35] Khan AM, Rajpoot N, Treanor D, Magee D. A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution. IEEE Trans Biomed Eng 2014;61(6):1729–38. [36] Beevi S, Nair MS, Bindu G. Automatic segmentation of cell nuclei using krill herd optimization based multithresholding and localized active contour model. Biocybern Biomed Eng 2016;36(4):584–96. [37] Vedaldi A, Lenc K. Matconvnet-convolutional neural networks for matlab. corr abs/1412.4564; 2014. [38] Sharif Razavian SC. Azizpour, Cnn features off-the-shelf: an astounding baseline for recognition. Computer Vision and Pattern Recognition Workshops, 2014; 2014. pp. 806–13. [39] Jolliffe I. Principal component analysis for special types of data. Principal Component Analysis. 2002;338–72. [40] Blum AL, Langley P. Selection of relevant features and examples in machine learning. Artif Intell 1997;97(1):245– 71.
634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688
Please cite this article in press as: Sabeena BK, et al. Automatic mitosis detection in breast histopathology images using Convolutional Neural Network based deep transfer learning. Biocybern Biomed Eng (2018), https://doi.org/10.1016/j.bbe.2018.10.007