Locality-constrained group lasso coding for microvessel image classification

ARTICLE IN PRESS

JID: PATREC

[m5G;February 15, 2019;11:34]

Pattern Recognition Letters xxx (xxxx) xxx

Contents lists available at ScienceDirect

Pattern Recognition Letters journal homepage: www.elsevier.com/locate/patrec

Locality-constrained group lasso coding for microvessel image classification Juan Chen a, Shijie Zhou b, Zhao Kang a, Quan Wen a,∗ a b

School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, 611731, China Glasgow College, University of Electronic Science and Technology of China, Chengdu, Sichuan, 611731, China

a r t i c l e

i n f o

Article history: Available online xxx Keywords: Image classification Group sparse coding Microvessel

a b s t r a c t Image-based classification of histology sections plays an important role in predicting clinical outcomes. In this paper, we propose a Locality-Constrained Group Lasso Coding (LCGLC) method for microvessel image classification, which realizes the automatic “hot spot” detection of angiogenesis for human liver carcinoma. First, we extract Scale-Invariant Feature Transform (SIFT) descriptors on the Single-Opponent (SO) feature map, which simulates the biological functionality of human visual systems. Then, we present the feature-biased dictionary learning to effectively generate the dictionary of SIFT descriptors. With the learned dictionary, our LCGLC method introduces the locality constraint in classical group lasso problem to encode SIFT descriptors. Furthermore, we apply the Spatial Pyramid Matching (SPM) for the code pooling of microvessel images. Finally, we use Support Vector Machine (SVM) to classify a tissue image as having angiogenesis or not. Comprehensive experiments on the microvessel dataset show that the proposed LCGLC method achieves better performance compared with other representative approaches. © 2019 Elsevier B.V. All rights reserved.

1. Introduction Blood vessel is composed of two types of interacting cells, namely endothelial cells and perivascular cells (also known as pericytes). Endothelial cells form the inner lining of the vessel wall, while pericytes envelop the surface of the vascular tube [17]. A sample image of a mature microvessel cut with endothelial cells and pericytes is shown in Fig. 1. Angiogenesis is the formation of new blood vessels consisting of both endothelial and perivascular cells from the pre-existing vasculature. It is necessary for tumor growth, invasion and metastasis to deliver oxygen and nutrients and eliminate metabolic wastes. Previous studies have shown that tumor angiogenesis correlates with the prognosis of cancers in many organs, such as breast, lung, stomach, colon, cervix, and urinary bladder [11], and can be quantified by Microvessel Density (MVD), which measures the growth of new blood vessels within the active regions of a tumor [19,29,43]. One of the most common procedures for determining MVD is to locate “hot spot” (region of high vessel concentration) first [42]. Consequently, the imagebased “hot spot” detection is often the first step to study tumor angiogenesis by computerized technologies. However, this task is challenging due to the presence of large technical variations (e.g.,

∗

Corresponding author. E-mail address: [email protected] (Q. Wen).

fixation, staining) and biological heterogeneities (e.g., tissue type, tissue state). Throughout this paper, we denote an image with angiogenesis formation, namely detected as having “hot spot”, as a positive image, and an image without angiogenesis as a negative image. By doing this, the “hot spot” detection task is turned into an image classification problem. Regarding the image classification methods, Bag-of-Words (BoW) based ones have presented outstanding simplicity and effectiveness in natural scene and object categorization [13,33,36]. Utilizing the pyramid match kernel [16], BoW for image classification is typically divided into the following steps: 1) dense feature extraction in the image; 2) dictionary learning and feature coding; 3) codes pooling and classification. It is characterized by the feature extraction-coding-pooling pipeline. Various BoW based methods with pyramid kernel are widely used and achieve impressive performances [41,46]. Following the similar path, in this paper, we propose a LocalityConstrained Group Lasso Coding (LCGLC) scheme for microvessel image classification, which also works under the feature extractioncoding-pooling framework. The flowchart of the LCGLC scheme is shown in Fig. 2. First, we extracts Scale-InvariantFeature Transform (SIFT) descriptors on the Single-Opponent (SO) feature map, which simulates the biological functionality of human visual systems. Then, we present the Feature-Biased Dictionary Learning to effectively generate the dictionary of SIFT descriptors. With the learned dictionary, our LCGLC method introduces the locality con-

https://doi.org/10.1016/j.patrec.2019.02.011 0167-8655/© 2019 Elsevier B.V. All rights reserved.

Please cite this article as: J. Chen, S. Zhou and Z. Kang et al., Locality-constrained group lasso coding for microvessel image classification, Pattern Recognition Letters, https://doi.org/10.1016/j.patrec.2019.02.011

JID: PATREC 2

ARTICLE IN PRESS

[m5G;February 15, 2019;11:34]

J. Chen, S. Zhou and Z. Kang et al. / Pattern Recognition Letters xxx (xxxx) xxx

adopt the Nesterov’s method to solve the 2,1 constrained smooth convex optimization problem efficiently.

Endothelial cells

This paper is organized as follows. Section 2 introduces the related work, including feature extraction and feature coding. Our method on the extraction of the SO feature map, learning of the Feature-Biased Dictionary and optimization of LCGLC is presented in Section 3. The experiments on the microvessel dataset are presented in Section 4. Finally, Section 5 concludes this work.

Pericytes

2. Related work

Fig. 1. Sample image of a mature microvessel cut with endothelium cells (stained in brown) and pericytes (stained in red). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

straint in classical Group Lasso problem to encode SIFT descriptors. Furthermore, we apply the Spatial Pyramid Matching (SPM) for the code pooling of microvessel images. Finally, we use Support Vector Machine (SVM) to decide if a tissue image has angiogenesis or not. Comprehensive experiments on the microvessel dataset show that the proposed LCGLC method achieves better performance compared with other representative approaches. The proposed framework on microvessel image classification serves as the first phase to automatically analyze microvessel density of human liver carcinoma, and can be easily applied to angiogenesis identification of other carcinomas. Its main contributions can be summarized as follows: 1) We propose a novel feature extraction method for microvessel images through generating SIFT descriptors on the SingleOpponent (SO) feature map. The SO map is generated by convolving both excitatory and inhibitory Gabor filters on the RGB channels of a microvessel image. It is quite discriminative by mimicking human visual systems and able to segment various types of microvessels from surrounding tissues in the image. 2) 2) We propose the Feature-Biased Dictionary Learning method, which relaxes the locality constraint in the dictionary learning process and approximates the strong similarity between input descriptor and basis vectors in the manifold structure. Compared with the “global” dictionary learning, such as Lagrange Dual [46] and Coordinate Descent [41] method, our dictionary learning method can avoid the iterative process between dictionary learning and code updating, thus reduces the computational cost for the diagnosis of tumor microvessel images. 3) 3) Finally, we propose the LCGLC algorithm to encode SIFT descriptors for microvessel image classification. In LCGLC, we

Computer-based pathology systems for tumor image analysis generally consist of a common sequence of steps, including image segmentation, feature extraction, feature coding/selection and pattern classification, although some steps may be missing in a specific task. While there are a considerable amount of literature published on feature selection [9,32], as the proposed LCGLC method follows the feature extraction-coding-pooling framework and mainly focuses on the improvement of feature extraction and feature coding, we briefly review the feature extraction and coding methods for medical image analysis in the following. 2.1. Hand-crafted features Pathologists often look for visual cues to categorize a tissue image as either healthy or diseased. Motivated by this, a variety of hand-crafted low-level image features have been developed for pathology image analysis based on texture, morphometric characteristics (shape and spatial arrangement) and image statistics. For example, the Gray Level Co-occurrence Matrix (GLCM) is applied by Chang et al. [7] to differentiate the benign and malignant tumors in the liver using computed tomography images. Morphological image features are used for nerve-fiber detection and quantification in corneal confocal microscopy images [10]. Twodimensional discrete wavelet transform (DWT) is utilized on brain magnetic resonance image for classification [30]. Fractal features are proposed to distinguish malignant pulmonary nodules from benign nodules [26]. Further, graph based topological features are extracted from skin tissue images to identify cancerous regions associated with basal cell carcinoma [34]. Although diverse low-level image feature extraction methods available right now, the performance of these hand-crafted features still relies largely on the sufficient domain knowledge for the specific medical problem to be solved. Furthermore, several recent feature descriptors in the general imaging domain have also been incorporated into the image-based disease diagnosis, including SIFT [12,15], Speeded Up Robust Features (SURF) [18,44], Local Binary Patterns (LBP) [1,38], and Histogram of Oriented Gradients (HOG) [35,37]. Such feature descriptors are comprehensive in extracting small image details, and also provide multi-resolution and histogram quantization properties that are especially useful for accommodating feature varia-

Fig. 2. The flowchart for microvessel image classification.

Please cite this article as: J. Chen, S. Zhou and Z. Kang et al., Locality-constrained group lasso coding for microvessel image classification, Pattern Recognition Letters, https://doi.org/10.1016/j.patrec.2019.02.011

ARTICLE IN PRESS

JID: PATREC

[m5G;February 15, 2019;11:34]

J. Chen, S. Zhou and Z. Kang et al. / Pattern Recognition Letters xxx (xxxx) xxx

tions. However, while most of these approaches deliver promising performances using the original feature descriptors, it is still highly possible to achieve even better results by enhancing the descriptor designs based on medical imaging characteristics.

Further, we define the Gabor filter as follows:



g(u, v ) = exp − with

 

2.2. Learned features Apart from the aforementioned hand-crafted feature extraction methods, algorithms using machine learning techniques to generate features are getting increasingly more attention. Being one of them, sparse coding has been popularly used as an effective data representation and feature coding method in various applications, such as computer vision, medical imaging and bioinformatics [21]. In Wang et al. [40], sparse coding is introduced for somatic mutations identification and breast tumor classification in ultrasonic images. It is also utilized to segment prostate in CT images by pixel-wise classification [14]. Further, convolutional sparse coding is employed to discover the morphometric signatures for classification of histology sections [49]. Recently, sparse coding based bagof-features is adopted to train a region classifier for infarct region recognition [48]. As another main stream in feature learning scheme, deep learning methods have the advantage of learning hierarchical representations from images, and demonstrate superior performances in a variety of visual recognition tasks. For instance, deep Convolutional Neural Networks (CNN) are applied in the classification of breast cancer histology images [3,4] and interstitial lung diseases [2]. Deep Belief Network (DBN) is utilized to predict the status and subtype of attention deficit hyperactivity disorder in functional Magnetic Resonance Imaging (fMRI) data [23], and also classify mitotic cells in Hematoxyline and Eosin stained images [5]. Nonetheless, the effectiveness of deep learning methods requires abundant labelled data available to overcome the overfitting problem at the training stage, which cannot be easily fulfilled in medical image analysis application (unless the dataset is artificially augmented).



u˜ cos θ = v˜ − sin θ

u˜2 + γ 2 v˜ 2 2σ 2 sin θ cos θ





cos 2π

u˜

λ



+ψ ,

(2)

  u

v

,

where λ is the wavelength of the sinusoidal function, θ is rotation angle, ψ is phase offset, γ is spatial aspect ratio, and σ is bandwidth of the Gaussian envelope function. Mimicking the biological functionality of single-opponent cell in Zhang et al. [47], we propose to extract the Single-Opponent (SO) feature map of a microvessel image in Algorithm 1. Note that we apply the half-wave Algorithm 1 Single-Opponent (SO) feature map extraction of microvessel image. Input: Microvessel image I of RGB channels IR , IG and IB , with corresponding weights ωR , ωG and ωB . The receptive field size [a×b] of Gabor filter, and its corresponding wavelength λ, rotation angle θ , phase offset ψ , spatial aspect ratio γ , and bandwidth σ. Output: The single-opponent feature map SO of I. 1: 2: 3: 4: 5: 6: 7: 8:

3. Proposed method

3

9: 10:

3.1. Extraction of Single-Opponent feature map 11:

Initialize the excitatory Gabor filter gP and inhibitory Gabor filter gN by zeros. Calculate the Gabor filter response g(u, v ) with receptive field [a×b] by Eq. (2). for each point (u, v ) in receptive field [a×b] do if g(u, v ) > 0 then gP (u, v )←g(u, v ), else gN (u, v )← − g(u, v ). end if end for Set Gabor filters on RGB channels by gR ←gP , gG ←gN and gB ←gN . Calculate the single-opponent response F (x, y ) by Eq. (1). Initialize the feature map SO by zeros. for each pixel (x, y ) in F (x, y ) do if F (x, y ) > 0 then SO(x, y )←F (x, y ). end if end for return SO.

In the human visual system, the cortical single-opponent cells respond much more strongly to equiluminant color than to blackwhite grating patterns, and are most responsive at low spatial frequencies [25]. For the recognition of scenes and color atmospheres, single-opponent cells have the advantage of responding to large areas of color and interiors of large patches. We formulate the visual response of a single-opponent cell centered on an image pixel (x, y) by:

12:

F (x, y ) = ωR ·[IR (x, y ) ∗ gR ] + ωG ·[IG (x, y ) ∗ gG ]

rectification on single-opponent response F(x, y) and generate the SO feature map with non-negative values in Algorithm 1. Some of the SO feature map results are shown in Fig. 3.

+ ωB ·[IB (x, y ) ∗ gB ],

(1)

13: 14: 15: 16: 17: 18:

∗

where is the image convolution operator, gR , gG and gB are Gabor filters with receptive field size [a × b] centered on the image pixel (x, y) in RGB channels IR , IG and IB , respectively, ωR , ωG and ωB are the corresponding weights on RGB channels. However, different weights will generate diverse color maps for the same microvessel image. As a microvessel is visualized by both endothelial cells in brown color and pericytes in red color (see Fig. 3 on the next page), we choose the R+ − C− (Red-Cyan) channel, generated by weight signs (+, +, − ) on RGB channels, which is appropriate for segmenting a microvessel from surrounding tissues in the image. Simulating the functionality of single-opponent cells, R+ −C− represents the color distribution with excitatory red center and inhibitory cyan surround.

3.2. Feature-biased dictionary learning Due to its advantages over SURF in salient point detection [20,45], we further extract SIFT features from each image block, which is a small fixed-size square patch of the SO feature map generated by the dense sampling method detailed in Section 4.1. Let yk ∈R p , k = 1, . . . , M, be the SIFT descriptor of the kth image block, where p is the feature dimension and M is the total number of image blocks. Given a dictionary A = [a1 , . . . , aN ] ∈ R p×N of g N base vectors, yk ’s code is xk ∈ RN , and let xk ∈Rng be its correp×n g sponding code of the gth group, and Ag ∈ R be the associated group dictionary, where nvarg is the number of base vectors related

Please cite this article as: J. Chen, S. Zhou and Z. Kang et al., Locality-constrained group lasso coding for microvessel image classification, Pattern Recognition Letters, https://doi.org/10.1016/j.patrec.2019.02.011

ARTICLE IN PRESS

JID: PATREC 4

[m5G;February 15, 2019;11:34]

J. Chen, S. Zhou and Z. Kang et al. / Pattern Recognition Letters xxx (xxxx) xxx

Fig. 3. SO feature map of microvessel images (Row 1: microvessel images on RGB channels. Row 2: their SO feature maps on R+ − C− (Red-Cyan) channels). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

to the gth group. With group information available, the classical group lasso problem is:

min A,xk

   2 G G   g  1 g   y − A x + ρ n x ,  k g k g k 2 k=1 g=1 g=1

M

2

(3)

2

where G is the total number of coding groups, ρ is the positive regularization hyper-parameter, and  · 2 is the 2 norm. As demonstrated by the feature coding scheme Localityconstrained Linear Coding (LLC) [41], the locality of an image code is more essential than its sparsity, as locality must lead to sparsity but not necessarily vice versa. In this paper, we propose a new feature coding algorithm called Locality-Constrained Group Lasso Coding (LCGLC). LCGLC incorporates the locality regularization into classical group lasso problem, which considers the underlying manifold structure of local features and ensures good approximation. We define the LCGLC problem as:

  2 G M G   g  1  min yk − Ag xgk  + ρ n g  xk   2 k=1 2 g=1 g=1 A,xk 2  2 + λdk xk 2 ,

dk.i = yk − ai 2 .

(4)

2

(6)

dk xk 22 ≤ λˆ ,

   2 G   g  G 1 g   ˆ yk − g=1 Ag (k )xk  + ρ g=1 ng xk 2 . k=1

M

2

1:

2:

5:

Apply K-means clustering algorithm on the Mt training SIFT Mt descriptors {yi }i=1 by N cluster centers to get the dictionary

A = [a1 , . . . , aN ] with ai being the ith cluster center. Initialize feature-biased dictionary Aˆ (k ) ← {0}. for each SIFT descriptor yk , k = 1, . . . , M, do Select the K-nearest neighbors {a j (k )}Kj=1 of SIFT descriptor yk from basis vectors ai of A. for each basis vector ai (k ) in A(k ) do

6:

ˆ is some positive number related to λ. where λ Let Aˆ (k ) = [aˆ 1 (k ), . . . , aˆ N (k )] ∈ R p×N be the feature-biased dic2 ˆ of tionary for yk that satisfies the locality constraint dk xk 2 ≤ λ LCGLC in Eq. (6). Then, Eq. (6) can be further relaxed as follows:

xk

Output: M feature-biased dictionary Aˆ (k ), k = 1, . . . , M.



if ai (k ) ∈ a j (k )

K

j=1

then

aˆ i (k ) ← ai (k ), end if 9: end for 10: end for ˆ (k ), k = 1, . . . , M. 11: return A 7:

8:

(5)

   2 M G   g  G 1  min yk − Ag xgk  + ρ ng xk   2 k=1 g=1 g=1 A,x 2

min

Input: Mt Training SIFT descriptors {yi }i=1 , SIFT descriptors {yk }M to be i=1 coded, and the size N of the dictionary Aˆ .

4:

The last regulation term in Eq. (4) can also be reformulated as constraint:

s.t.

Algorithm 2 Feature-Biased Dictionary Learning for LCGLC.

3:

where λ is a positive regularization hyper-parameter,  denotes the element-wise multiplication, and dk ∈RN is the locality adapter with its ith element dk.i measuring the Euclidean distance between the SIFT descriptor yk and each basis vector ai of dictionary A. Specifically, we have

k

be zero vector. Therefore, we say that the dictionary Aˆ (k ) is biased to feature yk . The construction of feature-biased dictionary Aˆ (k ) in Eq. (7) is described in Algorithm 2.

(7)

2

In practice, for each SIFT descriptor yk , Aˆ (k ) is constructed by selecting yk ’s K-nearest base vectors in A and setting other ai in A to

3.3. Optimization of LCGLC coding To optimize yk ’s coding using LCGLC, we consider the following equivalent problem of Eq. (7) for yk :

1     yg − Aˆ g (k )xg 2 + ρ ng xg  k k 2 k 2 g=1 2 xk    G  1  g 2 y − Aˆ g (k )xg  + γg xg  , = min k k 2 k 2 g=1

min

G

xk

(8)

2

√ where γg = ρ ng . The 2 norm regularization in Eq. (8) is actually the 2,1 norm as suggested by Nie et al. [32]. Unlike the common practice to solve the non-smooth convex 2,1 norm [9] or 2,1 norm like [28] regularization by iterative algorithms, applying the Lagrangian duality, the 2 norm regularization problem in Eq. (8) is equivalent to the following 2,1 constrained smooth convex opti-

Please cite this article as: J. Chen, S. Zhou and Z. Kang et al., Locality-constrained group lasso coding for microvessel image classification, Pattern Recognition Letters, https://doi.org/10.1016/j.patrec.2019.02.011

ARTICLE IN PRESS

JID: PATREC

[m5G;February 15, 2019;11:34]

J. Chen, S. Zhou and Z. Kang et al. / Pattern Recognition Letters xxx (xxxx) xxx

mization problem:

min

xk 2,1 ≤γˆg

G  g=1

 1 yg − Aˆ g xg 2 , k k 2 2

(9)

Table 1 Quantitative evaluation results of the image classification methods. Method

G

g g=1 xk 2 ,

where xk 2,1 = and γˆg is some positive number related to γ g . Finally, we adopt the Nesterov’s method [31] introduced in Liu et al. [27] to solve Eq. (9) efficiently. 4. Experiments

5

LLC ScSPM CNN DBN SPM LCGLC

Metrics (%) Precision

Recall

F-measure

77.64 77.16 47.51 79.03 90.55 83.04

82.40 81.20 94.80 73.60 46.80 93.60

79.83 79.02 63.28 74.82 61.54 87.86

4.1. Experimental settings For the parameters in Algorithm 1 to construct the Gabor filters, we set the wavelength λ = 5.6410, rotation angle θ ∈ {0, π /2}, phase offset ψ = 0, spatial aspect ratio γ = 0.3, bandwidth σ = 4.5128, and the receptive field size [a × b] as 11 × 11 pixels. Actually, we found that the classification performance is quite stable for different filter parameter values. We also √choose the weights √ √ on the RGB channels as ωR = 2/ 6, ωG = −1/ 6 and ωB = −1/ 6 to obtain the SO feature map. We use the average of the SO feature maps generated by different Gabor filters as the final SO feature map of an image. As in this experiment, we have two Gabor filters. In our experiments, the 128 dimensional SIFT features are extracted at single-scale from densely sampled patches of SO feature images. The patches are of size 16 × 16 pixels each and are centered over a grid with spacing of 6 pixels. In Algorithm 2, the feature-biased dictionary is initialized with N = 900 basis vectors by K-means, and tuned with 200-nearest neighbors. Here, the K-means method is initialized by choosing the cluster centers randomly from the data points, which is mentioned as MacQueen’s second method in Celebi et al. [6]. For the optimization of LCGLC, we sequentially and evenly divide all the basis vectors of dictionary Aˆ (k ) into groups. Specifically, we set ng = 10. In the experiments, the Spatial Pyramid Matching (SPM) is used by hierarchically partitioning each image into 1 × 1, 2 × 2, and 4 × 4 blocks on 3 levels, whose cumulative concatenations are denoted by SPM0, SPM1 and SPM2, respectively. In particular, SPM2 means that all three levels (from 0 to 2) are used by concatenating their max pooling vectors. Finally, a feature vector with 18,900 ((4 × 4 + 2 × 2 + 1 ) × 900) dimension is generated for each image. All obtained image representations are fed into the SVM in the training and testing phases (the LIBSVM package [8]), where the penalty parameter of SVM is fixed as C = 219 and the Radial Basis Function (RBF) kernel parameter is fixed as γ = 2−20 by grid search. 4.2. Evaluation metrics In the experiment, we adopt the recall, precision and F-measure to statistically analyze the performance. All the three measures are calculated by the classification quantities, namely, true positive (TP), false positive (FP), true negative (TN) and false negative (FN). Recall is defined as:

recall =

TP . TP + FN

(10)

Precision is defined as:

precision =

TP . TP + FP

(11)

F-measure is defined as the harmonic mean of precision and recall:

F −measure =

2 · precision · recall . precision + recall

(12)

As a result, higher values of precision, recall and F-measure indicate better performance. 4.3. Data preparation Two sections of liver with metastatic colon carcinoma, which are formalin-fixed and paraffin-embedded, are digitized into two virtual slides. The liver tissues are double-immunostained using both anti-smooth muscle actin and anti-CD31 antibodies. Smooth muscle actin labels pericytes. And CD31 labels endothelial cells. The liver tissues are scanned at 200 × magnification using an Aperio ScanScope T2 scanner, and archived in 24-bit color JPEG 20 0 0 format with a resolution of 21943 × 21726. The two virtual slides are divided into 521 images, including 247 positives and 274 negatives. Each microvessel image has a resolution of 300 × 300 pixels, which represents a 0.14 mm × 0.14 mm region of a glass slide. 4.4. Quantitative evaluation results We evaluate the proposed method on the microvessel dataset mentioned in Section 4.3. Some examples are shown in Fig. 4. In particular, the images possess small inter-class difference and large intra-class variance. It is notable that the appearances of different microvessel instances vary greatly. And some negatives and positives are too similar to distinguish even by the human exports. Such large intra-class variance and tiny inter-class difference make the classification quite challenging. We use 10-fold cross-validation to evaluate the performance. All the positive and negative images are randomly divided into 10 folds. In each iteration, the training set consists of 222 positive and 246 negative images, while the testing set consists of the remaining 25 positive and 28 negative images. We repeat the training and testing process for 10 times, and calculate the averaged performance. Table 1 gives the classification performance of different methods on the microvessel dataset. Here, we choose some representative methods in sparse coding and deep learning for comparison purpose, such as LLC [41], ScSPM [46], CNN [22], and DBN [39]. Besides, in order to demonstrate the effectiveness of our proposed method on improving the classification performance, Bag-of-Words (BoW) based method SPM [24] is also selected to compare with our LCLGC method on the SO feature map of microvessel dataset. For each comparison method mentioned above, we apply its hyperparameter setting used in the corresponding paper. The details are omitted here for space reasons. From Table 1, it can be observed that the proposed LCGLC method significantly outperforms the benchmark methods and generally achieves the top performance. This well verifies that the LCGLC method on SO feature map is able to more effectively capture the discriminative information in microvessel images. LLC and ScSPM have similar performance on this dataset, both producing the F-measure around 79%. Although CNN generates the highest recall rate of 94.80%, yet its poor precision rate drops the F-measure

Please cite this article as: J. Chen, S. Zhou and Z. Kang et al., Locality-constrained group lasso coding for microvessel image classification, Pattern Recognition Letters, https://doi.org/10.1016/j.patrec.2019.02.011

JID: PATREC 6

ARTICLE IN PRESS

[m5G;February 15, 2019;11:34]

J. Chen, S. Zhou and Z. Kang et al. / Pattern Recognition Letters xxx (xxxx) xxx

Fig. 4. Example images of microvessel dataset. Row 1 contains positive instances. Row 2 contains negative instances.

down to 63.28%. In contrast, DBN has the best balance between the precision rate of 79.03% and the recall rate of 73.60%, and reaches a F-measure of 74.82%. SPM produces the highest precision rate of 90.55%, but its low recall rate causes corresponding F-measure decline to 61.54%. The proposed LCGLC method achieves the highest F-measure at 87.86% with more than 8% above the best F-measure of other comparison methods, while maintaining the second best balance between a recall rate of 93.60% and a precision rate of 83.04%. 5. Conclusion In this paper, we propose a Locality-Constrained Group Lasso Coding (LCGLC) method for microvessel image classification, which realizes the automatic “hot-spot” detection of angiogenesis for human liver carcinoma. In fact, a novel pattern segmentation method is introduced to extract SIFT descriptors on the SingleOpponent (SO) feature map. Furthermore, the Feature-Biased Dictionary Learning method is utilized to relax the locality constraint in the dictionary learning process, which approximates the strong similarity between input descriptor and basis vectors in the manifold structure. Consequently, the LCGLC method is employed to encode SIFT descriptors for microvessel image classification. From the analysis and experiments, it is shown that the proposed LCGLC method on SO feature map is able to more effectively capture the discriminative information in microvessel images. Thus, the proposed method significantly outperforms other representative benchmark methods and generally achieves better performance. While investigating the classification results of the proposed LCGLC method, it is noticed that the large intra-class variance would significantly degrade the final classification performance. Inspired by this observation, we plan to investigate a new model to reduce the interference of intra-class variance aiming to improve the classification performance, e.g., embedding the image labels into the dictionary learning framework, which will form one of our future research directions. Conflict of interest The authors declare that there is no conflict of interest regarding the publication of this article. Acknowledgments This work was supported by the Fundamental Research Funds for the Central Universities (ZYGX2016J164). We would like to thank Dr. Mutlu Mete for providing their valuable image data. Our

gratitude also goes to the anonymous reviewers for their valuable comments. References [1] S. Abbasi, F. Tajeripour, Detection of brain tumor in 3D MRI images using local binary patterns and histogram orientation gradient, Neurocomputing 219 (2017) 526–535. [2] M. Anthimopoulos, S. Christodoulidis, L. Ebner, A. Christe, S. Mougiakakou, Lung pattern classification for interstitial lung diseases using a deep convolutional neural network, IEEE Trans. Med. Imaging 35 (5) (2016) 1207–1216. [3] T. Araujo, G. Aresta, E. Castro, J. Rouco, P. Aguiar, C. Eloy, A. Polonia, A. Campilho, Classification of breast cancer histology images using convolutional neural networks, PLoS ONE 12 (6) (2017). [4] A.S. Becker, M. Marcon, S. Ghafoor, M.C. Wurnig, T. Frauenfelder, A. Boss, Deep learning in mammography diagnostic accuracy of a multipurpose image analysis software in the detection of breast cancer, Invest. Radiol. 52 (7) (2017) 434–440. [5] K.S. Beevi, M.S. Nair, G.R. Bindu, A multi-classifier system for automatic mitosis detection in breast histopathology images using deep belief networks, IEEE J. Transl. Eng. Health Med. 5 (2017) 1–11. [6] M.E. Celebi, H.A. Kingravi, P.A. Vela, A comparative study of efficient initialization methods for the k-means clustering algorithm, Expert Syst. Appl. 40 (1) (2013) 200–210. [7] C.C. Chang, H.H. Chen, Y.C. Chang, M.Y. Yang, C.M. Lo, W.C. Ko, Y.F. Lee, K.L. Liu, R.F. Chang, Computer-aided diagnosis of liver tumors on computed tomography images, Comput. Methods Programs Biomed. 145 (2017) 45–51. [8] C.C. Chang, C.J. Lin, Libsvm : a library for support vector machines, ACM Trans. Intell. Syst. Technol. 2 (3) (2011) 1–27. [9] X. Chang, Y. Yang, Semisupervised feature analysis by mining correlations among multiple tasks, IEEE Trans. Neural Netw. Learn. Syst. 28 (10) (2017) 2294–2305. [10] X. Chen, J. Graham, M.A. Dabbah, I.N. Petropoulos, M. Tavakoli, R.A. Malik, An automatic tool for quantification of nerve fibers in corneal confocal microscopy images, IEEE Trans. Biomed. Eng. 64 (4) (2017) 786–794. [11] K. Dhanuthai, K. Sappayatosok, S. Yodsanga, S. Rojanawatsirivej, N.C. Pausch, P. Pitak-Arnnop, An analysis of microvessel density in salivary gland tumours: a single centre study, Surgeon 11 (2013) 147–152. [12] A. Farag, S. Elhabian, J. Graham, A. Farag, R. Falk, Toward precise pulmonary nodule descriptors for nodule type classification, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, 2010, pp. 626–633. [13] L. Fei-Fei, P. Perona, A Bayesian hierarchical model for learning natural scene categories, in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2, 2005, pp. 524–531. [14] Y. Gao, S. Liao, D. Shen, Prostate segmentation by sparse representation based classification, Med. Phys. 39 (10) (2012) 6372–6387. [15] V. Gonzalez-Castro, M.D.C.V. Hernandez, F.M. Chappell, P.A. Armitage, S. Makin, J.M. Wardlaw, Reliability of an automatic classifier for brain enlarged perivascular spaces burden and comparison with human performance, Clin. Sci. 131 (13) (2017) 1465–1481. [16] K. Grauman, T. Darrell, The pyramid match kernel: discriminative classification with sets of image features, in: IEEE International Conference on Computer Vision, Beijing, China, 2, 2005, pp. 1458–1465. [17] A. Holm, T. Heumann, H.G. Augustin, Microvascular mural cell organotypic heterogeneity and functional plasticity, Trends Cell Biol. (2018). [18] S. Hwang, Bag-of-visual-words approach based on SURF features to polyp detection in wireless capsule endoscopy videos, in: International Conference on Image Processing, Computer Vision, and Pattern Recognition, 2011, pp. 941–944.

Please cite this article as: J. Chen, S. Zhou and Z. Kang et al., Locality-constrained group lasso coding for microvessel image classification, Pattern Recognition Letters, https://doi.org/10.1016/j.patrec.2019.02.011

JID: PATREC

ARTICLE IN PRESS

[m5G;February 15, 2019;11:34]

J. Chen, S. Zhou and Z. Kang et al. / Pattern Recognition Letters xxx (xxxx) xxx [19] L.B. Jilaveanu, M. Puligandla, S.A. Weiss, V.X. Wang, C.R. Zito, K.T. Flaherty, M. Boeke, V.M. Neumeister, R.L. Camp, A. Adeniran, M. Pins, J. Manola, R.S. DiPaola, N. Haas, H.M. Kluger, Tumor microvessel density as a prognostic marker in high-risk renal cell carcinoma patients treated on ecog-acrin e2805, Clin. Cancer Res. (2017). [20] L. Juan, O. Gwun, A comparison of SIFT, PCA-SIFT and SURF, Int. J. Image Process. 3 (4) (2009) 143–152. [21] Z. Kang, C. Peng, Q. Cheng, Kernel-driven similarity learning, Neurocomputing 267 (2017) 210–219. [22] A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in: Annual Conference on Neural Information Processing Systems, 2012, pp. 1097–1105. [23] D. Kuang, X. Guo, X. An, Y. Zhao, L. He, Discrimination of ADHD based on fMRI data with deep belief network, in: International Conference on Intelligent Computing, 2014, pp. 225–232. [24] S. Lazebnik, C. Schmid, J. Ponce, Beyond bags of features: spatial pyramid matching for recognizing natural scene categories, in: IEEE Conference on Computer Vision and Pattern Recognition, 2, 2006, pp. 2169–2178. [25] P. Lennie, J. Krauskopf, G. Sclar, Chromatic mechanisms in striate cortex of macaque, J. Neurosci. 10 (1990) 649–669. [26] P.L. Lin, P.W. Huang, C.H. Lee, M.T. Wu, Automatic classification for solitary pulmonary nodule in ct image by fractal analysis based on fractional brownian motion model, Pattern Recognit. 46 (12) (2013) 3279–3287. [27] J. Liu, S. Ji, J. Ye, Multi-task feature learning via efficient 2,1 -norm minimization, in: Conference on Uncertainty in Artificial Intelligence, in: UAI’09, AUAI Press, Arlington, Virginia, United States, 2009, pp. 339–348. [28] M. Luo, X. Chang, Z. Li, L. Nie, A.G. Hauptmann, Q. Zheng, Simple to complex cross-modal learning to rank, Comput. Vis. Image Understand. 163 (2017) 67–77. [29] M. Mete, L. Hennings, H.J. Spencer, U. Topaloglu, Automatic identification of angiogenesis in double stained images of liver tissue, BMC Bioinform. 10(Suppl 11):S13 (2009). [30] D.R. Nayak, R. Dash, B. Majhi, Brain MR image classification using two-dimensional discrete wavelet transform and adaboost with random forests, Neurocomputing 177 (2016) 188–197. [31] Y. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course, Kluwer Academic Publishers, 2003. [32] F. Nie, H. Huang, X. Cai, C.H. Ding, Efficient and robust feature selection via joint 2,1 -norms minimization, in: Advances in Neural Information Processing Systems 23, 2010, pp. 1813–1821. [33] S. O’Hara, B.A. Draper, Introduction to the bag of features paradigm for image classification and retrieval, CoRR (2011). [34] D. Romo-Bucheli, G. Corredor, J.D. Garcia-Arteaga, V. Arias, E. Romero, Nuclei graph local features for basal cell carcinoma classification in whole slide images, in: E. Romero, N. Lepore, J. Brieva, I. Larrabide (Eds.), International Symposium on Medical Information Processing and Analysis, Proceedings of SPIE, 10160, 2017.

7

[35] S. Sedai, P.K. Roy, R. Garnavi, Right ventricle landmark detection using multiscale hog and random forest classifier, in: IEEE International Symposium on Biomedical Imaging, in: IEEE International Symposium on Biomedical Imaging, 2015, pp. 814–818. [36] J. Sivic, B.C. Russell, A.A. Efros, A. Zisserm, W.T. Freeman, Discovering objects and their location in images, in: IEEE International Conference on Computer Vision, 1, 2005, pp. 370–377. [37] Y. Song, W. Cai, J. Kim, D. Feng, A multistage discriminative model for tumor and lymph node detection in thoracic images, IEEE Trans. Med. Imaging 31 (5) (2012) 1061–1075. [38] L. Sorensen, S. Shaker, M. de Bruijne, Quantitative analysis of pulmonary emphysema using local binary patterns, IEEE Trans. Med. Imaging 29 (2) (2010) 559–569. [39] M. Tanaka, M. Okutomi, A novel inference of a restricted Boltzmann machine, in: International Conference on Pattern Recognition, 2014, pp. 1526–1531. [40] J. Wang, H. Bensmail, N. Yao, X. Gao, Discriminative sparse coding on multi– manifolds, Knowl. Based Syst. 54 (2013) 199–206. [41] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, Y. Gong, Locality-constrained linear coding for image classification, in: IEEE Conference on Computer Vision and Pattern Recognition, 2010, pp. 3360–3367. [42] N. Weidner, J. Semple, W. Welch, J. Folkman, Tumour angiogenesis and metastasis - correlation in invasive breast carcinoma, N. Engl. J. Med. 324 (1) (1991) 1–8. [43] S.A. Weiss, M. Puligandla, L. Jilaveanu, N.B. Haas, X.V. Wang, C. Zito, M. Boeke, V. Neumeister, J. Manola, R.S. DiPaola, H.M. Kluger, Microvessel density as a prognostic marker in high-risk renal cell carcinoma, J. Clin. Oncol. 35 (15_suppl) (2017) 4565. [44] A. Wojnar, A.M.G. Pinheiro, Annotation of medical images using the surf descriptor, in: IEEE International Symposium on Biomedical Imaging, 2012, pp. 130–133. [45] J. Wu, Z. Cui, V.S. Sheng, P. Zhao, D. Su, S. Gong, A comparative study of SIFT and its variants, Meas. Sci. Rev. 13 (3) (2013) 122–131. [46] J. Yang, K. Yu, Y. Gong, T. Huang, Linear spatial pyramid matching using sparse coding for image classification, in: IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 1794–1801. [47] J. Zhang, Y. Barhomi, T. Serre, A new biologically inspired color image descriptor, in: European Conference on Computer Vision, 2012, pp. 312–324. [48] X. Zhang, A. Elazab, Q. Hu, Segmentation of hyper-acute cerebral infarct based on random forest and sparse coding from diffusion weighted imaging, in: International Conference of the IEEE Engineering in Medicine and Biology Society, 2017, pp. 3425–3428. [49] Y. Zhou, H. Chang, K. Barner, P. Spellman, B. Parvin, Classification of histology sections via multispectral convolutional sparse coding, in: IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 3081–3088.

Please cite this article as: J. Chen, S. Zhou and Z. Kang et al., Locality-constrained group lasso coding for microvessel image classification, Pattern Recognition Letters, https://doi.org/10.1016/j.patrec.2019.02.011