- Email: [email protected]
Detection of cervical lesion region from colposcopic images based on feature reselection
Biomedical Signal Processing and Control 57 (2020) 101785
Contents lists available at ScienceDirect
Biomedical Signal Processing and Control journal homepage: www.elsevier.com/locate/bspc
Detection of cervical lesion region from colposcopic images based on feature reselection Bing Bai a , Yongzhao Du a,b,∗ , Peizhong Liu a,b , Pengming Sun c , Ping Li d , Yuchun Lv d a
College of Engineering, Huaqiao University, Quanzhou, China College of Medicine, Huaqiao University, Quanzhou, China Fujian Provincial Maternity and Children’s Hospital, Affiliated Hospital of Fujian Medical University, Fuzhou, China d Department of Gynecology and Obstetrics, The First Hospital of Quanzhou, Fujian Medical University, Quanzhou, China b c
a r t i c l e
i n f o
Article history: Received 27 May 2019 Received in revised form 11 October 2019 Accepted 16 November 2019 Keywords: Cervical lesion Deep learning Object detection SE-CNN CLDNet
a b s t r a c t Colposcopy is one of the important steps in the clinical screening of cervical intraepithelial neoplasia (CIN) and early cervical cancer. It directly affects the patient’s diagnosis and treatment program. Therefore, it is widely used for cervical cancer screening. The present work proposes a cervical lesion detection net (CLDNet) model based on the deep convolutional neural network (CNN). The Squeeze-Excitation convolutional neural network (SE-CNN) employed to extract depth features of the whole image. SE module for feature recalibration. Moreover, the region proposal network (RPN) generated a proposal box of the region of interest (ROI). Finally, the region of interest classified and proposal box regression performed to locate the cervical lesion region. The Squeeze-Excitation (SE block) strengthened important features and suppress non-primary features, improve feature extraction ability, which is beneficial to feature classification and proposal box regression in the regions of interest. It is found that the average precision of the model extraction lesion region is 92.53 % and the average recall rate is 85.56 %, which can play a good role in the auxiliary diagnosis. © 2019 Published by Elsevier Ltd.
1. Introduction Cervical cancer ranks as the fourth most common type of cancers worldwide in women aged 15–44 years [1]. Moreover, it is expected that the global cancer burden will further aggravated. According to the report, almost half of the world’s new cancer cases and more than half of all cancer deaths occurred in Asia in 2018 [2]. Among all types of cancer, cervical cancer incidence and mortality ranked fourth. However, most women with cervical cancer have never been screened or screened inadequately worldwide [3,4]. Studies show that about 50 % of cervical cancer patients have never undergone the cervical cytology and another 10 % have not screened for cervical cancer within five years before the cervical cancer diagnosis [5,6]. Therefore, a large-scale and standard screening for the general population is one of the most effective ways to reduce the incidence and death of cervical cancer. The process of the cervical cancer screening is highly complex and costly, resulting in the inability to popularize more advanced cervical cancer screening techniques in under-resourced settings.
∗ Corresponding author. E-mail addresses: [email protected] (Y. Du), [email protected] (P. Liu). https://doi.org/10.1016/j.bspc.2019.101785 1746-8094/© 2019 Published by Elsevier Ltd.
Therefore, the morbidity and mortality of cervical cancer are still high in underdeveloped regions [7,8]. According to recommendations of the World Health Organization, there are currently three types of cervical cancer/lesion screening methods, including human papillomavirus (HPV) testing, Pap smear testing and vinegar based colposcopy. The first two methods are relatively complex and expensive, especially for patients in developing countries [9]. Even in the first two methods, colposcopy is still a necessary step in the biopsy. Therefore, colposcopy is a necessary and most effective tool for cervical cancer screening. Colposcopy is the primary diagnostic method used to detect cervical cancer in women. It can identify and determine the severity of the lesion so that a biopsy of the highest level of abnormality detection can be taken, when necessary. Colposcopy includes a visual assessment of the system of the lower genital tract (cervical, vulva and vagina), primarily for the appearance of a metaplastic epithelium, consisting of a transformation zone on the cervix [10]. During the examination, a volumetric vinegar solution is applied to the cervix, resulting in abnormal whitening of the metaplastic epithelial tissue. Cervical precancerous lesions and invasive cancers exhibit significant abnormal morphological features that can be identified by the colposcopy. Pathological features of the cervical epithelial tissue, such as color characteristics, opacity, edge division
2
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785
Fig. 1. Five colposcopic images (Normal, CIN1, CIN2, CIN3 and Cancer).
and tissue shape, should be observed by an expert doctor expert to obtain a clinical diagnosis. Due to the subjective nature of the examination, the accuracy of the colposcopy is highly dependent on the doctor’s experience and expertise. Therefore, the colposcopy has low specificity and requires many unnecessary biopsies [11]. Cervical images observed by the colposcopy during the cervical cancer screening are divided into five categories, including normal, CIN1, CIN2, CIN3 and cancerous images. Fig. 1 showed that each type of colposcopic images has its own characteristics. 2. Related work Computer vision related technologies have been widely used in the field of the medical image analysis. They are mainly focusing on classification and reconstruction of lesion images, segmentation of lesions, detection and localization of lesions [12]. Currently, the application of the screening for cervical lesions are divided into cervical cancer screening techniques based on celllevel and colposcopy-based images [13]. Moreover, the cervical cancer screening based on colposcopy pictures is the most active research and includes the cervical segmentation and the cervical lesion classification. Based on conventional image processing methods and the application requirements of the clinical cervical cancer screening, NA Obukhova et al. [14] proposed a method to embed the artificial intelligence technology into the colposcope so that the colposcopic imaging device can automatically analyze the image, processed by the vinegar. The key premise is the segmentation of the cervical region and the development of a cervical region of interest (ROI) segmentation procedure. The device obtains colposcopy pictures for different light wavelengths and removes highlights from the image. Then the device obtains diagnostic results by performing a cross-analysis of the discrete Fourier transforms on different spectrograms. In the research project of Mr. Deepak. B. Patil et al. [15], a special camera with a green filter was used to collect the colposcopic image by setting the pixel with higher color gradient as the specular reflection region and performing filled pixels to remove the highlight region. Finally, the K-means was used to perform the ROI region segmentation on the colposcopic image. Moreover, Huang S et al. proposed a transformation learning algorithm based on the image pixel classification [16], which can roughly segment the cervical region. However, the proposed algorithm neglects the
removal and filling of the specular reflection region during the colposcopy imaging. Rama Praba et al. [17] applied a Gaussian mixture modeling method based on the mathematical morphology to eliminate irrelevant information and specular reflection in colposcopy pictures, prior to extracting features. Moreover, they applied the image segmentation based on the sparse reconstruction to segment the cervical region (ROI) [18]. Researches showed that the cervical region obtained by the above algorithm still needs to be classified and detected by the image processing technology, so as screening cervical cancer. Recently, CNN models have been widely used in the medical image analysis. For the classification of cervical lesion images, Tao Xu et al. [19] combined cervical images with Pap and HPV test results to construct a multimodal cervical lesion classification model. Similarly, they combined the PLAB, PHOG and PLBP multi-feature fusion with multiple machine learning classifiers to investigate the cervical cancer screening [20]. Correia et al. [21] proposed a regularized migration learning strategy to classify cervical images. The algorithm realized parameter sharing and improved the accuracy of the image classification. With the improvement of the auxiliary diagnosis technology, Toshiaki Hirasawa et al. [22] constructed a model for the endoscopic image detection based on the single shot multi-box detector (SSD) structure. They showed that the proposed algorithm is able to mark the lesion region in the gastroscopic image. In order to improve the accuracy of the cervical cancer intelligent assisted screening, a deep learning model based on the CNN is proposed in the present study. The proposed model can automatically calibrate cervical lesions. This model can mark the location of cervical lesions and estimate the lesion region. The algorithm applies the attention mechanism to the feature extraction of the lesion region, aiming to reduce the interference of the non-lesional region on the process. Combined with the advice of the expert doctor, relevant parameters of calibration lesion regions are tuned to achieve accurate labeling of the cervical lesion region. 3. Analysis of cervical lesions Colposcopy is an early tool for screening cervical cancer and precancerous lesions. Various colposcopic abnormal images have reference significance in the cervical screening. According to the risk degree, the hazard order of these abnormal images are [23]:
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785
3
Fig. 2. Multiple cervical lesion features. A: Punctate vessels, intraepithelial cells proliferate upwardly. B: Internal boundary, the internal vinegar white boundary of the lesion. C: Ridge-like sign, the opaque bulge of the white epithelium in the transformation zone.
Fig. 3. The overall framework of the model (CLDNet).
vinegar white epithelium, point blood vessels, cuff-like gland opening, inlay, iodine test without coloration, internal boundary, ridge-like signs and abnormal blood tube. Fig. 2 illustrates the risk degree for a cervical lesion. Colposcopy needs to initially identify the transformation zone. Then, in order to determine the lesion nature and necessity of a biopsy, it identifies whether there is an abnormal image in the transformation zone [24]. Dynamic observation of cervical epithelial vessels to find abnormal images to judge is an important task of the colposcopy. Different abnormal images reveal different risks and should be screened and treated. Acetowhite Epithelium original: The white change in the epithelium after application of the vinegar solution is called the white acetate epithelium, Punctate vessels: Fig. 2(A) showed that intraepithelial cells proliferate upwardly to compress blood vessels and form an image of fine red dots. Cuff-like gland opening: A thickened bulging white ring around the opening of the cervix gland is a signal that the lesion affects the gland. In this case, a deep biopsy should be performed, as shown in Fig. 2(B). Internal boundary: The internal vinegar white boundary of the lesion refers to the thickness of the Acetowhite Epithelium present in the same lesion region. Researches show that 7.6 % and 70 % of internal boundaries are pathologically confirmed as CIN2 and CIN3, respectively [25]. Ridge-like sign: refers to the opaque bulge of the white epithelium in the transformation zone, indicating abnormal proliferation of the lesion, as shown in Fig. 2(C). CIN1: Relatively thin white acetate white epithelium with blurred edges and small spotted blood vessels and small fuzzy inlays. CIN2: Acetate white epithelium is thick, dirty, opaque, oyster white or gray, the boundary of the lesion is sharp, well-defined, visible large point blood vessels and inlays. CIN3: A thick white epithelium appears in the columnar epithelial area, with thick punctate blood vessels and large irregular mosaics. Cancer: dense, thick grayish white or yellowish white acetate epithelium and singularly shaped blood vessels [26–30]. Clinical observations show that high-grade cervical lesions are mostly located in the cervical transformation region. It is extremely
important to observe the transformation of the transformation region in colposcopy. Lesions that are remote from the transformation zone are usually metastatic lesions or multicenter lesions. Colposcopy screening requires a description of the number of quadrants involved in the lesion and the percentage of lesion region occupying the surface region of the cervix, with a clock as a marker. It is found that the greater is the lesion extent, the greater is the likelihood of potentially high-grade lesions. Based on the above analysis on images of various lesions of the cervix, the CLDNet model proposed in the present study can effectively extract the cervical transformation region in the colposcopy image. 4. Methods 4.1. Model Fig. 3 illustrates the overall framework of the proposed CLDNet. The model is designed and improved according to the structure of the Faster RCNN [31], which is mainly composed of two parts. The first part is the deep feature extraction CNN, called SE-CNN network, which joins the SE block. The second part is the RPN that generates the region of interest (ROI). The region of interest is the cervical lesion region. The proposal box generated by the second part is mapped to feature maps, generated by the first part. Meanwhile, the RPN shares the weight with the SE-CNN to improve the training speed of the model. 4.2. Image features extraction The proposed feature extraction network of the lesion region aims to extract features of the lesion region in the cervical image. Since the classification network in the algorithm ultimately needs to determine whether the ROI belongs to the lesion region according to the feature, the CNN is selected. Shallow features of the model are used as the basis for the classification of lesions. In order to
4
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785
Fig. 5. LRPN implementation process.
be indicated that ı, W1 and W2 are the ReLU function, dimensionality reduction parameters required for the first and second full join operations, respectively. 4.3. Lesion region proposal network (LRPN)
Fig. 4. Feature Extraction Network (SE-CNN).
achieve better results, a feature reselection strategy is added, based on the basic CNN model [32]. It is intended to acquire the importance of each feature channel through learning and then enhance useful features according to this importance and suppress features that are of little use to the current task. Fig. 4 schematically illustrates the feature extraction network in the model of the present work. The basic CNN model of the feature extraction network adopts the ResNet deep learning model [33], selects feature maps of the first, third and fifth convolution operations of the model and selects non-adjacent feature layers for the purpose of avoiding the feature overlap [34]. Meanwhile, in order to improve the utilization of extracted features, SE block [32] is added between convolution operations. In order to reselection important features after obtaining the global receptive field, the module mainly includes Squeeze and Excitation operations. The Squeeze operation compresses each feature layer by a global average pooling operation after the convolution operation so that the C feature maps converted to the real number sequence of 1*1*C, as the following: zc = Fsq (uc ) =
1 W ×H
H W
The LRPN takes an image with arbitrary scale as the input and prepares a set of rectangular target suggestion boxes, as the output. Each box contains four positions, coordinate variables and a score. In order to generate a region suggestion box, the input image is generated through a shared convolution layer to generate a feature map. Then, a multi-scale convolution operation is performed on the feature map. The specific implementation process is as the following (Fig. 5): three kind of scales and three aspect ratios are used at the position of each sliding window. Centered on the center of the current sliding window. Moreover, corresponding to each scale and aspect ratio, mapping nine different scale candidate regions on the original image. For example, for a shared convolutional feature map size of W×H, there are W×H×9 proposal regions. Finally, the classifier prepares scores of W×H×2×9 proposal regions. Scores show probability that the region belongs to the lesion region. Moreover, the regression layer outputs W×H×4×9 parameters, which show coordinate parameters of the proposal region. When training the LRPN network, each candidate lesion region is assigned a binary label to indicate whether the region is lesion. The specific operations are as follows: 1) Candidate regions with the highest intersection-over-union (IOU) overlap with a true ground truth (GT). 2) IOU overlaps regions with any GT bounding box greater than 0.7. A negative label is assigned to a candidate region with an IOU ratio of less than 0.3 for all GT bounding boxes. 3) The region between two IOUs is discarded. The loss function defined in joint training: L({pi }, {ti }) =
1 1 ∗ Lcls (pi , p∗i ) + pi Lreg (ti , ti∗ ) Ncls Nreg i
uc (i, j)
(1)
i=1 j=1
Eq. (1) presents the average pooling operation of the convolved feature layer. Where c is the number of channels of the feature layer. In other words, it is the average pooling operation for each feature layer. W and H respectively indicate the width and height of the feature layer. Moreover, Zc is the weight of the feature layer to redistribute features of each layer to achieve enhancement of useful features and suppress unnecessary features, as shown in Eq. (2). Fex (z, W ) = (g(z, W )) = (W2 ı(W1 z))
(3)
i
(2)
Eq. (2) presents the Excitation operation in the SE block. It initially performs a full connection operation. Then it passes the ReLU function and finally performs a full connection operation. It should
In Eq. (3), i is the proposed frame index selected in one batch, and pi represents the probability that the proposed frame i is the target. If the proposed box is a positive label, its corresponding real area label pi * is 1, otherwise pi * is 0. ti represents the four parameterized coordinate vectors of the predicted bounding box, and ti * is the coordinate vector of the corresponding real region bounding box. The classification loss Lcls is the logarithmic loss for two categories (target and non-target), defined as: Lcls (pi , p∗i ) = − log[pi p∗i + (1 − pi )(1 − p∗i )]
(4)
The regression loss is defined as: Lreg (ti , ti∗ ) =
i ∈ {x,y,w,h}
smoothL1 (ti , ti∗ )
(5)
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785
5
Table 1 Distribution of colposcopic image data. Items
Nor/CIN1
CIN2
CIN3
Cancer
Total
Train Test Total
2528 722 3250
1213 346 1559
980 280 1260
374 106 480
5095 1441 6536
Table 2 Distribution of data in cervical lesions. Items
Nor/CIN1
CIN2
CIN3
Cancer
LR Total
Train Test LR Total
0 0 0
2061 589 2650
1666 476 2142
384 118 502
4111 1183 5294
Fig. 6. Distribution of TP, TN, FN and FP. TP, FP and TN, FN are the true positive rate, the false positive rate and the true negative rate,the false negative rate respectively.
smoothL1 (x) is:
S(x) =
0.5x2 , |x| < 1 |x| − 0.5, |x| ≥ 1
((6))
The sizes of bounding box seted as (28×28, 56×56, 112×112, 224×224) and four length and width ratios (1:1, 1:2, 1.5:1, 2:1). There are 16 anchor boxes in each position to improve the accuracy of the bounding box obtained by LRPN.
5. Experiments and analysis 5.1. Data We selected the cervical cancer screening data for the last two years. The volunteers were aged between 21 and 65 years old. The data included colposcopic images and complete pathological diagnosis report. Doctors who make standard data sets are colposcopy experts with more than 10 years of work experience. We have obtained approval from the local ethics committee and have been authorized by the volunteers for scientific research. A colposcope is an optical instrument that penetrates epithelial cells by intense light to amplify the cervical epithelium and subcutaneous blood vessels. By observing the reflected light, the doctor can judge the changes of the cells and blood vessels on the surface of the cervical epithelial cells, find out the possible areas of cervical lesions, and evaluate the nature and type of the lesions. In the present study, 1109 cases of cervical cancer screening par¨ ¨ ticipants are collected by the gynecologic oncology laboratoryof ¨ Fujian maternal and child health hospital¨. Then, 817 cases are selected for the experiment through screening by a colposcopy expert. A total of 6536 Colposcopic images are marked by experts, in this regards. Among them, the training data are 5095 sheets, including 2567 negative images. Moreover, in order to ensure the robustness of the model, 2528 cervical confirmed to be positive images are added to the training data. The experiment used five types of colposcopic images: Normal, CIN1, CIN2, CIN3 and Cancer. Totally, 1441 images are used in the present study. The total number of images (Total) presents the sum of images participating in the model training and testing, as shown in Table 1. The total number of cervical lesions region (LR Total) indicates the sum of the number of lesions in image data. It should be indicated that lesions of Normal and CIN1 types do not contain lesions, which play a role in optimizing the generalization ability of the model. The total number of lesions used for training, number of CIN2 and CIN3 types are 4111, 2061 and 1666, respectively. Moreover, there are few cervical cancer patients and 384 cancerous regions. 5294 lesion regions are used for training and testing, as shown in Table 2.
5.2. Experimental results In order to improve the speed of the model training, 224×224 RGB format colposcopy pictures are used as the experimental input. The experimental data are marked by experts to ensure the accuracy and authority of the experiment. In order to improve the robustness of the model, five types of Colposcopic images and annotation information of Normal, CIN1, CIN2, CIN3 and Cancer are used as training data. Considering the clinical application, test data are taken from the cervical lesion images. The CLDNet is designed to help doctors to identify region of the lesion for a more accurate diagnosis. To illustrate the superiority of the model in the extraction of cervical lesions, the HOG + SVM and Faster RCNN algorithms are used as comparative experiments. We use the non-maximum suppression method (NMS) to select 300 candidate frames with higher scores for the generated bounding box and the confidence score between 0.8 and 0.5. The precision and recall rates are set in accordance with the model evaluation criteria [33]. The precision rate (Precision) indicates the proportion of confirmed lesions in all identified lesions. While the recall rate (Recall) indicates the proportion of correctly identified lesions in the confirmed lesions. Fig. 6 shows the number of TP, FP, TN and FN contained in the CIN2, CIN3 and Cancer cervical lesion images in the test results. The purpose of this study is to detect and locate lesions in the cervical lesion image and to test the three types of cervical lesions of CIN2, CIN3 and Cancer. The CLDNet model is based on the improved structure of the cervical lesion region obtained by the Faster RCNN algorithm. This experiment uses the Faster RCNN algorithm as a comparative experiment. The algorithm takes deep features of the cervical image as the input and discriminates the lesion region. We have used the HOG + SVM algorithm to detect cervical lesions. The algorithm uses the histogram of oriented gradient (HOG) as the image feature and uses the SVM as the classifier to extract the cervical lesion region. The present study compares results of the golden truth (GT), cervical lesion detection algorithm (CLDNet) and two object detection algorithms mentioned above and displays them through relevant images, as shown in Fig. 7. A, B and C present results of CIN2, CIN3 and Cancer, respectively. In Fig. 7, the black box in the GT (Ground Truth) column diagram is an expert’s labeling of the lesion region. The red box in the CLDNet column diagram is the labeling of the lesion region by the algorithm in this paper. The text above the box indicates the lesion region, and the number indicates that the label box is the fit of the lesion region. The Faster RCNN column showed results similar to CLDNet. The HOG + SVM algorithm results in a green box to mark the lesion region.
6
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785
Fig. 7. Results of lesion region detection. A, B, and C are results of CIN2, CIN3 and Cancer, respectively.
In order to demonstrate that CLDNet algorithm is superior to Faster RCNN and HOG + SVM algorithm in extracting cervical lesions, the number of lesions diagnosed by each algorithm is com-
pared with results of the expert labeling (Ground Truth, GT). The true positive (TP), false positive (FP) and false negative rates (FN) are obtained [35]. Corresponding precision and the recall rates are
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785 Table 3 Detection precisions for different algorithms (Precision, %). Models
CIN2
CIN3
Cancer
Average
Hog + SVM Faster RCNN YOLOv3 SSD CLDNet(ours)
73.26 86.25 85.34 83.69 89.29
78.67 90.53 88.56 86.97 92.16
81.35 95.02 94.73 92.41 96.15
77.67 90.60 89.54 87.69 92.53
Table 4 Detection recall rates for different algorithms (Recall, %). Models
CIN2
CIN3
Cancer
Average
Hog + SVM Faster RCNN YOLOv3 SSD CLDNet(ours)
61.51 79.71 76.24 78.54 81.97
57.32 71.82 73.48 74.87 76.15
77.39 89.63 95.57 91.46 98.57
65.41 80.39 81.76 81.62 85.56
7
boundary features, and reduces the overlap of labeling boxs in the results. Analysis showed that the cervical boundary features and highlight regions have a greater impact on the generation of the labeling box. This paper proposes ways to suppress the useless features to enhance the characteristics of significant lesions, attenuate these effects and speed up the convergence of the labeled boxes. The algorithm in this paper improves the precision and recall rate of lesion detection. The feature recalibration mechanism in CLDNet redistributes the features of the lesion image based on the depth features. On the one hand, the utilization of the lesion features is improved, and on the other hand, the influence of the cervical border and the highlight region on the detection is suppressed. From the test performance of the model, the algorithm has a good auxiliary effect on cervical cancer screening.
6. Conclusion calculated, as shown in Tables 3 and 4. Table 3 presents the precision and average precision of three algorithms in diagnosing CIN2, CIN3 and Cancer. It is observed that the precision of the Cancer class is the highest in three algorithms and the average precision of the CLDNet is higher than the one for other methods. Tables 3 and 4 indicate that the precision and recall rates of the target detection algorithm based on the deep learning are higher than those from the HOG + SVM algorithm. Table 5 shows the mAP and IOU values in the detection. We have compared the More advanced detection algorithm, such as YOLOv3 and SSD. Our algorithm is improved according to the two-stage algorithm, so it is compared with the Faster RCNN algorithm. HOG + SVM is compared with our CNN algorithm as a classical feature extraction algorithm. The results of the different algorithms are shown in Tables 3–5. 5.3. Analysis and discussion This paper, a variety of algorithms were used to detect the lesions in CIN2, CIN3 and Cancer, the experimental results are shown in Fig. 7. Image of a CIN2 cervical lesion in A. The orange callout box generated by Faster RCNN showed that the incompleteness of the significant feature extraction leads to partial superimposition of the labeling box. Moreover, the incomplete suppression of the high-light region as the interference region causes the redundant region to be marked. The green label box generated by the HOG + SVM algorithm showed that the label boxes in the CIN2 image overlap heavily. The irregular shape of the lesion region leads to the inconspicuous convergence effect of the labeled box generated by HOG + SVM. In CIN3 and Cancer-like cervical lesions, Faster RCNN does not adequately suppress the characteristics of the cervical border and highlight regions, and boxs unused borders and highlights. Labeling box overlaps are prone to occur in Cancerlike cervical lesion images. The HOG + SVM algorithm also has a label box overlay when labeling CIN3, and it is missed by high light and some lesion regions are missed. Missing detection and overlap detection due to large boundary interference in Cancer images. The algorithm of this paper solves the defects of the above two types of algorithms, weakens the interference of highlight and cervical
The present work employs the deep learning algorithm based on the Faster RCNN structure to detect the cervical lesion region. The CLDNet algorithm applies the object detection algorithm to the detection of lesion region and improves the precision of the algorithm by improving the feature extraction process. The cervical lesion region detection algorithm (CLDNet) applies the feature recalibration method to enhance salient features and suppress the non-significant features, thereby achieving accurate detection and extraction of the lesion region. According to the performed experiment, the cervical lesion region extraction algorithm based on the feature recalibration can effectively detect the lesion region. The risk of developing precancerous lesions of the cervix as early invasive carcinoma increases with time and is unpredictable [24]. The researchers used the global features of the cervical image as the basis for judging the lesion level, which is not conducive to further clinical diagnosis [19–21]. Our method mitigates this risk by finding smaller lesions earlier and more accurately. The lesion region detected by the method provide a more accurate judgment basis for the doctor’s diagnosis, and the region includes important information for classification of the lesion level such as vinegar white and blood vessels.
Acknowledgments This work was supported by Promotion Program for Young and Middle-aged Teacher in Science and Technology Research of Huaqiao University (No. ZQN-PY518). The grants from National Natural Science Foundation of China (Grant No. 61605048) in part by Natural Science Foundation of Fujian Province, China under Grant 2015J01256, and Grant 2016J01300. In part by Fujian Provincial Big Data Research Institute of Intelligent Manufacturing, in part by the Quanzhou scientific and technological planning projects (No. 2018C113R, 2018N072S and No. 2017G024), and in part by the Subsidized Project for Postgraduates’ Innovative Fund in Scientific Research of Huaqiao University under Grant 17014084001.
Table 5 mAP and Average IOU for different algorithms. Models Hog + SVM Faster RCNN YOLOv3 SSD CLDNet(ours)
CIN2 69.22 73.84 75.92 70.37 78.39
CIN3
Cancer
mAP
Average IOU
66.28 75.59 73.67 72.57 79.27
71.46 78.25 81.73 79.86 83.46
68.99 75.89 77.11 74.27 80.37
68.24 74.51 77.65 76.18 79.47
8
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785
Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. References [1] WHO, Human Papillomavirus and Related Cancers in the World, in: Summary Report, ICO Information Centre on HPV and Cancer, 2014, August. [2] F. Bray, J. Ferlay, I. Soerjomataram, et al., Global Cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries, CA Cancer J. Clin. 68 (2018) 394–424. [3] H.Y. Sung, Papanicolaou smear history and diagnosis of invasive cervical carcinoma among members of a large prepaid health plan, J. Low. Genit. Tract Dis. 5 (2) (2010), 112-112. [4] W.A. Leyden, M.M. Manos, A.M. Geiger, et al., Cervical Cancer in Women With Comprehensive Health Care Access: Attributable Factors in the Screening Process, JNCI J. Natl. Cancer Inst. 97 (9) (2005) 675–683. [5] Centers for Disease Control and Prevention. Cervical cancer. Available at: https://www.cdc.gov/dotw/cervical-cancer/index.html. Retrieved April 2, 2015. [6] A.R. Spence, P. Goggin, E.L. Franco, Process of care failures in invasive cervical cancer: systematic review and meta-analysis, J. Low. Genit. Tract Dis. 45 (2) (2008) 93–106. [7] H.S. Cronjé, Screening for cervical cancer in developing countries, BMJ 84 (2) (1992) 101–108. [8] S.Y. Park, D. Sargent, R. Lieberman, et al., Domain-specific image analysis for cervical neoplasia detection based on conditional random fields, IEEE Trans. Med. Imaging 30 (3) (2011) 867–878. [9] C. Porras, N. Wentzensen, Ana C. Rodríguez, et al., Switch from cytology-based to human papillomavirus test-based cervical screening: implications for colposcopy, Int. J. Cancer 130 (8) (2012) 1879–1887. [10] Shiri Gordon, G. Zimmerman, H. Greenspan, Image segmentation of uterine cervix images for indexing in PACS, IEEE Symposium on Computer-Based Medical Systems IEEE Computer Society (2004). [11] Wenjing Li, A. Poirson, Detection and characterization of abnormal vascular patterns in automated cervical image analysis, in: International Symposium on Visual Computing, Springer, Berlin Heidelberg, 2006. [12] G. Litjens, T. Kooi, B.E. Bejnordi, et al., A survey on deep learning in medical image analysis, Med. Image Anal. 42 (2017) 60–88. [13] I. Vourlaki, C. Balas, G. Livanos, et al., Bootstrap clustering approaches for organization of data: application in improving grade separability in cervical neoplasia, Biomed. Signal Process. Control 49 (2019) 263–273. [14] N.A. Obukhova, A. Motyko, U. Kang, et al., Automated image analysis in multispectral system for cervical cancer diagnostic Conference of Open Innovations Association, 20, 2017, pp. 345–351. [15] D.B. Patil, M.S. Gaikwad, D.K. Singh, et al., Semi-automated lession grading in cervix images with specular reflection removal International Conference on Inventive Computation Technologies, 1, 2016, pp. 1–5. [16] S. Huang, M. Gao, D. Yang, et al., Unbalanced graph-based transduction on superpixels for automatic cervigram image Segmentation IEEE, International Symposium on Biomedical Imaging, ISBI, 12, 2015, pp. 1556–1559.
[17] P.S.R. Praba, H. Ranganathan, Wavelet transform based automatic lesion detection in cervix images using active contour, J. Comput. Sci. 9 (1) (2013) 30–36. [18] S. Zhang, J. Huang, W. Wang, et al., Cervigram image segmentation based on reconstructive sparse representations, Medical Imaging Image Process. (2015) 7623. [19] T. Xu, H. Zhang, X. Huang, et al., Multimodal deep learning for cervical dysplasia diagnosis International Conference on Medical Image Computing and Computer-Assisted Intervention, 19, Springer, Cham, 2016, pp. 115–123. [20] T. Xu, H. Zhang, C. Xin, et al., Multi-feature based benchmark for cervical dysplasia classification evaluation, Pattern Recognit. 63 (2017) 468–475. [21] K.A. Correia, J. Cardoso, Transfer learning with partial observability applied to cervical cancer screening Iberian Conference on Pattern Recognition and Image Analysis, 8, 2017, pp. 243–250. [22] T. Hirasawa, K. Aoyama, T. Tanimoto, et al., Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images, Gastric Cancer 21 (Suppl 1) (2018) 1–8. [23] M. Jahic, A. Balic, Corelation between cytologic, colposcopic and pathohistological findings of cervical intraepithelial lesions, Med. Arh. 66 (2) (2012) 122–124. [24] M. Munmany, L. Marimon, M. Cardona, et al., Small lesion size measured by colposcopy may predict absence of cervical intraepithelial neoplasia in a large loop excision of the transformation zone specimen, Bjog Int. J. Obstet. Gynaecol. 124 (3) (2016) 495–502. [25] A.H. El-Shalakany, M.M. Saeed, M.R. Abdel-Aal, et al., Direct visual inspection of the cervix with Lugol iodine for the detection of premalignant lesions, J. Low. Genit. Tract Dis. 12 (3) (2008) 193–198. [26] G.F. Vercellino, E. Erdemoglu, V. Chiantera, et al., Clinical rele-vance of objectifying colposcopy, Arch. Gynecol. Obstet. 291 (4) (2015) 907–915. [27] C. Scheungraber, K. Glutig, B. Fechtel, et al., Inner border-a specific and significant colposcopic sign for moderate or severe dysplasia (cervical intraepithelial neoplasia 2 or 3), J. Low. Genit. Tract Dis. 13 (1) (2009) 1–4. [28] G.F. Vercellino, E. Erdemoglu, V. Chiantera, et al., Validity of the colposcopic criteria inner border sign, ridge sign, and rag sign for detection of high-grade cervical intraepithelial neoplasia, Obstet. Gynecol. 121 (3) (2013) 624–631. [29] J. Sláma, The new colposcopic signs-ridge sign and inner border, Ceska Gynekol. 77 (1) (2012) 22–24. [30] C. Scheungraber, U. Koenig, B. Fechtel, et al., The colposcopic feature ridge sign is associated with the presence of cervical intraepithelial neoplasia 2/3 and human papillomavirus 16 in young women, J. Low. Genit. Tract Dis. 13 (1) (2009) 13–16. [31] S. Ren, K. He, R. Girshick, et al., Faster R-CNN: towards Real-time object detection with region proposal networks, in: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, MIT Press, 2015. [32] J. Hu, L. Shen, S. Albanie, et al., Squeeze-and-Excitation Networks, CVPR, 2018. [33] K. He, X. Zhang, S. Ren, et al., Deep Residual Learning for Image Recognition, CVPR, 2016. [34] T. Kong, A. Yao, Y. Chen, et al., HyperNet: Towards Accurate Region Proposal Generation and Joint Object Detection, CVPR, 2016. [35] Bing Bai, Pei-Zhong Liu, Yong-Zhao Du, Yan-Ming Luo, et al., Automatic segmentation of cervical region in colposcopic images using K-means, Australas. Phys. Eng. Sci. Med. 41 (2018) 1077–1085.
Contents lists available at ScienceDirect
Biomedical Signal Processing and Control journal homepage: www.elsevier.com/locate/bspc
Detection of cervical lesion region from colposcopic images based on feature reselection Bing Bai a , Yongzhao Du a,b,∗ , Peizhong Liu a,b , Pengming Sun c , Ping Li d , Yuchun Lv d a
College of Engineering, Huaqiao University, Quanzhou, China College of Medicine, Huaqiao University, Quanzhou, China Fujian Provincial Maternity and Children’s Hospital, Affiliated Hospital of Fujian Medical University, Fuzhou, China d Department of Gynecology and Obstetrics, The First Hospital of Quanzhou, Fujian Medical University, Quanzhou, China b c
a r t i c l e
i n f o
Article history: Received 27 May 2019 Received in revised form 11 October 2019 Accepted 16 November 2019 Keywords: Cervical lesion Deep learning Object detection SE-CNN CLDNet
a b s t r a c t Colposcopy is one of the important steps in the clinical screening of cervical intraepithelial neoplasia (CIN) and early cervical cancer. It directly affects the patient’s diagnosis and treatment program. Therefore, it is widely used for cervical cancer screening. The present work proposes a cervical lesion detection net (CLDNet) model based on the deep convolutional neural network (CNN). The Squeeze-Excitation convolutional neural network (SE-CNN) employed to extract depth features of the whole image. SE module for feature recalibration. Moreover, the region proposal network (RPN) generated a proposal box of the region of interest (ROI). Finally, the region of interest classified and proposal box regression performed to locate the cervical lesion region. The Squeeze-Excitation (SE block) strengthened important features and suppress non-primary features, improve feature extraction ability, which is beneficial to feature classification and proposal box regression in the regions of interest. It is found that the average precision of the model extraction lesion region is 92.53 % and the average recall rate is 85.56 %, which can play a good role in the auxiliary diagnosis. © 2019 Published by Elsevier Ltd.
1. Introduction Cervical cancer ranks as the fourth most common type of cancers worldwide in women aged 15–44 years [1]. Moreover, it is expected that the global cancer burden will further aggravated. According to the report, almost half of the world’s new cancer cases and more than half of all cancer deaths occurred in Asia in 2018 [2]. Among all types of cancer, cervical cancer incidence and mortality ranked fourth. However, most women with cervical cancer have never been screened or screened inadequately worldwide [3,4]. Studies show that about 50 % of cervical cancer patients have never undergone the cervical cytology and another 10 % have not screened for cervical cancer within five years before the cervical cancer diagnosis [5,6]. Therefore, a large-scale and standard screening for the general population is one of the most effective ways to reduce the incidence and death of cervical cancer. The process of the cervical cancer screening is highly complex and costly, resulting in the inability to popularize more advanced cervical cancer screening techniques in under-resourced settings.
∗ Corresponding author. E-mail addresses: [email protected] (Y. Du), [email protected] (P. Liu). https://doi.org/10.1016/j.bspc.2019.101785 1746-8094/© 2019 Published by Elsevier Ltd.
Therefore, the morbidity and mortality of cervical cancer are still high in underdeveloped regions [7,8]. According to recommendations of the World Health Organization, there are currently three types of cervical cancer/lesion screening methods, including human papillomavirus (HPV) testing, Pap smear testing and vinegar based colposcopy. The first two methods are relatively complex and expensive, especially for patients in developing countries [9]. Even in the first two methods, colposcopy is still a necessary step in the biopsy. Therefore, colposcopy is a necessary and most effective tool for cervical cancer screening. Colposcopy is the primary diagnostic method used to detect cervical cancer in women. It can identify and determine the severity of the lesion so that a biopsy of the highest level of abnormality detection can be taken, when necessary. Colposcopy includes a visual assessment of the system of the lower genital tract (cervical, vulva and vagina), primarily for the appearance of a metaplastic epithelium, consisting of a transformation zone on the cervix [10]. During the examination, a volumetric vinegar solution is applied to the cervix, resulting in abnormal whitening of the metaplastic epithelial tissue. Cervical precancerous lesions and invasive cancers exhibit significant abnormal morphological features that can be identified by the colposcopy. Pathological features of the cervical epithelial tissue, such as color characteristics, opacity, edge division
2
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785
Fig. 1. Five colposcopic images (Normal, CIN1, CIN2, CIN3 and Cancer).
and tissue shape, should be observed by an expert doctor expert to obtain a clinical diagnosis. Due to the subjective nature of the examination, the accuracy of the colposcopy is highly dependent on the doctor’s experience and expertise. Therefore, the colposcopy has low specificity and requires many unnecessary biopsies [11]. Cervical images observed by the colposcopy during the cervical cancer screening are divided into five categories, including normal, CIN1, CIN2, CIN3 and cancerous images. Fig. 1 showed that each type of colposcopic images has its own characteristics. 2. Related work Computer vision related technologies have been widely used in the field of the medical image analysis. They are mainly focusing on classification and reconstruction of lesion images, segmentation of lesions, detection and localization of lesions [12]. Currently, the application of the screening for cervical lesions are divided into cervical cancer screening techniques based on celllevel and colposcopy-based images [13]. Moreover, the cervical cancer screening based on colposcopy pictures is the most active research and includes the cervical segmentation and the cervical lesion classification. Based on conventional image processing methods and the application requirements of the clinical cervical cancer screening, NA Obukhova et al. [14] proposed a method to embed the artificial intelligence technology into the colposcope so that the colposcopic imaging device can automatically analyze the image, processed by the vinegar. The key premise is the segmentation of the cervical region and the development of a cervical region of interest (ROI) segmentation procedure. The device obtains colposcopy pictures for different light wavelengths and removes highlights from the image. Then the device obtains diagnostic results by performing a cross-analysis of the discrete Fourier transforms on different spectrograms. In the research project of Mr. Deepak. B. Patil et al. [15], a special camera with a green filter was used to collect the colposcopic image by setting the pixel with higher color gradient as the specular reflection region and performing filled pixels to remove the highlight region. Finally, the K-means was used to perform the ROI region segmentation on the colposcopic image. Moreover, Huang S et al. proposed a transformation learning algorithm based on the image pixel classification [16], which can roughly segment the cervical region. However, the proposed algorithm neglects the
removal and filling of the specular reflection region during the colposcopy imaging. Rama Praba et al. [17] applied a Gaussian mixture modeling method based on the mathematical morphology to eliminate irrelevant information and specular reflection in colposcopy pictures, prior to extracting features. Moreover, they applied the image segmentation based on the sparse reconstruction to segment the cervical region (ROI) [18]. Researches showed that the cervical region obtained by the above algorithm still needs to be classified and detected by the image processing technology, so as screening cervical cancer. Recently, CNN models have been widely used in the medical image analysis. For the classification of cervical lesion images, Tao Xu et al. [19] combined cervical images with Pap and HPV test results to construct a multimodal cervical lesion classification model. Similarly, they combined the PLAB, PHOG and PLBP multi-feature fusion with multiple machine learning classifiers to investigate the cervical cancer screening [20]. Correia et al. [21] proposed a regularized migration learning strategy to classify cervical images. The algorithm realized parameter sharing and improved the accuracy of the image classification. With the improvement of the auxiliary diagnosis technology, Toshiaki Hirasawa et al. [22] constructed a model for the endoscopic image detection based on the single shot multi-box detector (SSD) structure. They showed that the proposed algorithm is able to mark the lesion region in the gastroscopic image. In order to improve the accuracy of the cervical cancer intelligent assisted screening, a deep learning model based on the CNN is proposed in the present study. The proposed model can automatically calibrate cervical lesions. This model can mark the location of cervical lesions and estimate the lesion region. The algorithm applies the attention mechanism to the feature extraction of the lesion region, aiming to reduce the interference of the non-lesional region on the process. Combined with the advice of the expert doctor, relevant parameters of calibration lesion regions are tuned to achieve accurate labeling of the cervical lesion region. 3. Analysis of cervical lesions Colposcopy is an early tool for screening cervical cancer and precancerous lesions. Various colposcopic abnormal images have reference significance in the cervical screening. According to the risk degree, the hazard order of these abnormal images are [23]:
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785
3
Fig. 2. Multiple cervical lesion features. A: Punctate vessels, intraepithelial cells proliferate upwardly. B: Internal boundary, the internal vinegar white boundary of the lesion. C: Ridge-like sign, the opaque bulge of the white epithelium in the transformation zone.
Fig. 3. The overall framework of the model (CLDNet).
vinegar white epithelium, point blood vessels, cuff-like gland opening, inlay, iodine test without coloration, internal boundary, ridge-like signs and abnormal blood tube. Fig. 2 illustrates the risk degree for a cervical lesion. Colposcopy needs to initially identify the transformation zone. Then, in order to determine the lesion nature and necessity of a biopsy, it identifies whether there is an abnormal image in the transformation zone [24]. Dynamic observation of cervical epithelial vessels to find abnormal images to judge is an important task of the colposcopy. Different abnormal images reveal different risks and should be screened and treated. Acetowhite Epithelium original: The white change in the epithelium after application of the vinegar solution is called the white acetate epithelium, Punctate vessels: Fig. 2(A) showed that intraepithelial cells proliferate upwardly to compress blood vessels and form an image of fine red dots. Cuff-like gland opening: A thickened bulging white ring around the opening of the cervix gland is a signal that the lesion affects the gland. In this case, a deep biopsy should be performed, as shown in Fig. 2(B). Internal boundary: The internal vinegar white boundary of the lesion refers to the thickness of the Acetowhite Epithelium present in the same lesion region. Researches show that 7.6 % and 70 % of internal boundaries are pathologically confirmed as CIN2 and CIN3, respectively [25]. Ridge-like sign: refers to the opaque bulge of the white epithelium in the transformation zone, indicating abnormal proliferation of the lesion, as shown in Fig. 2(C). CIN1: Relatively thin white acetate white epithelium with blurred edges and small spotted blood vessels and small fuzzy inlays. CIN2: Acetate white epithelium is thick, dirty, opaque, oyster white or gray, the boundary of the lesion is sharp, well-defined, visible large point blood vessels and inlays. CIN3: A thick white epithelium appears in the columnar epithelial area, with thick punctate blood vessels and large irregular mosaics. Cancer: dense, thick grayish white or yellowish white acetate epithelium and singularly shaped blood vessels [26–30]. Clinical observations show that high-grade cervical lesions are mostly located in the cervical transformation region. It is extremely
important to observe the transformation of the transformation region in colposcopy. Lesions that are remote from the transformation zone are usually metastatic lesions or multicenter lesions. Colposcopy screening requires a description of the number of quadrants involved in the lesion and the percentage of lesion region occupying the surface region of the cervix, with a clock as a marker. It is found that the greater is the lesion extent, the greater is the likelihood of potentially high-grade lesions. Based on the above analysis on images of various lesions of the cervix, the CLDNet model proposed in the present study can effectively extract the cervical transformation region in the colposcopy image. 4. Methods 4.1. Model Fig. 3 illustrates the overall framework of the proposed CLDNet. The model is designed and improved according to the structure of the Faster RCNN [31], which is mainly composed of two parts. The first part is the deep feature extraction CNN, called SE-CNN network, which joins the SE block. The second part is the RPN that generates the region of interest (ROI). The region of interest is the cervical lesion region. The proposal box generated by the second part is mapped to feature maps, generated by the first part. Meanwhile, the RPN shares the weight with the SE-CNN to improve the training speed of the model. 4.2. Image features extraction The proposed feature extraction network of the lesion region aims to extract features of the lesion region in the cervical image. Since the classification network in the algorithm ultimately needs to determine whether the ROI belongs to the lesion region according to the feature, the CNN is selected. Shallow features of the model are used as the basis for the classification of lesions. In order to
4
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785
Fig. 5. LRPN implementation process.
be indicated that ı, W1 and W2 are the ReLU function, dimensionality reduction parameters required for the first and second full join operations, respectively. 4.3. Lesion region proposal network (LRPN)
Fig. 4. Feature Extraction Network (SE-CNN).
achieve better results, a feature reselection strategy is added, based on the basic CNN model [32]. It is intended to acquire the importance of each feature channel through learning and then enhance useful features according to this importance and suppress features that are of little use to the current task. Fig. 4 schematically illustrates the feature extraction network in the model of the present work. The basic CNN model of the feature extraction network adopts the ResNet deep learning model [33], selects feature maps of the first, third and fifth convolution operations of the model and selects non-adjacent feature layers for the purpose of avoiding the feature overlap [34]. Meanwhile, in order to improve the utilization of extracted features, SE block [32] is added between convolution operations. In order to reselection important features after obtaining the global receptive field, the module mainly includes Squeeze and Excitation operations. The Squeeze operation compresses each feature layer by a global average pooling operation after the convolution operation so that the C feature maps converted to the real number sequence of 1*1*C, as the following: zc = Fsq (uc ) =
1 W ×H
H W
The LRPN takes an image with arbitrary scale as the input and prepares a set of rectangular target suggestion boxes, as the output. Each box contains four positions, coordinate variables and a score. In order to generate a region suggestion box, the input image is generated through a shared convolution layer to generate a feature map. Then, a multi-scale convolution operation is performed on the feature map. The specific implementation process is as the following (Fig. 5): three kind of scales and three aspect ratios are used at the position of each sliding window. Centered on the center of the current sliding window. Moreover, corresponding to each scale and aspect ratio, mapping nine different scale candidate regions on the original image. For example, for a shared convolutional feature map size of W×H, there are W×H×9 proposal regions. Finally, the classifier prepares scores of W×H×2×9 proposal regions. Scores show probability that the region belongs to the lesion region. Moreover, the regression layer outputs W×H×4×9 parameters, which show coordinate parameters of the proposal region. When training the LRPN network, each candidate lesion region is assigned a binary label to indicate whether the region is lesion. The specific operations are as follows: 1) Candidate regions with the highest intersection-over-union (IOU) overlap with a true ground truth (GT). 2) IOU overlaps regions with any GT bounding box greater than 0.7. A negative label is assigned to a candidate region with an IOU ratio of less than 0.3 for all GT bounding boxes. 3) The region between two IOUs is discarded. The loss function defined in joint training: L({pi }, {ti }) =
1 1 ∗ Lcls (pi , p∗i ) + pi Lreg (ti , ti∗ ) Ncls Nreg i
uc (i, j)
(1)
i=1 j=1
Eq. (1) presents the average pooling operation of the convolved feature layer. Where c is the number of channels of the feature layer. In other words, it is the average pooling operation for each feature layer. W and H respectively indicate the width and height of the feature layer. Moreover, Zc is the weight of the feature layer to redistribute features of each layer to achieve enhancement of useful features and suppress unnecessary features, as shown in Eq. (2). Fex (z, W ) = (g(z, W )) = (W2 ı(W1 z))
(3)
i
(2)
Eq. (2) presents the Excitation operation in the SE block. It initially performs a full connection operation. Then it passes the ReLU function and finally performs a full connection operation. It should
In Eq. (3), i is the proposed frame index selected in one batch, and pi represents the probability that the proposed frame i is the target. If the proposed box is a positive label, its corresponding real area label pi * is 1, otherwise pi * is 0. ti represents the four parameterized coordinate vectors of the predicted bounding box, and ti * is the coordinate vector of the corresponding real region bounding box. The classification loss Lcls is the logarithmic loss for two categories (target and non-target), defined as: Lcls (pi , p∗i ) = − log[pi p∗i + (1 − pi )(1 − p∗i )]
(4)
The regression loss is defined as: Lreg (ti , ti∗ ) =
i ∈ {x,y,w,h}
smoothL1 (ti , ti∗ )
(5)
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785
5
Table 1 Distribution of colposcopic image data. Items
Nor/CIN1
CIN2
CIN3
Cancer
Total
Train Test Total
2528 722 3250
1213 346 1559
980 280 1260
374 106 480
5095 1441 6536
Table 2 Distribution of data in cervical lesions. Items
Nor/CIN1
CIN2
CIN3
Cancer
LR Total
Train Test LR Total
0 0 0
2061 589 2650
1666 476 2142
384 118 502
4111 1183 5294
Fig. 6. Distribution of TP, TN, FN and FP. TP, FP and TN, FN are the true positive rate, the false positive rate and the true negative rate,the false negative rate respectively.
smoothL1 (x) is:
S(x) =
0.5x2 , |x| < 1 |x| − 0.5, |x| ≥ 1
((6))
The sizes of bounding box seted as (28×28, 56×56, 112×112, 224×224) and four length and width ratios (1:1, 1:2, 1.5:1, 2:1). There are 16 anchor boxes in each position to improve the accuracy of the bounding box obtained by LRPN.
5. Experiments and analysis 5.1. Data We selected the cervical cancer screening data for the last two years. The volunteers were aged between 21 and 65 years old. The data included colposcopic images and complete pathological diagnosis report. Doctors who make standard data sets are colposcopy experts with more than 10 years of work experience. We have obtained approval from the local ethics committee and have been authorized by the volunteers for scientific research. A colposcope is an optical instrument that penetrates epithelial cells by intense light to amplify the cervical epithelium and subcutaneous blood vessels. By observing the reflected light, the doctor can judge the changes of the cells and blood vessels on the surface of the cervical epithelial cells, find out the possible areas of cervical lesions, and evaluate the nature and type of the lesions. In the present study, 1109 cases of cervical cancer screening par¨ ¨ ticipants are collected by the gynecologic oncology laboratoryof ¨ Fujian maternal and child health hospital¨. Then, 817 cases are selected for the experiment through screening by a colposcopy expert. A total of 6536 Colposcopic images are marked by experts, in this regards. Among them, the training data are 5095 sheets, including 2567 negative images. Moreover, in order to ensure the robustness of the model, 2528 cervical confirmed to be positive images are added to the training data. The experiment used five types of colposcopic images: Normal, CIN1, CIN2, CIN3 and Cancer. Totally, 1441 images are used in the present study. The total number of images (Total) presents the sum of images participating in the model training and testing, as shown in Table 1. The total number of cervical lesions region (LR Total) indicates the sum of the number of lesions in image data. It should be indicated that lesions of Normal and CIN1 types do not contain lesions, which play a role in optimizing the generalization ability of the model. The total number of lesions used for training, number of CIN2 and CIN3 types are 4111, 2061 and 1666, respectively. Moreover, there are few cervical cancer patients and 384 cancerous regions. 5294 lesion regions are used for training and testing, as shown in Table 2.
5.2. Experimental results In order to improve the speed of the model training, 224×224 RGB format colposcopy pictures are used as the experimental input. The experimental data are marked by experts to ensure the accuracy and authority of the experiment. In order to improve the robustness of the model, five types of Colposcopic images and annotation information of Normal, CIN1, CIN2, CIN3 and Cancer are used as training data. Considering the clinical application, test data are taken from the cervical lesion images. The CLDNet is designed to help doctors to identify region of the lesion for a more accurate diagnosis. To illustrate the superiority of the model in the extraction of cervical lesions, the HOG + SVM and Faster RCNN algorithms are used as comparative experiments. We use the non-maximum suppression method (NMS) to select 300 candidate frames with higher scores for the generated bounding box and the confidence score between 0.8 and 0.5. The precision and recall rates are set in accordance with the model evaluation criteria [33]. The precision rate (Precision) indicates the proportion of confirmed lesions in all identified lesions. While the recall rate (Recall) indicates the proportion of correctly identified lesions in the confirmed lesions. Fig. 6 shows the number of TP, FP, TN and FN contained in the CIN2, CIN3 and Cancer cervical lesion images in the test results. The purpose of this study is to detect and locate lesions in the cervical lesion image and to test the three types of cervical lesions of CIN2, CIN3 and Cancer. The CLDNet model is based on the improved structure of the cervical lesion region obtained by the Faster RCNN algorithm. This experiment uses the Faster RCNN algorithm as a comparative experiment. The algorithm takes deep features of the cervical image as the input and discriminates the lesion region. We have used the HOG + SVM algorithm to detect cervical lesions. The algorithm uses the histogram of oriented gradient (HOG) as the image feature and uses the SVM as the classifier to extract the cervical lesion region. The present study compares results of the golden truth (GT), cervical lesion detection algorithm (CLDNet) and two object detection algorithms mentioned above and displays them through relevant images, as shown in Fig. 7. A, B and C present results of CIN2, CIN3 and Cancer, respectively. In Fig. 7, the black box in the GT (Ground Truth) column diagram is an expert’s labeling of the lesion region. The red box in the CLDNet column diagram is the labeling of the lesion region by the algorithm in this paper. The text above the box indicates the lesion region, and the number indicates that the label box is the fit of the lesion region. The Faster RCNN column showed results similar to CLDNet. The HOG + SVM algorithm results in a green box to mark the lesion region.
6
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785
Fig. 7. Results of lesion region detection. A, B, and C are results of CIN2, CIN3 and Cancer, respectively.
In order to demonstrate that CLDNet algorithm is superior to Faster RCNN and HOG + SVM algorithm in extracting cervical lesions, the number of lesions diagnosed by each algorithm is com-
pared with results of the expert labeling (Ground Truth, GT). The true positive (TP), false positive (FP) and false negative rates (FN) are obtained [35]. Corresponding precision and the recall rates are
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785 Table 3 Detection precisions for different algorithms (Precision, %). Models
CIN2
CIN3
Cancer
Average
Hog + SVM Faster RCNN YOLOv3 SSD CLDNet(ours)
73.26 86.25 85.34 83.69 89.29
78.67 90.53 88.56 86.97 92.16
81.35 95.02 94.73 92.41 96.15
77.67 90.60 89.54 87.69 92.53
Table 4 Detection recall rates for different algorithms (Recall, %). Models
CIN2
CIN3
Cancer
Average
Hog + SVM Faster RCNN YOLOv3 SSD CLDNet(ours)
61.51 79.71 76.24 78.54 81.97
57.32 71.82 73.48 74.87 76.15
77.39 89.63 95.57 91.46 98.57
65.41 80.39 81.76 81.62 85.56
7
boundary features, and reduces the overlap of labeling boxs in the results. Analysis showed that the cervical boundary features and highlight regions have a greater impact on the generation of the labeling box. This paper proposes ways to suppress the useless features to enhance the characteristics of significant lesions, attenuate these effects and speed up the convergence of the labeled boxes. The algorithm in this paper improves the precision and recall rate of lesion detection. The feature recalibration mechanism in CLDNet redistributes the features of the lesion image based on the depth features. On the one hand, the utilization of the lesion features is improved, and on the other hand, the influence of the cervical border and the highlight region on the detection is suppressed. From the test performance of the model, the algorithm has a good auxiliary effect on cervical cancer screening.
6. Conclusion calculated, as shown in Tables 3 and 4. Table 3 presents the precision and average precision of three algorithms in diagnosing CIN2, CIN3 and Cancer. It is observed that the precision of the Cancer class is the highest in three algorithms and the average precision of the CLDNet is higher than the one for other methods. Tables 3 and 4 indicate that the precision and recall rates of the target detection algorithm based on the deep learning are higher than those from the HOG + SVM algorithm. Table 5 shows the mAP and IOU values in the detection. We have compared the More advanced detection algorithm, such as YOLOv3 and SSD. Our algorithm is improved according to the two-stage algorithm, so it is compared with the Faster RCNN algorithm. HOG + SVM is compared with our CNN algorithm as a classical feature extraction algorithm. The results of the different algorithms are shown in Tables 3–5. 5.3. Analysis and discussion This paper, a variety of algorithms were used to detect the lesions in CIN2, CIN3 and Cancer, the experimental results are shown in Fig. 7. Image of a CIN2 cervical lesion in A. The orange callout box generated by Faster RCNN showed that the incompleteness of the significant feature extraction leads to partial superimposition of the labeling box. Moreover, the incomplete suppression of the high-light region as the interference region causes the redundant region to be marked. The green label box generated by the HOG + SVM algorithm showed that the label boxes in the CIN2 image overlap heavily. The irregular shape of the lesion region leads to the inconspicuous convergence effect of the labeled box generated by HOG + SVM. In CIN3 and Cancer-like cervical lesions, Faster RCNN does not adequately suppress the characteristics of the cervical border and highlight regions, and boxs unused borders and highlights. Labeling box overlaps are prone to occur in Cancerlike cervical lesion images. The HOG + SVM algorithm also has a label box overlay when labeling CIN3, and it is missed by high light and some lesion regions are missed. Missing detection and overlap detection due to large boundary interference in Cancer images. The algorithm of this paper solves the defects of the above two types of algorithms, weakens the interference of highlight and cervical
The present work employs the deep learning algorithm based on the Faster RCNN structure to detect the cervical lesion region. The CLDNet algorithm applies the object detection algorithm to the detection of lesion region and improves the precision of the algorithm by improving the feature extraction process. The cervical lesion region detection algorithm (CLDNet) applies the feature recalibration method to enhance salient features and suppress the non-significant features, thereby achieving accurate detection and extraction of the lesion region. According to the performed experiment, the cervical lesion region extraction algorithm based on the feature recalibration can effectively detect the lesion region. The risk of developing precancerous lesions of the cervix as early invasive carcinoma increases with time and is unpredictable [24]. The researchers used the global features of the cervical image as the basis for judging the lesion level, which is not conducive to further clinical diagnosis [19–21]. Our method mitigates this risk by finding smaller lesions earlier and more accurately. The lesion region detected by the method provide a more accurate judgment basis for the doctor’s diagnosis, and the region includes important information for classification of the lesion level such as vinegar white and blood vessels.
Acknowledgments This work was supported by Promotion Program for Young and Middle-aged Teacher in Science and Technology Research of Huaqiao University (No. ZQN-PY518). The grants from National Natural Science Foundation of China (Grant No. 61605048) in part by Natural Science Foundation of Fujian Province, China under Grant 2015J01256, and Grant 2016J01300. In part by Fujian Provincial Big Data Research Institute of Intelligent Manufacturing, in part by the Quanzhou scientific and technological planning projects (No. 2018C113R, 2018N072S and No. 2017G024), and in part by the Subsidized Project for Postgraduates’ Innovative Fund in Scientific Research of Huaqiao University under Grant 17014084001.
Table 5 mAP and Average IOU for different algorithms. Models Hog + SVM Faster RCNN YOLOv3 SSD CLDNet(ours)
CIN2 69.22 73.84 75.92 70.37 78.39
CIN3
Cancer
mAP
Average IOU
66.28 75.59 73.67 72.57 79.27
71.46 78.25 81.73 79.86 83.46
68.99 75.89 77.11 74.27 80.37
68.24 74.51 77.65 76.18 79.47
8
B. Bai, Y. Du, P. Liu et al. / Biomedical Signal Processing and Control 57 (2020) 101785
Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. References [1] WHO, Human Papillomavirus and Related Cancers in the World, in: Summary Report, ICO Information Centre on HPV and Cancer, 2014, August. [2] F. Bray, J. Ferlay, I. Soerjomataram, et al., Global Cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries, CA Cancer J. Clin. 68 (2018) 394–424. [3] H.Y. Sung, Papanicolaou smear history and diagnosis of invasive cervical carcinoma among members of a large prepaid health plan, J. Low. Genit. Tract Dis. 5 (2) (2010), 112-112. [4] W.A. Leyden, M.M. Manos, A.M. Geiger, et al., Cervical Cancer in Women With Comprehensive Health Care Access: Attributable Factors in the Screening Process, JNCI J. Natl. Cancer Inst. 97 (9) (2005) 675–683. [5] Centers for Disease Control and Prevention. Cervical cancer. Available at: https://www.cdc.gov/dotw/cervical-cancer/index.html. Retrieved April 2, 2015. [6] A.R. Spence, P. Goggin, E.L. Franco, Process of care failures in invasive cervical cancer: systematic review and meta-analysis, J. Low. Genit. Tract Dis. 45 (2) (2008) 93–106. [7] H.S. Cronjé, Screening for cervical cancer in developing countries, BMJ 84 (2) (1992) 101–108. [8] S.Y. Park, D. Sargent, R. Lieberman, et al., Domain-specific image analysis for cervical neoplasia detection based on conditional random fields, IEEE Trans. Med. Imaging 30 (3) (2011) 867–878. [9] C. Porras, N. Wentzensen, Ana C. Rodríguez, et al., Switch from cytology-based to human papillomavirus test-based cervical screening: implications for colposcopy, Int. J. Cancer 130 (8) (2012) 1879–1887. [10] Shiri Gordon, G. Zimmerman, H. Greenspan, Image segmentation of uterine cervix images for indexing in PACS, IEEE Symposium on Computer-Based Medical Systems IEEE Computer Society (2004). [11] Wenjing Li, A. Poirson, Detection and characterization of abnormal vascular patterns in automated cervical image analysis, in: International Symposium on Visual Computing, Springer, Berlin Heidelberg, 2006. [12] G. Litjens, T. Kooi, B.E. Bejnordi, et al., A survey on deep learning in medical image analysis, Med. Image Anal. 42 (2017) 60–88. [13] I. Vourlaki, C. Balas, G. Livanos, et al., Bootstrap clustering approaches for organization of data: application in improving grade separability in cervical neoplasia, Biomed. Signal Process. Control 49 (2019) 263–273. [14] N.A. Obukhova, A. Motyko, U. Kang, et al., Automated image analysis in multispectral system for cervical cancer diagnostic Conference of Open Innovations Association, 20, 2017, pp. 345–351. [15] D.B. Patil, M.S. Gaikwad, D.K. Singh, et al., Semi-automated lession grading in cervix images with specular reflection removal International Conference on Inventive Computation Technologies, 1, 2016, pp. 1–5. [16] S. Huang, M. Gao, D. Yang, et al., Unbalanced graph-based transduction on superpixels for automatic cervigram image Segmentation IEEE, International Symposium on Biomedical Imaging, ISBI, 12, 2015, pp. 1556–1559.
[17] P.S.R. Praba, H. Ranganathan, Wavelet transform based automatic lesion detection in cervix images using active contour, J. Comput. Sci. 9 (1) (2013) 30–36. [18] S. Zhang, J. Huang, W. Wang, et al., Cervigram image segmentation based on reconstructive sparse representations, Medical Imaging Image Process. (2015) 7623. [19] T. Xu, H. Zhang, X. Huang, et al., Multimodal deep learning for cervical dysplasia diagnosis International Conference on Medical Image Computing and Computer-Assisted Intervention, 19, Springer, Cham, 2016, pp. 115–123. [20] T. Xu, H. Zhang, C. Xin, et al., Multi-feature based benchmark for cervical dysplasia classification evaluation, Pattern Recognit. 63 (2017) 468–475. [21] K.A. Correia, J. Cardoso, Transfer learning with partial observability applied to cervical cancer screening Iberian Conference on Pattern Recognition and Image Analysis, 8, 2017, pp. 243–250. [22] T. Hirasawa, K. Aoyama, T. Tanimoto, et al., Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images, Gastric Cancer 21 (Suppl 1) (2018) 1–8. [23] M. Jahic, A. Balic, Corelation between cytologic, colposcopic and pathohistological findings of cervical intraepithelial lesions, Med. Arh. 66 (2) (2012) 122–124. [24] M. Munmany, L. Marimon, M. Cardona, et al., Small lesion size measured by colposcopy may predict absence of cervical intraepithelial neoplasia in a large loop excision of the transformation zone specimen, Bjog Int. J. Obstet. Gynaecol. 124 (3) (2016) 495–502. [25] A.H. El-Shalakany, M.M. Saeed, M.R. Abdel-Aal, et al., Direct visual inspection of the cervix with Lugol iodine for the detection of premalignant lesions, J. Low. Genit. Tract Dis. 12 (3) (2008) 193–198. [26] G.F. Vercellino, E. Erdemoglu, V. Chiantera, et al., Clinical rele-vance of objectifying colposcopy, Arch. Gynecol. Obstet. 291 (4) (2015) 907–915. [27] C. Scheungraber, K. Glutig, B. Fechtel, et al., Inner border-a specific and significant colposcopic sign for moderate or severe dysplasia (cervical intraepithelial neoplasia 2 or 3), J. Low. Genit. Tract Dis. 13 (1) (2009) 1–4. [28] G.F. Vercellino, E. Erdemoglu, V. Chiantera, et al., Validity of the colposcopic criteria inner border sign, ridge sign, and rag sign for detection of high-grade cervical intraepithelial neoplasia, Obstet. Gynecol. 121 (3) (2013) 624–631. [29] J. Sláma, The new colposcopic signs-ridge sign and inner border, Ceska Gynekol. 77 (1) (2012) 22–24. [30] C. Scheungraber, U. Koenig, B. Fechtel, et al., The colposcopic feature ridge sign is associated with the presence of cervical intraepithelial neoplasia 2/3 and human papillomavirus 16 in young women, J. Low. Genit. Tract Dis. 13 (1) (2009) 13–16. [31] S. Ren, K. He, R. Girshick, et al., Faster R-CNN: towards Real-time object detection with region proposal networks, in: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, MIT Press, 2015. [32] J. Hu, L. Shen, S. Albanie, et al., Squeeze-and-Excitation Networks, CVPR, 2018. [33] K. He, X. Zhang, S. Ren, et al., Deep Residual Learning for Image Recognition, CVPR, 2016. [34] T. Kong, A. Yao, Y. Chen, et al., HyperNet: Towards Accurate Region Proposal Generation and Joint Object Detection, CVPR, 2016. [35] Bing Bai, Pei-Zhong Liu, Yong-Zhao Du, Yan-Ming Luo, et al., Automatic segmentation of cervical region in colposcopic images using K-means, Australas. Phys. Eng. Sci. Med. 41 (2018) 1077–1085.