Research on static path planning method of small obstacles for automatic navigation of agricultural machinery

15th Symposium Control in Transportation Systems JuneIFAC 6-8, 2018. Savona,on Italy 15th IFAC Symposium Control in Transportation Systems June 6-8, 2018. Savona,on Italy 15th Symposium Control in Transportation Systems JuneIFAC 6-8, 2018. Savona,on Italy Available online at www.sciencedirect.com JuneIFAC 6-8, 2018. Savona,on Italy 15th Symposium Control in Transportation Systems June 6-8, 2018. Savona, Italy

ScienceDirect

Multi-Objective Performance Evaluation of98–105 the Detection of Catenary Support IFAC PapersOnLine 51-9 (2018) Multi-Objective Performance Evaluation of the Detection of Catenary Support Multi-Objective Performance Evaluation of the Detection of Catenary Support Components Using DCNNs Components Using DCNNs Multi-Objective Performance Evaluation of the Detection of Catenary Support Components Using DCNNs Multi-Objective Performance Evaluation of the Detection of Catenary Support Wenqiang Liu*, Zhigang Liu *,Components Alfredo Núñez**,Using Liyou Wang*, Kai Liu*, Yang Lyu*, Hongrui Wang* DCNNs Wenqiang Liu*, Zhigang Liu *,Components Alfredo Núñez**, Liyou Wang*, Kai Liu*, Yang Lyu*, Hongrui Wang* DCNNs Wenqiang Liu*, Zhigang Liu *, Alfredo Núñez**,Using Liyou Wang*, Kai Liu*, Yang Lyu*, Hongrui Wang* 

Wenqiang Liu*, Zhigang Liu *, Alfredo Núñez**, Liyou Wang*, Kai Liu*, Yang Lyu*, Hongrui Wang* * School Engineering, Southwest Wenqiang Liu*, Zhigang Liu of *, Electrical Alfredo Nú ñez**, Liyou Wang*, Jiaotong Kai Liu*,University Yang Lyu*, Hongrui Wang* * School of Electrical Engineering, Southwest Jiaotong University China (e-mail: [email protected]).  * School of Electrical Engineering, Southwest Jiaotong University China (e-mail: [email protected]). ** SectionofofElectrical Railway Engineering, Delft University of Technology * School Southwest Jiaotong University China (e-mail: [email protected]). ** Section of Railway Engineering, Delft University of Technology Netherlands (e-mail: [email protected]) * School of Electrical Engineering, Southwest Jiaotong University China [email protected]). ** Section of Railway Engineering, Delft University of Technology Netherlands (e-mail: [email protected]) China (e-mail: [email protected]). ** Section of Railway Engineering, Delft University of Technology Netherlands (e-mail: [email protected]) ** Section of Railway Engineering, Delft University of Technology Netherlands (e-mail: [email protected]) Abstract: The goal of thisNetherlands paper is to evaluate a multi-objective perspective the performance on the (e-mail:from [email protected]) Abstract: goal ofsupport this paper is to evaluate multi-objectivedeep perspective the performance on the detection ofThe catenary components whenfrom usingaa state-of-the-art convolutional neural networks Abstract: The goal ofsupport this paper is to evaluate from multi-objectivedeep perspective the performance on the detection of catenary components when using state-of-the-art convolutional neural networks (DCNNs).ofThe The detection components is thefrom firsta state-of-the-art step towards adeep complete automatized monitoring Abstract: goal ofsupport thisof paper is to evaluate multi-objective perspective the performance on the detection catenary components when using convolutional neural networks (DCNNs). The detection of components is about thefrom first step towards adeep complete automatized monitoring system that willgoal provide information defects in the catenary support A series of Abstract: ofsupport thisactual paper is to evaluate a state-of-the-art multi-objective perspective thedevices. performance on the detection ofThe catenary components when using convolutional neural networks (DCNNs). The detection of components is the first step towards a complete automatized monitoring system that will provide actual information about defects in the catenary support devices. A series of experiments in an unified test environment for detection of components are performed using Faster-CNN, detection of catenary support components when using state-of-the-art deep convolutional neural networks (DCNNs). detection of is about the first step towards a complete automatized monitoring system thatThe will provide actual information defects in the catenary support devices. A series of experiments in an unified test environment for detection of components are performed using Faster-CNN, R-FCN, SSD, and YOLOv2. Through the comparison of different assessment indicators, such as (DCNNs). The detection of components is the first step towards a complete automatized monitoring system that will provide actual information defects in the catenary support devices. A series of experiments in an unified test environment forabout detection of components are performed using Faster-CNN, R-FCN, SSD, and YOLOv2. Through comparison of different assessment indicators, such as precision, recall, average precision and the mean average theare detection of the system that will provide actual information defects in the catenary support performance devices. A series of experiments in an unified test environment forabout detection of precision, components performed using Faster-CNN, R-FCN, SSD, and YOLOv2. Through the comparison of different assessment indicators, such as precision, recall, average precision and mean average precision, the detection performance of the different DCNNs methods for the components of the catenary support devices is analyzed, discussed and experiments in an unified test environment for detection of components are performed using Faster-CNN, R-FCN, SSD, and YOLOv2. Through the comparison of different assessment indicators, such as precision, recall, methods average for precision and mean precision, the detection performance of and the different DCNNs the components of average theall catenary devices is analyzed, evaluated. The experiment results show among considered methods, R-FCN is the discussed more such suitable R-FCN, SSD, and YOLOv2. Through the comparison of support different assessment indicators, as precision, recall, average precision andthat mean average precision, the detection performance of and the different DCNNs methods for the components of the catenary support devices is analyzed, discussed evaluated. The experiment results show that among all considered methods, R-FCN is the discussed more suitable for the detection of catenary support components. precision, recall, average precision and mean average precision, the detection performance of the different DCNNs methods for the components of the catenary support devices is analyzed, and evaluated. The experiment results show that among all considered methods, R-FCN is the more suitable for the detection ofmethods catenaryfor support components. different DCNNs the components of theall catenary support devicesR-FCN is analyzed, and evaluated. The experiment results show that among considered methods, is the discussed more suitable for the detection of catenary support components. © 2018, IFAC (International Federation of Automatic Control) Hostingmethods, by Elsevier Ltd. All rights reserved. Keywords:The Catenary, Railway Systems, Multi-Objective Performance Evaluation, Deep convolutional evaluated. experiment results show that among all considered R-FCN is the more suitable for the detection of catenary support components. Keywords: Catenary, Railway Systems, Multi-Objective Performance Evaluation, Deep convolutional neural (DCNNs) for the networks detection of catenary support components. Keywords: Catenary, Railway Systems, Multi-Objective Performance Evaluation, Deep convolutional neural networks (DCNNs) Keywords: Catenary, Railway Systems, Multi-Objective Performance Evaluation, Deep convolutional neural networks (DCNNs)  Keywords: Catenary, Railway Systems, Multi-Objective Performance Evaluation, Deep convolutional neural networks (DCNNs)  histogram of oriented gradients (HOG) feature to express the  neural networks (DCNNs) 1. INTRODUCTION histogram of oriented (HOG) to express the rotary double-ear, andgradients combined withfeature the support vector  1. INTRODUCTION histogram of oriented gradients (HOG) feature to express the rotary double-ear, and combined with the support vector 1. INTRODUCTION machine (SVM) to recognize it.(HOG) Then, thetheGabor wavelet is histogram of oriented gradients feature to express the With the rapid development of high-speed railway, a large rotary double-ear, and combined with support vector 1. INTRODUCTION machine (SVM) tothe recognize it.(HOG) Then, thetheGabor wavelet is With theofrapid development of high-speed railway, a large utilized to detect failure of the component. Authors in histogram of oriented gradients feature to express the rotary double-ear, and combined with support vector number new railway infrastructures will be constructed all machine to(SVM) tothe recognize it. the Then, the Gabor Authors wavelet in is 1. INTRODUCTION With theofrapid development of high-speed railway, a large utilized detect failure of component. number new railway infrastructures will be constructed all (Zhang et al., 2017) combined the difference histograms of rotary double-ear, and combined with the support vector machine (SVM) tothe recognize it. Then, the Gabor Authors wavelet in is over world. The larger infrastructure, thea more utilized to detect failure of the component. With theofrapid development ofthehigh-speed railway, large number new railway infrastructures will be constructed all (Zhang et al., 2017) combined the difference histograms of over theofrapid world. The infrastructure, thea more oriented gradients to detect machine (SVM) to(DHOG) recognize it.AdaBoost Then, thealgorithm Gabor wavelet is utilized to detect the failureandof the component. Authors in challenges arerailway facedlarger when deciding infrastructure With development ofthe large (Zhang et al., 2017) combined the difference histograms of number new infrastructures will berailway, constructed all over the the world. The larger thehigh-speed infrastructure, the more oriented gradients (DHOG) and AdaBoost algorithm to detect challenges are faced when deciding infrastructure the auxiliary catenary wire, and judged the fault of the utilized to detect the failure of the component. Authors in (Zhang etgradients al., 2017) combined the difference histograms of maintenance tasks. In larger thiswhen sense, automated monitoring number new railway infrastructures will be constructed all oriented (DHOG) andand AdaBoost algorithm toof detect over theof world. The the infrastructure, the more challenges are faced deciding infrastructure the auxiliary wire, judged the faultto the maintenance tasks. In larger this sense, automated monitoring auxiliary catenary wirecombined through the circular arc histograms detection and (Zhang etgradients al., catenary 2017) difference of oriented (DHOG) and AdaBoost algorithm detect systems capable to detect defects in the whole over the world. The the infrastructure, the more the auxiliary catenary wire, and judged the fault of the challenges are faced when deciding infrastructure maintenance tasks. In this sense, automated monitoring auxiliary catenary wire through the 2016), circular arc detection and systems capable to detect defects in the whole infrastructure segment clustering. In (Han et al., segment clustering oriented gradients (DHOG) and AdaBoost algorithm to detect the auxiliary catenary wire, and judged the fault of the will assure theare safe ofsense, theincomplete railway system. challenges faced when deciding infrastructure auxiliaryclustering. catenary wire through the circularsegment arc detection and maintenance tasks. In this automated monitoring systems capable to operation detect defects the whole infrastructure segment In (Han et and al., clustering is proposed tocatenary firstwire divide the independent area, and utilize will assure thetasks. safe ofsense, theincomplete railway system. the auxiliary wire, judgedsegment the fault of the auxiliary catenary through the 2016), circular arc detection and maintenance In this automated monitoring segment clustering. In (Han et al., 2016), clustering systems capable to operation detect defects the whole infrastructure will assure the safe operation of the complete railway system. is proposed to part firstwire divide the independent area, and utilize Two of capable the most important infrastructure components in the deformable models (DPM) and latent SVM to detect auxiliary catenary through the circular arc detection and segment clustering. In (Han et al., 2016), segment clustering systems to detect defects in the whole infrastructure is proposed to first models divide the independent area, and utilize will safe operation ofinfrastructure the complete railway system. Twoassure of system thethemost important components in the deformable (DPM) and latent SVM to utilize detect railway areoperation the catenary the track systems, as rod-insulator. The localthe (LN) method to segment clustering. In (Han etnormalization al., 2016), segment clustering is proposed to part first divide independent area, and will safe ofinfrastructure theand complete railway system. Twoassure of system thethemost important components in the the deformable part models (DPM) and latent SVM to detect railway are the catenary and the track systems, as the rod-insulator. The local normalization (LN) method to shown in Fig. 1. Potential failures of the catenary or track achieve the contrast enhancement of the rail image is is proposed to first divide the independent area, and utilize Two of system the most important infrastructure components in the partThe models and latent(LN) SVMmethod to detect railway are the catenary and the track systems, as the deformable rod-insulator. local(DPM) normalization to shown insystem Fig.threaten 1.are Potential failures of the the catenary or track achieve the contrast enhancement of the rail image is will severely the railway traffic safety. For ensuring Two of the most important infrastructure components in proposed in (Li and Ren, 2012), and the defect localization the deformable part models (DPM) and latent SVM to detect railway the catenary and track systems, as rod-insulator. The enhancement local normalization (LN) method shown in Fig.threaten 1. Potential failures of the catenary or track achieve the contrast of the raillocalization image to is will severely the railway traffic safety. For ensuring proposed in (Li and Ren, 2012), and the defect the safe operation ofthe high-speed railway, atrack large number of the railway catenary and systems, as based on the projection profile (DLBP) isof used to detect defects. rod-insulator. The local normalization (LN) method to shown insystem Fig.threaten 1.are Potential failures of the the catenary or track achieve contrast enhancement the rail image is will severely the railway traffic safety. For ensuring proposed in (Li and Ren, 2012), and the defect localization the safe of high-speed railway, abeen large number of based on the projection (DLBP) isof used to rail detect defects. monitoring and investigated, shown inoperation Fig. 1.detection Potential failures ofhave the catenary or track A probabilistic model to differentiate fasteners tolocalization track and contrast enhancement the image is will severely threaten the technology railway traffic safety. For ensuring proposed in (Li andprofile Ren, 2012), and defect the safe operation of high-speed railway, abeen large number of achieve based on projection profile (DLBP) is the used to detect defects. monitoring and detection technology have investigated, A probabilistic model to differentiate fasteners to track and and the corresponding detection equipment has been will severely threaten the railway traffic safety. For ensuring judge the fault of fasteners based on the likelihood proposed in (Li and Ren, 2012), and the defect localization the safe operation of high-speed railway, a large number of based on projection profile (DLBP) is used to detect defects. monitoring and detection technology have been investigated, A probabilistic model to differentiate fasteners to track and and the operation corresponding detection equipment has been theprojection faultmodel ofprofile fasteners based onto2014). the developed and Intechnology this paper the focus isinvestigated, on catenary the safe of high-speed railway, abeen large number of judge probability was established in (Feng al., In many based on (DLBP) is etused detect defects. monitoring andapplied. detection have A probabilistic to differentiate fasteners to likelihood track and and the corresponding detection equipment has been judge the fault of fasteners based on the likelihood developed and Intechnology this paper the focus isinvestigated, onhas catenary was established in (Feng etfasteners al., 2014). In many support relying on image processing technology. monitoring andapplied. detection have been aspects, these traditional feature extraction and machine A probabilistic model to differentiate to track and and thedevices corresponding detection equipment been probability judge the fault of fasteners based on the likelihood developed and applied. In this paper the focus is on catenary probability was traditional established feature in (Feng et al., 2014). In many support relying on image processing technology. aspects, extraction andlikelihood machine and thedevices corresponding detection equipment been judge learning methods object recognition have obtained thethese faulttraditional offor fasteners based on 2014). the developed and applied. In this paper the focus is onhas catenary probability was established in (Feng et al., In many support devices relying on image processing technology. aspects, these feature extraction and machine In (Cho andand Ko, 2015), Inthethis scale-invariant feature learning methods for object recognition have obtained developed applied. paper the focus is ontransform catenary positive results. However, performance been probability was traditional established intheir (Feng et al., 2014). In many support devices relying on processing technology. aspects, feature extraction and has machine In (Chois and Ko, 2015), theimage scale-invariant feature transform learning these methods for object recognition have obtained (SIFT) employed to track and locate the pantograph, and positive results. However, their performance has been support devices relying on image processing technology. In (Cho and Ko, 2015), the scale-invariant feature transform difficult to get improved. In recent years, with the rapid aspects, these traditional feature extraction and machine learning methods However, for objecttheir recognition have has obtained (SIFT) isand employed to track and locateoverhead the feature pantograph, and positive toresults. performance been assessing the reliability of railway power by difficult get improved. In recent years, with the rapid In (Cho Ko, 2015), the scale-invariant transform (SIFT) is employed to track and locateoverhead the pantograph, and development of deep there are methods applied in learning methods forlearning, object recognition have obtained positive results. However, their performance has been assessing the reliability of railway power by difficult to get improved. In recent years, with the rapid measuring the the pantograph and contact In (Choisand Ko,stagger 2015), theofscale-invariant transform development of detect deep learning, there are methods applied in (SIFT) employed to between track and locate the feature pantograph, and assessing the reliability railway overhead power by the industry to the fault of equipment. In (Faghihpositive results. However, their performance has been difficult to get improved. In recent years, with the rapid measuring the stagger between the pantograph and contact development of detect deep learning, there are methodsInapplied in wire. The speeded-up robust features (SURF) is proposed in (SIFT) is employed to track and locate the pantograph, and the industry to the fault of equipment. (Faghihassessing the reliability of railway overhead power by measuring the stagger robust between the pantograph and contact Roohi et to al.,get 2016), alearning, deep neural network is difficult improved. Inconvolutional recent years, with the rapid development of deep there are methods applied in wire. The speeded-up features (SURF) is proposed in the industry to detect the fault of equipment. In (Faghih(Yang et al., to extract features and detect the assessing the reliability of features railway overhead power by Roohi et al., 2016), alearning, deep convolutional neural network is measuring the 2013), stagger between the pantograph and contact wire. The speeded-up robust (SURF) is proposed in proposed for the analysis of image data for the detection of development of deep there are methods applied in the industry to detect the fault of equipment. In (Faghih(Yang et al., extract the features detect the Roohi et al., 2016), a deep convolutional neural network of is insulator. Then, the to state of the pantograph insulator evaluated measuring the 2013), stagger between contact for the analysis of image data for the detection wire. The speeded-up robust features (SURF) and isisand proposed in proposed (Yang et al., 2013), to extract the features and detect the rail surface defects. et data al., for 2017), anetwork detection the industry to detect the(Gibert fault of equipment. In (FaghihRoohi et al., 2016), aIndeep convolutional neural is insulator. Then, the state of the insulator is evaluated proposed for the analysis of image the detection of according to the vertical grayscale statistic distribution. In wire. The speeded-up robust features (SURF) is proposed in rail surface defects. (Gibert et data al., 2017), anetwork detection (Yang et al., 2013), extract and the insulator. Then, the tostate of the the features insulatordistribution. is detect evaluated method which is able improve Roohi etwas al., 2016), aIn deep neural is proposed forproposed the analysis of convolutional image for the the detection of according to vertical grayscale In rail surface defects. In (Gibert et to al., 2017), a detection detection (Zhang al.,the 2016), theextract contourlet transform along (Yang etet al., 2013), features and detect the method was proposed which is able to improve the detection insulator. Then, the tostate of the statistic insulator is(CT) evaluated according to the vertical grayscale statistic distribution. In performance through combining multiple detectors withinofa proposed for the analysis of image data for the detection rail surface defects. In (Gibert et toal., 2017),the a detection detection (Zhang et al., 2016), the contourlet transform (CT) along method was proposed which is able improve with theetChan-Vese model is statistic proposed to detect and insulator. Then, the (CV) state of the insulator is(CT) evaluated performance throughframework. combining multiple detectors withinthe a according to vertical distribution. In (Zhang al.,the2016), the grayscale contourlet transform along multi-task In et al.,the rail surface defects. In (Gibert et (Chen 2017), a2017), detection method waslearning proposed which is able toal., improve with theetChan-Vese (CV) model for is statistic proposed to(CT) detect and performance throughframework. combining multiple detectors withinthe a diagnose the insulator. A method failure detection of the according to the vertical grayscale distribution. In multi-task learning In (Chen et al., 2017), (Zhang al., 2016), the contourlet transform along with the the Chan-Vese (CV) model for is proposed to detect and steady arm base was recognized and located with regionmethod waslearning proposed which is able to improve the2017), detection performance through combining multiple detectors within a diagnose insulator. A method failure detection of the multi-task framework. In (Chen et al., the ear pieces isal., described in method (Han et for al., 2017), detection which used the (Zhang 2016),(CV) the contourlet transform along steady arm base was recognizedIn and located regionwith theetChan-Vese model is proposed to(CT) detect and diagnose the insulator. A failure of the performance through combining detectors within a multi-task framework. (Chen et al.,with 2017), the ear pieces is insulator. described in method (Han et for al., 2017), detection which used the steady armlearning base was recognizedmultiple and located with regionwith the Chan-Vese (CV) model is proposed to detect and diagnose the A failure of the ear pieces is described in (Han et al., 2017), which used the multi-task framework. (Chen et al.,with 2017), the steady armlearning base was recognizedInand located regiondiagnose insulator. of the 98 steady arm base was recognized and located with regionear pieces©the is described in method (Han et for al.,failure 2017), detection which used Copyright 2018 IFAC A Copyright © is 2018 IFAC ear pieces© described in (Han et Federation al., 2017),ofwhich usedControl) the 98 Hosting by Elsevier Ltd. All rights reserved. 2405-8963 2018, IFAC (International Automatic Copyright © 2018 IFAC 98 Peer review under responsibility of International Federation of Automatic Control. Copyright © 2018 IFAC 98 10.1016/j.ifacol.2018.07.017 Copyright © 2018 IFAC 98

2018 IFAC CTS June 6-8, 2018. Savona, Italy

Wenqiang Liu et al. / IFAC PapersOnLine 51-9 (2018) 98–105

based convolutional neural networks (RCNN) first, and then on the basis of the installation relationship, the steady arm was detected with Hough transform, and the slope of the angle of steady arm was calculated. Although the progress in recent research efforts in the literature (Jamshidi et al., 2017; Liu et al., 2017; Chen et al., 2018; Psuj, 2018), there is still a gap towards the application of DCNN techniques in the railway industry.

99

quickly and accurately using the traditional image detection technology all those types of defects. In this paper, catenary support device images are used to train DCNNs, and evaluate the detection performance of different DCNNs.

In the literature, different state-of-the-art deep convolutional neural networks (DCNNs) structures are available. However, for the specific task of detection in railway systems environments, not all the neural network structures will perform in the same way. Therefore, this paper proposes a simple assessment methodology based on multi-objective performance evaluation. The most advanced and the most representative structures are used for the detection of components in the catenary support, and the intention is to provide a systematic approach to support on their evaluation.

Fig. 2. XLN4C inspection vehicle

The rest of the paper is organized as follows. First, the inspection system is shown in Section 2. Four of the state-ofthe-art DCNNs are introduced in Section 3. Next, the experiment results are analyzed and discussed in Section 4. Last, some conclusions are summarized in Section 5.

12

11

Suspension Device

Support Device

Fig. 3. Catenary support device

Catenary

3. DCNN STRUCTURES OVERVIEW Currently, DCNNs-based deep learning method has been the mainstream for solving object detection problems. There are a lot of the state-of-the-art DCNNs structures developed, proposed and used in the various fields. Among all, regionbased and regression-based DCNNs structures are two main research directions, for which the representative algorithms are Faster RCNN, R-FCN, SSD, and YOLOv2. The core idea and network structure of these algorithms are described below.

Railway Tracks

Fig. 1. Catenary and track devices 2. INSPECTION SYSTEM XLN4C is a comprehensive inspection vehicle developed by the China Railway Inc., shown in Fig. 2. It is equipped with six detection and monitoring systems including comprehensive pantograph and catenary monitor system (CPCM-1C), catenary-checking video monitor system (CCVM-2C), catenary-checking on-line monitor system (CCLM-3C), high-precision catenary-checking monitor system (CCHM-4C), catenary and pantograph video monitor system (CPVM-5C), and ground monitor system for catenary and power supply equipment (CCGM-6C), and called as the 6C system. In particular, the 4C system is mainly used to detect and diagnose the condition of catenary by analyzing the collected 2D images as the one shown in Fig. 3. A large amount of catenary 2D images are acquired from this system and state of the art image processing technology used for the analysis.

3.1 Faster RCNN Faster RCNN (Ren et al., 2015) is a series of RCNN (Girshick et al., 2014). It is an integration of region proposal networks (RPNs) and Fast R-CNN (Girshick, 2015), which uses RPNs to extract the detection areas and then uses conventional neural networks (CNNs) to extract image features. Lastly, SoftMax function is used to achieve the classification and bounding boxes regression to locate the position of objects. The structure diagram is shown in Fig. 4. 3.2 Region-based fully convolutional networks (R-FCN) R-FCN (Dai et al., 2016) is seen as an improved version of the Faster RCNN. It moved several full connection layers behind the region of interest (RoI) pooling layer to the front, which greatly improved the detection speed. Meanwhile, the

Catenary support device has a complex structure with many components. In addition, its defect type and defect level are various, as shown in Appendix A. It is difficult to detect 99

2018 IFAC CTS June 6-8, 2018. Savona, Italy 100

Wenqiang Liu et al. / IFAC PapersOnLine 51-9 (2018) 98–105

feature extraction network was replaced with ResNet. The structure diagram is shown in Fig. 5.

Bboxes generation with dimension priors and lacation prediction

Objectscls

Feature extraction

3.3 Single shot multibox detector (SSD)

Objectsreg

SSD (Liu et al., 2016) can be simply seen as a combination of YOLO(v1) (Redmon et al., 2016) and anchor boxes idea, which is based on bounding boxes regression. By using a small convolution kernel on different feature maps to predict a series of box offsets of bounding boxes (Bboxes), the goal of the object detection can be achieved. The structure diagram is shown in Fig. 6.

YOLOv2

Fig. 7. Structure diagram of YOLOv2 4. EXPERIMENT AND RESULTS In order to evaluate the detection performance of the four different DCNNs methods presented in the previous section, different indexes are chosen to assess their performance. The experiment environment is as follows. Deep learning open source framework Caffe (Jia et al., 2014), Ubuntu 14.04, 32GB RAM, CPU clocked at 3.2 GHz, and GTX 1080 GPU with 8GB memory.

3.4 YOLOv2 YOLOv2 (Redmon and Farhadi, 2016) is an improved version of YOLO(v1), which is also based on the bounding boxes regression. YOLOv2 draws on the anchor ideas in the Faster R-CNN, which samples on the convolution feature map with sliding window. Then, each centre predicts nine different sizes and proportions of the proposed box. Since there is no need for reshaping the convolution layer, the spatial information is kept, solving the shortcomings of YOLO (v1). The structure diagram is shown in Fig. 7.

4.1 Dataset and Parameter settings The catenary dataset is made by the tool called “labelImg” provided from the website 1 . The total image amount of dataset is 5022, among which the training dataset is 2417, the validation dataset is 1036 and the test dataset is 1569. The experiment parameter settings are as follows. Momentum and weight decay are 0.9 and 0.0005, and learning rate is 0.001, iterations are 15, 000.

RPN

4.2 Evaluation Indexes

Objectscls

Feature extraction

Some indexes are chosen including the precision and recall, average precision (AP), mean average precision (mAP) and frames per second (FPS). Some curves and charts are drawn including precision and recall curve (P-R curve) and loss curve.

Objectsreg Faster RCNN

RoI pooling layer

Fig. 4. Structure diagram of Faster RCNN

precision 

TP

RPN

recall 

 100%

(1)

 100%

(2)

TP  FP TP

TP  FN 1

AP  0 p ( r ) d r

Objectscls

Feature extraction

mAP 

Objectsreg RFCN

Q

RoI pooling layer

(3) (4)

where TP is true positive, FP is false positive, TN is true negative, Q is the number of the component class.

Fig. 5. Structure diagram of R-FCN Bboxes generation with default fixed boxes

Feature extraction

Q  q 1 AP( q )

4.3 Experiment Results Objectscls

1) Examples of component detection effects

Objectsreg

In order to show the detection effect of the four different structures under the simple environment and complex environment, two sets of examples are shown in Fig. 8. One

SSD Extra feature extraction

Fig. 6. Structure diagram of SSD 1

100

https://github.com/tzutalin/labelImg

2018 IFAC CTS June 6-8, 2018. Savona, Italy

Wenqiang Liu et al. / IFAC PapersOnLine 51-9 (2018) 98–105

only includes one set of catenary support device, and the

101

other contains two sets of catenary support device.

(a)

(b)

(c)

(d) Figure. 8. Two examples of component detection effects for the four different DCNNs architecture. (a) row is the detection with Faster RCNN, (b) row is the detection with R-FCN, (c) row is the detection with SSD, and (d) row is the detection with YOLOv2.

101

2018 IFAC CTS June 6-8, 2018. Savona, Italy 102

Wenqiang Liu et al. / IFAC PapersOnLine 51-9 (2018) 98–105

(a)

(b)

(e)

(f)

(i)

(c)

(d)

(g)

(h)

(j)

(k)

(l)

Figure. 9. P-R Curve for the detection of all the parts of the catenary support device (a) Insulator, (b) Rotary double-ear, (c) Binaural sleeve, (d) Brace sleeve, (e) Steady arm base, (f) Bracing wire hook, (g) Double sleeve connector, (h) Messenger wire base, (i) Windproof wire ring, (j) Insulator base, (k) Isoelectric line, (l) Brace sleeve screw, among, (a)~(j) are the large objects, (k) and (l) are the small objects.

Table 1. Detection average precision on catenary dataset for the four different DCNNs architecture Brace Rotary Bracing Double Messen Catenary Binaural Brace Steady Windproof Insulator Isoelectric Insulator doublewire sleeve ger wire sleeve mAP FPS Dataset sleeve sleeve arm base wire ring base line ear hook connector base screw Faster 0.783 0.842 0.785 0.796 0.767 0.563 0.843 0.804 0.508 0.65 0.181 0 0.627 1.46 RCNN R-FCN

0.757

0.88

0.861

0.88

0.835

0.732

0.846

0.787

0.682

0.846

0.334

SSD

0.877

0.856

0.818

0.834

0.876

0.459

0.812

0.782

0.223

0.683

0.716

0.003 0.662 2.30

YOLOv2 0.886

0.261

0.381

0.225

0.73

0.587

0.703

0.615

0.586

0.743

0.511

0.027 0.521 3.70

102

0

0.703 2.02

2018 IFAC CTS June 6-8, 2018. Savona, Italy

Wenqiang Liu et al. / IFAC PapersOnLine 51-9 (2018) 98–105

103

Figure. 10. Detection average precision on catenary dataset for the four different DCNNs architecture

2) P-R curve

4.4 Results analysis

For analyzing the relationship between the false detection rate and missed detection rate of each component for four different DCNNs method, the curve of the precision and recall (P-R curve) of every part is drawn up, as shown in Fig. 9.

(1) From Fig. 8, the detection results based on regression method, SSD and YOLOv2 have the result of missed detection even if it is under the simple background environment (as shown in Fig. 8(c)-left and Fig. 8(d)left). Among the detection results with Faster RCNN and R-FCN, the latter performs better.

3) AP and mAP For comparing the detection accuracy and detection efficiency of different models, the mathematical statistics for the test dataset are carried out through AP and mAP as well as FPS, as show in Table 1 and Fig. 10.

(2) It can be found from Table 2 and Fig. 10 that the detection results with R-FCN are the best among these DCNNs, whether it is a single object AP or the whole mAP. For the smaller objects isoelectric line and brace sleeve screw, the regression-based SSD and YOLOv2 achieve some effects though they are not ideal. In addition, for the bigger objects, the SSD also performs relatively good and its speed is also very fast.

4) Training loss To measure the robustness of the training model, the training loss curve is plotted as the number of iterations increases, as shown in Fig. 11.

(3) Fig. 9 shows that the P-R curve of the YOLOv2 is the worst. It is difficult to achieve a good balance between the precision and recall. In other words, if the model wants to improve its false detection rate, then it has to sacrifice its missed detection rate. In contrast, the P-R curve of R-FCN which have a better performance. (4) As seen from Fig. 11, the robustness of the training model of the region-based DCNNs, Faster RCNN and R-FCN, are the best. Their convergence speed is also faster. To the opposite, the loss curve of SSD fluctuated greatly and YOLOv2 has a low convergence speed. 5. CONCLUSION Combined with the most advanced DCNNs learning method technology, the development of advanced detection methodologies is crucial for the railway infrastructure maintenance. This paper compared the state-of-the-art and the most representative DCNNs learning algorithms with different multi-objective evaluation indexes.

Fig. 11. Training loss for the four different DCNNs architecture 103

2018 IFAC CTS June 6-8, 2018. Savona, Italy 104

Wenqiang Liu et al. / IFAC PapersOnLine 51-9 (2018) 98–105

Through the above results analysis, for catenary image detection, the region-based R-FCN is more suitable for the detection of catenary support components. However, its shortcoming is a poor performance in detection of the smaller objects. A possible further research direction could be to incorporate the capabilities of SSD and YOLOv2 into an integrated scheme that make use the structure R-FCN achieve the desired results. SSD and YOLOv2 could be used as a RPNs, and the proposed regions would be sent into the RFCN to improve the detection effects of the DCNN. This paper does not study the characteristics of each network individually. It gives a general comparative result, which expects that the follow-up research work is able to focus on a core framework for optimization and improvement, and ultimately establishing a detection network that is more suitable for the maintenance of the railway infrastructure. REFERENCES Cho, C.J. and Ko, H. (2015). Video-based dynamic stagger measurement of railway overhead power lines using rotation-invariant feature matching. IEEE Transactions on Intelligent Transportation Systems, 16(3), 1294-1304. Yang, H.M., Liu, Z.G., Han, Y., and Han, Z.W. (2013). Defective condition detection of insulators in electrified railway based on feature matching of speeded-up robust features. Power System Technology, 37(8), 2297-2302. Zhang, G.N., Liu, Z.G., and Han, Y. (2016). Automatic recognition for catenary insulators of high-speed railway based on contourlet transform and Chan–Vese model. Optik - International Journal for Light and Electron Optics, 127(1), 215-221. Han, Ye., Liu, Z.G., Geng, X., and Zhong, J.P. (2017). Fracture detection of ear pieces in catenary support devices of high-speed railway based on HOG features and two-dimensional Gabor transform. Journal of the China Railway Society, 39(2), 52-57. Zhang, G.N., Liu, Z.G., Han, Y., and Han, Z.W. (2017). Loss fault detection for auxiliary catenary wire of high-speed railway catenary wire holder. Journal of the China Railway Society, 39(5), 40-46. Han, Y., Liu, Z.G., Lee, D.J., Zhang, G.N., and Deng, M. (2016). High-speed railway rod-insulator detection using segment clustering and deformable part models. In Image Processing (ICIP), 2016 IEEE International Conference on, 3852-3856, IEEE. Li, Q.Y. and Ren, S.W. (2012). A real-time visual inspection system for discrete surface defects of rail heads. IEEE Transactions on Instrumentation and Measurement, 61(8), 2189-2199. Feng, H., Jiang, Z.G., Xie, F.Y., Yang, P., Shi, J., and Chen, L. (2014). Automatic fastener classification and defect detection in vision-based railway inspection systems. IEEE Transactions on Instrumentation and Measurement, 63(4), 877-888. Faghih-Roohi, S., Hajizadeh, S., Núñez, A., Babuska, R., and De Schutter, B. (2016, July). Deep convolutional neural networks for detection of rail surface defects. In Neural Networks (IJCNN), 2016 International Joint Conference on, 2584-2589, IEEE. 104

Gibert, X., Patel, V., and Chellappa, R. (2017). Deep multitask learning for railway track inspection. IEEE Transactions on Intelligent Transportation Systems, 18(1), 153-164. Chen, D.J., Zhang, W.S., & Yang, Y. (2017). Detection and recognition of high-speed railway catenary locator based on Deep Learning. Journal of University of Science and Technology of China, 47(4), 320-327. Jamshidi, A., Faghih-Roohi, S., Hajizadeh, S., Núñez, A., Babuska, R., Dollevoet, R., Li, Z., and De Schutter, B. (2017). A big data analysis approach for rail failure risk assessment. Risk Analysis, 37(8), 1495-1507. Liu, Z.G., Wang, L.Y., Yang, C.J. and Han, Z.W. (2017). A high-precision loose strands diagnosis approach for isoelectric line in high-speed railway. IEEE Transactions on Industrial Informatics, DOI 10.1109/TII.2017.2774242. Chen, J.W., Liu, Z.G., Wang, H.R., Nunez, A. and Han, Z.W. (2018). Automatic defect detection of fasteners on the catenary support device using deep convolutional neural network. IEEE Transactions on Instrumentation and Measurement, 67(2), 257-269. Psuj, G. (2018). Multi-sensor data integration using deep learning for characterization of defects in steel elements. Sensors, 18(2), 292. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 580-587. Girshick, R. (2015). Fast R-CNN. In Proceedings of the IEEE international conference on computer vision, 1440-1448. Ren, S.Q., He, K.M., Girshick, R., and Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, 91-99. Dai, J.F., Li, Y., He, K.M., and Sun, J. (2016). R-FCN: Object detection via region-based fully convolutional networks. In Advances in neural information processing systems, 379-387. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016, October). SSD: Single shot multibox detector. In European conference on computer vision, 21-37, Springer, Cham. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 779-788. Redmon, J. and Farhadi, A. (2016). YOLO9000: better, faster, stronger. arXiv preprint arXiv:1612.08242. Jia, Y.Q., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014, November). Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia, 675-678, ACM.

2018 IFAC CTS June 6-8, 2018. Savona, Italy

Wenqiang Liu et al. / IFAC PapersOnLine 51-9 (2018) 98–105

Appendix A Table 2. Defect type and level of the components of catenary support device Part name 1 ○

Insulator

2 ○

Rotary double-ear

3 ○ 4 ○

Binaural sleeve

5 ○

Steady arm base

6 ○

Bracing wire hook

7 ○

Double sleeve connector

8 ○

Messenger wire base

Brace sleeve

Defect type

Defect level

Cracking Flashover Cracking Cotter pin losing Cracking

B B A

A A A A A A A A

B

9 ○

Windproof wire ring

10 ○

Insulator base

11 ○ 12 ○

Isoelectric line

Cracking Cracking Nut looseness Cracking Nut looseness Cracking Nut looseness Cracking Opposite direction Balance line losing Cracking Nut looseness Cotter pin losing Line looseness

Brace sleeve screw

Nut looseness

A A

C C A B B C

Note: Isoelectric line is the joint area between steady arm base and registration arm, and brace sleeve screw is subarea of the brace sleeve.

105

105