Train Rolling Stock Intelligent Monitoring with Computer Vision

Available online at www.sciencedirect.com

ScienceDirect Materials Today: Proceedings 4 (2017) 1730–1739

www.materialstoday.com/proceedings

5th International Conference of Materials Processing and Characterization (ICMPC 2016)

Train Rolling Stock Intelligent Monitoring with Computer Vision P.V.V. Kishorea*, Ch.Raghava Prasada a

K.L.University, E.C.E. Dept. . K.L.University E.C.E. Dept.

a

Abstract Computer automation of rolling stock involves determination of individual parts for examination for defect Identification from the videos of a moving train. Video frame segmentation using Chan Vese active contour model (CV-AC) results in a full bogie binary image making it impossible to track individual parts. To segment individual parts and track their shapes along the length of the train is a challenging task. This challenge is achieved by using shape prior seeds (SP-CV-AC) as destination contour from individual parts of the bogie for the Chan vese active contour model. Spatial distances are used to propel the initial contour towards final shape contour. The results demonstrate the quality of video segmentation algorithm based on destination seed shape priors. Calculating a factual segmentation score (FSS) between the shape prior segments and hand segmented portions of the rolling stock to access the quality of the proposed segmentation algorithm. We further compare shape prior segmentation model with no-shape prior active contours to specify the importance of shape prior models for complex image processing tasks related to intelligent maintenance systems with computer vision. ©2017 Elsevier Ltd. All rights reserved. Selection and peer-review under responsibility of Conference Committee Members of 5th International Conference of Materials Processing and Characterization (ICMPC 2016). Keywords: Train Rolling Stock Segmentation, Chan Vese Active Contour Segmentation, Shape Prior Seed Segmentation, Geodesic distance measure, Factual Segmentation Score.

1. Introduction The field of computer vision and its applications is of interest to many researcher’s around the world. The moving and rolling portions of a train are called rolling stock E-mail address: [email protected]

2214-7853 ©2017 Elsevier Ltd. All rights reserved. Selection and peer-review under responsibility of Conference Committee Members of 5th International Conference of Materials Processing and Characterization (ICMPC 2016).

P.V.V. Kishore et al / Materials Today: Proceedings 4 (2017) 1730–1739

1731

. Rolling examination is vital for passenger trains to identify defects that can possibly generated during movement of trains at high speeds. This process will ensure the safe movement of trains. Failure of the bogie questions the safety record of the railway organizations. To ensure safety of the train passengers, railways employ a technique called rolling stock examination (RSE). Examination involves trained vision and auditory senses of human operators by observing the train near stations. The train is moving at around 30kmph. The process followed by Indian railways can be best understood using figure 1. The results of the examination are a data sheet (office spread sheet) with defects and their intensities at various locations. This information is passed to the following station maintenance staff to attend the problem. Two personal for monitoring and a third person for noting the defects on either ends of the station make up a total team of SIX personal per train for examination. The following are the drawbacks identified:

Fig. 1. Manned Rolling examination with two personal on either side of the moving train and another person heading them.

Human indulgence, Heavy Personal work load, Weather dependent system. Over the decades the rail companies has formulated procedures and invested hugely in making the rail travel safer. In case of Indian Railways, rolling stock examination has brought down the accidents significantly but this incurs a huge financial burden on the rail company [1].This research proposes to replace the maned rolling examination with a high speed camera based computer vison automated rolling examination. For this purpose a 240 frames per second high speed camera is employed. Even though rolling stock examination is the primary task performed regularly, few systems have been developed for monitoring components of the rolling stock with computer vision models [2, 3]. A fuzzy model based weighted logarithmic least squares method with computer vision is proposed for concrete maintenance of train rolling stock. A model based on the triangular fuzzy number (TFN) weighted logarithmic least square method is given and corresponding evaluation process is described [4]. Papers [5]-[8] use similar methods of computer vision and classification techniques to determine and analyze the shape of brake shoes using a high speed digital camera installed on the railway tracks. Finally this paper encapsulates the core features of an experiment Technicatome that has been developed as a demonstrator for RATP (the Paris subway company) based on interconnected digital systems. This demonstrator is currently in operation on an MF 88 train set to the long existing and still operated with conventional relay-based systems [9]-[10]. From our work in the previous research, concentration was on segmentation of train rolling stock [3] as a whole to look for defects. This model is not fetching correct outcomes due to difficulty in identifying individual parts for further processing. Hence a need was felt to extract individual parts of the rolling stock. For this task, we propose a shape prior focused segmentation of rolling stock parts for development of a fully automated rolling stock examination. The rest of the paper will have a brief review on active contours followed by shape prior focused

P.V.V. Kishore et al / Materials Today: Proceedings 4 (2017) 1730–1739

1732

segmentation model. Finally results on various frames of rolling stock videos to validate the effectiveness of the algorithm by computing a factual segmentation score (FSS). 2. Shape Prior Active Contours 2.1. Active Contours Active contours are measurable curves that are used exclusively by image processing research community to extract object boundaries. Active contours come under a category of model based segmentation methods [11]-[12]. The fundamental design behind the active contours is the movement of a predefined contour within the domain of the image. Image domain is defined by the boundaries of objects in that particular image. Contour movement in the image domain is controlled by a parameter called energy function. The active contours model was first introduced by Terzopoulos [13]-[14]. Earlier models of active contours are prone to topological disturbances and are extremely susceptible to initial conditions. However with the development of level sets [15] topological changes in the objects of the image are involuntarily handled. Nevertheless all active contours depend on the gradient of the image for ending the growth of the curve. Evolution equation for active contours, called as snake initially was first proposed by Kass [16]. Let I xy : D → ℜ2 be an open and bounded 2D shape space consisting of a set of positive real numbers. The object subspace is S : D → R 2 such that it forms a subset of the defined image S ⊂ I . To achieve active segmentation the 1 energy function that is intended has to be minimized I (1) E ( S ) = E int (v( s )) + E xy (v( s )) d s I Where E ( S ) is the energy of the snake. E int0 and E xy are internal snake energy and image energy respectively. Snake’s location on the image is iteratively provided by (2) v( s ) = ( x( s ), y ( s ) )

∫{

}

The internal force E int of the snake is due to meandering of the contour and image external force E the snake towards the object boundaries. The internal force model is defined as

E int

2  ( ) d 2 v( s )  α ( s ) d vs + β (s)  ds ds 2  = 2

2

   

s ∈ [0,1]

I xy

will drive

(3)

dv( s ) gives the rate of change of the length of the contour and the degree of contraction is provided by the ds

coefficient α ( s ) .The second order derivative of v( s ) ,

d 2 v( s ) ds 2

gives rate of change of curvature of the contour. β ( s )

is called normal coefficient which regulates the rate of change of contour in the direction normal to the boundary. Adjusting these two coefficients allows the contour to move elastically to embrace the objects in the image. External image force is modeled as

E

I xy

= − ∇I ( x , y )

2

(4)

2.2 Chan Vese Active Contour Model Chan-Vese (CV) [17] active contour model discovers a contour Θ : D → ℜ2 defined on image space D consisting of a set of positive real numbers. The discovered contour optimally approximates the objects in a gray scale image I : D → ℜ2 to a single real gray value Φ ( I ) on the inside of the contour Θ and another single gray level value Φ ( E ) on the outside of the contour Θ . The basic idea of CV Active model is to find an optimal contour that fits the object boundaries. Alongside the best contour, the solution should also find a pair of optimal gray scale values Φ F = Φ ( I ) , Φ ( E ) that discriminates object pixels from background pixels. Mathematically the chan-vese active contour is formulated as an energy minimization problem

(

)

P.V.V. Kishore et al / Materials Today: Proceedings 4 (2017) 1730–1739

(

1733

)

E cv Θ F , Φ F = min E cv (Θ, Φ )

(5) Φ Where Θ is the final contour shape to be discovered and Θ is the initial contour chosen. The energy function or force function formulated by CV active contour model is minimized using piece wise linear Mumford-shah [18] function which estimates the pixel values of a gray scale image I ( x, y ) by a linear piece wise smooth contour Θ . The solution to the minimization problem is formulated as F

 1 E cv = χ1 ds + χ 2  I ( x, y ) − Φ ( I ) 2 Θ  int(Θ)

∫∫ (

∫

The first term in the eq.(6) indicates arc length

arg

min( Θ,Φ )

)

2

( χ1 × l (Θ) )

dxdy +

1 2

∫∫ ( I ( x, y) − Φ ) (E)

ext ( Θ )

2

 dxdy   

(6 )

which guarantees consistency of Θ .

Where l (Θ) defines the length of the contour. The second term in the eq.6 is a mixture of two integrals. The first integral function propels the contour Θ towards the objects in the image while the second integral function ensures the differentiability of the contour Θ . int(Θ) and ext (Θ) values representing internal and external to the contour Θ . The Mumford-Shah considers the edge of the image as the boundary. The weight parameters are chosen to be positive real quantities χ1 , χ 2 ≥ 0 . Solution for eq.6 is a complicated one. Hence a more simpler piecewise constant formulation of Mumford-Shah distance function is

∫

E cv = χ1 ds + χ 2 Θ

∫∫ ( I ( x, y) − Θ( x, y) )

2

(7)

dxdy

int( Θ )

Compared to Mumford-Shah model, Chen-Vese Model consists of an additional term imprisoning the area enclosed and on further simplification Φ is allowed to have two values corresponding to the mean values of the pixels inside and outside Θ .  Φ ( I ) , where ( x, y )lies inside  1  Θ= I ( x, y )dxdy  int(Θ) int( Θ )  Θ ( x, y ) =  (E) Φ , where ( x, y )lies outside  1  Θ= I ( x, y )dxdy ext (Θ)  ext ( Θ ) 

∫∫

(8)

∫∫

CV model calculates and finds the values of Θ , that best fit the image I ( x, y ) using the energy term in eq.8.   2 2 1 1 Θ ( x, y ) dxdy + χ 2  E cv = χ1 ds +ν I ( x, y ) − Φ ( I ) dxdy + I ( x, y ) − Φ ( E ) dxdy  (9) 2  2 int( Θ ) int( ) ( ) ext Θ Θ Θ   The first two terms are regularizing parameters for contours length and its area to control the size of the contour. Parameters χ1 > 0, χ 2 > 0 and ν > 0 should be the first two terms are regularizing parameters for contours length and its area to control the size of the contour. Parameters χ1 > 0, χ 2 > 0 and ν > 0 should be chosen by the user. The third and fourth terms make the model Θ ( x, y ) adapt to the objects in the image I ( x, y ) . Image segmentation deals with finding the global minimum to the problem defined in eq.9. The minimization problem in eq.9 is solved using the level set model [19] and is formulated in terms of level set function Θ ( x, y ) as   cv (I ) (E) (I ) 2 (E) 2 E ( Θ, Φ , Φ ) = min χ 2  ( I ( x, y ) − Φ ) H ( Θ( x, y ) ) + ( I ( x , y ) − Φ ) (1 − H ( Θ ( x , y ) )) d dx  y + χ1 ∇H ( Θ ( x , y )) d dx y (10) ∫∫ ∫∫ ∫ Θ,Φ ,Φ  int( Θ )  Θ ext ( Θ )

∫

∫∫ (

∫∫

(I )

)

∫∫ (

)

(E)

Where H (Θ) is Heaviside function. This minimization problem is solved by using Euler-Lagrange [17] equations and the level set function Θ ( x, y ) is updated iteratively by the gradient descent method as formulated below.

P.V.V. Kishore et al / Materials Today: Proceedings 4 (2017) 1730–1739

1734

Θt = −δ (Θ)(( I ( x, y ) − Φ ( I ) ) 2 − ( I ( x, y ) − Φ ( E ) ) 2 − χ1∇.

∇Θ( x, y ) ∇Θ( x, y )

(11)

Where x and y denote the locations of pixels in the image. δ (Θ) is the delta function and Φ ( I ) and Φ ( E ) are updated iteratively using the equations Φ

(I )

=

∫∫ I ( x, y) H (Θ( x, y))dxdy Θ

∫∫

H (Θ( x, y ))dxdy

(12)

Φ

(E)

=

∫∫ I ( x, y)(1 − H (Θ( x, y)))dxdy Θ

∫∫

(13)

(1 − H (Θ( x, y )))dxdy

Θ

Θ

Where H (Θ( x, y )) is a Heaviside function. The control on segmentation process plays a very important part in determining the quality of the segmented image objects. This under and over segmentation problem is best handled by applying seed contours and prior shape models to the already formulated level set function in eq’n 10. This method focus the segmentation on a single part of rolling stock enabling the user to identify defects in those parts or a defect classification model can be applied to build an intelligent rolling examination module. This system can also predict maintenance of various parts with their life span on board the moving bogie making maintenance intelligent. 2.3 Shape prior model Adding Chan Vese Active Contour (CV-AC) [no in previous] level set eq’n 10 with shape prior model proposed in [20], the shape induced energy term is represented as (14) E cv + shape = E cv (Θ, Φ ( I ) , Φ ( E ) ) + E Shape (Θ, Φ (SI ) , Φ (SE ) ) The first term is data term from CV level set in eq’n10 and the second term is prior shape energy model defined as 2 E Shape (Θ, Φ (SI ) , Φ (SE ) ) = H ( Φ ( x, y ) ) − H ( Φ S ( x, y ) ) dxdy (15 )

∫(

)

Θ

Where ( Φ S ( x, y ) ) is the shape prior term independent of position on the image. For multiple shape priors the representation of shape energy term is  d 2 H ( Φ ), H ( Φ S ( j ) ) )     N − (    1 2σ 2  E cv + shape( n ) = − log  e (16)   N j =1  

∑

For the above eq’n to produce meaningful segmentations the number of shape priors is typically small and it is difficult to capture the statistical structure of different shapes in the observational space. For this research we concentrated primarily on single object extractions and the simulations use eq’n 15. 3. Results and Discussion

Fig.2. Light Intensity variations at different times of the day.(a) at 6.30AM,(b)12.30 PM, (c) 4.30 PM

P.V.V. Kishore et al / Materials Today: Proceedings 4 (2017) 1730–1739

1735

Before giving the video frame as input to shape prior Chan vese active contour model (SP-CV-AC), a virtual image fusion based contrast enhancement algorithm [21] improves the contrast of the train bogie with the ballast(Gravel or ground).the light intensity variations at different times of the day is shown below in fig 2. Contrast in videos depends on shutter speed and F-stop of the digital SLR camera. Shutter speed controls the amount of reflected light reaching the image sensor in the camera. F-Stop controls the hole aperture that passes light to the sensor. Generally the F-Stop approximates to geometric sequence corresponding to powers of root 2 [3]. This concept is used to generate virtual video frames from non-uniform contrast frames. These frames are fused into one high contrast image in wavelet domain. The entire method is fast and robust to brightness variations in the video frame as proposed in our previous work [3]. The contrast enhanced video frames for a sample frame is shown in figure 3.

Fig.3. Virtual image contrast enhancement for compensating light intensity variations at different times of the day. (a) at 6.30AM,(b)12.30 PM, (c) 4.30 PM

Using regular Chan vese active contours on contrast enhanced video frames gives good segmented frames [3] compared to many edge detection techniques. Figure 7 shows the segmented output from the Chan vese active contours without shape priors. The initial contour covers the entire spatial domain for faster convergence of initial contour towards the gradient magnitude edges of the rolling frame. The segmentation is not focused on a particular part, but covers the entire video frame. Visual observations show the entire frame is covered during segmentation making it impossible to extract major parts during rolling examination for defect identification. Even classification algorithms failed to make sense of this particular segmented rolling stock of figure 4.

Fig.4. Chan Vese AC segmented video frame (a) Segmentation process (b) Final Result

Shape prior Chan vese active contours propel the contour towards the object of interest in the video frame. Our model does not insist on the shape of the initial contour Φ 0 ( x, y ) . The only constraint is to place the seed contour near to the rolling part the user wants to observe by selecting a particular frame of interest. The shape contour Φ S ( x, y ) is precisely cut from the original video frame in which full view of the bogie is visible. The spatial error between two contours will act as a controller for focusing the initial contour towards the rolling part. The following frame in figure 8 is selected for testing the algorithm.

1736

P.V.V. Kishore et al / Materials Today: Proceedings 4 (2017) 1730–1739

Fig.5 Test frame showing a bogie of train moving at 30KMPH during rolling stock examination by a high speed camera model.

Fig.6. Proposed segmentation model is tested on parts pointed by arrows on the bogie of a rolling stock.

Individual parts are hand extracted from the frame using multimedia tools to as shape contours in fig 5. Figure 6 shows the parts of the bogie that are extracted for testing the proposed shape prior AC model. The frame is contrast enhanced and is inputted to shape prior Chan vese algorithm. Figure 7 shows the initial contour which is a rectangular area consisting of seed pixels to choose the spatial pixels near to the rolling part. Using initial contour in figure 7, the extreme right binding screw is chosen. According to railway engineers this screw holds the bogie in place and does allow the bogie to move during transit.

Fig.7. Rolling stock frame showing the part to be segmented along with the initial contour acting as seed.

Fig.8. Shape prior model of right binding screw

The initial contour is propagates towards the shape prior model of the binding screw as shown in figure 8. A uniform distance transform gives the shape prior contour. By using eq’n 15, spatial error propels the initial seed contour towards the shape prior contour giving a focused segmentation result. Figure 9 shows step-by-step process during the binding screw extraction from 15th iteration, 18th, 29, 51, 73, 92, 107, 125, 144, 175, and 191 to 217th iteration. Figure 10 shows the segmentation outputs at previously mentioned iterations. Figure 11 gives the final segment of the binding screw after morphological dilation to close small unconnected pixels. Figure 11 is the compliment of last image in Figure 10. The mean square error between the two images is less than 7%, meaning the simulated result and ground truth are 93% similar. Further figure 12 gives the simulated results and ground truth for each part simulated Figure 12 provides insight into the similarities between ground truth rolling parts and simulated rolling parts using shape priors. The computation of mean square error between the two similar parts will determine the similarity between the simulated part and database hand segmented part. Mean square error calculation for an entire video of 10,000 frames with only seven dictionary parts produced an average mean square error of around 0.11. Which indicates average similarity between segmented parts is around 89%.

P.V.V. Kishore et al / Materials Today: Proceedings 4 (2017) 1730–1739

Fig.9. Shape prior evolution of CV Active contour targeting the binding screw.

Fig.10. Segmentation results of the binding screw at various iterations during propagation of initial contour

Fig.11. Compliment of Last segmented image in Fig.10.

1737

P.V.V. Kishore et al / Materials Today: Proceedings 4 (2017) 1730–1739

1738

4. Conclusion: This work provides a basic platform for automation and intelligent maintenance of rolling stock using image processing methods. A shape prior segmentation model is proposed for segmenting parts of a train rolling stock moving a 30KMPH. High speed video capture is used to to capture the movement of train without blurring. Chan Vese active contour model is upgraded with shape prior term. Seven parts of a train are segmented for 10,000 frames. Similarity index with mean square error between the simulated and dictionary segments is around 89% making the algorithm robust to brightness variations during video capture. Further this system can be upgraded to classify defects and point the location of defects in the bogies using pattern classifiers.

Fig.12. Comparing the simulated results of proposed segmentation algorithm with ground truth for various parts of train rolling stock.

5. Acknowledgements The authors like to thank personal of Indian Railways for encouragement throughout this work and explaining procedures of rolling stock examination. Also we want to thank K.L.University management for supporting our research at every stage. Finally railway personal in helping us capture videos of bogies.

P.V.V. Kishore et al / Materials Today: Proceedings 4 (2017) 1730–1739

1739

References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21]

http://www.intlrailsafety.com/capetown/3_024_amitabh.doc. Ashwin.T, S Ashok, Automation of Rolling Stock Examination, Proceeding of IEEE International Conference on Advanced Communication Control and Computing Technologies (ICACCCT),pp260-263,2014. Kishore, P.V.V.; Prasad, C.R., "Train rolling stock segmentation with morphological differential gradient active contours," in Advances in Computing, Communications and Informatics (ICACCI), 2015 International Conference on , vol., no., pp.1174-1178, 10-13 Aug. 2015. doi: 10.1109/ICACCI.2015.7275770 Wang Lingzhi; Xu Yugong; Zhang Jiadong, "Importance analysis on components in railway rolling stock based on fuzzy weighted logarithmic least square method," in Intelligent Computing and Intelligent Systems (ICIS), 2010 IEEE International Conference on , vol.1, no., pp.175-179, 29-31 Oct. 2010. doi: 10.1109/ICICISYS.2010.5658651 Boullie, J.-B.; Brun, M., "A new rolling stock architecture using safety computers and networks," in Dependable Systems and Networks, 2000. DSN 2000. Proceedings International Conference on , vol., no., pp.157-162, 2000. doi: 10.1109/ICDSN.2000.857529. HyunCheol Kim and Whoi-Yul Kim,Automated Inspection System for Rolling Stock Brake Shoes, IEEE transactions on instrumentation and measurement, Vol. 60, n. 8,pp.2835-2847,2011. H. Sato, H. Nishii, and S. Adachi, “Automatic thickness measuring system by image analysis for brake shoes of traveling rolling stock,” Kawasaki Steel Tech. Rep., vol. 27, pp. 77–83, 1992. H. Kim and W.-Y. Kim, Automated thickness measuring system for brake shoe of rolling stock, Proceedings of 9th IEEE Workshop Applications of Computer Vision, pp. 1–6, 2009. J.-W. Hwang, H. Kim, Y.-M. Baek, and W.-Y. Kim, “Image analysis system for measuring the thickness of train brakes,” in Proc 1st IEEEEastern Eur. Conf. Eng. Comput. Based Syst., 2009, pp. 83–87. Jungwon Hwang; Hu-Young Park; Whoi-Yul Kim, "Thickness measuring method by image processing for lining-type brake of rolling stock," in Network Infrastructure and Digital Content, 2010 2nd IEEE International Conference on , vol., no., pp.284-286, 24-26 Sept. 2010.doi: 10.1109/ICNIDC.2010.5657787. Ginmo Chung and Luminita Vese. (2003) “Image segmentation using a multilayer level set approach.Technical Report” 03-53, UCLA. R. Malladi, J.A. Sethian, and B.C. Vemuri. “Shape modeling with front propagation: A level set approach,”IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol.17,no.2,pp.158-175,Feb,1995. D. Terzopoulos and K. Fleischer, “Deformable models,”The Visual Computer, vol.4, no.6, pp.306-331,jun, 1988. D. Terzopoulos, J. Platt, A. Barr, and K. Fleischer, “Elastically deformable models” In Computer Graphics Processing, pp. 205-214, ACM Press/ ACM SIGGRAPH,1987. Luminita Vese and Tony Chan, “A multiphase level set framework for image segmentation using the mumford and shah model,”International Journal of Computer Vision, vol.50,No. 3, pp.271-293,mar,2002. M.Kass, A Witkin, D Terzopoulos ,“Snakes:active Contour Models”, International. Journal. of Computer Vision, pp. 321-331,1987. Chan.T, Vese.L.A, “Active contours without edges,”IEEE Transactions onImage Processing, vol.10, ,no.2, pp.266–277,feb,2001. D. Mumford and J. Shah. “Optimal approximation by piecewise smooth functions and associated variational problems,” Comm. Pure Appl. Math, vol.42,pp.577-685, 1989. Osher, S., Sethian, J,“Fronts propagating with curvature-dependant speed: algorithms based on Hammilton-Jacobi formulations,”Journal ofComputational Physics,vol.79, No.1, pp.12–49,jan,1988. D. Cremers, S.J. Osher, S. Soatto, Kernel density estimation and intrinsicalignment for shape priors in level set segmentation, International Journal of Computer Vision, Vol.69 .No. 3, pp.335–351, 2006. Chang-Hsing Lee; Ling-Hwei Chen; Wei-Kang Wang, "Image contrast enhancement using classified virtual exposure image fusion," Consumer Electronics, IEEE Transactions on , vol.58, no.4, pp.1253,1261, November 2012.