- Email: [email protected]
Image-Based Process Monitoring Using Deep Belief Networks
Advanced Control of Chemical Processes Proceedings, 10th IFAC International Symposium on Shenyang, Liaoning, China, July 25-27, 2018 online at www.sciencedirect.com Available Advanced Control Chemical Processes Proceedings, 10th of IFAC International Symposium on Proceedings, 10th IFAC International Symposium on Shenyang,Control Liaoning, July 25-27, 2018 Advanced of China, Chemical Processes Advanced Control of Chemical Processes Shenyang, Liaoning, China, July 25-27, 2018 Shenyang, Liaoning, China, July 25-27, 2018
ScienceDirect
IFAC PapersOnLine 51-18 (2018) 115–120 Image-Based Process Monitoring Using Deep Belief Networks Image-Based Process Monitoring Using Deep Belief Networks Yuting Lyu*, Junghui Chen**, Image-Based Monitoring Using Deep Belief Image-Based Process Process Monitoring UsingZhihuan DeepSong* Belief Networks Networks
Yuting Lyu*, Junghui Chen**, Zhihuan Song* Yuting Lyu*, Junghui Chen**, Song* Lyu*, Junghui Institute Chen**,ofZhihuan Zhihuan * State Key Laboratory of IndustrialYuting Control Technology, IndustrialSong* Process Control, Department of Control Science of and Engineering, Zhejiang University, Hangzhou 310027, Zhejiang, P. R.Department China * State Key Laboratory Industrial Control Technology, Institute of Industrial Process Control, of Control (e-mail: [email protected]; [email protected] ).Control, ** State Key Laboratory of Industrial Control Technology, Institute of Industrial Process Department Science and Engineering, Zhejiang University, Hangzhou 310027, Zhejiang, P. R. China State Key Laboratory of Industrial Control Technology, Institute of Industrial Process Control, Department of of Control Control ** Department of Chemical Engineering, Chung-Yuan Christian 310027, University, Chung-Li, Taiwan, R.O.C Science and Engineering, Zhejiang University, Hangzhou Zhejiang, P. R. China (e-mail: [email protected]; [email protected] ). Science and Engineering, Zhejiang University, Hangzhou 310027, Zhejiang, P. R. China (e-mail: [email protected] ) (e-mail: [email protected]; ). ** Department of Chemical Engineering, [email protected] Christian University, Chung-Li, Taiwan, R.O.C (e-mail: [email protected]; [email protected] ). ** Department of Chemical Engineering, Chung-Yuan Christian University, Chung-Li, (e-mail: [email protected] ) ** Department of Chemical Engineering, Chung-Yuan Christian University, Chung-Li, Taiwan, Taiwan, R.O.C R.O.C (e-mail: (e-mail: [email protected] [email protected] )) Abstract: With the advances in optical sensing and image capture systems, process images certainly offer new With perspectives to process monitoring. the process dataprocess collected by traditional Abstract: the advances in optical sensingCompared and imagetocapture systems, images certainly sensors at local regions, process images, which can capture more significant variations in the space, Abstract: With the advances in optical sensing and image capture systems, process images certainly offer new With perspectives to process monitoring. the process dataprocess collected bywhole traditional Abstract: the advances in optical sensingCompared and imagetocapture systems, images certainly enhance the monitoring performance in data-driven monitoring methods. In this paper, a popular deep offer new perspectives to process monitoring. Compared to the process data collected by traditional sensors at local regions, process images, which can space, capture more variations in thebywhole offer new perspectives to process monitoring. Compared to thesignificant process data collected traditional learningatthe method, namelyprocess deep belief network (DBN), is more applied to effectively extract useful features sensors local regions, images, which can capture significant variations in the whole space, enhance monitoring performance in data-driven monitoring methods. In this paper, a popular deep sensors at local regions, process images, which can capture more significant variations in the whole space, from the method, images. Meanwhile, new statistic is (DBN), developed for themethods. DBN model, which integrates feature enhance in data-driven monitoring In paper, popular deep learning namelyperformance deepa belief network is applied to effectively features enhance the the monitoring monitoring performance in data-driven monitoring methods. In this this extract paper, aauseful popular deep extraction and fault detection into onestatistic model israther thanisseparately accomplish them. The effectiveness learning method, namely deep belief network (DBN), applied to effectively extract useful features from the images. Meanwhile, a new developed for the DBN model, which integrates feature learning method, namely deep belief network (DBN), is applied to effectively extract useful features of the proposed DBN based monitoring method isdeveloped demonstrated in a real combustion system. from Meanwhile, aa new is for model, integrates feature extraction and fault detection into onestatistic model than separately accomplish them. The effectiveness from the the images. images. Meanwhile, new statistic israther developed for the the DBN DBN model, which which integrates feature extraction and fault detection into one model rather than separately accomplish them. The effectiveness of the proposed DBN based monitoring method is demonstrated in a real combustion system. extraction and faultMonitoring, detection into oneBelief ratherControl) than separately accomplish them. The © 2018, IFAC (International Federation ofmodel Automatic Hosting by Elsevier Ltd. All Process rightseffectiveness reserved. Keywords: Process Deep Network, Deep Learning, Fault Detection, Images. of of the the proposed proposed DBN DBN based based monitoring monitoring method method is is demonstrated demonstrated in in aa real real combustion combustion system. system. Keywords: Process Monitoring, Deep Belief Network, Deep Learning, Fault Detection, Process Images. Keywords: Deep Fault Detection, Process Images. Keywords: Process Process Monitoring, Monitoring, Deep Deep Belief Belief Network, Network, Deep Learning, Learning, Fault Detection, Images. process images is enriched, howProcess to convert the huge number 1. INTRODUCTION of images into usable information is convert an enormous challenge. process images is enriched, how to the huge number 1. INTRODUCTION process images is enriched, how to convert the huge number With the rapid development of the modern industry, of images into usable information is an enormous challenge. process images is are enriched, how to convert the huge spectral number 1. INTRODUCTION In general, there two types of image features: 1.are INTRODUCTION of images into usable information is an enormous challenge. industrial processes becoming more and more complex. With the rapid development of the modern industry, features of images and into usable information is an enormous challenge. spatial features. With these features, the The demand for safe and the stable product quality In general, there are two types of image features: spectral With rapid modern industry, industrial areoperation becomingof more complex. With the the processes rapid development development ofmore the and modern industry, In multivariate statistical process control (MSPC) model can be general, there are two types of image features: spectral features andthere spatial features. With these features, the general, are two types of image features: spectral givesdemand rise toforpromising development in online process industrial are becoming more and more The safe stable quality In industrial processes processes areoperation becomingand more and product more complex. complex. established. Multivariate image analysis (MIA) is the most features and spatial features. With these features, the multivariate canthe be features andstatistical spatial process features.control With(MSPC) these model features, monitoring fault diagnosis. Among all product the monitoring The safe and quality gives rise and tofor development in online process The demand demand forpromising safe operation operation and stable stable product quality multivariate widely used method ofprocess extracting spectral features. Bharati et statistical control (MSPC) model can be established. Multivariate image analysis (MIA) is the most multivariate statistical process control (MSPC) model can be methods, data-driven methods without physical models or gives to development process monitoring fault diagnosis. Among in all online the monitoring gives rise rise and to promising promising development in online process established. al. (2003) applied MIA on lumber images to isdetect the Multivariate image analysis (MIA) the most widely used method of extracting spectral features. Bharati et Multivariate image analysis (MIA) is the most expert knowledge have won massive popularity in recent monitoring and Among all methods, methods without models or established. monitoringdata-driven and fault fault diagnosis. diagnosis. Amongphysical all the the monitoring monitoring presence and quantity of common lumber defects in forest widely used method of extracting spectral features. Bharati al. (2003) applied MIA on lumber images to detect the widely used method of extracting spectral features. Bharati et et years. The most widely used techniquesin include methods, data-driven methods without physical or expert knowledge have won data-driven massive recent methods, data-driven methods without popularity physical models models or al. products. Yuapplied and Macgregor (2004) combined MIA andforest PLS (2003) MIA on lumber images to detect the presence and quantity of common lumber defects in al. (2003) applied MIA on lumber images to detect the principal component analysis (PCA), partial least square expert knowledge have won massive popularity in recent years. The most widely techniquesin include expert knowledge have used won data-driven massive popularity recent to efficiently extract information from the rapidly timepresence quantity of defects in Yu and Macgregor (2004) lumber combined MIA andforest PLS presence and and quantity of common common lumber defects in forest (PLS) and most their extensions (Qin(PCA), and Zheng, 2013, and products. years. The used techniques include principal least Ge square years. Thecomponent most widely widelyanalysis used data-driven data-drivenpartial techniques include varying flame images and predict the boiler performance. products. Yu and Macgregor (2004) combined MIA and to efficiently extract information from the rapidly timeproducts. Yu and Macgregor (2004) combined MIA and PLS PLS Song, and 2012). in (PCA), the methods, principal analysis partial least square (PLS) their However, extensions (Qin and data-driven Zheng, and Although principal component component analysis (PCA), partial2013, least Ge square the MIA method of interest in many industrial extract information from the timevarying flame images and is predict the boiler performance. to efficiently efficiently extract information from the rapidly rapidly timemonitoring an industrial process is data-driven realized2013, through the to (PLS) and their extensions (Qin and Zheng, Ge and Song, 2012). However, in the methods, (PLS) and their extensions (Qin and Zheng, 2013, Ge and processes, there aremethod some drawbacks. Firstly, the spatial varying images and predict the performance. Although the MIA of interest in many industrial varying flame flame images and is predict the boiler boiler performance. process 2012). measurements of physical properties, as flow Song, However, in methods, monitoring an industrial realized such through the Although Song, 2012). However, process in the the is data-driven data-driven methods, information is lost in the process of unfolding each image to the MIA method is of interest in many industrial processes, the there some isdrawbacks. spatial MIAaremethod of interest Firstly, in manythe industrial rates, temperatures, and which just provide a Although monitoring an process is through the process measurements of pressures, physical as flow monitoring an industrial industrial process properties, is realized realized such through the processes, one-dimensional vectors. Secondly, masks, aneach area selected there are some drawbacks. Firstly, the spatial information is lost in the process of unfolding image to limited representationof of the process because the processes, there are some drawbacks. Firstly, the spatial process measurements physical such as rates, which just provide processtemperatures, measurementsand of pressures, physical properties, properties, such as flow flowa one-dimensional in the score space tointrack the number of pixels, are manually information is lost the process of unfolding each image vectors. Secondly, masks, an area selected to measurements are sensed atpressures, some regions because instead of the the information is lost in the process of unfolding each image to rates, and which limitedtemperatures, representation thelocal process rates, temperatures, and of pressures, which just just provide provide aa in determined most one-dimensional vectors. Secondly, an area selected the score in space toMIA trackapplications. the numbermasks, of pixels, are manually one-dimensional vectors. Secondly, masks, an area selected whole region. Furthermore, if engineers are lack of limited representation of the process because the measurements are sensed atofsome regions because instead of the in the score space to track the number of pixels, are manually limited representation thelocal process in mosttoMIA applications. in the score space track the number of pixels,characterizing are manually process knowledge, the sensing points would be improper. measurements are sensed local the determined Besides thein spectral features, spatial features whole region. Furthermore, if engineers are instead lack ofof measurements are sensed at at some some local regions regions instead of the determined most MIA applications. determined in most MIA applications. The effect of poor monitoring and fault diagnosis is improper. notof easily whole region. Furthermore, if engineers are lack the the spatial variation in pixel intensities also play an important process knowledge, the sensing points would be whole region. Furthermore, if engineers are lack of the Besides the spectral features, spatial features characterizing avoided. process knowledge, the sensing points would be improper. role in process monitoring. The wavelet texture analysis The effect of poor monitoring and fault diagnosis is not easily Besides the spectral features, spatial features characterizing the spatial pixel intensities also playcharacterizing an important process knowledge, the sensing points would be improper. Besides thevariation spectralinfeatures, spatial features The effect of poor monitoring and fault diagnosis is not easily (WTA) and the grey level co-occurrence matrix (GLCM) are avoided. the spatial variation in pixel intensities also play an important rolespatial in process monitoring. The wavelet texture analysis The effect poor monitoring andsensing fault diagnosis is not image easily the variation in pixel intensities also play an important With the of advances in optical and digital avoided. regarded as state-of-the-art methods for spatial feature role in process monitoring. The wavelet texture analysis (WTA) the grey level co-occurrence matrix (GLCM) are avoided. in and process monitoring. The wavelet texture analysis processing techniques,in on-line based on process With the advances opticalmonitoring sensing and digital image role extraction inthe the machine vision area. They have also been (WTA) and grey level co-occurrence matrix (GLCM) are regarded as state-of-the-art methods for spatial feature (WTA) and the grey level co-occurrence matrix (GLCM) are images offers new perspectives to process monitoring in the With optical sensing digital image processing techniques,in based on process With the the advances advances in on-line opticalmonitoring sensing and and digital image regarded used to deal with process images. WangThey and Ren (2014) and as state-of-the-art methods for spatial feature extraction in the machine vision area. have also been regarded as state-of-the-art methods for spatial feature area of combustion processes, forest products, rolled steel processing techniques, on-line monitoring based on process images offers new perspectives to process monitoring in the Bai processing techniques, on-line monitoring based on process ettoal.deal (2017) analyzed the flame images combustion extraction in the vision They have also used process images. Wang andfrom Ren and extraction inwith the machine machine vision area. area. They have(2014) also been been sheets, and sonew on. Conventionally, process measurements images process monitoring the area of offers combustion processes, to forest products, rolledin images offers new perspectives perspectives to process monitoring insteel the with GLCM to characterize the second order statistics of used to deal with process images. Wang and Ren (2014) Bai et al. (2017) analyzed the flame images from combustion and collected at sotheon.local regions were used measurements for process used to deal with process images. Wang and Ren (2014) and area of combustion processes, products, rolled sheets, Conventionally, area of and combustion processes, forest forest process products, rolled steel steel pixel intensity. WTA is widely used in the fields of paper al. the images from combustion with to analyzed characterize the second of Bai et etGLCM al. (2017) (2017) analyzed the flame flame imagesorder from statistics combustion monitoring. In the this paper, process images are to Bai sheets, Conventionally, process measurements collected regions were usedused forinstead process sheets, and andat so so on. on.local Conventionally, process measurements formation (Reis and Bauer, 2009), manufactured countertop with GLCM to characterize the second order statistics of pixel intensity. WTA is widely used in the fields of paper with GLCM to characterize the second order statistics of help monitor larger spacesprocess and they can used recordforsignificant collected local were process monitoring. this paper, to pixel collected at atIn the the local regions regions images were are usedused forinstead process slabs (Liu and Macgregor, 2006) and so on. Obviously, these intensity. WTA is widely used in the fields of paper formation (Reis and Bauer, 2009), manufactured countertop pixel intensity. WTA is widely used in the fields of paper variations of the operating unit which may be neglected by monitoring. this images used instead help monitorIn spacesprocess and they can are record monitoring. Inlarger this paper, paper, process images are used significant instead to to formation methods features from images inflexible way, (Reis and 2009), countertop slabs (Liuextract and andmanufactured soinon.anObviously, these (ReisMacgregor, and Bauer, Bauer,2006) 2009), manufactured countertop sensors at oflocal regions. Although the information from help monitor larger spaces and they can record significant variations the operating unit which may be neglected by formation help monitor larger spaces and they can record significant because they may causefrom some important features tothese be slabs (Liu and Macgregor, 2006) and so on. Obviously, methods extract features images in an inflexible way, slabs (Liu and Macgregor, 2006) and so on. Obviously, these variations the unit may be by sensors regions. Although from variationsat of oflocal the operating operating unit which whichthe mayinformation be neglected neglected by because methods extract features from images in an inflexible way, they may cause some important features to be methods extract features from images in an inflexible way, sensors sensors at at local local regions. regions. Although Although the the information information from from because they may cause some important features to be because they may cause some important features to be
Copyright © 2018, 2018 IFAC 115 Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Peer review©under of International Federation of Automatic Copyright 2018 responsibility IFAC 115Control. 10.1016/j.ifacol.2018.09.285 Copyright © 2018 IFAC 115 Copyright © 2018 IFAC 115
2018 IFAC ADCHEM 116 Shenyang, Liaoning, China, July 25-27, 2018Yuting Lyu et al. / IFAC PapersOnLine 51-18 (2018) 115–120
omitted. A few approaches to combine spectral and spatial features such as multi-resolution MIA (Liu and Macgregor, 2007) were also proposed. Although they gave noticeable performance enhancement in applications, some of the aforementioned drawbacks remain unsolved. Recently, the deep learning (Lecun et al., 2015, Yoshua et al., 2004) has been attracting great attention as they can potentially extract different level features for better classification and recognition. One representation of deep learning is deep belief network (DBN) (Hinton et al., 2006). It has been widely used in a variety of fields, such as handwritten digits recognition, speech recognition, and many other applications (Zhao et al., 2017, Hao and Tian, 2017). In particular, DBN has also been applied to data-driven industrial processes for classification (Gao et al., 2014) and soft sensors (Shang et al., 2014). However, to our best knowledge, DBN has never been applied to online monitoring, particularly for the process images collected from industrial combustion processes.
Fig. 1. The diagram of RBM The probability of any possible visible and hidden vector is given based on the energy as: e E ( v ,h ) (2) P ( v, h) Z where Z v ,h e E ( v ,h ) is the normalizing factor. Since there are no direct connections within each layer, one is able to derive the conditional probabilities P ( v h) and P ( h v ) as follows:
In this paper, a novel DBN based framework is proposed to extract features from process images and to conduct online monitoring and detection of the current status of the combustion process. In this framework, the spectral and spatial features are extracted simultaneously. Meanwhile, a new statistic is first-ever proposed for DBN. Through the process image-based DBN model, feature extraction and process monitoring can be realized in one model. This opens a new area in image-based process monitoring.
j 1
(4)
i 1
P( v )
P ( v, h ) h
1 Z
e h
E ( v,h )
(5)
The training results can be summarized as follows.
wij ( vi h j data vi h j recon ) bi ( vi data vi recon ) c j ( h j data h j recon )
(6) (7) (8)
where is a learning rate, data denotes expectation over the training data, recon denotes expectation over the reconstructed data.
As an energy-based model, RBM has an energy given by i 1
nh
Then the stochastic gradient descent and contrastive divergence (Hinton, 2002) are used for training an RBM with the objective of maximizing the log-likelihood function of (5).
RBM is a type of undirected graphical model composed of a visible layer representing the input data and a hidden layer representing the latent variables. The visible and hidden units are connected using symmetrical weights with no visiblevisible or hidden-hidden connections as shown in Fig.1. nv
P ( vi 1 h) (bi wij h j )
2.1 RBM Model
nh
(3)
i 1
For RBM models, {W, b, c} are the parameters to be trained. Before training these parameters, the probability of a given input vector is assigned by RBM as follows:
2. BACKGROUND
nv
P ( h j 1 v ) ( c j wij vi )
where denotes the sigmoid function, ( x ) (1 e x ) 1 .
The remainder of this paper is organized as follows. The next section gives the background of the restricted Boltzmann machine (RBM) and DBN. In Section 3, image preprocessing techniques are presented. Then a new statistic for DBN is defined and kernel density estimation is used to determine its control limits. Next, the whole monitoring procedure using process images is summarized. In Section 4, the proposed method is evaluated using the flame images collected from a real combustion system. Finally, conclusions and the future research directions are presented.
nv
nh
E ( v, h) bi vi c j h j h j wij vi i 1 j 1
(1)
where vi and h j are the binary states of the visible unit i
and the hidden unit j , bi and c j are their biases respectively, and wi , j is the weight between them.
116
Although RBM with the energy function of (1) is beneficial for binary data, such as in the case of handwritten recognition where the image color is either black or white, it should be mentioned that the information in the industrial image is much more complex and the data are typically represented by real-valued vectors. In order to tackle those data, binary RBMs are extended to the Gaussian–Bernoulli restricted Boltzmann machine (GRBM) (Hinton, 2012). The corresponding normalized energy function is:
2018 IFAC ADCHEM Shenyang, Liaoning, China, July 25-27, 2018Yuting Lyu et al. / IFAC PapersOnLine 51-18 (2018) 115–120
E ( v , h)
nv nh （vi bi ) 2 nh c j h j h j w j ,i vi 2 1 j 1 i 1 j 1
nv
i
1 T v v bT v cT h hT Wv 2 In GRBM, (3) does not change, but (4) becomes
(9)
nh
P( vi h) ~ N (bi w j ,i h j ,1)
(10)
j 1
Note that the differences between GRBM and RBM only lie in the energy function and in the way of generating the reconstructions, but all update rules for weights and biases remain the same. In this paper, all the RBMs in DBN are selected as the GRBM.
117
arrange the three-dimensional images into one-dimensional vectors as shown in Fig.3. Suppose the image in each channel is of the size N1 N 2 , then the one-dimensional vector arranged from three channels is of the length M N1 N 2 3 . In other words, with a sequence of process images, each image corresponds to a one1, , , and the matrix input dimensional vector x k R M , k into DBN is X [x1 x ]T R M . Once the image data are used to train the network, the parameters of DBN can be determined and prepared for on-line monitoring.
xK
hL
h2
h*L
x2
v L ( h L1 )
x1
h1
h*2
h1*
v 2 ( h1 )
v 2 ( h1 )
v1
v1
v1
Fig. 3. Unfolded structures of the images with RGB channels 3.2 Construction of Statistic
Fig. 2. The training procedures of a multi-level DBN structure with smoothing
Like conventional process monitoring schemes, on-line process monitoring is carried out to detect the abnormal condition after off-line training DBN so that the operators can remove the faults early enough to avoid process deviation. Statistics play an important role in this procedure. Generally, for some traditional multivariate statistical methods like PCA, the Hotelling T 2 statistic and the SPE statistic are constructed to monitor the principal component space and its residual space, separately. Whether the test data are abnormal can be estimated by comparing the statistics of the data to the corresponding control limits.
2.2 DBN Model DBN is composed of a series of stacked RBMs. After an RBM has been trained, the activation probabilities of its hidden units can be computed and they are used as the input data for learning a higher-level RBM (Geoffrey and Hinton, 2007). In this way, each hidden layer in DBN performs a nonlinear transformation on its input data to actually describe the underlying features behind the input data. Fig.2 depicts the framework of DBN used in the present study. To avoid larger fluctuations of the process image data, the hidden variables h* are smoothed in the time domain to h before it is sent to the higher levels. This framework is designed for process image-based monitoring since the features extracted from the process images often involve frequent fluctuations caused by irregular blinks in the images.
However, in the procedure of images based process monitoring using DBN, the application of the T 2 statistic is limited because the distribution of the variables in hidden layers is inconclusive. Meanwhile, there are frequent fluctuations in the reconstruction value of each pixel. When they are added up to calculate the SPE statistic, huge fluctuations are accumulated. These huge fluctuations will cause perturbations to the determination of the control limit and consequently pose certain difficulty for fault detection.
3. METHODOLOGY
Thus, a new statistic, Energy , is introduced and it is defined as the energy of the top-level RBM:
3.1 Image Pre-processing
1 T v L v L b LT v L c LT h L h LT WL v L (11) 2 where L denotes the L-th layer on the top of DBN.
In a real industrial process, the colors of the process images obtained by digital color cameras are encoded in an RGB format. Nonetheless, the inputs and outputs of DBN have to be a one-dimensional vector. In order to extract useful features from the process images by DBN, one needs to
E ( v L , h L )
The principle of this statistic can be explained from the point of RBM training. In the training phase, RBM is trained with 117
2018 IFAC ADCHEM 118 Shenyang, Liaoning, China, July 25-27, 2018Yuting Lyu et al. / IFAC PapersOnLine 51-18 (2018) 115–120
the objective of maximizing the log-likelihood function of P ( v ) identified with minimizing the energy of model. In other words, when no faults occur in online monitoring, the energy of testing data is relatively low with small fluctuation. Once a fault occurs, the testing sample will produce a larger deviation with the statistical model and the energy of RBM will exceed its control limit.
and the air/fuel flow rates are automatically recorded through a SCADA system. Meanwhile, the concentrations of NOx, SOx, O2 and CO in the exhaust gas are measured by a gas analyzer. The flame images in the furnace are captured by a digital color camera. In this experiment, the camera with the specifications of 658 492 pixels and a resolution of 24 bits per pixel is used.
To construct a statistical model, the top-level RBM is chosen. The reason is obvious, because each hidden layer in DBN extracts features from the outputs of its previous layer, and the features extracted in the low-level contain more useless information. To avoid the features of the low level that deteriorate the monitoring effect, the top-level RBM is used here. It is more suitable and beneficial to construct statistics in the present study. Finally, to estimate whether fault information exists at a sample point, the confidence limit of Energy statistic should be determined. Because the distribution of this statistic is unknown, kernel density estimation is used here to calculate the control limit of the Energy statistic.
In the present study, the training dataset contains 1,500 flame images drawn from the normal condition and the testing dataset contains the normal and the abnormal conditions. Because of the space limit, only one fault condition resulted from the decrease of the ratio of air to fuel is applied here. A total of 2,100 flame images are used to detect the abnormal condition. For easy visualization and comparisons, five sequential images in the normal condition and the abnormal condition are respectively shown as examples in Fig.4(a) and Fig.4(b). Since the abnormal operation behavior is minor, the differences between the normal and the abnormal conditions cannot be distinguished by directly watching the images. Experimental furnace
Digital camera
3.3 Procedure of Process Monitoring based on DBN In summary, process image based DBN for process monitoring goes through the following steps:
Flame image
Burner Cooling system
Air compressor
Step 1. Acquire process images from the digital color camera under the normal operation; then pre-process the images to acquire a one-dimensional vector for each image.
Fuel tank
Step 2. Input the normal image data into the network; then train each individual RBM successively to constitute the DBN model.
Fig. 4. Scheme of the experimental combustion system (a)
Step 3. Calculate the Energy statistic of the normal image data and determine the control limit of Energy by kernel density estimation. Step 4. Acquire testing images and input them into DBN to extract useful features and calculate the monitoring statistic Energy after pre-processing.
(b)
Step 5. Determine whether a fault occurs by comparing the statistics of each testing image with the control limit of Energy .
Fig. 5. Five sequential images of (a) normal and (b) abnormal conditions Such abnormal conditions are detected by the proposed DBN. To demonstrate the effectiveness of the proposed DBN, three traditional approaches, namely, PCA, MIA and GLCM are applied to furnace monitoring for comparisons. The process variables (such as the concentration of CO, the concentration of NO, etc.) are used in PCA and process image data are applied to MIA and GLCM. The extracted features of MIA and GLCM are smoothed in the same way as the smoother of DBN.
4. CASE STUDY In order to evaluate the proposed framework of the processimage based DBN, flame images collected from a real combustion furnace are used. The experimental combustion system is shown in Fig.4. It consists of an experimental furnace, a burner, an air compressor, a fuel tank, a cooling system and a digital camera. The model of the burner is NA 5514-6 manufactured by North American Manufacturing Company. The volume of the experimental furnace is 2.5m 2.5m 1.5m . During the operation, the process data such as the steam pressure, the temperature in the furnace,
Because MIA and GLCM can merely extract spectral features and spatial features from flame images, further analysis of these features is still performed using PCA. Thus, the 118
2018 IFAC ADCHEM Shenyang, Liaoning, China, July 25-27, 2018Yuting Lyu et al. / IFAC PapersOnLine 51-18 (2018) 115–120
(b)
statistics T 2 and SPE in PCA, MIA-PCA and GLCM-PCA are constructed separately, but the Energy statistic of the proposed DBN model is used. The detailed monitoring results for the testing dataset are shown in Fig. 6. The blue curves represent the values of the statistics for each testing sample point and the red straight line represents the control limits. The significance level is 0.99. In Fig. 6(a), only process data are used, so the fault is almost not detected by the T 2 statistic because of the lack of adequate representation of the current status. The fault studied in this case is a step change occurs in the air-fuel ratio of the experimental system at the 1,300th point in the testing dataset. In Fig. 6(b) and Fig. 6(c), although the features extracted from the flame images are smoothed with the same time window as DBN, the statistics after the 1,300th point fluctuate around the control limit. In Fig. 6(d), obviously the Energy statistic of DBN performs better than the results of the other three traditional methods.
40
T2
30 20 10 0 0
300
600
900 1200 samples
1500
1800
2100
300
600
900 1200 samples
1500
1800
2100
SPE
15 10 5 0 0
(c)
In order to demonstrate the effectiveness of the proposed method explicitly, the false alarm rates for normal data and the fault detection rates for faulty data using different methods are summarized in Table 1. The best monitoring results obtained in this testing phase have been highlighted. From this table, one can see that the performance of the proposed method is much better than the other four methods. Significant improvements have been made in the fault detection rate on the premise that images are smoothed with the same window width in each method. Although the false alarm rate of the proposed method is not the smallest, it is very close to the smallest one obtained by the process data based PCA.
40
T2
30 20 10 0
0
300
600
900 1200 samples
1500
1800
2100
300
600
900 1200 samples
1500
1800
2100
1500
1800
2100
15
SPE
10 5 0
0
(a)
(d)
25
140.3
20 15 Energy
T2
119
10 5 0 0
300
600
900 1200 samples
1500
1800
140.2
140.1 0
2100
25
300
600
900 1200 samples
SPE
20
Fig. 6. Process monitoring results for the combustion furnace: (a) PCA, (b) MIA-PCA, (c) GLCM-PCA, and (d)DBN. PCA is based on process variable data, and MIA-PCA, GLCMPCA, and DBN are based on process image data.
15 10 5 0 0
300
600
900 1200 samples
1500
1800
2100
5. CONCLUSIONS This paper develops a novel monitoring method with process image based DBN to detect a combustion system. Instead of using process variables, which cover limited information of a process, this paper uses the process images captured by the digital camera to establish the DBN model and achieve process monitoring with the newly constructed statistic. The proposed method has two main advantages over the 119
2018 IFAC ADCHEM 120 Shenyang, Liaoning, China, July 25-27, 2018Yuting Lyu et al. / IFAC PapersOnLine 51-18 (2018) 115–120
experiment results show that the proposed method is superior to the other traditional image-based monitoring methods. In the future, the process data and process images will be combined to make up a complete dataset, which is beneficial for process monitoring.
traditional process image-based monitoring methods. First, it is more flexible in feature extraction and able to dig out more informative features via deep network structure. Second, the proposed method integrates feature extraction and abnormal condition detection into one model rather than separately accomplish them, making it more convenient and efficient. The effectiveness of the proposed method is demonstrated through the flame images obtained from a furnace. The
Table 1. Fault detection rates and false alarm rates of combustion furnace based on different methods Methods
Process variable data PCA
Process image data MIA-PCA
GLCM-PCA
DBN
Statistics
T2
SPE
T2
SPE
T2
SPE
Energy
Fault Detection Rate
0.0262
0.5938
0.4700
0.3187
0.5250
0.2338
0.9563
False Alarm Rate
0.0038
0.0108
0.0200
0.0138
0.0100
0.0246
0.0069
Hinton, G. E. (2012). A practical guide to training restricted boltzmann machines. Momentum, 9(1), 599-619. Lecun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444. Liu, J. J., & Macgregor, J. F. (2006). Estimation and monitoring of product aesthetics: application to manufacturing of “engineered stone” countertops. Machine Vision & Applications, 16(6), 374-383. Liu, J. J., & Macgregor, J. F. (2007). On the extraction of spectral and spatial information from images. Chemometrics & Intelligent Laboratory Systems, 85(1), 119-130. QIN, S. J. & ZHENG, Y. (2013). Quality-relevant and process-relevant fault monitoring with concurrent projection to latent structures. AIChE Journal, 59, 496504. Reis, M. S., & Bauer, A. (2009). Wavelet texture analysis of on-line acquired images for paper formation assessment and monitoring. Chemometrics & Intelligent Laboratory Systems, 95(2), 129-137. Shang, C., Yang, F., Huang, D., & Lyu, W. (2014). Datadriven soft sensor development based on deep learning technique. Journal of Process Control, 24(3), 223-233. Wang, J. S., & Ren, X. D. (2014). GLCM based extraction of flame image texture features and KPCA-GLVQ recognition method for rotary kiln combustion working conditions. International Journal of Automation and Computing, 11(01), 72-77. Yoshua Bengio, Aaron Courville, & Pascal Vincent. (2013). Representation learning: a review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8), 1798-828. Yu, H., & Macgregor, J. F. (2004). Monitoring flames in an industrial boiler using multivariate image analysis. Aiche Journal, 50(7), 1474–1483. Zhao, Z., Jiao, L., Zhao, J., Gu, J., & Zhao, J. (2017). Discriminant deep belief network for high-resolution sar image classification. Pattern Recognition, 61, 686-701.
ACKNOWLEDGEMENT This work is supported by National Key Research and Development Program of China (2017YFB0304203), National Nature Science Foundation of China (61573308), and Ministry of Science and Technology, Taiwan, R.O.C. under Grant MOST 106-2221-E-033-060-MY3. REFERENCES Bharati, M. H., Macgregor, J. F., & Tropper, W. (2003). Softwood lumber grading through on-line multivariate image analysis techniques. Industrial & Engineering Chemistry Research, 42(21). Bai, X., Lu, G., Hossain, M. M., Szuhánszki, J., Daood, S. S., & Nimmo, W., et al. (2017). Multi-mode combustion process monitoring on a pulverised fuel combustion test facility based on flame imaging and random weight network techniques. Fuel. Gao, X., Shang, C., Jiang, Y., Huang, D., & Chen, T. (2014). Refinery scheduling with varying crude: a deep belief network classification and multimodal approach. Aiche Journal, 60(7), 2525-2532. GE, Z. & SONG, Z. (2012). Multivariate Statistical Process Control: Process Monitoring Methods and Applications, Springer. Geoffrey E. Hinton. (2007). Learning multiple layers of representation. Trends Cogn Sci., 11(10), 428-434. Hao, X. L., & Tian, M. (2017). Deep belief network based on double weber local descriptor in micro-expression recognition. Hinton, G. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8), 1771-1800. Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18(7), 1527.
120
ScienceDirect
IFAC PapersOnLine 51-18 (2018) 115–120 Image-Based Process Monitoring Using Deep Belief Networks Image-Based Process Monitoring Using Deep Belief Networks Yuting Lyu*, Junghui Chen**, Image-Based Monitoring Using Deep Belief Image-Based Process Process Monitoring UsingZhihuan DeepSong* Belief Networks Networks
Yuting Lyu*, Junghui Chen**, Zhihuan Song* Yuting Lyu*, Junghui Chen**, Song* Lyu*, Junghui Institute Chen**,ofZhihuan Zhihuan * State Key Laboratory of IndustrialYuting Control Technology, IndustrialSong* Process Control, Department of Control Science of and Engineering, Zhejiang University, Hangzhou 310027, Zhejiang, P. R.Department China * State Key Laboratory Industrial Control Technology, Institute of Industrial Process Control, of Control (e-mail: [email protected]; [email protected] ).Control, ** State Key Laboratory of Industrial Control Technology, Institute of Industrial Process Department Science and Engineering, Zhejiang University, Hangzhou 310027, Zhejiang, P. R. China State Key Laboratory of Industrial Control Technology, Institute of Industrial Process Control, Department of of Control Control ** Department of Chemical Engineering, Chung-Yuan Christian 310027, University, Chung-Li, Taiwan, R.O.C Science and Engineering, Zhejiang University, Hangzhou Zhejiang, P. R. China (e-mail: [email protected]; [email protected] ). Science and Engineering, Zhejiang University, Hangzhou 310027, Zhejiang, P. R. China (e-mail: [email protected] ) (e-mail: [email protected]; ). ** Department of Chemical Engineering, [email protected] Christian University, Chung-Li, Taiwan, R.O.C (e-mail: [email protected]; [email protected] ). ** Department of Chemical Engineering, Chung-Yuan Christian University, Chung-Li, (e-mail: [email protected] ) ** Department of Chemical Engineering, Chung-Yuan Christian University, Chung-Li, Taiwan, Taiwan, R.O.C R.O.C (e-mail: (e-mail: [email protected] [email protected] )) Abstract: With the advances in optical sensing and image capture systems, process images certainly offer new With perspectives to process monitoring. the process dataprocess collected by traditional Abstract: the advances in optical sensingCompared and imagetocapture systems, images certainly sensors at local regions, process images, which can capture more significant variations in the space, Abstract: With the advances in optical sensing and image capture systems, process images certainly offer new With perspectives to process monitoring. the process dataprocess collected bywhole traditional Abstract: the advances in optical sensingCompared and imagetocapture systems, images certainly enhance the monitoring performance in data-driven monitoring methods. In this paper, a popular deep offer new perspectives to process monitoring. Compared to the process data collected by traditional sensors at local regions, process images, which can space, capture more variations in thebywhole offer new perspectives to process monitoring. Compared to thesignificant process data collected traditional learningatthe method, namelyprocess deep belief network (DBN), is more applied to effectively extract useful features sensors local regions, images, which can capture significant variations in the whole space, enhance monitoring performance in data-driven monitoring methods. In this paper, a popular deep sensors at local regions, process images, which can capture more significant variations in the whole space, from the method, images. Meanwhile, new statistic is (DBN), developed for themethods. DBN model, which integrates feature enhance in data-driven monitoring In paper, popular deep learning namelyperformance deepa belief network is applied to effectively features enhance the the monitoring monitoring performance in data-driven monitoring methods. In this this extract paper, aauseful popular deep extraction and fault detection into onestatistic model israther thanisseparately accomplish them. The effectiveness learning method, namely deep belief network (DBN), applied to effectively extract useful features from the images. Meanwhile, a new developed for the DBN model, which integrates feature learning method, namely deep belief network (DBN), is applied to effectively extract useful features of the proposed DBN based monitoring method isdeveloped demonstrated in a real combustion system. from Meanwhile, aa new is for model, integrates feature extraction and fault detection into onestatistic model than separately accomplish them. The effectiveness from the the images. images. Meanwhile, new statistic israther developed for the the DBN DBN model, which which integrates feature extraction and fault detection into one model rather than separately accomplish them. The effectiveness of the proposed DBN based monitoring method is demonstrated in a real combustion system. extraction and faultMonitoring, detection into oneBelief ratherControl) than separately accomplish them. The © 2018, IFAC (International Federation ofmodel Automatic Hosting by Elsevier Ltd. All Process rightseffectiveness reserved. Keywords: Process Deep Network, Deep Learning, Fault Detection, Images. of of the the proposed proposed DBN DBN based based monitoring monitoring method method is is demonstrated demonstrated in in aa real real combustion combustion system. system. Keywords: Process Monitoring, Deep Belief Network, Deep Learning, Fault Detection, Process Images. Keywords: Deep Fault Detection, Process Images. Keywords: Process Process Monitoring, Monitoring, Deep Deep Belief Belief Network, Network, Deep Learning, Learning, Fault Detection, Images. process images is enriched, howProcess to convert the huge number 1. INTRODUCTION of images into usable information is convert an enormous challenge. process images is enriched, how to the huge number 1. INTRODUCTION process images is enriched, how to convert the huge number With the rapid development of the modern industry, of images into usable information is an enormous challenge. process images is are enriched, how to convert the huge spectral number 1. INTRODUCTION In general, there two types of image features: 1.are INTRODUCTION of images into usable information is an enormous challenge. industrial processes becoming more and more complex. With the rapid development of the modern industry, features of images and into usable information is an enormous challenge. spatial features. With these features, the The demand for safe and the stable product quality In general, there are two types of image features: spectral With rapid modern industry, industrial areoperation becomingof more complex. With the the processes rapid development development ofmore the and modern industry, In multivariate statistical process control (MSPC) model can be general, there are two types of image features: spectral features andthere spatial features. With these features, the general, are two types of image features: spectral givesdemand rise toforpromising development in online process industrial are becoming more and more The safe stable quality In industrial processes processes areoperation becomingand more and product more complex. complex. established. Multivariate image analysis (MIA) is the most features and spatial features. With these features, the multivariate canthe be features andstatistical spatial process features.control With(MSPC) these model features, monitoring fault diagnosis. Among all product the monitoring The safe and quality gives rise and tofor development in online process The demand demand forpromising safe operation operation and stable stable product quality multivariate widely used method ofprocess extracting spectral features. Bharati et statistical control (MSPC) model can be established. Multivariate image analysis (MIA) is the most multivariate statistical process control (MSPC) model can be methods, data-driven methods without physical models or gives to development process monitoring fault diagnosis. Among in all online the monitoring gives rise rise and to promising promising development in online process established. al. (2003) applied MIA on lumber images to isdetect the Multivariate image analysis (MIA) the most widely used method of extracting spectral features. Bharati et Multivariate image analysis (MIA) is the most expert knowledge have won massive popularity in recent monitoring and Among all methods, methods without models or established. monitoringdata-driven and fault fault diagnosis. diagnosis. Amongphysical all the the monitoring monitoring presence and quantity of common lumber defects in forest widely used method of extracting spectral features. Bharati al. (2003) applied MIA on lumber images to detect the widely used method of extracting spectral features. Bharati et et years. The most widely used techniquesin include methods, data-driven methods without physical or expert knowledge have won data-driven massive recent methods, data-driven methods without popularity physical models models or al. products. Yuapplied and Macgregor (2004) combined MIA andforest PLS (2003) MIA on lumber images to detect the presence and quantity of common lumber defects in al. (2003) applied MIA on lumber images to detect the principal component analysis (PCA), partial least square expert knowledge have won massive popularity in recent years. The most widely techniquesin include expert knowledge have used won data-driven massive popularity recent to efficiently extract information from the rapidly timepresence quantity of defects in Yu and Macgregor (2004) lumber combined MIA andforest PLS presence and and quantity of common common lumber defects in forest (PLS) and most their extensions (Qin(PCA), and Zheng, 2013, and products. years. The used techniques include principal least Ge square years. Thecomponent most widely widelyanalysis used data-driven data-drivenpartial techniques include varying flame images and predict the boiler performance. products. Yu and Macgregor (2004) combined MIA and to efficiently extract information from the rapidly timeproducts. Yu and Macgregor (2004) combined MIA and PLS PLS Song, and 2012). in (PCA), the methods, principal analysis partial least square (PLS) their However, extensions (Qin and data-driven Zheng, and Although principal component component analysis (PCA), partial2013, least Ge square the MIA method of interest in many industrial extract information from the timevarying flame images and is predict the boiler performance. to efficiently efficiently extract information from the rapidly rapidly timemonitoring an industrial process is data-driven realized2013, through the to (PLS) and their extensions (Qin and Zheng, Ge and Song, 2012). However, in the methods, (PLS) and their extensions (Qin and Zheng, 2013, Ge and processes, there aremethod some drawbacks. Firstly, the spatial varying images and predict the performance. Although the MIA of interest in many industrial varying flame flame images and is predict the boiler boiler performance. process 2012). measurements of physical properties, as flow Song, However, in methods, monitoring an industrial realized such through the Although Song, 2012). However, process in the the is data-driven data-driven methods, information is lost in the process of unfolding each image to the MIA method is of interest in many industrial processes, the there some isdrawbacks. spatial MIAaremethod of interest Firstly, in manythe industrial rates, temperatures, and which just provide a Although monitoring an process is through the process measurements of pressures, physical as flow monitoring an industrial industrial process properties, is realized realized such through the processes, one-dimensional vectors. Secondly, masks, aneach area selected there are some drawbacks. Firstly, the spatial information is lost in the process of unfolding image to limited representationof of the process because the processes, there are some drawbacks. Firstly, the spatial process measurements physical such as rates, which just provide processtemperatures, measurementsand of pressures, physical properties, properties, such as flow flowa one-dimensional in the score space tointrack the number of pixels, are manually information is lost the process of unfolding each image vectors. Secondly, masks, an area selected to measurements are sensed atpressures, some regions because instead of the the information is lost in the process of unfolding each image to rates, and which limitedtemperatures, representation thelocal process rates, temperatures, and of pressures, which just just provide provide aa in determined most one-dimensional vectors. Secondly, an area selected the score in space toMIA trackapplications. the numbermasks, of pixels, are manually one-dimensional vectors. Secondly, masks, an area selected whole region. Furthermore, if engineers are lack of limited representation of the process because the measurements are sensed atofsome regions because instead of the in the score space to track the number of pixels, are manually limited representation thelocal process in mosttoMIA applications. in the score space track the number of pixels,characterizing are manually process knowledge, the sensing points would be improper. measurements are sensed local the determined Besides thein spectral features, spatial features whole region. Furthermore, if engineers are instead lack ofof measurements are sensed at at some some local regions regions instead of the determined most MIA applications. determined in most MIA applications. The effect of poor monitoring and fault diagnosis is improper. notof easily whole region. Furthermore, if engineers are lack the the spatial variation in pixel intensities also play an important process knowledge, the sensing points would be whole region. Furthermore, if engineers are lack of the Besides the spectral features, spatial features characterizing avoided. process knowledge, the sensing points would be improper. role in process monitoring. The wavelet texture analysis The effect of poor monitoring and fault diagnosis is not easily Besides the spectral features, spatial features characterizing the spatial pixel intensities also playcharacterizing an important process knowledge, the sensing points would be improper. Besides thevariation spectralinfeatures, spatial features The effect of poor monitoring and fault diagnosis is not easily (WTA) and the grey level co-occurrence matrix (GLCM) are avoided. the spatial variation in pixel intensities also play an important rolespatial in process monitoring. The wavelet texture analysis The effect poor monitoring andsensing fault diagnosis is not image easily the variation in pixel intensities also play an important With the of advances in optical and digital avoided. regarded as state-of-the-art methods for spatial feature role in process monitoring. The wavelet texture analysis (WTA) the grey level co-occurrence matrix (GLCM) are avoided. in and process monitoring. The wavelet texture analysis processing techniques,in on-line based on process With the advances opticalmonitoring sensing and digital image role extraction inthe the machine vision area. They have also been (WTA) and grey level co-occurrence matrix (GLCM) are regarded as state-of-the-art methods for spatial feature (WTA) and the grey level co-occurrence matrix (GLCM) are images offers new perspectives to process monitoring in the With optical sensing digital image processing techniques,in based on process With the the advances advances in on-line opticalmonitoring sensing and and digital image regarded used to deal with process images. WangThey and Ren (2014) and as state-of-the-art methods for spatial feature extraction in the machine vision area. have also been regarded as state-of-the-art methods for spatial feature area of combustion processes, forest products, rolled steel processing techniques, on-line monitoring based on process images offers new perspectives to process monitoring in the Bai processing techniques, on-line monitoring based on process ettoal.deal (2017) analyzed the flame images combustion extraction in the vision They have also used process images. Wang andfrom Ren and extraction inwith the machine machine vision area. area. They have(2014) also been been sheets, and sonew on. Conventionally, process measurements images process monitoring the area of offers combustion processes, to forest products, rolledin images offers new perspectives perspectives to process monitoring insteel the with GLCM to characterize the second order statistics of used to deal with process images. Wang and Ren (2014) Bai et al. (2017) analyzed the flame images from combustion and collected at sotheon.local regions were used measurements for process used to deal with process images. Wang and Ren (2014) and area of combustion processes, products, rolled sheets, Conventionally, area of and combustion processes, forest forest process products, rolled steel steel pixel intensity. WTA is widely used in the fields of paper al. the images from combustion with to analyzed characterize the second of Bai et etGLCM al. (2017) (2017) analyzed the flame flame imagesorder from statistics combustion monitoring. In the this paper, process images are to Bai sheets, Conventionally, process measurements collected regions were usedused forinstead process sheets, and andat so so on. on.local Conventionally, process measurements formation (Reis and Bauer, 2009), manufactured countertop with GLCM to characterize the second order statistics of pixel intensity. WTA is widely used in the fields of paper with GLCM to characterize the second order statistics of help monitor larger spacesprocess and they can used recordforsignificant collected local were process monitoring. this paper, to pixel collected at atIn the the local regions regions images were are usedused forinstead process slabs (Liu and Macgregor, 2006) and so on. Obviously, these intensity. WTA is widely used in the fields of paper formation (Reis and Bauer, 2009), manufactured countertop pixel intensity. WTA is widely used in the fields of paper variations of the operating unit which may be neglected by monitoring. this images used instead help monitorIn spacesprocess and they can are record monitoring. Inlarger this paper, paper, process images are used significant instead to to formation methods features from images inflexible way, (Reis and 2009), countertop slabs (Liuextract and andmanufactured soinon.anObviously, these (ReisMacgregor, and Bauer, Bauer,2006) 2009), manufactured countertop sensors at oflocal regions. Although the information from help monitor larger spaces and they can record significant variations the operating unit which may be neglected by formation help monitor larger spaces and they can record significant because they may causefrom some important features tothese be slabs (Liu and Macgregor, 2006) and so on. Obviously, methods extract features images in an inflexible way, slabs (Liu and Macgregor, 2006) and so on. Obviously, these variations the unit may be by sensors regions. Although from variationsat of oflocal the operating operating unit which whichthe mayinformation be neglected neglected by because methods extract features from images in an inflexible way, they may cause some important features to be methods extract features from images in an inflexible way, sensors sensors at at local local regions. regions. Although Although the the information information from from because they may cause some important features to be because they may cause some important features to be
Copyright © 2018, 2018 IFAC 115 Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Peer review©under of International Federation of Automatic Copyright 2018 responsibility IFAC 115Control. 10.1016/j.ifacol.2018.09.285 Copyright © 2018 IFAC 115 Copyright © 2018 IFAC 115
2018 IFAC ADCHEM 116 Shenyang, Liaoning, China, July 25-27, 2018Yuting Lyu et al. / IFAC PapersOnLine 51-18 (2018) 115–120
omitted. A few approaches to combine spectral and spatial features such as multi-resolution MIA (Liu and Macgregor, 2007) were also proposed. Although they gave noticeable performance enhancement in applications, some of the aforementioned drawbacks remain unsolved. Recently, the deep learning (Lecun et al., 2015, Yoshua et al., 2004) has been attracting great attention as they can potentially extract different level features for better classification and recognition. One representation of deep learning is deep belief network (DBN) (Hinton et al., 2006). It has been widely used in a variety of fields, such as handwritten digits recognition, speech recognition, and many other applications (Zhao et al., 2017, Hao and Tian, 2017). In particular, DBN has also been applied to data-driven industrial processes for classification (Gao et al., 2014) and soft sensors (Shang et al., 2014). However, to our best knowledge, DBN has never been applied to online monitoring, particularly for the process images collected from industrial combustion processes.
Fig. 1. The diagram of RBM The probability of any possible visible and hidden vector is given based on the energy as: e E ( v ,h ) (2) P ( v, h) Z where Z v ,h e E ( v ,h ) is the normalizing factor. Since there are no direct connections within each layer, one is able to derive the conditional probabilities P ( v h) and P ( h v ) as follows:
In this paper, a novel DBN based framework is proposed to extract features from process images and to conduct online monitoring and detection of the current status of the combustion process. In this framework, the spectral and spatial features are extracted simultaneously. Meanwhile, a new statistic is first-ever proposed for DBN. Through the process image-based DBN model, feature extraction and process monitoring can be realized in one model. This opens a new area in image-based process monitoring.
j 1
(4)
i 1
P( v )
P ( v, h ) h
1 Z
e h
E ( v,h )
(5)
The training results can be summarized as follows.
wij ( vi h j data vi h j recon ) bi ( vi data vi recon ) c j ( h j data h j recon )
(6) (7) (8)
where is a learning rate, data denotes expectation over the training data, recon denotes expectation over the reconstructed data.
As an energy-based model, RBM has an energy given by i 1
nh
Then the stochastic gradient descent and contrastive divergence (Hinton, 2002) are used for training an RBM with the objective of maximizing the log-likelihood function of (5).
RBM is a type of undirected graphical model composed of a visible layer representing the input data and a hidden layer representing the latent variables. The visible and hidden units are connected using symmetrical weights with no visiblevisible or hidden-hidden connections as shown in Fig.1. nv
P ( vi 1 h) (bi wij h j )
2.1 RBM Model
nh
(3)
i 1
For RBM models, {W, b, c} are the parameters to be trained. Before training these parameters, the probability of a given input vector is assigned by RBM as follows:
2. BACKGROUND
nv
P ( h j 1 v ) ( c j wij vi )
where denotes the sigmoid function, ( x ) (1 e x ) 1 .
The remainder of this paper is organized as follows. The next section gives the background of the restricted Boltzmann machine (RBM) and DBN. In Section 3, image preprocessing techniques are presented. Then a new statistic for DBN is defined and kernel density estimation is used to determine its control limits. Next, the whole monitoring procedure using process images is summarized. In Section 4, the proposed method is evaluated using the flame images collected from a real combustion system. Finally, conclusions and the future research directions are presented.
nv
nh
E ( v, h) bi vi c j h j h j wij vi i 1 j 1
(1)
where vi and h j are the binary states of the visible unit i
and the hidden unit j , bi and c j are their biases respectively, and wi , j is the weight between them.
116
Although RBM with the energy function of (1) is beneficial for binary data, such as in the case of handwritten recognition where the image color is either black or white, it should be mentioned that the information in the industrial image is much more complex and the data are typically represented by real-valued vectors. In order to tackle those data, binary RBMs are extended to the Gaussian–Bernoulli restricted Boltzmann machine (GRBM) (Hinton, 2012). The corresponding normalized energy function is:
2018 IFAC ADCHEM Shenyang, Liaoning, China, July 25-27, 2018Yuting Lyu et al. / IFAC PapersOnLine 51-18 (2018) 115–120
E ( v , h)
nv nh （vi bi ) 2 nh c j h j h j w j ,i vi 2 1 j 1 i 1 j 1
nv
i
1 T v v bT v cT h hT Wv 2 In GRBM, (3) does not change, but (4) becomes
(9)
nh
P( vi h) ~ N (bi w j ,i h j ,1)
(10)
j 1
Note that the differences between GRBM and RBM only lie in the energy function and in the way of generating the reconstructions, but all update rules for weights and biases remain the same. In this paper, all the RBMs in DBN are selected as the GRBM.
117
arrange the three-dimensional images into one-dimensional vectors as shown in Fig.3. Suppose the image in each channel is of the size N1 N 2 , then the one-dimensional vector arranged from three channels is of the length M N1 N 2 3 . In other words, with a sequence of process images, each image corresponds to a one1, , , and the matrix input dimensional vector x k R M , k into DBN is X [x1 x ]T R M . Once the image data are used to train the network, the parameters of DBN can be determined and prepared for on-line monitoring.
xK
hL
h2
h*L
x2
v L ( h L1 )
x1
h1
h*2
h1*
v 2 ( h1 )
v 2 ( h1 )
v1
v1
v1
Fig. 3. Unfolded structures of the images with RGB channels 3.2 Construction of Statistic
Fig. 2. The training procedures of a multi-level DBN structure with smoothing
Like conventional process monitoring schemes, on-line process monitoring is carried out to detect the abnormal condition after off-line training DBN so that the operators can remove the faults early enough to avoid process deviation. Statistics play an important role in this procedure. Generally, for some traditional multivariate statistical methods like PCA, the Hotelling T 2 statistic and the SPE statistic are constructed to monitor the principal component space and its residual space, separately. Whether the test data are abnormal can be estimated by comparing the statistics of the data to the corresponding control limits.
2.2 DBN Model DBN is composed of a series of stacked RBMs. After an RBM has been trained, the activation probabilities of its hidden units can be computed and they are used as the input data for learning a higher-level RBM (Geoffrey and Hinton, 2007). In this way, each hidden layer in DBN performs a nonlinear transformation on its input data to actually describe the underlying features behind the input data. Fig.2 depicts the framework of DBN used in the present study. To avoid larger fluctuations of the process image data, the hidden variables h* are smoothed in the time domain to h before it is sent to the higher levels. This framework is designed for process image-based monitoring since the features extracted from the process images often involve frequent fluctuations caused by irregular blinks in the images.
However, in the procedure of images based process monitoring using DBN, the application of the T 2 statistic is limited because the distribution of the variables in hidden layers is inconclusive. Meanwhile, there are frequent fluctuations in the reconstruction value of each pixel. When they are added up to calculate the SPE statistic, huge fluctuations are accumulated. These huge fluctuations will cause perturbations to the determination of the control limit and consequently pose certain difficulty for fault detection.
3. METHODOLOGY
Thus, a new statistic, Energy , is introduced and it is defined as the energy of the top-level RBM:
3.1 Image Pre-processing
1 T v L v L b LT v L c LT h L h LT WL v L (11) 2 where L denotes the L-th layer on the top of DBN.
In a real industrial process, the colors of the process images obtained by digital color cameras are encoded in an RGB format. Nonetheless, the inputs and outputs of DBN have to be a one-dimensional vector. In order to extract useful features from the process images by DBN, one needs to
E ( v L , h L )
The principle of this statistic can be explained from the point of RBM training. In the training phase, RBM is trained with 117
2018 IFAC ADCHEM 118 Shenyang, Liaoning, China, July 25-27, 2018Yuting Lyu et al. / IFAC PapersOnLine 51-18 (2018) 115–120
the objective of maximizing the log-likelihood function of P ( v ) identified with minimizing the energy of model. In other words, when no faults occur in online monitoring, the energy of testing data is relatively low with small fluctuation. Once a fault occurs, the testing sample will produce a larger deviation with the statistical model and the energy of RBM will exceed its control limit.
and the air/fuel flow rates are automatically recorded through a SCADA system. Meanwhile, the concentrations of NOx, SOx, O2 and CO in the exhaust gas are measured by a gas analyzer. The flame images in the furnace are captured by a digital color camera. In this experiment, the camera with the specifications of 658 492 pixels and a resolution of 24 bits per pixel is used.
To construct a statistical model, the top-level RBM is chosen. The reason is obvious, because each hidden layer in DBN extracts features from the outputs of its previous layer, and the features extracted in the low-level contain more useless information. To avoid the features of the low level that deteriorate the monitoring effect, the top-level RBM is used here. It is more suitable and beneficial to construct statistics in the present study. Finally, to estimate whether fault information exists at a sample point, the confidence limit of Energy statistic should be determined. Because the distribution of this statistic is unknown, kernel density estimation is used here to calculate the control limit of the Energy statistic.
In the present study, the training dataset contains 1,500 flame images drawn from the normal condition and the testing dataset contains the normal and the abnormal conditions. Because of the space limit, only one fault condition resulted from the decrease of the ratio of air to fuel is applied here. A total of 2,100 flame images are used to detect the abnormal condition. For easy visualization and comparisons, five sequential images in the normal condition and the abnormal condition are respectively shown as examples in Fig.4(a) and Fig.4(b). Since the abnormal operation behavior is minor, the differences between the normal and the abnormal conditions cannot be distinguished by directly watching the images. Experimental furnace
Digital camera
3.3 Procedure of Process Monitoring based on DBN In summary, process image based DBN for process monitoring goes through the following steps:
Flame image
Burner Cooling system
Air compressor
Step 1. Acquire process images from the digital color camera under the normal operation; then pre-process the images to acquire a one-dimensional vector for each image.
Fuel tank
Step 2. Input the normal image data into the network; then train each individual RBM successively to constitute the DBN model.
Fig. 4. Scheme of the experimental combustion system (a)
Step 3. Calculate the Energy statistic of the normal image data and determine the control limit of Energy by kernel density estimation. Step 4. Acquire testing images and input them into DBN to extract useful features and calculate the monitoring statistic Energy after pre-processing.
(b)
Step 5. Determine whether a fault occurs by comparing the statistics of each testing image with the control limit of Energy .
Fig. 5. Five sequential images of (a) normal and (b) abnormal conditions Such abnormal conditions are detected by the proposed DBN. To demonstrate the effectiveness of the proposed DBN, three traditional approaches, namely, PCA, MIA and GLCM are applied to furnace monitoring for comparisons. The process variables (such as the concentration of CO, the concentration of NO, etc.) are used in PCA and process image data are applied to MIA and GLCM. The extracted features of MIA and GLCM are smoothed in the same way as the smoother of DBN.
4. CASE STUDY In order to evaluate the proposed framework of the processimage based DBN, flame images collected from a real combustion furnace are used. The experimental combustion system is shown in Fig.4. It consists of an experimental furnace, a burner, an air compressor, a fuel tank, a cooling system and a digital camera. The model of the burner is NA 5514-6 manufactured by North American Manufacturing Company. The volume of the experimental furnace is 2.5m 2.5m 1.5m . During the operation, the process data such as the steam pressure, the temperature in the furnace,
Because MIA and GLCM can merely extract spectral features and spatial features from flame images, further analysis of these features is still performed using PCA. Thus, the 118
2018 IFAC ADCHEM Shenyang, Liaoning, China, July 25-27, 2018Yuting Lyu et al. / IFAC PapersOnLine 51-18 (2018) 115–120
(b)
statistics T 2 and SPE in PCA, MIA-PCA and GLCM-PCA are constructed separately, but the Energy statistic of the proposed DBN model is used. The detailed monitoring results for the testing dataset are shown in Fig. 6. The blue curves represent the values of the statistics for each testing sample point and the red straight line represents the control limits. The significance level is 0.99. In Fig. 6(a), only process data are used, so the fault is almost not detected by the T 2 statistic because of the lack of adequate representation of the current status. The fault studied in this case is a step change occurs in the air-fuel ratio of the experimental system at the 1,300th point in the testing dataset. In Fig. 6(b) and Fig. 6(c), although the features extracted from the flame images are smoothed with the same time window as DBN, the statistics after the 1,300th point fluctuate around the control limit. In Fig. 6(d), obviously the Energy statistic of DBN performs better than the results of the other three traditional methods.
40
T2
30 20 10 0 0
300
600
900 1200 samples
1500
1800
2100
300
600
900 1200 samples
1500
1800
2100
SPE
15 10 5 0 0
(c)
In order to demonstrate the effectiveness of the proposed method explicitly, the false alarm rates for normal data and the fault detection rates for faulty data using different methods are summarized in Table 1. The best monitoring results obtained in this testing phase have been highlighted. From this table, one can see that the performance of the proposed method is much better than the other four methods. Significant improvements have been made in the fault detection rate on the premise that images are smoothed with the same window width in each method. Although the false alarm rate of the proposed method is not the smallest, it is very close to the smallest one obtained by the process data based PCA.
40
T2
30 20 10 0
0
300
600
900 1200 samples
1500
1800
2100
300
600
900 1200 samples
1500
1800
2100
1500
1800
2100
15
SPE
10 5 0
0
(a)
(d)
25
140.3
20 15 Energy
T2
119
10 5 0 0
300
600
900 1200 samples
1500
1800
140.2
140.1 0
2100
25
300
600
900 1200 samples
SPE
20
Fig. 6. Process monitoring results for the combustion furnace: (a) PCA, (b) MIA-PCA, (c) GLCM-PCA, and (d)DBN. PCA is based on process variable data, and MIA-PCA, GLCMPCA, and DBN are based on process image data.
15 10 5 0 0
300
600
900 1200 samples
1500
1800
2100
5. CONCLUSIONS This paper develops a novel monitoring method with process image based DBN to detect a combustion system. Instead of using process variables, which cover limited information of a process, this paper uses the process images captured by the digital camera to establish the DBN model and achieve process monitoring with the newly constructed statistic. The proposed method has two main advantages over the 119
2018 IFAC ADCHEM 120 Shenyang, Liaoning, China, July 25-27, 2018Yuting Lyu et al. / IFAC PapersOnLine 51-18 (2018) 115–120
experiment results show that the proposed method is superior to the other traditional image-based monitoring methods. In the future, the process data and process images will be combined to make up a complete dataset, which is beneficial for process monitoring.
traditional process image-based monitoring methods. First, it is more flexible in feature extraction and able to dig out more informative features via deep network structure. Second, the proposed method integrates feature extraction and abnormal condition detection into one model rather than separately accomplish them, making it more convenient and efficient. The effectiveness of the proposed method is demonstrated through the flame images obtained from a furnace. The
Table 1. Fault detection rates and false alarm rates of combustion furnace based on different methods Methods
Process variable data PCA
Process image data MIA-PCA
GLCM-PCA
DBN
Statistics
T2
SPE
T2
SPE
T2
SPE
Energy
Fault Detection Rate
0.0262
0.5938
0.4700
0.3187
0.5250
0.2338
0.9563
False Alarm Rate
0.0038
0.0108
0.0200
0.0138
0.0100
0.0246
0.0069
Hinton, G. E. (2012). A practical guide to training restricted boltzmann machines. Momentum, 9(1), 599-619. Lecun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444. Liu, J. J., & Macgregor, J. F. (2006). Estimation and monitoring of product aesthetics: application to manufacturing of “engineered stone” countertops. Machine Vision & Applications, 16(6), 374-383. Liu, J. J., & Macgregor, J. F. (2007). On the extraction of spectral and spatial information from images. Chemometrics & Intelligent Laboratory Systems, 85(1), 119-130. QIN, S. J. & ZHENG, Y. (2013). Quality-relevant and process-relevant fault monitoring with concurrent projection to latent structures. AIChE Journal, 59, 496504. Reis, M. S., & Bauer, A. (2009). Wavelet texture analysis of on-line acquired images for paper formation assessment and monitoring. Chemometrics & Intelligent Laboratory Systems, 95(2), 129-137. Shang, C., Yang, F., Huang, D., & Lyu, W. (2014). Datadriven soft sensor development based on deep learning technique. Journal of Process Control, 24(3), 223-233. Wang, J. S., & Ren, X. D. (2014). GLCM based extraction of flame image texture features and KPCA-GLVQ recognition method for rotary kiln combustion working conditions. International Journal of Automation and Computing, 11(01), 72-77. Yoshua Bengio, Aaron Courville, & Pascal Vincent. (2013). Representation learning: a review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8), 1798-828. Yu, H., & Macgregor, J. F. (2004). Monitoring flames in an industrial boiler using multivariate image analysis. Aiche Journal, 50(7), 1474–1483. Zhao, Z., Jiao, L., Zhao, J., Gu, J., & Zhao, J. (2017). Discriminant deep belief network for high-resolution sar image classification. Pattern Recognition, 61, 686-701.
ACKNOWLEDGEMENT This work is supported by National Key Research and Development Program of China (2017YFB0304203), National Nature Science Foundation of China (61573308), and Ministry of Science and Technology, Taiwan, R.O.C. under Grant MOST 106-2221-E-033-060-MY3. REFERENCES Bharati, M. H., Macgregor, J. F., & Tropper, W. (2003). Softwood lumber grading through on-line multivariate image analysis techniques. Industrial & Engineering Chemistry Research, 42(21). Bai, X., Lu, G., Hossain, M. M., Szuhánszki, J., Daood, S. S., & Nimmo, W., et al. (2017). Multi-mode combustion process monitoring on a pulverised fuel combustion test facility based on flame imaging and random weight network techniques. Fuel. Gao, X., Shang, C., Jiang, Y., Huang, D., & Chen, T. (2014). Refinery scheduling with varying crude: a deep belief network classification and multimodal approach. Aiche Journal, 60(7), 2525-2532. GE, Z. & SONG, Z. (2012). Multivariate Statistical Process Control: Process Monitoring Methods and Applications, Springer. Geoffrey E. Hinton. (2007). Learning multiple layers of representation. Trends Cogn Sci., 11(10), 428-434. Hao, X. L., & Tian, M. (2017). Deep belief network based on double weber local descriptor in micro-expression recognition. Hinton, G. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8), 1771-1800. Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18(7), 1527.
120