- Email: [email protected]
Defect inspection for TFT-LCD images based on the low-rank matrix reconstruction
Neurocomputing 149 (2015) 1206–1215
Contents lists available at ScienceDirect
Neurocomputing journal homepage: www.elsevier.com/locate/neucom
Defect inspection for TFT-LCD images based on the low-rank matrix reconstruction Yi-Gang Cen a,b, Rui-Zhen Zhao a,b,n, Li-Hui Cen c, Li-Hong Cui d, Zhen-Jiang Miao a,b, Zhe Wei e a
School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China Beijing Key Laboratory of Advanced Information Science and Network Technology, Beijing 100044, China c School of Information Science and Engineering, Central South University, Changsha, Hunan 410083, China d Department of Mathematics, Beijing University of Chemical Technology, Beijing 100029, China e Electrical and Computer Engineering, McMaster University, Ontario, Canada b
art ic l e i nf o
a b s t r a c t
Article history: Received 13 May 2014 Received in revised form 12 July 2014 Accepted 2 September 2014 Communicated by Jinhui Tang Available online 16 September 2014
Surface defect inspection of TFT-LCD panels is a critical task in LCD manufacturing. In this paper, an automatic defect inspection method based on the low-rank matrix reconstruction is proposed. The textured background of the LCD image is a low-rank matrix and the foreground image with defects can be treated as a sparse matrix. By utilizing the Inexact Augmented Lagrange Multipliers (IALM) algorithm, the segmentation of a LCD image can be converted into the reconstruction of a low-rank matrix with a fraction of its entries arbitrarily corrupted. This low-rank matrix reconstruction problem can be exactly solved via convex optimization that minimizes a combination of the nuclear norm and the l1-norm. Also, adaptive parameter selection strategy is proposed by conducting deep analysis on the IALM algorithm, which improves the generality of the IALM algorithm for different defect types. Experiment results show that our inspection algorithm is robust for the defect shapes and types under different illumination conditions. The shapes and edges of defect areas in the LCD images can be well preserved and segmented from textured background by our detection algorithm. & 2014 Elsevier B.V. All rights reserved.
Keywords: Defect inspection Low-rank matrix reconstruction LCDs IALM Adaptive parameter selection
1. Introduction Thin film transistor-liquid crystal display (TFT-LCD) has become a major technology for flat panel display in recent years due to their full-color display capabilities, low power consumption and light weight. It can be used in a wide range of electronic devices such as cellular phones, PDAs, computer monitors and television sets. To ensure display quality and to improve the yield of LCD flat panels, the inspection of defects in TFT-LCD panels becomes a critical task in LCD manufacturing. As the TFT-LCD panel becomes dense, small defects can only be observed at an extremely high resolution. For fast imaging of a large-sized TFT-LCD panel at a high resolution, a one-dimensional (1D) line scan system is demanded. A TFT-LCD panel image at a fine resolution presents very complicated repetitive patterns, which increases the difficulty of the defect detection task. A TFT panel generally contains horizontal gate lines on one plane and vertical data lines on the other plane. At each pixel, the
n Corresponding author at: School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China. E-mail address: [email protected] (R.-Z. Zhao).
http://dx.doi.org/10.1016/j.neucom.2014.09.007 0925-2312/& 2014 Elsevier B.V. All rights reserved.
gate of the TFT is connected to the gate line and the source is connected to the data line. Since the TFT panel is composed of horizontal gate lines and vertical data lines, it forms a structural texture that contains horizontal and vertical line patterns. A typical structurally textured image of a defect-free TFT panel is shown in Fig. 1. It can be seen that the image involves repetitive, equally spaced horizontal and vertical lines. Surface defect detection using machine vision has been largely tackled by texture analysis techniques [1]. A set of textural features are generally extracted in the spatial domain or in the spectral domain, and then various analysis methods such as distance measures [2], Bayes probability [3], neural networks [4] and support vector machine [5] are used for discriminating defective areas from the background based on the extracted features. For the surface defects inspection, Tsai et al. conducted in-depth research and proposed some valuable approaches. For example, Ref. [6] proposed a fast regularity measure for defect detection in nontextured and homogeneously textured surfaces, with specific emphasis on ill defined subtle defects. In [7], the fractal signal of a 1D grey-level TFT-LCD image was divided into small segments. Then by calculating each divided segment's normalized cross correlation with its neighbouring segments and comparing the resulting correlation value with a predetermined threshold, the
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
1207
from D in the same time. This leads to the segmentation of the foreground (defects) and the background (structural texture image) of a TFT-LCD defect image. To be formal, this problem can be described as min rankðAÞ þ γ J E J 0 ; A;E
Fig. 1. A surface image of a defect-free TFT panel.
segments containing a defect can be effectively identified. In [8], according to the singular value decomposition (SVD) of the LCD panel image, proper singular values that represent the background texture of the surface were selected. Then the matrix without the selected singular values was reconstructed, which can eliminate periodical, repetitive patterns of the textured image, and preserve the anomalies in the restored image. Some other related works of Tsai can be found in [9–11]. The above-mentioned defect inspection methods resulted in a significant development for the online or the offline product defect inspection. The existing vision-based techniques generally need a pre-stored reference image for comparison. This requires a large volume of data for reference and precise environmental controls such as alignment and lighting for test images. Moreover, the techniques for LCD inspection mainly focus on final appearance checks for defects such as nap or dark/bright spots after the fabrication is completed. Finally, most techniques need to adjust some parameters. The selection of parameters is complex and mostly relies on the human experiences. This will limit the generality of the techniques. In the recent years, low-rank matrix reconstruction became a hot issue in the areas of face recognition, image recovery, de-noising and data reconstruction. Low-rank matrix reconstruction mainly concerns how to recover the low-rank matrix from a large but sparse error data. In different application cases, lowrank matrix reconstruction also be called as sparse and low-rank matrix decomposition, robust principle component analysis, ranksparsity incoherence, etc. In this paper, we mainly focus on how to segment the defect area from the textured background of the LCD panel images accurately. Because the LCD images generally involve repetitive, equally spaced horizontal and vertical lines, the defect detection problem can be boiled down to the low-rank matrix reconstruction problem. The textured background image corresponds to a low-rank matrix and the foreground defect image corresponds to a sparse matrix. Then according to the joint minimization of nuclear norm and l1-norm, the low-rank matrix and the sparse defect matrix can be segmented by the low-rank matrix reconstruction theory.
2. Problem formulation Suppose that the original M N defect image is D. The low-rank background matrix is AMN and the sparse foreground matrix is EMN . For the TFT panel image, the defects are sufficiently sparse, one can exactly recover the low-rank matrix A from D ¼ A þ E. Also, the sparse matrix E corresponding to the defects can be obtained
s:t: D ¼ A þ E
ð1Þ
Here A is a low rank matrix representing the background of a TFTLCD image. E denotes the sparse error matrix between A and D. For a TFT-LCD image, E denotes the defects, noise and outliers in the image. J J 0 denotes the l0-norm (number of non-zero entries in the matrix), and γ 4 0 is a parameter that trades off the rank of the solution A versus the sparsity of the error E. Unfortunately, (1) is not tractable since both rank and l0-norm are non-convex and discontinuous functions. The optimization problem of (1) is extremely difficult (NP-hard in general) to solve. Recently, Wright et al. [12] have proposed that the problem in (1) can be solved by replacing the cost function with its convex surrogate, provided that the rank of the matrix A is not too high and the number of non-zero entries in the matrix E is not too large. This convex relaxation, dubbed Principal Component Pursuit (PCP) in [12], replaces rankðÞ with the nuclear norm and the l0-norm with the matrix l1-norm. Under quite general conditions, it has been proved in [12] that the following optimization problem has the same optimal solution as (1): min J A J n þ λ J E J 1 ; A;E
s:t: D ¼ A þ E
ð2Þ
where J A J n ¼ ∑i σ i ðAÞ is the nuclear norm of A. σ i ðAÞ denotes the ith singular values of the matrix A. J J 1 represents the l1-norm, i.e., the sum of the absolute values of all entries of the matrix. λ 4 0 is a weighting parameter. Theoretical considerations in [12] pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi suggest that λ must be of the form C= maxfM; Ng, where C is a constant, typically set to unity. It is interesting to note that the equivalence between (1) and (2) is not affected by the magnitude of the singular values of the solution A or by the magnitude of the non-zero entries of the error matrix E. Due to the ability to exactly recover underlying low-rank structure in the data, even in the presence of large errors or outliers, the optimization problem (2) is referred to as Robust Principal Component Analysis (RPCA) in [12] (a popular term that has been used by a long line of work aims to render PCA robust to outliers and gross corruption). Several applications of RPCA, e.g. background modelling and removing shadows and specularities from face images, have been demonstrated in [12] to show the advantage of RPCA.
3. Solution overview and algorithm for matrix reconstruction The optimization problem (2) can be treated as a general convex optimization problem and solved by any off-the-shelf interior point solver [13]. However, although interior point methods normally take very few iterations to converge, they have difficulty in handling large matrices because the complexity of computing the step direction is Oðm6 Þ, where m is the dimension of the matrix. Recently, iterative thresholding (IT) algorithm showed its efficiency for l1-norm minimization problem in compressed sensing [14,15] and the nuclear norm minimization in matrix completion (MC) problem [16]. Also, in [17], we proposed a rank adaptive atomic decomposition algorithm for the low-rank matrix completion problem. From (2) we can see that matrix recovery (RPCA) problem involves minimizing a combination of both the l1-norm and the nuclear norm. In [12], the authors have also adopted the IT technology to solve (2) and obtained similar convergence and scalability properties. However, the iterative thresholding scheme proposed in [12] converges extremely slowly.
1208
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
In [18], Lin et al. proposed two new algorithms for solving the problem (2): the first one is an accelerated proximal gradient (APG) algorithm applied to the primal; the second one is a gradient-ascent algorithm applied to the dual of the problem (2). Experiments showed that both algorithms are at least 50 times faster than the IT method. In addition, by utilizing the augmented Lagrange multipliers (ALM), Lin et al. [19] proposed the exact ALM (EALM) algorithm and moreover, a slight improvement over the EALM, the inexact ALM (IALM) algorithm for (2). It showed that the IALM is at least five times faster than APG, and its precision is also higher. In particular, the number of non-zeros in E computed by IALM is much more accurate than that by APG, which often leave many small non-zero terms in E. For the TFT panel defect inspection, the inspection algorithm should be fast so that it can be applied to the online inspection in the manufacturing process. Thus, we choose the IALM method to solve the optimization problem (2). By identifying X ¼ ðA; EÞ;
f ðXÞ ¼ J A J n þ λ J E J 1 ;
s:t: hðXÞ ¼ D A E;
ð3Þ
we can apply the augmented Lagrange multiplier method for problem (2) [19]. The Lagrangian function is
μ
LðA; E; Y; μÞ ¼ J A J n þ λ J E J 1 þ 〈Y; D A E〉 þ J D A E J 2F : 2
ð4Þ
The detailed algorithm is described in Algorithm 1. Here, 1 JðDÞ ¼ maxð J D J 2 ; λ J D J 1 Þ. svdðÞ denotes the singular decomposition of a matrix. Sε ðxÞ is a soft-thresholding (shrink) operator
defined as 8 > < xε : Sε ðxÞ ¼ x þ ε > : 0
if x 4 ε
if x o ε
ð5Þ
otherwise
Algorithm 1. Inexact ALM method for RPCA Input: Observation matrix D A RMN , λ, ε Output: ðAk ; Ek Þ 1. Y 0 ¼ D=JðDÞ; E0 ¼ 0; μ0 4 0; ρ 4 1; k ¼0. 2. do 3. ðU; Σ ; VÞ ¼ svdðD Ek þ μk 1 Y k Þ; 4. Ak þ 1 ¼ USμ 1 ðΣ ÞV T ; k
5. Ek þ 1 ¼ Sλμ 1 ðD Ak þ 1 þ μk 1 Y k Þ; k
6. Y k þ 1 ¼ Y k þ μk ðD Ak þ 1 Ek þ 1 Þ; μk þ 1 ¼ ρμk ; 7. k ¼ k þ 1. 8. until
J D Ak þ 1 E k þ 1 J F o JDJF
ε
It has been shown in [19] that if μk does not increase too rapidly (ρ should not be too large), then ðAk ; Ek Þ converges to an optimal solution ðAn ; En Þ to the RPCA problem. Extensive numerical experiments have shown that for geometrically growing μk, IALM method converges Q-linearly. On the other hand, ρ is an important
Fig. 2. Four defect images. (a)–(c) are under fine image resolution (60 pixels/mm); (d) is under coarse image resolution (20 pixels/mm). (a) Pinhole, (b) Scratch, (c) Particle, (d) Fingerprint.
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
1209
Fig. 3. Reconstructed sparse defective foreground images of (a) fingerprint and (b) pinhole images under different values of ρ.
parameter in this algorithm. It can be seen in the next section that the different selections of ρ will lead to different segmentation results and running times. This problem will be analysed in detail in the next section.
4. Experiment results In this section, we firstly analyse the relations of the values of ρ and the segmentation results of different defect types. Then an
1210
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
Fig. 4. The plots of SLR and running time under different values of ρ for the 4 defect images shown in Fig. 2. (a) Relations of SLR and different values of ρ. (b) Relations of running time and different values of ρ.
adaptive parameter selection strategy is proposed to improve the generality of the IALM algorithm. 4.1. Sparsity low-rank measure function and analysis of the parameter selection We will present experimental results of the defect images used in [8] to evaluate the performance of the proposed defect detection scheme based on the low-rank matrix reconstruction. These images contain micro defects including pinholes, scratches, particles and fingerprints on TFT panel surfaces . The test image size is 256 256 with 8-bit grey levels. The computer used in the experiment is ThinkPad X61 with Duo 2 G CPU and 2 G memory. Fig. 2 shows these four defect images. We set μ0 ¼ 1:25= J D J 2 for all images. Now we will discuss the effects of the parameter ρ. According to our experiments, the convergence rate of IALM will decrease when ρ is increased. But the residuals of repetitive horizontal and vertical line patterns along with the defects will be retained in the reconstructed sparse defect images with the increase of ρ. Fig. 3 shows the experiment results of fingerprint and pinhole images under different values of ρ. It can be seen in the Fig. 3 that for the pinhole defect, ρ ¼2 may be the best choice for the reconstruction of the sparse defect image because the residuals of the textured background in the first image of Fig. 3(b) are less than others. Also the defect shape and edge are well preserved. For the fingerprint image, when ρ ¼2, the reconstructed sparse defect image is almost blank. With the increase of ρ, the fingerprint shape and edge become clear along with the residuals of the textured background. This is because the fingerprint area can also be treated as a kind of repetitive texture with low rank and hard to be segmented from the background. ρ ¼ 8; 11; 14; 17 may be the better choices for fingerprint defect image. How to evaluate the segmentation results is a difficult issue for image segmentation. To the best of my knowledge, there is no unified criterion for the image segmentation. For a defect image, one may not judge if a small area should belong to the defect or the background even by human visual inspection. In our experiment, the reconstructed defect image matrix E is sparse and the textured background image matrix A is a low-rank matrix. In general, compared with the whole image, defect area is relatively small or tiny (for example, the defects in Fig. 2(a)–(c), but Fig. 2(d) is an exception). Thus most elements in E should be 0 and A should be a low-rank matrix with repetitive textures. Combining these two properties, we define a Sparsity Low-rank (SLR) measure function as follows: SLR ¼ J E J 0 =rankðAÞ
ð6Þ
Fig. 4(a) presents the relations of SLR and ρ for the 4 images showed in Fig. 2. Here, let ρ ¼ f2; 5; 8; …; 78; 81g, the corresponding SLR set is denoted as fSLRk g, k ¼ 1; 2; …; lengthðρÞ. lengthðρÞ denotes the length of the vector ρ. In Fig. 4(a), the value of SLR corresponding to each ρ is normalized, i.e., for a given defect image and different values of ρ, the kth normalized SLR is given by SLRk =maxfSLRgk , k ¼ 1; 2; …; lengthðρÞ. Fig. 4(b) shows the running time for each image. On the one hand, we need to determine the values of ρ according to analysis of the values of SLR. On the other hand, running time of the program is an important issue that needs to be considered for the online defect inspection. By (6), small SLR implies that there are more zeros in the matrix E and E should be sparser. This means that the residuals of the background in E are less. In addition, a small SLR also means that the rank of A is big, i.e., there are more residuals of defect area retained in the low-rank textured background matrix (the residuals of defect area in A will increase the rank of A). By the experiment results, pinhole, scratch and particle are according with this case. But the situation is different for the fingerprint image. By Fig. 2(d) we can see that fingerprint area is textured and should be low-rank to some extent. In the optimization problem (2), fingerprint area will be treated as the textured background and is difficult to separate from the background. In order to segment the fingerprint area into the sparse matrix E, the value of ρ has to be increased such that the fingerprint area is treated as the residuals of background and segmented into E. But ρ cannot be increased too large since the residuals of the repetitive horizontal and vertical textures will seriously affect the detection result. Also, it can be seen by Fig. 2(d) that the fingerprint area occupies a large area of the whole image thus one cannot expect that there are too many zeros in the sparse matrix E. If E is too sparse, it means that the fingerprint area is not well segmented from the background. Thus the selection of ρ should be larger than the cases of pinhole, scratch and particle so that the fingerprint area can be well segmented from the background. In addition, running time of the program is an important issue that needs to be considered in the industrial manufacture process. By Fig. 4(b) we can see that the running time decreases quickly when ρ o 10. Thus, as long as the value of ρ is not too small (ρ o5 by Fig. 4(b)), the running time will be acceptable. For the real industry used computer, the running time will be significantly decreased. 4.2. Adaptive selection strategy of
ρ
According to the above analysis, it can be seen that if the parameter ρ is fixed as a constant, the defect detection approach
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
1211
Fig. 5. Reconstructed low-rank textured background and sparse defective foreground images. The images in the left column are the low-rank background images. Images in the middle column are the defect segmentation results (sparse foreground defect images) and the right column is the binarization results of the sparse foreground defect images. (a) Pinhole, (b) scratch, (c) particle, and (d) fingerprint.
1212
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
Fig. 6. Defect detection results obtained in [8]. The images in the left column are the original defect images. Images in the middle column are the defect segmentation results and the right column is the binarization results.
will not be suitable for all defect types. In order to improve the generality of our detection algorithm, adaptive selection strategy of the parameter ρ is proposed. Motivated by the selection of parameter μk in Algorithm 1, we also start ρ from a small value and increase it gradually during the iteration of Algorithm 1. By the discussions in Section 4.1, we initially start with ρ ¼ 2 in Algorithm 1. The value of ρ at each iteration of Algorithm 1 is
increasing until it reaches a pre-defined small positive number ρ . In other words, ρk þ 1 ¼ minðηρk ; ρ Þ for k ¼ 1; 2; …. In this way, we need not pre-define the value of ρ and this adaptive parameter selection strategy will improve the generality of Algorithm 1. The segmentation results of the defect images in Fig. 2 are shown in Fig. 5. Also, the binarization images are presented such that the defect shapes can be seen clearly. In this experiment, ρ was set as 10 and η ¼1.07. The
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
1213
Fig. 7. Defect detection results of the touch screen images. The image in the left of each sub-figure is the original defect image. Image in the middle of each sub-figure is the defect segmentation result and the right one is the binarization result. (a–c) Crack, (d, e) particle, (f, g) scratch, (h, i) stain, and (j) dust.
average running time of these four images is 2.23 s. Here, the segmentation results obtained in [8] are also presented for comparison, see Fig. 6. It can be seen that the defect shapes of our segmentation results are well preserved than the results obtained in [8]. In order to test the performance and robustness of our algorithm for low resolution images and different defect types, 10 defective touch screen images are selected for testing. Fig. 7 shows the defective touch screen images of crack, particle, scratch, stain and dust. The image resolution is 96 DPI (approximation 4 pixels/mm) and the image size is 100 100. By Fig. 7 we can see that although the resolution of the defect images is very low and the illumination condition of each image is different, the defect areas still can be well segmented by our algorithm. Also, because the image size is small, the running time decreased quickly and satisfied the real-time requirement of online inspection better. The average running time of these 10 images is 1.01 s. Thus, our inspection algorithm is robust for different defect shapes, types, resolutions and illumination conditions. 4.3. Rank of A and sparsity of E For the RPCA problem, the rank of A is difficult to be estimated as a prior knowledge. Fortunately, Theorem 2 in [19] proved that if μk
does not increase too rapidly, so that ∑kþ¼11 μk 2 μk þ 1 o þ1 and limk- þ 1 μk ðEk þ 1 Ek Þ ¼ 0, then ðAk ; Ek Þ converges to an optimal solution ðAnk ; Enk Þ to the RPCA problem. Thus, in the real applications of Algorithm 1, we only need to make sure that A is a low-rank matrix. But the rank of A need not to be considered or pre-determined. Let β r ¼ rankðAÞ=M, βs ¼ J E J 0 =M M (Here we assume that the testing image size is M M). For the 4 images in Fig. 2 and the 10 images in Fig. 7, the values of βr and βs in each iteration of Algorithm 1 are shown in Fig. 8(a) and (b) respectively. In Fig. 8, the convergence condition was set as ε ¼ 10 8 and a total of 40 iterations were plotted. This is because we want to show the relations among βr, βs and the number of iterations clearly. In fact, in the above experiments, the convergence condition was set as ε ¼ 10 7 . Thus, for the images shown in Figs. 2 and 7, an average of only 15 iterations is needed to reach the convergence condition. It also can be seen by Fig. 8 that the rank of A is observed to be monotonically increasing and after about 15 iterations, it becomes stable. The sparsity probability βs also becomes stable when βr is stable. When βr is stable, the ranks of the textured background image matrices of the 4 TFT-LCD images are almost same. This also can be observed for the touch screen images according to Fig. 8(b). Thus, by Theorem 2 in [19] and Fig. 8, in the real applications of
1214
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
Fig. 8. βr and βs in each iteration of Algorithm 1. (a) βr and βs of the 4 TFT-LCD defect images. (b) βr and βs of the 10 touch screen defect images.
Algorithm 1, the rank of A need not to be considered or prior determined.
the parameter ρ. Experiment results showed that our proposed algorithm can well segment the defect area from the textured LCD image and the running time can satisfy the requirement of online inspection.
5. Conclusion Defect inspection is a key issue for the quality control of the TFT-LCD panel manufacture process. The defect inspection problem of TFT-LCD images can be modelled as the low-rank matrix reconstruction since the LCD image containing repetitive patterns is a low-rank matrix. Also, the foreground image matrix containing defect areas is a sparse matrix. Thus, under the restriction of D¼ AþE, the segmentation of the LCD image can be achieved by joint minimization of the nuclear norm of A and the l1-norm of E. In order to improve the generality of our proposed detection algorithm, adaptive parameter selection strategy was proposed for
Acknowledgements The authors would like to thank Prof. Du-Ming Tsai for providing the TFT-LCD test images, who is with the department of Industrial Engineering and Management, Yuan-Ze University, Taiwan, R.O.C. http://machinevision.iem.yzu.edu.tw/vision/. Also, the authors would like to thank the anonymous reviewers for many valuable comments which contributed to the improvement of this paper. This work was supported by the National Natural Science Foundation of China (Grant nos. 61272028, 61473317, 61273274,
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
61104078); Fundamental Research Funds for the Central Universities of China (Grant no. 2013JBZ003); Specialized Research Fund for the Doctoral Program of Higher Education of China (Grant no. 20120009110008); Program for New Century Excellent Talents in University (Grant no. NCET-12-0768); National Key Technology R&D Program of China (Grant no. 2012BAH01F03); National Basic Research (973) Program of China (Grant no. 2011CB302203); The National High Technology Research and Development Program (863) of China (Grant no. 2014AA015202). Program for Changjiang Scholars and Innovative Research Team in University (Grant no. IRT201206).
References [1] X. Xie, A review of recent advances in surface defect detection using texture analysis techniques, Electron. Lett. Comput. Vis. Image Anal. 7 (3) (2008) 1–22. [2] A. Serdaroglu, A. Ertuzun, A. Ercil, Defect detection in textile fabric images using wavelet transforms and independent component analysis, Pattern Recognit. Image Anal. 16 (1) (2006) 61–64. [3] D. Brzakovic, H. Beck, N. Sufi, An approach to defect detection in materials characterized by complex textures, Pattern Recognit. 23 (1) (1990) 99–107. [4] M.M. Van Hulle, T. Tollenaere, A modular artificial neural network for texture processing, Neural Netw. 6 (1) (1993) 7–32. [5] G.A. Ruz, P.A. Estévez, P.A. Ramirez, Automated visual inspection system for wood defect classification using computational intelligence techniques, Int. J. Syst. Sci. 40 (2) (2009) 163–172. [6] D.M. Tsai, M.C. Chen, W.C. Li, et al., A fast regularity measure for surface defect detection, Mach. Vis. Appl. 23 (5) (2012) 869–886. [7] D.M. Tsai, S.T. Chuang, 1D-based defect detection in patterned TFT-LCD panels using characteristic fractal dimension and correlations, Mach. Vis. Appl. 20 (6) (2009) 423–434. [8] C.J. Lu, D.M. Tsai, Automatic defect inspection for LCDs using singular value decomposition, Int. J. Adv. Manuf. Technol. 25 (1–2) (2005) 53–61. [9] C.J. Lu, D.M. Tsai, Independent component analysis-based defect detection in patterned liquid crystal display surfaces, Image Vis. Comput. 26 (7) (2008) 955–970. [10] D.M. Tsai, H.Y. Tsai, Low-contrast surface inspection of mura defects in liquid crystal displays using optical flow-based motion analysis, Mach. Vis. Appl. 22 (4) (2011) 629–649. [11] D.M. Tsai, I.Y. Chiang, Y.H. Tsai, A shift-tolerant dissimilarity measure for surface defect detection, IEEE Trans. Ind. Informatics 8 (1) (2012) 128–137. [12] J. Wright, Y. Peng, Y. Ma, A. Ganesh, S. Rao, Robust principal component analysis: exact recovery of corrupted low-rank matrices by convex optimization, in: Proceedings of Neural Information Processing Systems, 2009, p. 3. [13] M. Grant, S. Boyd, CVX: Matlab Software for Disciplined Convex Programming (Web Page and Software). 〈http://stanford.edu/”boyd/cvx〉, 2009. [14] E.T. Hale, W. Yin, Y. Zhang, Fixed-point continuation for l1-minimization: methodology and convergence, SIAM J. Optim. 19 (3) (2008) 1107–1130. [15] W. Yin, S. Osher, D. Goldfarb, J. Darbon, Bregman iterative algorithms for l1-minimization with applications to compressed sensing, SIAM J. Imaging Sci. 1 (1) (2008) 143–168. [16] J.F. Cai, E.J. Cands, Z. Shen, A singular value thresholding algorithm for matrix completion, SIAM J. Optim. 20 (4) (2010) 1956–1982. [17] H.Y. Wang, R.Z. Zhao, Y.G. Cen, Rank Adaptive Atomic Decomposition for LowRank Matrix Completion and Its Application on Image Recovery, Neurocomputing 145 (5) (2014) 374–380. [18] Z. Lin, A. Ganesh, J. Wright, L. Wu, M. Chen, Y. Ma, Fast convex optimization algorithms for exact recovery of a corrupted low-rank matrix, in: Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), 2009, p. 61. [19] Z. Lin, M. Chen, L. Wu, Y. Ma, The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices, Technical Report UILU-ENG09-2215, University of Illinois at Urbana-Champaign, 2009.
Yi-Gang Cen received the Ph.D. degree in Control Science & Engineering, in 2006, from the HUST. In September 2006, he joined the Signal Processing Centre, EEE, NTU in Singapore as a research fellow. He is currently an associate professor and a supervisor of doctor student of BJTU. From January 2014 to January 2015, he was a visiting scholar of the department of Computer Science, University of Missouri in US. His research interests include Compressed Sensing, Sparse Representation, Low-rank Matrix Reconstruction, and Wavelet construction theory.
1215 Rui-Zhen Zhao was born in 1975. He got his Ph.D. degree from Xidian University. After that he was with a postdoctoral fellow in Institute of Automation, Chinese Academy of Sciences. He is currently a professor and a supervisor of doctor student. His research interests include wavelet transform and its applications, algorithms off image and signal processing, compressive sensing and sparse representation.
Li-Hui Cen received the Ph.D. degree in Control Theory and Control Engineering from the Shanghai Jiaotong University, Shanghai, China, in 2009. Now she is an associate professor with the School of Information Science and Engineering, Central South University, Hunan, Changsha, China. Her research interest includes control of open channels, signal processing, and predict control.
Li-Hong Cui received the B.S. degree from Henan Normal University, Xinxiang, China, in 1986, the M.S. degree Zhengzhou University, Zhengzhou, China, in 1999, both in mathematics, and the Ph.D. degree in mathematics and computing science from Xi'an Jiaotong University, Xi'an, China, in 2004. She is currently a professor with the Department of Mathematics and Computer Science, Beijing University of Chemical Technology, Beijing, China. She is currently working on implementing some of the new image processing algorithms and multiwavelet. Her present research interests include approximation theory, wavelet analysis theory and its application, signal and image processing.
Zhen-Jiang Miao, Ph.D. supervisor, got his master's and doctoral degrees in BJTU, in 1990 and 1994, respectively. In 1994–2004 he studied and worked in France and Canada. In France, at first he worked in France TOULOUSE National Institute of Technology (INPT) to carry out a post-doctoral study and then worked in the National Academy of Sciences (INRA); in Canada, at first he worked in the Canada National Research Council Institute of Information Technology (IIT-NRC), then he worked in the North Telecommunications (Nortel) and RIM company to develop wireless network for communications equipment DMS-MTX and 3G handheld device Blackberry 6750 system and the products now have a wide range of global sales and applications. He has published academic papers over 70 articles. At present, he has presided over the national 973 issue of “visual media interactive and integration deal” and other important national research projects.
Zhe Wei received the B.S. and M.S. degrees in computer science from Huaqiao University, Quanzhou, China, and the Ph.D. degree in electrical and electronics engineering from Nanyang Technological University, Singapore. During September 2011 to August 2013, he was a research associate and a research scientist in the Temasek Laboratories, Computer vision group of Nanyang Technology University. From September 2013 till now, he is a postdoctoral fellow in the Electrical and Computer Engineering of the McMaster University, Ontario, Canada. His current research interests include multimedia processing, image/video compression, multiple description coding, and multimedia communication.
Contents lists available at ScienceDirect
Neurocomputing journal homepage: www.elsevier.com/locate/neucom
Defect inspection for TFT-LCD images based on the low-rank matrix reconstruction Yi-Gang Cen a,b, Rui-Zhen Zhao a,b,n, Li-Hui Cen c, Li-Hong Cui d, Zhen-Jiang Miao a,b, Zhe Wei e a
School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China Beijing Key Laboratory of Advanced Information Science and Network Technology, Beijing 100044, China c School of Information Science and Engineering, Central South University, Changsha, Hunan 410083, China d Department of Mathematics, Beijing University of Chemical Technology, Beijing 100029, China e Electrical and Computer Engineering, McMaster University, Ontario, Canada b
art ic l e i nf o
a b s t r a c t
Article history: Received 13 May 2014 Received in revised form 12 July 2014 Accepted 2 September 2014 Communicated by Jinhui Tang Available online 16 September 2014
Surface defect inspection of TFT-LCD panels is a critical task in LCD manufacturing. In this paper, an automatic defect inspection method based on the low-rank matrix reconstruction is proposed. The textured background of the LCD image is a low-rank matrix and the foreground image with defects can be treated as a sparse matrix. By utilizing the Inexact Augmented Lagrange Multipliers (IALM) algorithm, the segmentation of a LCD image can be converted into the reconstruction of a low-rank matrix with a fraction of its entries arbitrarily corrupted. This low-rank matrix reconstruction problem can be exactly solved via convex optimization that minimizes a combination of the nuclear norm and the l1-norm. Also, adaptive parameter selection strategy is proposed by conducting deep analysis on the IALM algorithm, which improves the generality of the IALM algorithm for different defect types. Experiment results show that our inspection algorithm is robust for the defect shapes and types under different illumination conditions. The shapes and edges of defect areas in the LCD images can be well preserved and segmented from textured background by our detection algorithm. & 2014 Elsevier B.V. All rights reserved.
Keywords: Defect inspection Low-rank matrix reconstruction LCDs IALM Adaptive parameter selection
1. Introduction Thin film transistor-liquid crystal display (TFT-LCD) has become a major technology for flat panel display in recent years due to their full-color display capabilities, low power consumption and light weight. It can be used in a wide range of electronic devices such as cellular phones, PDAs, computer monitors and television sets. To ensure display quality and to improve the yield of LCD flat panels, the inspection of defects in TFT-LCD panels becomes a critical task in LCD manufacturing. As the TFT-LCD panel becomes dense, small defects can only be observed at an extremely high resolution. For fast imaging of a large-sized TFT-LCD panel at a high resolution, a one-dimensional (1D) line scan system is demanded. A TFT-LCD panel image at a fine resolution presents very complicated repetitive patterns, which increases the difficulty of the defect detection task. A TFT panel generally contains horizontal gate lines on one plane and vertical data lines on the other plane. At each pixel, the
n Corresponding author at: School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China. E-mail address: [email protected] (R.-Z. Zhao).
http://dx.doi.org/10.1016/j.neucom.2014.09.007 0925-2312/& 2014 Elsevier B.V. All rights reserved.
gate of the TFT is connected to the gate line and the source is connected to the data line. Since the TFT panel is composed of horizontal gate lines and vertical data lines, it forms a structural texture that contains horizontal and vertical line patterns. A typical structurally textured image of a defect-free TFT panel is shown in Fig. 1. It can be seen that the image involves repetitive, equally spaced horizontal and vertical lines. Surface defect detection using machine vision has been largely tackled by texture analysis techniques [1]. A set of textural features are generally extracted in the spatial domain or in the spectral domain, and then various analysis methods such as distance measures [2], Bayes probability [3], neural networks [4] and support vector machine [5] are used for discriminating defective areas from the background based on the extracted features. For the surface defects inspection, Tsai et al. conducted in-depth research and proposed some valuable approaches. For example, Ref. [6] proposed a fast regularity measure for defect detection in nontextured and homogeneously textured surfaces, with specific emphasis on ill defined subtle defects. In [7], the fractal signal of a 1D grey-level TFT-LCD image was divided into small segments. Then by calculating each divided segment's normalized cross correlation with its neighbouring segments and comparing the resulting correlation value with a predetermined threshold, the
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
1207
from D in the same time. This leads to the segmentation of the foreground (defects) and the background (structural texture image) of a TFT-LCD defect image. To be formal, this problem can be described as min rankðAÞ þ γ J E J 0 ; A;E
Fig. 1. A surface image of a defect-free TFT panel.
segments containing a defect can be effectively identified. In [8], according to the singular value decomposition (SVD) of the LCD panel image, proper singular values that represent the background texture of the surface were selected. Then the matrix without the selected singular values was reconstructed, which can eliminate periodical, repetitive patterns of the textured image, and preserve the anomalies in the restored image. Some other related works of Tsai can be found in [9–11]. The above-mentioned defect inspection methods resulted in a significant development for the online or the offline product defect inspection. The existing vision-based techniques generally need a pre-stored reference image for comparison. This requires a large volume of data for reference and precise environmental controls such as alignment and lighting for test images. Moreover, the techniques for LCD inspection mainly focus on final appearance checks for defects such as nap or dark/bright spots after the fabrication is completed. Finally, most techniques need to adjust some parameters. The selection of parameters is complex and mostly relies on the human experiences. This will limit the generality of the techniques. In the recent years, low-rank matrix reconstruction became a hot issue in the areas of face recognition, image recovery, de-noising and data reconstruction. Low-rank matrix reconstruction mainly concerns how to recover the low-rank matrix from a large but sparse error data. In different application cases, lowrank matrix reconstruction also be called as sparse and low-rank matrix decomposition, robust principle component analysis, ranksparsity incoherence, etc. In this paper, we mainly focus on how to segment the defect area from the textured background of the LCD panel images accurately. Because the LCD images generally involve repetitive, equally spaced horizontal and vertical lines, the defect detection problem can be boiled down to the low-rank matrix reconstruction problem. The textured background image corresponds to a low-rank matrix and the foreground defect image corresponds to a sparse matrix. Then according to the joint minimization of nuclear norm and l1-norm, the low-rank matrix and the sparse defect matrix can be segmented by the low-rank matrix reconstruction theory.
2. Problem formulation Suppose that the original M N defect image is D. The low-rank background matrix is AMN and the sparse foreground matrix is EMN . For the TFT panel image, the defects are sufficiently sparse, one can exactly recover the low-rank matrix A from D ¼ A þ E. Also, the sparse matrix E corresponding to the defects can be obtained
s:t: D ¼ A þ E
ð1Þ
Here A is a low rank matrix representing the background of a TFTLCD image. E denotes the sparse error matrix between A and D. For a TFT-LCD image, E denotes the defects, noise and outliers in the image. J J 0 denotes the l0-norm (number of non-zero entries in the matrix), and γ 4 0 is a parameter that trades off the rank of the solution A versus the sparsity of the error E. Unfortunately, (1) is not tractable since both rank and l0-norm are non-convex and discontinuous functions. The optimization problem of (1) is extremely difficult (NP-hard in general) to solve. Recently, Wright et al. [12] have proposed that the problem in (1) can be solved by replacing the cost function with its convex surrogate, provided that the rank of the matrix A is not too high and the number of non-zero entries in the matrix E is not too large. This convex relaxation, dubbed Principal Component Pursuit (PCP) in [12], replaces rankðÞ with the nuclear norm and the l0-norm with the matrix l1-norm. Under quite general conditions, it has been proved in [12] that the following optimization problem has the same optimal solution as (1): min J A J n þ λ J E J 1 ; A;E
s:t: D ¼ A þ E
ð2Þ
where J A J n ¼ ∑i σ i ðAÞ is the nuclear norm of A. σ i ðAÞ denotes the ith singular values of the matrix A. J J 1 represents the l1-norm, i.e., the sum of the absolute values of all entries of the matrix. λ 4 0 is a weighting parameter. Theoretical considerations in [12] pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi suggest that λ must be of the form C= maxfM; Ng, where C is a constant, typically set to unity. It is interesting to note that the equivalence between (1) and (2) is not affected by the magnitude of the singular values of the solution A or by the magnitude of the non-zero entries of the error matrix E. Due to the ability to exactly recover underlying low-rank structure in the data, even in the presence of large errors or outliers, the optimization problem (2) is referred to as Robust Principal Component Analysis (RPCA) in [12] (a popular term that has been used by a long line of work aims to render PCA robust to outliers and gross corruption). Several applications of RPCA, e.g. background modelling and removing shadows and specularities from face images, have been demonstrated in [12] to show the advantage of RPCA.
3. Solution overview and algorithm for matrix reconstruction The optimization problem (2) can be treated as a general convex optimization problem and solved by any off-the-shelf interior point solver [13]. However, although interior point methods normally take very few iterations to converge, they have difficulty in handling large matrices because the complexity of computing the step direction is Oðm6 Þ, where m is the dimension of the matrix. Recently, iterative thresholding (IT) algorithm showed its efficiency for l1-norm minimization problem in compressed sensing [14,15] and the nuclear norm minimization in matrix completion (MC) problem [16]. Also, in [17], we proposed a rank adaptive atomic decomposition algorithm for the low-rank matrix completion problem. From (2) we can see that matrix recovery (RPCA) problem involves minimizing a combination of both the l1-norm and the nuclear norm. In [12], the authors have also adopted the IT technology to solve (2) and obtained similar convergence and scalability properties. However, the iterative thresholding scheme proposed in [12] converges extremely slowly.
1208
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
In [18], Lin et al. proposed two new algorithms for solving the problem (2): the first one is an accelerated proximal gradient (APG) algorithm applied to the primal; the second one is a gradient-ascent algorithm applied to the dual of the problem (2). Experiments showed that both algorithms are at least 50 times faster than the IT method. In addition, by utilizing the augmented Lagrange multipliers (ALM), Lin et al. [19] proposed the exact ALM (EALM) algorithm and moreover, a slight improvement over the EALM, the inexact ALM (IALM) algorithm for (2). It showed that the IALM is at least five times faster than APG, and its precision is also higher. In particular, the number of non-zeros in E computed by IALM is much more accurate than that by APG, which often leave many small non-zero terms in E. For the TFT panel defect inspection, the inspection algorithm should be fast so that it can be applied to the online inspection in the manufacturing process. Thus, we choose the IALM method to solve the optimization problem (2). By identifying X ¼ ðA; EÞ;
f ðXÞ ¼ J A J n þ λ J E J 1 ;
s:t: hðXÞ ¼ D A E;
ð3Þ
we can apply the augmented Lagrange multiplier method for problem (2) [19]. The Lagrangian function is
μ
LðA; E; Y; μÞ ¼ J A J n þ λ J E J 1 þ 〈Y; D A E〉 þ J D A E J 2F : 2
ð4Þ
The detailed algorithm is described in Algorithm 1. Here, 1 JðDÞ ¼ maxð J D J 2 ; λ J D J 1 Þ. svdðÞ denotes the singular decomposition of a matrix. Sε ðxÞ is a soft-thresholding (shrink) operator
defined as 8 > < xε : Sε ðxÞ ¼ x þ ε > : 0
if x 4 ε
if x o ε
ð5Þ
otherwise
Algorithm 1. Inexact ALM method for RPCA Input: Observation matrix D A RMN , λ, ε Output: ðAk ; Ek Þ 1. Y 0 ¼ D=JðDÞ; E0 ¼ 0; μ0 4 0; ρ 4 1; k ¼0. 2. do 3. ðU; Σ ; VÞ ¼ svdðD Ek þ μk 1 Y k Þ; 4. Ak þ 1 ¼ USμ 1 ðΣ ÞV T ; k
5. Ek þ 1 ¼ Sλμ 1 ðD Ak þ 1 þ μk 1 Y k Þ; k
6. Y k þ 1 ¼ Y k þ μk ðD Ak þ 1 Ek þ 1 Þ; μk þ 1 ¼ ρμk ; 7. k ¼ k þ 1. 8. until
J D Ak þ 1 E k þ 1 J F o JDJF
ε
It has been shown in [19] that if μk does not increase too rapidly (ρ should not be too large), then ðAk ; Ek Þ converges to an optimal solution ðAn ; En Þ to the RPCA problem. Extensive numerical experiments have shown that for geometrically growing μk, IALM method converges Q-linearly. On the other hand, ρ is an important
Fig. 2. Four defect images. (a)–(c) are under fine image resolution (60 pixels/mm); (d) is under coarse image resolution (20 pixels/mm). (a) Pinhole, (b) Scratch, (c) Particle, (d) Fingerprint.
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
1209
Fig. 3. Reconstructed sparse defective foreground images of (a) fingerprint and (b) pinhole images under different values of ρ.
parameter in this algorithm. It can be seen in the next section that the different selections of ρ will lead to different segmentation results and running times. This problem will be analysed in detail in the next section.
4. Experiment results In this section, we firstly analyse the relations of the values of ρ and the segmentation results of different defect types. Then an
1210
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
Fig. 4. The plots of SLR and running time under different values of ρ for the 4 defect images shown in Fig. 2. (a) Relations of SLR and different values of ρ. (b) Relations of running time and different values of ρ.
adaptive parameter selection strategy is proposed to improve the generality of the IALM algorithm. 4.1. Sparsity low-rank measure function and analysis of the parameter selection We will present experimental results of the defect images used in [8] to evaluate the performance of the proposed defect detection scheme based on the low-rank matrix reconstruction. These images contain micro defects including pinholes, scratches, particles and fingerprints on TFT panel surfaces . The test image size is 256 256 with 8-bit grey levels. The computer used in the experiment is ThinkPad X61 with Duo 2 G CPU and 2 G memory. Fig. 2 shows these four defect images. We set μ0 ¼ 1:25= J D J 2 for all images. Now we will discuss the effects of the parameter ρ. According to our experiments, the convergence rate of IALM will decrease when ρ is increased. But the residuals of repetitive horizontal and vertical line patterns along with the defects will be retained in the reconstructed sparse defect images with the increase of ρ. Fig. 3 shows the experiment results of fingerprint and pinhole images under different values of ρ. It can be seen in the Fig. 3 that for the pinhole defect, ρ ¼2 may be the best choice for the reconstruction of the sparse defect image because the residuals of the textured background in the first image of Fig. 3(b) are less than others. Also the defect shape and edge are well preserved. For the fingerprint image, when ρ ¼2, the reconstructed sparse defect image is almost blank. With the increase of ρ, the fingerprint shape and edge become clear along with the residuals of the textured background. This is because the fingerprint area can also be treated as a kind of repetitive texture with low rank and hard to be segmented from the background. ρ ¼ 8; 11; 14; 17 may be the better choices for fingerprint defect image. How to evaluate the segmentation results is a difficult issue for image segmentation. To the best of my knowledge, there is no unified criterion for the image segmentation. For a defect image, one may not judge if a small area should belong to the defect or the background even by human visual inspection. In our experiment, the reconstructed defect image matrix E is sparse and the textured background image matrix A is a low-rank matrix. In general, compared with the whole image, defect area is relatively small or tiny (for example, the defects in Fig. 2(a)–(c), but Fig. 2(d) is an exception). Thus most elements in E should be 0 and A should be a low-rank matrix with repetitive textures. Combining these two properties, we define a Sparsity Low-rank (SLR) measure function as follows: SLR ¼ J E J 0 =rankðAÞ
ð6Þ
Fig. 4(a) presents the relations of SLR and ρ for the 4 images showed in Fig. 2. Here, let ρ ¼ f2; 5; 8; …; 78; 81g, the corresponding SLR set is denoted as fSLRk g, k ¼ 1; 2; …; lengthðρÞ. lengthðρÞ denotes the length of the vector ρ. In Fig. 4(a), the value of SLR corresponding to each ρ is normalized, i.e., for a given defect image and different values of ρ, the kth normalized SLR is given by SLRk =maxfSLRgk , k ¼ 1; 2; …; lengthðρÞ. Fig. 4(b) shows the running time for each image. On the one hand, we need to determine the values of ρ according to analysis of the values of SLR. On the other hand, running time of the program is an important issue that needs to be considered for the online defect inspection. By (6), small SLR implies that there are more zeros in the matrix E and E should be sparser. This means that the residuals of the background in E are less. In addition, a small SLR also means that the rank of A is big, i.e., there are more residuals of defect area retained in the low-rank textured background matrix (the residuals of defect area in A will increase the rank of A). By the experiment results, pinhole, scratch and particle are according with this case. But the situation is different for the fingerprint image. By Fig. 2(d) we can see that fingerprint area is textured and should be low-rank to some extent. In the optimization problem (2), fingerprint area will be treated as the textured background and is difficult to separate from the background. In order to segment the fingerprint area into the sparse matrix E, the value of ρ has to be increased such that the fingerprint area is treated as the residuals of background and segmented into E. But ρ cannot be increased too large since the residuals of the repetitive horizontal and vertical textures will seriously affect the detection result. Also, it can be seen by Fig. 2(d) that the fingerprint area occupies a large area of the whole image thus one cannot expect that there are too many zeros in the sparse matrix E. If E is too sparse, it means that the fingerprint area is not well segmented from the background. Thus the selection of ρ should be larger than the cases of pinhole, scratch and particle so that the fingerprint area can be well segmented from the background. In addition, running time of the program is an important issue that needs to be considered in the industrial manufacture process. By Fig. 4(b) we can see that the running time decreases quickly when ρ o 10. Thus, as long as the value of ρ is not too small (ρ o5 by Fig. 4(b)), the running time will be acceptable. For the real industry used computer, the running time will be significantly decreased. 4.2. Adaptive selection strategy of
ρ
According to the above analysis, it can be seen that if the parameter ρ is fixed as a constant, the defect detection approach
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
1211
Fig. 5. Reconstructed low-rank textured background and sparse defective foreground images. The images in the left column are the low-rank background images. Images in the middle column are the defect segmentation results (sparse foreground defect images) and the right column is the binarization results of the sparse foreground defect images. (a) Pinhole, (b) scratch, (c) particle, and (d) fingerprint.
1212
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
Fig. 6. Defect detection results obtained in [8]. The images in the left column are the original defect images. Images in the middle column are the defect segmentation results and the right column is the binarization results.
will not be suitable for all defect types. In order to improve the generality of our detection algorithm, adaptive selection strategy of the parameter ρ is proposed. Motivated by the selection of parameter μk in Algorithm 1, we also start ρ from a small value and increase it gradually during the iteration of Algorithm 1. By the discussions in Section 4.1, we initially start with ρ ¼ 2 in Algorithm 1. The value of ρ at each iteration of Algorithm 1 is
increasing until it reaches a pre-defined small positive number ρ . In other words, ρk þ 1 ¼ minðηρk ; ρ Þ for k ¼ 1; 2; …. In this way, we need not pre-define the value of ρ and this adaptive parameter selection strategy will improve the generality of Algorithm 1. The segmentation results of the defect images in Fig. 2 are shown in Fig. 5. Also, the binarization images are presented such that the defect shapes can be seen clearly. In this experiment, ρ was set as 10 and η ¼1.07. The
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
1213
Fig. 7. Defect detection results of the touch screen images. The image in the left of each sub-figure is the original defect image. Image in the middle of each sub-figure is the defect segmentation result and the right one is the binarization result. (a–c) Crack, (d, e) particle, (f, g) scratch, (h, i) stain, and (j) dust.
average running time of these four images is 2.23 s. Here, the segmentation results obtained in [8] are also presented for comparison, see Fig. 6. It can be seen that the defect shapes of our segmentation results are well preserved than the results obtained in [8]. In order to test the performance and robustness of our algorithm for low resolution images and different defect types, 10 defective touch screen images are selected for testing. Fig. 7 shows the defective touch screen images of crack, particle, scratch, stain and dust. The image resolution is 96 DPI (approximation 4 pixels/mm) and the image size is 100 100. By Fig. 7 we can see that although the resolution of the defect images is very low and the illumination condition of each image is different, the defect areas still can be well segmented by our algorithm. Also, because the image size is small, the running time decreased quickly and satisfied the real-time requirement of online inspection better. The average running time of these 10 images is 1.01 s. Thus, our inspection algorithm is robust for different defect shapes, types, resolutions and illumination conditions. 4.3. Rank of A and sparsity of E For the RPCA problem, the rank of A is difficult to be estimated as a prior knowledge. Fortunately, Theorem 2 in [19] proved that if μk
does not increase too rapidly, so that ∑kþ¼11 μk 2 μk þ 1 o þ1 and limk- þ 1 μk ðEk þ 1 Ek Þ ¼ 0, then ðAk ; Ek Þ converges to an optimal solution ðAnk ; Enk Þ to the RPCA problem. Thus, in the real applications of Algorithm 1, we only need to make sure that A is a low-rank matrix. But the rank of A need not to be considered or pre-determined. Let β r ¼ rankðAÞ=M, βs ¼ J E J 0 =M M (Here we assume that the testing image size is M M). For the 4 images in Fig. 2 and the 10 images in Fig. 7, the values of βr and βs in each iteration of Algorithm 1 are shown in Fig. 8(a) and (b) respectively. In Fig. 8, the convergence condition was set as ε ¼ 10 8 and a total of 40 iterations were plotted. This is because we want to show the relations among βr, βs and the number of iterations clearly. In fact, in the above experiments, the convergence condition was set as ε ¼ 10 7 . Thus, for the images shown in Figs. 2 and 7, an average of only 15 iterations is needed to reach the convergence condition. It also can be seen by Fig. 8 that the rank of A is observed to be monotonically increasing and after about 15 iterations, it becomes stable. The sparsity probability βs also becomes stable when βr is stable. When βr is stable, the ranks of the textured background image matrices of the 4 TFT-LCD images are almost same. This also can be observed for the touch screen images according to Fig. 8(b). Thus, by Theorem 2 in [19] and Fig. 8, in the real applications of
1214
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
Fig. 8. βr and βs in each iteration of Algorithm 1. (a) βr and βs of the 4 TFT-LCD defect images. (b) βr and βs of the 10 touch screen defect images.
Algorithm 1, the rank of A need not to be considered or prior determined.
the parameter ρ. Experiment results showed that our proposed algorithm can well segment the defect area from the textured LCD image and the running time can satisfy the requirement of online inspection.
5. Conclusion Defect inspection is a key issue for the quality control of the TFT-LCD panel manufacture process. The defect inspection problem of TFT-LCD images can be modelled as the low-rank matrix reconstruction since the LCD image containing repetitive patterns is a low-rank matrix. Also, the foreground image matrix containing defect areas is a sparse matrix. Thus, under the restriction of D¼ AþE, the segmentation of the LCD image can be achieved by joint minimization of the nuclear norm of A and the l1-norm of E. In order to improve the generality of our proposed detection algorithm, adaptive parameter selection strategy was proposed for
Acknowledgements The authors would like to thank Prof. Du-Ming Tsai for providing the TFT-LCD test images, who is with the department of Industrial Engineering and Management, Yuan-Ze University, Taiwan, R.O.C. http://machinevision.iem.yzu.edu.tw/vision/. Also, the authors would like to thank the anonymous reviewers for many valuable comments which contributed to the improvement of this paper. This work was supported by the National Natural Science Foundation of China (Grant nos. 61272028, 61473317, 61273274,
Y.-G. Cen et al. / Neurocomputing 149 (2015) 1206–1215
61104078); Fundamental Research Funds for the Central Universities of China (Grant no. 2013JBZ003); Specialized Research Fund for the Doctoral Program of Higher Education of China (Grant no. 20120009110008); Program for New Century Excellent Talents in University (Grant no. NCET-12-0768); National Key Technology R&D Program of China (Grant no. 2012BAH01F03); National Basic Research (973) Program of China (Grant no. 2011CB302203); The National High Technology Research and Development Program (863) of China (Grant no. 2014AA015202). Program for Changjiang Scholars and Innovative Research Team in University (Grant no. IRT201206).
References [1] X. Xie, A review of recent advances in surface defect detection using texture analysis techniques, Electron. Lett. Comput. Vis. Image Anal. 7 (3) (2008) 1–22. [2] A. Serdaroglu, A. Ertuzun, A. Ercil, Defect detection in textile fabric images using wavelet transforms and independent component analysis, Pattern Recognit. Image Anal. 16 (1) (2006) 61–64. [3] D. Brzakovic, H. Beck, N. Sufi, An approach to defect detection in materials characterized by complex textures, Pattern Recognit. 23 (1) (1990) 99–107. [4] M.M. Van Hulle, T. Tollenaere, A modular artificial neural network for texture processing, Neural Netw. 6 (1) (1993) 7–32. [5] G.A. Ruz, P.A. Estévez, P.A. Ramirez, Automated visual inspection system for wood defect classification using computational intelligence techniques, Int. J. Syst. Sci. 40 (2) (2009) 163–172. [6] D.M. Tsai, M.C. Chen, W.C. Li, et al., A fast regularity measure for surface defect detection, Mach. Vis. Appl. 23 (5) (2012) 869–886. [7] D.M. Tsai, S.T. Chuang, 1D-based defect detection in patterned TFT-LCD panels using characteristic fractal dimension and correlations, Mach. Vis. Appl. 20 (6) (2009) 423–434. [8] C.J. Lu, D.M. Tsai, Automatic defect inspection for LCDs using singular value decomposition, Int. J. Adv. Manuf. Technol. 25 (1–2) (2005) 53–61. [9] C.J. Lu, D.M. Tsai, Independent component analysis-based defect detection in patterned liquid crystal display surfaces, Image Vis. Comput. 26 (7) (2008) 955–970. [10] D.M. Tsai, H.Y. Tsai, Low-contrast surface inspection of mura defects in liquid crystal displays using optical flow-based motion analysis, Mach. Vis. Appl. 22 (4) (2011) 629–649. [11] D.M. Tsai, I.Y. Chiang, Y.H. Tsai, A shift-tolerant dissimilarity measure for surface defect detection, IEEE Trans. Ind. Informatics 8 (1) (2012) 128–137. [12] J. Wright, Y. Peng, Y. Ma, A. Ganesh, S. Rao, Robust principal component analysis: exact recovery of corrupted low-rank matrices by convex optimization, in: Proceedings of Neural Information Processing Systems, 2009, p. 3. [13] M. Grant, S. Boyd, CVX: Matlab Software for Disciplined Convex Programming (Web Page and Software). 〈http://stanford.edu/”boyd/cvx〉, 2009. [14] E.T. Hale, W. Yin, Y. Zhang, Fixed-point continuation for l1-minimization: methodology and convergence, SIAM J. Optim. 19 (3) (2008) 1107–1130. [15] W. Yin, S. Osher, D. Goldfarb, J. Darbon, Bregman iterative algorithms for l1-minimization with applications to compressed sensing, SIAM J. Imaging Sci. 1 (1) (2008) 143–168. [16] J.F. Cai, E.J. Cands, Z. Shen, A singular value thresholding algorithm for matrix completion, SIAM J. Optim. 20 (4) (2010) 1956–1982. [17] H.Y. Wang, R.Z. Zhao, Y.G. Cen, Rank Adaptive Atomic Decomposition for LowRank Matrix Completion and Its Application on Image Recovery, Neurocomputing 145 (5) (2014) 374–380. [18] Z. Lin, A. Ganesh, J. Wright, L. Wu, M. Chen, Y. Ma, Fast convex optimization algorithms for exact recovery of a corrupted low-rank matrix, in: Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), 2009, p. 61. [19] Z. Lin, M. Chen, L. Wu, Y. Ma, The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices, Technical Report UILU-ENG09-2215, University of Illinois at Urbana-Champaign, 2009.
Yi-Gang Cen received the Ph.D. degree in Control Science & Engineering, in 2006, from the HUST. In September 2006, he joined the Signal Processing Centre, EEE, NTU in Singapore as a research fellow. He is currently an associate professor and a supervisor of doctor student of BJTU. From January 2014 to January 2015, he was a visiting scholar of the department of Computer Science, University of Missouri in US. His research interests include Compressed Sensing, Sparse Representation, Low-rank Matrix Reconstruction, and Wavelet construction theory.
1215 Rui-Zhen Zhao was born in 1975. He got his Ph.D. degree from Xidian University. After that he was with a postdoctoral fellow in Institute of Automation, Chinese Academy of Sciences. He is currently a professor and a supervisor of doctor student. His research interests include wavelet transform and its applications, algorithms off image and signal processing, compressive sensing and sparse representation.
Li-Hui Cen received the Ph.D. degree in Control Theory and Control Engineering from the Shanghai Jiaotong University, Shanghai, China, in 2009. Now she is an associate professor with the School of Information Science and Engineering, Central South University, Hunan, Changsha, China. Her research interest includes control of open channels, signal processing, and predict control.
Li-Hong Cui received the B.S. degree from Henan Normal University, Xinxiang, China, in 1986, the M.S. degree Zhengzhou University, Zhengzhou, China, in 1999, both in mathematics, and the Ph.D. degree in mathematics and computing science from Xi'an Jiaotong University, Xi'an, China, in 2004. She is currently a professor with the Department of Mathematics and Computer Science, Beijing University of Chemical Technology, Beijing, China. She is currently working on implementing some of the new image processing algorithms and multiwavelet. Her present research interests include approximation theory, wavelet analysis theory and its application, signal and image processing.
Zhen-Jiang Miao, Ph.D. supervisor, got his master's and doctoral degrees in BJTU, in 1990 and 1994, respectively. In 1994–2004 he studied and worked in France and Canada. In France, at first he worked in France TOULOUSE National Institute of Technology (INPT) to carry out a post-doctoral study and then worked in the National Academy of Sciences (INRA); in Canada, at first he worked in the Canada National Research Council Institute of Information Technology (IIT-NRC), then he worked in the North Telecommunications (Nortel) and RIM company to develop wireless network for communications equipment DMS-MTX and 3G handheld device Blackberry 6750 system and the products now have a wide range of global sales and applications. He has published academic papers over 70 articles. At present, he has presided over the national 973 issue of “visual media interactive and integration deal” and other important national research projects.
Zhe Wei received the B.S. and M.S. degrees in computer science from Huaqiao University, Quanzhou, China, and the Ph.D. degree in electrical and electronics engineering from Nanyang Technological University, Singapore. During September 2011 to August 2013, he was a research associate and a research scientist in the Temasek Laboratories, Computer vision group of Nanyang Technology University. From September 2013 till now, he is a postdoctoral fellow in the Electrical and Computer Engineering of the McMaster University, Ontario, Canada. His current research interests include multimedia processing, image/video compression, multiple description coding, and multimedia communication.