Fabric defect inspection based on lattice segmentation and lattice templates

Accepted Manuscript

Fabric Defect Inspection Based on Lattice Segmentation and Lattice Templates Liang Jia , Junguo Zhang , Shuyue Chen , Zhenjie Hou PII: DOI: Reference:

S0016-0032(18)30477-0 10.1016/j.jfranklin.2018.07.005 FI 3548

To appear in:

Journal of the Franklin Institute

Received date: Revised date: Accepted date:

11 December 2017 18 May 2018 16 July 2018

Please cite this article as: Liang Jia , Junguo Zhang , Shuyue Chen , Zhenjie Hou , Fabric Defect Inspection Based on Lattice Segmentation and Lattice Templates , Journal of the Franklin Institute (2018), doi: 10.1016/j.jfranklin.2018.07.005

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

ACCEPTED MANUSCRIPT

Fabric Defect Inspection Based on Lattice Segmentation and Lattice Templates Liang Jia1 Junguo Zhang2* Shuyue Chen1* Zhenjie Hou1* 1

School of Information Science & Engineering, Changzhou University, China 1*

[email protected]

School of Technology, Beijing Forestry University, China 2* [email protected]

CR IP T

2

Abstract

PT

ED

M

AN US

Automated fabric inspection is a challenging task due to the unpredictable visual forms of the fabric defects and their scarcity compared with the tremendous amount of defect-free fabric products. This paper proposes a novel method based on lattice segmentation and lattice templates which automatically identifies the defects of fabric images. With the proposed method, a fabric image is segmented to lattices by inferring the placement rule of the texture primitives categorized to distinct texture classes. Each texture class is modeled by multiple templates inferred from the defect-free samples based on some metrics determined a priori according to their inspection efficiencies. For a lattice segmented from a given image, the most similar template is identified through a template matching process which compensates the local deformations around the lattice, and the distances between the lattice and the identified template are estimated based on the selected metrics. The lattices of distances exceeding the learnt distance range are identified as defective. The performance of the proposed method is evaluated based on two databases respectively providing pixel-level and image-level evaluations. For both databases, the receiver operating characteristic curves are plotted and the average areas under curves are 0.86 and 0.95 respectively for pixel-level and image-level databases. The proposed method is further tested on the blurred and noisy version of images from pixel-level database and the resulting area is 0.81 on average. The proposed method outperforms the state-of-the-art methods by comparing corresponding areas.

CE

Keywords: Fabric defect inspection; Lattice segmentation; Image Decomposition; Patterned texture

AC

1. Introduction

As an industrial product possessing the most diverse two-dimensional surfaces, fabric (textile) serves many fields of human civilization and is inseparable from our daily lives. The number of fabric products is tremendous and the quality control thus plays an important role in saving cost [1]. A critical aspect of quality control is inspecting fabric defects of unpredictable visual forms which randomly occur in the automatic manufacturing process. Consequently, it is difficult to collect lots of defective fabric samples. Hence, Automated Fabric Inspection (AFI) which identifies defects of unpredictable visual forms is always developed in absence of defective samples, which makes AFI as a challenging task. As a result, there are numerous AFI methods developed for various fabrics. These methods may be categorized from different perspectives such as the texture representations and the fabric types. 1

ACCEPTED MANUSCRIPT

AC

CE

PT

ED

M

AN US

CR IP T

According to the texture representations, AFI methods may be categorized to four classes [2][3][4], i.e., statistical [5][6][7], spectral [8][9][10], model-based methods [11][12][13] and Structural Analysis (SA) [14][15][16]. Statistical methods based on the gray values are found not good at identifying subtle defects [8]. Spectral methods mainly focus on the patterned fabric images consisting of the repeated texture primitives whose periodicity can be revealed more easily in frequency domain than in spatial domain. Therefore, spectral methods commonly require the fabric should be explicitly periodic [4]. Model-based methods represent texture characteristics as the parameters of some statistical models like autoregressive models and Markov Random Field (MRF). It is found that small defects cannot be efficiently identified by using model-based methods [8]. SA methods represent the fabric texture by texture primitives (texels) [3][4] repeated according to some placement rules which can be random [16] or specific [14][15]. The texture primitives are modeled differently in SA methods. For instance, texture primitive is defined as the runs of the foreground pixels in the binarized fabric image in [14], and the defects are identified based on the analyses of the histograms associated with the run locations and lengths. Run locations serve as the placement rule to assist the defect inspection. In [15], texture primitive is defined as texture blobs enclosed by the rectangular regions within a grid overlapping the binarized image. The defects are identified by comparing the maximum frequency differences of the texture primitives. Unlike [14], the inference of the placement rule is omitted in [15] because texture primitives are assumed to be arranged in a rigid grid. Contrary to the rigid grid adopted globally for all images as reported in [15], an inference process revealing a dynamic grid for a single given image is adopted to segment texture primitives in both [17] and [18]. The inference is derived based on the locations of texture blobs and thus the deformation of texture primitives are partially taken into account. Therefore, the dynamic grid changes correspondingly to conform the texture blob positions which differ from image to image. The rectangular region determined by the dynamic grid is named lattice in [17] and [18]. The advantage of lattice-based methods is that a given image is represented by hundreds of lattices instead of millions of pixels, which makes computationally-expansive operations practical in AFI. On the basis of fabric types, AFI methods may be categorized to two classes [3], i.e., methods capable or incapable of processing fabrics categorized as groups other than p1 group of wallpaper group [19]. Wallpaper groups categorize two dimensional repetitive patterns according to their symmetries. The symmetry defines a set of rules to generate the pattern based on some smallest texture called motif. Essentially, there are four basic rules to build symmetry, i.e., translations; rotations, reflections and glide reflections. The group p1 contains only translations; there are no rotations, reflections, or glide reflections. Most of the aforementioned methods except [17] and [18] aim at the plain or twill fabric categorized as p1 group [19]. A few are able to process fabrics of groups other than p1 group [3], e.g., Wavelet-pre-processed Golden Image Subtraction (WGIS) [20], Bollinger Bands (BB) [21], Regular Bands (RB) [22], Image Decomposition (ID) [23], Motif-Based (MB) method [19] and Elo Rating (ER) method [24]. BB is developed based on the observation that defects break the regularity of runs of the pixel values along rows and columns in fabric image. The regularities of the horizontal and vertical runs of pixels are measured by computing the values named upper and lower bands. The defects are identified as the pixels whose bands exceed the band ranges learnt from defect-free samples only. RB inherits the spirit of BB, and its framework is very similar to that of BB except that the regularity is evaluated by computing the light and dark regular bands instead of the lower and 2

ACCEPTED MANUSCRIPT

AC

CE

PT

ED

M

AN US

CR IP T

upper bands. Besides the regularity of pixel runs, templates are another popular means to inspect defects. WGIS works based on the subtraction between a manually-selected template and pixels enclosed in a window. The subtraction is conducted according to the level-1 approximation yielded by Haar wavelet for reducing the noises. The defects are identified as the pixels whose subtraction exceeds a range learnt from the defect-free samples only. Another template-based method named ER [24] is similar to WGIS except that the subtraction in [20] is replaced by a matching process inspired by the Elo rating system for evaluating the chess players’ capability. Essentially, a patch of predefined size is compared with several randomly-chosen patches for a given fabric image by computing scores involving their subtractions. The defects are identified as the pixels which are centers of the patches with scores exceeding the range learnt from the defect-free samples only. This paper proposes a method based on lattice segmentation and modular metric framework capable of processing images of fabrics from groups other than p1. The fabric image is assumed to consist of patterns repeated in a manner consistent horizontally and vertically in the image, the patterns lead to texture blobs in the binary version of the image, and the texture blobs are aligned with the image axes. For a given fabric image meeting this assumption, lattice segmentation infers a grid overlapping on the image to capture the patterns in a fine texture granularity. Especially for the alternatively-changed patterns, textures enclosed by the lattices also change periodically. For each type of texture during a period, it is modeled by a group of templates generated based on lattices of this type. The modeling process incorporates distance metrics for measuring lattice texture similarities, and the metrics serve as replaceable module in the process. Thus, the proposed method can be flexibly adjusted according to its efficiency of the application on hand. Generally, the contributions of this paper are summarized as follows. 1) A novel fabric inspection algorithm based on Lattice Segmentation and Lattice Templates (LSLT) is proposed. Contrary to the traditional methods involving some feature extractions of the fixed structures, the proposed algorithm can flexibly combine the distance metrics found efficient for the current application, which leads to the great adaptability. 2) The proposed algorithm is tested on multiple databases including one containing the blurred and noisy fabric images rarely found in fabric inspection literatures. The experiment results are represented as ROC curves which are rarely adopted but effectively reflect the performances of AFI methods throughout fabric inspection literatures. The following parts of this paper are organized as follows. In Section 2, the reported algorithms involved in the proposed method are briefly introduced. In Section 3, the novel lattice segmentation and the modular framework of the proposed method are outlined. In section 4, the optimal metric combination and the corresponding performance are evaluated. Lastly, Section 5 is the conclusion of the paper.

2. Related Works The basics of the proposed method involve several reported algorithms which can be categorized to two classes based on their functionalities. The two classes are image decomposition, and lattice segmentation. Image decomposition differs from the common image enhance techniques which 3

ACCEPTED MANUSCRIPT

AN US

CR IP T

indistinguishably enhance the edges and textures, and it generates an edge-enhanced version called carton or structural image in which only edges are enhanced and the rest are blurred. Thus, Image decomposition may serve as an ideal preprocessing step for segmentation tasks. There are various image decomposition techniques adopted in AFI methods, e.g., Morphological Component Analysis (MCA) in [17] and [18], the method in [26] provides inseparable functionality in [23]. In this paper, Relative Total Variation (RTV) proposed in [25] is chosen as the preprocessing step of the proposed method. Lattice segmentation (LS) serves for inferring the placement rule of texture primitives. The rule is commonly represented as a grid determining the boundaries between lattices, which represents an image by hundreds of texture-homogeneous lattices instead of millions of unrelated pixels. Hence, LS is particularly suitable for processing the fabric images containing repeated fabric patterns. LS methods may be roughly categorized to two classes [27], i.e., local feature-based and Global Feature-based Methods (GFM). The former identifies the texture primitives before inferring the placement rule, while the latter reverses the order. Generally, GFM are suitable for images containing large texture primitives of sophisticated appearances [27]. Consequently, GFM are commonly adopted in AFI, e.g., Liu et al’s method in [28] and LS method in [17] and [18]. Liu et al’s method infers the placement rule by analyzing the peaks yielded by autocorrelation function and introduces dominant peaks to filtrate spurious peaks disturbing the inference. However, the number of the dominant peaks cannot be automatically determined [27]..

3. The LSLT Method

AC

CE

PT

ED

M

Suppose there are 𝑰1 , 𝑰2 … 𝑰𝑁 defect-free fabric images consisting of orthogonally-repeated texture primitives categorized to 𝑡 distinct classes, the classes may be modeled by some templates reflecting the most common characteristics shared by the texture primitives. However, there are three issues about this modeling process: 1) how the fabric image can be segmented to lattices representing the texture primitives of different classes; 2) there may be pathological data in some training samples; 3) the diverse characteristics of the lattices categorized to the same class may not be fully captured by a single template. To solve the aforementioned issues, a lattice-segmentation-based modular framework is derived. The framework consists of three components, i.e., 1) lattice segmentation, 2) template generating and 3) defects inspection. 1) Lattice segmentation serves for tackling the first issue, i.e., representing texture primitives to rectangular lattices enclosing textures categorized to different classes. 2) Template generating aims at the second and the third issues, i.e., each class is modeled by multiple templates whose sizes and textures are similar to most of the lattices related with the class. The similarities are individually measured through distinct metrics. Hence, for each class of lattices, it is represented by multiple templates for each metric. The characteristic diversity of the lattices categorized to a single class is thus reflected through multiple templates related with the different metrics, and the pathological data may be avoided by eliminating the templates found to be dissimilar to all lattices of the same class. 3) Defects inspection identifies defects as the lattices of distances exceeding the learnt thresholds. The whole procedure of LSLT is illustrated in Fig. 1. As shown in the figure, for a given fabric image, it is first segmented to non-overlapping lattices, and then lattices flow through multiple paths linked by sequential arrows. Each path adopts the same steps but different metrics to identify defects except the distance metric. Although the steps of paths are the same, the templates and thresholds inferred on different metrics are very different 4

ACCEPTED MANUSCRIPT

as shown in the third and fourth columns on the left of Fig. 1. For the bottom path in the figure, there are one template inferred for texture primitive class 1 and two templates for class 2. For the top path, the number of templates for a specific class differs from the bottom one. For any branch, each lattice segmented from the input image is compared with the templates, and then the template most similar to the lattice is identified. The lattice is thus categorized to the texture primitive class the same as the template. The fourth column on the left of Fig. 1 illustrates the class index of each lattice with the digit 1 and 2 on a gray block, and its gray scale reflects the distance. The defects are identified as lattices with abnormal class indices and distances. Finally, all defects are merged to yield the final detection result.

AN US

segment lattices

...

Output

...

Lattices

...

Input Image

Templates of distinct metrics

CR IP T

Class Index Map & Distances

compare each lattice with templates

identify lattices of class indices or distances beyond the thresholds

M

find the template most similar to each lattice, estimate distance and class index

merge all identified lattices

Fig. 1 Flowchart of LSLT.

3.1 Lattice Segmentation

AC

CE

PT

ED

Lattice segmentation is built on the assumptions that the given fabric image contains texture primitives repeated periodically and the contrast between patterns and the background is strong enough to yield area-similar texture primitives covering the patterns. Suppose the assumption is met, then whatever the textures of the fabric patterns are, the background pixels always concentrate between two adjacent texture primitives. If the background pixels along rows or columns are counted, then there are local maximums between any two adjacent texture primitives, which form ―peaks‖. The regular locations of peaks suggest the separators of texture primitives, and the resulting regions segmented by separators are called lattices. However, when the fabric image contains defects, the defects may disturb the regularity of peaks. Lattice segmentation method illustrated in Fig. 2 is developed to overcome such disturbances to find separators for lattices. As shown in Fig. 2, the main steps of lattice segmentation are projection estimation, finding and expanding initial separators. These steps are introduced in the following subsections sequentially.

3.1.1 Background Pixel Projections The first step of lattice segmentation is estimating background pixel projections based on the image decomposition technique which reduces the noises and the disturbances originating from defects. The main steps involved in the projection estimation are illustrated in Fig. 3. As shown in the figure, for a given grayscale fabric image 𝑰, the Relative Total Variation (RTV)-based method [25] is employed to blur the regions filled with slowly-changed gray values and to preserve the 5

ACCEPTED MANUSCRIPT

CR IP T

regions of abruptly-changed gray values like edges. The resulting image is called cartoon image 𝑰𝑐 , and Bradley’s method [29] is adopted to generate the binary cartoon image 𝑰𝑡𝑐 , i.e., Bradley’s method employs the mean in the neighborhood of each pixel in 𝑰𝑐 to threshold the pixel. In Fig. 3, 𝑰 is represented as a 3D mesh diagram, and some thresholds are visualized as the gray planes. The darkness of the plane changes from block to block, which illustrates the change of the local adaptive thresholds associated with the blocks. The regions consisting of 8-connected foreground pixels in 𝑰𝑡𝑐 are identified as texture blobs through Moore-Neighbor tracing algorithm. Texture blob area is defined as the number of foreground pixels covered by the texture blob. According to the fabric image assumptions made in Section 3.1, the areas of texture primitives are expected to be roughly the same. Due to the disturbances like noises or defects, there may be areas differing a lot from the texture primitive areas, i.e., the ones not in the range ((1 − 𝛼) ∙ 𝑚𝑎 , (1 + 𝛼) ∙ 𝑚𝑎 ) where 0 ≤ 𝛼 ≤ 1, 𝛼 is a hyper parameter named area factor and 𝑚𝑎 denotes the median of all areas in 𝑰𝑡𝑐 . However, 𝑚𝑎 may also be biased if there are numerous noise-like small areas. Hence, morphological methods erosion and dilation are employed to remove the noise-like texture blobs. binary objects

and

final separators

and

AN US

initial separators

along rows

grayscale fabric image

M

along columns

3.1.1 project background pixels along rows and columns

3.1.2 find initial separators

3.1.3 expand initial separators

CE

PT

ED

Fig. 2 Lattice segmentation. Once the disturbances are reduced, the separator estimation starts by finding the ―peaks‖ of the background pixel projections 𝓟𝑟 and 𝓟𝑐 shown as the ―waves‖ in Fig. 3, i.e., background pixel numbers along rows and columns. The most well-spaced peaks are then identified as the initial separators 𝑺𝑟 and 𝑺𝑐 . Since the initial separators may only cover parts of 𝑰 as shown in Fig. 3, they are further expanded to separators 𝑺∗𝑟 and 𝑺∗𝑐 for fully covering 𝑰. The processes of estimating 𝑺𝑟 , 𝑺𝑐 , 𝑺∗𝑟 and 𝑺∗𝑐 are introduced in the following two subsections.

3.1.2 Initial Separator Estimation

AC

The second step of lattice segmentation is finding initial separators 𝑺𝑟 and 𝑺𝑐 , i.e., the positions of regularly-distributed peaks. This process is illustrated in Fig. 4. As shown in the figure, for a given projection like 𝓟𝑐 , the peaks are defined as the positions in a projection where the background pixel number changes from the rise to the drop or vice versa. However, only some of peaks may serve as candidates. As shown in Fig. 4, the peaks near maximal peak value seem promising candidates and some lower peaks may also contribute information about the ideal lattice size whose definitions are given below. A simple global threshold of peaks may leave out such information. For emphasizing the high peaks while probing the low peaks, the dominance value 𝓓(𝑝) for a peak 𝑝 serves to filtrate peaks. 𝓓(𝑝) is defined as 𝓓(𝑝) = ∑ arg max |*𝑖 + 𝑠ℓ|𝑝𝑖+𝑠ℓ ≤ 𝑝𝑖 , ℓ = 1,2 … 𝑛+|, 𝑠=±1

𝑛

6

ACCEPTED MANUSCRIPT where 𝑝𝑖 denotes the peak value w. r. t. the index 𝑖. 𝓓(𝑝) is the number of consecutive peaks lower than 𝑝. For peaks shown in Fig. 4, peaks of distinct dominance values are marked by using different colors.

compute cartoon image through SET method final separators

and

mesh diagram of

cartoon image

binarize cartoon image through Bradley's method

initial separators

with adaptive thresholds

adopt morphological methods to remove noises

binary objects of similar areas

and

binarized cartoon image

CR IP T

grayscale fabric image

along rows

AN US

compute areas and centroids

binary objects of

along columns

expand initial separators

find initial separators

remove binary objects of abnormal areas

M

project background pixels along rows and columns

AC

CE

PT

ED

Fig. 3 Projection estimation. If the peaks are filtrated based on their dominance values sorted in the decreasing order, we acquire peaks at different dominance levels, e.g., there are six distinct values 𝓓∗ = *27, 11, 4, 2, 1, 0+ in Fig. 4, and eight peaks are above the first dominance level. For the peaks at ℓth level, it is easy to compute distances 𝓭 between two adjacent peaks and frequency 𝓕(ℓ) (occurrence number) of the median distance 𝓜(ℓ). The frequencies 𝓕 of the median distances 𝓜 are accumulated from top level to lower level, and thus if the underlying fabrics have repeated patterns, then the pattern borders lead to regular distance values. When lowering the dominance threshold and accumulating the distance frequencies, these regular distances may lead to large accumulations, because these distance values may be discovered at different levels regularly while other values appear randomly. The value in 𝓜 of the maximal frequency is defined as the initial lattice size ℒ𝑑′ where 𝑑 can be 𝑟 or 𝑐 denoting the row or column dimension, i.e., |𝓓∗ |

ℒ𝑑′ = arg max ∑ 𝓕(ℓ) δ(𝓜(ℓ) − 𝑚), 𝑚∈𝓜∗

ℓ=1

where δ is Dirac delta function and 𝓜 denotes the distinct values in 𝓜. In practice, ℒ𝑑′ is employed for learning ideal lattice size based on training samples and thus ideal lattice size is known ahead of estimating 𝑺𝑟 and 𝑺𝑐 . For example, suppose there are training samples 𝑰1 , 𝑰2 … 𝑰𝑁 of the same defect-free fabric patterns, then the ideal lattice size, i.e., the lattice height ℒ𝑟∗ and width ℒ𝑐∗ are defined as the medians of the lattice sizes computed based on each training ∗

7

ACCEPTED MANUSCRIPT sample. In this case, the known ℒ𝑑∗ serves as the reference for finding 𝑺𝑟 and 𝑺𝑐 . The process of finding initial separators reverses the procedure of estimating ℒ𝑑∗ , i.e., for a given ℒ𝑑∗ , we focus on the dominance level ℓ∗ whose distance median ℒ𝑑 is closest to ℒ𝑑∗ and of maximal frequency if there are multiple levels whose medians share the same minimal distance to ℒ𝑑∗ , i.e., ℓ∗ = arg max 𝓕(ℓ) ∙ δ (𝓜(ℓ) − arg max|𝑚 − ℒ𝑑∗ |). ℓ=1,2,…|𝓓∗ |

𝑚∈𝓜∗

∗

CR IP T

At the level ℓ , the longest run of the consecutive peaks whose distances approximating ℒ𝑟 (ℒ𝑐 ) is defined as the initial separators 𝑺𝑟 (𝑺𝑐 ). Let 𝒹 denote the distance between two neighboring peaks in level ℓ∗, and then 𝑺𝑟 (𝑺𝑐 ) is defined as the row (column) indices associated with the peaks whose indices are 𝑗 ∗ + 1, 𝑗 ∗ + 2 … 𝑗 ∗ + 𝑛∗ where 𝑗 ∗ and 𝑛∗ for row indices are defined as below, 𝑗 ∗ , 𝑛∗ = arg max|{𝑗 + 𝑘||𝒹𝑗+𝑘 − ℒ𝑑 | ≤ 𝛽ℒ𝑑 , 𝑘 = 0,1,2 … 𝑛}|, 𝑗,𝑛

Peaks at the ≥27

≥11

th Dominance Level

at the

th Dominance Level

Initial Column Separators

M

≥4

≥2

Column Index

find dominance values for each peak

Background Pixel Number

Background Pixel Projection Along Columns

AN US

where 0 < 𝛽 < 1 and 𝛽 is a hyper parameter named tolerance. For column indices, 𝑗 ∗ and 𝑛∗ are defined similarly. To refine 𝑺𝑟 (𝑺𝑐 ), the median centroids of texture primitives between two adjacent separators in S𝑟 (𝑺𝑐 ) is computed, and each separator is adjusted to have equal distances to two neighboring median centroids.

ED

≥1

PT

≥0

CE

filrate

s w.r.t.

find the most frequent distance 1 between ps at the lth dominance level

find initial column separators based on

s

Fig. 4 Finding initial separators.

3.1.3 Initial Separator Expansion

AC

The final step of lattice segmentation is expanding the initial separators. The main steps of expanding are illustrated in Fig. 5. Because the initial separators may only cover parts of fabric image 𝑰, expanding 𝑺𝑟 and 𝑺𝑐 is necessary to fully cover 𝑰. A simple strategy like inserting separators to 𝑺𝑟 and 𝑺𝑐 at the fixed step of ideal lattice size ℒ𝑟∗ and ℒ𝑐∗ may not yield satisfactory segmentation due to the deformed fabric surface. One of the possible means to tackle the deformation is adjusting 𝑺𝑟 and 𝑺𝑐 according to the centroids of texture blobs reflecting the texture primitives as illustrated in Fig. 5 which depicts the steps of expanding 𝑺𝑐 , i.e., for a given fabric image 𝑰 and the corresponding 𝑺𝑐 , the expanding starts at both sides of 𝑺𝑐 . For either side, two separators ahead or behind the marginal separators of 𝑺𝑐 are predicted at the fixed steps of length ℒ𝑐∗ , and then the median centroids shown as crosses in Fig. 5 between the predicted and the marginal separators are estimated. The predicted separator just one step ahead or behind the 8

ACCEPTED MANUSCRIPT

marginal separator is adjusted to have equal distances to two neighboring median centroids. This procedure is repeated until the image border is reached. Initial Column Separators

Predicted Separators Centroids Between Separators

Adjusted Separators

Expanded Separators

...

adjust separators compute centroids between according to centroids predicted separators

repeat predicting and adjusting until reach the image borders

CR IP T

predict next two separators on both sides based on

Fig. 5 Expanding initial separators.

3.2 Modular Framework

AN US

Although lattice segmentation yields lattices for a given fabric image, there are yet unknown knowledge which is necessary for inspecting defective lattices, e.g., the details of the classes associated with the texture patterns enclosed by the lattices. Generally, there are three different kinds of parameters representing the unknown knowledge: 1) lattice period, i.e., the number of classes, 2) the templates of each class and 3) thresholds, i.e., the limits of distances between the lattices of defect-free lattices with the templates. Conceptually, there are two phases, i.e., training and testing phases. Training phase aims at estimating unknown parameters and each parameter is learnt based on the knowledge gained in the previous learning step. Testing phase conducts defect inspection. Unknown knowledge learning and fabric defect inspection are sequentially introduced in the following subsections.

M

3.2.1 Lattice Period

AC

CE

PT

ED

Suppose the lattices of the fabric image 𝑰 are segmented based on the ideal lattice size, and that the textures enclosed in two lattices of the same row or column separated by (𝑡 − 1) lattices are the same, then 𝑡 is defined as the lattice period of 𝑰, i.e., the lattices can be categorized to 𝑡 different lattice classes based on their textures. 𝑡 may be estimated by investigating the periods of the lattice rows and columns. Specifically, for a training sample, feature vectors of lattices can be computed through HOG [30]. For a lattice 𝑳, it is easy to estimate the Euclidean distances between its feature vector and feature vectors of all other lattices in the same row or column of 𝑳. If the distances are arranged in the spatial order of the corresponding lattices, then the resulting arrangement may reflect the regularity of texture changes throughout lattices because the textures for every 𝑡 lattices are the same. Such regularity 𝑡 ′ can be easily detected through one dimensional Fourier transformation. The median of 𝑡 ′s associated with all lattices should closely approximate 𝑡. Attention should be paid to the case where 𝑡 = 1, i.e., the textures of all lattices are the same. In this case, the predicted periods of row and column may differ erroneously, and the inconsistent row and column periods suggest the true value of 𝑡 is 1.

3.2.2 Lattice Template Suppose there are training samples 𝑰1 , 𝑰2 … 𝑰𝑁 , each 𝑰𝑖 (1 ≤ 𝑖 ≤ 𝑁) can be segmented to lattices categorized to 𝑡 different classes based on the lattice textures. The process of inferring templates comprises three main steps conceptually, i.e., 1) inferring representative lattices of each training sample, 2) sorting the representative lattices and 3) generating templates. All three steps need to estimate the similarity of textures enclosed by lattices; the similarity can be reflected by comparing either the feature vectors extracted from lattices through some feature extraction 9

ACCEPTED MANUSCRIPT

methods or the distances directly computed based on the underlying pixels of lattices. However, feature extraction methods are commonly computationally expansive, while the pixel-based distance computation is much faster. In this paper, six metrics are chosen to estimate the lattice distances, i.e., correlation coefficient, cosine similarity, Chebychev distance, Euclidean distance, MSE (Mean-squared error) and Spearman’s rank correlation coefficient. These metrics then serve as the combinable modules in the proposed method.

Step 1 Inferring Representative Lattices of Each Sample

AN US

CR IP T

If the lattice period is 𝑡, there should be 𝑡 different classes of the fabric textures enclosed by the lattices segmented through lattice segmentation. When 𝑡 > 1, the fabric texture periodically repeats every 𝑡 lattices, e.g., the middle of Fig. 6 illustrates the case 𝑡 = 3. Hence, to infer representatives for all 𝑡 different classes for a given image 𝑰𝑖 , the lattices of the same class should be retrieved accordingly. The retrieving process is illustrated in Fig. 6, i.e., for a specific class of the fabric texture, the column indices of every 𝑡 lattices in every 𝑡 rows are the same. Hence, retrieving the lattices of the same column indices from every 𝑡 rows and arranging them row by row, a lattice matrix can be formed. For a specific class, there are at most 𝑡 such matrices, i.e., 𝑴1 , 𝑴2 …𝑴𝑡 . For instance, there are three matrices of class 3, i.e., the matrices formed by lattices at row 1 and row 4 (shown on the left of Fig. 6), at row 2 and row 5 (shown on the right of Fig. 6), and at the row 3 (not shown). Suppose the lattice matrices of the 𝑘th (1 ≤ 𝑘 ≤ 𝑡) class of 𝐼𝑖 have been obtained, then a representative lattice may be inferred to represent the 𝑘th class based on a specific metric. Specifically, for each lattice matrix and the 𝑗th metric 𝑇𝑗 among |𝑇| available metrics, a lattice most similar to all other lattices of the matrix can be found. These lattices may serve as the candidates for inferring the representative of the 𝑘th class based on 𝐼𝑖 .

3

1

2

3

1

2

3

1

2

3

extract lattices at row 2 and 5

2

3

1

2

3

1

2

3

3

3

1

2

3

1

2

3

1

3

3

3

3

1

2

3

1

2

3

CE

extract lattices at row 1 and 4

1

M

3

3

3

ED

3

2

PT

3

1

AC

Fig. 6 Extracting lattices of the same class. However, because the fabric image may be slightly deformed, the local deformation of a specific lattice may not be correctly captured in the lattice segmentation. Consequently, the local deformation may bias the distance measured based on metric 𝑻𝑗 , hence some compensation for the local deformation should be made before estimating the distance between lattices 𝑳 and 𝑳′, i.e., the following equation has to be solved for finding the deformation-compensated version of 𝑳. 𝑳~ = arg min 𝑻𝑗 (𝑓(𝑳), 𝑳′ ), 𝑓(𝑳)

(1)

where 𝑓(𝑳) represents the compensation made for lattice 𝑳, and 𝑻𝑗 (𝑳, 𝑳′ ) denotes the distance between 𝑳 and 𝑳′ measured based on metric 𝑻𝑗 . Ideally, 𝑓 may consist of a series of affine transformation, contrast adjustment and so on. In practice, 𝑓 is restricted to the combination of the simple translations applied to the frame of 𝑳 in 𝑰𝑖 , i.e., moving the frame in the orthogonal directions and generating a compensated version 𝑓(𝑳) by retrieving the pixels enclosed by the 10

ACCEPTED MANUSCRIPT frame. The moving frame is restricted around 𝑳 by requiring the resulting 𝑓(𝑳) should shares at least half of pixels with 𝑳. However, because the correct movements leading to 𝑳~ is unknown prior, 𝑻𝑗 (𝑓(𝑳), 𝑳′ ) is estimated at each movement and the specific series of movements leading

AN US

CR IP T

to the least metric are accepted as the correct movements. This process is illustrated in Fig. 7. As shown in Fig. 7, assuming 𝑳 and 𝑳′ are lattices of dotted and solid frames respectively and 𝑳 is selected to be compensated while keeping 𝑳′ fixed, then the distance between the original 𝑳 and 𝑳′ is estimated to serve as a distance reference. Next, the lattice frame of 𝑳 is moved orthogonally, i.e., upwards, downwards, leftwards and rightwards as shown in Fig. 7. Each movement leads to a compensated version of 𝑳, i.e., the region enclosed by the moved lattice frame. Thus, four orthogonal directions result in four compensated 𝑳s. Then the distance between the compensated 𝑳s and 𝑳′ is estimated and compared with the distance reference. If there are any distances less than the reference, then the correct movement is the one leading to the minimal distance among these distances. The distance reference is then replaced by the minimal distance, and the movement continues at 𝑳 compensated only by the correct movements. This process continues until no movement leading to distances smaller than the distance reference. Finally, the version of 𝑳 compensated by the sequential correct movements serves as 𝑳~ corresponding to formula (1). Hereinafter, the distance between a lattice 𝑳 and a reference lattice 𝑳′ refers to 𝑻𝑗 (𝑳~ , 𝑳′ ). If the movement step is one pixel, this process is computationally expansive. In

move upwards Compensated Lattice

PT

Original Lattice

ED

M

practice, the computational costs may be reduced by adopting the correct movements of previous lattices instead of estimating movements for each lattice from scratch. For example, for lattices in the same row, the correct movements are estimated at the first lattice in the row, then the estimated movements are directly applied to the next lattice and finally the aforementioned process starts. This may save computation because the deformation of the fabric surface may be global, i.e., the correct movements of neighboring lattices are roughly the same.

AC

CE

move downwards

move leftwards

move rightwards

move lattice frame

find the compensated lattice of the least distance

Fig. 7 Template matching. For the 𝑘th class based on 𝐼𝑖 , the lattice matrices 𝑴1 , 𝑴2 …𝑴𝑡 can be retrieved by means illustrated in Fig. 6. For each matrix 𝑴ℓ , a lattice can be generated based on the 𝑗th metric 𝑻𝑗 for generating the representative of the 𝑘th class of 𝐼𝑖 . This lattice is called candidate which is the average of the lattices of the minimal distances to all other lattices in rows or columns of 𝑴ℓ . Suppose 𝑳ℓ denotes the candidate of 𝑴ℓ , then the representative lattice 𝑳𝑖,𝑗,𝑘 of the 𝑘th class of 𝑰𝑖 based on 𝑻𝑗 is defined as 11

ACCEPTED MANUSCRIPT

𝑳𝑖,𝑗,𝑘 = ∑ ℓ

𝑳ℓ . 𝑡

(5)

As illustrated in Fig. 8, for a given fabric image, the lattices of the same texture class are retrieved and arranged to construct the matrix 𝑴ℓ s, and then a lattice is chosen for each row or column in 𝑴ℓ based on its distance to all other lattices in the same row or column. The chosen lattices of 𝑴ℓ are then averaged to yield the candidate of 𝑴ℓ . All candidates associated with the 𝑘th class are then averaged to generate the representative 𝑳𝑖,𝑗,𝑘 of the class based on the 𝐼𝑖 and metric 𝑻𝑗 .

CR IP T

candidates

𝑳𝑖,𝑗 ,𝑡

𝑴ℓ of the 𝑘th class

AN US

segmented lattices

retrieve 𝑴1 , 𝑴2 … 𝑴𝑡 for the 𝑘th class find lattice of the minimal distance in a row or a column of 𝑴ℓ

M

average candidates to yield representative lattice 𝑳𝑖,𝑗 ,𝑘

Fig. 8 Generating representative lattices.

Step 2 Sort Feature Statistics

AC

CE

PT

ED

Suppose each 𝑰𝑖 of 𝑰1 , 𝑰2 … 𝑰𝑁 is segmented to lattices of 𝑡 classes, there is a possibility that the arrangement of the lattices from each sample may not be consistent based on the class order, e.g., if there are four training samples 𝑰1 , 𝑰2 , 𝑰3 and 𝑰4 segmented to lattices of two classes as shown in Fig. 9, the lattices of 𝑰1 , 𝑰2 and 𝑰4 share the class order 1-2 while 𝑰3 has a class order 2-1. However, the representative lattices are arranged according to the local class order by default. Therefore, the representative lattices should be sorted based on a consistent class order.

1

2

1

2

2

1

1

2

2

1

2

1

1

2

2

1

Fig. 9 Lattice class order. If the samples are in the same order, then the sorting is unnecessary and may lead to chaos. Thus, the sum of the mutual distances between two means of adjacent indices is computed before and after the sorting. If the orders of samples are different indeed, then the distance sum should decrease; otherwise, the sorting should be cancelled. The distance sum is computed based on

12

ACCEPTED MANUSCRIPT 𝑁−1 ‖𝑻𝑗 (𝑳𝑖,𝑗,1 , 𝑳𝑖+1,𝑗,1 )‖

2 , (6) 𝑁 − 1 𝑖=1 where 𝑁 denotes the number of training samples; notice 𝑘 of 𝑳𝑖,𝑗,𝑘 defined by (5) is fixed to 1,

𝑑𝑗 = ∑

i.e., distances are measured w. r. t. a fixed class. For the 𝑘th class and the 𝑗th metric, there is a representative lattice 𝑳𝑖 ∗ ,𝑗,𝑘 most similar to 𝑳1,𝑗,𝑘 , 𝑳2,𝑗,𝑘 … 𝑳𝑁,𝑗,𝑘 estimated based on 𝑰1 , 𝑰2 … 𝑰𝑁 . The sample index 𝑖 ∗ of 𝑳𝑖 ∗ ,𝑗,𝑘 is defined as follows. 𝑁−1 ∗

𝑖 = arg min ∑ 𝑻𝑗 (𝑳𝑖,𝑗,𝑘 , 𝑳𝑖 ′ ,𝑗,𝑘 ) . 1≤𝑖≤𝑁

(7)

𝑖 ′ =1

CR IP T

The lattice 𝑳𝑖 ∗ ,𝑗,𝑘 serves as the reference based on 𝑻𝑗 . For a specific class of index 𝑘 ′, each 𝑳𝑖,𝑗,𝑘 ′ of 𝑳1,𝑗,𝑘 ′ , 𝑳2,𝑗,𝑘 ′ … 𝑳𝑁,𝑗,𝑘 ′ is compared with all references 𝑳𝑖 ∗ ,𝑗,𝑘 s w. r. t. 𝑻𝑗 , and the most similar reference leads to a class index 𝑘 ∗ indicating 𝑳𝑖,𝑗,𝑘 ′ is consistent with class 𝑘 ∗. 𝑘 ∗ is defined as below.

𝑘 ∗ = arg min 𝑻𝑗 (𝑳𝑖,𝑗,𝑘 ′ , 𝑳𝑖 ∗ ,𝑗,𝑘 ) 1≤𝑘≤𝑡

(8)

AN US

If 𝑘 ∗ ≠ 𝑘 ′ which means the original class 𝑘 ′ of 𝑳𝑖,𝑗,𝑘 ′ is inconsistent with class 𝑘 ∗, then 𝑳𝑖,𝑗,𝑘 ′ and 𝑳𝑖,𝑗,𝑘 ∗ are exchanged. This comparison repeats for all samples and all classes to complete sorting. Fig. 10 illustrates the sorting procedure of a training set in which 𝑰3 is inconsistent. The sorting begins by finding references 𝑳𝑖 ∗ ,𝑗,𝑘 s highlighted by solid frames, and then 𝑘 ∗ is computed for 𝑳𝑖,𝑗,1 s based on 𝑳𝑖 ∗ ,𝑗,𝑘 s. The 𝑘 ∗ s of 𝑳𝑖,𝑗,1 s except 𝑳3,𝑗,1 of 𝑰3 all agree with 𝑘 = 1, and thus 𝑳3,𝑗,1 and 𝑳3,𝑗,2 are exchanged. This repeats for class 𝑘 = 2, and all

M

ED

references

find references for all classes

find

class 2 class 1

AC

CE

PT

class 2 class 1

𝑘 ∗s agree with 𝑘 = 2 after the exchanging. Hence, no exchanging is conducted, which terminates the sorting.

interchange

and

Fig. 10 Sorting representative lattices.

Step 3 Generate Lattice Templates For the 𝑘th class, there are sorted representatives 𝑳1,𝑗,𝑘 , 𝑳2,𝑗,𝑘 … 𝑳𝑁,𝑗,𝑘 generated based on the training samples 𝑰1 , 𝑰2 … 𝑰𝑁 w. r. t. the 𝑗th metric 𝑻𝑗 . To infer the template of the 𝑘th class based on 𝑳𝑖,𝑗,𝑘 s, 𝑳𝑖,𝑗,𝑘 s may be simply averaged to yield a single template, however, this result may be distorted by the pathological data. Meanwhile, a single template may not fully 13

ACCEPTED MANUSCRIPT reflect the variations of the training data. Hence, the 𝑘th class should be modeled by multiple templates,. This modeling procedure is a typical unsupervised learning which can be approximately solved through the clustering method like K-means, and the pathological data may be detected by analyzing the statistics of the clustering results. Therefore, a voting process based on the cluster labels of the repetitive K-means is developed, i.e., the priorities of 𝑳𝑖,𝑗,𝑘 s are reflected by the weights generated according to the statistical analysis of their labels, and the weighted 𝑳𝑖,𝑗,𝑘 s contained by the clusters are adopted to infer the templates. A representative 𝑳𝑖,𝑗,𝑘 can be reshaped to a vector 𝒗𝑖,𝑗,𝑘 by sequentially concatenating its rows. All 𝒗𝑖,𝑗,𝑘 s of the 𝑘th class can be grouped through adaptive K-means algorithm [18] and

CR IP T

the grouping leads to the clusters identified by labels 1, 2 … |𝑮| where 𝑮 denotes the label set, and |𝑮| is a hyper parameter called cluster number limit. Each 𝒗𝑖,𝑗,𝑘 belongs to one of the clusters and is marked by the corresponding cluster label 𝑔𝑖,𝑗,𝑘 ∈ 𝑮. If the grouping is repetitively conducted on the representatives through K-means of K fixed to |𝑮|, and the labels are somewhat maintained to be consistent throughout all repetitions, i.e., the cluster centers of the same label in different repetitions are very close, then the weights can be generated by analyzing these labels. The consistency of the labels throughout ℛ repetitions is guaranteed if labels of any two

AN US

repetitions are consistent, i.e., if 𝑮(𝑟) denotes the label set of the 𝑟th repetition, 𝑐 (𝑟) and 𝑐 (𝑟+1) respectively denote the cluster centers corresponding to the clusters of label 𝑔(𝑟) ∈ 𝑮(𝑟) and 𝑔(𝑟+1) ∈ 𝑮(𝑟+1) where 𝑔(𝑟) = 𝑔(𝑟+1) , and then 𝑐 (𝑟) and 𝑐 (𝑟+1) meet the following condition. 𝑐 (𝑟) = arg min 𝑻𝑗 (𝑐 (𝑟+1) , 𝒞), 𝒞∈𝓒(𝑟)

(9)

M

where 𝓒(𝑟) denotes the cluster centers of the 𝑟th repetition, and a cluster center 𝑐 corresponding to the cluster identified by a label 𝑔 is defined as the average of the 𝒗𝑖,𝑗,𝑘 s whose labels 𝑔𝑖,𝑗,𝑘 s

ED

agree with 𝑔 ∈ 𝑮. Assuming the consistency of labels is maintained throughout all repetitions according to (9), then for the 𝑘th class and the 𝑗th metric, the representative 𝑳𝑖,𝑗,𝑘 of the 𝑖th (1)

(2)

(ℛ)

(𝑟)

sample owns labels 𝑔𝑖,𝑗,𝑘 , 𝑔𝑖,𝑗,𝑘 …𝑔𝑖,𝑗,𝑘 given by ℛ times of repetitive clustering, where 𝑔𝑖,𝑗,𝑘

PT

denotes the label of 𝑳𝑖,𝑗,𝑘 in the 𝑟th repetition. The median of labels owned by 𝑳𝑖,𝑗,𝑘 is defined as the benchmark 𝑏𝑖,𝑗,𝑘 ∈ 𝑮. Thus, if 𝑏𝑖,𝑗,𝑘 = 𝑏𝑖 ′ ,𝑗,𝑘 holds for 𝑖 ≠ 𝑖 ′, then 𝑳𝑖,𝑗,𝑘 and 𝑳𝑖 ′ ,𝑗,𝑘 is considered similar and should be adopted together for generating a template. For a specific 𝑳𝑖,𝑗,𝑘

AC

CE

similar to 𝑳𝑖 ′ ,𝑗,𝑘 s where 𝑖, 𝑖 ′ ∈ 𝓘 = {𝑖|arg𝑖 (𝑏𝑖,𝑗,𝑘 = 𝑔)} and 𝑔 is a constant label in 𝑮, its weight is defined as 𝑤𝑖,𝑗,𝑘 = (1)

max (𝑒𝑖 ′ ,𝑗,𝑘 ) − 𝑒𝑖,𝑗,𝑘 ′ 𝑖 ∈𝓘

|∑𝑖 ′ ∈𝓘(𝑒𝑖 ′ ,𝑗,𝑘 )|

(2)

,

(10)

(ℛ)

(12)

(ℛ)

where 𝑒𝑖,𝑗,𝑘 denotes the entropy of {𝑔𝑖,𝑗,𝑘 𝑔𝑖,𝑗,𝑘 …𝑔𝑖,𝑗,𝑘 }, i.e., (1)

(2)

𝑒𝑖,𝑗,𝑘 = entropy (𝑔𝑖,𝑗,𝑘 , 𝑔𝑖,𝑗,𝑘 … 𝑔𝑖,𝑗,𝑘 ).

Basically, 𝑤𝑖,𝑗,𝑘 reflects the normalized ―purity‖ of the labels associated with 𝑳𝑖,𝑗,𝑘 throughout all repetitions. Finally, the template associated with a label 𝑔 is defined as follows. ∗ 𝑳𝑗,𝑘,𝑔 = ∑ δ(𝑏𝑖,𝑗,𝑘 − 𝑔) ∙ 𝑤𝑖,𝑗,𝑘 𝑳𝑖,𝑗,𝑘 ,

(13)

𝑖

where δ(∙) is Dirac delta function. Therefore, the 𝑘th class w. r. t. 𝑻𝑗 is modeled by templates 14

ACCEPTED MANUSCRIPT ∗ ∗ ∗ 𝑳𝑗,𝑘,1 , 𝑳𝑗,𝑘,2 …𝑳𝑗,𝑘,|𝑮| . For example, in Fig. 11, the representatives 𝑳1,𝑗,𝑘 , 𝑳2,𝑗,𝑘 … 𝑳6,𝑗,𝑘 are

grouped to 2 (|𝑮| = 2) clusters for 3 times (ℛ = 3) for the 𝑘th class w. r. t. 𝑻𝑗 , i.e., the index 𝑗 and 𝑘 are both fixed. 𝑳3,𝑗,𝑘 and 𝑳4,𝑗,𝑘 contain pathological data reflected by the inconsistent labels, while that the rest 𝑳𝑖,𝑗,𝑘 s have consistent labels indicates their data can be safely adopted for generating templates. Although 𝑳3,𝑗,𝑘 and 𝑳4,𝑗,𝑘 may still be taken into account, their weights are much lower than the rest 𝑳𝑖,𝑗,𝑘 s, which is indicated by the dotted arrows.

#1 #1 #1

#2 #1 #2

#1 #1 #2

#2 #2 #2

#2 #2 #2

CR IP T

#1 #1 #1

Fig. 11 Inferring templates based on learnt weights.

AN US

3.2.3 Learn Thresholds

∗ Suppose the lattice templates 𝑳𝑗,𝑘,𝑔 s are obtained based on the aforementioned learning

involving training samples 𝑰1 , 𝑰2 … 𝑰𝑁 , then for each lattice 𝑳 from the 𝑖th training sample 𝑰𝑖 ∗ there is a template 𝑳𝑗,𝑘 ∗ ,𝑔∗ most similar to 𝑳 among all templates based on the 𝑗th metric 𝑻𝑗 , i.e., ∗ ~ ′ 𝑳𝑗,𝑘 ∗ ,𝑔∗ = arg min (𝑻𝑗 (𝑳 , 𝑳 )),

(14)

𝑳′ ∈𝑪∗

M

where 𝑳~ denotes the compensated version of 𝑳 defined by the formula (1), 𝑪∗ represents the ∗ set of 𝑳𝑗,𝑘,𝑔 s, 𝑘 ∗ and 𝑔∗ respectively denote the class index and the template index of the

PT

ED

∗ ~ ∗ template most similar to 𝑳. The distance between 𝑳~ and 𝑳𝑗,𝑘 ∗ ,𝑔∗ , i.e., 𝑻𝑗 (𝑳 , 𝑳𝑗,𝑘 ∗ ,𝑔∗ ), is called ∗ least distance. In most cases, a template 𝑳𝑗,𝑘,𝑔 may be the most similar template shared by ∗ several lattices, and it is also possible that 𝑳𝑗,𝑘,𝑔 may not serve for any lattices. Thus, there will ∗ be either a series of the least distances or none related with 𝑳𝑗,𝑘,𝑔 . According to the least distances ∗ related with 𝑳𝑗,𝑘,𝑔 , the lower and upper bounds of the least distances may serve as the metric ∗ thresholds associated with 𝑳𝑗,𝑘,𝑔 . However, the pathological data in the training samples should be

CE

avoided for estimating thresholds. To avoid the pathological data, the statistics of the least distances should be probed and only the ones of stable statistics are taken into account, i.e., (𝑖)

∗ suppose 𝑫𝑗,𝑘,𝑔 denotes the set of the least distances involving both a specific template 𝑳𝑗,𝑘,𝑔 and

AC

∗ the lattices in 𝑰𝑖 most similar to 𝑳𝑗,𝑘,𝑔 , then the set 𝓓𝑗,𝑘,𝑔 of stable distances based on all

samples is defined as below. (𝑖)

(𝑖)

(1)

(𝑁)

𝓓𝑗,𝑘,𝑔 = {𝑫𝑗,𝑘,𝑔 |std (𝑫𝑗,𝑘,𝑔 ) < 𝛾 ∙ med (std (𝑫𝑗,𝑘,𝑔 ) … std (𝑫𝑗,𝑘,𝑔 ))},

(15)

where med(∙) and std(∙) respectively represent finding the median and standard deviation, and 𝛾 ≥ 1 denotes a hyper parameter named standard deviation factor. According to 𝓓𝑗,𝑘,𝑔 , the ∗ ∗ ∗∗ metric thresholds of 𝑳𝑗,𝑘,𝑔 are jointly defined by the minimum 𝑑𝑗,𝑘,𝑔 and maximum 𝑑𝑗,𝑘,𝑔 of the distances in 𝓓𝑗,𝑘,𝑔 , i.e., ∗ 𝑑𝑗,𝑘,𝑔 = min 𝓓𝑗,𝑘,𝑔 { ∗∗ . (16) 𝑑𝑗,𝑘,𝑔 = max 𝓓𝑗,𝑘,𝑔 15

ACCEPTED MANUSCRIPT

The whole process of finding metric thresholds is illustrated in Fig. 12. For a given lattice as ∗ shown in the figure, 𝑳𝑗,𝑘 ∗ ,𝑔∗ is found among all templates. For each lattice, the class index map ∗ shown in Fig. 12 depicts the distance to 𝑳𝑗,𝑘 ∗ ,𝑔∗ , i.e., the color denotes the distance, and the first and the second digits respectively indicate 𝑘 ∗ and 𝑔∗. After the distance estimation, the template not related with any lattices is eliminated, and the metric thresholds are evaluated for the survived templates. Templates

𝑳∗|𝑻|,2,1 𝑳∗|𝑻|,2,2

𝑳∗|𝑻|,2,1 𝑳∗|𝑻|,2,2

∗ ∗∗ 𝑑|𝑻|,2,2 ; 𝑑|𝑻|,1,2

CR IP T

Segmented Lattices

Class Index Map

∗ ∗∗ 𝑑|𝑻|,2,1 ; 𝑑|𝑻|,1,1

...

...

∗ ∗ 𝑳1,2,1 𝑳1,2,2

AN US

∗ ∗ 𝑳1,2,2 compare with 𝑳1,2,1 ∗ 𝑳𝑗 ,𝑘,𝑔 each 111111

∗ ∗ ∗ 𝑳1,1,2 𝑳1,1,3 𝑳1,1,1 ∗

find 𝑳𝑗 ,𝑘 ∗,𝑔 ∗ leading to the minimal distance

∗ ∗ 𝑳1,1,2 𝑳1,1,1

∗ eliminate templates (𝑳1111in this 1,1,3 example) not related with any lattices

∗ ∗∗ 𝑑|𝑻|,1,2 ; 𝑑|𝑻|,1,2 ∗ ∗∗ 𝑑|𝑻|,1,1 ; 𝑑|𝑻|,1,1

...

𝑳∗|𝑻|,1,1 𝑳∗|𝑻|,1,2

𝑳∗|𝑻|,1,1 𝑳∗|𝑻|,1,2

∗ 𝑑1,2,2 ; ∗ 𝑑1,2,1 ; ∗ 𝑑1,1,2 ; ∗ 𝑑1,1,1 ;

∗∗ 𝑑1,1,2 ∗∗ 𝑑1,1,1 ∗∗ 𝑑1,1,2 ∗∗ 𝑑1,1,1

∗

𝑑𝑗 ,𝑘,𝑔 find 1111 𝑑𝑗∗∗,𝑘,𝑔 and 1111 related 𝑳𝑗∗,𝑘,𝑔 with1111

M

Fig. 12 Finding metric thresholds. Assuming the period (class number) 𝑡 > 1, then the class index may also serve for ∗ estimating period thresholds. Because every lattice 𝑳 of 𝑰𝑖 is attached to a template 𝑳𝑗,𝑘,𝑔

ED

through the formula (14), there is an unique map between 𝑳 and the class index 𝑘 specified by ∗ 𝑳𝑗,𝑘,𝑔 . If 𝑘s are arranged according to the position of 𝑳s in 𝑰𝑖 as shown in Fig. 12, then the

AC

CE

PT

differences of the indices in class index map 𝑲𝑖 can be adopted for inspecting defects because the indices of defective lattices may violate the period, e.g., the original class index map in the middle of Fig. 13 illustrates this case in which the defective lattice indicated by the white cross is mapped to class index 2 instead of the correct index 3. Such violation can be easily detected by subtracting the synchronized indices in 𝑲𝑖 as illustrated in Fig. 13. There are two kinds of synchronizations, i.e., the row and column synchronizations respectively depicted by the left and the right of Fig. 13. For the row synchronization, the columns of 𝑲𝑖 are moved upwards to synchronize the indices of rows. The movement leaves blanks indicated by dotted frames in Fig. 13. The blanks are then filled according to the period pattern. Subtracting the neighboring indices of a row only containing defect-free lattices, the result will be zero, while any none-zero result indicates the existence of defective lattices. The column synchronization works similarly. Suppose a lattice 𝑳 corresponds to a period index of (𝑟, c) in 𝑲𝑖 , then its period difference 𝑑𝑳 is defined as the sum of differences involving the neighboring indices in the 𝑟th row of the row-synchronized version of 𝑲𝑖 and the neighboring indices in the 𝑐th column of the column-synchronized version of 𝑲𝑖 . In practice, if any one of the two kinds of differences is zero, then the period difference is set to zero. For each 𝑲𝑖 based on the 𝑗th metric, there is a maximum period difference. The median of all maximums associated with 𝑲𝑖 s is defined as the period threshold 𝑑𝑗∗.

16

ACCEPTED MANUSCRIPT

synchronized class index map

original class index map

synchronized class index map

2

2

2

2

2

2

2

1

2

3

1

2

3

1

2

3

1

2

3

1

2

1

1

1

1

1

1

1

3

1

2

3

1

2

3

2

3

1

2

3

1

2

3

3

3

3

2

3

3

2

3

1

2

2

1

2

2

3

1

2

2

1

2

2

2

2

2

2

2

2

1

2

3

1

2

3

1

2

3

1

2

3

1

2

1

1

1

1

1

1

1

3

1

2

3

1

2

3

2

3

1

2

3

1

2

synchronize columns of class index map

synchronize rows of class index map

CR IP T

Fig. 13 Computing period differences

3.2.4 Defect Inspection

AN US

Suppose the templates and thresholds have been obtained from the aforementioned steps, then for lattices segmented from a given 𝑰, both the similarities between lattices and the templates and the resulting class indices are estimated to compare with the two types of thresholds learnt from training samples, i.e., the metric and the period thresholds. The lattice which violates the thresholds is identified as defective. For applying the metric thresholds to a lattice 𝑳 in 𝑰, the ∗ most similar template 𝑳𝑗,𝑘 ∗ ,𝑔∗ defined by formula (14) w. r. t. the metric 𝑻𝑗 for 𝑳 is estimated, ∗ ~ and the distance 𝑻𝑗 (𝑳~ , 𝑳𝑗,𝑘 ∗ ,𝑔∗ ) involving 𝑳 defined by formula (1) can be computed and ∗ ∗∗ adopted for the comparison with metric thresholds 𝑑𝑗,𝑘 ∗ ,𝑔∗ and 𝑑𝑗,𝑘 ∗ ,𝑔∗ defined by formula (16). ∗ The class index 𝑘 ∗ specified by 𝑳𝑗,𝑘 ∗ ,𝑔∗ leads to the class index map 𝑲, and the period difference 𝑑𝑳 can thus be estimated and compared with period threshold 𝑑𝑗∗ .

M

∗ ~ ∗ On the basis of 𝑻𝑗 (𝑳~ , 𝑳𝑗,𝑘 ∗ ,𝑔∗ ) and 𝑑𝑳 , 𝑳 is identified as defect-free if 𝑻𝑗 (𝑳 , 𝑳𝑗,𝑘 ∗ ,𝑔∗ ) ∗ ∗∗ lies in the range restricted by 𝑑𝑗,𝑘 ∗ ,𝑔∗ and 𝑑𝑗,𝑘 ∗ ,𝑔∗ , and meanwhile 𝑑𝑳 does not exceed the period threshold 𝑑𝑗∗, i.e., the set 𝑪𝑗 containing defect-free lattices w. r. t. the metric 𝑻𝑗 is defined as

ED

follows.

∗ ∗∗ ∗ ~ ∗ 𝑪𝑗 = arg(𝑑𝑗,𝑘 ∗ ,𝑔∗ ≤ 𝑻𝑗 (𝑳 , 𝑳𝑗,𝑘 ∗ ,𝑔∗ ) ≤ 𝑑𝑗,𝑘 ∗ ,𝑔∗ ) ∩ arg(𝑑𝑳 ≤ 𝑑𝑗 ), 𝑳∈𝑰

𝑳∈𝑰

(19)

PT

Lattice 𝑳 ∉ 𝑪𝑗 is identified as defective. All 𝑪𝑗 s lead to the set of defective lattices 𝑪× .

4. Performance Evaluation

AC

CE

The performance evaluation of the proposed method (LSLT) essentially involves two fabric image databases providing pixel- and image-level evaluations respectively. Pixel-level evaluation is conducted based on Fabric Image Database (FID) provided by Industrial Automation Research Laboratory from Dept. of Electrical and Electronic Engineering of Hong Kong University. Image-level evaluation is implemented on Fabric Defect Detection Database (FDDD) [31][32][33] provided by Department of Computer Engineering of Bingol University. For further exploring the performance of LSLT in the case of degraded fabric images, the samples in FID are blurred and polluted by salt & pepper noises. The set of the degraded samples is called Degraded Fabric Image Database (DFID). Therefore, LSLT is tested on three databases: FID, DFID and FDDD. For each database, we sequentially repeated four types of experiments, i.e., typeⅠ evaluating effects of hyper parameters on lattice segmentation, type Ⅱchoosing distance metrics, type Ⅲ determining the optimal values of hyper parameters and type Ⅳ evaluating the performance of LSLT parameterized by the optimal values. Experiments of typeⅠinvolve hyper parameters area factor (0.4 by default) and tolerance 17

ACCEPTED MANUSCRIPT

AC

CE

PT

ED

M

AN US

CR IP T

(0.1 by default) in lattice segmentation. Because there’s no standard to measure how well lattices are segmented based on different values of hyper parameters, we defined a measurement based on Kullback–Leibler divergence which reflects the differences between lattice size distributions determined by the default and none-default parameter values. The benchmark for computing the divergences is chosen as the size distribution of the lattices from the defect-free samples based on the default values, because the default values result in a known working solution leading to high accuracy. The distributions of none-default values are computed based on lattices segmented from the defective samples. For the experiments of the aforementioned types except typeⅠ, area factor and tolerance are fixed to their default values. Experiments of typeⅡ involves six distance metrics. Although combining more metrics may lead to high accuracy, too many metrics lead to heavy computational burden. Thus, we explored the performances of LSLT parameterized by one, two and three metrics while other hyper parameters are fixed to default values. The performance is visualized by Receiver Operating Characteristic (ROC) curves and qualified by Area Under Curve (AUC). If AUC involving combinations of a few metrics approximates AUC involving combinations of many metrics, then the former is preferred. For each database, once the optimal metrics are determined, they are constantly employed throughout the following experiments. Experiments of type Ⅲ involves hyper parameters: cluster number limit (5 by default) and standard deviation factor (1.5 by default). For a value of one parameter, the experiment is repeated twelve times while the other parameter is fixed to its default value. For each repetition, the measurements named Accuracy (ACC), True Positive Rate (TPR) and False Positive Rate (FPR) [23] are computed. For each measurement, the mean and standard deviation are estimated based on the results of repetitions. The means and standard deviations of values w. r. t. a hyper parameter are then visualized as error bars. The values stably leading to high TPR and low FPR are selected as the optimal values. For each database, once the optimal values are determined, they are constantly employed throughout the following experiments. Experiments of type Ⅳ illustrate the performance and runtime of LSLT parameterized by optimized hyper parameters. Besides LSLT, we also evaluated four other methods LSG, RB, ER and WGIS for comparison. For each method and a specific database, the experiment is repeated twelve times, and the detection results and runtime are collected for measuring its performance. The detection results are visualized as ROC curves and runtime is visualized by box plots. The settings of four methods for comparison are the same as [17]. All experiments are performed on a tower server with two Intel Xeon E5-2665 processors and 32.00 GB of memory. The software includes Windows 10 and Matlab 8.4. The experiments results and related analyses are introduced individually w. r. t. each database.

4.1 Results Based on FID FID consists of 247 images of size 256-by-256. Among these images, 166 images are in 24-bit depth and the rest are binary images. All images are categorized to 3 fabric patterns, i.e., dot pattern (30 defect-free, 30 defective and 30 ground-truth images), star pattern (25 defect-free, 25 defective and 25 ground-truth images) and box pattern (30 defect-free, 26 defective and 26 ground-truth images). For each defective image, there is a binary ground-truth image whose foreground pixels denote the defects for measuring the detection rate in pixel level. Results of experiments of typeⅠ are shown in Fig. 14 and Fig. 15. For area factor, the divergences of box and dot patterns approach zero when area factor exceeds 0.374. On the other 18

ACCEPTED MANUSCRIPT

CR IP T

hand, the divergence of star pattern is large except points around the default value, which implies the segmented lattices are different from the lattices of default value. For tolerance, divergences of star and box patterns approximate zero when tolerance is larger than 0.1, however, divergence of dot pattern is large except the range from 0.1 to 0.17. In short, dot and box patterns are insensitive to the changes of area factor; star and box pattern are insensitive to the changes of tolerance.

AN US

Fig. 14 Divergence of area factor based on FID.

AC

CE

PT

ED

M

Fig. 15 Divergence of tolerance based on FID.

Fig. 16 ROC curves based on box-patterned images of FID. 19

ACCEPTED MANUSCRIPT

PT

ED

M

AN US

CR IP T

Results of experiments of typeⅡ are illustrated from Fig. 16 to Fig. 18. For these figures, colors of curves correspond to color bar in which colors from top to bottom correspond to AUCs decreasingly. The text legend in each subfigure lists the method names according to their AUCs decreasingly. For box pattern, correlation and MSE are metrics of the optimal AUCs throughout all subfigures in Fig. 16. For dot pattern, the optimal AUCs are also related with correlation and MSE. For star pattern, the optimal AUCs are associated with correlation and cosine. To a great extent, the optimal metrics are concluded to be correlation and MSE.

AC

CE

Fig. 17 ROC curves based on dot-patterned images of FID.

20

M

AN US

CR IP T

ACCEPTED MANUSCRIPT

AC

CE

PT

ED

Fig. 18 ROC curves based on star-patterned images of FID. Results of experiments of type Ⅲ are shown in Fig. 19 and Fig. 20. As shown in the figure, the performances related with box and star patterns are quite stable compared with dot pattern while both hyper parameters are varying. For dot pattern, its TPR increases as cluster number limit increases and it starts to become stable at value 13; its TPR decreases as standard deviation factor decreases, and the optimal TPR occurs at value 1.05. However, FPR is a little higher at this value than at the value 1.1. Hence, the optimal values for cluster number limit and standard deviation factor are concluded to be 13 and 1.1 respectively.

Fig. 19 Performances w. r. t. cluster number limit based on FID.

21

CR IP T

ACCEPTED MANUSCRIPT

AC

CE

PT

ED

M

AN US

Fig. 20 Performances w. r. t. standard deviation factor based on FID. Results of experiments of type Ⅳ are illustrated in Fig. 21 and Fig. 22. For box pattern, AUC of LSLT is higher than any other methods. AUC of WGIS is suboptimal. Although AUC of LSG is much lower than LSLT, it is higher than RB and ER. AUCs of RB and ER indicate they are incapable of identifying defects in box-patterned images. For star pattern, AUC of LSLT is also higher than any other methods. AUCs of LSG and WGIS are similar and higher than RB and ER. AUCs of RB and ER reflect their inefficiencies of identifying defects in star-patterned images. For dot pattern, AUCs of RB is higher than LSLT, and LSLT achieves the suboptimal AUC. AUC of WGIS is close to but lower than LSLT. AUCs of LSG and ER suggest the two methods are inefficient for dot pattern. Generally, the proposed LSLT achieves the optimal AUCs for star and box patterns, and its AUC is suboptimal for dot pattern.

Fig. 21 Performances w. r. t. patterned images of FID. 22

ACCEPTED MANUSCRIPT

Fig. 22 Runtime based on FID.

AC

CE

PT

ED

M

AN US

CR IP T

Runtime of all methods are illustrated by box plots in Fig. 22. The meaning of box plot is the same as [18]. The runtime of WGIS is optimal throughout all fabric patterns as shown in Fig. 22. The runtime of ER is suboptimal for all patterns, and runtime of WGIS and ER for individual samples approximate the medians. However, the runtime RB, LSG and LSLT varies a lot w. r. t. their medians. The runtime of most methods except LSLT are nearly consistent throughout all patterns, e.g., WGIS always leads to runtime close to 0 regardless of the fabric patterns. However, runtime of LSLT changes drastically from pattern to pattern. For box pattern, its median runtime approximates RB and runtime variance is very low. For star and dot pattern, its runtime is very high, especially for dot pattern which leads to lattices of alternatively-changed textures. This reflects that the runtime of LSLT is deeply affected by the number of different lattice textures and texture complexity.

4.2 Results Based on DFID DFID is generated by blurring and adding noises to samples of FID. For blurring, the point spread function (PSF) is defined to be 2D Gaussian function 𝑕(𝑠1 , 𝑠2 ) of varying sizes (𝑠1 , 𝑠2 ) and varying standard deviation 𝜎, i.e., 𝑕(𝑥, 𝑦) =

𝑒

−(𝑥2 +𝑦 2 ) 2 𝜎2

−((𝑥 ′ )2 +(𝑦 ′ )2 ) 2 𝜎2 ∑𝑥 ′ ∑𝑦 ′ 𝑒

,

where 𝑥, 𝑦 ∈ ℤ+, and −⌊𝑠1 /2⌋ ≤ 𝑥 ≤ ⌊𝑠1 /2⌋, −⌊𝑠2 /2⌋ ≤ 𝑦 ≤ ⌊𝑠2 /2⌋. The defective samples of the dataset are convolved with PSF to yield blurred images. For each defective sample, the size (𝑠1, 𝑠2 ) is randomly drew from the range (3, 3) to ideal lattice size, e.g., if ideal size is (17, 22), 23

ACCEPTED MANUSCRIPT

Blurred & noisy

Restored

Ground-truth

CE

PT

ED

M

AN US

Original

CR IP T

then 𝑠1 and 𝑠2 are randomly chosen from 3 to 17 and 3 to 22 respectively. The standard deviation 𝜎 ranges from 3 to 20, and for each defective sample, 𝜎 is randomly chosen from this range. The blurred defective samples are then polluted by salt & pepper noises of parameters randomly chosen from the range from 0 to 0.15. Ground-truth images are left unchanged. Fig. 23 illustrates some exemplars of the defective samples of dot, box and star patterns. For each defective sample, two blurred and noisy versions are generated accordingly. Hence, the fabric image database employed in experiments consists of 494 images of size 256-by-256. The images are categorized to 3 fabric patterns, i.e., dot pattern (60 defect-free, 60 defective and 60 ground-truth images), star pattern (50 defect-free, 50 defective and 50 ground-truth images) and box pattern (60 defect-free, 52 defective and 52 ground-truth images).

AC

Fig. 23 Polluted defective samples. For degraded images in this dataset, we add an additional preprocessing step. Namely, the training and testing procedures are the same as the original LSLT except that the input samples are preprocessed to remove the noises and the blurring. The preprocessing applies the blind deconvolution based on Biggs’s method to obtain the restored images as shown in the third column of Fig. 23, and then the restored images are processed by LSLT to identify the defects. Results of experiments of typeⅠare shown in Fig. 24 and Fig. 25. For area factor, divergences of star and box patterns approach zero when area factor exceeds 0.221, while divergence of dot pattern differs from zero for all values. For tolerance, the case is similar to area factor, i.e., star and box patterns are insensitive to the changes of tolerance when the tolerance is larger than 0.068, while divergence of dot pattern differs from zero. Generally, star and box patterns are insensitive to the changes of both parameters. 24

ACCEPTED MANUSCRIPT

CR IP T

Fig. 24 Divergence of area factor based on DFID.

AC

CE

PT

ED

M

AN US

Fig. 25 Divergence of tolerance based on DFID.

Fig. 26 ROC curves for restored box-patterned images of DFID. Results of experiments of typeⅡ are illustrated from Fig. 26 to Fig. 28. For box pattern shown in Fig. 26, the optimal AUCs of different numbers of metrics are associated with correlation, and the optimal AUCs of two or more metrics are not better than the optimal one of a single metric. For dot pattern shown in Fig. 27, the optimal AUCs are associated with MSE. For star pattern shown in Fig. 28, the situation is similar to the case of dot pattern. Therefore, 25

ACCEPTED MANUSCRIPT

M

AN US

CR IP T

correlation and MSE are necessary to achieve the optimal AUCs in most cases. Hence, the metrics of LSLT for degraded samples are concluded to be correlation and MSE.

AC

CE

PT

ED

Fig. 27 ROC curves for restored dot-patterned images of DFID.

26

AN US

CR IP T

ACCEPTED MANUSCRIPT

AC

CE

PT

ED

M

Fig. 28 ROC curves for restored star-patterned images of DFID. Results of experiments of type Ⅲ are shown in Fig. 29 and Fig. 30. For each measurement, its mean is denoted by the point decorated by a marker, and the vertical bar at each point reflects its standard deviation. As shown in Fig. 29, the performance associated with star pattern is independent of the parameter changes. FPR of dot pattern is so high that the prediction is no better than guessing. The TPR of dot pattern becomes stable at value 17. The minimal FPR of box pattern occurs at value 16, and the error bars are also very short at values 16 and 17. Therefore, the optimal value of cluster number limit is concluded to be 17. Fig. 30 depicts the results of standard deviation factor ranging from 1 to 3 and cluster number limit fixed to 5. The points and the error bars are obtained in the same way described above. As shown in Fig. 30, TPRs of star and box patterns become stable as the parameter increases, and meanwhile, TPR of dot pattern decreases. The highest TPRs of all patterns occur at value 1.05, however, FPRs at this value are relatively large and they drop at value 1.1. Hence, the optimal value of standard deviation factor is concluded to be 1.1.

27

ACCEPTED MANUSCRIPT

AN US

CR IP T

Fig. 29 Error bars of cluster number limit based on DFID.

AC

CE

PT

ED

M

Fig. 30 Error bars of standard deviation factor based on DFID. Results of experiments of type Ⅳ are illustrated in Fig. 31 and Fig. 32. For methods of comparison, the blurriness and noises in the degraded images are removed by using preprocessing the same as LSLT. As shown in the figure, LSLT achieves the optimal AUCs for both box patterns (0.88) and star patterns (0.87) and its AUC of dot pattern (0.69) is very close to suboptimal (0.71). Compared with the AUCs based on the original dataset shown in Fig. 21, AUCs of LSLT decrease by 1, 7 and 5 respectively for degraded images of box, dot and star pattern. AUCs of LSG also decrease for all patterns. AUCs of ER remain unchanged throughout all patterns. AUCs of WGIS and RB decrease by 4 for dot pattern only. RB is the only method whose AUCs increase for degraded images of box and star pattern. Generally, although the performance of LSLT decreases more or less when fabric images are blurred and noisy, its performance related with star and box patterns is still acceptable.

28

AN US

CR IP T

ACCEPTED MANUSCRIPT

AC

CE

PT

ED

M

Fig. 31 ROC curves based on DFID. Runtime is shown in Fig. 32. Compared with runtime of FID illustrated in Fig. 22, the runtime of LSLT increases. The medians of runtime increase roughly by 4 for box pattern and by 35 for dot pattern. Counter-intuitively, its runtime decreases roughly by 2 for star pattern. This may because the local contrast of restored image is higher than the original image as shown in Fig. 23. The difference between defect-free and defective textures may be larger than the original textures, which makes the inspection easier than before for star pattern. This is also true for box and dot patterns, i.e., the contrasts of the deblurred and denoised images are lower than the original versions as shown in Fig. 23, which makes inspection harder than before. Consequently, the runtime of dot and box patterns increase significantly.

29

AN US

CR IP T

ACCEPTED MANUSCRIPT

4.3 Results Based on FDDD

M

Fig. 32 Runtime based on DFID

AC

CE

PT

ED

FDDD consists of 3,002 defective images and 10,835 defect-free images. There’s no ground-truth image to label the defects in pixel level. Hence, the method performance can only be evaluated at image level. Some samples in FDDD violate the assumption stated in Introduction, and Fig. 33 illustrates some exemplars including uneven illumination, extreme low contrast, texture blobs not aligned with image axes and even no texture blobs. Samples like the ones shown in Fig. 33 are removed from FDDD. After removal, FDDD contains 1,136 defective images and 6,217 defect-free images.

Fig. 33 Inappropriate FDDD samples. Fig. 34 illustrates some samples randomly chosen from FDDD. As shown in the figure, the contrast drastically varies from sample to sample, and texture blobs in some samples are not strictly aligned with image axes, e.g. the second-to-last column in the figure.

30

CR IP T

ACCEPTED MANUSCRIPT

M

AN US

Fig. 34 Selected FDDD samples. Results of experiments of typeⅠare illustrated in Fig. 35 and Fig. 36. For area factor, divergence fluctuates as the parameter changes and the fluctuation is relatively stable when the parameter exceeds 0.374. For tolerance, the divergence is near to zero around 0.1. In brief, the segmentation results of FDDD are relatively insensitive to the changes of area factor and sensitive to the changes of tolerance.

PT

ED

Fig. 35 Divergence of area factor based on FDDD.

AC

CE

Fig. 36 Divergence of tolerance based on FDDD. Results of experiments of typeⅡ are illustrated in Fig. 37. Cosine is the metric involved in optimal AUCs throughout all subfigures, hence, the optimal metric of FDDD is concluded to be cosine.

31

AN US

CR IP T

ACCEPTED MANUSCRIPT

AC

CE

PT

ED

M

Fig. 37 ROC curves based on FDDD. Results of experiments of type Ⅲ are shown in Fig. 38 and Fig. 39. As shown in Fig. 38, the performance is nearly independent from the changes of cluster number limit. TPR decreases a little at value 9; hence, the optimal value of cluster number limit is concluded to be 8. As shown in Fig. 39, 1.1 and 1.15 are ideal values because the TPRs are almost the highest at these values and TPR decreases as the value increases. Thus, the optimal value of standard deviation factor is concluded to be 1.15.

Fig. 38 Error bars of cluster number limit based on FDDD.

32

ACCEPTED MANUSCRIPT

ED

M

AN US

CR IP T

Fig. 39 Error bars of standard deviation factor based on FDDD. Results of experiments of type Ⅳ are illustrated in Fig. 40. The left subfigure of Fig. 40 depicts the ROC curves of LSLT parameterized by optimized hyper parameters. All methods except LSLT and WGIS lead to AUCs lower than 0.5, which means these methods do not work at all. For LSLT, its AUC achieves 0.95 which is even higher than its performances for FID. The runtime is shown on the right subfigure of Fig. 40. Although there are some outliers for LSLT, the median runtime of LSLT is lower than LSG and RB.

PT

Fig. 40 ROC curves and runtime based on FDDD.

5. Conclusion

AC

CE

In this paper, a novel AFI method is proposed to identify fabric defects based on the defect-free fabric images. Because the defective samples are assumed unavailable, LSLT method is derived purely based on the defect-free samples. For a given fabric image, with the LSLT method, the image is segmented to non-overlapping lattices through a novel lattice segmentation procedure and then the lattices are compared with the lattice templates inferred from defect-free samples. The lattice segmentation infers a placement rule of texture blobs from the structural image of a given fabric image. For each defect-free sample, the lattices are segmented based on the dynamically revealed placement rule, and the period of the resulting lattices is estimated. The lattices of the same texture are extracted based on the period and adopted for generating a representative lattice of the texture involving a specific metric. The representative lattices of each defect-free sample are then sorted to guarantee they are consistent with a global texture order. For each texture class, its representative lattices from all samples are grouped, and each group yields a template. Hence, each texture class is modeled by multiple templates based on a specific metric and different metrics lead to distinct sets of templates for modeling a single texture class in turn. 33

ACCEPTED MANUSCRIPT

CR IP T

For each metric, the range of the distances between defect-free lattices and templates is inferred. If the lattice textures are alternatively changed, then an additional period threshold is also learnt. Finally, for a given test sample, the placement rule is inferred, and the lattices are segmented and compared with the templates by estimating their distances according to the specific metric. The defective lattices are identified as the ones of distances out of the learnt range or exceeding the period threshold. For all metrics, their identified defective lattices are collected as the final output of LSLT. The combinations of metrics and values of hyper parameters leading to high efficiency are determined through experiments. The performance of LSLT parameterized by the optimal metric combination and hyper parameter values is evaluated based on three datasets containing dot-, boxand star-patterned fabric images. According to the comparison of the experiment results, LSLT outperforms BB, RB, ER and WGIS. However, its runtime is not ideal compared with any other methods. It should be noticed that the runtime varies for different fabric patterns. This research may be suitable for not only the fabrics but also some 2D surfaces like ceramic tile of repetitive textures.

AN US

Acknowledgments

This research is financially supported by the National Natural Science Foundation of China (Grant No. 31670553).

References

AC

CE

PT

ED

M

[1] K. Srinivasan, P. H. Dastoor, P. Radhakrishnaiah, S. Jayaraman, FDAS : a knowledge-based framework for analysis of defects in woven textile structures, J. Text. Inst. 83 (1992) 431–448. [2] X. Xie, A review of recent advances in surface defect detection using texture analysis techniques, Electron. Lett. on Comput. Vision and Image Anal. 7(3) (2008) 1-22. [3] H. Y. T. Ngan , G. K. H. Pang , N. H. C. Yung , Automated fabric defect detection—a review, Image Vis. Comput. 29 (7) (2011) 442–458. [4] K. Hanbay, M. F. Talu, Ö. F. Özgüven, Fabric defect detection systems and methods—A systematic literature review, Int. J. for Light and Electron Optics 127(24) (2016) 11960–11973. [5] H. F. Ng, Automatic thresholding for defect detection, Pattern Recognit. Lett. 27 (14) (2006) 1644–1649. [6] A. Latif-Amet, A. Ertüzün, A. Erc¸ il, An efficient method for texture defect detection: sub-band domain co-occurrence matrices, Image Vision Comput.18 (2000) 543–553. [7] Y. F. Zhang, R. R. Bresee, Fabric defect detection and classification using image analysis, Text. Res. J. 65 (1995) 1–9. [8] L. Tong, W. K. Wong, C. K. Kwong, Differential evolution-based optimal Gabor filter model for fabric inspection, Neurocomputing 173 (2016) 1386-1401. [9] D. M. Tsai, C. Y. Heish, Automated surface inspection for directional textures, Image Vision Comput. 18 (1999) 49–62. [10] M. Tabassian, R. Ghaderi, R. Ebrahimpour, Knitted fabric defect classification for uncertain labels based on Dempster–Shafer theory of evidence,Expert Syst Appl. 38 (2011) 5259–5267. [11] O. Alata, C. Ramananjarasoa, Unsupervised textured image segmentation using 2-D quarter plane autoregressive model with four predictionsupports, Pattern Recognit. Lett. 26 (2005) 1069– 1081. 34

ACCEPTED MANUSCRIPT

AC

CE

PT

ED

M

AN US

CR IP T

[12] A. Bouhamidi, K. Jbilou, An iterative method for Bayesian Gauss–Markov image restoration, Appl. Math. Modell. 33 (2009) 361–372. [13] J. P. Monaco, A. Madabhushi, Class-specific weighting for Markov random field estimation: application to medical image segmentation, Med. ImageAnal. 16 (2012) 1477–1489. [14] J. Chen, A. K. Jain, A Structural Approach To Identify Defects In Textured Images, IEEE Int. Conf. on Syst., Man, and Cybern. 1998: 29-32. [15] A. Bodnarova, M. Bennamoun, K.K. Kubik, Defect Defection in Textile Materials Based on Aspects on The HVS, Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics, 1998: 4423–4428. [16] K. Y. Song, J. Kittler, M. Petrou, Defect detection in random colour textures, Image and Vision Computing 14 (1996) 667-683. [17] L. Jia , C. Chen , J. Liang , Z. Hou , Fabric defect inspection based on lattice segmentation and Gabor filtering, Neurocomputing 238 (2017) 84–102. [18] L. Jia, J. Liang, Fabric defect inspection based on isotropic lattice segmentation, Journal of the Franklin Institute 354 (13) (2017) 5694-5738. [19] H. Y. T. Ngan, G. K. H. Pang, N. H. C. Yung, Motif-based defect detection for patterned fabric, Pattern Recognit. 41 (6) (2008) 1878–1894. [20] H. Y. T. Ngan, G. K.H. Pang, N. H.C. Yung, et al., Wavelet based methods on patterned fabric defect detection, Pattern Recognit. 38 (4) (2005) 559–576. [21] H. Y. T. Ngan, G. K. H. Pang, Novel method for patterned fabric inspection using bollinger bands, Opt. Eng. 45 (8) (2006) 087202-1–087202-15. [22] H. Y. T. Ngan, G. K. H. Pang, Regularity analysis for patterned texture inspection, IEEE Trans. Autom. Sci. Eng. 6 (1) (2009) 131–144. [23] M. K. Ng, H. Y. T. Ngan, X. Yuan, et al., Patterned fabric inspection and visualization by the method of image decomposition, IEEE Trans. Autom. Sci. Eng. 11 (3) (2014) 943–947. [24] C. S.C. Tsang, H. Y. T. Ngan, G. K. H. Pang, Fabric inspection based on the Elo rating method, Pattern Recognit. 51 (2016) 378–394. [25] L. Xu, Q. Yan, Y. Xia, J. Jia, Structure Extraction from Texture via Relative Total Variation, ACM Transactions on Graphics 31 (6) 2012 Article 139 [26] M. Ng, X. M. Yuan, W. X. Zhang, A coupled variational image decomposition and restoration model for blurred cartoon-plus-texture images with missing pixels, IEEE Trans. Image Process. 22 (6) (2013) 2233–2246. [27] J. Han, S. J. McKenna, Lattice estimation from images of patterns that exhibit translational symmetry, Image and Vision Computing 32 (1) (2014) 64–73. [28] Y. Liu , R. T. Collins , Y. Tsin, A computational model for periodic pattern perception based on frieze and wallpaper groups, IEEE Trans. on Pattern Anal. and Mach. Intell. 26 (3) (2004) 354– 371. [29] D. Bradley, G. Roth, Adaptive Thresholding Using the Integral Image, Journal of Graphics Tools 12 (2) 2007 13-21. [30] N. Dalal, B. Triggs, Histograms of Oriented Gradients for Human Detection, IEEE Comput. Soc. Conf. on Comput. Vision and Pattern Recognition 1 (2005) 886–893. [31] K. Hanbay, M.F. Talu, Ö. F. Özgüven, Real time fabric defect detection by using fourier transform, Journal of the Faculty of Engineering and Architecture of Gazi University 32 (1) 2017 151-158. [32] K. Hanbay, M.F. Talu, Ö. F. Özgüven, D. Öztürk, Fabric defect detection methods for circular 35

ACCEPTED MANUSCRIPT

AC

CE

PT

ED

M

AN US

CR IP T

knitting machines, 23th Signal Processing and Communications Applications Conference (SIU), 2015 735-738. [33] K. Hanbay, M. F. Talu, Ö. F. Özgüven, Fabric defect detection systems and methods—A systematic literature review, Optik - International Journal for Light and Electron Optics 127 (24) 2016 11960-11973. [34] D. Biggs, M. Andrews, Acceleration of Iterative Image Restoration Algorithms, Applied Optics 36 (8) 1997 1766-1775

36