Fabric Defect Detection Method Using SA-Pix2pix Network and Transfer Learning

Abstract

This paper proposes a fabric defect detection algorithm based on the SA-Pix2pix network and transfer learning to address the issue of insufficient accuracy in detecting complex pattern fabric defects in scenarios with limited sample data. Its primary contribution lies in treating defects as disruptions to the fabric’s texture. It leverages a generative adversarial network to reconstruct defective images, restoring them to images of normal fabric texture. Subsequently, the reconstituted images are subjected to dissimilarity calculations against defective images, leading to image segmentation for the purpose of defect detection. This approach addresses the issues of poor defect image reconstruction accuracy due to the limited ability of remote dependency modeling within the generator’s convolutional neural network. It also tackles deficiencies in the generative adversarial network’s loss function in handling image details. To enhance the structure and loss function of the generative adversarial network, it introduces self-attention mechanisms, L1 loss, and an improved structural loss, thus mitigating the problems of low defect image reconstruction accuracy and insufficient image detail handling by the network. To counteract the issue of declining model training accuracy in the face of sparse complex fabric defect samples, a channel-wise domain transfer learning approach is introduced. This approach constrains the training of the target network through feature distribution, thereby overcoming the problem of target network overfitting caused by limited sample data. The study employs three methods to experimentally compare and investigate five distinct complex pattern fabric defects. The results demonstrate that, when compared to two other defect detection methods, the approach advocated in this paper exhibits superior detection accuracy in scenarios with limited sample data.

1. Introduction

Fabric defect detection stands as a pivotal component within textile quality control processes. Manual inspection methods fall short of meeting the demands of actual production. The integration of machine vision technology has emerged as a promising avenue, offering high-precision automatic inspection capabilities to supplant manual scrutiny. This transition holds substantial significance for enhancing textile production quality and efficiency [1].

In the realm of machine vision technology, the efficiency of fabric defect detection algorithms plays a decisive role in enabling automated detection. Current fabric defect detection methodologies can be broadly categorized into three types [2]. Firstly, frequency domain algorithms, such as wavelet transform [3,4,5] and Gabor filtering [6,7], leverage Fourier transformations [8] of images to extract spectral features for defect identification, primarily effective for simple background features, like solid-color fabrics. However, their efficacy diminishes when detecting defects in fabrics with intricate textures. Secondly, model-based algorithms, including Gaussian mixture modeling [9,10], utilize autoregressive modeling to extract error regions in fabric images. While effective for simpler textures, these methods often falter in detecting defects within more complex fabric textures. Lastly, spatial domain-based methods like grayscale symbiotic matrix algorithms [11,12] attempt to discern normal texture from defects by learning texture features. However, their extensive computational demands and lack of robustness hinder practical application. Despite significant progress in traditional fabric defect detection algorithms, limitations persist in terms of detection accuracy, generalization, and real-time performance. These shortcomings impede their practical deployment in production settings.

In recent years, the evolution of deep learning has propelled convolutional neural networks (CNNs) to remarkable heights in the domain of image recognition, fostering their swift integration across diverse industries. Within the sphere of defect detection, CNNs manifest through two predominant avenues: target detection algorithms and semantic segmentation.

Defect detection based on target detection algorithms unfolds along two principal trajectories, characterized by Regional Convolutional Neural Networks [13] (R-CNN) and models rooted in You Only Look Once [14] (YOLO) architectures. R-CNN, initially designed for specific target determination, has found application in fabric defect detection, notably through improved iterations like Faster R-CNN [15]. Notably, Chen et al. [16] introduced the Faster GG R-CNN model, employing a two-stage training approach that integrates Gabor filters into the Faster R-CNN framework via genetic algorithms and backpropagation. This innovation effectively addresses the challenge posed by complex background textures in fabric defect detection. In a parallel development, Jin et al. [17] enhanced YOLOv5 using attention mechanisms to augment defect-related features. They proposed a lightweight fabric defect detection method exhibiting promising results when tested on fabric blemish images. Despite the expediency and high accuracy of target detection-based defect detection methods, their efficacy diminishes when faced with fabrics adorned with intricate patterns and when training samples are scarce. The reliance of these methods on the volume of training samples poses a significant limitation, especially given the challenges associated with collecting defect samples. In scenarios where the number of training samples is insufficient, these methods may encounter limitations and potential failures.

The defect detection approach based on image segmentation involves dividing an image into distinct regions and extracting areas of interest. In this method, the fabric image to be inspected is reconstructed using a pre-trained reconstruction model, resulting in a reconstructed image. Subsequently, image segmentation is performed on the reconstructed image in comparison with the original fabric image to obtain a defect map containing defect information. This image reconstruction algorithm only requires training with normal and defect samples, eliminating the need to label defect samples and thus reducing the manual effort involved in annotating defects. Within image segmentation methods, autoencoder networks [18] or generative adversarial networks [19] are commonly employed for the pixel-level segmentation of fabric defect images. Autoencoders are typically trained in an unsupervised manner, using defect-free samples, avoiding the necessity to collect a large number of defect samples for network training. Liu et al. [20] introduced the CU-Net model, which incorporates attention mechanisms into the classical U-Net architecture and employs a novel composite loss function for training. They evaluated the model using the publicly available AITEX fabric defect dataset, demonstrating an enhancement in fabric defect detection accuracy. Meanwhile, Yu et al. [21] proposed an unsupervised defect detection algorithm based on a reconstruction network (ReNet-D). This approach involves dividing images into equally sized patches for training, enabling defect detection in texture images of varying complexities. Although autoencoder-based defect detection methods require only normal samples during network training, they can only learn the features of normal samples. During defect reconstruction, since defects may resemble the texture background of the original image, it can lead to the retention of excessive defect features in the reconstructed image, thereby reducing detection accuracy. Furthermore, autoencoders lose a significant amount of image information during training and cannot reconstruct clear images, making it challenging to apply this method to the detection of complex pattern fabric defects.

Since the inception of Generative Adversarial Networks (GANs), their application has been extensive across diverse domains including steel [22], photoplates [23], and medicine [24] for image defect detection. Liu et al. [25] introduced a GAN-based framework specifically tailored for fabric defect detection. This framework commences by training a GAN to explore the conditional distribution of defects across varied texture backgrounds, utilizing a multilevel GAN structure to synthesize imperceptible defects within defect-free fabric samples. Subsequently, the synthesized defects are integrated into specific locations within new fabric instances. The final stage involves training a deep semantic segmentation network tasked with detecting defects within the generated defect images. Zhang et al. [26] proposed a framework for generating adversarial networks based on attentional feature fusion for unsupervised detection of defects in dyed fabrics, which enhances the model’s attention to texture details through multilevel information fusion and attentional mechanisms to improve the detection of defects in yarn-dyed fabrics.

The utilization of GAN networks presents augmented image feature extraction capabilities due to their inherent architecture. Moreover, their competitive adversarial-based training mechanism proves particularly adept in processing intricate patterned images [27], effectively diminishing the requisite number of training samples [28]. In contrast to self-encoders, GANs exhibit greater suitability in processing fabric images characterized by complex textures.

The intricate nature of fabric production, characterized by a diversity of species, small batch sizes, and high product yields, poses a significant challenge in acquiring ample sample data. This scarcity severely constrains the accuracy of detection model training. Introducing migration learning [29,30], a technique that migrates feature extraction capabilities to a designated network by weight sharing within the network model, stands as a promising solution. This approach demonstrates potential in amplifying model training efficiency and mitigating the reliance on extensive data during training. Presently, migration learning methods for blemish detection encompass two primary categories: pre-trained model methods and domain adaptive methods. Pre-trained model approaches [31,32] often overlook the intrinsic similarity between source and target domain data, leading the target network to acquire task-irrelevant features, thereby impeding the effectiveness of migration. Conversely, domain adaptive methods facilitate the transfer of pertinent features learned by the source domain model to the target task by identifying shared features or similarities between the domains [33]. Early domain adaptive techniques predominantly leveraged shallow machine learning methodologies, such as Support Vector Machines (SVM) [34] and Maximum Mean Discrepancy (MMD) [35], aiming to mitigate distributional disparities between source and target domains by altering feature spaces or re-weighting source domain data. Nevertheless, these methods exhibited limitations in handling high-dimensional data and complex tasks. As research evolved, methodologies centered on feature selection and transformation emerged to better accommodate diverse domain data. These encompassed techniques like feature selection [36] and principal component analysis (PCA) [37] aimed at reducing feature space dimensionality or extracting discriminative features. Advancements in the field witnessed a fusion of deep neural networks with domain-adaptive tasks. Li et al. [38] proposed a domain-adaptive DAYOLOv5 model specifically tailored for surface defect detection in tiles, exhibiting commendable performance even with limited datasets. Similarly, Zhao S et al. [39] introduced a fabric defect detection algorithm founded on two-stage deep migration learning, successfully transferring defect features from solid-color cotton linen to a checkered fabric detection model. This migration encapsulated defect size features and notably reduced the requisite training samples. Despite the strides made by these migration methods in enhancing detection performance and model training efficiency, challenges persist, particularly in achieving high detection accuracy when confronted with intricately patterned fabric defects.

This research considers blemishes as distortions to the original fabric, where regardless of their specific type, the texture within the blemished region diverges from the original fabric texture. To address the challenges posed by limited sample sizes and intricate patterns affecting blemish reconstruction accuracy, a novel fabric blemish detection algorithm combining the SA-Pix2pix network and migration learning is proposed. The primary objective is the restoration of defective regions to their pristine state. Subsequently, dissimilarity computations between the reconstructed and defective images are performed, followed by image segmentation to facilitate defect detection. To mitigate the poor reconstruction accuracy attributed to the Pix2pix network generator’s limited sensory field within convolutional layers and its inadequacy in establishing long-range dependencies within images, the concept of image self-attention is integrated into the network architecture. This innovation leads to the formulation of the SA-Pix2pix network, effectively remedying the network’s inability to extract comprehensive global features from images, thus enhancing reconstruction accuracy. Moreover, recognizing the inefficacy of the adversarial loss function within generative adversarial neural networks for processing fabric images with complex pattern textures, we introduce an improved structural loss function. This, combined with the original loss function, forms the target loss function, aimed at alleviating the network’s struggles with intricate fabric textures. A systematic investigation into the impact of each loss function on defect detection accuracy reveals that the fusion of L1 loss and the enhanced structural loss significantly enhances the network’s capacity to process image intricacies, thereby improving overall image reconstruction accuracy. Building upon this foundation, migration learning is introduced to constrain the training of the target network by aligning feature distributions, effectively addressing potential overfitting issues stemming from the limited influence of small sample sizes during training. This holistic approach aims to enhance the accuracy and robustness of fabric blemish detection by synergizing the SA-Pix2pix network, an adapted loss function, and migration learning to surmount challenges posed by small sample sizes and complex fabric textures in blemish reconstruction.

2. Materials and Methods

2.1. The SA-Pix2pix Model

2.1.1. Network Structure

The Pix2pix network [40], comprising a generator and discriminator, features a generator employing a U-Net model [41] that prioritizes direct feature fusion over pooled indexing methods. This design choice avoids memory-intensive full connectivity between downsampling and upsampling, thereby preserving crucial image structural information. Particularly beneficial for non-periodic and complex patterned texture fabrics, this framework efficiently shares essential underlying information between inputs and outputs, expediting image reconstruction.

To augment the network’s ability to extract global features crucial for defect detection within intricate patterned fabrics, this study integrates the image Self-Attention mechanism (Self-Attention) [42] into the generator network structure. Focused on reconstructing the blemished region necessitating less information, the addition of the Self-Attention mechanism selectively incorporates it solely within the input and intermediate layers of the generator’s upsampling and downsampling layers. This strategic placement enables the network to associate global features of the patterned fabric image with local features surrounding the blemish, enhancing the network’s capacity to comprehend both the broader fabric context and localized defect characteristics. This targeted integration aims to bolster the network’s ability to extract comprehensive features crucial for detecting defects within complex patterned fabrics while curbing computational complexity and training time.

The generator and discriminator components of the architecture employ the “convolution-batch-activation function” module for both upsampling and downsampling operations. Figure 1 illustrates the specific structure of the generator network. The discriminator, derived from the Pix2pix network, is structured as a “Patch GAN” and depicted in Figure 2.

2.1.2. Loss Function

In addressing the intricacies of image reconstruction, this section adopts a composite approach involving multiple loss functions to train the model effectively. Initially, the traditional pixel-by-pixel loss, known as the averaged absolute value error (L1 loss function) [43], is employed in this study. This loss function serves to constrain the generated images by the generator, ensuring minimal deviation from true values. Compared to the mean squared loss function (L2 loss function), the L1 loss function offers advantages in terms of rapid convergence and stable gradients, rendering it the preferred choice as the pixel-by-pixel loss function for the network.

Simultaneously, to empower the generator to process images with a higher level of detail and to ensure proficient reconstruction of defective regions within fabric images, the structural loss function (SSIM loss function) is introduced. Recognizing that L1 loss fails to encapsulate regional structural characteristics in images, especially in scenarios involving irregular textures, the SSIM loss function emerges as a critical addition. Leveraging luminance, contrast, and structure (structural) elements, the SSIM loss function operates in tandem with the L1 loss function to optimize the network model. It specifically accentuates image details more effectively than the L1 loss alone, as shown in Equation (1):

$$G^{*}=\begin{array}{c}\arg \\ \end{array}\begin{array}{c}\min \\ G\end{array}\begin{array}{c}\max \\ D\end{array}\begin{array}{c}{L}_{CGAN}(D,G)+\lambda (L1+L^{\ast }{}_{SSIM})\\ \end{array}$$

(1)

In the equation, λ represents a weight parameter regulating the balance between the generator’s Conditional Generative Adversarial Network (CGAN) loss function and the L1 and SSIM loss functions. L_CGAN (D, G) denotes the CGAN target loss [44], as expressed in Equation (2):

$$\begin{array}{c}\arg \\ \end{array}\begin{array}{c}\min \\ G\end{array}\begin{array}{c}\max \\ D\end{array}\begin{array}{c}{L}_{CGAN}(D,G)\\ \end{array}={{\rm E}}_{x\sim p_{data}(x)}[\log D(x|y)]+{{\rm E}}_{z\sim p_{data}(z)}[\log (1-D(G(z|y)))]$$

(2)

In the equation, x represents the label, which corresponds to normal sample images, y stands for the additional input information (in this context, the input condition is defect images), and z denotes the defect images. D represents the discriminator, while G represents the generator. p_data(x) signifies the dataset of normal samples, and p_data(z) represents the dataset of defect samples.

Typically, the SSIM loss function [45] is formulated as follows:

$$L_{SSIM}(x,y)=l{(x,y)}^{\alpha }\cdot c{(x,y)}^{\beta }\cdot s{(x,y)}^{\gamma }$$

(3)

In the equation, x and y represent normal samples and images generated by the generator, respectively. α, β, and γ are constants. Here, l(x, y) denotes the comparison of the brightness of the two images, c(x, y) signifies the contrast comparison between the two images, and s(x, y) represents the structural comparison of the two images.

The reconstruction process yields images with a subtle difference in brightness and contrast compared to the originals, leading to a diminished value of the Structural Similarity Index (SSIM) loss function. This diminutive value, as encountered in Equation (1), curtails the efficacy of the SSIM loss function and renders it susceptible to noise. To address this limitation, an enhancement is introduced to the SSIM loss function, as delineated in Equation (4). This modification serves to amplify the weight of the SSIM loss function within Equation (1), thereby augmenting the sensitivity of the Conditional Generative Adversarial Network (CGAN) to image disparities. The refined SSIM loss function, as per Equation (4), is instrumental in elevating the network’s discernment of subtle variations, promoting improved performance in capturing nuanced image differences during the reconstruction process.

$$L^{\ast }{}_{SSIM}=1-\min (l(x,y),c(x,y),s(x,y))$$

(4)

2.2. Channel Domain Transfer Learning

The technical roadmap for achieving complex pattern defect reconstruction using the SA-Pix2pix network and transfer learning is depicted in Figure 3. It primarily comprises two stages:

Channel Weight Estimation: Artificial defect patterns, referred to as “artificial defects”, are created to resemble real defects closely. These artificial defects are utilized to train the SA-Pix2pix model in the source domain. The channel weights are estimated by calculating the variation in the image reconstruction loss function. This process helps identify the channels and features that have the most significant impact on defect image reconstruction accuracy.

Construction of Metric Criteria: Real defect images are treated as target domain data and are input into both the source domain network (the pre-trained SA-Pix2pix model) and the target domain network (the SA-Pix2pix model under training). These networks map the source domain data to the same feature space and measure the distance between the corresponding features of the source and target domain networks. The feature distance is then multiplied by the channel weights. A smaller sum of the feature distance and channel weight product indicates greater consistency in feature distribution between the source and target domains, facilitating the transfer of similar features between the two domains. However, relying solely on “consistency in feature distribution” cannot completely address the issue of inadequate defect reconstruction accuracy in the target domain network. To improve the reconstruction accuracy of complex pattern defect images, both the feature distance and the channel weight product sum must be incorporated into the loss function of the target domain network along with the image reconstruction loss function.

2.2.1. Channel Weight Estimation

Each convolutional layer comprises multiple kernels, with each kernel corresponding to a channel. To assess the impact of each channel on defect image reconstruction accuracy, we set all the elements of each kernel to zero separately. The change in the loss function (Equation (1)) is computed to gauge the significance of each channel. This loss can be interpreted as the error between the generator’s output and the real labels, indicating the extent to which the images generated by the generator are mistakenly recognized as real images by the discriminator.

The Pix2pix network generators are separately trained using source domain data, one with the j-th channel, and one without. The loss function with the j-th channel is denoted as $l_j$, and without the j-th channel as ${l^{\prime }}_j$. To normalize the change in loss functions, the softmax function is applied, ensuring that the weight for the j-th channel is non-negative, as presented in Equation (3). From Equation (5), it is evident that if the removal of a specific channel leads to a larger change in the loss function, the channel weight becomes greater, signifying a more substantial impact of that channel on the defect image reconstruction accuracy.

$${\mathrm{w}}_j=soft\max (l_j-{l^{\prime }}_j)$$

(5)

$$Z_{\mathrm{j}}=l_j-{l^{\prime }}_j$$

(6)

$$\mathrm{soft}\max (z_j)=\frac{\exp (z_j)}{{\displaystyle {\sum }_{t=1}^T\exp (z_t)}}$$

(7)

In Equation (7), T represents the total number of convolution kernels in the network.

2.2.2. Construction of Metric Criteria

For any defect image xi in the target domain (1 ≤ i ≤ n), it is input into the generator of the source domain network. After undergoing computation by the j-th convolution kernel of the k-th convolutional layer, it produces a feature map $F^{jk}$. Subsequently, xi is input into the generator of the target domain network, and after undergoing computation by the j-th convolution kernel of the k-th convolutional layer, it yields a feature map $F^{\prime /jk}$. The consistency in feature distribution between the source and target domains is quantified using Equation (8).

$$L_C={\displaystyle {\sum }_{k=1}^M{\displaystyle {\sum }_{j=1}^N(w_j\cdot ||F^{jk}-F^{\prime /jk}|{|}_2^2)}}$$

(8)

In this context, $w_j$ represents the weight allocated to the j-th convolution kernel, N denotes the total number of convolution kernels, and M represents the total number of convolutional layers. $\parallel F^{jk}-F^{\prime /jk}{\parallel }_2^2$ signifies the Euclidean distance between corresponding features of the source and target networks.

From Equation (8), it can be observed that when the features in the source and target domains are dissimilar, despite having a large Euclidean distance between features, the channel weights corresponding to dissimilar features are small. Channel weights effectively mitigate the influence of dissimilar features on the measure of feature distribution consistency. When the features in the source and target domains are similar, although the channel weights corresponding to similar features are relatively large, the Euclidean distance between similar features is very small, resulting in a minimal value for the measure of feature distribution consistency. Therefore, Equation (8) is applicable for assessing feature distribution consistency, ensuring convergence, and facilitating the transfer of similar features between the source and target domains, thereby reducing the impact of a small sample size on defect image reconstruction accuracy.

However, the measure of feature distribution consistency, L_c, is still influenced by dissimilar features. To further enhance defect image reconstruction accuracy, Equations (1) and (8) are both incorporated as the loss function of the target domain network, as illustrated in Equation (9). Equation (9) not only accounts for the transfer learning of similar features but also considers the impact of the generator’s output error on reconstruction accuracy, ultimately improving defect reconstruction precision.

$$\min (G^{*}+\alpha L_C)$$

(9)

In the equation, the recommended value for $\alpha$ is 0.1.

2.3. Defect Detection

The dissimilarity computation of reconstructed images and the defect localization method are as follows.

2.3.1. Image Dissimilarity Calculation

Defect images, as depicted in Figure 4a, are fed into the defect reconstruction network, yielding images as shown in Figure 4b. The subtraction of these two images on a per-pixel basis results in defective images with the texture background removed, as illustrated in Figure 4c. These images encapsulate information about the location of defect regions.

2.3.2. Denoising and Defect Localization

The dissimilarity computation of reconstructed images and the defect localization method are as follows:

In Figure 4c, a significant amount of noise is present, giving rise to spurious defects, necessitating image denoising and grayscale conversion. This paper employs the enhanced FT algorithm [46] for denoising and grayscale conversion, replacing the FT algorithm’s filter with a mean filter. The processing outcome is depicted in Figure 4d. Finally, by binarizing the image, the defect detection results are obtained, as illustrated in Figure 4e.

2.4. Experimental Environment and Dataset

The experimental data were acquired using a CCD camera, and a dataset was generated by shifting a cropping box of size 256 × 256 on the captured images. To augment the dataset, the cropping box had a stride of less than 256. We collected five different types of fabric defect patterns, as depicted in Figure 5. Samples 1 and 2 are of cotton and linen fabric, while samples 3, 4, and 5 are made of cotton silk fabric. All five samples are woven fabrics, and the defect types primarily include oil stains, abrasions, foreign objects, tears, and missing warp threads. Both the test and training set images are obtained from different positions on the same fabric roll. Within the training set, each sample category was obtained through cropping and contains a small number of positive samples. According to statistics, the approximate ratio of positive to negative samples is 1:4.

A significant number of artificial defect images were created individually on the flawless images of five distinct patterns, as illustrated in Figure 6. In the transfer learning experiments, only 200 genuine defect images per pattern class were employed as the target domain dataset, with 1000 artificial defect images used as the source domain dataset.

The computer system configuration is detailed in

Table 1. Computer system configuration.

System	Memory	GPU	CPU	Deep Learning Framework
Windows 10	64 G	NVIDIA GTX-1080Ti	Intel Core i7-8700K @ 3.70 GHz	Pytorch, CUDA10.1, CUDNN7.6

. In the experiments, the Adam optimizer [47] was utilized using the parameters β1 = 0.5 and β2 = 0.999. During training, the initial learning rate was set to 0.0002, which linearly decayed to 0 over the last 200 epochs.

3. Results

3.1. Impact of Self-Attention Mechanism on Defect Detection Accuracy

In order to assess the impact of the self-attention mechanism and loss functions on defect detection accuracy, separate experimental studies were conducted to investigate the effects of the self-attention mechanism and loss functions on defect detection accuracy.

The convolutional neural network’s local receptive field within GAN networks has limitations, restricting its focus to local image features, thereby diminishing image reconstruction accuracy. As depicted in Figure 7a, the defect appears as a relatively large black stain. In the absence of the self-attention mechanism, the reconstructed image (Figure 7b) retains small black areas, causing missed defect regions. Therefore, the self-attention mechanism was introduced into the generator network. To reduce the network’s complexity and training time, the self-attention mechanism was added only to the input and intermediate layers of the generator’s upsampling and downsampling layers, as depicted in Figure 1. This enables the network to capture global features of fabric patterns and local features of the defect’s surrounding area. The introduction of the self-attention mechanism empowers the generator with the ability to extract global features, effectively addressing the issue of inadequate image reconstruction accuracy. The reconstructed image with the self-attention mechanism, as shown in Figure 7d, exhibits significant improvement in reconstruction accuracy compared to the version without the self-attention mechanism. Furthermore, it enables the complete detection of defect areas, as demonstrated in Figure 7e.

3.2. Impact of Loss Function on Defect Detection Accuracy

Existing GAN networks have limited capability in handling image details, resulting in incorrect reconstruction of non-defective areas. This issue leads to increased dissimilarity errors in images and a decrease in defect segmentation accuracy, potentially causing missed and false defect detection, as illustrated in Figure 8b. In CGAN training, we introduced both the L1 loss function and an enhanced SSIM loss function as the target loss functions. As depicted in Figure 8d, when using only the L1 loss function, the edge regions of defect images are not entirely detected. This might be due to the L1 loss function’s capability to perform pixel-wise comparisons, adapting to the complex requirements of reconstructing fabric patterns in images, but it requires an enhancement in image detail reconstruction ability. When using only the improved SSIM loss function, the results, as shown in Figure 8e, indicate that some defect regions are missed. Combining both loss functions can address the aforementioned limitations, as illustrated in Figure 8f. The weight parameter λ = 100 balances the CGAN loss function with the L1 and SSIM loss functions effectively.

3.3. Comparison of Experimental Results for Different Defect Detection Methods

Under the same testing conditions, comparative experiments were conducted between the ReNet-D [21] method, the SDDM-PS [48] method, and the SA-Pix2pix model. Initially, these three methods were employed to reconstruct defect images, followed by pixel-wise dissimilarity calculations between the reconstructed and defect images. Finally, the dissimilarity results underwent image segmentation, as illustrated in Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13.

In Figure 8, which displays the experimental results for oil stain defects using the three methods, the second column reveals the ReNet-D method’s significantly poor detection accuracy for oil stain defects. In the third column, it can be observed that the SDDM-PS method provides incomplete detection of oil stain defect regions, and in some cases, it fails to detect the defect in sample 4. However, the fourth column demonstrates that the method recommended in this paper effectively detects oil stain defects.

Figure 10 displays the experimental results for various methods of detecting foreign object defects. In the second column, it is evident that the ReNet-D method exhibits notably poor accuracy in detecting scratch defects. In the third column, the SDDM-PS method yields results with a substantial amount of noise points and even misinterprets the image texture background as defects, failing to effectively detect scratch defects. However, in the fourth column, our proposed approach demonstrates effective scratch defect detection. Nevertheless, for sample 4, the detection results exhibit a tendency for smaller detection areas.

Figure 11 displays the experimental results for foreign object defects using various methods. From the second column, it is evident that the ReNet-D method exhibits significantly poor detection accuracy for foreign object defects. The third column reveals that the SDDM-PS method can detect most of the defect areas but performs poorly in the detection of defects in samples 1, 4, and 5. However, in the fourth column, the method recommended in this paper effectively detects foreign object defects, with only minor undetected areas in sample 1.

Figure 12 depicts the experimental results for pinhole defects using various methods. From the second column, it is evident that the ReNet-D method exhibits significantly poor detection accuracy for pinhole defects. The third column reveals that the SDDM-PS method detects pinhole defect areas but with some noise and incomplete detection areas. In the fourth column, the method recommended in this paper effectively detects pinhole defects.

Figure 13 illustrates the experimental results for missing warp defects using various methods. From the second column, it is evident that the ReNet-D method exhibits significantly poor detection accuracy for the missing warp defects. The third column reveals that the SDDM-PS method fails to detect the missing warp defects completely. For samples 3, 4, and 5, it only detects a small portion of the defects, and for sample 1, it does not detect any defects. Sample 2 also experiences a few cases of false detection. In the fourth column, it is evident that the method recommended in this paper can detect missing warp defects almost entirely, although discontinuous results are observed for sample 3 and sample 4.

In summary, the ReNet-D methodologies exhibit poor accuracy in blemish detection, occasionally even incorporating the original image background. This may be attributed to the self-encoder’s loss of critical image information during training, hindering the ReNet-D method’s ability to effectively reconstruct images and subsequently perform precise blemish image segmentation.

While the SDDM-PS method demonstrates a degree of proficiency in detecting oil stains, foreign objects, and broken holes, limitations persist. This may be attributed to its reliance solely on a GAN network for sample training, as GANs exhibit limited capacity in processing image details. Their focus tends to be confined to local areas within images, lacking the capability to detect defects closely resembling the original fabric texture background.

In contrast, the method advocated in this paper showcases notable completeness and accuracy in detecting various defect types compared to the aforementioned methodologies. Notably, it achieves high accuracy in detecting oil stains, foreign objects, and holes. However, it does encounter errors when identifying scuffs and warp defects. This is primarily due to the minor damage degree within the edge regions of scuff defects, which bears a slight difference from the fabric background. Consequently, the subtle dissimilarity between the reconstructed and original images in these areas impedes effective segmentation, leading to smaller detected regions. Similarly, warp defects present challenges akin to scuffing defects. The low contrast between defective areas and the fabric background further diminishes differences between the reconstructed and original images. Consequently, this results in intermittent segmentation outcomes.

In addition, we conducted an efficiency comparison of the algorithms using an in-house dataset.

Table 2. Comparison of processing time.

Methods	ReNet-D	SDDM-PS	SA-Pix2pix
Time (ms)	38.25	43.82	46.15

illustrates the processing elapsed time of the three methods under uniform computational performance. The method advocated in this paper demonstrates an average detection time of 46.15 ms, slightly exceeding the processing times of the other two methods. However, it leads to higher detection accuracy.

4. Discussion

This paper proposes a novel blemish detection methodology leveraging an SA-Pix2pix network combined with migration learning. This approach synthesizes the strengths of GAN networks, self-attention mechanisms, and migration learning to notably enhance the reconstruction accuracy of blemish images, especially when working with limited samples. Addressing the limitations of generative adversarial neural networks in processing image details, we introduce the L1 loss function and an enhanced structural loss function to form a comprehensive target loss function, augmenting the CGAN network’s capacity to handle image intricacies.

Through comparative analysis between networks employing and omitting the self-attention mechanism and distinct loss functions, our findings verify that the self-attention mechanism significantly amplifies the network’s global feature extraction capabilities. Its introduction enables the network to reconstruct defects based on global image features, markedly improving reconstruction accuracy. Conversely, networks lacking the self-attention mechanism rely solely on local features for defect reconstruction. Consequently, they may fail to reconstruct intermediate regions, leading to errors in the final detection results.

Moreover, our experiments confirm that integrating both the L1 loss function and the improved SSIM loss function notably enhances the network’s ability to process intricate fabric images with complex pattern textures. However, employing either loss function in isolation or without introducing any additional loss function diminishes the network’s defect detection accuracy.

Comparative experimental studies highlight the superior performance of our method over ReNet-D and SDDM-PS methodologies in enhancing both the completeness of defective regions and the precision of their localization, particularly in scenarios with limited sample sizes. While the incorporation of the self-attention mechanism and improved loss functions into the Pix2pix network elevates its image reconstruction capabilities, further enhancements are needed to improve detection accuracy for abrasions and missing warp defects.

While this paper successfully achieves defect detection through image reconstruction, there exist certain limitations in the obtained results, leading to errors in defect detection in some images. To further enhance defect detection accuracy, additional research avenues warrant exploration in the following areas:

Firstly, investigating the implementation of deformable convolutions instead of conventional convolutions for image sampling presents a promising direction. Deformable convolutions introduce positional offset parameters to each sampling point, allowing the convolution to autonomously learn sampling point selection within the network. This transformation alters the receptive field from rectangular to polygonal, enabling better fitting to object contours with multi-layer polygonal receptive fields. This enhancement could significantly aid in learning fabric pattern features, potentially playing a pivotal role in image reconstruction. However, leveraging deformable convolutions might present challenges in model training and convergence. Subsequent research could focus on optimizing the integration of deformable convolutions within the model to improve image reconstruction accuracy and consequently enhance defect detection performance.

Secondly, augmenting the network by integrating Transformer models, renowned for their capacity to directly focus on global image features, holds promise. Properly integrating Transformer models with CNNs could further elevate image reconstruction accuracy, especially in scenarios with limited sample sizes. This integration might bolster the network’s ability to capture comprehensive global features crucial for defect detection. Investigating the synergy between Transformer models and CNNs could be instrumental in advancing defect detection methodologies.

5. Conclusions

This paper presents a novel defect detection approach leveraging the SA-Pix2pix network and migration learning to address the issue of insufficient accuracy in detecting blemishes within complex patterned fabrics, especially when dealing with limited sample sizes. The central concept involves treating blemishes as damage to fabric texture, utilizing a conditional generative adversarial neural network to reconstruct the blemished regions within images, aiming to restore them to their normal fabric texture. Subsequently, dissimilarity computation between the reconstructed and defective images is conducted, followed by image segmentation based on the dissimilarity result, ultimately facilitating accurate blemish detection.

Our contributions revolve around three key points: Firstly, we address the challenge of establishing remote dependencies in images due to the limited receptive field of the convolutional neural network within the generator. We introduce the image self-attention mechanism into the Pix2pix neural network, exploring its impact on the network’s image reconstruction accuracy. This leads to the proposal of the SA-Pix2pix neural network model, significantly enhancing the accuracy of defective image reconstruction.

Secondly, to combat the Pix2pix neural network’s limited capacity in processing image details, we introduce the L1 loss function and an improved structural loss function. This exploration assesses the impact of each loss function on image reconstruction accuracy, culminating in the construction of a comprehensive loss function tailored specifically for defective image reconstruction, overcoming the network’s limitations in processing image details.

Thirdly, we introduce migration learning to regulate the training of the target network through feature distribution, mitigating potential overfitting issues arising from small sample sizes.

Furthermore, a comparative experimental study is conducted, pitting the ReNet-D model, SDDM-PS model, and our proposed model against each other. The study involves the detection of five different complex pattern fabric blemishes, demonstrating that our method outperforms both the ReNet-D and SDDM-PS models in terms of blemish detection accuracy.