SeRNet: Segmentation Helps Reconstruction for Anomaly Detection

Abstract

With production growth and improvements in production speed in modern industries, accurate anomaly detection is becoming increasingly important to improve quality inspection efficiency and help minimize production costs. Existing reconstruction-based methods achieve promising anomaly detection results in some scenarios. However, when large-scale anomalies exist, the generalization ability of these methods is limited, and it is difficult to reconstruct the anomalies effectively into normal areas, which may lead to unsatisfactory results. To address this issue, this paper proposes a novel network architecture called SeRNet, which comprises three components: a segmentation sub-network, a reconstruction sub-network, and a repair module. SeRNet addresses the challenge of large-scale anomaly reconstruction by utilizing the segmentation sub-network for pre-segmentation, the repair module for repairing the large-scale anomalies using normal images similar to the test images, and the reconstruction sub-network for processing small-scale anomalies and anomalies in the repaired splices. Additionally, this paper introduces two methods for generating pseudo-anomalies at different scales. SeRNet leverages the advantages of both the segmentation sub-network, which is effective at segmenting large-scale anomalies, and the reconstruction sub-network, which can effectively reconstruct small-scale anomalies. Experiments on the MVTec AD industrial dataset demonstrate that SeRNet delivers outstanding performance, achieving an image-level AUROC score of 99.6.

1. Introduction

Anomaly detection is a crucial research field in computer vision, with widespread applications across a range of industries [1,2,3,4]. Traditional anomaly detection methods typically rely on supervised learning approaches [5,6] and have demonstrated promising detection results. However, the scarcity of anomalous samples and the diversity of anomaly types limit the generalizability of these approaches. Thus, with the development of deep learning, unsupervised anomaly detection methods [7,8,9,10] have attracted considerable attention. Unsupervised anomaly detection methods can be categorized into two main classes: embedding-based and reconstruction-based methods.

Embedding-based methods [11,12,13] typically utilize a pre-trained model to extract features from samples and model the feature space to detect anomalies. Although these methods achieve higher detection accuracy, they also lack interpretability. In contrast, reconstruction-based methods [14,15,16,17] rely on restoration techniques to detect anomalies. During the training phase, reconstruction-based methods employ a pseudo-anomaly generation strategy to obtain anomalous images, and the reconstruction model then learns how to reconstruct these pseudo-anomalous samples as normal patterns. The resulting model then attempts to reconstruct real anomalous images during testing. Anomaly detection is performed by comparing the differences between the input and reconstructed images. However, these models typically have limited generalization ability, which prevents them from fully reconstructing all anomalies, particularly in the case of large-scale anomalies. The convolutional layer in the reconstruction model has a constrained receptive field, resulting in poor reconstruction ability and affecting the final detection accuracy. To address the challenges faced by reconstruction networks, previous research has alleviated this issue by introducing more powerful reconstruction networks. For example, anomaly diffusion [18] incorporates a diffusion model, while MUTAD [19] utilizes a multi-scale Transform model as the reconstruction network. Although these methods enhance reconstruction capabilities, they also increase the complexity of the detection algorithms, resulting in decreased efficiency.

To address the challenges described above, this paper proposes a network architecture called SeRNet, which comprises three parts: a segmentation sub-network, a reconstruction sub-network, and a repair module. Previous work often simply combines segmentation and reconstruction, using the difference images from the reconstruction output of the segmentation network for anomaly detection. This approach does not fully leverage the potential of the segmentation network, treating it merely as a discriminator for differences before and after image reconstruction, and it does not address the reconstruction difficulties faced by the reconstruction network when large-scale anomalies exist. The segmentation sub-network is responsible for detecting suspected large-scale anomalies, whereas the reconstruction sub-network focuses on reconstructing small-scale anomalies and repairing splices of anomalies. The repair module, guided by the prediction masks, uses similar images to effectively repair large-scale anomalies, thus helping to reconstruct anomalous regions. The above structure allows SeRNet to reconstruct anomalous regions of different scales more effectively. Two pseudo-anomaly generation methods, namely Perlin Pseudo-Anomaly (PPA) and Diffusion Pseudo-Anomaly (DPA), are employed to guide the learning of each sub-network. The PPA approach generates large-scale anomalies to guide the segmentation sub-network, whereas DPA generates small-scale anomalies to enable the reconstruction sub-network to focus on reconstructing the details of identified anomalous regions. This design enables each sub-network to focus on learning the specific anomaly patterns or features, thereby enhancing the model’s overall detection performance. Figure 1c illustrates the reconstructed image of SeRNet. Compared with Draem, the classical reconstruction method [9], SeRNet can more effectively handle large-scale anomalies. SeRNet does not adopt conventional innovative methods often found in previous work, such as replacing the reconstruction network model. Instead, SeRNet approaches the problem at its root, recognizing that the reconstruction network encounters receptive field limitations when handling large-scale anomalies. Therefore, SeRNet first locates these large-scale anomalies using the segmentation network and then utilizes a repair module for reconstruction. This way, we effectively avoid the receptive field limitations faced by the reconstruction network during the rebuilding process, thereby enhancing reconstruction quality and improving anomaly detection accuracy. Overall, this paper’s contributions can be summarized as follows:

(1) This paper presents SeRNet, an innovative network architecture that addresses the reconstruction challenge in large-scale anomaly detection through the collaborative design of the segmentation sub-network and the repair module, providing a new theoretical framework for industrial anomaly detection.

(2) This paper introduces two novel pseudo-anomaly generation methods, PPA and DPA, which optimize pseudo-anomaly generation strategies for segmentation and reconstruction tasks, significantly enhancing the model’s ability to capture anomalous features across varying scales.

(3) Extensive experiments validate the accuracy and robustness of SeRNet. The results indicate that SeRNet performs exceptionally well on the MVTec AD dataset, achieving an image-level AUROC score of up to 99.6. SeRNet demonstrates high performance across multiple benchmarks, showcasing superior generalization in complex industrial environments with a robust and efficient framework for anomaly detection.

The rest of this paper is organized as follows: Section 2 reviews the related work. Section 3 introduces the proposed SeRNet. Section 4 presents comparative and ablation experiments. Conclusions are provided in Section 5.

2. Related Work

2.1. Embedding-Based Methods

Embedding-based methods compress normal features into a lower-dimensional space in which the anomalous features are noticeably distant from the clusters formed by the normal features. PatchCore [20] utilizes a pre-trained model to extract features and construct a memory bank of patch features. This approach then calculates the difference between the test samples and the feature bank using the nearest neighbor method. The incorporation of a memory bank boosts algorithmic performance but also raises space and time complexity. Padim [21] introduced a patch distribution modeling framework that utilizes a pre-trained patch feature embedding model and a multivariate Gaussian distribution, enabling a probabilistic representation of normal samples to be obtained. This method models the fixed positions of feature maps, and it is only adapted to aligned datasets. Tsai et al. [22] proposed a representation learning method that extracts features from normal samples based on multi-scale patches and also evaluated feature similarity among the patches to achieve superior anomaly segmentation results. Through multi-scale feature extraction, the model can capture anomalies at different scales, thereby achieving satisfactory results in anomaly detection. The PyramidFlow [12] methods employ flow models to learn the spatial distribution of features in normal samples. Both multi-scale and feature pyramid tricks are often used to extract image features and perceive multi-scale anomalies. Compared to traditional convolutional neural networks, flow models require more computational resources, which, to some extent, limits their applicability. RD4AD [13] uses knowledge distillation to compare the differences between anomalous and normal samples in the feature space. RD4AD [13] differs from standard knowledge distillation methods [23,24] by adopting an encoder-decoder structure, where knowledge from the pre-trained teacher model is conveyed to the student model in an inverse direction. This approach effectively filters out anomalous features, reducing their interference with the student network’s input and enabling it to learn normal feature representations more accurately. Recently, several hybrid methods have emerged. For example, MemSeg [25] combines an embedding-based feature repository with segmentation, introducing pseudo-anomalies, multi-scale feature fusion modules, and spatial attention modules. This approach simplifies semi-supervised anomaly detection into an end-to-end semantic segmentation task, making semi-supervised surface defect detection more flexible.

2.2. Reconstruction-Based Methods

Reconstruction-based methods are applied in anomaly detection by training a model to reconstruct anomalous regions as normal regions. Due to the limited availability of anomaly samples, many current reconstruction-based methods employ pseudo-anomaly generation strategies. These strategies generate synthetic anomaly samples that are then used for model training. Draem [9] uses Perlin noise [26] to generate pseudo-anomaly masks. This approach reconstructs anomalous images into normal images using a reconstruction network and utilizes a discriminative network to distinguish anomalous regions from the input and reconstructed images. However, since Draem [9] introduces an additional dataset as the source of anomalous data, this leads to a discrepancy between the distribution of pseudo-anomalies and that of real anomalies, which hinders the model’s ability to effectively learn real anomalous features. RIAD [16] uses a random mask strategy to generate pseudo-anomalies and utilizes the Structural Similarity Index Measure (SSIM) and the Gradient Magnitude Similarity (GMS) as reconstruction loss functions, thereby improving detection accuracy compared to traditional Mean Square Error (MSE) loss functions. SSIM and GMS effectively guide the image reconstruction task. However, the introduction of the mask strategy increases the number of reconstructions, potentially leading to higher algorithmic time complexity. EdgRec [15] transforms images with pseudo-anomalies into edge images and feeds them into the reconstruction network. This model discards the discriminative network and uses a new anomaly evaluation function to identify differences between the input and reconstructed images. The function takes into account both color and gradient differences to detect anomalies. Employing an anomaly evaluation function to identify image differences not only enhances the model’s interpretability but also effectively mitigates the risk of network overfitting. Xian et al. [27] proposed a new framework called the Dual-Siamese network. The first Siamese architecture captures distinguishing features from normal samples and anomalous samples generated by the pseudo-defect generation module. The subsequent second Siamese architecture reconstructs and maps the dual-density features. This framework avoids the inaccuracies present in existing image reconstruction-based methods and addresses issues found in discriminative embedding-based approaches. Behrendt et al. [28] introduced diffusion models into anomaly detection and proposed a conditioning mechanism that incorporates additional feature representations of the input image into the denoising process of DDPMs. This effectively addresses the challenges faced by DDPMs in capturing precise intensity characteristics and domain adaptation. As a result, this method has achieved remarkable success in medical anomaly detection. CFRDC [29] employs a reconstruction-based approach, effectively addressing the issue of class increment in anomaly detection scenarios. It demonstrates the ability to learn features of new classes while retaining knowledge of previously learned classes, underscoring the strong adaptability and practicality of reconstruction-based methods.

2.3. Pseudo-Anomaly-Based Methods

The pseudo-anomaly-based approach generates simulated anomaly samples during the training phase, enabling the model to learn the differences between normal and anomalous samples. CutPaste [30] and AnoSeg [31] simulate anomalous samples by randomly copying small rectangular regions from the input image and pasting them elsewhere within the image. While this approach is simple and efficient, the generated anomalies often differ significantly from real anomalies. GLASS [32] integrates global anomalies with local anomalies, employing gradient ascent and truncated projection to generate anomaly samples that closely resemble the normal distribution, thereby significantly improving anomaly detection and localization performance. Hu et al. [18] proposed a pseudo-anomaly generation strategy based on diffusion models. This method leverages a pre-trained latent diffusion model and incorporates spatial anomaly embedding and adaptive attention reweighting mechanisms, enabling the generation of realistic and diverse anomaly samples even with limited anomalous data.

3. Methods

Figure 2 shows the overall structure of the proposed SeRNet, which comprises a segmentation sub-network, a reconstruction sub-network, and a repair module that connects the two sub-networks. To facilitate the introduction of SeRNet,

Table 1. Symbol descriptions.

Symbol	Definition
$I_t$, $I_a$	The input image, the (pseudo) anomalous image,
$I_s$, $I_b$	The similar image, the repaired image
$I_p$, I	The reconstructed image, the normal image
$M_t$, $M_p$	The ground truth mask, the prediction mask
$M_a$, $M_d$	The pseudo-anomaly mask of PPA and DPA
$t{h}_p$, $t{h}_a$	Binarization threshold, filtering threshold
$I_p^a$, $I_p^b$	The a-channel and b-channel of $I_p$ in CLELAB color space.

lists the key symbols along with their definitions.

3.1. Pseudo-Anomaly Generation Strategy

SeRNet’s segmentation sub-network is responsible for segmenting large-scale anomalies, while its reconstruction sub-network is responsible for reconstructing small-scale anomalies. In this study, two pseudo-anomaly generation methods are used. The PPA method is used to generate large-scale pseudo-anomalies for the segmentation sub-network, while the DPA method is used to generate small-scale pseudo-anomalies for the reconstruction sub-network.

Inspired by [9], PPA uses Perlin noise [26] to generate anomaly masks. However, unlike [9], the PPA method restricts the generation of small-scale pseudo-anomalies by filtering the pseudo-anomaly masks such that the scale of the generated pseudo-anomalies is consistent with the segmentation sub-network’s large-scale anomaly detection task. Furthermore, the PPA approach improves upon the choice of pseudo-anomalous data sources compared to Draem [9]. Draem [9] utilizes the DTD dataset [33] as a source for pseudo-anomalies. However, the data distribution of the DTD dataset differs significantly from that of industrial products in practice as the distribution of real defects, such as breaks, scratches, and misalignment, is similar to that of normal areas. PPA addresses this issue by employing images from the same class as the anomaly data source. This approach allows pseudo-anomalies to be generated that better align with the true distribution of anomalies. To ensure diversity among the pseudo-anomaly data sources, images of the same class are first divided into four pieces and randomly combined to create spatial differences. The resulting images are then randomly enhanced with brightness, contrast, color, and affine transformations. After these processing steps, images from the same class can be used as a data source for pseudo-anomaly generation. Figure 3a shows the process of generating pseudo-anomalies using PPA, which can be expressed as:

$$I_a=\left(1-\beta \right)\left(M_a\odot I\right)+\beta \left(M_a\odot I_A\right)+{\overline{M}}_a\odot I$$

(1)

where $M_a$ denotes the pseudo-anomaly mask generated by Perlin noise, ${\overline{M}}_a$ denotes the inverse result of $M_a$, I represents the normal image, $I_A$ represents the (enhanced) image of the same class, ⊙ indicates the element-by-element multiplication operation, and $\beta$ represents a randomized weight in the range [0, 0.8].

In contrast, the DPA method creates one or two rectangular masks that cover less than 2.5% of the image area. It then generates two to four randomly proportioned rectangular masks immediately adjacent to these rectangles to form an irregular region. These irregular regions represent pseudo-anomaly masks. The process of gradually changing from a single rectangular mask to an irregular mask can be vividly summarized as diffusion, which is the origin of the name DPA.

DPA then randomly adjusts the color and lightness of these image regions to generate pseudo-anomalies. To train the reconstruction sub-network, large-scale anomalies can be generated using PPA to ensure the creation of splicing anomalies during the repair phase. Additionally, the DPA method can be used to generate small-scale pseudo-anomalies, thus enabling the reconstruction sub-network to prioritize image details.

DPA anomaly generation process is shown in Figure 3b and can be expressed as:

$$I_a={\overline{M}}_d\odot I+\left[\left(M_d\odot I\right)+\Delta bri+\Delta col\right]$$

(2)

where $M_d$ denotes the pseudo-anomaly mask randomly generated by DPA, ${\overline{M}}_d$ represents the inverse result of $M_d$, I is the normal image, $\Delta bri$ denotes the random brightness variation value, $\Delta col$ denotes the random color variation value, and ⊙ signifies the elementwise multiplication operation.

3.2. Segmentation Sub-Network and Reconstruction Sub-Network

As shown in Figure 4, both the segmentation sub-network and the reconstruction sub-network of SeRNet adopt the classical U-net architecture, which comprises an encoder and a decoder. The encoder comprises convolutional and pooling layers that extract image features and reduce the image’s scale to create a latent feature space. The decoder consists of convolutional and upsampling layers that decode the image’s spatial dimensions and details from the latent feature space.

The reconstruction sub-network suppresses the expression of anomalous features through the downsampling layers of the encoder and then reconstructs the normal image from the latent feature space via the upsampling layers of the decoder. However, large-scale anomalies still behave as anomalous features in the latent feature space. During the decoder’s upsampling process, the restricted receptive field causes the current pixel values to be influenced by surrounding pixels, making large-scale anomalies more likely to be reconstructed as anomalies. Consequently, the reconstruction sub-network performs well in reconstructing small-scale anomalies but struggles with large-scale anomalies. To assist in reconstructing large-scale anomalies, the segmentation sub-network is responsible for localizing them. This is attributed to the fact that large-scale anomalies exhibit more distinctive features and stronger boundaries, making them easier for the segmentation sub-network to identify.

The segmentation sub-network of SeRNet only produces an anomaly mask as its output. In addition, to reduce the total number of parameters in SeRNet, the encoder in the segmentation sub-network uses only two max-pooling layers and three convolutional layers, while the decoder contains two upsampling layers, two convolutional layers, and one output convolutional layer.

Unlike the segmentation sub-network, the reconstruction sub-network must output reconstructed three-channel color images, which is a relatively complex task. Thus, the reconstruction sub-network’s encoder comprises five convolutional layers and four max-pooling layers, while its decoder consists of four convolutional layers, four upsampling layers, and one output convolutional layer. However, increasing the number of layers may also result in a loss of image information during downsampling, leading to distortion of the reconstructed image. To address this issue, this study introduces skip connections, which link the feature maps of each layer of the encoder directly to the feature maps of the corresponding layer of the decoder. This enables the decoder to reconstruct fine-grained features from the feature maps of lower layers via the skip connection path. Reconstruction sub-networks with skip connections are correspondingly more effective at capturing image details.

The segmentation sub-network and the reconstruction sub-network are optimized separately. The segmentation sub-network is trained first, and after the training is complete, it infers and outputs the anomaly segmentation map, which assists in training the reconstruction sub-network. To ensure the consistency of SeRNet during the training phase, the same loss function is used to assess both sub-networks. The total loss comprises two components: the MSE loss and the SSIM loss [16]. These components can be expressed as the following:

$$Loss\left(y_p,y_t\right)=SSIM\left(y_p,y_t\right)+MSE\left(y_p,y_t\right)$$

(3)

where $y_p$ denotes the reconstructed image or predicted mask, and $y_t$ denotes ground truth image or mask.

3.3. Repair Module

The repair module acts as a crucial link that connects both sub-networks. This module has multiple functions, including prediction mask binarization, filtering, and large-scale anomaly repair. The pseudo code for the repair module is provided in Algorithm 1. The inputs are the prediction mask $M_p$, the anomalous image (or pseudo-anomalous image) $I_a$, and the similar image $I_s$ of $I_a$. Firstly, $M_p$ is binarized in steps 1–5. Subsequently, in steps 6–10, prediction masks with small-scale anomalies are filtered and the anomalies are eliminated. Finally, in steps 11–17, $I_a$ is repaired utilizing $I_s$ under the guidance of $M_p$, and the resulting repaired image $I_b$ is returned.

Algorithm 1 Repair strategy

Require:  $M_p$: the prediction mask. $I_a$: the (pseudo) anomalous image. $I_s$: the similar image of $I_a$. Ensure:  $I_b$: the repaired image of $I_a$.        # Prediction mask filtering   1:     if  $M_p[i,j]<t{h}_p$  then   2:          $M_p[i,j]\leftarrow 0$   3:     else   4:          $M_p[i,j]\leftarrow 1$   5:     end if # Prediction mask filtering # $S_w$ denotes the connected region in $M_p$ # $AREA\left(S_w\right)$ means calculate the area of $S_w$   6:     for $S_w$ in $M_p$ do   7:         if $AREA\left(S_w\right)<t{h}_a$ then   8:             $M_p\left[S_w\right]\leftarrow 0$   9:         end if 10:     end for 11:     for $item$ in $M_p$ do 12:         if $M_p\left[item\right]==0$ then 13:             $I_b\left[item\right]\leftarrow I_a\left[item\right]$ 14:         else 15:             $I_b\left[item\right]\leftarrow I_s\left[item\right]$ 16:         end if 17:     end for

The pixels in the prediction mask $M_p$ output from the segmentation sub-network all have values in the range [0, 1], which indicate the probability of a pixel belonging to an anomalous region. This paper utilizes binarization to handle the continuous distribution of pixel values in $M_p$. Specifically, a threshold $t{h}_p$ is introduced, where pixel values below the threshold are set to 0, and those above it are set to 1. This treatment prevents anomalous regions with low probability from introducing additional anomalies during the repair phase. The binarized version of $M_p$ must also be filtered. To do so, if the area of an anomalous region in $M_P$’s is below a set threshold $t{h}_a$, the corresponding pixel values of the connected area should be set to 0, thus indicating that these small-scale anomalies should be excluded. Since the reconstruction sub-network can reconstruct these small-scale anomalies into normal regions, it is unnecessary to repair them, thus avoiding potentially introducing additional anomalies. Figure 5 shows a comparison of the original and processed prediction masks. As shown, the non-binarized and unfiltered masks contain very high noise levels, while the processed masks contain only large-scale masks representing suspected anomalous regions.

The repair operation replaces anomalous regions of $I_a$ with corresponding regions from similar image $I_s$ under the guidance of $M_p$. The sources of $I_s$ are explained in Section 3.4. The repair result is shown in Figure 6. Repairing anomalous regions using the similar images approach effectively assists in the reconstruction of large-scale anomalies.

The prediction masks assist in repairing large-scale anomalies by providing location information about these anomalies, thereby supporting the reconstruction network in transforming anomalous areas into normal ones. Without the aid of the prediction masks, the reconstruction network would struggle to effectively achieve this transformation, potentially leading to errors in anomaly detection. Therefore, the prediction masks play a crucial role in the repair module. The repair module consists of two main operations: first, processing the prediction masks generated by the segmentation sub-network, and second, using the processed prediction masks to replace anomalous regions with normal ones to repair large-scale anomalies. The objective of the repair module is to minimize the introduction of additional anomalies while effectively repairing large-scale anomalies.

3.4. Inference

The inference phase of SeRNet includes the following additional operations.

3.4.1. Similar Image Search

As noted in Section 3.3, SeRNet needs to repair large-scale anomalies using normal regions from similar images. During the training phase, the image $I_t$ generates the pseudo-anomalous image $I_a$; the corresponding region of $I_t$ is then used to repair the large-scale anomalies in $I_a$.

During the inference phase, SeRNet searches for the training image that is the most similar to the test image. To achieve this, a perceptual hash algorithm [34] is used. The algorithm generates a unique perceptual hash value for each image based on the appearance and location of the target object, ensuring that the test image can accurately retrieve similar images from the training set. In the experiments, perceptual hashing for encoding images is implemented using the default settings of the version 4.3.1 of ImageHash (ImageHash library: https://pypi.org/project/ImageHash/ (accessed on 1 February 2026)) library.

For the MVTec AD [35] dataset, which contains 3416 training images, only 217 KB of storage is needed for its hash values, equating to an average of 0.06 KB per image. This is because the perceptual hash algorithm can convert an image into a 16-bit hash string, and computers are highly efficient at storing strings, allowing for significant reductions in storage space.

Algorithm 2 outlines the steps for executing a similar image search algorithm. This algorithm takes the set of training samples $D_t$ and the anomalous image $I_a$ as inputs. Before the inference process, steps 1–4 are executed to compute the perceptual hash value of each training image and save it in the corresponding file, which avoids the need for repeated computation during the inference stage. During inference, Step 5 is executed to calculate the hash value of the current test image. Step 6 is then performed to search the file containing the stored hash value to find the image with the smallest Hamming distance from the $I_a$ hash value, i.e., the most similar image $I_s$. Figure 7 illustrates the result of this search. A similar image search algorithm can find similar images for screws in the first column and hazelnuts in the fourth column of Figure 7 when both have different orientations, thus playing a crucial role in ensuring accurate repairs for large-scale anomalies.

Algorithm 2 Similar image search

Require:  $D_t$: the set of training samples in the same category. $I_a$: the anomalous image. Ensure:  $F_h$: perceptual hash value storage files. $I_s$: the similar image of $I_a$.        # Compute the perceptual hash of the training samples in advance and store it. 1:      for I in $D_t$ do 2:          $has{h}_I\leftarrow CalculateHash\left(I\right)$ 3:          $SaveFile(has{h}_I,Name\left(I\right),F_h)$ 4:      end for # At the point of inference, the perceptual hash of the present image $I_a$ is calculated. 5:      $has{h}_{I_a}\leftarrow CalculateHash\left(I_a\right)$ #$MinHashDis\left(\right)$ means to find the minimum Hamming distance 6:      $I_s\leftarrow MinHashDis(has{h}_{I_a},F_h)$

3.4.2. Anomaly Evaluation

SeRNet reconstructs the test image during the inference phase and then uses an anomaly evaluation function [15] to compare the differences between the original and reconstructed images. This comparison will determine the presence of any anomalies. The anomaly evaluation function measures the color differences and structural differences between the test image and the reconstructed image. To represent the color difference, the anomaly evaluation function calculates the difference between the a-channel and b-channel of the two images in the CIELAB color space, which can be expressed as:

$$D_{color}=\left[{\left(I_p^a-I_t^a\right)}^2+{\left(I_p^b-I_t^b\right)}^2\right]\ast F_{mean}$$

(4)

where $F_{mean}$ denotes the filtering process using a convolutional kernel of size 21.

The anomaly evaluation function calculates the image’s structural differences based on the image’s multi-scale gradient magnitude similarity (MSGMS) [16], which can be expressed as:

$$D_{structure}=1_{H\times W}-\mathrm{MSGMS}\left(I_p,I_t\right)\ast F_{mean}$$

(5)

where $MSGMS\left(I_p,I_t\right)$ denotes the computation of the multiscale gradient magnitude similarity of $I_p$ and $I_t$, with the convolutional kernel size of $F_{mean}$ set to 21.

The anomaly evaluation function can be expressed as:

$$D_{score}=c{D}_{color}+D_{structure}$$

(6)

where c is taken as 0.001 to balance the difference between the orders of magnitude of $D_{color}$ and $D_{structure}$. The maximum value of $D_{score}$ represents the anomaly score of the test image.

4. Experiments

4.1. Experimental Details

Datasets: MVTec AD [35] is widely used for detecting anomalies in industrial products and consists of fifteen classes. Its training set comprises 3629 normal images, while its test set comprises 1725 images, including multiple anomalies for each class.

VisA [36] comprises a total of 10,821 images across twelve industrial classes. The training set includes 8659 normal images, while the test set consists of 962 normal images and 1200 anomalous images.

MPDD [37] contains 1346 images of metal parts categorized into six groups. The training set includes 888 normal images, and the test set contains 176 normal images and 282 anomalous images.

BTAD [38] contains three classes of industrial products. These three classes contain 400, 1000, and 399 normal images, respectively. The scale of anomalies in the same class varies widely, creating a challenge for anomaly detection.

The types of anomalies in the four datasets mainly include scratches, wear, cracks, and missing parts, with the ground truth represented by mask images indicating the actual anomalies.

Implementation Details: The SeRNet model was implemented using Python 3.8 and the PyTorch 1.11 frameworks on an Intel(R) Core i7-9700K CPU and an Nvidia RTX 2080 Ti GPU. The training process was conducted for 600 epochs, with an initial learning rate of 0.0001. The experimental setup has a batch size of four, and the optimizer used is Adam. The generation of pseudo-anomalies has been enhanced using methods from the imgaug library, with specific methods, including GammaContrast, MultiplyAndAddToBrightness, AddToHueAndSaturation, Autocontrast, and Affine Transformation. The input training images were resized to a resolution of $256\times 256$ pixels. Section 3.3 employs a binarization threshold of 0.8 and an area filtering threshold of 7000.

4.2. Experimental Results

4.2.1. Comparison with State-of-the-Art Methods

Table 2 shows the experimental results of SeRNet on the MVTec AD dataset. Notably, SeRNet achieves a perfect I-AUROC score of 100 across seven different item classes, representing a significant performance improvement over reconstruction-based methods [9,18]. Specifically, SeRNet surpasses MUTAD [19] by 3.1 in I-AUROC on average, exceeds DTAM [39] by 1.8, outperforms AnomalyDiffusion [18] by 1.1 on average in texture categories, and surpasses SuperSimpleNet [40] by 1.7 on average in object categories. The AP scores shown in Table 3 also demonstrate the strong anomaly localization capability of SeRNet. Compared to the knowledge distillation-based STPM [23] and ST [41], SeRNet surpassed their scores by 20.5 and 29.3, respectively. In comparison to the reconstruction-based RIAD [16], SeRNet showed an AP improvement of 26.6.

As shown in Table 4, SeRNet performs well on the VisA dataset. In terms of the average I-AUROC, SeRNet outperforms LightFlow [42] and FastFlow [43], which are based on the flow model, by 2.8 and 15.8, respectively. Table 5 shows the experimental results for the MPDD dataset, where SeRNet obtains an I-AUROC of 100.0 for both connector and metal plate classes. Its average I-AUROC and P-AUROC are 12.5 and 1.2 higher than those of EffAD [44], and 10.8 and 0.8 higher than those of CFlow [45], respectively.

Table 2. Results of anomaly detection for different methods on the MVTec AD dataset (I-AUROC), with the best results highlighted in bold.

Class	Draem [9]	DTAM [39]	AnomalyDiffusion [18]	MUTAD [19]	MSPBA [22]	FastRecon [46]	DBKD [47]	SuperSimpleNet [40]	SeRNet (Ours)
Carpet	97.0	98.0	96.7	97.7	93.4	99.9	97.1	98.4	98.8
Grid	99.9	96.0	98.4	98.3	100.0	99.7	88.8	99.3	100.0
Leather	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Tile	99.6	100.0	100.0	99.9	96.2	100.0	99.5	99.7	100.0
Wood	99.1	100.0	98.4	99.9	99.7	99.3	99.1	99.3	100.0
Avg. Text.	99.1	98.8	98.7	99.2	97.7	97.6	98.0	99.3	99.8
Bottle	99.2	100.0	99.8	100.0	100.0	99.4	100.0	100.0	100.0
Capsule	98.5	85.4	99.7	96.5	97.2	90.1	98.0	98.1	98.4
Pill	98.9	96.8	98.0	95.7	97.7	93.5	98.3	98.1	98.7
Transistor	93.1	96.8	100.0	95.1	98.9	97.7	99.8	99.9	99.9
Zipper	100.0	99.5	99.9	98.5	99.5	97.0	98.6	99.6	99.2
Cable	91.8	95.3	100.0	98.3	98.8	93.8	98.0	98.1	99.7
Hazelnut	100.0	100.0	99.8	99.3	99.6	99.3	100.0	99.8	99.9
Metal_nut	98.7	100.0	100.0	97.4	97.8	99.1	100.0	99.5	100.0
Screw	93.9	99.8	96.8	87.2	94.1	62.5	95.5	92.9	99.9
Toothbrush	100.0	100.0	100.0	83.8	100.0	93.6	92.5	92.2	100.0
Avg. Obj.	97.4	97.4	99.4	95.2	98.4	92.6	98.1	97.9	99.6
Average	98.0	97.8	99.2	96.5	98.2	94.2	98.1	98.4	99.6

Table 2. Results of anomaly detection for different methods on the MVTec AD dataset (I-AUROC), with the best results highlighted in bold.

Class	Draem [9]	DTAM [39]	AnomalyDiffusion [18]	MUTAD [19]	MSPBA [22]	FastRecon [46]	DBKD [47]	SuperSimpleNet [40]	SeRNet (Ours)
Carpet	97.0	98.0	96.7	97.7	93.4	99.9	97.1	98.4	98.8
Grid	99.9	96.0	98.4	98.3	100.0	99.7	88.8	99.3	100.0
Leather	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Tile	99.6	100.0	100.0	99.9	96.2	100.0	99.5	99.7	100.0
Wood	99.1	100.0	98.4	99.9	99.7	99.3	99.1	99.3	100.0
Avg. Text.	99.1	98.8	98.7	99.2	97.7	97.6	98.0	99.3	99.8
Bottle	99.2	100.0	99.8	100.0	100.0	99.4	100.0	100.0	100.0
Capsule	98.5	85.4	99.7	96.5	97.2	90.1	98.0	98.1	98.4
Pill	98.9	96.8	98.0	95.7	97.7	93.5	98.3	98.1	98.7
Transistor	93.1	96.8	100.0	95.1	98.9	97.7	99.8	99.9	99.9
Zipper	100.0	99.5	99.9	98.5	99.5	97.0	98.6	99.6	99.2
Cable	91.8	95.3	100.0	98.3	98.8	93.8	98.0	98.1	99.7
Hazelnut	100.0	100.0	99.8	99.3	99.6	99.3	100.0	99.8	99.9
Metal_nut	98.7	100.0	100.0	97.4	97.8	99.1	100.0	99.5	100.0
Screw	93.9	99.8	96.8	87.2	94.1	62.5	95.5	92.9	99.9
Toothbrush	100.0	100.0	100.0	83.8	100.0	93.6	92.5	92.2	100.0
Avg. Obj.	97.4	97.4	99.4	95.2	98.4	92.6	98.1	97.9	99.6
Average	98.0	97.8	99.2	96.5	98.2	94.2	98.1	98.4	99.6

Table 3. Results of anomaly location for different methods on the MVTec AD dataset (AP), with the best results highlighted in bold.

Method	RIAD [16]	Padim [21]	ST [41]	UTRAD [48]	STPM [23]	SeRNet
Carpet	61.4	60.7	52.2	17.2	65.3	75.7
Grid	36.4	35.7	10.1	15.4	45.4	58.5
Leather	49.1	53.5	40.9	11.9	42.9	77.4
Tile	52.6	52.4	65.3	52.0	61.7	87.7
Wood	38.2	46.3	53.3	18.3	47.0	76.2
Avg. Text.	47.5	49.7	44.4	23.0	52.5	75.1
Bottle	76.4	77.3	74.2	37.8	80.6	89.1
Capsule	38.2	46.7	25.9	25.2	35.9	55.6
Pill	51.6	61.2	62.0	52.5	63.3	80.8
Transistor	39.2	72.0	27.1	49.7	44.4	75.9
Zipper	63.4	58.2	36.1	35.6	54.9	72.4
Cable	24.4	45.4	48.2	62.4	58.0	74.6
Hazelnut	33.8	61.1	57.8	45.3	60.3	85.9
Metal_nut	64.3	77.4	83.5	65.7	79.3	89.6
Screw	43.9	21.7	7.8	1.7	26.9	47.8
Toothbrush	50.6	54.7	37.7	30.6	48.8	75.1
Avg. Obj	48.6	57.6	46.0	40.7	55.2	74.7
Average	48.2	55.0	45.5	34.8	54.3	74.8

Table 3. Results of anomaly location for different methods on the MVTec AD dataset (AP), with the best results highlighted in bold.

Method	RIAD [16]	Padim [21]	ST [41]	UTRAD [48]	STPM [23]	SeRNet
Carpet	61.4	60.7	52.2	17.2	65.3	75.7
Grid	36.4	35.7	10.1	15.4	45.4	58.5
Leather	49.1	53.5	40.9	11.9	42.9	77.4
Tile	52.6	52.4	65.3	52.0	61.7	87.7
Wood	38.2	46.3	53.3	18.3	47.0	76.2
Avg. Text.	47.5	49.7	44.4	23.0	52.5	75.1
Bottle	76.4	77.3	74.2	37.8	80.6	89.1
Capsule	38.2	46.7	25.9	25.2	35.9	55.6
Pill	51.6	61.2	62.0	52.5	63.3	80.8
Transistor	39.2	72.0	27.1	49.7	44.4	75.9
Zipper	63.4	58.2	36.1	35.6	54.9	72.4
Cable	24.4	45.4	48.2	62.4	58.0	74.6
Hazelnut	33.8	61.1	57.8	45.3	60.3	85.9
Metal_nut	64.3	77.4	83.5	65.7	79.3	89.6
Screw	43.9	21.7	7.8	1.7	26.9	47.8
Toothbrush	50.6	54.7	37.7	30.6	48.8	75.1
Avg. Obj	48.6	57.6	46.0	40.7	55.2	74.7
Average	48.2	55.0	45.5	34.8	54.3	74.8

Table 4. Results of anomaly detection and location for different methods on the VisA (I-AUROC/P-AUROC), with the best results highlighted in bold.

Method	Candle	Capsules	Cashew	Chewing Gum	Fryum	Macaroni1	Macaroni2	PCB1	PCB2	PCB3	PCB4	Pipe Fryum	Average
SPD [36]	89.1/97.3	68.1/86.3	90.5/86.1	99.3/96.9	89.8/88.0	85.7/98.8	70.8/96.0	92.7/97.7	87.9/97.2	85.4/96.7	99.1/89.2	95.6/95.4	87.8/93.8
OminAL [49]	85.1/90.5	87.9/98.6	97.1/98.9	94.9/98.7	97.0/89.3	96.9/98.9	89.9/99.1	96.6/98.7	99.4/83.2	96.9/98.4	97.4/98.5	91.4/99.1	94.2/96.0
FastFlow [43]	92.8/94.9	71.2/75.3	91.0/91.4	91.4/98.6	88.6/97.3	98.3/97.3	86.3/89.2	77.4/75.2	61.9/67.3	74.3/94.8	80.9/89.9	72.0/87.3	82.2/88.2
SimpleNet [50]	96.9/98.6	89.5/99.2	94.8/99.0	100.0/98.5	96.6/94.5	97.6/99.6	83.4/96.4	99.2/99.8	99.2/98.8	98.6/99.2	98.9/98.6	99.2/99.3	96.2/98.5
LightFlow [42]	96.2/98.9	93.6/99.5	93.5/99.1	98.3/98.9	95.8/95.1	97.4/99.5	92.5/99.1	95.1/95.4	94.1/98.5	91.1/98.2	96.5/97.1	97.8/98.6	95.2/98.5
PatchCore [20]	98.7/99.2	68.8/96.5	97.7/99.2	99.1/98.9	91.6/95.9	90.1/98.5	63.4/93.5	96.0/99.8	95.1/98.4	93.0/98.9	99.5/98.3	99.0/99.3	91.0/98.1
SeRNet (Ours)	97.0/98.8	96.4/99.8	99.0/98.8	99.6/99.0	98.3/95.9	98.5/99.8	94.2/99.6	98.4/99.7	98.5/98.9	99.4/99.2	98.8/99.2	98.2/99.6	98.0/99.0

Table 4. Results of anomaly detection and location for different methods on the VisA (I-AUROC/P-AUROC), with the best results highlighted in bold.

Method	Candle	Capsules	Cashew	Chewing Gum	Fryum	Macaroni1	Macaroni2	PCB1	PCB2	PCB3	PCB4	Pipe Fryum	Average
SPD [36]	89.1/97.3	68.1/86.3	90.5/86.1	99.3/96.9	89.8/88.0	85.7/98.8	70.8/96.0	92.7/97.7	87.9/97.2	85.4/96.7	99.1/89.2	95.6/95.4	87.8/93.8
OminAL [49]	85.1/90.5	87.9/98.6	97.1/98.9	94.9/98.7	97.0/89.3	96.9/98.9	89.9/99.1	96.6/98.7	99.4/83.2	96.9/98.4	97.4/98.5	91.4/99.1	94.2/96.0
FastFlow [43]	92.8/94.9	71.2/75.3	91.0/91.4	91.4/98.6	88.6/97.3	98.3/97.3	86.3/89.2	77.4/75.2	61.9/67.3	74.3/94.8	80.9/89.9	72.0/87.3	82.2/88.2
SimpleNet [50]	96.9/98.6	89.5/99.2	94.8/99.0	100.0/98.5	96.6/94.5	97.6/99.6	83.4/96.4	99.2/99.8	99.2/98.8	98.6/99.2	98.9/98.6	99.2/99.3	96.2/98.5
LightFlow [42]	96.2/98.9	93.6/99.5	93.5/99.1	98.3/98.9	95.8/95.1	97.4/99.5	92.5/99.1	95.1/95.4	94.1/98.5	91.1/98.2	96.5/97.1	97.8/98.6	95.2/98.5
PatchCore [20]	98.7/99.2	68.8/96.5	97.7/99.2	99.1/98.9	91.6/95.9	90.1/98.5	63.4/93.5	96.0/99.8	95.1/98.4	93.0/98.9	99.5/98.3	99.0/99.3	91.0/98.1
SeRNet (Ours)	97.0/98.8	96.4/99.8	99.0/98.8	99.6/99.0	98.3/95.9	98.5/99.8	94.2/99.6	98.4/99.7	98.5/98.9	99.4/99.2	98.8/99.2	98.2/99.6	98.0/99.0

Table 5. Results of anomaly location and location for different methods on the MPDD (I-AUROC/P-AUROC), with the best results highlighted in bold.

Class	RD4AD [13]	CFLOW [45]	EffAD [44]	SeRNet
Bracket black	50.7/96.4	96.5/97.6	85.5/97.2	94.1/98.1
Bracket brown	68.7/97.3	49.7/95.0	85.3/92.5	99.1/97.2
Bracket white	83.7/98.1	79.9/96.7	96.5/97.7	94.6/96.3
Connector	93.1/98.6	99.5/97.4	50.0/99.3	100.0/99.5
Metal plate	91.9/97.5	97.4/98.6	100.0/96.5	100.0/98.6
Tubes	89.7/99.1	99.6/99.2	95.3/99.0	99.5/99.6
Average	79.6/97.8	87.1/97.4	85.4/97.0	97.9/98.2

Table 5. Results of anomaly location and location for different methods on the MPDD (I-AUROC/P-AUROC), with the best results highlighted in bold.

Class	RD4AD [13]	CFLOW [45]	EffAD [44]	SeRNet
Bracket black	50.7/96.4	96.5/97.6	85.5/97.2	94.1/98.1
Bracket brown	68.7/97.3	49.7/95.0	85.3/92.5	99.1/97.2
Bracket white	83.7/98.1	79.9/96.7	96.5/97.7	94.6/96.3
Connector	93.1/98.6	99.5/97.4	50.0/99.3	100.0/99.5
Metal plate	91.9/97.5	97.4/98.6	100.0/96.5	100.0/98.6
Tubes	89.7/99.1	99.6/99.2	95.3/99.0	99.5/99.6
Average	79.6/97.8	87.1/97.4	85.4/97.0	97.9/98.2

SeRNet also performs well on BTAD. The experimental results are shown in

Table 6. Results of anomaly detection and location for different methods on the BTAD dataset (I-AUROC/P-AUROC), with the best results highlighted in bold.

Method	Class 01	Class 02	Class 03	Average
Fastflow [43]	99.4/97.1	82.4/93.6	91.1/98.3	91.0/96.3
PatchCore [20]	90.9/95.5	79.3/94.7	99.8/99.3	90.0/96.5
DRA [51]	-/-	-/-	-/-	94.2/75.4
MKD [52]	-/-	-/-	-/-	93.5/96.5
CFA [53]	98.1/95.9	85.5/96.0	99.0/98.6	94.2/96.8
VT-ADL [38]	97.6/99.0	71.0/94.0	82.6/77.0	83.7/90.0
PatchSVDD [54]	95.7/91.6	72.1/93.6	82.1/91.0	83.3/92.1
SeRNet (ours)	98.1/95.7	85.2/95.2	99.9/99.7	94.4/96.9

, where SeRNet achieves an excellent score of 99.9 for I-AUROC and 99.7 for P-AUROC on class 03. For both anomaly detection and anomaly localization, SeRNet achieves the highest average accuracy. This demonstrates that SeRNet is still effective for detecting anomalies with large-scale variations.

4.2.2. Qualitative Analysis

Although existing reconstruction-based methods perform well in many categories, they struggle in categories with large-scale anomalies, such as cables, bottles, and pills. This is mainly due to the limitations of auto-encoders. Figure 8 compares the reconstructed images and detection results of Draem, the reconstruction-based method [9], and SeRNet. Compared to the Draem method, SeRNet shows better reconstruction quality, leading to higher accuracy in anomaly localization. This improvement is attributed to the segmentation sub-network’s capability to locate large-scale anomalies and the repair module’s effectiveness. Furthermore, thanks to the reconstruction ability of the reconstruction sub-network, SeRNet still achieves satisfactory performance in categories containing many small-scale anomalies, such as carpets and capsules. This is illustrated in Figure 9, which presents the specific reconstruction detection results of SeRNet across four datasets, encompassing both large-scale and small-scale anomalies.

4.2.3. Complexity Analysis

Table 7. Comparative results of model testing time (s), parameters (M), and performance (I-AUROC/P-AUROC) on the MVTec AD dataset, with the best results highlighted in bold.

Method	Testing Time	Parameters	Performance
Draem [9]	0.072	138.1	98.0/96.0
CSFlow [55]	0.116	275.5	98.7/-
PatchCore [20]	0.261	116.4	99.0/98.1
AST [24]	0.823	223.6	96.0/95.2
SPADE [56]	1.256	68.9	85.5/96.5
Padim [21]	1.625	68.9	95.5/97.5
SeRNet (ours)	0.085	81.0	99.6/97.6

shows the test time for a single image using SeRNet. Compared to CSFlow [55], PatchCore [20], AST [24], SPADE [56] and Padim [21], SeRNet has a significantly reduced inference time. This is because CSFlow [55] and AST [24] use complex flow models, whereas Patchcore [20], Padim [21], and SPADE [56] rely on image patches, leading to multiple computations per image. Additionally, since Padim [21] is a modification of SPADE [56], both share the same backbone network and parameters. Draem [9] uses an end-to-end anomaly detection algorithm where the discriminative network directly outputs anomaly masks, thereby reducing the inference time. The similar image search is a key step in SeRNet. As the scale of the dataset increases, similar image search does not significantly increase the inference time. The specific reasons are as follows. Firstly, the perceptual hash values of the training images have been pre-computed, so only the perceptual hash value of the test image needs to be computed during the inference process, and the time to compute the hash value of a single image is only 1 to 2 ms. In addition, SeRNet performs image search by calculating the Hamming distance of the perceptual hash values between images, and the Hamming distance can be calculated using Numpy’s broadcast mechanism instead of a loop operation, which makes the image search operation very efficient. Secondly, when working with large datasets, although each category may contain a large number of images, these images typically display only a limited number of standardized appearances. This is because industrial products tend to have uniform designs, resulting in a high degree of visual consistency. Therefore, we can create a representative sample set by selecting a small number of images with typical appearances from the training set, using either hand-picking or cluster analysis. During the testing phase, we only need to search for similar images within this representative sample set, eliminating the need for a comprehensive search across the entire dataset.

4.3. Ablation Study

4.3.1. Model Structure

SeRNet leverages pre-segmentation results to repair large-scale anomalies, which in turn helps the image reconstruction process. This section presents an ablation study to demonstrate the efficacy of SeRNet’s segmentation and reconstruction sub-network structure. The segmentation sub-network, reconstruction sub-network, and SeRNet were tested on the MVTec AD dataset, the results of which are shown in

Table 8. Results of ablation experiments on the MVTec AD dataset with the SeRNet structure (Image/Piexl-level AUROC). The checkmark (✓) denotes settings included in the experiment, and bold formatting indicates the best performance.

Segmentation Sub-Network	Reconstruction Sub-Network	Image/Pixel-Level
✓		77.6/75.3
	✓	98.1/95.1
✓	✓	99.6/97.6

. The experimental results show that the segmentation network alone achieves poor performance. This likely relates to the simple structure of SeRNet’s segmentation sub-network, which is designed for large-scale anomaly detection and may struggle to detect small-scale anomalies. In comparison, the performance of the reconstruction sub-network when used alone is better than that of the segmentation sub-network when used alone, as the reconstruction sub-network has a more complex structure and more parameters, resulting in better generalization performance. SeRNet, which combines both the segmentation and reconstruction sub-networks, achieved an I-AUROC that is 1.5 higher and a P-AUROC that is 2.5 higher than the reconstruction sub-network alone, demonstrating the significant guiding role of the segmentation sub-network.

4.3.2. Repair Module

This section explores the role of SeRNet’s repair module, conducting ablation experiments on the MVTecAD dataset and the BTAD dataset, with results shown in

Table 9. Results of ablation experiments on the MVTec AD dataset with the repair module (Image/Piexl-level AUROC). When the similar images search approach is not used, random images from the same class are used instead of similar images. The checkmark (✓) denotes settings included in the experiment, and bold formatting indicates the best performance.

Similar Images Search	Prediction Masks Processing	Image/Pixel-Level
		93.0/96.5
✓		94.1/96.9
	✓	98.3/97.1
✓	✓	99.6/97.6

and

Table 10. Results of ablation experiments on the BTAD dataset with the repair module (Image/Piexl-level AUROC). When the similar images search approach is not used, random images from the same class are used instead of similar images. The checkmark (✓) denotes settings included in the experiment, and bold formatting indicates the best performance.

Similar Images Search	Prediction Masks Processing	Image/Pixel-Level
		87.7/92.5
✓		90.3/94.5
	✓	92.1/95.7
✓	✓	94.4/96.9

. When similar image searching is not utilized, random normal images from the same class are used as substitutes. As shown, SeRNet performs worst when the prediction mask is not processed, and random image regions are used for anomaly repair. This mainly occurs because the absence of prediction masks introduces abundant fine-scale noise that hampers the image reconstruction process. Both using only similar images and processing the prediction masks individually led to improved performance. However, neither approach was able to fully eliminate the extra noise. The optimal performance of SeRNet is achieved only when both components are used together, thus demonstrating the effectiveness of the repair module. Connecting the two sub-networks without processing would not significantly improve the model’s overall performance.

Additionally, the results of the ablation experiments for the threshold values $t{h}_p$ and $t{h}_a$ in Algorithm 1 are shown in

Table 11. Results of ablation experiments on the MVTec AD dataset with the thresholds during prediction masks processing (Image/Piexl-level AUROC). Bold formatting indicates the best performance.

	${\mathit{th}}_{\mathit{p}}$	0.3	0.6	0.8
${\mathit{th}}_{\mathit{a}}$
1000		96.7/97.5	98.3/97.4	99.1/97.4
3000		99.5/97.4	99.5/97.5	99.4/97.3
5000		99.5/97.5	99.5/97.5	99.5/97.5
7000		99.6/97.5	99.6/97.5	99.6/97.6
8000		99.6/97.4	99.6/97.4	99.5/97.5

and

Table 12. Results of ablation experiments on the BTAD dataset with the thresholds during prediction masks processing (Image/Piexl-level AUROC). Bold formatting indicates the best performance.

	${\mathit{th}}_{\mathit{p}}$	0.3	0.6	0.8
${\mathit{th}}_{\mathit{a}}$
1000		91.2/92.1	92.6/92.5	93.3/93.9
3000		92.3/94.3	93.5/95.6	93.7/96.2
5000		93.7/94.7	94.0/97.5	94.4/96.3
7000		94.0/96.2	94.2/96.9	94.4/96.9
8000		94.2/95.4	93.8/96.4	94.2/96.6

. Table 11 indicates that when $t{h}_p$ and $t{h}_a$ are set to 0.8 and 7000, the image-level AUROC increases by 2.9, while the detection accuracy significantly declines when $t{h}_a$ is set to less than 1000 or greater than 8000. The best performance is achieved when $t{h}_a$ is set to 7000. This is because relatively larger values of $t{h}_p$ and $t{h}_a$ effectively repair larger anomalies, while reducing the additional anomalies introduced by the segmentation sub-network. However, if $t{h}_a$ is too large, it may ignore the masks generated by the segmentation network, leading to a decrease in accuracy. The ablation experiment results on the BTAD dataset shown in Table 12 are similar, indicating that the choice of thresholds is not sensitive and is closely related to both the segmentation sub-network and the reconstruction sub-network, with less dependence on the dataset. The threshold $t{h}_p$ is used to filter out segmentation regions with low confidence, so its setting is related to the segmentation sub-network and varies with changes in it. The threshold $t{h}_a$ is associated with the size of the input image and the reconstruction sub-network. When the input image size is fixed, and the reconstruction sub-network remains unchanged, the threshold $t{h}_a$ also remains constant.

4.3.3. Pseudo-Anomaly Generation Strategy

The SeRNet model training is performed using normal samples that contain pseudo-anomalies, and the generation process of these pseudo-anomalies is crucial to the model’s performance. This section explores the impact of the classical pseudo-anomaly generation methods CutPaste [30], PPA, and DPA on the detection accuracy of SeRNet. The basic idea of CutPaste involves cropping and pasting a random area of the original image into another area, generating anomalies that are obvious but insufficiently realistic. The experimental results are presented in

Table 13. Results of ablation experiments on the MVTec AD dataset with different pseudo-anomaly generation strategies (Image/Piexl-level AUROC). The checkmark (✓) denotes settings included in the experiment, and bold formatting indicates the best performance.

CutPaste [30]	PPA	DPA	Image/Pixel-Level
✓			96.7/95.8
	✓		98.1/95.2
		✓	97.9/96.3
	✓	✓	99.6/97.6

. The results indicate that the strategies proposed in this paper are highly effective for producing appropriate pseudo-anomalies that can be used to train the various sub-networks of SeRNet.

PPA can generate large-scale pseudo-anomalies, which are not lost during the downsampling process of the segmentation sub-network. As a result, the segmentation sub-network can better learn and recognize anomalous regions, thus improving the performance of the model. While DPA focuses on the generation of small-scale pseudo-anomalies, which enables the upsampling process of the reconstruction sub-network to focus on local features and better reconstruct image details. This training combination achieves the highest image-level and pixel-level AUROC. Using only CutPaste [30], PPA, or DPA methods failed to achieve optimal results with SeRNet. This finding reveals that when formulating a pseudo-anomaly generation strategy, it is essential to consider both the similarity between pseudo-anomalies and real anomalies and the characteristics of the algorithm being trained.

4.3.4. Anomaly Evaluation Function

The anomaly evaluation function consists of $D_{structure}$ and $D_{color}$. We conducted ablation experiments on these two components, with results shown in

Table 14. Results of ablation experiments on the MVTec AD dataset with different anomaly evaluation functions (Image/Piexl-level AUROC). Bold formatting indicates the best performance.

	${\mathit{D}}_{\mathit{structure}}$	${\mathit{D}}_{\mathit{color}}$	${\mathit{D}}_{\mathit{structure}}+{\mathit{cD}}_{\mathit{color}}$
Image-level	98.0	82.3	99.6
Pixel-level	96.7	89.9	97.6

. The experimental results demonstrate that utilizing $D_{structure}$ and $D_{color}$ individually results in a notable decline in the algorithm’s accuracy. Specifically, image-level AUROCs exhibited a reduction of 1.6 and 17.3, while pixel-level AUROCs demonstrated a decrease of 0.9 and 7.7, respectively. The combination of $D_{structure}$ and $D_{color}$ is most effective as it enables the comprehensive coverage of diverse anomaly types, leading to enhanced detection performance.

5. Conclusions

This paper proposes an unsupervised image anomaly detection model called SeRNet to address the challenge of reconstructing large-scale anomalies in current reconstruction sub-networks. The key contribution of this paper is to combine the advantages of segmentation and reconstruction in order to reconstruct large-scale anomalies. SeRNet employs a serial fusion approach of segmentation followed by reconstruction. The segmentation sub-network performs pre-segmentation of anomalous images to detect suspected anomalous regions, while the repair module employs prediction masks and similar images to repair the suspected large-scale anomalies. The reconstruction sub-network then reconstructs the repaired image. In addition, two methods for generating pseudo-anomalies at different scales are introduced for training SeRNet. The extensive experiments performed in this work demonstrate that SeRNet surpasses other state-of-the-art anomaly detection methods in terms of both performance and robustness.

However, SeRNet also faces some limitations and challenges. While the introduction of segmentation does enhance the performance of the reconstruction network, it also brings the risk of over-reliance. Additionally, when working with non-uniform textures, the detection accuracy of SeRNet may decline, which is a common challenge faced by most industrial anomaly detection methods. To address the limitations of SeRNet, we will leverage diffusion models [57] or transform networks in our future work to enhance reconstruction capabilities and tackle complex anomaly detection problems.