Self-Supervised Autoencoders for Visual Anomaly Detection

Abstract

We focus on detecting anomalies in images where the data distribution is supported by a lower-dimensional embedded manifold. Approaches based on autoencoders have aimed to control their capacity either by reducing the size of the bottleneck layer or by imposing sparsity constraints on their activations. However, none of these techniques explicitly penalize the reconstruction of anomalous regions, often resulting in poor detection. We tackle this problem by adapting a self-supervised learning regime that essentially implements a denoising autoencoder with structured non-i.i.d. noise. Informally, our objective is to regularize the model to produce locally consistent reconstructions while replacing irregularities by acting as a filter that removes anomalous patterns. Formally, we show that the resulting model resembles a nonlinear orthogonal projection of partially corrupted images onto the submanifold of uncorrupted examples. Furthermore, we identify the orthogonal projection as an optimal solution for a specific regularized autoencoder related to contractive and denoising variants. In addition, orthogonal projection provides a conservation effect by largely preserving the original content of its arguments. Together, these properties facilitate an accurate detection and localization of anomalous regions by means of the reconstruction error. We support our theoretical analysis by achieving state-of-the-art results (image/pixel-level AUROC of 99.8/99.2%) on the MVTec AD dataset—a challenging benchmark for anomaly detection in the manufacturing domain.

1. Introduction

The task of anomaly detection (AD) in a broad sense corresponds to searching for patterns that considerably deviate from some concept of normality. The criteria for what is normal and what is an anomaly can be very subtle and depend heavily on the application. Visual AD specifically aims to detect and locate anomalous regions in imagery data, with practical applications in the industrial, medical, and other domains [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17]. The continuous research in this area has produced a variety of methods ranging from classical unsupervised approaches like PCA [18,19,20], one-class SVM [21], SVDD [22], nearest neighbor algorithms [23,24], and KDE [25] to more recent methods including various types of autoencoders [5,12,26,27,28,29,30,31,32], deep one-class classification [33,34,35], generative models [6,13], self-supervised approaches [1,3,36,37,38,39,40,41], and others [3,7,8,9,10,11,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79].

The one-class classifiers [21,22], for example, aim at learning a tight decision boundary around the normal examples in the feature space and define a distance-based anomaly score relative to the center of the training data. The success of this approach strongly depends on the availability of suitable features. Therefore, it usually allows only for the detection of outliers that greatly deviate from the normal structure. In practice, however, we are often interested in more subtle deviations, which require a good representation of the data manifold. The same applies to combined approaches based on training deep neural networks (DNNs) with a one-class objective [35,80,81]. Although this objective encourages the network to concentrate the training data in a small region in the feature space, there is no explicit motivation for anomalous examples to be mapped outside of the decision boundary. In fact, the one-class objective gives preference to models that map its input domain to a narrower region and does not explicitly focus on separating anomalous examples from the normal data.

Recently, deep autoencoders (AEs) have been used for the task of anomaly detection in the visual domain [1,40,41]. Unlike the one-class approaches, they additionally enable the localization of anomalous regions by exploiting the pixel-wise nature of a corresponding training objective. By optimizing for the reconstruction error using anomaly-free examples, a common belief is that the corresponding network should fail to reconstruct anomalous regions in the application phase. Typically, this goal is achieved by controlling the capacity of the model either directly by reducing the size of the bottleneck layer or implicitly by imposing sparsity constraints on parts of the corresponding network [32,82,83,84,85]. However, neither of these techniques explicitly penalizes the reconstruction of anomalous regions, often resulting in poor detection. This is similar to the problem of training with the one-class objective, where no explicit mechanism exists for preventing anomalous examples from being mapped to the normal region. In fact, unsupervised trained AEs aim to compress and accurately reconstruct the input images and do not care much about the actual distinction between normal and anomalous samples. As a result, the reconstruction errors for the anomalous and normal regions can be very similar, preventing reliable detection and localization.

In this paper, we propose a self-supervised learning framework, which introduces discriminative information during training to prevent the good reconstruction of anomalous patterns in the testing phase. We begin with an observation that the abnormality of a region in the image is partially characterized by how well it can be reconstructed from the context of the surrounding pixels. Imagine an image where a small patch has been cut out by setting the corresponding pixel values to zeros. We can try to reconstruct this patch by interpolating the surrounding pixel values according to our knowledge about the distribution of the training data. If the reconstruction significantly deviates from the original content, we consider a corresponding patch to be anomalous and normal otherwise. Following this idea, we feed partially distorted images to the network during training while forcing it to recreate the original content—similar to the task of neural image completion [86,87,88,89,90,91]. However, instead of setting the individual pixel values to zeros—as for the completion task—we apply a patch transformation, which avoids introducing easily detectable artifacts. To succeed, our model must accomplish two different tasks: (a) detection of regions deviating from the expected pattern and (b) recreation of the original content from the uncorrupted areas. The second task, in particular, imposes an important regularization effect on the model. Together, they provide a powerful training signal for an accurate AD.

Technically, our approach can be seen as training a denoising autoencoder (DAE) on artificially corrupted inputs following a specific noise model. Although the general objective of DAEs allows for arbitrary stochastic corruptions, previous works [85,92,93] only considered simple unstructured noise, which can be characterized by the i.i.d. assumption on the distribution of the individual pixels. In contrast, we consider structured noise with spatial dependencies between the corruption variables implemented by a specific form of partial occlusions. Altogether, the resulting training effect regularizes the model to produce locally consistent reconstructions while replacing irregularities, therefore acting as a filter that removes anomalous patterns. Figure 1 illustrates a few examples to give a sense of visual quality of the resulting heatmaps when training according to our approach.

Figure 1. Anomaly detection results of our approach on a few images from the MVTec AD dataset. The first row shows the input images and the second row an overlay with the predicted anomaly heatmap.

To support our idea, we performed a theoretical analysis of the proposed method, which provided a number of interesting insights. Specifically, we show that the resulting model approximates a mapping, which, with an increasing number of image pixels, converges (stochastically) to the orthogonal projection of partially corrupted images onto the submanifold of normal examples. This is consistent with the findings in [85] showing that, in a non-degenerate case (i.e., for distributions $p\left(\mathit{x}\right)$ with support of nonzero volume) with i.i.d. pixel noise, the optimal reconstruction $r^{*}$ is given by a mapping that corrects the noisy inputs by pushing them toward the region of high probability density according to $r^{*}\left(\mathit{x}\right)=\mathit{x}+{\sigma }^2\nabla logp\left(\mathit{x}\right)+o\left({\sigma }^2\right)$ as ${\sigma }^2\to 0$. These results approach a point mass distribution collapsing to a lower-dimensional manifold, where we talk about projections and establish here the following complementary connection. While the gradient $\nabla logp\left(\mathit{x}\right)$ points in the direction of the maximal increase in probability at point $\mathit{x}$, the orthogonal projection onto the data manifold is characterized by the shortest distance between $\mathit{x}$ and its projected image. Therefore, we can naturally measure the abnormality of anomalous samples according to their distance from the manifold of normal examples.

Additionally, we investigate the effect of projection mappings on the segmentation accuracy of anomalous regions. Specifically, we analyze the conservation property of the orthogonal projection, which removes anomalous patterns in a way that largely preserves the original content. Furthermore, we establish a close connection of our approach to the previous autoencoding models including the contractive and denoising variants. In particular, we show that the orthogonal projection provides an optimal solution for the reconstruction contractive autoencoder.

The rest of this paper is organized as follows: In Section 2, we outline the main differences between our approach and related methods, concluding with a concise summary of our contributions. In Section 3, we formally introduce the proposed framework including training objective, data generation, and model architecture, followed by a theoretical analysis in Section 4. In Section 5, we evaluate the performance of our approach and provide a conclusion in Section 6.

2. Related Works

The plethora of existing AD methods (see [78] for an overview) can be roughly divided in three groups: probabilistic models, one-class classification methods, and reconstruction models; our approach is a member of the latter. Without going deeper into the details, we note that the first two groups appear less suitable in our case, where the data distribution is supported by a lower-dimensional embedded manifold. Due to this fact, the Lebesgue measure of the data manifold is zero excluding the existence of a density function over the input space, which takes positive values only on the manifold. On the other hand, one-class approaches implicitly assume a nonzero volume of the corresponding data in order to provide accurate detection results, which again violates our assumption about the data distribution.

Here, we focus on the reconstruction methods represented by the regularized AEs including the contractive [84] and denoising [92] variants and a number of self-supervised methods inspired by the task of neural image completion [1,40,41]. The central idea behind the contractive autoencoder (CAE), for example, is to learn a reconstruction mapping $r\left(\mathit{x}\right)=d\left(e\right(\mathit{x}\left)\right)$ composed of encoder and decoder parts by adding a regularization term $\parallel D_{\mathit{x}}{e\parallel }_F^2$ to the objective, which implicitly promotes a reduction in the magnitude of gradients ${\partial }_{\mathit{v}}e\left(\mathit{x}\right)$ in direction $\mathit{v}$ pointing outside the tangent plane on the data manifold. In contrast, the denoising autoencoder (DAE) aims to minimize the reconstruction error without regularization terms but augments the training procedure by adding stochastic noise to the inputs.

Previous work [85] demonstrated that a DAE with small noise corruption of variance ${\sigma }^2$ is similar to a CAE with penalty coefficient $\lambda ={\sigma }^2$ but where the contraction is imposed explicitly on the whole reconstruction function rather than on the encoder part alone. Our approach is also based on training a model to reconstruct original content from modified inputs but differs from the previous works on DAE by using more involved input modifications beyond the i.i.d. corruption noise. Specifically, the corruptions introduced during training may represent additive noise, stochastic occlusions, and even deterministic modifications (e.g., geometric transformations), allowing for a wider range of anomalous patterns to be detectable during the application phase.

Another group of related methods has been inspired by the task of neural image completion [86,87,88,89,90,91]. This group includes a number of self-supervised methods like LSR [3], RIAD [39], CutPaste [40], InTra [41], DRAEM [38], and SimpleNet [94], which (similar to our approach) aim at training a model to reconstruct the original content from corrupted inputs in either the latent or the input space. The main differences of these methods from our approach are in the choice of data augmentation during training and the specific network architecture of our model, which in summary ked to higher performance in our experiments on the MVTec AD dataset [95]. Another two recent studies (PatchCore) [10] and (PNI) [68] reported impressive results on this benchmark. Both are based on the usage of memory banks comprising locally aware nominal patch-level feature representations extracted from pretrained networks. In contrast, our approach neither requires pretraining nor additional storage room for the memory bank and performs on par with the two methods. In the following, we briefly summarize our contributions.

As our first contribution, we formulated an effective AD framework for a special use case (e.g., natural images), where the normal examples exist on a lower-dimensional submanifold embedded in the input space and are restricted by an additional assumption on the covariance between the spatially close components to vanish with increasing distance. While similar ideas have been used in different domains, our approach distinguishes itself by the specific form of the noise model (implemented by partial occlusions with complex shapes) and the network architecture (SDC-AE and SDC-CCM) that together produce a solution which consistently outperforms previous methods. In particular, we achieved state-of-the-art results for both detection (AUROC of 99.8%) and localization (AUROC of 99.2%) tasks on the MVTec AD dataset—a challenging benchmark for visual anomaly detection in the manufacturing domain.

As our second contribution, we performed a rigorous theoretical analysis of our method, which provided a number of interesting insights. Specifically, we show that with an increasing number of input pixels, the corresponding model resembles the orthogonal projection of partially corrupted images onto the submanifold of uncorrupted examples. This covers various forms of input modifications including partial occlusions and additive noise (see Theorem 1 and Theorem 2). At the same time, orthogonal projection acts as a filter for irregularities by removing anomalous patterns from the inputs in a way that largely preserves the original content (see Proposition 2 and Proposition 3). Together, these properties facilitate the accurate detection and localization of the anomalous regions by means of the reconstruction error, supporting our hypothesis around the training procedure.

As our third contribution, we improved upon the previous understanding of the connections between contractive and denoising autoencoders. Specifically, we identified the nonlinear orthogonal projection as an optimal solution (see Proposition 1), minimizing the training objective of the reconstruction contractive autoencoder (RCAE). On the other hand, our training objective essentially corresponds to training a DAE with a specific noise model, imposing strong spatial correlation on the corruption variables. A corresponding model, in turn, approximates a mapping that (with the increasing number of input pixels) converges stochastically to the orthogonal projection—the optimal solution of the RCAE. In contrast to the previous results [85], our findings extend beyond the i.i.d. variable noise and are not limited by the assumption of small variance.

3. Methodology

In the following, we formally introduce our self-supervised framework for training a model to detect and localize anomalous regions in images. We provide a detailed description of the objective function, the structure of artificially generated anomalies used during training, and the rationale behind our choice of model architecture.

3.1. Training Objective

We identify an autoencoder with input and output tensors corresponding to color images with a parameterized map $f_{\mathit{\theta }}:{[0,1]}^{h\times w\times 3}\to {[0,1]}^{h\times w\times 3}$, $h,w\in \mathbb{N}$. Furthermore, $\mathit{x}\in {[0,1]}^{h\times w\times 3}$ denotes an original (anomaly-free) image, and $\widehat{\mathit{x}}$ denotes a copy of $\mathit{x}$ that has been partially modified. The modified regions within $\widehat{\mathit{x}}$ are encoded through a real-valued mask $\mathit{M}\in {[0,1]}^{h\times w\times 3}$, while $\overline{\mathit{M}}:=\mathbf{1}-\mathit{M}$ denotes a corresponding complement, with $\mathbf{1}\in {\left\{1\right\}}^{h\times w\times 3}$ being a tensor of ones. Our goal is to train a model $f_{\mathit{\theta }}$ that projects arbitrary inputs from the data space to the submanifold of normal images by minimizing for each triple $(\mathit{x},\widehat{\mathit{x}},\mathit{M})$, with the following objective:

$$\begin{array}{c}\hfill \mathcal{L}(\widehat{\mathit{x}},\mathit{x},\mathit{M})={\displaystyle \frac{(1-\lambda )}{\parallel \overline{\mathit{M}}{\parallel }_1}}·\parallel \overline{\mathit{M}}\odot \left(f_{\mathit{\theta }}\left(\widehat{\mathit{x}}\right)-\mathit{x}\right){\parallel }_2^2+{\displaystyle \frac{\lambda }{{\parallel \mathit{M}\parallel }_1}}·{\parallel \mathit{M}\odot \left(f_{\mathit{\theta }}\left(\widehat{\mathit{x}}\right)-\mathit{x}\right)\parallel }_2^2,\end{array}$$

(1)

where ⊙ denotes an element-wise tensor multiplication, and $\lambda \in [0,1]$ is a hyperparameter controlling the importance of the two terms during training. Here, ${\parallel ·\parallel }_p$ denotes the $l^p$-norm on a corresponding tensor space. In terms of supervised learning, we feed a partially corrupted image $\widehat{\mathit{x}}$ as input to the network, which is trained to recreate the original image $\mathit{x}$ representing the ground truth label.

By minimizing the above objective, the corresponding autoencoder aims to interpolate between two different tasks. The first term steers the model to reproduce uncorrupted image regions $\overline{\mathit{M}}\odot \mathit{x}$, while the second term requires the network to correct the corrupted regions $\mathit{M}\odot \widehat{\mathit{x}}$ by recreating the original content. Altogether, the objective in (1) encourages the model to produce a locally consistent reconstruction of the input while replacing irregularities, acting as a filter for anomalous patterns. Figure 2 shows a few reconstruction examples produced by our model $f_{\mathit{\theta }}$. We can see how normal regions are accurately replicated, while irregularities (e.g., scratches or threads) are replaced by locally consistent patterns. During training, we use a specific procedure to generate corrupted images $\widehat{\mathit{x}}$ from normal examples $\mathit{x}$ based on randomly generated masks $\mathit{M}$ marking the corrupted regions. We provide a detailed description of this process in the next subsection.

Figure 2. Illustration of the reconstruction effect of our model trained either on the wood, carpet, or grid images (without defects) from the MVTec AD dataset.

Given a trained model $f_{\mathit{\theta }}$, we can perform AD on an input image $\widehat{\mathit{x}}$ as follows: First, we compute the difference map $\mathrm{Diff}[\widehat{\mathit{x}},f_{\mathit{\theta }}\left(\widehat{\mathit{x}}\right)]\in {\mathbb{R}}^{h\times w}$ between the input $\widehat{\mathit{x}}$ and its reconstruction $f_{\mathit{\theta }}\left(\widehat{\mathit{x}}\right)$ by averaging the pixel-wise squared difference ${(\widehat{\mathit{x}}-f_{\mathit{\theta }}\left(\widehat{\mathit{x}}\right))}^2$ over the color channels according to

$$\mathrm{Diff}[\widehat{\mathit{x}},f_{\mathit{\theta }}\left(\widehat{\mathit{x}}\right)]:={\displaystyle \frac{1}{3}}\sum _{c=1}^3\left[{({\widehat{\mathit{x}}}_{\ast ,\ast ,c}-f{\left(\widehat{\mathit{x}}\right)}_{\ast ,\ast ,c})}^2\right].$$

(2)

The binary segmentation mask can be computed by thresholding the difference map. To obtain a more robust result, we smooth the difference map before thresholding according to the following formula:

$${\mathrm{anomap}}_{f_{\mathit{\theta }}}^{n,k}\left(\widehat{\mathit{x}}\right):=G_k^n\left(\mathrm{Diff}[\widehat{\mathit{x}},f_{\mathit{\theta }}\left(\widehat{\mathit{x}}\right)]\right),$$

(3)

where $G_k^n={\underbrace{G_k\circ \cdots \circ G_k}}_{n-\mathrm{times}}$ denotes a repeated application of a convolution mapping $G_k$ defined by an averaging filter of size $k\times k$ with all entries set to $1/k^2$. We treat the numbers $n,k\in \mathbb{N}$, $k⩾1$ as hyperparameters, where $G_k^0$ is the identity mapping. By thresholding ${\mathrm{anomap}}_{f_{\mathit{\theta }}}^{n,k}\left(\widehat{\mathit{x}}\right)$, we obtain a binary segmentation mask for the anomalous regions. We compute the anomaly score for the entire image $\widehat{\mathit{x}}$ from ${\mathrm{anomap}}_{f_{\mathit{\theta }}}^{n,k}\left(\widehat{\mathit{x}}\right)$ by summing the scores for the individual pixels. Note that for $n=0$, if we skip the averaging step over the color-channels in (2), this reduces to $\parallel \widehat{\mathit{x}}-f_{\theta }\left(\widehat{\mathit{x}}\right){\parallel }_2^2$. Alternatively to the summation, we could take the maximum over the pixel scores, making the anomaly score potentially less sensitive to size variations in the anomalous regions. The complete detection procedure is summarized in Figure 3. In Section 4, we investigate which image corruptions approximately preserve the orthogonality of the corresponding transformations.

Figure 3. Illustration of our anomaly detection process after the training. Given input $\widehat{\mathit{x}}$, we first, see (1) compute an output $f_{\mathit{\theta }}\left(\widehat{\mathit{x}}\right)$ by replicating normal regions and replacing irregularities with locally consistent patterns. Then, see (2), we compute a pixel-wise squared difference ${(\widehat{\mathit{x}}-f_{\mathit{\theta }}\left(\widehat{\mathit{x}}\right))}^2$, which is subsequently averaged over the color channels to produce the difference map $\mathrm{Diff}[\widehat{\mathit{x}},f_{\mathit{\theta }}\left(\widehat{\mathit{x}}\right)]\in {\mathbb{R}}^{h\times w}$. In the last step, see (3), we apply a series of averaging convolutions $G_k$ to the difference map to produce our final anomaly heatmap ${\mathrm{anomap}}_{f_{\mathit{\theta }}}^{n,k}\left(\widehat{\mathit{x}}\right)$.

3.2. Generating Artificial Anomalies for Training

We use a self-supervised approach representing the training data as input–output $(\widehat{\mathit{x}},\mathit{x})$ pairs. Here, the ground truth outputs are given by the original images $\mathit{x}$ corresponding to the normal examples. The inputs $\widehat{\mathit{x}}$ are generated from these images by partially modifying some regions according to the procedure illustrated in Figure 4.

Figure 4. Illustration of data generation for training. After randomly choosing the number and locations of the patches to be modified, we create new content by gluing the extracted patches with the corresponding replacements. Given a real-valued mask $\mathit{M}\in {[0,1]}^{\tilde{h}\times \tilde{w}\times 3}$ marking corrupted regions within a patch, an original image patch $\mathit{x}$, and a corresponding replacement $\mathit{y}$, we create the next corrupted patch by merging the two patches together according to the formula $\widehat{\mathit{x}}:=\mathit{M}\odot \mathit{y}+\overline{\mathit{M}}\odot \mathit{x}$. All mask shapes $\mathit{M}$ are created by applying Gaussian distortion to the same (static) mask, representing a filled disk at the center of the patch with a smoothly fading boundary toward the exterior of the disk.

For each normal example $\mathit{x}$, we first randomly sample the number, the size, and the location of the patches to be modified. In the next step, each randomly selected patch is modified according to the following procedure: First, we create a real-valued mask $\mathit{M}\in {[0,1]}^{\tilde{h}\times \tilde{w}\times 3}$ based on the elastic deformation technique to mark the corrupted regions within the selected patch. Precisely, we start with a static mask shaped as a disk at the center of the patch with a smoothly fading boundary toward the exterior of the disk. We then apply Gaussian distortion (with random parameters) to this mask, resulting in the varying shapes illustrated on the right side in Figure 4. These masks are used to smoothly merge the original patches with the new content given by the replacement patches. Here, we consider two types of replacements. On the one hand, we can use any natural image (sufficiently different from the original patch) as a potential replacement. In our experiments, we used the publicly available Describable Textures Dataset (DTD) [96], consisting of images of varying backgrounds. On the other hand, we can use the extracted patches as replacements after passing them through a Gaussian distortion process, similar to how we create the masking shapes. An important aspect here is that the corresponding corruptions approximately preserve the original color distribution.

Given an input $\mathit{x}$ and a replacement $\mathit{y}$, we create a corrupted image $\widehat{\mathit{x}}$ by smoothly gluing the two images together according to the formula $\widehat{\mathit{x}}:=\mathit{M}\odot \mathit{y}+\overline{\mathit{M}}\odot \mathit{x}$. In the last step, the patches extracted at the beginning are replaced with their modified versions in the original image. The individual shape masks are embedded in a two-dimensional zero-array at the corresponding locations to create a global mask $\mathit{M}\in {[0,1]}^{h\times w\times 3}$. During testing phase, there are no input modifications, and images are passed unchanged to the network. Anomalous regions are detected by thresholding the anomaly heatmap ${\mathrm{anomap}}_{f_{\mathit{\theta }}}^{n,k}\left(\widehat{\mathit{x}}\right)$ defined in Equation (3).

3.3. Model Architecture

We adopt a well-established encoder–decoder architecture commonly used in computer vision tasks such as semantic segmentation [97,98], image inpainting [87,88,89], representation learning [86], and anomaly detection [1,2,91]. Specifically, the encoder is composed of convolutional layers that progressively reduce spatial dimensions while increasing the number of feature maps. Conversely, the decoder gradually restores spatial dimensions while reducing the number of output channels. In contrast to common practices, we also incorporate dilated convolutions [99], which enable the model to efficiently capture long-range dependencies between individual pixels and their local neighborhoods. This modification has two key effects: it allows access to richer contextual information without increasing the network depth and helps address larger anomalous regions that would otherwise be affected by a blind spot phenomenon, as shown in Figure A3 in Appendix I. Due to the interplay between the training objective and the network architecture, the model naturally adopts a filtering behavior by replacing anomalous patterns in the input images with locally consistent reconstructions. We demonstrate the reconstruction effect with several examples in Figure 2.

The overall architecture is summarized in Table 1.

Table 1. Our network architecture SDC-AE.

Layer Type	Number of Filters	Filter Size	Output Size
Conv	64	$3\times 3$	$512\times 512\times 64$
Conv	64	$3\times 3$	$512\times 512\times 64$
MaxPool		$2\times 2$/2	$256\times 256\times 64$
Conv	128	$3\times 3$	$256\times 256\times 128$
Conv	128	$3\times 3$	$256\times 256\times 128$
MaxPool		$2\times 2$/2	$128\times 128\times 128$
Conv	256	$3\times 3$	$128\times 128\times 256$
Conv	256	$3\times 3$	$128\times 128\times 256$
MaxPool		$2\times 2$/2	$64\times 64\times 256$
${\mathrm{SDC}}_{1,2,4,8,16,32}$	$64\times 6$	$3\times 3$	$64\times 64\times 384$
${\mathrm{SDC}}_{1,2,4,8,16,32}$	$64\times 6$	$3\times 3$	$64\times 64\times 384$
${\mathrm{SDC}}_{1,2,4,8,16,32}$	$64\times 6$	$3\times 3$	$64\times 64\times 384$
${\mathrm{SDC}}_{1,2,4,8,16,32}$	$64\times 6$	$3\times 3$	$64\times 64\times 384$
TranConv	256	$3\times 3$/2	$128\times 128\times 256$
Conv	256	$3\times 3$	$128\times 128\times 256$
Conv	256	$3\times 3$	$128\times 128\times 256$
TranConv	128	$3\times 3$/2	$256\times 256\times 128$
Conv	128	$3\times 3$	$256\times 256\times 128$
Conv	128	$3\times 3$	$256\times 256\times 128$
TranConv	64	$3\times 3$/2	$512\times 512\times 64$
Conv	64	$3\times 3$	$512\times 512\times 64$
Conv	64	$3\times 3$	$512\times 512\times 64$
Conv	3	$1\times 1$	$512\times 512\times 3$

Here, after each convolution Conv (except in the last layer), we use batch normalization and rectified linear units (ReLUs) as the activation function. Max-pooling MaxPool is applied to reduce the spatial dimension of the intermediate feature maps. TranConv denotes the convolution transpose operation and also uses batch normalization with ReLUs. In the last layer, we use a sigmoid activation function without batch normalization. ${\mathrm{SDC}}_{1,2,4,8,16,32}$ refers to a stacked dilated convolution block in which multiple dilated convolutions are stacked together. The corresponding subscript $\{1,2,4,8,16,32\}$ denotes the dilation rates of the six individual convolutions in each stack. After each stack we add an additional convolutional layer with the kernel size $3\times 3$ and the same number of feature maps as in the stack followed by a batch normalization and ReLU activation. We refer to this baseline architecture as SDC-AE, indicating an autoencoder (AE) that relies on stacked dilated convolutions (SDCs).

In our experiments, we observed that SDC-AE struggled to reproduce finer visual patterns, sometimes resulting in false positive detections. In order to improve the reconstruction ability, we proposed the following adjustment illustrated in Figure 5. Similar to the approach commonly used in image segmentation, we introduce skip connections into the network. However, direct access to the feature maps from earlier stages appears counterproductive, as it may interfere with other parts of the computational flow that are essential for suppressing corrupted regions in intermediate representations. Instead, we combine information from different layers of the network using a specific type of attention mechanism. Specifically, we first compute a tensor of coefficients ranging from zero to one, which we then use to calculate a component-wise convex combination of the individual feature maps. The corresponding procedure is illustrated in Figure 6. The idea behind this approach is that it provides the model with a more explicit mechanism for reusing, refining, or replacing regions of the input based on its assessment of the abnormality of a given region. We consolidate the corresponding computations into a single convex combination module (CCM), highlighted in brown in Figure 5. We refer to this architecture as SDC-CCM.

Figure 5. Illustration of our network architecture SDC-CCM including the convex combination module (CCM) marked in brown and the skip-connections represented by the horizontal arrows. Without these additional elements, we obtain our baseline architecture SDC-AE.

Figure 6. Illustration of the CCM module. The module receives two inputs: ${\mathit{x}}_s$ along the skip connection and ${\mathit{x}}_c$ from the current layer below. In the first step (image on the left), we compute the squared difference of the two and stack it together with the original values $[{\mathit{x}}_s,{\mathit{x}}_c,{({\mathit{x}}_s-{\mathit{x}}_c)}^2]$. This combined feature map is processed by two convolutional layers. The first layer uses batch normalization with ReLU activation. The second layer uses batch normalization and a sigmoid activation function to produce a coefficient matrix $\mathit{\beta }$. In the second step (image on the right), we compute the output of the module as a (component-wise) convex combination ${\mathit{x}}_o=\mathit{\beta }·{\mathit{x}}_s+(\mathbf{1}-\mathit{\beta })·{\mathit{x}}_{\mathit{c}}$, where $\mathbf{1}$ is a tensor of ones.

In our experiments on the MVTec AD dataset, we observed a significant performance boost in some object categories when using SDC-CCM over SDC-AE. See Figure A4 in Appendix J for a comparison of the reconstruction quality between the models.

4. Theoretical Analysis

In this section, we provide a number of theoretical insights supporting our idea behind the proposed AD method. We begin by establishing a close connection to the related regularized AEs by identifying the orthogonal projection as an optimal solution for the optimization problem of the RCAE. We then investigate the conservation properties of the orthogonal projection with respect to the segmentation accuracy in the context of AD. As our main result, we show that under certain conditions the resulting model approximates a mapping that stochastically converges (with an increasing number of input pixels) to the orthogonal projection of partially corrupted images onto the submanifold of uncorrupted examples.

For the purpose of the following analysis, we identify each projecting autoencoder (PAE) with an idempotent mapping $f:\mathcal{U}\to {\mathbb{R}}^n$ from an input space $\mathcal{U}\subseteq {\mathbb{R}}^n$, $n\in \mathbb{N}$ to a differentiable manifold $\mathcal{D}:=f\left(\mathcal{U}\right)\subseteq \mathcal{U}$, $\mathit{dim}\left(\mathcal{D}\right)<n$. In the context of AD, we refer to the manifold $\mathcal{D}$ as the (nonlinear) subspace of normal examples, while each $\widehat{\mathit{x}}\in {\mathbb{R}}^n\setminus \mathcal{D}$ is considered anomalous. In particular, we exploit a generalization of orthogonal projection to nonlinear mappings defined below and illustrated in Figure 7.

Figure 7. Illustration of the general concept of the orthogonal projection f onto a data manifold $\mathcal{D}$. Here, anomalous samples $\widehat{\mathit{x}}\in {\mathbb{R}}^n$ (red dots) are projected to points $\mathit{x}:=f\left(\widehat{\mathit{x}}\right)\in \mathcal{D}$ (blue dots) in a way that minimizes the distance $d(\widehat{\mathit{x}},\mathit{x})={inf}_{\mathit{y}\in \mathcal{D}}d(\widehat{\mathit{x}},\mathit{y})$.

Definition 1.

For $n\in \mathbb{N}$ in a metric space $({\mathbb{R}}^n,d(·,·))$, we call a (nonlinear) mapping $f:\mathcal{U}\to {\mathbb{R}}^n$, $\mathcal{U}\subseteq {\mathbb{R}}^n$ the  orthogonal projection   onto $\mathcal{D}\subseteq {\mathbb{R}}^n$ if it satisfies the equality

$$d(\widehat{\mathit{x}},f\left(\widehat{\mathit{x}}\right))=\underset{\mathit{y}\in \mathcal{D}}{inf}d(\widehat{\mathit{x}},\mathit{y})$$

(4)

for all $\widehat{\mathit{x}}\in \mathcal{U}$.

Note that the expression in (4) can be written as a set of inequalities. Namely, for $d(\widehat{\mathit{x}},f\left(\widehat{\mathit{x}}\right)):={\parallel \widehat{\mathit{x}}-f\left(\widehat{\mathit{x}}\right)\parallel }_2$, we obtain the following alternative description:

$$\forall \mathit{y}\in \mathcal{D}:\parallel \widehat{\mathit{x}}-f\left(\widehat{\mathit{x}}\right){\parallel }_2⩽{\parallel \widehat{\mathit{x}}-\mathit{y}\parallel }_2.$$

(5)

We use form (5) in Section 4.3.

In order to train a PAE, we minimize the following objective

$${\mathcal{L}}_{\mathrm{PAE}}\left[f\right]={\mathbf{E}}_{\mathit{x},T}\left[{\parallel f\left(T\left(\mathit{x}\right)\right)-\mathit{x}\parallel }_p^p\right],$$

(6)

where $p\in {\mathbb{N}}_{+}$, and $T:\mathcal{D}\to {\mathbb{R}}^n$ denotes some random data transformation. That is, the expectation in (6) is taken with respect to $\mathit{x}\in \mathcal{D}$ and $T\in \mathcal{T}$. Note how the above objective is related to our training objective in (1). There, we explicitly use a mask $\mathit{M}$ only to balance the training with respect to the reconstruction accuracy of normal and anomalous regions. If we fix the value of ${\parallel \mathit{M}\parallel }_1$ and set $\lambda ={\parallel \mathit{M}\parallel }_1/{(\parallel \mathit{M}\parallel }_1+\parallel \overline{\mathit{M}}{\parallel }_1)$, the objective (1) reduces to minimizing the loss $\mathcal{L}(\widehat{\mathit{x}},\mathit{x})={\parallel f_{\mathit{\theta }}\left(\widehat{\mathit{x}}\right)-\mathit{x}\parallel }_2^2$. Here, the shape of $\mathcal{T}$ mainly determines the properties of the projection map to be learned, which in turn simulates the inverse mapping of the corresponding input corruptions T. Depending on our goal, $\mathcal{T}$ can be very specific ranging from additive noise (e.g., Gaussian noise) to partial occlusions and elastic deformations. In Section 4.3, we identify a number of input transformations that (asymptotically) preserve the orthogonality of the corresponding projections.

4.1. Connections to Regularized Autoencoders

Typically, an autoencoder is composed from two building blocks: the encoder $e(·)$, which maps the input $\mathit{x}$ to some internal representation, and the decoder $d(·)$, which maps $e\left(\mathit{x}\right)$ back to the input space. The composition $f\left(\mathit{x}\right)=d\left(e\right(\mathit{x}\left)\right)$ is often referred to as the reconstruction mapping. Most of the regularized autoencoders aim to capture the structure of the training distribution based on an interplay between the reconstruction error and the regularization term. They are trained by minimizing the reconstruction loss on the training data either by directly adding a regularization term to the objective or by introducing the regularization through some kind of data augmentation.

Specifically, the contractive autoencoder (CAE) [84] is trained to minimize the following regularized reconstruction loss

$${\mathcal{L}}_{\mathrm{CAE}}\left[f\right]={\mathbf{E}}_{\mathit{x}}\left[{\parallel f\left(\mathit{x}\right)-\mathit{x}\parallel }_2^2+\lambda {\parallel D_{\mathit{x}}e\parallel }_F^2\right],$$

(7)

where $\lambda \in {\mathbb{R}}_{+}$ is a weighting hyperparameter, and $\parallel D_{\mathit{x}}{e\parallel }_F$ is the Frobenius norm of the Jacobian of the encoder $e(·)$.

The denoising autoencoder (DAE) [92], on the other hand, is trained to minimize the following denoising criterion

$${\mathcal{L}}_{\mathrm{DAE}}\left[f\right]={\mathbf{E}}_{\mathit{x},\epsilon }\left[{\parallel f(\mathit{x}+\epsilon )-\mathit{x}\parallel }_2^2\right],$$

(8)

where $\epsilon$ represents some additive noise, and the expectation is over the training distribution and the corruption noise. Specifically, the term noise includes additive (e.g., isotropic Gaussian noise) and non-additive transformation (e.g., masking pepper-noise). However, the unifying feature of the considered corruptions is that the transformations of the individual entries in the input are statistically independent. In contrast, in (6), we consider more complex modifications, which result in the strong spatial correlation of the corruption variables). Note that the general objective of the DAE allows for arbitrary stochastic corruptions. However, previous theoretical investigations including [85] focused on (mostly additive) unstructured noise, which can be characterized by the i.i.d. assumption on the distribution of the individual pixels. The results in [85] demonstrate that there is close connection between the CAE and DAE when the standard deviation of the corruption noise approaches zero $\sigma \to 0$. More precisely, under some technical assumptions, the objective of the DAE can be written as

$$\mathcal{L}\mathrm{DAE}\left[f\right]={\mathbf{E}}_{\mathit{x}}\left[{\parallel f\left(\mathit{x}\right)-\mathit{x}\parallel }_2^2+{\sigma }^2{\parallel D_{\mathit{x}}f\parallel }_F^2\right]$$

(9)

as $\sigma \to 0$. That is, for a small variance ${\sigma }^2$ of the corruption noise, the DAE becomes similar to a CAE with penalty coefficient $\lambda ={\sigma }^2$ but where the contraction is imposed explicitly on the whole reconstruction mapping f. This connection motivated the authors to define the RCAE, a variation of the CAE by minimizing the following objective:

$${\mathcal{L}}_{\mathrm{RCAE}}\left[f\right]={\mathbf{E}}_{\mathit{x}}\left[{\parallel f\left(\mathit{x}\right)-\mathit{x}\parallel }_2^2+\lambda {\parallel D_{\mathit{x}}f\parallel }_F^2\right],$$

(10)

where the regularization term affects the whole reconstruction mapping. Finally, a common variant of the SAE applies an $l^1$ penalty on the hidden unit activations, effectively making them saturate toward zero—similar to the effect in the CAE.

We now show that a PAE, realized by the orthogonal projection onto the submanifold of normal examples, is an optimal solution that minimizes the objective of the RCAE in (10). We provide a formal proof in Appendix F.

Proposition 1.

Let $f^{*}:\mathcal{U}\subseteq {\mathbb{R}}^n\to {\mathbb{R}}^n,n\in \mathbb{N}$ be an orthogonal projection onto $\mathcal{D}:=f^{*}\left(\mathcal{U}\right)$ with respect to an $l^p$-norm, $p\in {\mathbb{N}}_{+}$. Then, $f^{*}$ is an optimal solution to the following optimization problem:

$$\underset{f\in C^1}{\mathit{minimize}}{\mathbf{E}}_{\mathit{x}\sim \mathcal{D}}\left[{\parallel f\left(\mathit{x}\right)-\mathit{x}\parallel }_2^2+\lambda {∥D_{\mathit{x}}f∥}_q^2\right],$$

(11)

where $\lambda \in {\mathbb{R}}_{+}$, and $q\in \{F,2\}$ is a placeholder denoting either the Frobenius or the spectral norm.

Note that the objective in the above proposition is slightly more general than that in (10) allowing the use of the spectral norm. Although being closely related to each other, the Frobenius and the spectral norm might have different effects on the optimization. Namely, the spectral norm measures the maximal scale by which a unit vector is stretched by a corresponding linear transformation and is determined by the maximal singular value, while the Frobenius norm measures the overall distortion of the unit circle taking into account all the singular values. In a more general sense, Proposition 1 implies a close connection between the PAE and other types of regularized autoencoders (see Figure 8 for an overview).

Figure 8. Illustration of the connections between the different types of regularized autoencoders. For a small variance of the corruption noise, the DAE becomes similar to the CAE. This, in turn, gives rise to the RCAE, where the contraction is imposed explicitly on the whole reconstruction mapping. A special instance of PAE given by the orthogonal projection yields an optimal solution for the optimization problem of the RCAE. On the other hand, the training objective for PAE can be seen as an extension of DAE to more complex input modifications beyond additive noise. Finally, a common variant of the sparse autoencoder (SAE) applies an $l^1$ penalty on the hidden units, resulting in saturation toward zero similar to the CAE.

Orthogonal projection, in particular, appears to be an appropriate choice with respect to the shared goals of the autoencoding models discussed in this section. In Section 4.3.2, we show that in the limit when the dimensionality of the inputs goes to infinity, the DAE converges stochastically to the orthogonal projection when restricted to the specific noise corruptions.

4.2. Conservation Effect of Orthogonal Projections

Here, we focus on the subtask of AD regarding the segmentation of anomalous regions in the inputs. In the following, we identify each image tensor with a column vector $\mathit{x}\in {\mathbb{R}}^n$, where $S\subset \{1,\dots ,n\}$ and $\overline{S}:=\{1,\dots ,n\}\setminus S$ denote a set of pixel indices and its complement, respectively. We write ${\mathit{x}}_S$ to denote a restriction of a vector $\mathit{x}$ to the indices in S.

Consider an $\widehat{\mathit{x}}\in {\mathbb{R}}^n$ that has been generated by a (partial) modification $\widehat{\mathit{x}}:=T\left(\mathit{x}\right)$, $T\in \mathcal{T}$ from an $\mathit{x}\in \mathcal{D}$. We define the modified region as the set of disagreeing indices according to

$$S(\widehat{\mathit{x}},\mathit{x}):=\left\{i\in \{1,\dots ,n\}:{\widehat{x}}_i\ne x_i\right\}.$$

(12)

There is some ambiguity about what might be considered an anomalous region, which motivates the following definition:

Definition 2.

Let $\mathcal{D}\in {\mathbb{R}}^n$, $n\in \mathbb{N}$ be a data manifold. Given $\widehat{\mathit{x}}\in {\mathbb{R}}^n\setminus \mathcal{D}$, we define the set of anomalous regions in $\widehat{\mathit{x}}$ as the areas of smallest disagreement according to

$$\mathcal{A}\left(\widehat{\mathit{x}}\right):=\left\{S(\widehat{\mathit{x}},\mathit{x}):\mathit{x}\in \underset{\mathit{y}\in \mathcal{D}}{\mathrm{argmin}}\left|S(\widehat{\mathit{x}},\mathit{y})\right|\right\}.$$

(13)

We refer to each $S\in \mathcal{A}\left(\widehat{\mathit{x}}\right)$ as an anomalous region to the pair $(S,{\widehat{\mathit{x}}}_S)$ as an anomalous patternand to $\widehat{\mathit{x}}$ as an anomalous sample.

We now provide some explanation of the motivation of our definition of anomalous patterns. Consider an example of binary sequences $\mathit{x}\in {\{0,1\}}^8$ restricted by a condition that the pattern “11” is forbidden. For example, the sequence (0, 1, 0, 1, 1, 1, 1, 0) is invalid because it contains “11”. If we define the anomalous region as the smallest subset of indices that need to be corrected in order for the point to be projected to the data manifold, we encounter the following ambiguity problem. Namely, there are three different ways to correct the above example using minimal number of changes:

(0, 1, 0, 1, 0, 1, 0, 0); (0, 1, 0, 0, 1, 0, 1, 0); and (0, 1, 0, 1, 0, 0, 1, 0), which we denote as ${\mathit{y}}_1$, ${\mathit{y}}_2$ and ${\mathit{y}}_3$, respectively. All three sequences correspond to orthogonal projections to the feasible set, where $\parallel \widehat{\mathit{x}}-{\mathit{y}}_1{\parallel }_2=\parallel \widehat{\mathit{x}}-{\mathit{y}}_2{\parallel }_2={\parallel \widehat{\mathit{x}}-{\mathit{y}}_3\parallel }_2=\sqrt{2}$. However,

$$\bigcap _{i=1}^{3}S(\widehat{\mathit{x}},{\mathit{y}}_i)\ne \bigcup _{i=1}^{3}S(\widehat{\mathit{x}},{\mathit{y}}_i).$$

(14)

That is, there are multiple anomalous patterns giving rise to the same corrupted point $\widehat{\mathit{x}}$. In particular, the anomalous regions in $\widehat{\mathit{x}}$ are not uniquely determined and depend on the structure of $\mathcal{D}$. Based on this observation, we highlight a special projection map below, which maximally preserves the content of its arguments.

Definition 3.

For $n\in \mathbb{N}$, $p\in {\mathbb{N}}_{+}$, we call an idempotent mapping $f:\mathcal{U}\to {\mathbb{R}}^n$, $\mathcal{U}\subseteq {\mathbb{R}}^n$ the  conservative projection   onto $\mathcal{D}:=f\left(\mathcal{U}\right)\subseteq \mathcal{U}$ with respect to the $l^p$-norm if, for each pair $(\widehat{\mathit{x}},\mathit{x})\in \mathcal{U}\times \mathcal{D}$ with $f\left(\widehat{\mathit{x}}\right)=\mathit{x}$ and $S:=S(\widehat{\mathit{x}},\mathit{x})$, it satisfies the following properties:

$$\begin{array}{cc}\hfill \left(a\right)& S\in \mathcal{A}\left(\widehat{\mathit{x}}\right)\hfill \\ \hfill \left(b\right)& \parallel {\widehat{\mathit{x}}}_S-f_S\left(\widehat{\mathit{x}}\right){\parallel }_p⩽{inf}_{\mathit{x}\in \mathcal{D}:{\widehat{\mathit{x}}}_{\overline{S}}={\mathit{x}}_{\overline{S}}}{\parallel {\widehat{\mathit{x}}}_S-{\mathit{x}}_S\parallel }_p.\hfill \end{array}$$

The next proposition relates the conservation properties of orthogonal projections for different $l^p$-norms. See Figure 9 for an illustration. A corresponding proof is provided in Appendix G.

Figure 9. Illustration of the conservation effect of the orthogonal projections with respect to different $l^p$-norms. Here, the anomalous sample $\widehat{\mathit{x}}$ is orthogonally projected onto the manifold $\mathcal{D}$ (depicted by a red ellipsoid) according to $\parallel \widehat{\mathit{x}}-{\mathit{y}}_p^{*}{\parallel }_p={inf}_{\mathit{y}\in \mathcal{D}}{\parallel \widehat{\mathit{x}}-\mathit{y}\parallel }_p$ for $p\in \{1,2,\infty \}$. The remaining three colors (green, blue, and yellow) represent rescaled unit circles around $\widehat{\mathit{x}}$ with respect to $l^1$, $l^2$ and $l^{\infty }$-norms. The intersection points of each circle with $\mathcal{D}$ mark the orthogonal projection of $\mathit{x}$ onto $\mathcal{D}$ for the corresponding norm. We can see that projections ${\mathit{y}}_p^{*}$ for lower p-values better preserve the content in $\widehat{\mathit{x}}$ according to the higher sparsity of the difference $\widehat{\mathit{x}}-{\mathit{y}}_p^{*}$, which results in smaller modified regions $S(\widehat{\mathit{x}},{\mathit{y}}_p^{*})$.

Proposition 2.

Let $\mathcal{D}\subseteq {\mathbb{R}}^n$, $n\in \mathbb{N}$ be a data manifold and $\widehat{\mathit{x}}\in {\mathbb{R}}^n\setminus \mathcal{D}$ be an anomalous sample. Furthermore, let $S\in \mathcal{A}\left(\widehat{\mathit{x}}\right)$ and $S_p:=S(\widehat{\mathit{x}},{\mathit{y}}_p^{*})$, where ${\mathit{y}}_p^{*}\in {inf}_{\mathit{y}\in \mathcal{D}}{\parallel \widehat{\mathit{x}}-\mathit{y}\parallel }_p$ corresponds to the orthogonal projection of $\widehat{\mathit{x}}$ onto $\mathcal{D}$ with respect to the $l^p$-norm. For all $p,q\in {\mathbb{N}}_{+}$, $p<q$, the following statements are true:

$$\begin{array}{cc}\hfill \left(a\right)& S\subseteq S_p\subseteq S_q\hfill \\ \hfill \left(b\right)& \exists \mathcal{D}\subseteq {\mathbb{R}}^n:S_1\ne S_2.\hfill \end{array}$$

The above proposition shows that orthogonal projection with respect to the $l^p$-norm is more conservative for lower $p⩾1$ values regarding the preservation of normal regions. However, $l^2$-norm has practical advantages over the $l^1$-norm regarding the optimization process and is, therefore, often a better choice. Furthermore, the orthogonal projection with respect to the $l^2$-norm (unlike the conservative projection) is not maximally preserving in general. We show later, however, that within the limit of the increasing input dimensionality, the conservative and the orthogonal projections (with respect to $l^2$-norm) disagree only on a zero-measure probability set.

While Proposition 2 describes the conservation properties of orthogonal projections relative to each other, the following proposition specifies (asymptotically) how accurate the corresponding reconstruction error is in describing the anomalous region. As an auxiliary concept, we here introduce the notion of a transition set (illustrated in Figure 10), which glues the disconnected parts of an input together. We provide more details in Appendix H.

Figure 10. Illustration of the concept of a transition set. Consider a 2D image tensor identified with a column vector $\mathit{x}\in {\mathbb{R}}^n$, $n={20}^2$, which is partitioned according to $S\subseteq \{1,\dots ,n\}$ (gray area) and $\overline{S}:=\{1,\dots ,n\}\setminus S$ (union of light blue and dark blue areas). The transition set B (dark blue area) glues the two disconnected sets S and $\overline{S}\setminus B$ together such that $\mathit{x}\in \mathcal{D}$ is feasible.

Proposition 3.

Let $f:{[0,1]}^n\to \mathcal{D}\subseteq {[0,1]}^n$, $n\in \mathbb{N}$ be the orthogonal projection with respect to an $l^p$-norm, $p\in {\mathbb{N}}_{+}$ and $\mathit{x}\in \mathcal{D}$ with a finite set of states $x_i\in I\subseteq [0,1]$, $\left|I\right|<\infty$. Consider an $\widehat{\mathit{x}}\in {[0,1]}^n$ that has been generated from $\mathit{x}$ via partial modification with ${\widehat{\mathit{x}}}_{\overline{S}}={\mathit{x}}_{\overline{S}}$ for some $S\subset \{1,\dots ,n\}$, $\overline{S}:=\{1,\dots ,n\}\setminus S$. Furthermore, let $B\subseteq \{1,\dots ,n\}$ denote a transition set from S to $\overline{S}\setminus B$. For all $p\in {\mathbb{N}}_{+}$, the following holds true:

$$\parallel f_{\overline{S}}\left(\widehat{\mathit{x}}\right)-{\mathit{x}}_{\overline{S}}{\parallel }_p⩽{\left|B\right|}^{{\displaystyle \frac{1}{p}}},$$

(15)

where $\left|B\right|$ grows asymptotically according to $O\left(\sqrt{\left|S\right|}\right)$.

When projecting corrupted points onto the data manifold, we would like the transition set B around the anomalous region S to be as small as possible. The inequality in (15) implicitly upper-bounds the number of entries in the normal region $\overline{S}$ that are not preserved by the projection. This is mostly practical for lower values of p and least informative in the case $p=\infty$. Namely, $\underset{n\to \infty }{\lim }{\left|B\right|}^{{\displaystyle \frac{1}{p}}}=1$ provides a trivial upper bound, since ${\parallel \mathit{x}\parallel }_{\infty }⩽1$ for $\mathit{x}\in {[0,1]}^n$. For $p=1$, for example, the interpretation is the simplest in the case of finite sequences of binary values. The number of disagreeing components is then upper-bounded by the number $\left|B\right|$, corresponding to the size of the transition set, which, in turn, is determined by the form of the underlying data distribution.

To summarize, we showed that orthogonal projection with respect to the $l^p$-norm preserves the normal regions more accurately for smaller p-values. Furthermore, we identified the conservative projection as the one that is maximally preserving (unlike the orthogonal projection). As previously mentioned, we show in Section 4.3 that within the limit (when the dimensionality of the input vectors goes to infinity), the $l^2$-conservative and the $l^2$-orthogonal projection are the same up until a zero-measure probability set.

4.3. Convergence Guarantees for Input Corruptions

In the following, we specify a range of input modifications that (approximately) preserve the orthogonality of the corresponding projections. In the context of image processing, the plethora of existing data augmentation techniques can be roughly divided into five groups: affine transformations, color jittering, mixing strategies, elastic deformations, and additive noise. Here, we characterize each image transformation either as $\left(a\right)$ a partial modification (of any type) or as $\left(b\right)$ a modification affecting the entire image represented by the additive noise methods. Image mixing strategies like MixUp can simply be seen as a transformation with additive noise, while CutMix is an example of partial modification. On the other hand, linear and affine data transformations like shift, rotation, or color-channel permutation, in general, do not preserve orthogonality.

For the purpose of the subsequent analysis, we identify the image tensors in our input space with a multivariate random variable $\mathit{x}\in {\mathbb{R}}^n$ corresponding to Markov random fields (MRFs) [100,101,102,103,104,105,106,107,108]. In particular, we assume that the variables representing the individual pixels are organized in a two-dimensional grid. Based on this view, we consider sequences ${\mathcal{D}}^{\left(n\right)}\subseteq {[0,1]}^n$ of spaces of increasing dimensionality $n\in \mathbb{N}$, representing images of gradually increasing size by adding new nodes along the rows and columns of the grid. Furthermore, we use the notation $d_G(x_i,x_j)$ to denote the distance between the variables $x_i$ and $x_j$ in a corresponding MRF graph G defined by the number of edges on a shortest path connecting the two nodes.

4.3.1. Partial Modification

The following theorem describes the technical conditions under which a corresponding partial modification corresponds to the orthogonal projection. We provide a formal proof in Appendix D.

Theorem 1.

Consider a pair $\mathit{x},\mathit{y}\in {\mathcal{D}}^{\left(n\right)}$ of independent MRFs over identically distributed variables $x_i,y_j$ with finite fourth moment $\mathbf{E}\left[x_i^4\right]<\infty$, variance ${\sigma }^2:=\mathbf{Var}\left[x_i\right]$ and vanishing covariance

$$\mathbf{Cov}[x_i^k,x_j^l]⟶0for{d}_G(x_i,x_j)⟶\infty$$

(16)

for all $k,l\in \{1,2\}$. Furthermore, let $\widehat{\mathit{x}}\in {[0,1]}^n$ be a copy of $\mathit{x}$ that has been partially modified, where ${\widehat{\mathit{x}}}_{\overline{S}}={\mathit{x}}_{\overline{S}}$ for some $S\subset \{1,\dots ,n\}$, $\overline{S}:=\{1,\dots ,n\}\setminus S$. The following is true,

$$\underset{n\to \infty }{\mathit{lim}}{\mathbf{P}}_{\mathit{x},\mathit{y}}\left(\parallel \widehat{\mathit{x}}{-\mathit{x}\parallel }_2⩽{\parallel \widehat{\mathit{x}}-\mathit{y}\parallel }_2\right)=1,$$

(17)

provided the inequality

$$\left|S\right|⩽2·{\sigma }^2·\left|\overline{S}\right|$$

(18)

holds for all $n\in \mathbb{N}$.

Given two samples $\mathit{x},\mathit{y}\in \mathcal{D}$ and a corrupted copy $\widehat{\mathit{x}}\in {[0,1]}^n$, Theorem 1 describes how the expression $\parallel \widehat{\mathit{x}}{-\mathit{x}\parallel }_2⩽{\parallel \widehat{\mathit{x}}-\mathit{y}\parallel }_2$ approaches a true statement when the dimensionality of the embedding space $n\in \mathbb{N}$ goes to infinity. In particular, $\mathit{x}$ can be identified with the image of $\widehat{\mathit{x}}$ under the conservative projection $f\left(\widehat{\mathit{x}}\right)=\mathit{x}$. That is,

$$\underset{n\to \infty }{\lim }{\mathbf{P}}_{\mathit{x},\mathit{y}}\left(\parallel \widehat{\mathit{x}}-f\left(\widehat{\mathit{x}}\right){\parallel }_2⩽{\parallel \widehat{\mathit{x}}-\mathit{y}\parallel }_2\right)=1.$$

(19)

Therefore, the conservative projection converges stochastically to the orthogonal projection (compare to the definition of the orthogonal projection in (5)), while the inequality in (18) controls the maximal size of corrupted regions $\left|S\right|$ by taking into account the distribution of normal examples. On the other hand, Equation (19) suggests that (in the limit) the orthogonal and the conservative projection disagree at most on a subset of the data manifold $\mathcal{D}$, which has a zero probability measure under the training distribution.

4.3.2. Additive Noise

The following theorem describes the technical conditions under which the denoising process (with additive noise) corresponds to the orthogonal projection. We provide a formal proof in Appendix E.

Theorem 2.

Consider a pair $\mathit{x},\mathit{y}\in {\mathcal{D}}^{\left(n\right)}$ of independent MRFs over identically distributed variables $x_i,y_j$ with finite fourth moment $\mathbf{E}\left[x_i^4\right]<\infty$, variance ${\sigma }^2:=\mathbf{Var}\left[x_i\right]$, and vanishing covariance

$$\mathbf{Cov}[x_i^k,x_j^l]⟶0for{d}_G(x_i,x_j)⟶\infty$$

(20)

for all $k,l\in \{1,2\}$. Furthermore, let $\widehat{\mathit{x}}:=\mathit{x}+\epsilon$ be another MRF, where $\epsilon =({ϵ}_1,\dots ,{ϵ}_n)$ is an additive noise vector of i.i.d. variables with ${\mu }_{\epsilon }:=\mathbf{E}\left[{ϵ}_i\right]$ and ${\sigma }_{\epsilon }^2:=\mathbf{Var}\left[{ϵ}_i\right]$. Then, the following holds true:

$$\underset{n\to \infty }{\mathit{lim}}{\mathbf{P}}_{\mathit{x},\mathit{y}}\left(\parallel \widehat{\mathit{x}}{-\mathit{x}\parallel }_2⩽{\parallel \widehat{\mathit{x}}-\mathit{y}\parallel }_2\right)=1,$$

(21)

provided

$${\mu }_{\epsilon }^2+{\sigma }_{\epsilon }^2⩽{\displaystyle \frac{1}{2}}{\sigma }^2.$$

(22)

In the special example of isotropic Gaussian noise $\epsilon \sim \mathcal{N}(\mathbf{0},{\sigma }_{\epsilon }^2·I)$, the condition in (22) is reduced to ${\sigma }_{\epsilon }^2⩽{\displaystyle \frac{1}{2}}{\sigma }^2$. Here, again, $\mathit{x}$ can be identified with the image of the conservative projection, which removes the specific type of noise from its arguments. The equation in (21) then implies its convergence to the orthogonal projection.

5. Experiments

In our experiments, we used the publicly available MVTec AD dataset [95,109]—a popular benchmark for anomaly detection in the manufacturing domain. It provides a diverse dataset with significant variations in materials and anomaly types, organized into 5 texture categories and 10 object categories. Each category encompasses multiple classes of anomalies, with the number of training examples ranging up to several hundred images. For a detailed technical description of the dataset, we direct the reader to the original publication [109]. During training, we artificially increased the amount of data by applying simple data augmentations, such as rotation and flipping, depending on the category. We used 5% of the data as a validation set. For the sake of reproducibility, we mention that for the transistor category, we used a random image rotation by multiples of 90 degrees, in addition to the artificial corruptions, during training. This made the training procedure considerably more challenging, but it proved essential for detecting missing objects.

For each category, we trained a separate model based on the proposed architectures (SDC-AE and SDC-CCM). For both models, we used the same image resolution of $512\times 512$ pixels for texture and $256\times 256$ pixels for object categories and the Adam optimizer [110] with initial learning rate of ${10}^{-4}$. Note that our model architecture makes no assumptions about the size of the input images. The SDC-AE architecture, in particular, is fully convolutional and can be applied to any input size that is a multiple of the network’s stride after training. However, the input size is an important hyperparameter that influences the final performance of the anomaly detection. The resolution choices of 512 × 512 and 256 × 256 pixels appeared to work well with the considered dataset.

We report the pixel-level and the image-level AUROC metric to illustrate both segmentation and recognition performance in Table 2 and Table 3, respectively. We compare our results (for SCD-AE and SDC-CCM) with a number of previous methods like AnoGAN [6], VAE [9], and LSR [3], including top-ranking algorithms such as RIAD [39], CutPaste [40], InTra [41], DRAEM [38], SimpleNet [94], PatchCore [10], MSFlow [111], and PNI [68]. Our method achieves high performance with both architectures. However, SDC-CCM significantly outperforms SDC-AE in the cable, capsule, and transistor categories due to a reduction in false positive detections, supporting our idea about the CCM module. To further validate our choice of network architecture, we describe an additional ablation study in Appendix I. This study demonstrates the importance of dilated convolutions and the use of fine-grained information from earlier layers in subsequent layers, either through skip connections or the proposed CCM module. To give a sense of the visual quality of the resulting anomaly heatmaps produced by our method, we show a few additional examples in Figure 11.

Table 2. Experimental results for anomaly segmentation measured with pixel-level AUROC on the MVTec AD dataset.

Category	AnoGAN	VAE	LSR	RIAD	CutPaste	InTra	DRAEM	SimpleNet	PatchCore	MSFlow	PNI	SDC-AE	SDC-CCM
carpet	54	78	94	96.3	98.3	99.2	95.5	98.2	98.7	99.4	99.4	99.7	99.8
grid	58	73	99	98.8	97.5	98.8	99.7	98.8	98.8	99.4	99.2	99.7	99.8
leather	64	95	99	99.4	99.5	99.5	98.6	99.2	99.3	99.7	99.6	99.7	99.7
tile	50	80	88	89.1	90.5	94.4	99.2	97.0	96.3	98.2	98.4	99.2	99.2
wood	62	77	87	85.8	95.5	88.7	96.4	94.5	95.2	97.1	97.0	98.4	98.4
avg. tex.	57.6	80.6	93.4	93.9	96.3	96.1	97.9	97.5	97.7	98.8	98.7	99.3	99.4
bottle	86	87	95	98.4	97.6	97.1	99.1	98.0	98.6	99.0	98.9	98.6	98.9
cable	86	87	95	94.2	90.0	91.0	94.7	97.6	98.7	98.5	99.1	98.2	98.5
capsule	84	74	93	92.8	97.4	97.7	94.3	98.9	99.1	99.1	99.3	99.1	99.1
hazelnut	87	98	95	96.1	97.3	98.3	99.7	97.9	98.8	98.7	99.4	98.9	99.1
metal nut	76	94	91	92.5	93.1	93.3	99.5	98.8	99.0	99.3	99.3	98.5	98.5
pill	87	83	91	95.7	95.7	98.3	97.6	98.6	98.6	98.8	99.0	99.3	99.3
screw	80	97	96	98.8	96.7	99.5	97.6	99.3	99.5	99.1	99.6	99.7	99.7
toothbrush	90	94	97	98.9	98.1	98.9	98.1	98.5	98.9	98.5	99.1	99.1	99.4
transistor	80	93	91	87.7	93.0	96.1	90.9	97.6	97.1	98.3	98.0	98.6	98.9
zipper	78	78	98	97.8	99.3	99.2	98.8	98.9	99.0	99.2	99.4	99.5	99.6
avg. obj.	83.4	88.5	94.6	95.3	95.8	96.9	97.0	98.4	98.7	98.8	99.1	99.0	99.1
avg. all	74.8	85.9	94.2	94.8	96.0	96.7	97.3	98.1	98.4	98.8	99.0	99.1	99.2

Table 3. Experimental results for anomaly recognition measured with image-level AUROC on the MVTec AD dataset.

Category	AnoGAN	VAE	LSR	RIAD	CutPaste	InTra	DRAEM	SimpleNet	PatchCore	MSFlow	PNI	SDC-AE	SDC-CCM
carpet	49	78	71	84.2	93.1	98.8	97.0	99.7	98.2	100	100	100	100
grid	51	73	91	99.6	99.9	100	99.9	99.7	98.3	99.8	98.4	100	100
leather	52	95	96	100	100	100	100	100	100	100	100	100	100
tile	51	80	95	93.4	93.4	98.2	99.6	99.8	98.9	100	100	100	100
wood	68	77	96	93.0	98.6	97.5	99.1	100	99.9	100	99.6	100	100
avg. textures	54.2	80.6	89.8	94.0	97.0	98.9	99.1	99.8	99.0	99.9	99.6	100	100
bottle	69	87	99	99.9	98.3	100	99.2	100	100	100	100	100	100
cable	53	90	72	81.9	80.6	70.3	91.8	99.9	99.7	99.5	99.8	98.0	99.9
capsule	58	74	68	88.4	96.2	86.5	98.5	97.7	98.1	99.2	99.7	98.2	98.8
hazelnut	50	98	94	83.3	97.3	95.7	100	100	100	100	100	100	100
metal nut	50	94	83	88.5	99.3	96.9	98.7	100	100	100	100	100	100
pill	62	83	68	83.8	92.4	90.2	98.9	99.0	97.1	99.6	96.9	99.5	99.6
screw	35	97	80	84.5	86.3	95.7	93.9	98.2	99.0	97.8	99.5	98.9	98.9
toothbrush	57	94	92	100	98.3	100	100	99.7	98.9	100	99.7	100	100
transistor	67	93	73	90.9	95.5	95.8	93.1	100	99.7	100	100	98.8	100
zipper	59	78	97	98.1	99.4	99.4	100	99.9	99.7	100	99.9	100	100
avg. objects	56.0	88.8	82.6	89.9	94.4	93.1	97.4	99.5	99.2	99.6	99.5	99.3	99.7
avg. all	55.4	86.1	85.0	91.3	95.2	95.0	98.0	99.6	99.2	99.7	99.6	99.6	99.8

Figure 11. Illustration of our anomaly segmentation results (with SDC-CCM) as an overlay of the original image and the anomaly heatmap. Each row shows three random examples from a category (carpet, grid, leather, transistor, and cable) in the MVTec AD dataset. In each pair, the first image represents the input to the model and the second image a corresponding anomaly heatmap.

6. Conclusions

We focused on an important use case for anomaly detection, where the data distribution is supported by a lower-dimensional manifold, and the covariance between neighboring input components vanishes as the distance increases. Our self-supervised approach aims to learn a reconstruction model that repairs artificially corrupted inputs based on a specific form of stochastic occlusions. The resulting training effect regularizes the model to produce locally consistent reconstructions while replacing irregularities, thus acting as a filter that removes anomalous patterns. We demonstrated the effectiveness of our approach by achieving state-of-the-art results on the MVTec AD dataset, a challenging benchmark for visual anomaly detection in the manufacturing domain.

Additionally, we performed a theoretical analysis of the proposed method, providing several interesting insights. As our main result, we showed that as the dimensionality of the inputs increases, the corresponding model approximates a mapping that stochastically converges to the orthogonal projection of partially corrupted inputs onto the submanifold of uncorrupted examples. (According to our training objective in (1), an optimal solution is given by a model $f^{*}$, such that $f_S^{*}\left(\widehat{\mathit{x}}\right)={\mathbf{E}}_{\mathit{x}\sim \mathcal{D}}\left[{\mathit{x}}_S\right|{\widehat{\mathit{x}}}_{\overline{S}}]$. Therefore, $f^{*}\left(\widehat{\mathit{x}}\right)$ does not necessarily lie in $\mathcal{D}$ and the corresponding approximation error depends on the variance of ${\mathit{x}}_S$ given ${\mathit{x}}_{\overline{S}}$). If the covariance between the input variables (with increasing distance) rapidly approaches zero, the orthogonal projection maps its input in a way that largely preserves the original content, supporting our intuition about the filtering behavior of the model. Therefore, we can jointly perform the detection and localization of corrupted regions using the pixel-wise reconstruction error.

Furthermore, we deepened the understanding of the relationship between regularized autoencoders. Specifically, we showed that the orthogonal projection provides an optimal solution for the RCAE, which was demonstrated to be equivalent to the DAE for small variance in the corruption noise. Our results extend this equivalence to more complex input modifications beyond i.i.d. pixel corruptions and provide a unifying perspective on the regularized autoencoders that is not limited by the assumption of small variance.