Learning Energy-Based Models in High-Dimensional Spaces with Multiscale Denoising-Score Matching

Abstract

Energy-based models (EBMs) assign an unnormalized log probability to data samples. This functionality has a variety of applications, such as sample synthesis, data denoising, sample restoration, outlier detection, Bayesian reasoning and many more. But, the training of EBMs using standard maximum likelihood is extremely slow because it requires sampling from the model distribution. Score matching potentially alleviates this problem. In particular, denoising-score matching has been successfully used to train EBMs. Using noisy data samples with one fixed noise level, these models learn fast and yield good results in data denoising. However, demonstrations of such models in the high-quality sample synthesis of high-dimensional data were lacking. Recently, a paper showed that a generative model trained by denoising-score matching accomplishes excellent sample synthesis when trained with data samples corrupted with multiple levels of noise. Here we provide an analysis and empirical evidence showing that training with multiple noise levels is necessary when the data dimension is high. Leveraging this insight, we propose a novel EBM trained with multiscale denoising-score matching. Our model exhibits a data-generation performance comparable to state-of-the-art techniques such as GANs and sets a new baseline for EBMs. The proposed model also provides density information and performs well on an image-inpainting task.

1. Introduction and Motivation

Treating data as stochastic samples from a probability distribution and developing models that can learn such distributions is at the core of solving a large variety of application problems, such as error correction/denoising [1]; outlier/novelty detection [2,3]; sample generation [4,5]; invariant pattern recognition; Bayesian reasoning [6], which relies on suitable data priors; and many others.

Energy-based models (EBMs) [7,8] assign energy $E\left(\mathit{x}\right)$ to each data point $\mathit{x}$, which implicitly defines a probability density via the Boltzmann distribution $p_m\left(\mathit{x}\right)=e^{-E\left(\mathit{x}\right)}/Z$. Sampling from this distribution can be used as a generative process that yields plausible samples of $\mathit{x}$.

Compared to other generative models like GANs [9], flow-based models [10,11] or autoregressive models [12,13], energy-based models have significant advantages. First, they provide explicit (unnormalized) density information, compositionality [14,15], better mode coverage [16] and flexibility [5]. Further, they do not require special model architecture, unlike autoregressive and flow-based models. Recently, energy-based models have been successfully trained with maximum likelihood [4,5], but training can be very computationally demanding due to the need for a sampling model distribution. Variants with a truncated sampling procedure have been proposed, such as contrastive divergence [17]. Such models learn much faster with the drawback of not exploring the state space thoroughly [18].

Score Matching, Denoising-Score Matching and Deep-Energy Estimators

Score matching (SM) [19] circumvents the requirement of sampling the model distribution. In score matching, the score function is defined as the gradient of the log density or the negative energy function. The expected $L2$ norm of the difference between the model score function and the data score function is minimized.

One convenient way of using score matching is learning the energy function corresponding to a Gaussian kernel Parzen density estimator [20] of the data: $p_{{\sigma }_0}\left(\tilde{\mathit{x}}\right)=\int q_{{\sigma }_0}\left(\tilde{\mathit{x}}\right|\mathit{x})p\left(\mathit{x}\right)d\mathit{x}$. Though hard to evaluate, the data score is well defined, $s_d\left(\tilde{\mathit{x}}\right)={\nabla }_{\tilde{\mathit{x}}}log\left(p_{{\sigma }_0}\left(\tilde{\mathit{x}}\right)\right)$, and the corresponding objective is

$$L_{SM}\left(\theta \right)={\mathbb{E}}_{p_{\sigma 0}\left(\tilde{\mathit{x}}\right)}{\Vert {\nabla }_{\tilde{\mathit{x}}}log\left(p_{{\sigma }_0}\left(\tilde{\mathit{x}}\right)\right)+{\nabla }_{\tilde{\mathit{x}}}E(\tilde{\mathit{x}};\theta )\Vert }^2$$

(1)

$L_{SM}$ is also known as the Fisher divergence or the Fisher information distance [21,22], $L_{SM}=D_{FD}(p_{\sigma 0}\left|\right|p_m)$, where $p_m\left(\mathit{x}\right)=e^{-E\left(\mathit{x}\right)}/Z$ is the normalized distribution from the model energy function. While the KL divergence requires the ratio between two density functions, this metric does not depend on the normalizing constant Z, which for an energy-based model needs global integration through sampling and is rarely accurately available.

Vincent [23] studied the connection between a denoising autoencoder and score matching and proved the remarkable result that the following objective, named denoising-score matching (DSM), is equivalent to the objective above:

$$L_{DSM}\left(\theta \right)={\mathbb{E}}_{p_{{\sigma }_0}(\tilde{\mathit{x}},\mathit{x})}{\Vert {\nabla }_{\tilde{\mathit{x}}}log\left(q_{\sigma 0}\left(\tilde{\mathit{x}}\right|\mathit{x})\right)+{\nabla }_{\tilde{\mathit{x}}}E(\tilde{\mathit{x}};\theta )\Vert }^2$$

(2)

Note that in (2), the Parzen density score is replaced by the derivative of the log density of the single noise kernel ${\nabla }_{\tilde{\mathit{x}}}log(q_{{\sigma }_0}\left(\tilde{\mathit{x}}\right|\mathit{x}))$, which is much easier to evaluate. In the particular case of Gaussian noise, $log\left(q_{{\sigma }_0}\left(\tilde{\mathit{x}}\right|\mathit{x})\right)=-\frac{{(\tilde{\mathit{x}}-\mathit{x})}^2}{2{\sigma }_0^2}+C$, and therefore

$$L_{DSM}\left(\theta \right)={\mathbb{E}}_{p_{\sigma 0}(\tilde{\mathit{x}},\mathit{x})}{\Vert \mathit{x}-\tilde{\mathit{x}}+{{\sigma }_0}^2{\nabla }_{\tilde{\mathit{x}}}E(\tilde{\mathit{x}};\theta )\Vert }^2$$

(3)

The intuition behind objective (3) is simple: it forces the energy gradient to align with the vector pointing from the noisy sample to the clean data sample.

To optimize an objective involving the derivative of a function defined by a neural network, Kingma and LeCun [24] proposed double backpropagation [25]. Deep-energy estimator networks [26] first applied this technique to learn an energy function defined by a deep neural network. In this work and similarly in Saremi and Hyvärinen [27], an energy-based model was trained to match a Parzen density estimator of data with a particular noise magnitude. The previous models could perform denoising tasks, but they could not generate high-quality data samples from a random input initialization. Recently, Song and Ermon [28] trained an excellent generative model by fitting a series of score estimators coupled together in a single neural network, each matching the score of a Parzen estimator with a different noise magnitude.

The questions we address here are why learning energy-based models with a single noise level does not permit high-quality sample generation and what can be performed to improve such energy-based models. Our work builds on key ideas from Saremi et al. [26], Saremi and Hyvärinen [27] and Song and Ermon [28].

Section 2 provides a geometric view of the learning problem in denoising-score matching and provides a theoretical explanation of why training with one noise level is insufficient if the data dimension is high.

Section 3 presents a novel method for training an energy-based model, multiscale denoising-score matching (MDSM). Section 4 describes the empirical results of the MDSM model and comparisons with other models.

2. A Geometric View of Denoising-Score Matching

Song and Ermon [28] used denoising-score matching with a range of noise levels, achieving great empirical results. The authors explained that large noise perturbations are required to enable the learning of the score in low-data density regions. But, it is still unclear why a series of different noise levels are necessary, rather than one single noise level that is large enough. Following Saremi and Hyvärinen [27], we analyze the learning process in denoising-score matching based on the measure concentration properties of high-dimensional random vectors.

We adopt the common assumption that the data distribution to be learned is high-dimensional but only has support around a relatively low-dimensional manifold [29,30,31]. If the assumption holds, it causes a problem for score matching: the density, or the gradient of the density, is then undefined outside the manifold, making it difficult to train a valid density model for the data distribution defined on the entire space. Saremi and Hyvärinen [27] and Song and Ermon [28] discussed this problem and proposed to smooth the data distribution with a Gaussian kernel to alleviate the issue.

To further understand the learning in denoising-score matching when the data lie on a manifold $\mathcal{X}$ and the data dimension is high, two elementary properties of random Gaussian vectors in high-dimensional spaces are helpful: First, the length distribution of random vectors becomes concentrated at $\sqrt{d}\sigma$ [32], where ${\sigma }^2$ is the variance of a single dimension. Second, a random vector is always close to orthogonal to a fixed vector [33]. With these premises, one can visualize the configuration of noisy and noiseless data points that enter the learning process: A data point $\mathit{x}$ sampled from $\mathcal{X}$ and its noisy version $\tilde{\mathit{x}}$ always lie on a line which is almost perpendicular to the tangent space $T_{\mathit{x}}\mathcal{X}$ and intersects $\mathcal{X}$ at $\mathit{x}$. Further, the distance vectors between $(\mathit{x},\tilde{\mathit{x}})$ pairs all have a similar length $\sqrt{d}\sigma$. As a consequence, the set of noisy data points concentrates on a set ${\tilde{\mathcal{X}}}_{\sqrt{d}\sigma ,ϵ}$ that has a distance of $(\sqrt{d}\sigma -ϵ,\sqrt{d}\sigma +ϵ)$ from the data manifold $\mathcal{X}$, where $ϵ\ll \sqrt{d}\sigma$.

Therefore, performing denoising-score-matching learning with $(\mathit{x},\tilde{\mathit{x}})$ pairs generated with a fixed noise level $\sigma$, which is the approach taken previously except in [28], will match the score in the set ${\tilde{\mathcal{X}}}_{\sqrt{d}\sigma ,ϵ}$ and enable the denoising of noisy points in the same set. However, the learning provides little information about the density outside this set, farther or closer to the data manifold, as noisy samples outside ${\tilde{\mathcal{X}}}_{\sqrt{d}\sigma ,ϵ}$ rarely appear in the training process. An illustration is presented in Figure 1A.

Let ${\tilde{\mathcal{X}}}_{\sqrt{d}\sigma ,ϵ}^C$ denote the complement of the set ${\tilde{\mathcal{X}}}_{\sqrt{d}\sigma ,ϵ}$. Even if $p_{{\sigma }_0}(\tilde{\mathit{x}}\in {\tilde{\mathcal{X}}}_{\sqrt{d}\sigma ,ϵ}^C)$ is tiny in a high-dimensional space, the score in ${\tilde{\mathcal{X}}}_{\sqrt{d}\sigma ,ϵ}^C$ still plays a critical role in sampling from random initialization. This analysis may explain why models based on denoising-score matching, trained with a single noise level, encounter difficulties in generating data samples with random initialization. For empirical support of this explanation, see our experiments with models trained with single noise magnitudes (Appendix B). To remedy this problem, one has to apply a learning procedure proposed in [28], in which samples with different noise levels are used. Depending on the dimension of the data, the different noise levels have to be spaced narrowly enough to avoid empty regions in the data space. In the following, we will use Gaussian noise and employ a Gaussian scale mixture to produce the noisy data samples for the training (for details, see Section 3.1 and Appendix A).

Another interesting property of denoising-score matching was suggested in the denoising autoencoder literature [1,34]. With an increasing noise level, the learned features tend to have a larger spatial scale. In our experiment, we observe a similar phenomenon when the training model experiences denoising-score matching with a single noise scale. If one compares the samples in Figure A1, Appendix B, it is evident that a noise level of 0.3 produced a model that learned a short-range correlation that spans only a few pixels, a noise level of 0.6 produced a more extended stroke structure without a coherent overall structure and a noise level of 1 produced a more coherent long-range structure without details such as stroke-width variations. This suggests that training with a single noise level for denoising-score matching is insufficient for learning in a model capable of high-quality sample synthesis. For that, a model has to capture data structures at all scales.

3. Learning Energy-Based Model with Multiscale Denoising-Score Matching

3.1. Multiscale Denoising-Score Matching

Motivated by the analysis in Section 2, we strive to develop an EBM based on denoising-score matching that can be trained with noisy samples in which the noise level is not fixed but drawn from a distribution. The model should approximate the Parzen density estimator of the data $p_{{\sigma }_0}\left(\tilde{\mathit{x}}\right)=\int q_{{\sigma }_0}\left(\tilde{\mathit{x}}\right|\mathit{x})p\left(\mathit{x}\right)dx$. Specifically, the learning should minimize the difference between the derivative of the energy and the score of $p_{{\sigma }_0}$ under the expectation ${\mathbb{E}}_{p_M\left(\tilde{\mathit{x}}\right)}$ rather than ${\mathbb{E}}_{p_{{\sigma }_0}\left(\tilde{\mathit{x}}\right)}$, the expectation taken in standard denoising-score matching. Here, $p_M\left(\tilde{\mathit{x}}\right)=\int q_M\left(\tilde{\mathit{x}}\right|\mathit{x})p\left(\mathit{x}\right)dx$ is chosen to cover the signal space more evenly to avoid the measure concentration issue described above. The resulting multiscale score matching (MSM) objective is

$$L_{MSM}\left(\theta \right)={\mathbb{E}}_{p_M\left(\tilde{\mathit{x}}\right)}{\Vert {\nabla }_{\tilde{\mathit{x}}}log\left(p_{{\sigma }_0}\left(\tilde{\mathit{x}}\right)\right)+{\nabla }_{\tilde{\mathit{x}}}E(\tilde{\mathit{x}};\theta )\Vert }^2$$

(4)

Compared to the objective of denoising-score matching (1), the only change in the new objective (4) is the expectation. Both objectives are consistent if $p_M\left(\tilde{\mathit{x}}\right)$ and $p_{{\sigma }_0}\left(\tilde{\mathit{x}}\right)$ have the same support, as shown formally in Proposition A1 of Appendix A. In Proposition A2, we prove that Equation (4) is equivalent to the following denoising-score-matching objective:

$$L_{MDS{M}^{*}}={\mathbb{E}}_{p_M\left(\tilde{\mathit{x}}\right)q_{{\sigma }_0}\left(\mathit{x}\right|\tilde{\mathit{x}})}{\Vert {\nabla }_{\tilde{\mathit{x}}}log\left(q_{\sigma 0}\left(\tilde{\mathit{x}}\right|\mathit{x})\right)+{\nabla }_{\tilde{\mathit{x}}}E(\tilde{\mathit{x}};\theta )\Vert }^2$$

(5)

The above results hold for any noise kernel $q_{{\sigma }_0}\left(\tilde{\mathit{x}}\right|\mathit{x})$, but Equation (5) contains the reversed expectation, which is difficult to evaluate in general. To proceed, we choose $q_{{\sigma }_0}\left(\tilde{\mathit{x}}\right|\mathit{x})$ to be Gaussian, and also choose $q_M\left(\tilde{\mathit{x}}\right|\mathit{x})$ to be a Gaussian scale mixture: $q_M\left(\tilde{\mathit{x}}\right|\mathit{x})=\int q_{\sigma }\left(\tilde{\mathit{x}}\right|\mathit{x})p\left(\sigma \right)d\sigma$ and $q_{\sigma }\left(\tilde{\mathit{x}}\right|\mathit{x})=\mathcal{N}(\mathit{x},{\sigma }^2{I}_d)$. After algebraic manipulation and one approximation (see the derivation following Proposition A2 in Appendix A), we can transform Equation (5) into a more convenient form, which we call multiscale denoising-score matching (MDSM):

$$L_{MDSM}={\mathbb{E}}_{p\left(\sigma \right)q_{\sigma }\left(\tilde{\mathit{x}}\right|\mathit{x})p\left(\mathit{x}\right)}{\Vert {\nabla }_{\tilde{\mathit{x}}}log\left(q_{\sigma 0}\left(\tilde{\mathit{x}}\right|\mathit{x})\right)+{\nabla }_{\tilde{\mathit{x}}}E(\tilde{\mathit{x}};\theta )\Vert }^2$$

(6)

The square loss term evaluated at noisy points $\tilde{\mathit{x}}$ at larger distances from the true data points $\mathit{x}$ will have a much larger magnitude. Therefore, in practice, it is necessary to add a monotonically decreasing term $l\left(\sigma \right)$ to balance the loss in different noise scales, e.g., $l\left(\sigma \right)=\frac{1}{{\sigma }^2}$. Ideally, we want our model to learn the correct gradient everywhere, so we need to add noise at all levels. However, learning denoising-score matching at very large or very small noise levels is useless. At huge noise levels, the information of the original sample is completely lost. Conversely, in the limit of small noise, the noisy sample is virtually indistinguishable from real data. In neither case, one can learn an informative gradient about the data structure. Thus, the noise range must only be broad enough to encourage the learning of data features over all scales. Particularly, we do not sample $\sigma$ but instead choose a series of fixed $\sigma$ values ${\sigma }_1\cdots {\sigma }_K$. Further, substituting $log(q_{{\sigma }_0}\left(\tilde{\mathit{x}}\right|\mathit{x}))=-\frac{{(\tilde{\mathit{x}}-\mathit{x})}^2}{2{\sigma }_0^2}+C$ into Equation (4), we arrive at the final objective:

$$L\left(\theta \right)=\sum _{\sigma \in \{{\sigma }_1\cdots {\sigma }_K\}}{\mathbb{E}}_{q_{\sigma }\left(\tilde{\mathit{x}}\right|\mathit{x})p\left(\mathit{x}\right)}l\left(\sigma \right){\Vert \mathit{x}-\tilde{\mathit{x}}+{\sigma }_0^2{\nabla }_{\tilde{\mathit{x}}}E(\tilde{\mathit{x}};\theta )\Vert }^2$$

(7)

It may seem that ${\sigma }_0$ is an important hyperparameter for our model. Still, after our approximation, ${\sigma }_0$ becomes just a scaling factor in front of the energy function. It can be set to one as long as the temperature range during sampling is scaled accordingly (see Section 3.2). Therefore, the only hyperparameter is the range of noise levels used during training.

On the surface, Objective (7) looks similar to the one in Song and Ermon [28]. The important difference is that Equation (7) approximates a single distribution, namely $p_{{\sigma }_0}\left(\tilde{\mathit{x}}\right)$, the data smoothed with one fixed kernel $q_{{\sigma }_0}\left(\tilde{\mathit{x}}\right|\mathit{x})$. In contrast, Song and Ermon [28] approximate the score of multiple distributions, the family of distributions $\{p_{{\sigma }_i}\left(\tilde{\mathit{x}}\right):i=1,...,n\}$, resulting from the data smoothed by kernels of different widths ${\sigma }_i$. Because our model learns only a single target distribution, it does not require noise magnitude as the input.

3.2. Sampling by Annealed Langevin Dynamics

Langevin dynamics has been used to sample from neural network energy functions [4,5]. However, those studies described difficulties with mode exploration unless many sampling steps were used. We propose incorporating simulated annealing in the Langevin dynamics to improve mode exploration. Simulated annealing [35,36] improves mode exploration by sampling first at a high temperature and then cooling down gradually. This has been successfully applied to challenging computational problems like combinatorial optimization.

To apply simulated annealing to Langevin dynamics, note that in a model of the Brownian motion of a physical particle, the temperature in the Langevin equation enters as a factor $\sqrt{T}$ in front of the noise term; some literature uses $\sqrt{{\beta }^{-1}}$, where $\beta =1/T$ [37]. Adopting the $\sqrt{T}$ convention, the Langevin sampling process [38] is given by:

$${\mathit{x}}_{t+1}={\mathit{x}}_t-\frac{{ϵ}^2}{2}{\nabla }_{\mathit{x}}E({\mathit{x}}_t;\theta )+ϵ\sqrt{T_t}\mathcal{N}(0,I_d)$$

(8)

where $T_t$ follows some annealing schedule and $ϵ$ denotes the step length, which is fixed. During sampling, samples behave like physical particles under Brownian motion in a potential field. Because the particles have average energies close to their current thermic energy, they explore the state space at different distances from the data manifold depending on the temperature. Eventually, they settle somewhere on the data manifold. The behavior of the particle’s energy value during a typical annealing process is depicted in Appendix F in Figure A7B.

If the obtained sample is still slightly noisy, we can apply a single-step gradient denoising jump [27] to improve the sample quality:

$${\mathit{x}}_{clean}={\mathit{x}}_{noisy}-{\sigma }_0^2{\nabla }_{\mathit{x}}E({\mathit{x}}_{noisy};\theta )$$

(9)

This denoising procedure can be applied to noisy samples with any level of Gaussian noise because, in our model, the gradient automatically has the correct magnitude to denoise the sample. This process is justified by the Empirical Bayes interpretation of this denoising process, as studied in [27].

Song and Ermon [28] also call their sample-generation process annealed Langevin dynamics. It should be noted that their sampling process does not coincide with Equation (8). Their sampling procedure is best understood as sequentially sampling a series of distributions corresponding to data distribution corrupted by different noise levels.

4. Image Modeling Results

Training and Sampling Details. The proposed energy-based model is trained on standard image datasets, specifically MNIST, Fashion MNIST, CelebA [39] and CIFAR-10 [40]. During training, we set ${\sigma }_0=0.1$ and train over a noise range of $\sigma \in [0.05,1.2]$, with the different noise uniformly spaced on the batch dimension. For MNIST and Fashion MNIST, we used geometrically distributed noise in the range $[0.1,3]$. The weighting factor $l\left(\sigma \right)$ is always set to $1/{\sigma }^2$ to make the square term roughly independent of $\sigma$. We use a batch size of 128 and the Adam optimizer with a $5\times {10}^{-5}$ learning rate. For MNIST and Fashion MNIST, we use a 12-layer ResNet with 64 filters. For the CelebA and CIFAR-10 datasets, we used a 18-layer ResNet with 128 filters [41,42]. No normalization layer was used in any of the networks. We designed the output layer of all networks to take a generalized quadratic form [43]. Because the energy function is anticipated to be approximately quadratic with respect to the noise level, this modification boosted the performance significantly. For more detail on the training and model architecture, see Appendix D. One notable result is that since our training method does not involve sampling, we achieved a speed up of roughly an order of magnitude compared to the maximum-likelihood training using Langevin dynamics (for example, on a single GPU, training MNIST with a 12-layer ResNet takes 0.3 s per batch with our method, while maximum-likelihood training with a modest 30 Langevin steps per weight update takes 3 s per batch. Both methods need a similar number of weight updates to train). Our approach thus enables the training of energy-based models even when limited computational resources prohibit maximum-likelihood methods.

We found that the choice of the maximum noise level has little effect on learning as long as it is large enough to encourage the learning of the longest range features in the data. However, as expected, learning with too small or too large noise levels is not beneficial and can even destabilize the training process. Further, our method appeared to be relatively insensitive to how the noise levels are distributed over a chosen range. Geometrically spaced noise as in [28] and linearly spaced noise both work, although in our case, learning with linearly spaced noise was somewhat more robust.

To sample the learned energy function, we used annealed Langevin dynamics with an annealing schedule where the temperature varies continuously. See Figure A7B for the particular shape of our annealing schedule. In contrast, annealing schedules with a theoretical guaranteed convergence property takes extremely long [44]. The range of temperatures to use in the sampling process depends on the choice of ${\sigma }_0$ as the equilibrium distribution contains rough images with a Gaussian noise of magnitude $\sqrt{T}{\sigma }_0$ added on top. To ease traveling between modes far apart and ensure even sampling, the initial temperature needs to be high enough to inject noise of sufficient magnitude. The choice of $T=100$, corresponding to added noise of magnitude $\sqrt{100}\times 0.1=1$, is an adequate starting point. For step length $ϵ$, we generally used $0.02$, and $[0.015,0.05]$ appeared to be a reasonable range for this parameter. After annealing, we performed single-step denoising to enhance the sample quality slightly.

Unconditional Image Generation. We demonstrate the generative ability of our model by displaying samples obtained by annealed Langevin sampling and single-step denoising jump. We evaluated 50k sampled images after training on CIFAR-10 with two performance scores, Inception [45] and FID [46]. We achieved an Inception Score of 8.31 and an FID of 31.7, comparable to modern GAN approaches. In Figure 2, we display some samples for visual inspection. The scores of the CelebA dataset are not reported here as they are not commonly reported and may depend on the specific preprocessing used. More samples and training images are provided in the Appendix E for visual inspection. We believe the visual assessment is still essential because of the possible issues with the Inception Score [47]. Indeed, we also found that the visually impressive samples were not necessarily the ones achieving the highest Inception Score.

Although overfitting is not a common concern for generative models, we still tested our model for overfitting. We found no indication of overfitting by comparing the model samples with their nearest neighbors in the dataset. See Figure A2 in Appendix C.

Mode Coverage. We repeated, with our model, the three-channel MNIST mode coverage experiment similar to the one in [16]. An energy-based model was trained on three-channel data where each channel is a random MNIST digit. Then, 8000 samples were taken from the model, and each channel was classified by using a small MNIST classifier network. We obtained the results of the 966 modes, comparable to GAN approaches. The training was successful, and our model assigned low energy to all the learned modes. But, some modes were not accessed during sampling, likely due to the Langevin dynamics failing to explore these modes. A better sampling technique such as HMC [48] or a Maximum Entropy Generator [16] could improve this result.

Image Inpainting. Image inpainting can be achieved with our model by clamping a part of the image to ground truth and performing the same annealed Langevin and jump-sampling procedure on the missing part of the image. Noise appropriate for the sampling temperature must be added to the clamped inputs. The quality of the inpainting results of our model trained on CelebA and CIFAR-10 can be assessed in Figure 3. For the CIFAR-10 inpainting results, we used the test set.

Log-Likelihood Estimation. For energy-based models, the log density can be obtained after estimating the partition function with Annealed Importance Sampling (AIS) [49] or Reverse AIS [50]. In our experiment on the CIFAR-10 model, similar to reports in [5], there is still a substantial gap between AIS and Reverse AIS estimation, even after significant computational effort. In

Table 1. Unconditional Inception Score, FID scores and likelihoods for CIFAR-10. Arrow indicates the better direction of the score.

Model	IS ↑	FID ↓	Likelihood	NLL (bits/dim) ↓
iResNet [51]	-	65.01	Yes	3.45
PixelCNN [12]	4.60	65.93	Yes	3.14
PixelIQN [13]	5.29	49.46	Yes	-
Residual Flow [52]	-	46.37	Yes	3.28
GLOW [11]	-	46.90	Yes	3.35
EBM (ensemble) [5]	6.78	38.2	Yes	- 1
MDSM (Ours)	8.31	31.7	Yes	7.04 2
SNGAN [53]	8.22	21.7	No	-
NCSN [28]	8.91	25.32	No	-

¹ Author-reported difficulties evaluating likelihood. ² Upper bound obtained by Reverse AIS.

, we report the results from Reverse AIS, as it tends to overestimate the partition function, thus underestimating the density. Our reported density value on the CIFAR dataset underperforms compared to other models, likely due to two reasons: The model is approximating a Gaussian kernel density estimator of the data distribution, which is not a very good model on its own. Also, the lower bound obtained by Reverse AIS may not be tight due to the difficulty in sampling.

We also report a density of 6.79 bits/dim on the MNIST dataset, again not comparable to other density models. The density reported here follows the convention of measuring the density of pixel values between $[0,255]$. More details on this experiment are provided in the Appendix D.

Outlier Detection. Choi et al. [3] and Nalisnick et al. [54] reported the intriguing behavior of high-dimensional density models on out-of-distribution samples. Specifically, they showed that many models assign a higher likelihood to out-of-distribution samples than real data samples. We investigated whether our model behaves similarly.

Our energy function is only trained outside the data manifold where samples are noisy, so the energy value at clean data points may not always be well-behaved. Therefore, we added noise with magnitude ${\sigma }_0$ before measuring the energy value. Our network behaves similarly to previous likelihood models; it assigns lower energy, and thus a higher density, to some OOD samples. We show one example of this phenomenon in Appendix A in Figure A7.

We also attempted to use the denoising performance, or the objective function, to perform outlier detection. Intriguingly, the results are similar to using the energy value. The denoising performance correlates more with the original image’s variance than the image’s content.

5. Discussion

The central goal of our work is to investigate how to build EMBs in high-dimensional spaces with an objective function similar to denoising-score matching or the “Fisher divergence”. We first provided analyses and empirical results for understanding the limitations of learning the structure of high-dimensional data with denoising-score matching. We found that the objective function $L_{SM}$ confines learning to a small set due to the measure concentration phenomenon in high-dimensional random vectors. Therefore, sampling the learned distribution starting from outside the set, where the gradient is learned more accurately, does not produce good results. In our opinion, the expectation with respect to the target distribution ${\mathbb{E}}_{p_{\sigma 0}\left(\tilde{\mathit{x}}\right)}$ is not of critical importance, and it only enforces the score matching in the confined high-probability region. Since sampling from a random initial location requires the model score to match the target distribution score everywhere, we propose that ${\mathbb{E}}_{p_{\sigma 0}\left(\tilde{\mathit{x}}\right)}$ should be replaced by ${\mathbb{E}}_{p_M\left(\tilde{\mathit{x}}\right)}$, where $p_M\left(\tilde{\mathit{x}}\right)$ covers a much larger range in the signal space than $p_{\sigma 0}\left(\tilde{\mathit{x}}\right)$. This leads to the multiscale denoising-score matching, which can be viewed as “multiscale Fisher Divergence”. The resulting multiscale denoising-score matching (MDSM) EBM model can denoise, produce high-quality samples from random noise, perform image inpainting, etc. While also providing density information, our model learns an order of magnitude faster than the models based on maximum likelihood and sampling.

Previous efforts to learn energy-based models with score matching [24,55] cannot produce high-quality samples and sometimes are computationally intensive. Saremi et al. [26] and Saremi and Hyvärinen [27] trained energy-based models with the denoising-score-matching objective. Their method is computationally efficient, but the resulting models cannot perform sample synthesis from random noise initialization. The NCSN recently proposed by Song and Ermon [28] is not an EBM, but it is capable of high-quality sample synthesis. This model learns a sequence of score functions. Each approximates the data distribution smoothed by a different-sized Gaussian. Sample generation in NCSN is achieved by sequential sampling from this set of distributions. Our MDSM method instead learns an energy-based model corresponding to $p_{{\sigma }_0}\left(\tilde{\mathit{x}}\right)$ for a fixed ${\sigma }_0$. This method improves score matching in high-dimensional space by matching the gradient of an energy function to the score of $p_{{\sigma }_0}\left(\tilde{\mathit{x}}\right)$ in the whole space and avoids measure-concentration issues.

All told, we offer a novel EBM model that achieves high-quality sample synthesis, which provides a new state-of-the-art approach among EBM approaches. Compared to the NCSN model, our model is more parsimonious and can support single-step denoising without prior knowledge of the noise magnitude. Our model performs slightly worse than the NCSN model in terms of sample quality, which could have several reasons. First, the derivation of Equation (6) requires an approximation to keep the training procedure tractable, which could be inaccurate. Second, the NCSN’s output is a vector that, at least during optimization, does not always have to be the derivative of a scalar function. In contrast, in our model, the network output is scalar. Thus, the NCSN model may perform better because it explores a more extensive set of functions during optimization.