How Much Is Enough? A Study on Diffusion Times in Score-Based Generative Models

Abstract

Score-based diffusion models are a class of generative models whose dynamics is described by stochastic differential equations that map noise into data. While recent works have started to lay down a theoretical foundation for these models, a detailed understanding of the role of the diffusion time T is still lacking. Current best practice advocates for a large T to ensure that the forward dynamics brings the diffusion sufficiently close to a known and simple noise distribution; however, a smaller value of T should be preferred for a better approximation of the score-matching objective and higher computational efficiency. Starting from a variational interpretation of diffusion models, in this work we quantify this trade-off and suggest a new method to improve quality and efficiency of both training and sampling, by adopting smaller diffusion times. Indeed, we show how an auxiliary model can be used to bridge the gap between the ideal and the simulated forward dynamics, followed by a standard reverse diffusion process. Empirical results support our analysis; for image data, our method is competitive with regard to the state of the art, according to standard sample quality metrics and log-likelihood.

1. Introduction

Diffusion-based generative models [1,2,3,4,5,6,7] have recently gained popularity due to their ability to synthesize high-quality audio [8,9], image [10,11] and other data modalities [12], outperforming known methods based on Generative Adversarial Networks (GANs) [13], normalizing flows (NFs) [14] or Variational Autoencoders (vaes) and Bayesian autoencoders (BAEs) [15,16].

Diffusion models learn to generate samples from an unknown density $p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}$ by reversing a diffusion process which transforms the distribution of interest into noise. The forward dynamics injects noise into the data following a diffusion process that can be described by a Stochastic Differential Equation (SDE) of the form

$$\mathrm{d}{\mathit{x}}_t=\mathit{f}({\mathit{x}}_t,t)\mathrm{d}t+g\left(t\right)\mathrm{d}{\mathit{w}}_t\mathrm{with}{\mathit{x}}_0\sim p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}},$$

(1)

where ${\mathit{x}}_t$ is a random variable at time t, $\mathit{f}(·,t)$ is the drift term, $g(·)$ is the diffusion term and ${\mathit{w}}_t$ is a Wiener process (or Brownian motion). We also consider a special class of linear SDEs, for which the drift term is decomposed as $\mathit{f}({\mathit{x}}_t,t)=\alpha \left(t\right){\mathit{x}}_t$, where the function $\alpha \left(t\right)\le 0$ for all t, and the diffusion term is independent of ${\mathit{x}}_t$. This class of parameterizations of SDEs is known as affine and it admits analytic solutions. We denote the time-varying probability density by $p(\mathit{x},t)$, where, by definition $p(\mathit{x},0)=p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)$, and the conditional on the initial condition ${\mathit{x}}_0$ by $p(\mathit{x},t|{\mathit{x}}_0)$. The forward SDE is usually considered for a “sufficiently long” diffusion time T, leading to the density $p(\mathit{x},T)$. In principle, when $T\to \infty$, $p(\mathit{x},T)$ converges to Gaussian noise, regardless of initial conditions.

For generative modeling purposes, we are interested in the inverse dynamics of such process, i.e., transforming samples of the noisy distribution $p(\mathit{x},T)$ into $p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)$. Such dynamics can be obtained by considering the solutions of the inverse diffusion process [17],

$$\begin{array}{c}\hfill \mathrm{d}{\mathit{x}}_t=\left[-\mathit{f}({\mathit{x}}_t,t^{\prime })+g^2\left(t^{\prime }\right)\nabla logp({\mathit{x}}_t,t^{\prime })\right]\mathrm{d}t+g\left(t^{\prime }\right)\mathrm{d}{\mathit{w}}_t,\end{array}$$

(2)

where $t^{\prime }\stackrel{\mathrm{def}}{=}T-t$, with the inverse dynamics involving a new Wiener process. Given $p(\mathit{x},T)$ as the initial condition, the solution of Equation (2) Equation (2) after a reverse diffusion time T, will be distributed as $p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)$. We refer to the density associated to the backward process as $q(\mathit{x},t^{\prime })$. The simulation of the backward process is referred to as sampling and, differently from the forward process, this process is not affine and a closed form solution is out of reach.

Practical considerations on diffusion times. In practice, diffusion models are challenging to work with [3]. Indeed, a direct access to the true score function $\nabla logp({\mathit{x}}_t,t)$ required in the dynamics of the reverse diffusion is unavailable. This can be solved by approximating it with a parametric function ${\mathit{s}}_{\mathsf{\theta }}({\mathit{x}}_t,t)$, e.g., a neural network, which is trained using the following loss function:

$$\begin{array}{c}\hfill \mathcal{L}\left(\mathsf{\theta }\right)={\int }_0^T{\mathbb{E}}_{\sim \left(1\right)}\lambda \left(t\right)\Vert {{\mathit{s}}_{\mathsf{\theta }}({\mathit{x}}_t,t)-\nabla logp({\mathit{x}}_t,t|{\mathit{x}}_0)\Vert }^2\end{array}$$

(3)

where $\lambda \left(t\right)$ is a positive weighting factor and the notation ${\mathbb{E}}_{\sim \left(1\right)}$ means that the expectation is taken with respect to the random process ${\mathit{x}}_t$ in Equation (1) for a generic function h, ${\mathbb{E}}_{\sim \left(1\right)}\left[h({\mathit{x}}_t,{\mathit{x}}_0,t)\right]=\int h(\mathit{x},\mathbf{z},t)p(\mathit{x},t|\mathbf{z})p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathbf{z}\right)\mathrm{d}\mathit{x}\mathrm{d}\mathbf{z}$. The loss in Equation (3), usually referred to as score matching loss, is the cost function considered in [18] (Equation (4)). The condition $\lambda \left(t\right)=g{\left(t\right)}^2$, which we use in this work, is referred to a likelihood reweighting. Due to the affine property of the drift, the term $p({\mathit{x}}_t,t|{\mathit{x}}_0)$ is analytically known and normally distributed for all t (expression available in Table 1, and in Särkkä and Solin [19]). Intuitively, the estimation of the score is akin to a denoising objective, which operates in a challenging regime. Later, we will quantify the difficulty of learning the score, as a function of T.

While the forward and reverse diffusion processes are valid for all T, the noise distribution $p(\mathit{x},T)$ is analytically known only when the diffusion time is $T\to \infty$. Then, the common solution is to replace $p(\mathit{x},T)$ with a simple (i.e., easy to sample) distribution $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)$, which, for the classes of SDEs that we consider in this work, is a Gaussian distribution.

In the literature, the discrepancy between $p(\mathit{x},T)$ and $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)$ has been neglected, under the informal assumption of a sufficiently large diffusion time. Unfortunately, while this approximation seems a valid approach to simulate and generate samples, the reverse diffusion process starts from an initial condition $q(\mathit{x},0)$ which is different from $p(\mathit{x},T)$ and, as a consequence, it will converge to a solution $q(\mathit{x},T)$ that is different from the true $p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)$. Later, we will expand on the error introduced by this approximation, but for illustration purposes, Figure 1 shows quantitatively this behavior for a simple 1D toy example where we set the data distribution equal to a mixture of normal ($\mathcal{N}$) distributions as $p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)=\pi \mathcal{N}(1,0.1^2)+(1-\pi )\mathcal{N}(3,0.5^2)$, with $\pi =0.3$. When T is small, the distribution $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)$ is very different from $p(\mathit{x},T)$ and samples from $q(\mathit{x},T)$ exhibit very low likelihood of being generated from $p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)$.

Crucially, Figure 1 (zoomed region) illustrates an unknown behavior of diffusion models, which we unveil in our analysis. The right balance between efficient score estimation and sampling quality can be achieved by diffusion times that are smaller than common best practices. Moreover, even excessively large diffusion times can be detrimental. This is a key observation that we explore in our work.

Contributions. An appropriate choice of the diffusion time T is a key factor that impacts training convergence, sampling time and quality. On the one hand, the approximation error introduced by considering initial conditions for the reverse diffusion process drawn from a simple distribution $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)\ne p(\mathit{x},T)$ increases when T is small. This is why the current best practice is to choose a sufficiently long diffusion time. On the other hand, training convergence of the score model ${\mathit{s}}_{\mathsf{\theta }}({\mathit{x}}_t,t)$ becomes more challenging to achieve with a large T, which also imposes extremely high computational costs both for training and for sampling. This would suggest to choose a smaller diffusion time. Given the importance of this problem, in this work, we set off to study the existence of suitable operating regimes to strike the right balance between computational efficiency and model quality. The main contributions of this work are the following:

Contribution 1: 

We use an evidence lower bound (ELBO) decomposition which allows us to study the impact of the diffusion time T. This ELBO decomposition emphasizes the roles of (i) the discrepancy between the “ending” distribution of the diffusion and the “starting” distribution of the reverse diffusion processes, and (ii) of the score matching objective. Crucially, our analysis does not rely on assumptions on the quality of the score models. We explicitly study the existence of a trade-off and explore experimentally, for the first time, current approaches for selecting the diffusion time T.

Contribution 2: 

In Section 3, we propose a novel method to improve both the training and sampling efficiency of diffusion-based models, while maintaining high sample quality. Our method introduces an auxiliary distribution, allowing us to transform the simple “starting” distribution of the reverse process used in the literature so as to minimize the discrepancy to the “ending” distribution of the forward process. Then, a standard reverse diffusion can be used to closely match the data distribution. Intuitively, our method allows to build “bridges” across multiple distributions, and to set T toward the advantageous regime of small diffusion times.

In addition to our methodological contributions, in Section 4, we provide experimental evidence of the benefits of our method, in terms of sample quality and log likelihood. Finally, we conclude this work in Section 5.

Related Work. A concurrent work by Zheng et al. [20] presents an empirical study of a truncated diffusion process but lacks a rigorous analysis and a clear justification for the proposed approach. Recent attempts by Lee et al. [9] to optimize $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$, or the proposal to do so [21], have been studied in different contexts. Related work focus primarily on improving sampling efficiency (but not training efficiency), using a wide array of techniques. Sample generation times can be drastically reduced considering adaptive step-size integrators [22]. Such methods are complementary to our approach, and can be used in combination with the techniques we propose in this work. Other popular choices are based on merging multiple steps of a pretrained model through distillation techniques [23] or by taking larger sampling steps with GANs [24]. Approaches closer to ours modify the sde, or the discrete time processes, to obtain inference efficiency gains. In particular, Song et al. [7] considers implicit non-Markovian diffusion processes, while Watson et al. [25] changes the diffusion processes by optimal scheduling selection, and Dockhorn et al. [26] considers overdamped SDEs. Finally, hybrid techniques combining VAEs and diffusion models [4] or simple auto encoders and diffusion models [27] have positive effects on training and sampling times.

Moreover, we remark that a simple modification of the noise schedule to steer the diffusion process toward a small diffusion time [5,28] is not a viable solution. As we discuss in Section 2.4, the optimal value of the ELBO, in the case of affine SDEs, is invariant to the choice of the noise schedule. Naively selecting a faster noise schedule does not provide any practical benefit in terms of computational complexity, as it requires smaller step sizes to keep the same accuracy of the original noise schedule simulation. However, the optimization of the noise schedule can have important practical effects on the stability of training and variance of estimations [5]. Finally, few other works in the literature attempt to study the convergence properties of diffusion models. For instance, De Bortoli et al. [29] obtain a total variation bound between the generated and data distribution under maximum error assumptions between true and approximated score. De Bortoli [30] relaxes this requirement obtaining a bound in terms of Wasserstein distance. Lee et al. [31] show how the total variation bound can be expressed as a function of the maximum score error and find that the bound is optimized for a diffusion time that depends on this error. Our work, on the other hand, does not make any assumption and aims at selecting the smallest possible diffusion time to maximize training and sampling efficiency.

2. A Tradeoff on Diffusion Time

The dynamics of a diffusion model can be studied through the lens of variational inference, which allows us to bound the (log-)likelihood using an evidence lower bound (ELBO) [32]. The interpretation we consider in this work (see also [18], Theorem 1) emphasizes the two main factors affecting the quality of sample generation: an imperfect score and a mismatch, measured in terms of $\mathrm{KL}\left[logp(\mathit{x},T)\parallel p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)\right]$, the Kullback-Leibler (KL) divergence between the noise distribution $p(\mathit{x},T)$ of the forward process and the distribution $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$ used to initialize the backward process.

2.1. Preliminaries: The ELBO Decomposition

Our goal is to study the quality of the generated data distribution as a function of the diffusion time T. Instead of focusing on the log-likelihood bounds for single datapoints $logq(\mathit{x},T)$, we consider the average over the data distribution, i.e., the cross-entropy ${\mathbb{E}}_{p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)}logq(\mathit{x},T)$. By rewriting the ${\mathcal{L}}_{\mathrm{ELBO}}$ derived in Huang et al. [32] [Equation (25)] (details of the steps in Appendix B), we have that

$$\begin{array}{cc}\hfill & {\mathbb{E}}_{p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)}logq(\mathit{x},T)\ge {\mathcal{L}}_{\mathrm{ELBO}}({\mathit{s}}_{\mathsf{\theta }},T)={\mathbb{E}}_{\sim \left(1\right)}log{p}_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left({\mathit{x}}_T\right)-\mathit{I}({\mathit{s}}_{\mathsf{\theta }},T)+R\left(T\right)\hfill \end{array}$$

(4)

where $R\left(T\right)=\frac{1}{2}\underset{t=0}{\overset{T}{\int }}{\mathbb{E}}_{\sim \left(1\right)}\left[g^2\left(t\right){\nabla logp({\mathit{x}}_t,t|{\mathit{x}}_0)}^2-2{\mathit{f}}^{\top }({\mathit{x}}_t,t)\nabla logp({\mathit{x}}_t,t|{\mathit{x}}_0)\right]\mathrm{d}t$, and $\mathit{I}({\mathit{s}}_{\mathsf{\theta }},T)=\frac{1}{2}\underset{t=0}{\overset{T}{\int }}{g}^2\left(t\right){\mathbb{E}}_{\sim \left(1\right)}\left[{{\mathit{s}}_{\mathsf{\theta }}({\mathit{x}}_t,t)-\nabla logp({\mathit{x}}_t,t|{\mathit{x}}_0)}^2\right]\mathrm{d}t$ is equal to the loss term Equation (3) when $\lambda \left(t\right)=g^2\left(t\right)$.

Note that $R\left(T\right)$ depends neither on ${\mathit{s}}_{\mathsf{\theta }}$ nor on $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$, while $\mathit{I}({\mathit{s}}_{\mathsf{\theta }},T)$, or an equivalent reparameterization [18,32] [Equation (1)], is used to learn the approximated score , by optimization of the parameters $\mathsf{\theta }$. It is then possible to show that

$$\mathit{I}({\mathit{s}}_{\mathsf{\theta }},T)\ge \underset{\stackrel{\mathrm{def}}{=}K\left(T\right)}{\underbrace{\mathit{I}(\nabla logp,T)}}=\frac{1}{2}\underset{t=0}{\overset{T}{\int }}{g}^2\left(t\right){\mathbb{E}}_{\sim \left(1\right)}{\left[\nabla logp({\mathit{x}}_t,t)-\nabla logp({\mathit{x}}_t,t|{\mathit{x}}_0)\right]}^2\mathrm{d}t.$$

(5)

Note that the term $K\left(T\right)=\mathit{I}(\nabla logp,T)$ does not depend on $\mathsf{\theta }$. Consequently, we can define $\mathcal{G}({\mathit{s}}_{\mathsf{\theta }},T)=\mathit{I}({\mathit{s}}_{\mathsf{\theta }},T)-K\left(T\right)$ (see Appendix C for details), where $\mathcal{G}({\mathit{s}}_{\mathsf{\theta }},T)$ is a positive term that we call the gap term, accounting for the practical case of an imperfect score , i.e., ${\mathit{s}}_{\mathsf{\theta }}({\mathit{x}}_t,t)\ne \nabla logp({\mathit{x}}_t,t)$. It also holds that

$$\begin{array}{cc}\hfill {\mathbb{E}}_{\sim \left(1\right)}log{p}_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left({\mathit{x}}_T\right)& =\int \left[log{p}_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)-logp(\mathit{x},T)+logp(\mathit{x},T)\right]p(\mathit{x},T)\mathrm{d}\mathit{x}=\hfill \\ \hfill & ={\mathbb{E}}_{\sim \left(1\right)}logp({\mathit{x}}_T,T)-1.10kl\left[logp(\mathit{x},T)\parallel p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)\right].\hfill \end{array}$$

(6)

Therefore, we can substitute the cross-entropy term ${\mathbb{E}}_{\sim \left(1\right)}log{p}_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left({\mathit{x}}_T\right)$ of the ELBO in Equaiton (4) to obtain

$$\begin{array}{cc}\hfill & {\mathbb{E}}_{p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)}logq(\mathit{x},T)\ge \hfill \\ & -\mathrm{KL}\left[p(\mathit{x},T)\parallel p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)\right]+{\mathbb{E}}_{\sim \left(1\right)}logp({\mathit{x}}_T,T)-K\left(T\right)+R\left(T\right)-\mathcal{G}({\mathit{s}}_{\mathsf{\theta }},T).\hfill \end{array}$$

(7)

Before concluding our derivation, we show how to combine different terms of Equation (7) into the negative entropy term ${\mathbb{E}}_{p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)}log{p}_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)$. Given the stochastic dynamics defined in Equtaion (1), it holds that (see derivation and details in Appendix D)

$$\begin{array}{c}\hfill {\mathbb{E}}_{\sim \left(1\right)}logp({\mathit{x}}_T,T)-K\left(T\right)+R\left(T\right)={\mathbb{E}}_{p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)}log{p}_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right).\end{array}$$

(8)

Finally, we can now bound the value of ${\mathbb{E}}_{p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)}logq(\mathit{x},T)$ as

$${\mathbb{E}}_{p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)}logq(\mathit{x},T)\ge \underset{{\mathcal{L}}_{\mathrm{ELBO}}({\mathit{s}}_{\mathsf{\theta }},T)}{\underbrace{{\mathbb{E}}_{p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)}log{p}_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)-\mathcal{G}({\mathit{s}}_{\mathsf{\theta }},T)-\mathrm{KL}\left[p(\mathit{x},T)\parallel p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)\right]}}.$$

(9)

Equation (9) clearly emphasizes the roles of an approximate score function, through the gap term $\mathcal{G}(·)$, and the discrepancy between the noise distribution of the forward process and the initial distribution of the reverse process through the KL term. The (negative) entropy term ${\mathbb{E}}_{p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)}log{p}_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)$, which is constant with regard to T and $\mathsf{\theta }$, is the best value achievable by the ELBO. Indeed, by rearranging Equation (9), $\mathrm{KL}\left[p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)\parallel q(\mathit{x},T)\right]\le \mathcal{G}({\mathit{s}}_{\mathsf{\theta }},T)+\mathrm{KL}\left[p(\mathit{x},T)\parallel p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)\right]$. Optimality is achieved when (i) we have perfect score matching and (ii) the initial conditions for the reverse process are ideal, i.e., $q(\mathit{x},0)=p(\mathit{x},T)$.

Next, we show the existence of a tradeoff: the KL decreases with T, while the gap increases with T.

2.2. The Tradeoff on Diffusion Time

We begin by showing that the KL term in Equation (9) decreases with the diffusion time T, which induces to select large T to maximize the ELBO.

We consider the two main classes of SDEs for the forward diffusion process defined in Equation (1): SDEs whose steady state distribution is the standard multivariate Gaussian, referred to as Variance Preserving (VP), and SDEs without a stationary distribution, referred to as Variance Exploding (VE), which we summarize in Table 1. The standard approach to generate new samples relies on the backward process defined in Equation (2), and consists in setting $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$ in agreement with the form of the forward process SDE. The following result bounds the discrepancy between the noise distribution $p(\mathit{x},T)$ and $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$.

Table 1. Two main families of diffusion processes, where ${\sigma }^2\left(t\right)={\left(\frac{{\sigma }_{\mathit{max}}^2}{{\sigma }_{\mathit{min}}^2}\right)}^t$ and $\beta \left(t\right)={\beta }_0+({\beta }_1-{\beta }_0)t$.

	Diffusion Process	$\mathit{p}({\mathit{x}}_{\mathit{t}},\mathit{t}|{\mathit{x}}_0)=\mathcal{N}(\mathit{m},\mathit{s}\mathit{I})$	${\mathit{p}}_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)$
Variance Exploding	$\alpha \left(t\right)=0$, $g\left(t\right)=\sqrt{\frac{{\sigma }^2\left(t\right)}{\mathrm{d}t}}$	$m={\mathit{x}}_0$, $\mathit{s}={\sigma }^2\left(t\right)-{\sigma }^2\left(0\right)$	$\mathcal{N}(0,({\sigma }^2\left(T\right)-{\sigma }^2\left(0\right))\mathit{I})$
Variance Preserving	$\alpha \left(t\right)=-\frac{1}{2}\beta \left(t\right)$, $g\left(t\right)=\sqrt{\beta \left(t\right)}$	$m=e^{-\frac{1}{2}{\int }_0^t\beta \left(\mathrm{d}\tau \right)}{\mathit{x}}_0$, $\mathit{s}=1-e^{-{\int }_0^t\beta \left(\mathrm{d}\tau \right)}$	$\mathcal{N}(0,\mathit{I})$

Table 1. Two main families of diffusion processes, where ${\sigma }^2\left(t\right)={\left(\frac{{\sigma }_{\mathit{max}}^2}{{\sigma }_{\mathit{min}}^2}\right)}^t$ and $\beta \left(t\right)={\beta }_0+({\beta }_1-{\beta }_0)t$.

	Diffusion Process	$\mathit{p}({\mathit{x}}_{\mathit{t}},\mathit{t}|{\mathit{x}}_0)=\mathcal{N}(\mathit{m},\mathit{s}\mathit{I})$	${\mathit{p}}_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)$
Variance Exploding	$\alpha \left(t\right)=0$, $g\left(t\right)=\sqrt{\frac{{\sigma }^2\left(t\right)}{\mathrm{d}t}}$	$m={\mathit{x}}_0$, $\mathit{s}={\sigma }^2\left(t\right)-{\sigma }^2\left(0\right)$	$\mathcal{N}(0,({\sigma }^2\left(T\right)-{\sigma }^2\left(0\right))\mathit{I})$
Variance Preserving	$\alpha \left(t\right)=-\frac{1}{2}\beta \left(t\right)$, $g\left(t\right)=\sqrt{\beta \left(t\right)}$	$m=e^{-\frac{1}{2}{\int }_0^t\beta \left(\mathrm{d}\tau \right)}{\mathit{x}}_0$, $\mathit{s}=1-e^{-{\int }_0^t\beta \left(\mathrm{d}\tau \right)}$	$\mathcal{N}(0,\mathit{I})$

Lemma 1. 

For the classes of SDEs considered (Table 1), the discrepancy between $p(\mathit{x},T)$ and the $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)$ can be bounded as follows.

For Variance Preserving SDEs, it holds that:$\mathrm{KL}\left[p(\mathit{x},T)\parallel p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)\right]\le C_{1}exp(-{\int }_0^T\beta \left(t\right)dt)$.

For Variance Exploding SDEs, it holds that:$\mathrm{KL}\left[p(\mathit{x},T)\parallel p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)\right]\le C_2\frac{1}{{\sigma }^2\left(T\right)-{\sigma }^2\left(0\right)}$.

Our proof uses results from Villani [33], the logarithmic Sobolev Inequality and Gronwall inequality (see Appendix E for details). The consequence of Lemma 1 is that to maximize the ELBO, the diffusion time T should be as large as possible (ideally, $T\to \infty$), such that the KL term vanishes. This result is in line with current practices for training score-based diffusion processes, which argue for sufficiently long diffusion times [29]. Our analysis, on the other hand, highlights how this term is only one of the two contributions to the ELBO.

Now, we focus our attention on studying the behavior of the second component, $\mathcal{G}(·)$. Before that, we define a few quantities that allow us to write the next important result.

Definition 1. 

We define the optimal score ${\widehat{\mathit{s}}}_{\mathsf{\theta }}$ for any diffusion time T, as the score obtained using parameters that minimize $\mathit{I}({\mathit{s}}_{\mathsf{\theta }},T)$. Similarly, we define the  optimal score gap $\mathcal{G}({\widehat{\mathit{s}}}_{\mathsf{\theta }},T)$ for any diffusion time T, as the gap attained when using the optimal score.

The optimal score gap term $\mathcal{G}({\widehat{\mathit{s}}}_{\mathsf{\theta }},T)$ is a non-decreasing function in T. That is, given $T_2>T_1$, and ${\mathsf{\theta }}_1={arg\; min}_{\mathsf{\theta }}\mathit{I}({\mathit{s}}_{\mathsf{\theta }},T_1),{\mathsf{\theta }}_2={arg\; min}_{\mathsf{\theta }}\mathit{I}({\mathit{s}}_{\mathsf{\theta }},T_2)$, then $\mathcal{G}({\mathit{s}}_{{\mathsf{\theta }}_2},T_2)\ge \mathcal{G}({\mathit{s}}_{{\mathsf{\theta }}_1},T_1)$. The proof (see Appendix F) is a direct consequence of the definition of $\mathcal{G}$ and the optimality of the score.

Note that Section 2.2 does not imply that $\mathcal{G}({\mathit{s}}_{{\mathsf{\theta }}_a},T_2)\ge \mathcal{G}({\mathit{s}}_{{\mathsf{\theta }}_b},T_1)$ holds for generic parameters ${\mathsf{\theta }}_a,{\mathsf{\theta }}_b$.

2.3. Is There an Optimal Diffusion Time?

While diffusion processes are generally studied for $T\to \infty$, diffusion times in score-based models have been arbitrarily set to be “sufficiently large” in the literature. Here we formally argue about the existence of an optimal diffusion time, which strikes the right balance between the gap $\mathcal{G}(·)$ and the KL terms of the ELBO in Equation (9).

Before proceeding any further, we clarify that our final objective in this work is not to find and use an optimal diffusion time. Instead, our result on the existence of optimal diffusion times (which can be smaller than the ones set by than popular heuristics) serves the purpose of motivating the choice of small diffusion times, which, however, calls for a method to overcome approximation errors. For completeness, in Appendix H, we show that optimizing the ELBO to obtain an optimal diffusion time $T^{★}$ is technically feasible, without resorting to exhaustive grid search.

Consider the ELBO decomposition in Equation (9). We study it as a function of time T, and seek its optimal argument $T^{★}={arg\; max}_T{\mathcal{L}}_{\mathrm{ELBO}}({\widehat{\mathit{s}}}_{\mathsf{\theta }},T)$. Then, the optimal diffusion time $T^{★}\in {\mathbb{R}}^{+}$, and thus not necessarily $T^{★}=\infty$. Additional assumptions on the gap term $\mathcal{G}(·)$ can be used to guarantee strict finiteness of $T^{★}$.

It is trivial to verify that, since the optimal gap term $\mathcal{G}({\widehat{\mathit{s}}}_{\mathsf{\theta }},T)$ is a non decreasing function in T (Section 2.2), we have $\frac{\partial \mathcal{G}}{\partial T}\ge 0$. Then, we study the sign of the KL derivative, which is always negative as shown in Appendix G. Moreover, we know that that $\underset{T\to \infty }{lim}\frac{\partial \mathrm{KL}}{\partial T}=0$. Consequently, the function $\frac{\partial {\mathcal{L}}_{\mathrm{ELBO}}}{\partial T}=\frac{\partial \mathcal{G}}{\partial T}+\frac{\partial \mathrm{KL}}{\partial T}$ has at least one zero in its domain ${\mathbb{R}}^{+}$. To guarantee a stricter bounding of $T^{★}$, we could study asymptotically the growth rates of $\mathcal{G}$ and the KL terms for large T. The investigation is technically involved and outside the scope of this paper. Nevertheless, as discussed hereafter, the numerical investigation carried out in this work suggests finiteness of $T^{★}$.

Empirically, we use Figure 2 to illustrate the tradeoff and the optimality arguments through the lens of the same toy example we use in Section 1. On the first and third columns, we show the ELBO decomposition. We can verify that $\mathcal{G}({\mathit{s}}_{\mathsf{\theta }},T)$ is an increasing function of T, whereas the KL term is a decreasing function of T. Even in the simple case of a toy example, the tension between small and large values of T is clear. On the second and fourth columns, we show the values of the ELBO and of the likelihood as a function of T. We then verify the validity of our claims: the ELBO is neither maximized by an infinite diffusion time, nor by a “sufficiently large” value. Instead, there exists an optimal diffusion time which, for this example, is smaller than $T=1.0$, which is typically used in practice.

In Section 3, we present a new method that admits much smaller diffusion times and we show that the ELBO of our approach is at least as good as the one of a standard diffusion model, configured to use its optimal diffusion time $T^{★}$.

2.4. Relation with Diffusion Process Noise Schedule

We remark that a simple modification of the noise schedule to steer the the diffusion process toward a small diffusion time [5,28] is not a viable solution. In Appendix J, we discuss how the optimal value of the ELBO, in the case of affine SDEs, is invariant to the choice of the noise schedule. Indeed, its value depends uniquely on the relative level of corruption of the initial data at the considered final diffusion time T, that is, the Signal-to-Noise Ratio . Naively, we could think that, by selecting a twice as fast noise schedule, we would be able to obtain the same ELBO of the original schedule by diffusing only for half the time. While true, this does not provide any practical benefit in terms of computational complexity. If the noise schedule is faster, the drift terms involved in the reverse process changes more rapidly. Consequently, to simulate the reverse SDE with a numerical integration scheme, smaller step sizes are required to keep the same accuracy of the original noise schedule simulation. The effect is that, while the diffusion time for the continuous time dynamics is smaller, the number of integration steps is larger, inducing no computational gains. The optimization of the noise schedule can, however, have important practical effects in terms of stability of the training and variance of the estimations, which we do not tackle in this work [5].

2.5. Relation with Literature on Bounds and Goodness of Score Assumptions

Few other works in the literature attempt to study the convergence properties of Diffusion models. In the work of De Bortoli et al. [29] (Theorem 1), a total variation (TV) bound between the generated and data distribution is obtained in the form $C_{1}exp\left(a_{1}T\right)+C_{2}exp(-a_{2}T)$, where the constant $C_1$ depends on the maximum error over $[0,T]$ between the true and approximated score, i.e., ${max}_{t\in [0,T]}\Vert {\mathit{s}}_{\mathsf{\theta }}(\mathit{x},t)-\nabla logp(\mathit{x},t)\Vert$. In the work of De Bortoli [30], the requirement is relaxed by setting ${max}_{t\in [0,T]}\frac{{\sigma }^2\left(t\right)}{1+\Vert \mathit{x}\Vert }\Vert {\mathit{s}}_{\mathsf{\theta }}(\mathit{x},t)-\nabla logp(\mathit{x},t)\Vert$, where the 1-Wasserstein distance between generated and true data is bounded as $C_1+C_{2}exp(-a_{2}T)+C_3$ (Theorem 1). Other works consider the more realistic average square norm instead of the infinity norm, which is consistent with standard training of diffusion models. Moreover, Lee et al. [31] show how the TV bound can be expressed as a function of ${max}_{t\in [0,T]}\mathbb{E}\left[\Vert {{\mathit{s}}_{\mathsf{\theta }}({\mathit{x}}_t,t)-\nabla logp({\mathit{x}}_t,t)\Vert }^2\right]$ (Theorems 2.2, 3.1 and 3.2). Related to our work, Lee et al. [31] find that the TV bound is optimized for a diffusion time that depends, among others, on the maximum score error. Finally, the work by Chen et al. [34] (Theorem 2), which is concurrent to ours, shows that if ${max}_{t\in [0,T]}\mathbb{E}\left[\Vert {{\mathit{s}}_{\mathsf{\theta }}({\mathit{x}}_t,t)-\nabla logp({\mathit{x}}_t,t)\Vert }^2\right]$ is bounded, then the TV distance between true and generated data can be bound as $C_{1}exp(-a_{1}T)+\sqrt{ϵT}$, plus a discretization error.

All prior approaches require assumptions on the maximum score error, which implicitly depends on: (i) the maximum diffusion time T and (ii) the class of parametric score networks considered. Hence, such methods allow for the study of convergence properties, but with the following limitations. It is not clear how the score error behaves as the fitting domain ($[0,T]$) is increased, for generic class of parametric functions and generic $p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}$. Moreover, it is difficult to link the error assumptions with the actual training loss of diffusion models. In this work, instead, we follow a more agnostic path, as we make no assumptions about the error behavior. We notice that the optimal gap term is always a non decreasing function of T. First, we question whether the current best practice for setting diffusion times is adequate: we find that, in realistic implementations, diffusion times are larger than necessary. Second, we introduce a new approach, with provably the same performance of standard diffusion models but lower computational complexity, as highlighted in Section 3.

3. A New, Practical Method for Decreasing Diffusion Times

The ELBO decomposition in Equation (9) and the bounds in Lemma 1 and Section 2.2 highlight a dilemma. We thus propose a simple method that allows us to achieve both a small gap $\mathcal{G}({\mathit{s}}_{\mathsf{\theta }},T)$ and a small discrepancy $\mathrm{KL}\left[p(\mathit{x},T)\parallel p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)\right]$. Before that, let us use Figure 3 to summarize all densities involved and the effects of the various approximations, which will be useful to visualize our proposal.

The data distribution $p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)$ is transformed into the noise distribution $p(\mathit{x},T)$ through the forward diffusion process. Ideally, starting from $p(\mathit{x},T)$, we can recover the data distribution by simulating using the exact score $\nabla logp$. Using the approximated score ${\mathit{s}}_{\mathsf{\theta }}$ and the same initial conditions, the backward process ends up in $q^{\left(1\right)}(\mathit{x},T)$, whose discrepancy ① to $p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)$ is $\mathcal{G}({\mathit{s}}_{\mathsf{\theta }},T)$. However, the distribution $p(\mathit{x},T)$ is unknown and replaced with an easy distribution $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)$, accounting for an error ⓐ measured as $\mathrm{KL}\left[p(\mathit{x},T)\parallel p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)\right]$. With the score and initial distribution approximated, the backward process ends up in $q^{\left(3\right)}(\mathit{x},T)$, where the discrepancy ③ from $p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}$ is the sum of the terms $\mathcal{G}({\mathit{s}}_{\mathsf{\theta }},T)+\mathrm{KL}\left[p(\mathit{x},T)\parallel p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\right]$.

Multiple bridges across densities. In a nutshell, our method allows us to reduce the gap term by selecting smaller diffusion times and by using a learned auxiliary model to transform the initial density $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)$ into a density ${\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)$, which is as close as possible to $p(\mathit{x},T)$, thus avoiding the penalty of a large KL term. To implement this, we first transform the simple distribution $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$ into the distribution ${\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)$, whose discrepancy ⓑ $\mathrm{KL}\left[p(\mathit{x},T)\parallel {\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)\right]$ is smaller than ⓐ. Then, starting from from the auxiliary model ${\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)$, we use the approximate score ${\mathit{s}}_{\mathsf{\theta }}$ to simulate the backward process reaching $q^{\left(2\right)}(\mathit{x},T)$. This solution has a discrepancy ② from the data distribution of $\mathcal{G}({\mathit{s}}_{\mathsf{\theta }},T)+\mathrm{KL}\left[p(\mathit{x},T)\parallel {\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)\right]$, which we will quantify later in the section. Intuitively, we introduce two bridges. The first bridge connects the noise distribution $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$ to an auxiliary distribution ${\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)$ that is as close as possible to that obtained by the forward diffusion process. The second bridge—a standard reverse diffusion process—connects the smooth distribution ${\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)$ to the data distribution. Notably, our approach has important guarantees, which we discuss next.

3.1. Auxiliary Model Fitting and Guarantees

We begin by stating the requirements we consider for the density ${\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)$. First, as it is the case for $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$, it should be easy to generate samples from ${\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)$ in order to initialize the reverse diffusion process. Second, the auxiliary model should allow us to compute the likelihood of the samples generated through the overall generative process, which begins in $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$, passes through ${\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)$, and arrives in $q(\mathit{x},T)$.

The fitting procedure of the auxiliary model is straightforward. First, we recognize that minimizing $\mathrm{KL}\left[p(\mathit{x},T)\parallel {\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)\right]$ with respect to $\mathsf{\varphi }$ also minimizes ${\mathbb{E}}_{p(\mathit{x},T)}\left[log{\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)\right]$, which we can use as loss function. To obtain the set of optimal parameters ${\mathsf{\varphi }}^{★}$, we require samples from $p(\mathit{x},T)$, which can be easily obtained even if the density $p(\mathit{x},T)$ is not available. Indeed, by sampling from $p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}$, and $p(\mathit{x},T|{\mathit{x}}_0)$, we obtain an unbiased Monte Carlo estimate of ${\mathbb{E}}_{p(\mathit{x},T)}\left[log{\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)\right]$, and optimization of the loss can be performed. Note that, due to the affine nature of the drift, the conditional distribution $p(\mathit{x},T|{\mathit{x}}_0)$ is easy to sample from, as shown in Table 1. From a practical point of view, it is important to notice that the fitting of ${\nu }_{\mathsf{\varphi }}$ is independent from the training of the score-matching objective, i.e., the result of $\mathit{I}\left({\mathit{s}}_{\mathsf{\theta }}\right)$ does not depend on the shape of the auxiliary distribution ${\nu }_{\mathsf{\varphi }}$. This implies that the two training procedures can be run in parallel, thus enabling considerable time savings.

Next, we show that the first bridge in our model reduces the KL term, even for small diffusion times.

Proposition 1. 

Let us assume that $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)$ is in the family spanned by ${\nu }_{\mathsf{\varphi }}$, i.e., there exists $\tilde{\mathsf{\varphi }}$ such that ${\nu }_{\tilde{\mathsf{\varphi }}}=p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$. Then we have that

$$\mathrm{KL}\left[p(\mathit{x},T)\parallel {\nu }_{{\mathsf{\varphi }}^{\ast }}\left(\mathit{x}\right)\right]\le \mathrm{KL}\left[p(\mathit{x},T)\parallel {\nu }_{\tilde{\mathsf{\varphi }}}\left(\mathit{x}\right)\right]=\mathrm{KL}\left[p(\mathit{x},T)\parallel p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)\right].$$

(10)

Since we introduce the auxiliary distribution $\nu$, we shall define a new ELBO for our method:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{KL}}^{\mathsf{\varphi }}({\mathit{s}}_{\mathsf{\theta }},T)={\mathbb{E}}_{p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)}log{p}_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right)-\mathcal{G}({\mathit{s}}_{\mathsf{\theta }},T)-\mathrm{KL}\left[p(\mathit{x},T)\parallel {\nu }_{\mathsf{\varphi }}\left(\mathit{x}\right)\right]\end{array}$$

(11)

Recalling that ${\widehat{\mathit{s}}}_{\mathsf{\theta }}$ is the optimal score for a generic time T, Proposition 1 allows us to claim that ${\mathcal{L}}_{\mathrm{ELBO}}^{{\mathsf{\varphi }}^{★}}({\widehat{\mathit{s}}}_{\mathsf{\theta }},T)\ge {\mathcal{L}}_{\mathrm{ELBO}}({\widehat{\mathit{s}}}_{\mathsf{\theta }},T)$.

Then, we can state the following important result:

Proposition 2. 

Given the existence of $T^{★}$, defined as the diffusion time such that the ELBO is maximized (Section 2.3), there exists at least one diffusion time $\tau \le T^{★}$, such that ${\mathcal{L}}_{\mathrm{ELBO}}^{{\mathsf{\varphi }}^{★}}({\widehat{\mathit{s}}}_{\mathsf{\theta }},\tau )\ge {\mathcal{L}}_{1.10elbo}({\widehat{\mathit{s}}}_{\mathsf{\theta }},T^{\ast })$.

Proposition 2, which we prove in Appendix I, has two interpretations. On the one hand, given two score models optimally trained for their respective diffusion times, our approach guarantees an ELBO that is at least as good as that of a standard diffusion model configured with its optimal time $T^{★}$. Our method achieves this with a smaller diffusion time $\tau$, which offers sampling efficiency and generation quality. On the other hand, if we settle for an equivalent ELBO for the standard diffusion model and our approach, with our method we can afford a sub-optimal score model, which requires a smaller computational budget to be trained, while guaranteeing shorter sampling times. We elaborate on this interpretation in Section 4, where our approach obtains substantial savings in terms of training iterations.

A final note is in order. The choice of the auxiliary model depends on the selected diffusion time. The larger the T, the “simpler” the auxiliary model can be. Indeed, the noise distribution $p(\mathit{x},T)$ approaches $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$, so that a simple auxiliary model is sufficient to transform $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$ into a distribution ${\nu }_{\mathsf{\varphi }}$. Instead, for a small T, the distribution $p(\mathit{x},T)$ is closer to the data distribution. Then, the auxiliary model requires high flexibility and capacity. In Section 4, we substantiate this discussion empirically on synthetic and real data.

3.2. Comparison with Schrödinger Bridges

In this section, we briefly compare our method with the Schrödinger bridges approach [29,35,36], which allows one to move from an arbitrary $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$ to $p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}$ in any finite amount of time T. This is achieved by simulating the SDE

$$\mathrm{d}{\mathit{x}}_t=\left[-\mathit{f}({\mathit{x}}_t,t^{\prime })+g^2\left(t^{\prime }\right)\nabla log\widehat{\psi }({\mathit{x}}_t,t^{\prime })\right]\mathrm{d}t+g\left(t^{\prime }\right)\mathrm{d}{\mathit{w}}_t,{\mathit{x}}_0\sim p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}},$$

(12)

where $\widehat{\psi },\psi$ solve the Partial Differential Equation (PDE) system

$$\left\{\begin{array}{c}\frac{\partial \psi (\mathit{x},t)}{\partial t}=-{\nabla }^{\top }\left(\mathit{f}(\mathit{x},t)\right)\psi (\mathit{x},t)-\frac{g^2\left(t\right)}{2}\Delta \left(\psi (\mathit{x},t)\right),\hfill \\ \frac{\partial \widehat{\psi }}{\partial t}=-{\nabla }^{\top }\left(\widehat{\psi }(\mathit{x},t)\mathit{f}(\mathit{x},t)\right)+\frac{g^2\left(t\right)}{2}\Delta \left(\widehat{\psi }(\mathit{x},t)\right),\hfill \end{array}\right.$$

(13)

with boundary conditions $\psi (\mathit{x},0)\widehat{\psi }(\mathit{x},0)=p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}}\left(\mathit{x}\right),\psi (\mathit{x},T)\widehat{\psi }(\mathit{x},T)=p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)$. In the above equation, ${}^{\top }\left(\mathit{f}(\mathit{x},t)\right)={\sum }_{\mathit{I}=1}^N\frac{\partial {\mathit{f}}^{\mathit{I}}(\mathit{x},t)}{\partial {\mathit{x}}^{\mathit{I}}}$, being N the dimension of the vectors $\mathit{x},\mathit{f}$ and the notation $f^{\mathit{I}},x^{\mathit{I}}$ indicating their ${\mathit{I}}_{th}$ component. This approach presents drawbacks compared to classical Diffusion models. First, the functions $\psi ,\widehat{\psi }$ are not known, and their parametric approximation is costly and complex. Second, it is much harder to obtain quantitative bounds between true and generated data as a function of the quality of such approximations.

The $\widehat{\psi },\psi$ estimation procedure simplifies considerably in the particular case where $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)=p(\mathit{x},T)$, for arbitrary T. The solution of Equation (13) is indeed $\psi (\mathit{x},t)=1,\widehat{\psi }(\mathit{x},t)=p(\mathit{x},t)$. The first PDE of the system is satisfied when $\psi$ is a constant. The second PDE is the Fokker–Planck equation, satisfied by $\widehat{\psi }(\mathit{x},t)=p(\mathit{x},t)$. Boundary conditions are also satisfied. In this scenario, a sensible objective is the score-matching, as getting $log\widehat{\psi }$ equal to the true score $\nabla logp$ allows perfect generation.

Unfortunately, it is difficult to generate samples from $p(\mathit{x},T)$, the starting conditions of Equation (12). A trivial solution is to select $T\to \infty$ in order to have $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$ as the simple and analytically known steady state distribution of Equation (1). This corresponds to the classical diffusion models approach, which we discussed in the previous sections. An alternative solution is to keep T finite and cover the first part of the bridge from $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}$ to $p(\mathit{x},T)$ with an auxiliary model. This provides a different interpretation of our method, which allows for smaller diffusion times while keeping good generative quality.

3.3. An Extension for Density Estimation

Diffusion models can be also used for density estimation by transforming the diffusion SDE into an equivalent Ordinary Differential Equation (ODE) whose marginal distribution $p(\mathit{x},t)$ at each time instant coincide to that of the corresponding SDE [3]. The exact equivalent ODE requires the score $\nabla logp({\mathit{x}}_t,t)$, which in practice is replaced by the score model ${\mathit{s}}_{\mathsf{\theta }}$, leading to the following ODE

$$\mathrm{d}{\mathit{x}}_t=\left(\mathit{f}({\mathit{x}}_t,t)-\frac{1}{2}g{\left(t\right)}^2{\mathit{s}}_{\mathsf{\theta }}({\mathit{x}}_t,t)\right)\mathrm{d}t\mathrm{with}{\mathit{x}}_0\sim p_{\mathit{d}\mathit{a}\mathit{t}\mathit{a}},$$

(14)

whose time varying probability density is indicated with $\tilde{p}(\mathit{x},t)$. Note that the density $\tilde{p}(\mathit{x},t)$, is in general not equal to the density $p(\mathit{x},t)$ associated to Equation (1), with the exception of perfect score matching [18]. The reverse time process is modeled as a Continuous Normalizing Flow (cnf) [37,38] initialized with distribution $p_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left(\mathit{x}\right)$; then, the likelihood of a given value ${\mathit{x}}_0$ is

$$\begin{array}{c}\hfill log\tilde{p}\left({\mathit{x}}_0\right)=log{p}_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left({\mathit{x}}_T\right)+\underset{t=0}{\overset{T}{\int }}\nabla ·\left(\mathit{f}({\mathit{x}}_t,t)-\frac{1}{2}g{\left(t\right)}^2{\mathit{s}}_{\mathsf{\theta }}({\mathit{x}}_t,t)\right)\mathrm{d}t.\end{array}$$

(15)

To use our proposed model for density estimation, we also need to take into account the ODE dynamics. We focus again on the term $log{p}_{\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}}\left({\mathit{x}}_T\right)$ to improve the expected log likelihood. For consistency, our auxiliary density ${\nu }_{\mathsf{\varphi }}$ should now maximize ${\mathbb{E}}_{\sim \left(14\right)}log{\nu }_{\mathsf{\varphi }}\left({\mathit{x}}_T\right)$ instead of ${\mathbb{E}}_{\sim \left(1\right)}log{\nu }_{\mathsf{\varphi }}\left({\mathit{x}}_T\right)$. However, the simulation of Equation (14) requires access to ${\mathit{s}}_{\mathsf{\theta }}$ which, in the endeavor of density estimation, is available only once the score model has been trained. Consequently, optimization with respect to $\mathsf{\varphi }$ can only be performed sequentially, whereas, for generative purposes, it could be done concurrently. While the sequential version is expected to perform better, experimental evidence indicates that improvements are marginal, justifying the adoption of the more efficient concurrent version.

4. Experiments

We now present numerical results on the MNIST and CIFAR10 datasets, to support our claims in Section 2 and Section 3. We follow a standard experimental setup [5,7,18,32]: we use a standard U-Net architecture with time embeddings [6] and we report the log-likelihood in terms of bit per dimension (BPD) and the Fréchet Inception Distance (FID) scores (uniquely for CIFAR10). Although the FIDscore is a standard metric for ranking generative models, caution should be used against over-interpreting FIDimprovements [39]. Similarly, while the theoretical properties of the models we consider are obtained through the lens of ELBO maximization, the log-likelihood measured in terms of BPD should be considered with care [40]. Finally, we also report the number of neural function evaluations (NFE) for computing the relevant metrics. We compare our method to the standard score-based model [3]. The full description on the experimental setup is presented in Appendix K.

On the existence of $T^{★}$. We look for further empirical evidence of the existence of a $T^{★}<\infty$, as stated in Section 2.3. For the moment, we shall focus on the baseline model [3], where no auxiliary models are introduced. Results are reported in

Table 2. Optimal T in [3].

Dataset	Time T	BPD (↓)
MNIST	$1.0$	$1.16$
	$0.6$	$\mathbf{1}.\mathbf{16}$
	$0.4$	$1.25$
	$0.2$	$1.75$
CIFAR10	$1.0$	$3.09$
	$0.6$	$\mathbf{3}.\mathbf{07}$
	$0.4$	$3.09$
	$0.2$	$3.38$

. For MNIST, we observe how times $T=0.6$ and $T=1.0$ have comparable performance in terms of BPD, implying that any $T\ge 1.0$ is at best unnecessary and generally detrimental. Similarly, for CIFAR10, it is possible to notice that the best value of BPD is achieved for $T=0.6$, outperforming all other values.

Our auxiliary models. In Section 3, we introduced an auxiliary model to minimize the mismatch between initial distributions of the backward process. We now specify the family of parametric distributions we have considered. Clearly, the choice of an auxiliary model also depends on the data distribution, in addition to the choice of diffusion time T.

For our experiments, we consider two auxiliary models: (i) a Dirichlet process Gaussian mixture model (DPGMM) [41,42] for MNIST and (ii) Glow [43], a flexible normalizing flow for CIFAR10. Both of them satisfy our requirements: they allow exact likelihood computation and they are equipped with a simple sampling procedure. As discussed in Section 3, auxiliary model complexity should be adjusted as a function of T.

This is confirmed experimentally in Figure 4, where we use the number of mixture components of the DPGMM as a proxy to measure the complexity of the auxiliary model.

Reducing T with auxiliary models. We now show how it is possible to obtain a comparable (or better) performance than the baseline model for a wide range of diffusion times T. For MNIST, setting $\tau =0.4$ produces good performance both in terms of BPD (

Table 3. Experiment results on MNIST. For our method, (S) is for the extension in Section 3.3.

	NFE (↓)	BPD (↓)
Model	(ODE)
ScoreSDE	300	$1.16$
ScoreSDE ($T=0.6$)	258	$1.16$
Our ($T=0.6$)	258	$1.16$	$1.14$ (S)
ScoreSDE ($T=0.4$)	235	$1.25$
Our ($T=0.4$)	235	$1.17$	$1.16$ (S)
ScoreSDE ($T=0.2$)	191	$1.75$
Our ($T=0.2$)	191	$1.33$	$1.31$ (S)

) and visual sample quality (Figure 5). We also consider the sequential extension (S) to compute the likelihood, but remark marginal improvements compared to a concurrent implementation. Similarly for the CIFAR10 dataset, in

Table 4. Experimental results on CIFAR10, including other relevant baselines and sampling efficiency enhancements from the literature.

	FID $(\downarrow )$	BPD $(\downarrow )$	NFE $(\downarrow )$	NFE $(\downarrow )$
Model			(SDE)	(ODE)
ScoreSDE [3]	$3.64$	$3.09$	1000	221
ScoreSDE ($T=0.6$)	$5.74$	$3.07$	600	200
ScoreSDE ($T=0.4$)	$24.91$	$3.09$	400	187
ScoreSDE ($T=0.2$)	$339.72$	$3.38$	200	176
Our ($T=0.6$)	$3.72$	$3.07$	600	200
Our ($T=0.4$)	$5.44$	$3.06$	400	187
Our ($T=0.2$)	$14.38$	$3.06$	200	176
ARDM [44]	−	$2.69$	3072
VDM [5]	$4.0$	$2.49$	1000
D3PMs [21]	$7.34$	$3.43$	1000
DDPM [6]	$3.21$	$3.75$	1000
Gotta Go Fast [22]	$2.44$	−	180
LSGM [4]	$2.10$	$2.87$	$120/138$
ARDM-P [44]	−	$2.68/2.74$	$200/50$

we observe how our method achieves better BPD than the baseline diffusion for $T=1$. Moreover, our approach outperforms the baselines for the corresponding diffusion time in terms of FIDscore (Figure 6 and additional non-curated samples in the Appendix K). In Figure A3 we provide a non curated subset of qualitative results, showing that our method for a diffusion time equal to 0.4 still produces appealing images, while the vanilla approach fails. We finally notice how the proposed method has comparable performance with regard to several other competitors, while stressing that many orthogonal to our solutions (like diffusion in latent space [4], or the selection of higher order schemes [22]) can actually be combined with our methodology.

Training and sampling efficiency. In Figure 7, the horizontal line corresponds to the best performance of a fully trained baseline model for $T=1.0$ [3]. To achieve the same performance of the baseline, variants of our method require fewer iterations, which translate in training efficiency. For the sake of fairness, the total training cost of our method should account for the auxiliary model training, which, however, can be done concurrently to the diffusion process. As an illustration for CIFAR10, using four GPUs, the baseline model requires ∼6.4 days of training. With our method we trained the auxiliary and diffusion models for ∼2.3 and 2 days, respectively, leading to a total training time of $max\{2.3,2\}=2.3$ days. Similar training curves can be obtained for the MNIST dataset, where the training time for dpgmms is negligible.

Sampling speed benefits are evident from Table 3 and Table 4. When considering the SDE version of the methods the number of sampling steps can decrease linearly with T, in accordance with theory [45], while retaining good BPD and FIDscores. Similarly, although not in a linear fashion, the number of steps of the ODE samplers can be reduced by using a smaller diffusion time T.

Finally, we test the proposed methodology on the more challenging CELEBA 64x64 dataset. In this case, we use a variance exploding diffusion and we consider again Glow as the auxiliary model. The results, presented in

Table 5. Experimental results on CELEBA 64.

	BPD $(\downarrow )$	NFE $(\downarrow )$
Model	(ODE)
ScoreSDE [3]	$2.13$	68
ScoreSDE ($T=0.5$)	$8.06$	15
ScoreSDE ($T=0.2$)	$12.1$	9
Our ($T=0.5$)	$2.48$	16
Our ($T=0.2$)	$2.58$	9
Our with pretrain diffusion ($T=0.5$)	$2.36$	16
Our with pretrain diffusion ($T=0.2$)	$2.32$	9
Glow [43]	$3.74$	1

, report the log-likelihood performance of different methods (qualitative results are reported in Appendix K). On the two extremes of the complexity we have the original diffusion (VE, $T=1.0$) with the best BPD and the highest complexity, and Glow which provides a much simpler scheme with worse performance. In the table we report the BPD and the NFE metrics for smaller diffusion times, in three different configurations: naively neglecting the mismatch (ScoreSDE) or using the auxiliary model (Our). Interestingly, we found that the best results are obtained by using a combination of diffusion models pretrained for $T=1.0$. The summary of the content of this table is the following: by accepting a small degradation in terms of BPD, we can reduce the computational cost by almost one order of magnitude. We think it would be interesting to study more performing auxiliary models to improve performance of our method on challenging datasets.

5. Conclusions

Diffusion-based generative models emerged as an extremely competitive approach for a wide range of application domains. In practice, however, these models are resource-hungry, both for their training and for sampling new data points. In this work, we have introduced the key idea of considering diffusion times T as a free variable which should be chosen appropriately. We have shown that the choice of T introduces a trade-off, for which a “sweet spot” exists. In standard diffusion-based models, smaller values of T are preferable for efficiency reasons, but sufficiently large T are required to reduce approximation errors of the forward dynamics. Thus, we devised a novel method that allows for an arbitrary selection of diffusion times, where even small values are allowed. Our method closes the gap between practical and ideal diffusion dynamics, using an auxiliary model. Our empirical validation indicated that the performance of our approach was comparable and often superior to standard diffusion models, while being efficient both in training and in sampling.