The Information Dynamics of Generative Diffusion

Abstract

Generative diffusion models have emerged as a powerful class of models in machine learning, yet a unified theoretical understanding of their operation is still developing. This paper provides an integrated perspective on generative diffusion by connecting the information-theoretic, dynamical, and thermodynamic aspects. We demonstrate that the rate of conditional entropy production during generation (i.e., the generative bandwidth) is directly governed by the expected divergence of the score function’s vector field. This divergence, in turn, is linked to the branching of trajectories and generative bifurcations, which we characterize as symmetry-breaking phase transitions in the energy landscape. Beyond ensemble averages, we demonstrate that symmetry-breaking decisions are revealed by peaks in the variance of pathwise conditional entropy, capturing heterogeneity in how individual trajectories resolve uncertainty. Together, these results establish generative diffusion as a process of controlled, noise-induced symmetry breaking, in which the score function acts as a dynamic nonlinear filter that regulates both the rate and variability of information flow from noise to data.

1. Introduction

Generative diffusion models have rapidly become one of the most successful frameworks for high-dimensional generation across images, audio, and video [1,2,3,4,5]. In practice, they synthesize samples by iteratively denoising from simple noise, and they support flexible conditioning and editing via guidance mechanisms that trade off fidelity and diversity [3,6,7]. They were introduced in Sohl-Dickstein et al. [1] in analogy with stochastic thermodynamics and later unified under score-based/SDE formulations that clarify training objectives and reverse-time sampling [2,8,9] (the diffusion formalism, notation, and the SDE viewpoint can be found in Section 3). Despite these efforts, a unified conceptual understanding of their behavior is still emerging. Several perspectives on information theory, stochastic thermodynamics, and the statistical physics of symmetry breaking have each shed light on different aspects of diffusion models, but their interrelations remain fragmented. The purpose of this perspective paper is to integrate these viewpoints into a single coherent theoretical picture.

Our central thesis is that generation in diffusion models proceeds through a sequence of noise-driven symmetry-breaking transitions. These transitions determine when and how the model commits to a specific generative outcome, structure the flow of information, regulate entropy production, and shape the geometry of trajectories in state space. We refer to this synthesis as the information thermodynamics of generative diffusion.

• Information theory and entropy-based perspectives

A growing line of work has examined diffusion models from the standpoint of information theory, focusing especially on how information about the clean sample $x_0$ is progressively revealed as noise is removed. Recent works have proposed information-theoretic decompositions of diffusion dynamics [10,11] and have explored the role of conditional entropy in designing improved training and sampling schedules [12,13]. Furthermore, Franzese et al. [14] show how information-theoretic tools reveal the mechanisms by which latent abstractions guide generation. These approaches treat diffusion as a sequential information transfer process and highlight that the effectiveness of generation depends on how rapidly uncertainty about $x_0$ can be reduced. Central to these results is the observation that the conditional entropy rate is directly linked to geometric quantities such as the divergence of the score and the curvature of the log-density. This suggests that information flow is deeply connected to the underlying dynamical and geometric structure of the generative process.

• Phase transitions, associative memories, and symmetry breaking

Parallel developments in statistical physics have revealed that diffusion models exhibit noise-driven symmetry-breaking events, where the score field undergoes bifurcations and the generative trajectories split into distinct modes [15,16]. High-dimensional analyses have linked these transitions to mean-field phase transitions [17] and to dynamical behavior captured by stochastic localization [18,19,20,21]. These bifurcations correlate with sharp changes in the Hessian of the log-density, revealing a connection between symmetry breaking and information geometry. Similar mechanisms have been studied in hierarchical generative settings [22,23] and in analyses of memorization, mode formation, and semantic emergence [24,25,26,27]. Generative diffusion models have also been directly connected to modern Hopfield networks and other associative memory networks [28,29,30,31], whereas generalization has been associated with the emergence of spurious states [32]. Across these domains, the key unifying insight is that the Hessian (or Jacobian) score mediates both stability of generative trajectories and the structure of the data manifold.

• Thermodynamics and the role of entropy

The connection between diffusion models and stochastic thermodynamics was first made explicit in Sohl-Dickstein et al. [1], motivating a thermodynamic view of generation. Furthermore, this connection was strengthened with a mathematical framework based on stochastic differential equations (SDEs) formulated in Rombach et al. [7], Song et al. [8] and is central to the modern understanding of diffusion models. However, the notion of entropy that is commonly used in stochastic thermodynamics [33,34] measures the irreversibility of the forward process. While such quantities yield elegant speed–accuracy tradeoffs [35], they characterize the evolution of the distribution of trajectories rather than the uncertainty relevant for generating a single sample. Instead, we argue that what matters during generation is the uncertainty about the clean sample.

• Contributions

We synthesize several existing lines of work on information flow, symmetry breaking, and instability in diffusion models into a single coherent narrative, and we clarify how these viewpoints fit together. The main new technical contributions appear in Section 6 and Section 7, where we connect entropy rate to trajectory divergence and analyze a variance-based signature of decision points (including its speciation-time scaling), respectively. Concretely, we provide the following:

• Entropy production as a signature of symmetry breaking: We review the known expression for conditional entropy production in diffusion dynamics and provide an intuitive interpretation (see [10,12] for a practical use of the entropy production expression). We then connect this information-theoretic quantity to symmetry-breaking phenomena reported in [15,16], emphasizing that bifurcations manifest as pronounced, ensemble-level changes in information measures. In particular, symmetry breaking induces a transient loss of identifiability of $x_0$, which appears as a peak in the conditional entropy rate.
• Noise-driven decisions via posterior geometry: We interpret these entropy-rate signatures through the geometry of the score field. Near low-curvature directions, the score weakens and temporarily loses its ability to suppress stochastic fluctuations, so noise becomes the effective selector of the branch that the generative trajectory follows. This provides a unified, mechanism-level explanation for “decision” events during generation.
• Local divergence of trajectories under reverse-time dynamics: We show that the same loss of curvature is reflected in the local linearization of the reverse-time flow. The Jacobian of the score develops expanding directions, implying local exponential amplification of small differences between nearby generative trajectories over finite horizons. This explains how noise fluctuations can propagate and shape the final generative outcome.
• Variance peak as a speciation-time marker: Motivated by tools from stochastic thermodynamics, we introduce the variance of pathwise conditional entropy. Along individual trajectories, the pathwise conditional entropy need not decrease monotonically and may transiently increase, reflecting temporary ambiguity in the resolution of uncertainty about $x_0$. This trajectory heterogeneity produces a pronounced peak in the variance, and we show how this peak concentrates around the speciation time [24].

Together, these results organize existing insights into a unified picture in which entropy production and local stability provide complementary perspectives on the same noise-driven branching mechanism. At the same time, our variance-based analysis offers a new lens on trajectory-level generative dynamics, highlighting a form of pathwise heterogeneity and decision-making that is fundamentally absent in autoregressive generation.

2. Information Theory

We start by presenting an introduction to the information theory of sequential generative modeling, which will open the door to the analysis of generative diffusion.

Consider a game of Twenty Questions where an interrogator player may ask twenty binary questions concerning a set to an “oracle” player in order to gradually reveal the identity of a predetermined element ${\mathbf{y}}^{*}$ in a finite set $\Omega =\{{\mathbf{y}}_1,\dots ,{\mathbf{y}}_{N_0}\}$ with $N_0$ elements. We denote the size of the possible set ${\Omega }_{j-1}\left(a_{1:j-1}\right)$ after $j-1$ questions as $N_{j-1}\left(a_{1:j-1}\right)$. The answer $a_j$ to the j-th question $q_j$ then divides the set into two possible subsets with sizes $N_j^1\left(a_{1:j-1}\right)=N_j(a_j=1,a_{1:j-1})$ and $N_j^0\left(a_{1:j-1}\right)=N_{j-1}\left(a_{1:j-1}\right)-N_j(a_j=1,a_{1:j-1})$. Assuming a fixed set of questions, the expected uncertainty experienced by the player after the j-th question can be quantified by the conditional entropy:

$$\mathbf{H}[{\mathbf{y}}^{*}\mid a_{1:j}]=-{\mathbb{E}}_{{\mathbf{y}}^{*},a_{1:j}}\left[{log}_{2}p(\mathbf{y}\mid a_{1:j})\right]={\mathbb{E}}_{a_{1:j}}\left[{log}_2{N}_j\left(a_{1:j}\right)\right]$$

(1)

where ${\mathbf{y}}^{*}$ is sampled uniformly from $\Omega$. Under these conditions, the expected entropy reduction associated to a given question is given by

$$\Delta {\mathbf{H}}_j={\mathbb{E}}_{a_{1:j}}\left[{log}_2{N}_{j-1}-{\displaystyle \frac{N_j^0}{N_{j-1}}}{log}_2{N}_j^0+{\displaystyle \frac{N_j^1}{N_{j-1}}}{log}_2{N}_j^1\right],$$

(2)

where we left the dependence on the set of answers implicit to unclutter the notation. It is easy to see that the maximum bit rate is 1, which is achieved when $N_j^0=N_{j-1}/2$. Assuming that 20 questions are enough to fully identify the value of ${\mathbf{y}}^{*}$, we can encode each $\mathbf{y}$ in the string of binary values $a_{1:20}$, which makes clear that the question-answering process consists of gradually filling in this string. Using the language of generative diffusion, we can re-frame this process in terms of a ‘forward’ process, where the string $a_{1:20}$ corresponding to an element of $\Omega$ is sampled in advance and then transmitted to the j-th ‘time point’ through the following non-injective forward process

$$R_j\left(a_{1:20}\right)=a_{1:j},$$

(3)

which deterministically suppresses information by masking the values of the string. The solution to a Twenty Question game can then be seen as inverting this ‘forward process’. Note that the forward process leads to a sequence of monotonically non-decreasing marginal conditional entropies $\mathbf{H}({\mathbf{y}}^{*}\mid a_{1:j})<\mathbf{H}({\mathbf{y}}^{*}\mid a_{1:j-1})$, which is a fundamental feature of a forward process in diffusion models that captures the fact that information is lost by the forward transformation.

Now consider the case where a lazy oracle forgot to select a word in advance and decides instead to answer the questions at random under the probability determined by the sizes $N_j^0$ and $N_j^1$, which we assume to be fixed given the questions. Strikingly, this reformulation does not make any observable difference from the point of view of the interrogator as each (randomly sampled) answer equally reduces the space of possible words and it results in the same entropy reduction, until a final guess can be offered. Therefore, the game of Twenty Questions with a random oracle can be interpreted as a sequential generative process where the state at ‘time’ j is given by a binary string $a_{1:j}$ with Markov transition probabilities

$$p(a_{j+1}=0\mid a_{1:j})={\displaystyle \frac{N_{j+1}^0\left(a_{1:j}\right)}{N_j\left(a_{1:j}\right)}}$$

(4)

The conditional entropy rate $\Delta H_j$ determines how much information is transferred from ‘time’ j to the final generation.

As we shall see, the reverse diffusion process can be seen as analogous to this ‘generative game’ with the score function playing the part of the interrogator and the noise ${ϵ}_t$ playing the role of the oracle. Like in the interrogator in the generative Twenty Questions game, the score function can reduce the information transfer by tilting the probabilities of the stochastic increments out of uniformity, which reduces the impact of the noise. This phenomenon is related to the divergence of the vector field induced by the score function, which amplifies small perturbations during generative dynamics. We will also see that the phenomenon is connected to the branching of paths of fixed points of the score and consequently to the phenomenon of generative phase transitions and spontaneous symmetry breaking [15].

3. Diffusion Models

The sequential generation example outlined above is analogous to the masked diffusion models [36,37]. On the other hand, continuous diffusion models cast generation as the time reversal of a noising process that transports a complex data distribution $p_0$ to a simple reference distribution (typically an isotropic Gaussian) through a sequence of progressively corrupted marginals ${\left\{p_t\right\}}_{t\in [0,T]}$ [1,2,8]. In continuous time, the forward process is specified by an SDE that gradually destroys information about $X_0$ by injecting noise, while the reverse-time process progressively restores structure. Therefore, intermediate states $X_t$ can be interpreted as partially noised versions of the data.

Formally, an SDE specifies the evolution of a random variable as

$$d{X}_t=f(X_t,t)dt+g\left(t\right)d{W}_t,$$

(5)

where $W_t$ is a standard Wiener process, f is a drift field, and g controls the noise scale. The associated marginal densities $p_t\left(x\right)$ evolve according to the Fokker–Planck equation

$${\partial }_t{p}_t\left(x\right)=\sum _{j=1}^d{\partial }_{x_j}\left(-f_j(x,t)+{\displaystyle \frac{g^2\left(t\right)}{2}}{\partial }_{x_j}\right)p_t\left(x\right).$$

(6)

Different diffusion formulations correspond to different choices of $(f,g)$, or equivalently, different parameterizations of Gaussian perturbation kernels $p(x_t\mid x_0)$. For the standard choices used in diffusion models, the forward noising admits a closed-form Gaussian kernel, which we write as

$$p(x_t\mid x_0)=\mathcal{N}\left(x_t;{\alpha }_t{x}_0,{\sigma }_t^{2}I\right),$$

(7)

with ${\alpha }_t$ controlling the attenuation of the signal and ${\sigma }_t$ the noise level.

In the variance-preserving (VP) setting, the forward process is the Ornstein–Uhlenbeck process,

$$d{X}_t=-\frac{1}{2}\beta \left(t\right)X_{t}dt+\sqrt{\beta \left(t\right)}d{W}_t,$$

(8)

which implies the closed-form corruption $X_t={\alpha }_t{X}_0+{\sigma }_{t}Z$ with $Z\sim \mathcal{N}(0,I)$ and ${\alpha }_t^2+{\sigma }_t^2=1$ [2,8].

In the variance-exploding (VE) setting, the drift is set to zero and the noise scale increases,

$$d{X}_t=g\left(t\right)d{W}_t⟺X_t=X_0+{\sigma }_{t}Z,$$

(9)

so $\mathrm{Var}\left(X_t\right)$ grows with t [8].

Finally, the EDM parameterization [38] can be viewed as a VE-type SDE written directly in terms of the noise level $\sigma$ (rather than physical time), with the canonical choice ${\sigma }_t=t$ and $X_t=X_0+tZ$. In this paper, we will mostly adopt this EDM convention for notational convenience. The main identities translate broadly across VP/VE/EDM formulations. When the specific choice of forward process becomes essential, notably in our discussion of sharp speciation-time behavior and the variance peak, we will make this explicit (there, we focus on the VP/OU setting).

Generation is obtained by reversing the forward process. Reversing an SDE introduces an additional drift term involving the score ${\nabla }_{x}log{p}_t\left(x\right)$ [39],

$$d{X}_t=\left(f(X_t,t)-g^2\left(t\right){\nabla }_{x}log{p}_t\left(X_t\right)\right)dt+g\left(t\right)d{\tilde{W}}_t.$$

(10)

The fundamental mathematical object that determines the reverse dynamics is the score function, which in this case can be expressed as $\nabla log{p}_t\left({\mathit{x}}_t\right)={\mathbb{E}}_{\mathit{y}\mid {\mathit{x}}_t}\left[{\displaystyle \frac{\mathit{y}-{\mathit{x}}_t}{{\sigma }^2\left(t\right)}}\right],$ where the expectation is taken with respect to the conditional distribution $p(\mathbf{y}\mid {\mathit{x}}_t)\propto p({\mathit{x}}_t\mid \mathbf{y})\rho \left(\mathbf{y}\right)$. This expression can be further simplified by noticing that ${\mathit{x}}_t=\mathbf{y}+\sigma \left(t\right){\mathbf{z}}_t$:

$$\nabla log{p}_t\left({\mathit{x}}_t\right)=-{\mathbb{E}}_{{\mathbf{z}}_t\mid {\mathit{x}}_t}\left[{\displaystyle \frac{{\mathbf{z}}_t}{\sigma \left(t\right)}}\right]$$

(11)

where $\mathbf{z}$ is a standard normal vector. In other words, the score is the negative of the average (rescaled) noise, and it therefore provides the optimal (infinitesimal) denoising direction.

If the score were known exactly, sampling (10) from the (approximately Gaussian) terminal distribution of the forward process would produce exact samples from $p_0$. However, in practice, the score is not available and is approximated by a neural network $s({\mathbf{x}}_t;\theta )$ trained by (denoising) score matching [8,40,41]. In other words, it is trained to minimize

$${\mathcal{L}}_{\mathrm{sm}}(\theta ,t)={\mathbb{E}}_{{\mathit{x}}_t}\left[{∥\sigma \left(t\right)\nabla log{p}_t\left({\mathit{x}}_t\right)-s({\mathit{x}}_t;\theta )∥}^2\right]$$

(12)

This loss function cannot be computed directly because the true score is not available. However, Equation (12) can be re-written using Equation (11) and expanding the square:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{sm}}(\theta ,t)& ={\mathbb{E}}_{{\mathit{x}}_t}\left[{∥{\mathbb{E}}_{{\mathbf{z}}_t\mid {\mathit{x}}_t}\left[{\mathbf{z}}_t\right]+s({\mathit{x}}_t;\theta )∥}^2\right]\hfill \\ \hfill & ={\mathbb{E}}_{{\mathbf{z}}_t,\mathbf{y}}\left[{∥{\mathbf{z}}_t+s(\mathit{y}+\sigma \left(t\right){\mathbf{z}}_t;\theta )∥}^2\right]-{\mathbb{E}}_{{\mathbf{z}}_t,\mathbf{y}}\left[{∥{\mathbf{z}}_t+\sigma \left(t\right)\nabla log{p}_t(\mathit{y}+\sigma \left(t\right){\mathbf{z}}_t)∥}^2\right].\hfill \end{array}$$

(13)

Note that the second term is constant in $\theta$, which means that the gradient solely depends on the denoising loss:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{d}}(\theta ,t)& ={\mathbb{E}}_{{\mathbf{z}}_t,\mathbf{y}}\left[{∥{\mathbf{z}}_t+s(\mathit{y}+\sigma \left(t\right){\mathbf{z}}_t;\theta )∥}^2\right].\hfill \end{array}$$

(14)

The constant term

$$C_t={\mathbb{E}}_{{\mathbf{z}}_t,\mathbf{y}}\left[{∥{\mathbf{z}}_t+\sigma \left(t\right)\nabla log{p}_t(\mathit{y}+\sigma \left(t\right){\mathbf{z}}_t)∥}^2\right]$$

(15)

is of high importance for our current purposes. It quantifies the loss of the denoiser obtained from the score function. This is therefore the unavoidable part of the denoising error that is still present given a perfectly trained network. With a few manipulations, it is possible to show that this term is in fact equal to the variance of the posterior denoising distribution:

$$C_t={\mathbb{E}}_{\mathit{y},{\mathit{x}}_t}\left[\mathrm{var}(\mathit{y}\mid {\mathit{x}}_t)\right],$$

(16)

which allows us to interpret this term as a measure of uncertainty at time t on the final outcome of the generative trajectory.

In contrast, throughout this paper, we analyze the oracle (perfect-score) dynamics, assuming access to the exact score ${\nabla }_{x}log{p}_t\left(x\right)$ in order to isolate intrinsic information-theoretic and dynamical mechanisms of the generative process. In practical models, approximation error means that equalities derived under the oracle assumption may only hold approximately. Moreover, the learned vector field used in sampling need not coincide with the gradient of any true log-density (i.e., it may be non-integrable), which can introduce inconsistencies relative to the idealized score-driven dynamics.

4. Generative Information Transfer in Score Matching Diffusion

To characterize the generative information transfer we need to compute the conditional entropy rate $\dot{\mathbf{H}}[\mathit{y}\mid {\mathit{x}}_t]$, which is analogous to the discrete entropy reduction we gave in Equation (2). The conditional entropy is defined as

$$\mathbf{H}[\mathit{y}\mid {\mathit{x}}_t]=-{\mathbb{E}}_{\mathit{y},{\mathit{x}}_t}\left[logp(\mathit{y}\mid {\mathit{x}}_t)\right]$$

(17)

To find the entropy rate, we can take the temporal derivative of Equation (17) and use the Fokker–Planck equation, which in our case is just the heat equation:

$${\partial }_t{p}_t\left({\mathit{x}}_t\right)={\displaystyle \frac{1}{2}}{\nu }^2\left(t\right){\nabla }^2{p}_t\left({\mathit{x}}_t\right).$$

(18)

Using integration by parts, this results in

$$\begin{array}{cc}\hfill \dot{\mathbf{H}}\left[\mathit{y}\right|{\mathit{x}}_t]& ={\displaystyle \frac{{\nu }^2\left(t\right)}{2}}\left({\mathbb{E}}_{p({\mathit{x}}_t,{\mathit{x}}_0)}\left[{∥\nabla logp\left({\mathit{x}}_t\right|{\mathit{x}}_0)∥}^2\right]-{\mathbb{E}}_{p_t\left({\mathit{x}}_t\right)}\left[{∥\nabla logp\left({\mathit{x}}_t\right)∥}^2\right]\right)\hfill \\ \hfill & ={\displaystyle \frac{{\nu }^2\left(t\right)}{2}}\left({\displaystyle \frac{D}{{\sigma }^2\left(t\right)}}-{\mathbb{E}}_{p_t\left({\mathit{x}}_t\right)}\left[{∥\nabla logp\left({\mathit{x}}_t\right)∥}^2\right]\right),\hfill \end{array}$$

(19)

where D is the dimensionality of the ambient space. From this formula, we can see that the maximal bandwidth is reached when the Euclidean norm of the score function is minimized.

4.1. Score Norm and Posterior Concentration

To gain some insight into the significance of the square norm and the expression for the conditional entropy, we will consider the following case. We assume a discrete data distribution $p_0\left(y\right)={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N\delta (y-y_i)$ with an empirical mean equal to zero.

At time t, the marginal density is given by a Gaussian smoothing of the data,

$$p_t\left(x\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\phi }_{\sigma \left(t\right)}(x-y_i),$$

(20)

where ${\phi }_{\sigma \left(t\right)}$ denotes an isotropic Gaussian with variance ${\sigma }^2\left(t\right)$. The posterior distribution over datapoints is

$$p(y_i\mid x_t)={\displaystyle \frac{{\phi }_{\sigma \left(t\right)}(x_t-y_i)}{{\sum }_{k=1}^N{\phi }_{\sigma \left(t\right)}(x_t-y_k)}}.$$

(21)

The score function can then be written as the posterior average

$$\nabla log{p}_t\left(x_t\right)={\mathbb{E}}_{y\mid x_t}\left[{\displaystyle \frac{y-x_t}{{\sigma }^2\left(t\right)}}\right]={\displaystyle \frac{1}{{\sigma }^2\left(t\right)}}\left(\mu \left(x_t\right)-x_t\right),\mu \left(x_t\right):=\mathbb{E}[y\mid x_t].$$

(22)

In addition, we assume that the data vectors satisfy

$$y_i^{\top }{y}_j\approx 0(i\ne j),{\parallel y_i\parallel }^2\approx R^2,$$

(23)

i.e., datapoints are approximately orthogonal and lie at a common distance R from the mean. Even though this holds only in special cases (e.g., randomly sampled data points from a Gaussian in high dimension) and is not representative of the commonly encountered datasets, we proceed with the setup as an intuitive approach to understanding the meaning of the norm of the score and its role in the entropy expression (Equation (19)).

Under these assumptions, the squared norm of the posterior mean simplifies to

$$\parallel \mu \left(x_t\right){\parallel }^2=\parallel \sum _{i=1}^{N}p(y_i\mid x_t)y_i{\parallel }^2\approx R^2\sum _{i=1}^{N}p{(y_i\mid x_t)}^2.$$

(24)

Taking expectations with respect to $p_t\left(x_t\right)$, we obtain

$${\mathbb{E}}_{x_t}\left[\parallel \nabla log{p}_t\left(x_t\right){\parallel }^2\right]={\displaystyle \frac{1}{{\sigma }^4\left(t\right)}}\left({\mathbb{E}}_{x_t}\left[\parallel \mu \left(x_t\right){\parallel }^2\right]-2{\mathbb{E}}_{x_t}\left[x_t^{\top }\mu \left(x_t\right)\right]+{\mathbb{E}}_{x_t}[\parallel x_t{\parallel }^2]\right).$$

(25)

The first term captures the data-dependent structure of the score and, using Equation (24), can be written as

$${\mathbb{E}}_{x_t}\left[\parallel \mu \left(x_t\right){\parallel }^2\right]\approx R^2{\mathbb{E}}_{x_t}\left[\sum _{i=1}^{N}p{(y_i\mid x_t)}^2\right].$$

(26)

The quantity ${\sum }_{i}p{(y_i\mid x_t)}^2$ measures the concentration of the posterior over datapoints. It satisfies $1/N\le {\sum }_{i}p{(y_i\mid x_t)}^2\le 1$, interpolating between a fully diffuse posterior and complete concentration on a single datapoint.

The remaining two terms in Equation (25) can be estimated explicitly under the forward model $x_t=y+\sigma \left(t\right)z$, where $z\sim \mathcal{N}(0,I)$ is independent of y. We have

$${\mathbb{E}}_{x_t}\left[x_t^{\top }\mu \left(x_t\right)\right]={\mathbb{E}}_{x_t}\left[x_t^{\top }\mathbb{E}[y\mid x_t]\right]={\mathbb{E}}_{x_t,y}\left[x_t^{\top }y\right]=\mathbb{E}{\parallel y\parallel }^2\approx R^2,$$

(27)

$${\mathbb{E}}_{x_t}\left[\parallel x_t{\parallel }^2\right]={\mathbb{E}\parallel y\parallel }^2+{\sigma }^2\left(t\right)\mathbb{E}{\parallel z\parallel }^2\approx R^2+D{\sigma }^2\left(t\right),$$

(28)

where D denotes the ambient dimensionality.

Substituting Equations (26)–(28) into Equation (25) yields

$${\mathbb{E}}_{x_t}\left[\parallel \nabla log{p}_t\left(x_t\right){\parallel }^2\right]\approx {\displaystyle \frac{R^2}{{\sigma }^4\left(t\right)}}\left({\mathbb{E}}_{x_t}\left[\sum _{i=1}^{N}p{(y_i\mid x_t)}^2\right]-1\right)+{\displaystyle \frac{D}{{\sigma }^2\left(t\right)}}.$$

(29)

The second term coincides with the expected squared norm of the score of the forward Gaussian kernel and therefore represents a data-independent baseline contribution. The first term encodes the deviation from pure diffusion induced by the structure of the dataset and depends solely on the posterior distribution over datapoints.

Using the bound $1/N\le {\sum }_{i=1}^{N}p{(y_i\mid x_t)}^2\le 1$, we obtain the inequality

$$-{\displaystyle \frac{(N-1)R^2}{N{\sigma }^4\left(t\right)}}\le {\displaystyle \frac{R^2}{{\sigma }^4\left(t\right)}}\left({\mathbb{E}}_{x_t}\left[\sum _{i=1}^{N}p{(y_i\mid x_t)}^2\right]-1\right)\le 0.$$

(30)

As a consequence, the expected squared norm of the score is always bounded above by the forward kernel contribution, ensuring that the marginal entropy remains a monotonically increasing function of time.

Further insight can be gained by rewriting

$${\mathbb{E}}_{x_t}\left[\sum _{i=1}^{N}p{(y_i\mid x_t)}^2\right]={\displaystyle \frac{1}{N}}\sum _{i=1}^N\int p(y_i\mid x_t)p(x_t\mid y_i)d{x}_t.$$

(31)

This expression makes explicit that the deviation from the diffusion baseline is controlled by the overlap of the forward kernels. If, at time t, the datapoints have effectively merged into m indistinguishable groups (with identical posteriors), the term evaluates to $m/N$, yielding

$${\mathbb{E}}_{x_t}\left[\sum _{i=1}^{N}p{(y_i\mid x_t)}^2\right]-1={\displaystyle \frac{m-N}{N}}.$$

(32)

Therefore, increasing mixing among datapoints (smaller m) makes the data-dependent term more negative, reducing the expected score norm and increasing the conditional entropy rate.

This result allows us to interpret the magnitude of the score vector as a quantitative estimate of uncertainty in the denoising process: when multiple datapoints are compatible with the noisy state $x_t$, posterior averaging suppresses the score, leading to enhanced entropy production (Equation (19)). As we shall see in the rest of the paper, we can associate peaks in the entropy rates with symmetry-breaking bifurcations that correspond to noise-induced ’choices’ between possible data points.

4.2. Conditional Entropy Production as Optimal Error

The conditional entropy rate quantifies the instantaneous generative information transfer at any given moment in time. It can be shown (see [12]) that this quantity is closely connected to the optimal denoising squared error, which is the variance of the denoising distribution:

$$\dot{\mathbf{H}}\left[\mathit{y}\right|{\mathit{x}}_t]={\displaystyle \frac{1}{2}}{\displaystyle \frac{{\nu }^2\left(t\right)}{{\sigma }^4\left(t\right)}}{\mathbb{E}}_{\mathit{y},{\mathit{x}}_t}\left[\mathrm{var}(\mathit{y}\mid {\mathit{x}}_t)\right].$$

(33)

Intuitively, this means that the information rate is directly related to the denoising uncertainty at a given time.

Using this relation, we can now re-express the denoising score matching formula in Equation (13) in terms of the conditional entropy rate:

$${\mathbb{E}}_{{\mathit{x}}_t}\left[{∥{\mathbb{E}}_{{\mathbf{z}}_t\mid {\mathit{x}}_t}\left[\mathbf{z}\right]-s({\mathit{x}}_t;\theta )∥}^2\right]+{\displaystyle \frac{2{\sigma }^4\left(t\right)}{{\nu }^2\left(t\right)}}\dot{\mathbf{H}}\left[\mathit{y}\right|{\mathit{x}}_t]={\mathbb{E}}_{{\mathbf{z}}_t,\mathbf{y}}\left[{∥{\mathbf{z}}_t-s(\mathit{y}+\sigma \left(t\right){\mathbf{z}}_t;\theta )∥}^2\right],$$

(34)

which implies that the entropy rate can be estimated from the training loss if we assume that the network is well-trained.

4.3. Generative Bandwidth

It is insightful to investigate under what circumstances the score-matching diffusion model can achieve the maximum possible generative bandwidth. From Equation (19), it is clear that this happens when $\mathbb{E}\left[∥\nabla log{p}_t\left({\mathit{x}}_t\right)∥\right]=0$, which in turn is obtained if the score vanishes almost everywhere.

To realize this situation, we can consider a data distribution $p_h\left(\mathit{y}\right)$ to be a centered multivariate normal with variance $h^2$. In this case, the score function is just

$$\nabla log{p}_t\left({\mathit{x}}_t\right)=-{\displaystyle \frac{{\mathit{x}}_t}{{\sigma }^2\left(t\right)+h^2}},$$

(35)

which vanishes everywhere for $h\to \infty$, giving a maximum entropy rate:

$$\dot{\mathbf{H}}[{\mathit{x}}_0\mid {\mathit{x}}_t]={\displaystyle \frac{1}{2}}{\displaystyle \frac{D{\nu }^2\left(t\right)}{{\sigma }^2\left(t\right)}}.$$

(36)

This corresponds to a setting where the particles are free to diffuse since every possible generation is equally likely. From this, we can conclude that the score function has the negative role of suppressing fluctuations along ’unwanted directions’ to preserve the statistics of the data and that peaks in the information transfer comes from periods where noise fluctuations are not suppressed. Note that the maximum bandwidth scales with the dimensionality D.

Now consider the case where the distribution of the data is a centered Gaussian in a $D_{\mathrm{data}}$-dimensional subspace with $D_{\mathrm{data}}\le D$. In this case, the expected norm of the score decomposes as follows

$$\mathbb{E}\left[{∥\nabla log{p}_t\left({\mathit{x}}_t\right)∥}^2\right]={\displaystyle \frac{D_{\mathrm{data}}}{{\sigma }^2\left(t\right)+h^2}}+{\displaystyle \frac{D-D_{\mathrm{data}}}{{\sigma }^2\left(t\right)}}\to {\displaystyle \frac{D-D_{\mathrm{data}}}{{\sigma }^2\left(t\right)}}$$

(37)

which leads to the entropy rate

$$\dot{\mathbf{H}}[{\mathit{x}}_0\mid {\mathbf{x}}_t]={\displaystyle \frac{1}{2}}{\displaystyle \frac{D_{\mathrm{data}}{\nu }^2\left(t\right)}{{\sigma }^2\left(t\right)}}.$$

(38)

In this case, the score function suppresses entropy reduction in the subspace orthogonal to the data and therefore acts as a linear analog filter. Note that the entropy rate is zero when $D_{\mathrm{data}}$ is equal to zero since all the distribution is in this case collapsed into a single point and no ‘decision’ needs to be made.

5. Statistical Physics, Order Parameters and Phase Transitions

In this section, we will connect the information-theoretical concepts we outlined above with concepts from statistical physics such as order parameters, phase transitions and spontaneous symmetry breaking. We will start by studying the paths of fixed-points of the score function and use them to track ‘generative decisions’ (i.e., bifurcations) along the denoising trajectories. As we will see, the stability of these fixed-points paths is regulated by the Jacobian of the score and it is deeply connected with the conditional entropy production.

Branching Paths of Fixed-Points and Spontaneous Symmetry Breaking

The fixed-points of the score function are defined by the equation

$$\nabla log{p}_t\left({\mathit{x}}_t^{*}\right)=\mathbf{0}.$$

(39)

We denote the set of fixed-points at time t as ${\Psi }_t$. The solutions of this fixed-point equation can be organized in a set $\Omega$ of piecewise continuous paths $\gamma :{\mathbb{R}}^{+}\to {\mathbb{R}}^d\in \Omega$. To remove ambiguities, we assume that, if $\gamma \left(\tau \right)$ is discontinuous at ${\tau }_0$, then the one-sided limit exists and $\gamma \left({\tau }_0\right)$ is equal to ${lim}_{t\to {\tau }_0^{+}}\gamma \left(t\right)={arg\; min}_{\mathbf{x}\in {\Psi }_{{\tau }_0}}{lim}_{t\to {\tau }_0^{+}}\left[∥\mathbf{x}-\gamma \left(t\right)∥\right]$. We know that ${lim}_{t\to \infty }\gamma \left(t\right)=\mathbf{0}$ for all paths since the zero vector is the only fixed-point of the score of the asymptotic Gaussian distribution. Any two paths ${\gamma }_1\left(t\right)$ and ${\gamma }_2\left(t\right)$ can be proven to overlap for a finite range of time, meaning that ${\gamma }_1\left(t\right)={\gamma }_2\left(t\right)$ if $t\ge {\tau }_{1,2}\in {\mathbb{R}}^{+}$ (this follows from the results in [42,43,44] on the number of modes of mixture of normal distributions). We refer to ${\tau }_{1,2}$ as the branching time of the two paths. The branching time of two paths of fixed points can roughly be interpreted as a decision time in the generative process, where the sample will be ‘pushed’ by the noise in either one or the other path during the reverse dynamics. It is therefore insightful to study the behavior of the paths at the branching times. In general, this can happen if there is a discontinuous jump in a path $\gamma \left(t\right)$. Perhaps more interestingly, two paths can also branch continuously at a finite time. This can be studied by analyzing the Jacobian matrix of the score function:

$$J_t\left({\mathit{x}}_t^{*}\right)={\nabla }^T\nabla log{p}_t\left({\mathit{x}}_t^{*}\right).$$

(40)

We call a path point $\gamma \left(t\right)$ stable at time t if $J_t\left(\gamma \left(t\right)\right)$ is negative-definite. We say that the path is stable if this is true for all $t\in {\mathbb{R}}^{+}$ except for a countable set of time points $t_j$ where the Jacobian is negative semi-definite. Now consider two stable paths ${\gamma }_1\left(t\right)$ and ${\gamma }_2\left(t\right)$ that branch continuously at time ${\tau }_{1,2}$. Given the asymptotic separation vector

$${\mathit{v}}_{1,2}=\underset{t\to {\tau }_{1,2}^{-}}{lim}{\displaystyle \frac{({\gamma }_2\left(t\right)-{\gamma }_1\left(t\right))}{∥{\gamma }_2\left(t\right)-{\gamma }_1\left(t\right)∥}},$$

it can be shown that ${\mathit{v}}^T{J}_t\left(\gamma \left(t\right)\right)\mathit{v}<0$ in a finite interval $({\tau }_{1,2},{\tau }_{1,2}+ϵ)$ and that

$$\underset{t\to {\tau }_{1,2}^{+}}{lim}{\mathit{v}}^T{J}_t\left({\gamma }_1\left(t\right)\right)\mathit{v}=0,$$

which implies that the second directional derivative of $D_{\mathit{v}}^{2}log{p}_t\left({\mathit{x}}_t\right)$ along $\mathit{v}$ vanishes at the branching point.

Consider now a generative diffusion with an initial distribution given as

$$p_0\left(\mathit{y}\right)={\displaystyle \frac{1}{K}}\sum _{j=1}^K\delta ({\mathit{y}}^{\left(j\right)}-\mathit{y}),$$

(41)

with K distinct data-points ${\mathit{y}}^{\left(j\right)}\in {\mathbb{R}}^d$. In this case, there are exactly K distinct stable fixed-point paths ${\gamma }_j\left(t\right)$, with ${\gamma }_j\left(0\right)={\mathit{y}}^{\left(j\right)}$. Again, any two paths branch at a finite time ${\tau }_{j,k}$. For a given t, we can partition the set of data-points in equivalence classes, where two data-points ${\mathit{y}}^{\left(j\right)}$ and ${\mathit{y}}^{\left(k\right)}$ share the same class if their associated path coincide at t. Importantly, each equivalence class corresponds to an individual fixed-point, which allows us to associate each fixed-point ${\mathit{x}}^{*}\in {\Psi }_t$ with a sub-set of data-points that are, using colorful language, fused together. More precisely, we can express the fixed-points as weighted averages of data-points obtained by solving the self-consistency equation:

$${\mathit{x}}^{*}=\sum _{j=1}^K{w}_j\left({\mathit{x}}^{*}\right){\mathit{y}}^{\left(j\right)}$$

(42)

where

$$w_j\left(\mathit{x}\right)={\displaystyle \frac{e^{\left(-{∥{\mathit{y}}^{\left(j\right)}∥}^2/2+{\mathit{x}}^T{\mathit{y}}^{\left(j\right)}\right)/{\sigma }^2\left(t\right)}}{{\sum }_{k=1}^K{e}^{\left(-{∥{\mathit{y}}^{\left(k\right)}∥}^2/2+{\mathit{x}}^T{\mathit{y}}^{\left(k\right)}\right)/{\sigma }^2\left(t\right)}}}.$$

(43)

Note that this average has non-zero weight on all data points, which is why we cannot find the location of the fixed-point solely based on its equivalence class. However, usually the weights corresponding to data-points in the equivalence class will be substantially larger than the other weights and will therefore dominate the average. In summary, we can interpret the set of fixed-points as a decision tree where each branching point roughly coincides with a split between two sets of data points.

An example of spontaneous symmetry breaking happens when the generative path needs to ’decide’ between two isolated data-points. Consider again the mixture of the delta case and two neighboring data-points ${\mathit{y}}_1=\mathit{v}$ and ${\mathit{y}}_2=-\mathit{v}$. If the distance between the center of mass of these two points and the nearest external data-point is much larger than $\sigma \left(t\right)$, there will be a fixed point approximately located along the line segment connecting the two points. In these conditions, we can consider the fixed-point equation restricted to the projections on $\mathit{v}$:

$$x_v^{*}=tanh\left({\displaystyle \frac{x_v^{*}+\varphi (x_v^{*},t)}{{\sigma }^2\left(t\right)}}\right)$$

(44)

where $\varphi ({\mathit{x}}_v^{*},t)$ encapsulates the interference due to all other data-points, which we, in this example, assume to be small relative to the norm of the separation vector:

$$\varphi (x_v^{*},t)={\displaystyle \frac{{\sigma }^2\left(t\right)}{2}}\left(log\left(e^{x_v^{*}}+\sum _{j\ne 1,2}^K{y}_v^{\left(k\right)}{e}^{\left(-{∥{\mathit{y}}^{\left(k\right)}∥}^2/2+x_v{y}_v^{\left(k\right)}\right)/{\sigma }^2\left(t\right)}\right)-x_v^{*}\right).$$

If we approximate the interference function with constant $\varphi$ using a zero-th order Taylor expansion, Equation (44) becomes the self-consistency equation of a Curie-=Weiss model of magnetism, with temperature $T={\sigma }^2\left(t\right)$ and external magnetic field $\varphi$. The solutions of this equation can be visualized as intersection points between a straight line and a hyperbolic tangent (see [15,16] for a detailed analysis). When $\varphi$ is finite, the system transitions discontinuously from one to two fixed-points, which corresponds to a first-order phase transition in the magnetic system. However, the size of the discontinuity vanishes when $\varphi =0$, when there is an exact symmetry between the two data-points (see Figure 1). This gives rise to a so-called critical phase transition, where a single fixed point at $x^{*}=0$ continuously splits into two paths ${\mathit{x}}_1\left(t\right)$ and $x_2\left(t\right)$ with ${\mathit{x}}_{1,2}(t-t_c)\sim \pm {(t-t_c)}^{1/2}$ for $t\to 0$. The loss of stability of the fixed-point at the origin corresponds to the vanishing of the quadratic well around the point:

$${\displaystyle \frac{{\partial }^2}{\partial x_v^2}}log{p}_{t_c}\left(x_{t_c}^{*}\right)=0,$$

(45)

where, in this case, ${\mathit{x}}_{t_c}^{*}=0$ for $t<t_c$. The analysis we just carried out involves the breaking of the permutation symmetry between two isolated data-points. On the other hand, if the symmetry is broken along all directions like in the case where the data manifold is a sphere centered at ${\mathit{x}}_t^{*}$, Equation (45) implies that

$$\mathrm{Tr}\left[{\nabla }^T\nabla log{p}_t\left({\mathit{x}}_{t_c}^{*}\right)\right]=\nabla ·\nabla log{p}_t\left({\mathit{x}}_{t_c}^{*}\right)=\mathbf{0}$$

(46)

Therefore, the change in stability condition can be reformulated as the local vanishing (or suppression in a less symmetric case) of the divergence of the vector field that drives the generative dynamics. The transition from the super-critical ($t>t_c$) and the sub-critical ($t<t_c)$ phases then corresponds to a sign change in the divergence of the vector field (i.e., the score) in the spherically symmetric case, or a sign change of the divergence restricted to a sub-space in the general case, with the sub-critical regime being characterized by positive eigenvalues of the Jacobi matrix that lead to divergent local trajectories (see Figure 2).

6. Dynamics of the Generative Trajectories

Around a point ${\mathit{x}}_t$, the local behavior of the generative trajectories under the deterministic ODE flow dynamics can be characterized by the eigenvalues of the symmetric part of the vector field, which quantify the separation rate of infinitesimally close trajectories.

Let $\dot{\mathit{x}}=f\left(\mathit{x}\right)$ and denote by ${\mathit{x}}_{t+\tau }(·)$ the flow map that advances an initial condition at time t by an increment $\tau$. For a small perturbation $\mathit{w}$ at ${\mathit{x}}_t$, define the path difference

$$\delta {\mathit{x}}_t\left(\tau \right):={\mathit{x}}_{t+\tau }({\mathit{x}}_t+\mathit{w})-{\mathit{x}}_{t+\tau }\left({\mathit{x}}_t\right).$$

(47)

In the limit of vanishing $\parallel \mathit{w}\parallel$ and small $\tau$, a first-order expansion yields

$$\delta {\mathit{x}}_t\left(\tau \right)\approx \left(I+J_t\tau \right)\delta {\mathit{x}}_t\left(0\right),J_t:=Df\left({\mathit{x}}_t\right),$$

(48)

and therefore the perturbation obeys

$${\left.{\displaystyle \frac{d\delta {\mathit{x}}_t}{d\tau }}\right|}_{\tau =0}=J_t\delta {\mathit{x}}_t\left(0\right).$$

(49)

The evolution of the perturbation magnitude is controlled by the symmetric part of $J_t$ (note that for the gradient flows, like in the case of the diffusion, the Jacobian is fully symmetric). Letting $S_t:=\frac{1}{2}(J_t+J_t^{\top })$, we obtain

$${\left.{\displaystyle \frac{d}{d\tau }}{\parallel \delta {\mathit{x}}_t\left(\tau \right)\parallel }^2\right|}_{\tau =0}=2\delta {\mathit{x}}_t{\left(0\right)}^{\top }{S}_t\delta {\mathit{x}}_t\left(0\right).$$

(50)

Maximizing over directions gives the local maximal stretching rate ${\lambda }_{max}\left(S_t\right)$, while individual directions $\mathit{w}$ experience the direction-dependent stretching rate given by the equation above. When the flow is integrated backward in time (as in reverse-time generation), these instantaneous stretching rates flip sign, so directions that are locally contracting forward in time become locally expanding under the reverse dynamics, leading to exponential sensitivity to perturbations over sufficiently short horizons.

To connect this local stretching picture to symmetry breaking at $t_c$, consider the reverse-time generative dynamics in the immediate sub-critical regime $t=t_c-ϵ$ with $ϵ\ll 1$ and linearize around the symmetric (unstable) branch ${\mathit{x}}_t^{*}$. Writing the perturbation dynamics as $\dot{\delta \mathit{x}}=-J_t\left({\mathit{x}}_t^{*}\right)\delta \mathit{x}$, we obtain the finite-horizon propagation

$$\delta \mathit{x}\left(\tau \right)=e^{-\tau J_t\left({\mathit{x}}_t^{*}\right)}\delta \mathit{x}\left(0\right),$$

(51)

so that perturbations with overlap on expanding modes of the reverse dynamics grow exponentially over $\tau$ (equivalently, along directions that are locally contracting in the forward-time flow, i.e., modes associated with negative eigenvalues for the forward dynamics). In the immediate sub-critical phase of a symmetry-breaking phase transition, we know that there is a non-empty subspace spanned by the eigenvector of the Jacobian corresponding to negative eigenvalues. Therefore, perturbations along this unstable eigen-space will be exponentially amplified by the generative dynamics. In the stochastic case, this can be seen as a critical ‘macroscopic amplification’ of the infinitesimal noise input, where the noise breaks the symmetry of the generative model. In the deterministic dynamics, the symmetry is instead broken by the amplification of small differences between the generative trajectories.

In general, we will refer to the spectrum of Jacobian eigenvalues ${\lambda }_j({\mathit{x}}_t,t)$ as the local Lyapunov spectrum. As we shall see, this spectrum can be directly related to the conditional entropy production.

6.1. The Global Perspective on Generative Bifurcations

In the previous sections, we characterized the generative dynamics of diffusion models by studying the associated paths of fixed points in term of their stability and bifurcations, which led us to establish formal connections with the statistical physics of phase transitions and symmetry breaking. However, in high dimensions, small volumes around a fixed point have vanishingly low probability of being visited. In fact, due to the dispersive effect of the noise, the generative trajectories are concentrated on fixed-variance shells around the fixed points. More formally, these set of “typical” points form tubular neighborhoods of the set of fixed-points (see Figure 1). It is therefore unclear how a bifurcation in a path of fixed-points affects the behavior of the generative trajectories, since the analysis we presented in the previous sections was purely local.

To gain insight into the global behavior of the typical generative trajectories, we can study the expected divergence of the vector field at time t

$$\mathrm{div}\left(t\right)={\mathbb{E}}_{{\mathit{x}}_t}\left[\nabla ·\nabla log{p}_t\left({\mathit{x}}_t\right)\right]={\mathbb{E}}_{{\mathit{x}}_t}\left[\mathrm{Tr}\left[{\nabla }^T\nabla log{p}_t\left({\mathit{x}}_t\right)\right]\right].$$

(52)

If $\mathrm{div}\left(t\right)$ is negative, the separation between the generative trajectories will, on average, be contracted by the generative dynamics. The simplest example of this contractive behavior can be studied by considering a data distribution with a single point: $p_0\left({\mathit{x}}_t\right)=\delta (\mathit{y}-\mathit{c})$. In this case, all trajectories converge to $\mathit{c}$ for $t\to 0$, and we have

$${\mathrm{div}}_1\left(t\right)=-{\displaystyle \frac{D}{{\sigma }^2\left(t\right)}}.$$

(53)

where D is the dimensionality of the space. In the reverse dynamics, the negative sign implies that the forward process produces a stable dynamics where the particles ‘fall’ towards the data points.

In the general case, this quantity can be identified with the ‘trivial component’ of the expected divergence since it does not depend on the data but only on the forward process. In the general case, it can be expressed as

$${\mathrm{div}}_1\left(t\right)={\mathbb{E}}_{{\mathit{x}}_t}\left[\mathrm{Tr}\left[{\nabla }^T\nabla log{p}_t({\mathit{x}}_t\mid \mathit{y})\right]\right].$$

(54)

We can therefore study the purely data-dependent part of the expected divergence by subtracting this ‘trivial component’:

$$\Delta \mathrm{div}\left(t\right)=\mathrm{div}\left(t\right)-{\mathrm{div}}_1\left(t\right).$$

(55)

Intuitively, $\Delta \mathrm{div}\left({\mathit{x}}_t\right)$ encodes the separation of the typical trajectories in the reverse process due to bifurcations in the generative process, which mirrors the local analysis we carried out in the previous sections at the level of the fixed points.

Using integration by parts, it is straightforward to connect the expected divergence with the conditional entropy rate

$$\dot{\mathbf{H}}[\mathit{y}\mid {\mathit{x}}_t]={\displaystyle \frac{{\nu }^2\left(t\right)}{2}}\Delta \mathrm{div}\left(t\right)$$

(56)

Therefore, the expected data-dependent divergence of the generative trajectories directly determines the conditional entropy rate. From this identity, we can immediately deduce that $\Delta \mathrm{div}\left({\mathit{x}}_t\right)$ is non-negative-valued and consequently that $\mathrm{div}\left(t\right)\ge {\mathrm{div}}_1\left(t\right)$.

We can also show that the marginal entropy is produced by the expected divergence

$$\dot{\mathbf{H}}\left[{\mathit{x}}_t\right]=-{\displaystyle \frac{{\nu }^2\left(t\right)}{2}}\mathrm{div}\left(t\right),$$

(57)

which implies that $\mathrm{div}\left(t\right)\le 0$ since the marginal entropy is a monotonically increasing function of t under our forward process. This reflects the fact that the forward process always leads to a dispersion of the trajectories, regardless to the nature of the initial distribution. From this, we can conclude that the maximum bandwidth is achieved when

$$\mathrm{div}\left(t\right)={\mathbb{E}}_{{\mathit{x}}_t}\left[\mathrm{Tr}\left[{\nabla }^T\nabla log{p}_t\left({\mathit{x}}_t\right)\right]\right]\to 0.$$

(58)

This gives us a clear connection between the local vanishing of the Jacobian in spontaneous symmetry breaking (Equation (46)) with the expected vanishing that corresponds to saturation of the generative bandwidth.

6.2. Information Geometry

The derivation in the previous sub-section suggests a deep connection between the information production and the geometry of the data manifold. We can further analyze this connection by using concepts from information geometry [45]. The key connection is that the conditional entropy rate is in fact just the expected value of the trace of the Fisher information matrix, which can be defined as follows:

$${\mathcal{I}}_t\left({\mathit{x}}_t\right)=-{\mathbb{E}}_{\mathit{y}\mid {\mathit{x}}_t}\left[\nabla {\nabla }^{T}logp(\mathit{y}\mid {\mathit{x}}_t)\right].$$

(59)

This quantity quantifies the sensitivity of the posterior distribution $p(\mathit{y}\mid {\mathit{x}}_t)$ to changes in ${\mathit{x}}_t$ and can be interpreted as a natural metric tensor on the variable ${\mathit{x}}_t$. Using Bayes’ theorem and our simplified forward process, the expression can be rewritten as

$${\mathcal{I}}_t\left({\mathit{x}}_t\right)={\sigma }^{-2}\left(t\right)\left(I+{\sigma }^2\left(t\right)J\left({\mathit{x}}_t\right)\right),$$

(60)

Geometric information such as the manifold dimensionality is encoded in the spectrum of this matrix [26,46,47,48]. The Fisher information metric provides information on the (local) manifold structure of the data $\mathit{y}$ as seen through the lenses of the noisy state ${\mathit{x}}_t$. This is easy to see in the case where the data is Gaussian with covariance matrix ${\Sigma }_0$, which gives the formula

$${\mathcal{I}}_t={\sigma }^{-2}\left(t\right)I-{\left({\Sigma }_0+{\sigma }^2\left(t\right)I\right)}^{-1}.$$

(61)

When $\mathit{y}$ is supported on a $D_{\mathrm{data}}$ manifold, the (degenerate) eigenvalue ${\lambda }_{\left|\right|}$ corresponding to the orthogonal complement is equal to zero. On the other hand, in the flat limit, the tangent eigenvalues become equal to ${\Sigma }_0^{-1}$. This implies that the dimensionality of the manifold is given by the dimensionality of the eigenspace corresponding to the eigenvalue ${\lambda }_{\Vert }={\sigma }^{-2}\left(t\right)$. In the general case, the eigen-decomposition of $\mathcal{I}\left({\mathit{x}}_t\right)$ characterizes the local tangent structure of the manifold [47,48].

We can now use these expressions to cast light on the geometry of entropy production. The conditional entropy rate is directly related to the trace of the Fisher information matrix:

$$\dot{\mathbf{H}}[\mathit{y}\mid {\mathit{x}}_t]={\displaystyle \frac{1}{2}}{\nu }^2\left(t\right){\mathbb{E}}_{{\mathbf{x}}_t}\left[\mathrm{Tr}\left[\mathcal{I}\left({\mathbf{x}}_t\right)\right]\right],$$

(62)

which reduces to Equation (38) in the linear manifold case we just considered. From this perspective, it is clear that the reduction in bandwidth is the result of the suppression of the eigenvalues of $\mathcal{I}\left({\mathit{x}}_t\right)$. This can also be seen in the general case by re-expressing the entropy rate in terms of the expected eigenvalues of the Jacobi matrix:

$$\dot{\mathbf{H}}[\mathit{y}\mid {\mathit{x}}_t]={\displaystyle \frac{{\nu }^2\left(t\right)}{2{\sigma }^2\left(t\right)}}\left(D+{\sigma }^2\left(t\right)\sum _j\mathbb{E}\left[{\lambda }_j\left({\mathbf{x}}_t\right)\right]\right).$$

(63)

This equation shows that the entropy production is directly regulated by the spectrum of expected local Lyapunov exponents, as studied in our local analysis.

We can better understand this formula by rewriting it as follows:

$$\dot{\mathbf{H}}[\mathit{y}\mid {\mathit{x}}_t]={\displaystyle \frac{{\nu }^2\left(t\right)}{2}}\sum _j\left(1/{\sigma }^2\left(t\right)+\mathbb{E}\left[{\lambda }_j\left({\mathbf{x}}_t\right)\right]\right).$$

(64)

From this, we can see that conditional entropy production in an eigenspace is fully suppressed when $\mathbb{E}\left[{\lambda }_j\left({\mathbf{x}}_t\right)\right]=-1/{\sigma }^2\left(t\right)$, which is the eigenvalue of the Jacobian of the conditional score under the isotropic forward process.

7. A Stochastic Thermodynamic Perspective

A central question in generative diffusion is how uncertainty about the clean sample $x_0$ is resolved as the model evolves from the noisy state $x_t$ toward the data manifold. As argued throughout this paper, the appropriate notion of inferential uncertainty is the previously discussed conditional entropy $\mathbf{H}[{\mathbf{x}}_0\mid {\mathbf{x}}_t]$ and, more fundamentally, its pathwise realization. The study of pathwise entropy is naturally motivated by ideas from stochastic thermodynamics. However, we believe that the commonly used entropy in stochastic thermodynamics [33,34] is not the correct quantity for understanding generative dynamics. It measures the irreversibility of the forward diffusion, not the uncertainty relevant to generating a single outcome.

For a given point on the trajectory ${\mathbf{x}}_t$, we define its path-dependent conditional entropy as

$$h_t\left({\mathbf{x}}_t\right)=-\int p(x_0\mid {\mathbf{x}}_t)logp(x_0\mid {\mathbf{x}}_t)d{x}_0.$$

(65)

This quantity measures the uncertainty experienced along a single generative path. Its expectation is the usual conditional entropy,

$$\mathbb{E}\left[h_t\left({\mathbf{x}}_t\right)\right]=H[{\mathbf{x}}_0\mid {\mathbf{x}}_t],$$

but its fluctuations encode a structure that is invisible to marginal entropies. In particular, as illustrated in Figure 3, the pathwise conditional entropy $h_t\left({\mathbf{x}}_t\right)$ can locally increase along individual generative trajectories even as the mean conditional entropy decreases, a behavior reminiscent of entropy fluctuations in stochastic thermodynamics. Such effects do not arise in autoregressive models, where each generation step reduces uncertainty about the final sequence by revealing one token, since $\mathbf{H}[{\mathbf{x}}_{i+1:n}\mid {\mathbf{x}}_{1:i}]\le \mathbf{H}[{\mathbf{x}}_{i+1:n}\mid {\mathbf{x}}_{1:i}]+\mathbf{H}[{\mathbf{x}}_i\mid {\mathbf{x}}_{1:i-1}]=\mathbf{H}[{\mathbf{x}}_{i:n}\mid {\mathbf{x}}_{1:i-1}]$. Whether these entropy fluctuations in diffusion-based generation have any practical advantage, however, remains an open question.

To expose this dynamical heterogeneity, we consider the variance of the pathwise conditional entropy,

$${\mathcal{V}}_h\left(t\right):=Var\left[h_t\left({\mathbf{x}}_t\right)\right].$$

(66)

7.1. Variance of Pathwise Conditional Entropy as a Signature of Symmetry Breaking

We find that the variance captures symmetry-breaking transitions (see Figure 3). To gain a better understanding, we explore the general behavior of the variance and demonstrate a connection with the speciation time [24].

Two limits are immediate. At very early times, the noise scale is negligible compared to the curvature of the data manifold. The posterior $p(x_0\mid x_t)$ is effectively confined to the local tangent plane, behaving as an isotropic Gaussian whose shape is determined solely by the intrinsic dimension $D_{data}$ (we assume that the dimension is uniform across the manifold). Because the entropy of this Gaussian depends on k and t but is insensitive to the specific location on the manifold,

$$h_t\left(x_t\right)\approx \mathrm{const}\Rightarrow {\mathcal{V}}_h\left(t\right)\approx 0.$$

(67)

At very late times, the diffusion has effectively mixed the data distribution: $x_t$ carries little discriminative information about the origin $x_0$ and the posterior becomes approximately independent of $x_t$, again implying

$$h_t\left(x_t\right)\approx \mathrm{const}\Rightarrow {\mathcal{V}}_h\left(t\right)\approx 0.$$

(68)

Thus, nontrivial variance can only arise in an intermediate regime where different trajectories resolve uncertainty in different ways.

Furthermore, near a bifurcation/decision time $t_c$, the ensemble contains a substantial fraction of trajectories that are already decisively committed to a branch and a substantial fraction that remain ambiguous. In this regime, $h_t\left(x_t\right)$ becomes broadly distributed (some paths yield low entropy, others high entropy), and ${\mathcal{V}}_h\left(t\right)$ is therefore maximized.

Connection with the Speciation Time

As already hinted at, for Gaussian mixture models, the variance of the pathwise conditional entropy develops a pronounced peak on timescales of the order of the speciation time in the sense of Biroli et al. [24]. Interestingly, this behavior is observed for the VP SDE, but is absent for the EDM and VE SDEs. We provide a brief, intuitive argument of the proof for the VP case here. More information about the setup and detailed derivation is deferred to Appendix A.

We consider a high-dimensional Gaussian mixture in the regime where, as $d\to \infty$, the inter-class mean separations scale as $\parallel {\mu }_i-{\mu }_j\parallel =\Theta \left(\sqrt{d}\right)$ while within-class covariances remain $O\left(1\right)$. Throughout this discussion we focus on the Ornstein–Uhlenbeck (VP) forward process, since it yields a sharp speciation crossover. In contrast, the EDM parametrization does not generally exhibit the same sharp transition under an analogous rescaling.

Introduce a latent class label $\mathbf{z}\in \{1,\dots ,N\}$ so that ${\mathbf{x}}_0\mid (\mathbf{z}=i)$ is drawn from the i-th component. Conditioning on a realization $x_t$, the conditional entropy satisfies

$$\mathbf{H}[{\mathbf{x}}_0\mid {\mathbf{x}}_t=x_t]=\mathbf{H}[{\mathbf{x}}_0\mid {\mathbf{x}}_t=x_t,\mathbf{z}]+\mathbf{H}[\mathbf{z}\mid {\mathbf{x}}_t=x_t]-\mathbf{H}[\mathbf{z}\mid {\mathbf{x}}_0,{\mathbf{x}}_t=x_t],$$

(69)

i.e., the chain rule for $({\mathbf{x}}_0,\mathbf{z})$ given ${\mathbf{x}}_t=x_t$. In the well-separated regime (as is the case for $d\to \infty$), $\mathbf{z}$ is essentially fixed once ${\mathbf{x}}_0$ is known, so $H[\mathbf{z}\mid {\mathbf{x}}_0,{\mathbf{x}}_t=x_t]$ is negligible. For Gaussian components and Gaussian forward kernels, the first term is non-random, $p(x_0\mid x_t,\mathbf{z}=i)$ is Gaussian with a covariance that does not depend on $x_t$, and in the shared-covariance case, it is the same across i. Thus the dominant source of fluctuations in $h_t\left({\mathbf{x}}_t\right)$ is the class-uncertainty term $H[\mathbf{z}\mid {\mathbf{x}}_t]$. In regions where the posterior concentrates on one component, $H[\mathbf{z}\mid x_t]\approx 0$, whereas in the maximally mixed region $H[\mathbf{z}\mid x_t]\approx H\left(\pi \right)$.

Under OU noising,

$${\mathbf{x}}_t={\alpha }_t{\mathbf{x}}_0+{\sigma }_t\epsilon ,{\alpha }_t=e^{-t},{\sigma }_t^2=1-e^{-2t}.$$

(70)

With $\parallel {\mu }_i-{\mu }_j\parallel =\Theta \left(\sqrt{d}\right)$ and $O\left(1\right)$ within-class covariance, the posterior over $\mathbf{z}$ depends on $(t,d)$ essentially through a single effective SNR,

$${\eta }_d\left(t\right):={\displaystyle \frac{d{\alpha }_t^2}{{\sigma }_t^2}}={\displaystyle \frac{d}{e^{2t}-1}}.$$

(71)

Equivalently, the variance is a deterministic function that only depends on the effective SNR,

$${\mathcal{V}}_h\left(t\right)=Var\left(h_t\left({\mathbf{x}}_t\right)\right)\approx F\left(\eta \left(t\right)\right),$$

(72)

where F is determined by the geometry of the mixture. The endpoints are deterministic; $\eta \to \infty$ (early times) makes $\mathbf{z}$ effectively known and $h_t\left({\mathbf{x}}_t\right)$ nearly constant, while $\eta \to 0$ (late times) makes $\mathbf{z}$ effectively independent of ${\mathbf{x}}_t$ and $h_t\left({\mathbf{x}}_t\right)$ again nearly constant. Hence $F(\infty )=F\left(0\right)=0$, and nontrivial variance can only occur when ${\eta }_d\left(t\right)=O\left(1\right)$.

This is exactly the speciation crossover. Using the dynamical-regimes rescaling $t_S\left(d\right)=\frac{1}{2}logd$ and $u=t/t_S\left(d\right)$, we have $e^{2t}=d^u$ and therefore

$${\eta }_d\left(u{t}_S\left(d\right)\right)={\displaystyle \frac{d}{d^u-1}},$$

(73)

which collapses to ∞ for $u<1$ and to 0 for $u>1$, while staying $O\left(1\right)$ at $u=1$. Thus, as d grows, the variance profile concentrates into a peak at $u=1$, matching the phase-transition behavior observed in Figure 4.

The same mechanism is not specific to Gaussians. What matters is (i) a latent ‘class’ structure with separation $\Theta \left(\sqrt{d}\right)$ and (ii) $O\left(1\right)$ within-class fluctuations so that, after the normalization implicit in (71), component-specific details are subleading and the posterior over $\mathbf{z}$ is controlled by a single effective SNR. For strongly non-Gaussian components, $Var\left[h_t\left({\mathbf{x}}_t\right)\right]$ may additionally show an early-time feature associated with rapid local Gaussianization under the forward kernel. This effect is absent in the ideal Gaussian case and is suppressed when each component is already close to Gaussian.

8. Discussion and Conclusions

This paper has presented a unified framework that connects the dynamics, information theory, and statistical physics of generative diffusion. We have shown that the generative process is governed by the conditional entropy rate, which is directly tied to the expected divergence of the score function’s vector field and, equivalently, to the expected squared norm of the score. This quantity captures how uncertainty about the clean sample is resolved during denoising and reveals when the score is suppressed, allowing noise to drive the dynamics. In this view, the branching of generative trajectories arises from noise-induced symmetry-breaking transitions that occur when multiple datapoints remain compatible with the noisy state, and the model is forced to commit to a specific outcome.

By analyzing the fixed points of the score function and their stability, we showed that these generative decisions are formalized as bifurcations of the score field, which can be mapped onto classical symmetry-breaking phase transitions such as those described by mean-field models like the Curie–Weiss magnet. Peaks in the conditional entropy rate coincide with these bifurcation points, marking moments of maximal posterior mixing and heightened sensitivity to noise, where small fluctuations determine the generative branch taken by the system.

Our results also clarify the relationship between generative diffusion and stochastic thermodynamics. While stochastic thermodynamic entropy characterizes the irreversibility of the process, the conditional entropy studied here captures the inferential uncertainty relevant to generating a single sample. At the trajectory level, the pathwise conditional entropy and its variance reveal heterogeneity in how different generative paths resolve uncertainty, with variance peaks emerging precisely during symmetry-breaking events. From this perspective, entropy fluctuations are not incidental but constitute an information-theoretic signature of generative decisions.

In conclusion, generative diffusion can be understood as a dynamical system that progressively breaks symmetries in the energy landscape while regulating the flow of information through posterior mixing. The score function acts as a dynamic filter that suppresses noise along resolved directions while leaving unresolved directions weakly constrained, thereby controlling the generative bandwidth. This perspective provides a coherent explanation of how diffusion models transform noise into structured data and connects the learning dynamics of modern generative models to fundamental principles of information theory and statistical physics.

Beyond conceptual unification, this framework suggests practical implications for model design and analysis. Because entropy production and posterior mixing are directly linked to the score norm, they offer principled signals for identifying critical periods of high information transfer, motivating adaptive training and sampling strategies that target generative decision points [12]. More broadly, the information-thermodynamic perspective developed here provides a natural language for studying memorization, mode formation, and generalization, and may guide the development of future generative models that explicitly leverage controlled symmetry breaking to represent hierarchical and semantic structure.