Stochastic Thermodynamics of Learning Parametric Probabilistic Models

Abstract

We have formulated a family of machine learning problems as the time evolution of parametric probabilistic models (PPMs), inherently rendering a thermodynamic process. Our primary motivation is to leverage the rich toolbox of thermodynamics of information to assess the information-theoretic content of learning a probabilistic model. We first introduce two information-theoretic metrics, memorized information (M-info) and learned information (L-info), which trace the flow of information during the learning process of PPMs. Then, we demonstrate that the accumulation of L-info during the learning process is associated with entropy production, and the parameters serve as a heat reservoir in this process, capturing learned information in the form of M-info.

1. Introduction

Starting nearly half a century ago, physicists began to learn that information is a physical entity [1,2,3]. Today, the information-theoretic perspective has significantly impacted various fields of physics, including quantum computing [4], cosmology [5], and thermodynamics [6]. Simultaneously, recent years have witnessed the remarkable success of an algorithmic approach known as machine learning, which is adept at learning information from data. This paper is propelled by a straightforward proposition: if “information is physical”, then the process of learning information must inherently be a physical process.

The concepts of memory, prediction, and information exchange between subsystems have undergone extensive exploration within the realms of thermodynamics of information [6] and stochastic thermodynamics [7]. For instance, Still et al. [8] delved into the thermodynamics of prediction, and the role of information exchange between thermodynamic subsystems has been studied by Sagawa and Ueda [9] and Esposito et al. [10]. This rich toolbox of thermodynamics of information is our main venue to study the physics of the machine learning process, with motivation to assess the information content of the learning process.

The type of machine learning problem we consider in this study encompasses any algorithmic approach that evolves a parametric probabilistic model (PPM) or simply the model toward a desirable target distribution through gradient-based loss function minimization. To establish our notation, consider a set of observations denoted by the training dataset B, drawn from an unknown target distribution $p^{\ast }$. The PPM, without lose of generality, can be written as the follows:

$$p_{\theta }(X=x)=e^{-{\varphi }_{\theta }\left(x\right)}$$

(1)

This distribution is parameterized by a set of parameters $\theta \in {\mathbb{R}}^M$. The objective of learning is to find a set of parameters such that the samples drawn from the model exhibit a desirable statistical characteristics. In machine learning practice, one constructs the function ${\varphi }_{\theta }\left(x\right)$ with a (deep) neural network and leaves the parameter selection task to an optimizer that minimizes a loss function. Examples include energy-based models [11], Large Language Models (LLMs), Softmax classifiers, and Variational Autoencoders (VAEs) [12].

While the information-theoretic approach to this problem is prevalent in the field [13,14,15,16], it has also faced criticism [17]. Our primary motivation for framing learning in a PPM as a thermodynamic process is to facilitate the assessment of the information content inherent in the learning process. The structure of this paper is as follows. Section 2 briefly discusses prior information-theoretic approaches to the learning problem with PPM and the challenges they encounter. Subsequently, we introduce our own information-theoretic metrics. Finally, Section 3 and Section 6 employ the thermodynamic framework to address these information-theoretic inquiries.

2. Information Content of PPMs

Locating information within the parametric model (i.e., the neural network) remains a fundamental question in machine learning [18]. This challenge is central to any information-theoretic perspective on machine learning problems. In a pioneering study, Shwartz-Ziv et al. [19] quantified the internal information within neural networks by estimating the mutual information between inputs and the activities of hidden neurons. Moreover, they employed the information bottleneck theory to interpret the decrease in this mutual information as evidence of data compression during learning. This perspective garnered significant attention in the field [15,16], reinforcing the view of neural networks as an information channel. However, the study encountered critiques [17,20]. A primary problem was that the hidden neurons’ activity in a neural network constitutes a deterministic function of the input. Such determinism inherently possesses a trivial mutual information value, even prior to any learning. The challenge of defining a well-defined and interpretable (Shannon) information metric in deterministic neural networks has prompted the proposition that neural network information processing is geometric in nature [17] (given that inputs are mapped to a latent space of differing dimensions) rather than information-theoretical.

In a distinct research direction, the authors of [21] addressed the significance of assessing the information content of the model’s parameters. In our study, we echo this view, emphasizing that the parameters are the primary carriers of learned information within neural networks. Consequently, any information-theoretic measure of learned information by the model should be grounded in the parameters rather than the deterministic activity of hidden neurons. However, quantifying the information within parameters poses challenges, primarily due to the elusive nature of their distribution [22]. In this section, we introduce two information-theoretic metrics crafted to assess the information content within the learning process of a PPM. This paves the way for computing these quantities within the thermodynamic framework.

To avoid introducing new notation, we also denote B as the ground truth random variable associated with the target distribution $p^{\ast }$ from which the training dataset is sampled. Subsequently, we represent the action of the optimizer as a map between this ground truth random variable and the desired set of parameters after n optimization steps:

$${\Theta }_{t_n}={\Lambda }_n\left(B\right)$$

(2)

The map ${\Lambda }_n$ incorporates the structure of the loss function, the optimization algorithm, and any hyperparameters related to the optimizer’s action. We exclude the initial parameters’ value from this map’s argument under the assumption that as n increases, the final set of parameters becomes independent of its initial condition. In information theory terminology, this map corresponds to a statistic of the ground truth random variable [23]. Moreover, the outcome of this map defines a model from which the final model-generated sample is sampled: $X_{t_n}\sim p\left(x\right|{\theta }_{t_n})$. Considering that the model-generated sample becomes independent of the ground truth random variable given the parameters, we can express the following Markov chain governing the learning process:

$$B\to {\Theta }_{t_n}\to X_{t_n}.$$

(3)

The data processing inequality (DPI) associated with this Markov chain serves as our framework to define two information-theoretic metrics that gauge the information content of the model:

$$\underset{\mathrm{M}\text{-}\mathrm{info}}{\underbrace{I_{\Theta ;B}\left(t_n\right)}}\ge \underset{\mathrm{L}\text{-}\mathrm{info}}{\underbrace{I_{X;B}\left(t_n\right)}}.$$

(4)

We used the notations presented in

Table 1. A list of notations used in this paper.

${\Delta }_{t_n}f\left(t\right):=f\left(t_n\right)-f\left(0\right)$	Change over the interval $[0,t_n]$
$<f\left(x\right){>}_{p\left(x\right)}:=\int dxp\left(x\right)f\left(x\right)$	Average over $p\left(x\right)$
$s_X\left(t\right):=s\left[p_t\left(x\right)\right]:=-ln{p}_t\left(x\right)$	Surprisal of $p_t\left(x\right)$
$S_X\left(t\right):=S\left[p_t\left(x\right)\right]:=<-ln{p}_t\left(x\right){>}_{p_t\left(x\right)}$	Shannon entropy of $p_t\left(x\right)$
$I_{X;\Theta }\left(t\right):=I[X_t;{\Theta }_t]:=S_X\left(t\right)-S_{X|\Theta }\left(t\right)$	Mutual information between X and $\Theta$ at time t

. The left-hand side of this inequality quantifies the accumulation of mutual information between the parameters and the training dataset, while the right-hand side characterizes the performance of the generative model, which gauges the accumulation of mutual information between the model’s generated samples and the training dataset. We refer to the former as memorized information (M-info) and the latter as learned information (L-info). We also note that both of these quantities start at zero before the training begins. Thus, their measurements at $t_n$ reveal accumulation of information during the learning process.

In the context of the learning problem, the DPI, as referenced in Equation (4), suggests that what is memorized is always greater than or equal to what is learned. The L-info metric is task-oriented. For example, in the realm of image generation, it quantifies the statistical resemblance between the model’s outputs and the genuine images. In the case of classification tasks, L-info would encapsulate only the pertinent information for label prediction. In contrast, M-info can encompass information not directly pertinent to the current task. For instance, it might capture intricate pixel configurations in an image dataset, which are not crucial for identifying distinct patterns like human faces. The DPI in Equation (4) neatly illustrates the risk of overfitting, which is when a model starts to incorporate extraneous information that does not align with the learning objective. The necessity of constraining the information in a model’s parameters is highlighted in [21], echoing the minimum description length principle [24]. Additionally, studies suggest that the SGD optimizer tends to favor models with minimal information in their parameters [22]. The recent work by the authors of [25] even proposed an upper limit for minimizing parameter information to bolster generalization capabilities. These findings suggest that the learning process seeks to minimize the left-hand side of the DPI inequality while simultaneously maximizing the right-hand side, which measures the model performance. This leads us to an ideal scenario where $I_{\Theta ;B}\left(t_n\right)=I_{X;B}\left(t_n\right)$, signifying that all memorized information is relevant to the learning task.

We now take one step further in our definition of M-info and L-info. First, the presence of the optimizer map, as referenced in Equation (2) and connecting the ground truth source of the training dataset to the parameters, allows us to simplify M-info as follows:

$$\mathrm{M}\text{-}\mathrm{info}:=I_{\Theta ;B}\left(t_n\right)=S\left({\Theta }_{t_n}\right)$$

(5)

Thus, the parameters naturally emerge as the model’s memory, where its Shannon entropy measures the stored information during the learning process.

Second, we swap B for ${\Theta }_{t_n}$ in the definition of L-info at the cost of losing some information:

$$\begin{array}{cc}\hfill \mathrm{L}\text{-}\mathrm{info}:=I_{X;B}\left(t_n\right)& =I_{X;{\Lambda }_{t_n}\left(B\right)}+ϵ\hfill \\ \hfill & =I_{X;\Theta }\left(t_n\right)+ϵ\hfill \end{array}$$

(6)

where $ϵ$ is a non-negative number that equals zero only when ${\Lambda }_n$ is a sufficient statistic of B. For the above expression, the condition of a sufficient statistic can be eased when ${\Theta }_{t_n}$ becomes sufficient only with respect to X. In other words, when ${\Lambda }_n$ preserves all information in B that is also mutual in $X_{t_n}$, i.e., preserving all task-oriented information. Indeed, we are interested in this type of preservative map in machine learning practice. Therefore, for a well-chosen optimizer, we consider $I_{X;\Theta }$ to be a reasonable proxy to L-info, and we use the two interchangeably:

$$\mathrm{L}\text{-}\mathrm{info}:=I_{X;\Theta }\left(t_n\right)$$

(7)

3. The Learning Trajectory of a PPM

The time evolution of the PPM is the first clue to frame the learning process as a thermodynamic process. To illustrate this, consider a discretized time interval $[0,t_n]$, which represents the time needed for n optimization steps of the parameters. During this time, the optimizer algorithm draws a sequence of independent and identically distributed samples from the training dataset. We denote this sequence by ${\mathit{b}}_n:=\{b_{t_1},b_{t_2},\cdots ,b_{t_n}\}$ and refer to it as the “input trajectory”. Then, the outcome of the optimization defines a sequence of parameters called the “parameters’ trajectory”: ${\mathit{\theta }}_n:=\{{\theta }_0,{\theta }_{t_1},{\theta }_{t_2},\cdots ,{\theta }_{t_n}\}$. Each realization of parameters defines a specific PPM. Consequently, the parameters’ trajectory produces a sequence of PPMs:

$$\mathcal{T}:=\{p\left(X\right|{\theta }_0),p\left(X\right|{\theta }_{t_1}),p\left(X\right|{\theta }_{t_2}),\cdots ,p\left(X\right|{\theta }_{t_n})\}$$

(8)

We refer to this sequence as the learning trajectory, depicted in Figure 1. On the other hand, a thermodynamic process can be constructed solely from the time evolution of a distribution [26]. Therefore, we see $\mathcal{T}$ as a thermodynamic process. The physics of this process is encoded in the transition rates governing the master equation of this time evolution. Finding the transition rate associated to learning a PPM is our main task in this section.

3.1. The Model Subsystem

We refer to the subsystem that goes under the thermodynamic process $\mathcal{T}$ as the model subsystem. This subsystem has X degrees of freedom, and its microscopic states’ realization along the learning trajectory represent model-generated samples at each time step. We denote the stochastic trajectory of the model subsystem (model-generated samples) by ${\mathit{x}}_n:=\{x_{t_1},x_{t_2},\cdots ,x_{t_n}\}$. To avoid confusion with our notation, we consider the probability functions $p\left(x_{t_i}\right|{\theta }_{t_i})$ and $p\left(x_{t_{i-1}}\right|{\theta }_{t_i})$, which represent the probability of observing $x_{t_i}\in {\mathit{x}}_n$ and $x_{t_{i-1}}\in {\mathit{x}}_n$ at time $t=t_i$, respectively. Here, the time index of $\theta$ aligns with the time index of the PPM (i.e., $p_{t_i}\left(X\right|{\theta }_{t_i})\equiv p\left(X\right|{\theta }_{t_i})$) because the PPM is fully defined upon observing the parameters. In contrast, the time index on x denotes a specific observation within ${\mathit{x}}_n$. To simplify our notation, the absence of a time index on x denotes a generic realization of the random variable X, and we write $p\left(x\right|{\theta }_{t_i})$ instead of $p\left(X\right|{\theta }_{t_i})$.

3.2. The Parameter Subsystem

The parameters of the neural network at each step of optimization represent realization of the parameter subsystem with $\Theta$ degrees of freedom and the stochastic trajectory ${\mathit{\theta }}_n$. The statistical state of the parameter subsystem is given with the marginal $p\left({\theta }_{t_i}\right)$ at time step $t=t_i$. This marginal state represents the statistic of all possible outcomes of training a PPM on a specific learning objective. We can think of training an ensemble of computers on the same machine learning task. This allows us to think about the time evolution of the marginal $p\left({\theta }_{t_i}\right)$ and the joint distribution $p\left(x\right|{\theta }_{t_i})p\left({\theta }_{t_i}\right)$ during the learning process. We refer to this view as the ensemble view of the learning process. In practice, however, we train the PPM only once, and we do not have access to the marginal $p\left({\theta }_{t_i}\right)$. Thus, our model-generated samples are conditioned on specific observations of parameters ${\theta }_{t_i}\sim {\Theta }_{t_i}$. This defines the conditional view of the learning process, which is fully described by the learning trajectory of the PPM.

In machine learning practice, it is desirable for a training process to exhibit a robust outcome, regardless of who is running the code. One way to achieve this is by imposing a low variance condition on the statistics of the parameters across the ensemble of all learning trials. This condition asserts that the parameters’ trajectory across the ensemble is confined to a small region $D_n\subset {\mathcal{R}}^M$. As n grows larger, this region shrinks and becomes associated with the area surrounding the target distribution as depicted in Figure 1. Under this condition, we can express

$$<f\left(\theta \right){>}_{p\left({\theta }_n\right)}\approx f\left({\theta }^{\ast }\right),\forall {\theta }^{\ast }\in D_n$$

(9)

The above approximation becomes exact when $p\left({\theta }_n\right)$ assumes the form of a Dirac delta function, indicating a zero-variance condition in the parameters’ dynamics.

The low-variance condition proves invaluable when computing the information-theoretic measurements introduced in Section 2. This is because the computation of the M-info $I_{B;\Theta }$ and L-info $I_{X;\Theta }$ necessitates averaging over the parameters’ distribution. However, since we typically train our model just once, we lack direct access to the parameters’ distribution throughout the learning trajectory. To overcome this challenge, we introduce the partially averaged L-info:

$$I_{X:\Theta }\left({\theta }_t\right)=\int dxp\left(x\right|{\theta }_t)ln\frac{p\left(x\right|{\theta }_t)}{p_t\left(x\right)}$$

(10)

Subsequently, under the low-variance condition of the parameter subsystem, we can measure the partially averaged L-info as a proxy for the L-info, where $I_{X:\Theta }\left(t\right)=<I_{X:\Theta }\left(\theta \right){>}_{p_t\left(\theta \right)}\approx I_{X:\Theta }\left({\theta }_t\right)$. In Section 6, we will delve deeper into the evidence supporting the low-variance condition of the subsystem $\Theta$.

We refer to the joint $(X,\Theta )$ as the learning system, which embodies the thermodynamic process of learning a PPM. In this section, we will demonstrate that the thermodynamic exchange between the model subsystem and parameter subsystem is the primary source for producing M-info and L-info during the learning process. Before delving deeper, we establish two interconnected assumptions about the parameter subsystem: (1) The PPM is over-parameterized; specifically, the subsystem $\Theta$ has a much higher dimension compared with the subsystem X. (2) The parameter subsystem evolves in a quasi-static fashion (slow dynamics).

The foundation for these assumptions in machine learning is clear. Training over-parameterized models represents a pivotal achievement of machine learning algorithms, and the slow dynamics (often termed as lazy dynamics) of these over-parametrized models are well documented [27,28]. These characteristics underscore the significant role of the parameter subsystem in the learning process, akin to that of a heat reservoir. Over-parameterization implies a higher heat capacity for this subsystem compared with the model subsystem. Additionally, the quasi-dynamics align with the behavior of an ideal heat reservoir, which does not contribute to entropy production [29]. The role of the parameter subsystem as a reservoir aligns with the assumption of a low-variance condition for this subsystem. This is because we expect the stochastic dynamics of the reservoir to be low variance across the ensemble of all trials.

In this study, we attribute the role of an ideal heat reservoir to the parameter subsystem with an inverse temperature ${\beta }^{-1}=1$. In Section 6, we delve deeper into the rationale behind this assumption by examining the stochastic dynamics of parameters under a vanilla stochastic gradient descent optimizer and highlighting potential limitations of this assumption.

3.3. Lagged Bipartite Dynamics

We want to emphasize that the dynamics of subsystem X are not a mere conjecture or an arbitrary component in this study. Rather, it is an integral part of training a generative PPM. These dynamics are inherent in the optimizer action, necessitating a fresh set of model-generated samples to compute the loss function or its gradients after each parameter update. For instance, in the context of EBM, a Langevin Monte Carlo (LMC) sampler can be employed to generate new samples from the model [30]. The computational cost of producing a fresh set of model-generated samples introduces a time delay in the parameter dynamics. For instance, when using an LMC sampler, the number of Monte Carlo steps dictates this lag time. Conversely, in the case of a language model, since the computation of the loss function relies on inferring subsequent tokens, the inference latency signifies the time delay.

We denote the lag time parameters as $\tau$. Here, the model subsystem evolves on the timescale $\delta t$, while the parameter subsystem evolves on the timescale $\alpha =\tau \delta t$. In the thermodynamic context, this parameter represents the relaxation time of the subsystem X under the fixed microscopic state of the subsystem $\Theta$. Conceptually, parameter $\tau$ acts as a complexity metric, quantifying the computational resources required for each parameter optimization step. Moreover, the dynamics of the joint $(X,\Theta )$ exhibit a bipartite property. This implies that simultaneous transitions in the states of X and $\Theta$ are not allowed, given that the observation of a new set of model-generated samples occurs only after a parameter update.

The lagged bipartite dynamics described above can be represented using two time resolutions: $\delta t$ and $\alpha$. In the finer time resolution of $\delta t$, the Markov chain within the time interval $[t_i,t_{i+1}]$ is as follows:

$$\begin{array}{c}\hfill (x_{t_i},{\theta }_{t_i})\to (x_{t_i},{\theta }_{t_{i+1}})\to (x_{t_i+\delta t},{\theta }_{t_{i+1}})\cdots \to (x_{t_i+\tau \delta t},{\theta }_{t_{i+1}})\equiv (x_{t_{i+1}},{\theta }_{t_{i+1}}).\end{array}$$

(11)

We can also analyze this dynamics at a coarser time resolution of $\alpha$. Within the interval $[t_0,t_n]$, the Markov Chain appears as

$$\begin{array}{c}\hfill (x_0,{\theta }_0)⤏(x_{t_1},{\theta }_{t_1})\cdots (x_{t_{n-1}},{\theta }_{t_{n-1}})⤏(x_{t_n},{\theta }_{t_n}).\end{array}$$

(12)

In the above Markov chain, the dashed arrows remind us of the ignorance of the intermediate steps in the high resolution picture (Equation (11)). Figure 2 illustrates the lagged bipartite dynamics of the learning system. An important observation is that the learning trajectory $\mathcal{T}$, as defined in Equation (8), is written in the low-resolution picture. Therefore, studying the learning trajectory means studying the dynamics of the system $(X,\Theta )$ in the low- resolution picture.

3.4. Trajectory Probabilities

To set the stage for the application of the fluctuation theorem (FT) to learning a PPM, we define the trajectory probability of the joint $({\mathit{x}}_n,{\mathit{\theta }}_n)$ as the probability of observing the trajectory of model-generated samples and parameters during the learning process:

$$\begin{array}{c}\hfill P[{\mathit{x}}_n,{\mathit{\theta }}_n]:=p(x_0,x_{t_1},\cdots ,x_{t_n},{\theta }_0,{\theta }_{t_1},\cdots ,{\theta }_{t_n})\end{array}$$

(13)

Additionally, we can consider the time reversal of the model’s trajectory and parameters’ trajectory as ${\tilde{\mathit{x}}}_n:=\{x_{t_n},x_{t_{n-1}},\cdots ,x_{t_1}\}$ and ${\tilde{\mathit{\theta }}}_n:=\{{\theta }_{t_n},{\theta }_{t_{n-1}},\cdots ,{\theta }_{t_1}\}$, respectively. Then, the probability of observing the backward trajectory is denoted by $P[{\tilde{\mathit{x}}}_n,{\tilde{\mathit{\theta }}}_n]$.

Here, $P[{\mathit{x}}_n,{\mathit{\theta }}_n]$ represents the trajectory probability of the learning system in the ensemble view. In practice, however, we typically train our model only once, and we often lack access to the parameters’ distribution. Therefore, our model is conditioned on the observation of a specific parameters’ trajectory ${\mathit{\theta }}_n$. This defines the trajectory probability in the conditional view:

$$P\left[{\mathit{x}}_n\right|{\mathit{\theta }}_n]:=\frac{P[{\mathit{x}}_n,{\mathit{\theta }}_n]}{P\left[{\mathit{\theta }}_n\right]}$$

(14)

where

$$P\left[{\mathit{\theta }}_n\right]=p({\theta }_0,{\theta }_{t_1},\cdots ,{\theta }_{t_n}).$$

(15)

Similarly, the backward conditional trajectory probability is the probability of observing the time reversal model’s trajectory, conditioned on the observation of the time reversal parameters’ trajectory $P\left[{\tilde{\mathit{x}}}_n\right|{\tilde{\mathit{\theta }}}_n]=\frac{P[{\tilde{\mathit{x}}}_n,{\tilde{\mathit{\theta }}}_n]}{P\left[{\tilde{\mathit{\theta }}}_n\right]}$.

We now use the Markov property of the Markov chain in Equation (12) to decompose the conditional trajectory probability and the marginal trajectory probability, respectively, as follows:

$$\begin{array}{cc}\hfill P\left[{\mathit{x}}_n\right|{\mathit{\theta }}_n]& =p\left(x_{t_n}\right|x_{t_{n-1}},{\theta }_{t_n})\cdots p\left(x_{t_1}\right|x_0,{\theta }_{t_1})p\left(x_0\right|{\theta }_0),\hfill \\ \hfill P\left[{\mathit{\theta }}_n\right]& =p\left({\theta }_{t_n}\right|{\theta }_{t_{n-1}})\cdots p\left({\theta }_{t_1}\right|{\theta }_{t_0})p\left({\theta }_0\right),\hfill \end{array}$$

(16)

where expressions such as $p\left(x_{t_n}\right|x_{t_{n-1}},{\theta }_{t_n})$ and $p\left({\theta }_{t_n}\right|{\theta }_{t_{n-1}})$ represent the transition probabilities that determine the probability of moving from one microscopic state to another. Additionally, we define two probability trajectories conditioned on the initial conditions, which will be used later in the formulation of the FT:

$$\begin{array}{cc}\hfill P\left[\left({\mathit{x}}_n\right|{\mathit{\theta }}_n)\right|\left(x_0\right|{\theta }_0)]& :=P\left[\left({\mathit{x}}_n\right|{\mathit{\theta }}_n)\right]/p\left(x_0\right|{\theta }_0),\hfill \\ \hfill P\left[{\mathit{\theta }}_n\right|{\theta }_0]& :=P\left[{\mathit{\theta }}_n\right|{\theta }_0]/p\left({\theta }_0\right).\hfill \end{array}$$

(17)

3.5. Local Detailed Balance (LDB) for Learning PPMs

The transition probabilities, represented in Equation (16), capture the physics of the learning problem. When considering a Markov property (i.e., memoryless process) for time evolution of the model subsystem, the transition rate for this subsystem is reduced to the PPM:

$$p\left(x_{t_i}\right|x_{t_{i-1}},{\theta }_{t_i})=p\left(x_{t_i}\right|{\theta }_{t_i}).$$

(18)

The above expression suggests that the transition rate between two microscopic states $x_{t_{i-1}}$ and $x_{t_i}$ under the fixed ${\theta }_{t_i}$ is equivalent to the probability of observing $x_{t_i}$ by the PPM itself at $t=t_i$. To reiterate, this is the Markov property that suggests the elements inside ${\mathit{x}}_n$ are independently and freshly drawn from the PPM specified with given parameters along the learning trajectory $\mathcal{T}$. This is especially true when $\tau >>1$. We can generalize this observation for the backward transition probability $p\left(x_{t_{i-1}}\right|x_{t_i},{\theta }_{t_i})$, which represents the probability of the backward transition $(x_{t_i},{\theta }_{t_i})⤏(x_{t_{i-1}},{\theta }_{t_i})$ under a fixed ${\theta }_i$, as follows:

$$p\left(x_{t_{i-1}}\right|x_{t_i},{\theta }_{t_i})=p\left(x_{t_{i-1}}\right|{\theta }_{t_i}).$$

(19)

The above expression tells us that the probability of backward transition is equivalent to the probability of observing the sample generated at $t=t_{i-1}$ in ${\mathit{x}}_n$ with the PPM at time $t=t_i$.

Finally, we write the log ratio of the forward and backward transitions:

$$ln\frac{p\left(x_{t_i}\right|x_{t_{i-1}},{\theta }_{t_i})}{p\left(x_{t_{i-1}}\right|x_{t_i},{\theta }_{t_i})}=ln\frac{p\left(x_{t_i}\right|{\theta }_{t_i})}{p\left(x_{t_{i-1}}\right|{\theta }_{t_i})}=-\left({\varphi }_{{\theta }_{t_i}}\left(x_{t_i}\right)-{\varphi }_{{\theta }_{t_i}}\left(x_{t_{i-1}}\right)\right),$$

(20)

where the second equality is due to Equation (1). The above expression resembles the celebrated local detailed balance (LDB) [31], which relates the log ratio of the forward and backward transition probabilities to the difference in potential energy of the initial and final states in the transition. The heat reservoir that supports the legitimacy of the above LBD expression for the learning PPM is the parameter subsystem, whose temperature has been set to one, as we will discuss in more detail in Section 6. We emphasize that the above LBD emerged naturally under assumption of the Markov property and a relaxation time for learning a generic generative PPM. It is also important to note that the above LBD is only valid in the low-resolution picture.

The LBD relation, presented in Equation (20), has a profound consequence. This allows us to write the forward conditional probability trajectory $P\left[{\mathit{x}}_n\right|{\mathit{\theta }}_n]$ and the backward conditional probability trajectory $P\left[{\tilde{\mathit{x}}}_n\right|{\tilde{\mathit{\theta }}}_n]$ solely based on the series of PPMs in the learning trajectory $\mathcal{T}$:

$$\begin{array}{cc}\hfill P\left[{\mathit{x}}_n\right|{\mathit{\theta }}_n]& =p\left(x_{t_n}\right|{\theta }_{t_n})\cdots p\left(x_{t_1}\right|{\theta }_{t_1})p\left(x_0\right|{\theta }_0)\hfill \\ \hfill P\left[{\tilde{\mathit{x}}}_n\right|{\tilde{\theta }}_n]& =p\left(x_0\right|{\theta }_{t_1})\cdots p\left(x_{t_{n-1}}\right|{\theta }_{t_n})p\left(x_{t_n}\right|{\theta }_{t_n})\hfill \end{array}$$

(21)

This is significant because it renders the application of the FT framework to the learning PPMs practically, as we have access to elements of the learning trajectory during the learning process.

4. L-Info and Entropy Production

The version of the fluctuation theorem we are about to apply to the learning PPMs is known as the detailed fluctuation theorem (DFT) [32]. We also note that the machinery we are about to present for measuring the information flow in PPMs was developed to study the information exchange between thermodynamic subsystems [9]. The novelty here lies merely in the application of this machinery to the learning process of a PPM. In this section, we extensively use the notations presented in Table 1. Also, note that the temperature of the parametric reservoir is set to one. Applying the DFT in the conditional view (i.e., the conditional forward and backward trajectories defined in Equation (21)) results in

$$\begin{array}{cc}\hfill {\sigma }_{{\mathit{x}}_n|{\mathit{\theta }}_n}& =ln\frac{P\left[{\mathit{x}}_n\right|{\mathit{\theta }}_n]}{P\left[{\tilde{\mathit{x}}}_n\right|{\tilde{\mathit{\theta }}}_n]}\hfill \\ \hfill & =ln\frac{P\left[\left({\mathit{x}}_n\right|{\mathit{\theta }}_n)\right|\left(x_0\right|{\theta }_0)]}{P\left[\left({\tilde{\mathit{x}}}_n\right|{\tilde{\mathit{\theta }}}_n)\right|\left(x_{t_n}\right|{\theta }_{t_n})]}+ln\frac{p\left(x_0\right|{\theta }_0)}{p\left(x_{t_n}\right|{\theta }_{t_n})}\hfill \\ \hfill & =-q_{{\mathit{x}}_n}\left({\mathit{\theta }}_n\right)+s\left[p\left(x_{t_n}\right|{\theta }_{t_n})\right]-s\left[p\left(x_{t_0}\right|{\theta }_{t_0})\right]\hfill \end{array}$$

(22)

The first line is due to the DFT, which defines the stochastic Entropy Production (EP) to be the logarithm of the ratio of the forward and backward trajectory probabilities. The second line is due to the decomposition presented in Equation (17). Finally, the third line is the consequence of the LDB relation in Equation (20), and the definition of the stochastic heat flow $q_{{\mathit{x}}_n}\left({\mathit{\theta }}_n\right)$ is the change in the energy of the subsystem X due to alterations in its microscopic state configuration:

$$q_{{\mathit{x}}_n}\left({\mathit{\theta }}_n\right):=-ln\frac{P\left[\left({\mathit{x}}_n\right|{\mathit{\theta }}_n)\right|\left(x_0\right|{\theta }_0)]}{P\left[\left({\tilde{\mathit{x}}}_n\right|{\tilde{\mathit{\theta }}}_n)\right|\left(x_{t_n}\right|{\theta }_{t_n})]}=\sum _{i=1}^n{\varphi }_{{\theta }_{t_i}}\left(x_i\right)-{\varphi }_{{\theta }_{t_i}}\left(x_{i-1}\right).$$

(23)

Note that our convention defines $q_{{\mathit{x}}_n}>0$ as the heat absorbed by the subsystem X.

The second law arises from averaging Equation (22) over the forward trajectory distribution $P\left[{\mathit{x}}_n\right|{\mathit{\theta }}_n]$ and recalling the nonnegativity property of the Kl divergence to establish the nonnegativity of the averaged EP; ${\Sigma }_{X|\Theta }\left({\mathit{\theta }}_n\right):=<ln\frac{P_F\left[{\mathit{x}}_n\right|{\mathit{\theta }}_n]}{P_B\left[{\tilde{\mathit{x}}}_n\right|{\tilde{\mathit{\theta }}}_n]}{>}_{P_F\left[{\mathit{x}}_n\right|{\mathit{\theta }}_n]}\ge 0$. We note that this still a partially averaged EP, conditioned on the stochastic trajectory of the parameters. This is indeed the consequence of working in the conditional view.

Motivated to compute the L-info, in the next step, we rearrange Equation (22) as follows:

$$\mathcal{I}[x_{t_n}:{\theta }_{t_n}]-\mathcal{I}[x_0:{\theta }_0]=-q_{{\mathit{x}}_n}\left({\mathit{\theta }}_n\right)+s\left[p\left(x_{t_n}\right)\right]-s\left[p\left(x_{t_0}\right)\right]-{\sigma }_{{\mathit{x}}_n|{\mathit{\theta }}_n},$$

(24)

where $\mathcal{I}[x_{t_n}:{\theta }_{t_n}]:=s\left[p\left(x_{t_n}\right)\right]-s\left[p\left(x_{t_n}\right|{\theta }_{t_n})\right]$ is the mutual content (or stochastic mutual information) at $t=t_n$. We now arrive at the partially averaged L-info (Equation (10)) by averaging Equation (24) over $P\left[{\mathit{x}}_n\right|{\mathit{\theta }}_n]$:

$$\begin{array}{cc}\hfill I_{X;\Theta }\left({\theta }_{t_n}\right)-I_{X;\Theta }\left({\theta }_0\right)& =-Q_X\left({\mathit{\theta }}_{\mathit{n}}\right)+\left(S_X\left({\theta }_{t_n}\right)-S_X\left({\theta }_0\right)\right)-{\Sigma }_{X|\Theta }\left({\mathit{\theta }}_{\mathit{n}}\right)\hfill \\ \hfill & ={\Sigma }_X\left({\mathit{\theta }}_{\mathit{n}}\right)-{\Sigma }_{X|\Theta }\left({\mathit{\theta }}_{\mathit{n}}\right).\hfill \end{array}$$

(25)

This defines a series of Partially Averaged (PA) quantities:

$$\begin{array}{ccc}\hfill Q_X\left({\mathit{\theta }}_{\mathit{n}}\right)& :=\sum _{i=1}^n<{\varphi }_{{\theta }_{t_i}}\left(x\right){>}_{p\left(x\right|{\theta }_{t_i})}-<{\varphi }_{{\theta }_{t_i}}\left(x\right){>}_{p\left(x\right|{\theta }_{t_{i-1}})}\hfill & \hfill \left(\mathrm{PA\; Heat\; flow}\right)\\ \hfill S_{X|\Theta }\left({\theta }_{t_i}\right)& :=<-log\left(p\left(x\right|{\theta }_{t_i})\right){>}_{p\left(x\right|{\theta }_{t_i})}\hfill & \hfill \left(\mathrm{PA\; Conditinal\; Entropy}\right)\\ \hfill S_X\left({\theta }_{t_i}\right)& :=<-log\left(p\left(x\right)\right){>}_{p\left(x\right|{\theta }_{t_i})}\hfill & \hfill \left(\mathrm{PA\; Marginal\; Entropy}\right)\\ \hfill {\Sigma }_X\left({\mathit{\theta }}_{\mathit{n}}\right)& :=\left(S_X\left({\theta }_{t_n}\right)-S_X\left({\theta }_0\right)\right)-Q_X\left({\mathit{\theta }}_{\mathit{n}}\right)\hfill & \hfill \left(\mathrm{PA\; Marginal\; EP}\right)\end{array}$$

We note that all PA quantities are conditioned on the parameters’ trajectory (i.e., the choice of ${\mathit{\theta }}_n$ from the ensemble). This is a direct consequence of working in the conditional view. However, this also signifies that all thermodynamic quantities mentioned above are computable in the practice of machine learning, as they only require access to the time evolution of one PPM. Fortunately, thanks to the low-variance condition in Equation (9), we can use the partially averaged L-info as a proxy to the L-info.

Equation (25) equates the L-info to the difference between the marginal EP, and the conditional EP. We refer to this difference as the ignorance EP:

$${\Sigma }_{ign}\left({\mathit{\theta }}_{\mathit{n}}\right):={\Sigma }_X\left({\mathit{\theta }}_{\mathit{n}}\right)-{\Sigma }_{X|\Theta }\left({\mathit{\theta }}_{\mathit{n}}\right)$$

(26)

It is important to note that both the marginal EP and the conditional EP measure the EP of the same process, which is the time evolution of the subsystem X. However, the conditional EP measures this quantity with a lower time resolution of $\alpha$, which is conditioned on the parameter’s subsystem. On the other hand, the marginal EP measures this quantity with a higher time resolution of $\delta t$, including the relaxation time of the subsystem X between each parameter’s update. Therefore, the term “ignorance” refers to ignorance of the full dynamic of X, and the origin of the L-info is the EP between each consecutive parameter’s update (i.e., the EP of generating fresh samples represented with the Markov chain in Equation (11)).

5. M-Info and the Parameter Subystem

We can also apply the DFT to subsystem $\Theta$:

$$\begin{array}{cc}\hfill {\sigma }_{{\mathit{\theta }}_{\mathit{n}}}& =\log \frac{P\left[{\theta }_n\right]}{P\left[\widetilde{{\theta }_n}\right]}\hfill \\ \hfill & =-q_{{\mathit{\theta }}_{\mathit{n}}}+s\left[p\left({\theta }_{t_n}\right)\right]-s\left[p\left({\theta }_{t_0}\right)\right].\hfill \end{array}$$

(27)

In the above expression, the second line is due to the decomposition in Equation (17), and the definition of the stochastic heat flow for the parameter subsystem is $q_{{\mathit{\theta }}_{\mathit{n}}}:=\log P\left[{\theta }_n\right|{\theta }_0]/P\left[\widetilde{{\theta }_n}\right|{\theta }_{t_n}]$.

Under the assumption that the subsystem $\Theta$ evolves quasi-statically, the EP of this subsystem is zero, as expected for an ideal reservoir. This results in $q_{{\mathit{\theta }}_{\mathit{n}}}={\Delta }_{t_n}s\left[p\left({\theta }_t\right)\right]$. Furthermore, in the closed system of $(X,\Theta )$, the heat flow of the subsystem X must be provided with an inverse flow of the subsystem $\Theta$ (i.e., $q_{{\mathit{x}}_n}\left({\mathit{\theta }}_n\right)=-q_{{\mathit{\theta }}_{\mathit{n}}}$). Thus, we arrive at the stochastic version of the Clausius relation for the heat reservoir:

$${\Delta }_{t_n}s\left[p\left({\theta }_t\right)\right]=-q_{{\mathit{x}}_n}\left({\mathit{\theta }}_n\right)$$

(28)

This relation states that the heat dissipation in subsystem X ($q_{{\mathit{x}}_n}\left({\mathit{\theta }}_n\right)<0$) is compensated by an increase in the parameter subsystem entropy. We recall the definition of M-info from Equation (5) as the entropy of the subsystem $\Theta$. Since heat dissipation is a source of L-info accumulation (see Equation (25)), the above Clausius relation states that this information is stored in the parameters by increasing the entropy of this subsystem (i.e., the M-info), confirming the role of parameters as the memory space of the PPM.

We can also take the ensemble average of Equation (28) (i.e., averaging over $P[{\mathit{x}}_n,{\mathit{\theta }}_n]$):

$${\Delta }_{t_n}S\left[{\Theta }_t\right]=-Q_X\left(t_n\right),$$

(29)

where $Q_X\left(t_n\right):={\sum }_{{\mathit{x}}_n,{\mathit{\theta }}_n}P[{\mathit{x}}_n,{\mathit{\theta }}_n]q_{{\mathit{x}}_n}\left({\mathit{\theta }}_n\right)={\sum }_{{\mathit{\theta }}_n}P\left[{\mathit{\theta }}_n\right]Q_X\left({\mathit{\theta }}_n\right)$ is the fully averaged dissipated heat from the subsystem X. However, under the low-variance condition of learning (Equation (9)), we expect $Q_X\left({\mathit{\theta }}_n\right)$ to be independent of a specific parameters’ trajectory from the ensemble. Thus, we can write $Q_X\left(t_n\right)\approx Q_X\left({\mathit{\theta }}_n\right)$.

The Ideal Learning Process

The learning objective necessitates an increase in L-info to enhance the model’s performance while simultaneously reducing the M-info to minimize the generalization error and prevent overfitting. As previously mentioned in Section 2, the ideal scenario is achieved when all the stored information in the parameters (M-info) matches the task-oriented information learned by the model (L-info). Now that we have studied the machinery for computing these two information-theoretic quantities through the computation of entropy production, we can formally examine this optimal learning condition.

Maximizing the L-info, as described in Equation (25), is equivalent to maximizing the marginal EP while minimizing the conditional EP. Given that the conditional EP is always nonnegative, the “ideal” scenario would involve achieving a conditional EP of zero (i.e., ${\Sigma }_{X|\Theta }\left(t_n\right)=0$). This condition can be realized through a quasi-static time evolution of the PPM occurring on the lower-resolution timescale $\alpha$, presented in the Markov chain in Equation (12). In the context of generative models, this condition is akin to achieving perfect sampling. Under these circumstances, all EP of the subsystem X transforms into L-info, resulting in ${\Delta }_{t_n}{I}_{X;\Theta }\left({\theta }_t\right)={\Sigma }_X\left(t_n\right)$.

Thermodynamically, the condition of quasi-static time evolution of the PPM (and consequently the zero conditional EP) can be realized by having a large relaxation parameter $\tau \gg 1$, which allows the model to reach equilibrium after each optimization step. However, a high relaxation parameter comes at the cost of requiring more computational resources and longer computation times. This introduces a fundamental trade-off between the time required to run a learning process and its efficiency, a concept central to thermodynamics and reminiscent of the Carnot cycle, representing an ideal engine that requires an infinite operation time.

6. The Parameters’ Reservoir

In the formulation of the previous sections, we make the assumption that the subsystem $\Theta$ behaves as an ideal reservoir. In this section, we delve deeper into the premises of this assumption by studying the dynamics of the parameter subsystem. To facilitate our formulation, we adapt a negative log-likelihood as a fairly general form for the loss function,

$$\ell (b_t,\theta ):=-\frac{1}{|b_t|}\sum _{x\in b_t}\log \left(p_{\theta }\left(x\right)\right)={\varphi }_{\theta }\left(b_t\right).$$

(30)

Here, the loss function is computed according to the empirical average of a random mini-batch $b_t\in B$ drawn from the training dataset at time step t. The last equality is due to the PPM defined in Equation (1) and ${\varphi }_{\theta }\left(b_t\right):=\frac{1}{|b_t|}{\sum }_{x\in b_t}{\varphi }_{\theta }\left(x\right)$, where the notation $|·|$ shows the size of a set. We also use a vanilla stochastic gradient descent (SGD) optimizer with the learning rate r to take gradient steps iteratively for n steps in the direction of loss function minimization:

$${\theta }_{t+1}={\theta }_t-r{\nabla }_{\theta }{\varphi }_{\theta }\left(b_t\right){|}_{\theta ={\theta }_t}$$

(31)

To render the dynamics of parameters in the form of a conventional overdamped Langevin dynamic, we introduce the following conservative potential, defined by the entire training dataset B:

$$U_B\left(\theta \right):=\frac{1}{\left|B\right|}\sum _{x\in B}{\varphi }_{\theta }\left(x\right).$$

(32)

The negative gradient of this potential gives rise to a deterministic vector force. Additionally, we define the fluctuation term, which represents the source of random forces due to selection of a mini-batch at time step $t_n$:

$$\begin{array}{cc}\hfill \eta \left(t_n\right)& :=-{\nabla }_{\theta }{\varphi }_{{\theta }_t}\left(b_{t_n}\right)+{\nabla }_{\theta }{U}_B\left({\theta }_{t_n}\right).\hfill \end{array}$$

We now reformulate the SGD optimizer in Equation (31) in the guise of overdamped Langevin dynamics, dividing it by the parameters’ update timescale $\alpha$ to convert the learning protocol into a dynamic over time:

$$\begin{array}{c}\hfill \frac{{\theta }_{t_{n+1}}-{\theta }_{t_n}}{\alpha }=-\mu {\nabla }_{\theta }{U}_B\left({\theta }_{t_n}\right)+\mu \eta \left(t_n\right),\end{array}$$

(33)

where $\mu :=r/\alpha$ is known as the mobility constant in the context of Brownian motion.

We note that Equation (33) is merely a rearrangement of the standard SGD. For us to interpret it as a Langevin equation, the term $\eta \left(t_n\right)$ must represent a stationary stochastic process to serve as the noise term in the Langevin equation. To demonstrate this property of $\eta \left(t_n\right)$, we must examine the characteristic of its time correlation function (TCF) [33]: $C_{i,j}(t,t-t^{\prime }):={\delta }_{i,j}<{\eta }_i\left(t\right){\eta }_j\left(t^{\prime }\right)>$, where the indices $i,j$ represent different components of the vector $\theta$ and ${\delta }_{i,j}$ is the Kronecker delta.

If the fluctuation term $\eta$ satisfies the condition of the white noise (uncorrelated stationary random process), and assuming that Equation (33) describes a motion akin to Brownian motion, then we can apply the fluctuation-dissipation theorem to write

$$<{\eta }_i\left(t\right){\eta }_j\left(t^{\prime }\right)>=\frac{2k_{B}T}{\mu }\delta (t-t^{\prime }){\delta }_{i,j}$$

(34)

Here, $\delta (t-t^{\prime })$ is a delta Dirac, $k_B$ stands for the Boltzmann constant, and the constant T symbolizes the temperature. To render our framework as unitless, we treat the product of the Boltzmann factor and temperature as dimensionless. Moreover, regardless of the noise width, we set $T=1$, and henceforth it will not appear in our formulation. This is possible by adjusting the Boltzmann factor according to the noise width (i.e., $k_B=\mu <{\eta }_i\left(t\right){\eta }_i\left(t\right)>/2$).

We still need to investigate if the fluctuation term indeed describes an uncorrelated stationary random process, as presented in Equation (34). To this end, we conducted an experiment by training an ensemble of 50 models for the classification of the MNIST dataset. To induce different levels of stochastic behavior (i.e., different “temperatures”), we considered three different mini-batch sizes. A smaller mini-batch size leads to a bigger deviation in the fluctuation term, consequently amplifying the influence of random forces. The results are presented in Figure 3.

The plot in Figure 3c represents the TCF function at no time lag $t=t^{\prime }$ (i.e., a variance of $\eta \left(t\right)$) as a function of time. The constant value of variance suggests the stationary property of $\eta \left(t\right)$. Moreover, Figure 3d illustrates the autocorrelation of $\eta \left(t\right)$ at different time lags, indicating the white noise characteristic for this term.

However, it would be naive to draw a generic conclusion regarding the nature of the fluctuation term as an uncorrelated stationary random process solely based on a simple experiment. Indeed, research has demonstrated that the noise term can be influenced by the Hessian matrix of the loss function [34]. This observation aligns with our definition of the fluctuation term presented in Equation (33), where $\eta$ is defined in relation to the gradient of the loss itself. Consequently, as the optimizer explores the landscape of the loss function, the characteristics of the fluctuation term $\eta$ can vary. We can grasp this concept in the context of Brownian motion by envisioning a Brownian particle transitioning from one medium to another, each with distinct characteristics. This implies that there could be intervals during training where $\eta$ stays independent of the loss function and exhibits a stationary behavior.

Moreover, we overlooked the fact that $\eta \left(t\right)$ is also a function of $\theta$ itself. This could potentially jeopardize its stationary property. To address this issue, we refer to the slow dynamic (lazy dynamic) [27,28] of over-parameterized models under SGD optimization. This slow dynamic allows us to write the Taylor expansion (similar to what has been performed in neural tangent kernel theory [35], but with a different purpose) of the loss function around a microscopic state ${\theta }^{\ast }$, sampled from its current state $p_t\left(\theta \right)$:

$${\varphi }_{{\theta }_t}\left(b_t\right)={\varphi }_{{\theta }^{\ast }}\left(b_t\right)+({\theta }_t-{\theta }^{\ast }){\nabla }_{\theta }{\varphi }_{{\theta }^{\ast }}\left(b_t\right)$$

(35)

As a result, the gradient of the loss ${\nabla }_{\theta }{\varphi }_{{\theta }_t}\left(b_t\right)={\nabla }_{\theta }{\varphi }_{{\theta }^{\ast }}\left(b_t\right)$, signifying an independent behavior from the specific value of the parameter ${\theta }_t$ at a given time t. We can extend this concept to the deterministic force $-{\nabla }_{\theta }{U}_B\left({\theta }_t\right)=-{\nabla }_{\theta }{U}_B\left({\theta }^{\ast }\right)=F\left({\theta }^{\ast }\right)$, which indicates a conservative force in the lazy dynamics regime, denoted as $F\left({\theta }^{\ast }\right)$. This indicates a conservative force. The key point here is that the value of this force is independent of the exact microscopic state ${\theta }_t$, but rather on any typical sample ${\theta }^{\ast }$ from ${\Theta }_t$. In Appendix A, we illustrate how the condition of lazy dynamics leads to a thermodynamically reversible dynamic of the subsystem $\Theta$.

6.1. Naive Parametric Reservoir

The stationary state of the subsystem $\Theta$, under the dynamic of Equation (33) and satisfying the fluctuation-dissipation relation in Equation (34), corresponds to the thermal equilibrium state (the canonical state):

$$p^{eq}=e^{-U_B\left(\theta \right)+F_{\Theta }}$$

(36)

where $F_{\Theta }:=-\log (\int d\theta e^{-U_B\left(\theta \right)})$ is the free energy of the subsystem $\theta$. Recall that the temperature was set to one. This state also satisfies the detailed balance condition, which defines the log ratio between the forward and backward transition probabilities (presented in Equation (16)) as follows:

$$\log \frac{p\left({\theta }_{t_i}\right|{\theta }_{t_{i-1}})}{p\left({\theta }_{t_{i-1}|{\theta }_{t_i}}\right)}=-\left(U_B\left({\theta }_{t_i}\right)-U_B\left({\theta }_{t_{i-1}}\right)\right)$$

(37)

The standard plot of the loss function versus the optimization steps in machine learning practice can help us to visualize the dynamics of the subsystem $\Theta$. A rapid decline in the loss function signals a swift relaxation of the subsystem $\Theta$ to its equilibrium state. It is important to note that this self-equilibrating property is determined by the training dataset B through the definition of the potential function $U_B\left(\theta \right)$. These swift and self-equilibrating properties mirror the characteristics of a heat reservoir in thermodynamics [29].This observation supports the foundations on which we designated the subsystem $\Theta$ as the “parametric reservoir”. After a swift decline, a gradual reduction of the loss function can be a sign of a quasi-statistic process, where the subsystem $\Theta$ evolves from one equilibrium state to another. This can be due to the lazy dynamic condition, as discussed in Appendix A. Additionally, the requirement of a high heat capacity for the reservoir ($dim(\Theta )>>dim\left(X\right)$), offers a thermodynamic justification for the use of over-parameterized models in machine learning.

6.2. Realistic Parametric Reservoir

We refer to the assumption of the parametric reservoir with an equilibrium state expressed in Equation (36) as the “naive assumption” due to several issues that were previously sidestepped. The first issue stems from the assumption that all components of the parameter vector $\theta$ are subject to the same temperature (i.e., $<{\eta }_i\left(t\right){\eta }_i\left(t\right)>=\frac{2k_{B}T}{\mu }$ for all indexes i). In practice, we might find different values for the noise width, particularly with respect to different layers of a deep neural network. Furthermore, the weights or biases within a specific layer might experience different amounts of fluctuation. This scenario is entirely acceptable if we consider each group of parameters as a subsystem that contributes to the formation of the parametric reservoir. Consequently, each subsystem possesses different environmental temperatures and distinct stationary states. This observation may explain, in thermodynamic terms, why a deep neural network can offer a richer model. As it encompasses multiple heat reservoirs at varying temperatures, it presents a perfect paradigm for the emergence of non-equilibrium thermodynamic properties.

Second, the fluctuation term $\eta$ may exhibit an autocorrelation property that characterizes colored noise, as presented in [36]. While this introduces additional richness to the problem, potentially displaying non-Markovian properties, it does not impede us from deriving the equilibrium state, as demonstrated in [37].

We also overlooked the irregular behavior of the loss function, such as spikes or step-like patterns. These irregularities are considered abnormal as we typically expect the loss function to exhibit a monotonous decline, but in practice, such behaviors are quite common. These anomalies may be associated with a more intricate process experienced by the reservoir, such as a phase transition or a shock. Nevertheless, we can still uphold the basic parametric reservoir assumption during the time intervals between these irregular behaviors.

The mentioned issues are attributed to a richer and more complex dynamic of the subsystem $\Theta$ and do not fundamentally contradict the potential role of subsystem $\Theta$ as a reservoir. Examples of these richer dynamics can be found in a recent study [38], which shows the limitation of the Langevin formulation of the SGD, and Ref. [39], which investigates the exotic non-equilibrium characteristic of parameters’ dynamics under SGD optimization.

Before closing this section, it is worth mentioning that the experimental results presented in Figure 3 support the assumption of a low-variance condition for the stochastic dynamics of the subsystem $\Theta$. For instance, panel (a) shows that even in the high noise regime ($|b_t|=1$), the dynamics of the parameters remained confined to a small region across the ensemble. Furthermore, panel (b) demonstrates the low-variance characteristics of the model’s performance accuracy. Finally, the large magnitude of the deterministic force (dashed line in panel (c)) to the random force is evidence of low-variance dynamics.

7. Discussion

In this study, we delved into the thermodynamic aspects of machine learning algorithms. Our approach involved first formulating the learning problem as the time evolution of a PPM. Consequently, the learning process naturally emerged as a thermodynamic process. This process is driven by the work of the optimizer, which can be considered a thermodynamic work since parameter optimization constantly changes the system’s energy landscape through ${\varphi }_{\theta }\left(x\right)$. The optimizer action is fueled by the input trajectory and a series of samples drawn from the ground truth system. The work and heat exchange of the subsystem X can be computed practically along the learning trajectory $\mathcal{T}$ as outlined below:

$$\begin{array}{cc}\hfill W_X\left({\mathit{\theta }}_n\right)& =\sum _{t=1}^n<{\varphi }_{{\theta }_t}\left(x\right){>}_{p\left(x\right|{\theta }_{t-1})}-<{\varphi }_{{\theta }_{t-1}}\left(x\right){>}_{p\left(x\right|{\theta }_{t-1})}\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill Q_X\left({\mathit{\theta }}_n\right)& =\sum _{t=1}^n<{\varphi }_{{\theta }_t}\left(x\right){>}_{p\left(x\right|{\theta }_t)}-<{\varphi }_{{\theta }_t}\left(x\right){>}_{p\left(x\right|{\theta }_{t-1})}\hfill \end{array}$$

(39)

We use the term “practically” because when running a machine learning algorithm, we have access to the function ${\varphi }_{\theta }\left(x\right)$ at each training instance. We also note that these quantities are conditioned on a specific parameter’s trajectory as a result of working in the conditional view. Finally, the learning process can be summarized as follows: The model learns by dissipating heat, and the dissipated heat increases the entropy of the parameters, which act as the heat reservoir (a memory space) for learned information. This means the learning process must be irreversible, as this is the only way to increase the mutual information between the two subsystem X and $\Theta$ [10].

It is important to note that despite the wealth of research highlighting the significance of information content in parameters [21,22,25], calculating these quantities remains difficult due to the lack of access to the parameter distribution. In contrast, the thermodynamic approach computes the information-theoretic metrics indirectly as the heat and work of the process. Moreover, the mysterious success of over-parametrized models can be explained within the thermodynamic framework, where over-parameterization plays a crucial role in allowing the parameter subsystem to function as a heat reservoir.

At the same time, we are aware of the strong assumptions made during this study. Addressing each of these assumptions or providing justifications for them represents a direction for future research. For instance, we assumed slow dynamics for over-parameterized models using the SGD optimizer. This formed the basis for treating the parameters’ degrees of freedom as an ideal heat reservoir, evolving in a thermodynamically reversible manner. Breaking this assumption leads to entropy production of parameter subsystem, changing contribution of entropy production to accumulation of L-info.

We also sidestepped the role of changes in the marginal entropy of the model’s subsystem ${\Delta }_{t_n}{S}_X\left(t\right)$. This term can be estimated by computing the entropy of the empirical distribution of generated samples. For a model initialized randomly, this term is always negative, as the initial model produces uncorrelated patterns with maximum entropy. Then, the negative value of this term must converge when the entropy of the generated patterns reaches the entropy of the training dataset, or a representation of the training dataset that is most relevant to the learning objective. However, if we look at Equation (25) as an optimization objective to maximize the L-info, then an increase in the model’s generated samples $S_X\left(t\right)$ is favorable. This might act as a regularization term to improve the generalization power of the model by forcing it to avoid easy replication of the dataset.