Entropic Dynamics for Learning in Neural Networks and the Renormalization Group ^†

Abstract

We study the dynamics of information processing in the continuous depth limit of deep feed-forward Neural Networks (NN) and find that it can be described in language similar to the Renormalization Group (RG). The association of concepts to patterns by NN is analogous to the identification of the few variables that characterize the thermodynamic state obtained by the RG from microstates. We encode the information about the weights of a NN in a Maxent family of distributions. The location hyper-parameters represent the weights estimates. Bayesian learning of new examples determine new constraints on the generators of the family, yielding a new pdf and in the ensuing entropic dynamics of learning, hyper-parameters change along the gradient of the evidence. For a feed-forward architecture the evidence can be written recursively from the evidence up to the previous layer convoluted with an aggregation kernel. The continuum limit leads to a diffusion-like PDE analogous to Wilson’s RG but with an aggregation kernel that depends on the the weights of the NN, different from those that integrate out ultraviolet degrees of freedom. Approximations to the evidence can be obtained from solutions of the RG equation. Its derivatives with respect to the hyper-parameters, generate examples of Entropic Dynamics in Neural Networks Architectures (EDNNA) learning algorithms. For simple architectures, these algorithms can be shown to yield optimal generalization in student- teacher scenarios.

1. Introduction

Neural networks are information processing systems that learn from examples. Loosely inspired in biological neural systems, they have been used for several types of problems such as classification, regression, dimensional reduction and clustering [1]. Biological systems selection is based on a measure of performance that combines not only accuracy but also ease of computation and implementation. Predictions based on expectations over posterior Bayesian distributions may lead to saturating bounds for optimal accuracy learning but will typically lack in ease of computation and speed in reaching a result. Neural networks are parametric models and if we don’t address the determination of the architecture, which we don’t in this paper, the problem of learning from examples is reduced to obtaining fast estimates of the weights or parameters, avoiding the integration over large dimensional spaces. The spectacular explosion of applications in several areas is witness to the fact that several training methods and large data sets are available. Despite these victories, the mechanisms of information dynamics processing remain obscure and despite several decades of theoretical analysis using methods of Statistical Mechanics, much remains to be understood. Here we study on-line learning in feed-forward architectures, where (input,output) examples are presented one at a time. Theoretical analysis is easier than for batch or off-line learning where the cost function depends on a large number of example pairs, however on-line accuracy performance remains high. This is in part due to the fact that since the cost function changes from example to example, the local minima of the cost function that plague off-line learning are not so important. Local stationary points of the learning dynamics are still a problem, but good performances are possible. An important problem to be addressed is what cost function is the most appropriate. If an algorithm is going to be successful it has to approach Bayesian estimates for the available information. But any Bayes algorithm leads to high, even in the millions, dimensional integrals. Monte Carlo strategies cannot be used if simplicity is a requirement. The strategy to determine optimized algorithms for on-line learning has been studied in the past for restricted scenarios and architectures. We present a more general approach, with the following strategy. We are in a situation of incomplete information, thus a probability distribution represents, at a given point in the dynamics, what is known about the parameters. We have to commit to a family of distributions and we choose a Maxent family. Location hyperparameters give the current estimate of the weights. A new (input,output) example pair arrives and Bayes rule permits an update. The choice of the likelihood is a reflection of what we know about the architecture of the NN. In general it is not conjugated to the chosen family.

Still, the Bayes posterior, while not in the family, points to a unique member of the family, since it imposes new constraints on the expected values of the generators.

The resulting learning algorithm is the entropic dynamics imposed by the arrival of information in the examples that induces a change of the hyperparameters of the family. It turns out that changes in the weights are in the direction of decreasing the model Bayesian evidence and it is a stochastic gradient descent algorithm, where the cost function is the log evidence of the model.

The denominator of the Bayes update can be interpreted either as the evidence of the model or alternatively as the predictive probability distribution of the output conditioned on the input and the weights. Once it is written as the marginalization over the internal representation, i.e. the activation values of the internal units, of the joint distribution of activities of the whole network, and under the supposition that the information flows only from one layer to the next, a Markov chain structure follows. Recursion relations of the partial evidence up to a given internal layer are obtained and in the continuous depth limit (CDL) a Fokker-Planck parabolic partial differential equation is obtained. It generalizes Wilson’s Renormalization Group [2] diffusion equation for general kernels. The usual, e.g., majority rule that eliminates high frequency degrees of freedom are replaced by the weights of the NN. The RG dynamics can be seen as a classifier of Statistical Mechanics microstates into thermodynamics states. A NN extracts the relevant degrees of freedom that describe the macroscopic concept onto which an input pattern is to be assigned. The first authors to relate the RG and NN were [3] and [4] generating a large flow of ideas into the possible connections between these two areas [5,6,7].

2. Maxent Distributions and Bayesian Learning

Let $f_a\left(\mathit{w}\right)$, for $a=1,...K$, $\mathit{w}\in I{R}^N$, be the generators of a family $\mathcal{Q}$ of distributions $Q\left(\mathit{w}\right|\mathit{\lambda })$. If information about $\mathit{w}$ is given in the form of constraints $I{E}_Q\left(f_a\right)=F_a,$ for the set of numbers ${\left\{F_a\right\}}_{a=1,K}$, the Maxent distribution is

$$Q\left(\mathit{w}\right|\mathit{\lambda })=\frac{1}{z}exp\left(-\sum _{i=1}^K{\lambda }_a{f}_a\left(\mathit{w}\right)\right),$$

(1)

where z ensures normalization. Then

$$\frac{\partial lnz}{\partial {\lambda }_a}=-F_a\mathrm{and}\frac{\partial Q\left(\mathit{w}\right|\mathit{\lambda })}{\partial {\lambda }_a}=(-f_a+F_a)Q\left(\mathit{w}\right|\mathit{\lambda }).$$

(2)

Now consider a system learning a map from inputs x to outputs y, and the model is a known function which depends on a parameter array $\mathit{w}$: $y=T(x;\mathit{w})$. The aim of learning is to obtain the parameters from the information in the learning set ${\mathcal{D}}_n={\left\{(x_i,y_i)\right\}}_{i=1,n}$. We want to obtain a distribution for the parameters and consider that up to $n-1$ examples the information is coded in a member of the Q family: $Q\left(\mathit{w}\right|{\mathit{\lambda }}_{n-1})=Q_{n-1}$. Calling the likelihood of the problem $L_n=P\left(y_n\right|x_n,\mathit{w})$, the product rule permits the Bayesian updating

$$\begin{array}{ccc}\hfill P_n=P\left(\mathit{w}\right|{\mathcal{D}}_n)& =& \frac{Q_{n-1}{L}_n}{Z_n},\hfill \end{array}$$

(3)

where the partition function or the evidence is $Z\left(y_n\right|x_n,{\mathit{\lambda }}_{n-1})=\int Q_{n-1}{L}_{n}d\mathit{w}=P\left(y_n\right|x_n{\mathit{\lambda }}_{n-1})$. The Bayes posterior given by eq. 3 in general doesn’t belong to the $\mathcal{Q}$ family. We have to choose the member of the family that is closest to the Bayes posterior. This is the Maxent posterior. The way to proceed is based on the fact that a member of the $\mathcal{Q}$ family is determined solely by the values of the constraints $\left\{F_a\right\}$. The Bayes posterior defines a set of values for the constraints $\left\{\langle f_a\rangle \right\}$. It points in a unique way to the Maxent posterior $Q_n$ within the family $\mathcal{Q}$, obtained at the extreme of

$$S[Q_n\left|\right|Q_{n-1}]=-\int Q_{n}log\frac{Q_n}{Q_{n-1}}d\mathit{w}-\mathsf{\Delta }{\lambda }_a\left(I{E}_n\left(f_a\right)-\langle f_a\rangle \right),$$

(4)

subject to the only possible constraints on its expected values $I{E}_n\left(f_a\right)$ which are taken to be the Bayes posterior expected values $\langle f_a\rangle$. Then for every generator

$$\begin{array}{ccc}\hfill {IE}_{Q_n}\left(f_a\right)& =& \int \frac{Q_{n-1}{L}_n}{Z_n}{f}_a\left(\mathit{w}\right)d\mathit{w}={IE}_{P_n}\left(f_a\right)=F_a^n.\hfill \end{array}$$

(5)

Subtract from both sides $F_a^{n-1}$, and use equation 2, then

$$\begin{array}{ccc}\hfill F_a^n-F_a^{n-1}& =& -\frac{\partial lnZ}{\partial {\lambda }_a^{n-1}}\hfill \end{array}$$

(6)

since the likelihood is independent of the Lagrange multiplier. This learning dynamics is deduced from entropy maximization and thus will be called Entropic dynamics. Learning occurs along the gradient of the log evidence. It will turn out that the sign is such that typically the evidence for the new model is higher than before learning. These equations hold for any family, but it is interesting to consider the case that will be most likely to be useful in practice, where the family is determined by the functions $f_0=1$, $f_i=w_i$ and $f_{ij}=w_i{w}_j$ , for $i,j=1,N$. The constraints after n examples are the normalization, $IE\left(w_i\right)={\widehat{w}}_{ni}$ and $IE\left(w_i{w}_j\right)={\left({\mathit{C}}_n\right)}_{ij}+{\widehat{w}}_{ni}{\widehat{w}}_{nj}.$ The result is the gaussian family $Q\propto exp(-{\lambda }_0-{\sum }_i{\lambda }_i{w}_i-{\sum }_{ij}{\lambda }_{ij}{w}_i{w}_j)$. The entropic dynamics update equations, driven by the arrival of the $n^{th}$ example are

$$\begin{array}{ccc}\hfill {\widehat{\mathit{w}}}_n& =& {\widehat{\mathit{w}}}_{n-1}+{\mathit{C}}_{n-1}.{\nabla }_{{\widehat{\mathit{w}}}_{n-1}}{logZ}_n,\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill {\mathit{C}}_n& =& {\mathit{C}}_{n-1}+{\mathit{C}}_{n-1}.{\nabla }_{{\widehat{\mathit{w}}}_{n-1}}^{2}log{Z}_n.{\mathit{C}}_{n-1}.\hfill \end{array}$$

(8)

For a layered network, these are the equations associated to the update of the weights afferent to a particular unit in layer d from unit i in layer $d-1$ and of the component of the covariance matrix describing the correlation between weights coming from units i and j. The update equations, induced by a maximum entropy approximation to Bayesian learning is the learning algorithm of the neural network which implements the map $y=T(x;\widehat{\mathit{w}})$.

An approximation to this scheme was found for simple networks with no hidden units using a variational procedure ([8]) and applied to several architectures [9,10,11,12,13]. Then Opper [14] showed the Bayesian connection, explored elsewhere [15]. Recently it has been applied to societies of interacting neural networks [16,17,18,19]. While [12] attacked the neural network with a hidden layer, the challenge remains to study networks with deep architectures.

3. Deep Multilayer Perceptron

In this section we show that the evidence for a multilayer feedforward neural network can be written recursively as a map. Actually we will get two maps that are essentially the same. This type of map is typical of Renormalization Group transformations and in a continuous limit representation of the neural network as a field theory, we will show that the map leads to a partial differential equation analogous to Wilson’s diffusion-like RG equation.

We fix our attention at the $n^{th}$ example, and hence don’t write temporal (lower) indices anymore. A layer (upper) index now appears and $x^d$ is the internal representation at the the unit layer d. Layers start with $d=0$ and the depth of the network is D. Layer d weights are collectively denoted ${\mathit{w}}^d$ and individually $w_{ij}^d$ is the weight connecting unit i at layer $d-1$ to unit j at layer d. The data pair used for the learning step are $X_0$ and y. The distributions of the representation at the input is $\delta (x^0-X^0)$ and at the output $\delta (x^D-y)$. The partition function $Z\left(y_n\right|x_n,{\mathit{\lambda }}_{n-1})$ in Equation (3) is $Z\left(X^D\right|x^0,\mathit{\lambda })=\int Q\left(\mathit{w}\right|\mathit{\lambda })Ld\mathit{w}$, where $Q\left(\mathit{w}\right|\mathit{\lambda })$ is the prior joint distribution of the weights over all the layers. We will eventually take this to be a product over layers, $Q\left(\mathit{w}\right|\mathit{\lambda })={\prod }_{d=1}^{D-1}Q\left({\mathit{w}}^d\right|{\mathit{\lambda }}_d).$ which will permit a simpler analytical treatment, but it is not a necessity at this moment. To obtain the likelihood we marginalize the joint distribution of the internal representations $P(x^D,x^{D-1}....x^1|x^0,{\mathit{w}}^1,\dots {\mathit{w}}^D)$ over all internal representations at the hidden units doing the same trick that leads to the Chapman-Kolmogorov equation

$$\begin{array}{c}\hfill L=P\left(x^D\right|x^0=X^0,{\mathit{w}}^1,\dots {\mathit{w}}^D)=\int P(x^D,x^{D-1},\dots x^1|x^0=X^0,{\mathit{w}}^1,\dots {\mathit{w}}^D)\prod _{d=1}^{D-1}d{x}^d.\\ \hfill \end{array}$$

(9)

The evidence can be written as

$$Z_D\left(x^D\right|X^0,\mathit{\lambda })=\int Q^T(x^D,x^{D-1}....x^1|x^0=X^0,\mathit{\lambda })\prod _{d=1}^{D-1}d{x}^d.$$

(10)

where

$$\begin{array}{c}\hfill Q^T(x^D,x^{D-1}....x^1|x^0=X^0,\mathit{\lambda })=\int P(x^D,x^{D-1}\dots x^1|x^0=X^0,{\mathit{w}}^1,...{\mathit{w}}^D)\\ \hfill \times \prod _{d=1}^{D-1}Q\left({\mathit{w}}^d\right|{\mathit{\lambda }}^d)d{\mathit{w}}^d\end{array}$$

(11)

is the joint transition distribution. Define the partially integrated $Z_d$ for any $d=1....D$

$$Z_d(x^D,x^{D-1},....x^d|x^0,\mathit{\lambda })=\int Q^T(x^D,x^{D-1}....x^1|x^0=X^0,\mathit{\lambda })\prod _{d^{\prime }=1}^{d-1}d{x}^{d^{\prime }}.$$

(12)

It satisfies the recursion

$$Z_d=\int Z_{d-1}d{x}^{d-1}.$$

(13)

and the evidence is

$$Z_D=\int Z_d\prod _{d^{\prime }=d}^{D-1}d{x}^{d^{\prime }}$$

(14)

At this point this is analogous to a Statistical Mechanics (SM) or euclidean field theory (EFT) partition function in which all field configurations with momentum components above a cutoff have been integrated out. The equivalent of the effective action of the EFT, or the renormalized hamiltonian in the SM is $-log{Z}_d$.

Now we get a similar map, where the renormalization group transformation of the internal representations can be seen. Recall the likelihood in equation 9 and use the product rule

$$L=P\left(x^D\right|x^0,{\mathit{w}}^1,\dots {\mathit{w}}^D)=\int P\left(x^D\right|x^{D-1}{w}_D)P(x^{D-1}....x^1|x^0,{\mathit{w}}^1,\dots {\mathit{w}}^D)\prod _{d=1}^{D-1}d{x}^d$$

and finally

$$L=P\left(x^D\right|x^0,{\mathit{w}}^1,\dots {\mathit{w}}^D)=\int \prod _{d=1}^{D-1}P\left(x^{d+1}\right|x^d,{\mathit{w}}^{d+1})d{x}^d$$

Since the prior is also a product, then the partition function $Z_D=Z_D(x^D=y|x^0=X^0,\left\{{\lambda }^d\right\})$ is given by

$$Z_D=\int \prod _{d=1}^D{Q}_d\left({\mathit{w}}^d\right|{\lambda }^d)P\left(x^d\right|x^{d-1},{\mathit{w}}^d)\prod _{d=1}^{D}d{x}^{d-1}d{\mathit{w}}^d$$

(15)

We integrate over $x_0$ and $x^D$ with the constraints that their distribution are deltas at the input $X^0$ and output y.

$$Z_D=\prod _{d=1}^D\int d{\mathit{w}}^d\left[\int d{x}^{d-1}{Q}_d\left({\mathit{w}}^d\right|{\lambda }^d)P\left(x^d\right|x^{d-1},{\mathit{w}}^d)\right]$$

Define the evidence up to a given layer $\rho \left(x^d\right)$, with initial condition $\rho \left(x^0\right)=\delta (x^0-X^0)$ and the map

$$\rho \left(x^{d+1}\right)=\int \rho \left(x^d\right)P\left(x^{d+1}\right|x^d{\mathit{w}}^{d+1})Q_{d+1}\left({\mathit{w}}^{d+1}\right|{\lambda }^{d+1})d{x}^{d}d{\mathit{w}}^{d+1}$$

(16)

The last step for the map of a network of depth D is for $x^D=y$ leading to the evidence of the model defined by the architecture of the network with weight and hyperparameters given by the set of ${\lambda }_d$:

$$Z_D\left(y\right)=\rho \left(x^D\right)=\int \rho \left(x^{D-1}\right)P\left(x^D\right|x^{D-1}{\mathit{w}}^D)Q_D\left({\mathit{w}}^D\right|{\lambda }^D)d{x}^{D-1}d{\mathit{w}}^D$$

(17)

Define a layer to layer transition distribution

$$\begin{array}{ccc}\hfill Q_{d-1}^T\left(x^d\right|x^{d-1}{\lambda }^d)& =& \int P\left(x^d\right|x^{d-1},{\mathit{w}}^d)Q_d\left({\mathit{w}}^d\right|{\lambda }^d)d{\mathit{w}}^d\hfill \end{array}$$

(18)

$$\begin{array}{c}\hfill \end{array}$$

(19)

then, we have a map that gives the evidence after d layers as an integral over internal representations at layer $d-1$ of the evidence at layer $d-1$ with a kernel $Q^T$ that implements an aggregation RG-like step:

$$\begin{array}{ccc}\hfill \rho \left(x^d\right)& =& \int d{x}^{d-1}\rho \left(x^{d-1}\right)Q_{d-1}^T\left(x^d\right|x^{d-1},{\lambda }^d)\hfill \end{array}$$

(20)

We have obtained two RG-like maps, Equations (13) and (20). $Z_d$ depends on all internal representations from layer d to D and on all the hyperparameters $\mathit{\lambda }$. The simpler ${\rho }_d$ only depends on the internal representation at layer d and on the hyperparameters of the previous layers. The map for $Z_d$ is simpler and the map for ${\rho }_d$ requires, at each step the input on the transition distribution $Q^T\left(x^d\right|x^{d-1},{\lambda }^d)$. The transition distribution describes the renormalization group like transformation implemented by the neural network that takes the internal representation at one layer to the next. It is simple to see that

$$Z_d=\rho \left(x^d\right)\prod _{d^{\prime }\ge d}^D{Q}^T\left(x^{d^{\prime }+1}\right|x^{d^{\prime }}{\lambda }^d)$$

(21)

3.1. Generalized RG Differential Equation of a Neural Network in the Continuous Depth Limit

The layer index is obviously discrete, but we can take the continuous limit, where now layers are represented by a time like $\tau$ variable. A discrete variable i still labels the units. The evidence at depth $\tau$ is related to the evidence at depth ${\tau }_0$ by a generalization of Equation (20):

$$\begin{array}{ccc}\hfill \rho (\mathit{x},\tau )& =& \int Q^T\left(\mathit{x}\left(\tau \right)\right|{\mathit{x}}^{\prime }\left({\tau }_0\right),\lambda )\rho ({\mathit{x}}^{\prime },{\tau }_0)D{\mathit{x}}^{\prime },\hfill \end{array}$$

(22)

where the integration measure $D\mathit{x}={\prod }_{i}d{x}_i$. The distribution $Q^T\left(\mathit{x}\left(\tau \right)\right|{\mathit{x}}^{\prime }\left({\tau }_0\right),\lambda )$ is the probability, that a network with parameters $\lambda$, conditional on being in state ${\mathit{x}}^{\prime }$ at ${\tau }_0$ has an internal representation $\mathit{x}$ at depth $\tau$. It must satisfy the composition law

$$\begin{array}{ccc}\hfill Q^T\left(\mathit{x}(\tau +\mathsf{\Delta }\tau )\right|{\mathit{x}}^{\prime }\left({\tau }_0\right),\lambda )& =& \int Q^T\left(\mathit{x}(\tau +\mathsf{\Delta }\tau )\right|\mathit{z}\left(\tau \right),\lambda )Q^T\left(\mathit{z}\left(\tau \right)\right|{\mathit{x}}^{\prime }\left({\tau }_0\right),\lambda )D\mathit{z}\hfill \end{array}$$

For a deterministic neural network, conditional on the weights $\mathit{w}$, the evolution of the internal representation is given by the transfer function. To obtain a well behaved limit it is supposed to vary slowly:

$$x_i(\tau +\mathsf{\Delta }\tau )=T_i(\mathit{x}\left(\tau \right),\mathit{w})=x_i\left(\tau \right)+\mathsf{\Delta }\tau {\tilde{b}}_i(\mathit{x}\left(\tau \right),\mathit{w}),$$

(23)

so that interpretation of $\tilde{\mathit{b}}$ is the gradient of the transfer function. The transition distribution is

$$\begin{array}{ccc}\hfill Q^T\left(\mathit{x}\right|\tau ,{\mathit{x}}^{\prime },{\tau }_0,\lambda )& =& \int \prod _{{\tau }^{\prime }\in [{\tau }_0,\tau ]}\delta \left(\mathit{x}(\tau +\mathsf{\Delta }\tau )-T({\mathit{x}}^{\prime }\left(\tau \right),\mathit{w})\right)Q\left(\mathit{w}\right|\lambda ,\tau )d{\mathit{w}}_{{\tau }^{\prime }},\hfill \end{array}$$

(24)

obtained by integrating over all configuration of the weights in the slice. We have chosen a Gaussian family to represent the informational state of the network, which now takes the form of a product of Gaussians for all $\tau$ slices:

$$Q\left(\mathit{w}\right|\lambda ,\tau )\propto \prod _{\tau }exp-\frac{1}{2}\{\mathsf{\Delta }\mathit{w}·C_{\tau }^{-1}·\mathsf{\Delta }\mathit{w}\}$$

where $\mathsf{\Delta }\mathit{w}=\mathit{w}-{\widehat{\mathit{w}}}_{\tau }$ and $\lambda =\{{\widehat{\mathit{w}}}_{\tau },C_{\tau }\}$ for all values of $\tau$, but only the hyperparameters of the particular slice under consideration matters. To define the continuous limit we impose that the limits below exit:

$$\begin{array}{ccc}\hfill \underset{\mathsf{\Delta }\tau \downarrow 0}{lim}\frac{1}{\mathsf{\Delta }\tau }\int Q^T\left(\mathit{x}\right|\tau +\mathsf{\Delta }\tau ,{\mathit{x}}^{\prime },\tau ,\lambda )(\mathit{x}-{\mathit{x}}^{\prime })D\mathit{x}& =& \hfill \\ \hfill I{E}_{\mathit{w}}[\tilde{b}(\mathit{x}\left(\tau \right),\mathit{w}]=\mathit{b}({\mathit{x}}^{\prime },\tau ,\lambda ),\\ \hfill \underset{\mathsf{\Delta }\tau \downarrow 0}{lim}\frac{1}{\mathsf{\Delta }\tau }\int Q^T\left(\mathit{x}\right|\tau +\mathsf{\Delta }\tau ,{\mathit{x}}^{\prime },\tau ,\lambda )(x_i-x_i^{\prime })(x_j-x_j^{\prime })D\mathit{x}& =& \hfill \\ \hfill I{E}_{\mathit{w}}\left[{\tilde{b}}_i(\mathit{x}\left(\tau \right),w){\tilde{b}}_j(\mathit{x}\left(\tau \right),\mathit{w})\right]=B_{ij}({\mathit{x}}^{\prime },\tau ,\lambda ).\end{array}$$

(25)

At each layer the drift vector $\mathit{b}({\mathit{x}}^{\prime },\tau ,\lambda )$ is the expected value of the change in internal representation and the diffusion matrix $B_{ij}({\mathit{x}}^{\prime },\tau ,\lambda )$ to the expectation of quadratic change, which are related to the expected values of the gradient and Hessian of the transfer function respectively. As usual, take the time derivative of the expected value, with respect to $Q^T\left(\mathit{x}\right|{\mathit{x}}^{\prime },\lambda )$ of a well behaved test function $g\left(\mathit{x}\right)$. Taylor expand $g\left(\mathit{x}\right)$ around ${\mathit{x}}^{\prime }$ and integrate by parts, use that $g\left(\mathit{x}\right)$ is arbitrary and obtain that $Q^T$ satisfies a parabolic PDE and so does the evidence (see Equation (22))

$$\begin{array}{c}\hfill \frac{\partial \rho (\mathit{x},\tau )}{\partial \tau }=-\frac{\partial }{\partial x_i}\left(b_i(\mathit{x},\tau ,\lambda )\rho (\mathit{x},\tau )\right)+\frac{1}{2}\frac{{\partial }^2}{\partial x_i\partial x_j}\left(B_{ij}(\mathit{x},\tau ,\lambda )\rho (\mathit{x},\tau )\right).\end{array}$$

(26)

The long time limit of Equation (26) is the predictive distribution $\rho (\mathit{y},\tau =D)=P\left(\mathit{y}\right|{\mathit{x}}_0,\lambda )$. Equation (26) is a generalization of an analogous diffusion equation which appears in Wilson’s incomplete integration formulation of the renormalization group (e.g., [2]). It extends the type of transformation by permitting that the transformations that leads from τ to τ + dτ are not a simple spatial average, which would eliminate high spatial frequency components. Instead, the transformations are mediated by the weights $\widehat{w}$. It differs from the usual statistical mechanics or field theories also in the following sense. In those approaches, the transformation $\widehat{w}$ is known and uniform and the aim is to obtain the final ρ_D, which describes the infrared limit or the thermodynamics of the theory. In supervised learning in neural networks, the starting point, defined by the input X⁰ and the output Y are given. The problem is to find the correct set of weights $\widehat{w}$ that implements the correct input-output association. There are two regimes for the neural network. In the learning phase the set of examples is a set of microscopic-macroscopic variables that describe a task. The aim of learning is to determine the appropriate generalized RG transformation that maps from the microscopic description to the macroscopic. After learning, the network is used to find out, for the current RG transformation, the unknown macroscopic generalized thermodynamics or infrared properties associated to the microstate. The next step is to derive optimized learning algorithms, from the solutions of Equation (26) and the EDNNA learning described by (7) and (8).