Dataset-Learning Duality and Emergent Criticality

Abstract

In artificial neural networks, the activation dynamics of non-trainable variables are strongly coupled to the learning dynamics of trainable variables. During the activation pass, the boundary neurons (e.g., input neurons) are mapped to the bulk neurons (e.g., hidden neurons), and during the learning pass, both bulk and boundary neurons are mapped to changes in trainable variables (e.g., weights and biases). For example, in feedforward neural networks, forward propagation is the activation pass and backward propagation is the learning pass. We show that a composition of the two maps establishes a duality map between a subspace of non-trainable boundary variables (e.g., dataset) and a tangent subspace of trainable variables (i.e., learning). In general, the dataset-learning duality is a complex nonlinear map between high-dimensional spaces. We use duality to study the emergence of criticality, or the power-law distribution of fluctuations of the trainable variables, using a toy and large models at learning equilibrium. In particular, we show that criticality can emerge in the learning system even from the dataset in a non-critical state, and that the power-law distribution can be modified by changing either the activation function or the loss function.

1. Introduction

Dualities provide a highly useful technique for solving complex problems that have found applications in many branches of science, most notably in physics. For example, well-known dualities include electric–magnetic duality [1], wave–particle duality [2], target–space dualities [3], Kramers–Wannier duality [4,5], etc. More recent, but less well-known examples include a quantum–classical duality [6], dual path integral [7], and pseudo-forest duality [8], to name a few. The key idea of all physical dualities is to establish a mapping (i.e., a duality mapping) between two systems (e.g., physical theories), which can then be used to study properties (e.g., obtaining solutions) of one system by analyzing the other system, or vice versa.

Perhaps the most well-studied example of a physical duality is the so-called bulk-boundary or holographic duality [9,10], such as AdS–CFT [11,12,13,14]. In AdS–CFT, the mapping is established between the bulk, representing a gravitational theory on the anti-de Sitter (AdS) background, and the boundary, representing a conformal field theory (CFT) without gravity.

Mathematical dualities focus on more formal, abstract transformations preserving algebraic or geometric structures, but are also very useful in physics. For instance, dual vector space duality [15], duality of points and lines in projective geometry [16], Hom and Tensor Duality [17], etc.

In this article, we shall consider a learning duality, which we shall refer to as the dataset-learning duality. It is often convenient to view the dataset as representing the states of non-trainable variables on the ‘boundary’, and learning as representing the dynamics (or changes in states) of trainable variables in the ‘bulk’. Thus, the dataset-learning duality may be considered to be an example of a bulk-boundary duality in the context of learning theory. As we shall argue, the duality is a complex nonlinear mapping between very high-dimensional spaces, but near the equilibrium state, the analysis is greatly simplified. This simplification allows us to apply the dataset-learning duality to study the emergence of criticality in learning systems.

Criticality in physical systems refers to the behavior of these systems at or near critical points, where they undergo phase transitions. These critical points are characterized by dramatic changes in physical properties due to the collective behavior of the system’s components. Understanding phase transitions and criticality is essential for explaining many physical phenomena [18] as well as complex biological phenomena such as biological phase transitions [19]. Recent studies have shown that many physical [20,21,22] and biological [23,24] systems can be modeled as learning systems, such as artificial neural networks. Therefore, understanding the criticality and phase transitions in artificial neural networks may also shed light on the emergence of criticality in physical and biological systems.

The phenomenon of self-organized criticality was investigated using biologically inspired discrete network models, which adapt their topology based on local rules without any central control to achieve a balance between stability and flexibility. In papers [25,26,27,28,29,30], networks reach a critical state, where they exhibit power-law distributions of avalanche sizes and durations, indicative of self-organized criticality (SOC), where the criticality is described by a scale-invariant and power-law distribution of fluctuations.

As for the study of artificial neural networks that can be used to solve real applied problems, significant investigation has been carried out in the works [31,32]. There, the eigenvalue spectra of the weight matrices of a trained neural network were empirically examined, and it was found that well-generalizing models exhibit heavy-tailed (power-law) spectral distributions, suggesting that learning guides networks toward a form of self-organized criticality that acts as an implicit regularization. In another paper [33], the researchers carefully analyzed the learning dynamics of two-layer neural networks in the mean-field limit, showing that as the network width increases, the loss landscape becomes increasingly convex and the Hessian spectrum becomes concentrated, implying that wide networks self-organize into a near-critical regime characterized by flat minima and increased sensitivity to disturbances. At the same time, the authors of another paper [34] analytically demonstrated that even in randomly initialized neural networks, nonlinear activation functions can cause significant deviations from the classical spectra of random matrices, suggesting that architectural nonlinearity alone can initiate the emergence of structured, critical behavior in the neural network.

Our own study of criticality concerns the study of fluctuations in trainable variables and the possibility of their adoption by a power-law distribution in learning equilibrium. The first step in this direction was made in Ref. [35], where the criticality, or a power-law distribution of fluctuations, was derived analytically using a macroscopic, or thermodynamic, description developed in [36]. These results were then confirmed numerically [35] for a classification learning task involving handwritten digits [37]. In this paper, we take another step towards developing a theory of emergent criticality by providing a more microscopic description of the phenomena using the dataset-learning duality. We then test our predictions numerically for a classification task.

The paper is organized as follows. In Section 2, a theoretical model of neural networks and basic notations are introduced. Section 3 is devoted to developing a statistical description of a local dataset-learning duality. In Section 4, the distribution of the variables of the tangent space dual to the boundary (dataset) space is analyzed in the multidimensional case. In Section 5, a toy model with two trainable and two non-trainable variables is introduced, and in Section 6, it is solved for power-law fluctuations of the trainable variables. In Section 7, we present numerical results for some specific power-law distributions obtained for specific compositions of activation and loss functions for a toy and a realistic large model. In Section 8, we summarize and discuss the main results of the paper.

2. Neural Networks

Consider an artificial neural network defined as a neural septuple $(\mathbf{x},\widehat{P},p_{\partial },\widehat{w},\mathbf{b},\mathbf{f},H)$ [36], where:

• $\mathbf{x}\in {\mathbb{R}}^N$, is a vector state of N neurons,
• $\widehat{P}$, is a projection to subspace of $N_{\partial }=\mathrm{Tr}\left(\widehat{P}\right)$ boundary neurons,
• $p_{\partial }\left({\mathbf{x}}_{\partial }\right)$, is a probability distribution which describes the training dataset,
• $\widehat{w}\in {\mathbb{R}}^{N^2}$, is a weight matrix which describes connections between neurons,
• $\mathbf{b}\in {\mathbb{R}}^N$, is a bias vector which describes bias in inputs of individual neurons,
• $\mathbf{f}\left(\mathbf{y}\right)$, is an activation map which describes a nonlinear part of the dynamics where $\mathbf{y}=\widehat{w}\mathbf{x}+\mathbf{b}$,
• $H(\mathbf{x},\widehat{w},\mathbf{b})$, is a loss function of both trainable and non-trainable variables.

It is assumed that the bias vector $\mathbf{b}$ and weight matrix $\widehat{w}$ are the only trainable parameters which can be combined into a single trainable vector $\mathbf{q}$ via transformation tensors $W_{ij}^l$, and $B_i^l$, (Einstein summation convention over repeated indices is implied here and throughout the manuscript unless stated otherwise.)

$$\begin{array}{ccc}\mathrm{(1)}& \hfill w_{ij}& =& W_{ij}^l{q}_l,\hfill \hfill b_i& =& B_i^l{q}_l.\hfill \end{array}$$

In standard neural networks, including feedforward [38], convolutional [39], auto-encoder [40], transformers [41], etc., certain trainable variables can be shared, fixed, or set to zero, but all such architectures can be described using appropriate choices of constant tensors $W_{ij}^l$ and $B_i^l$. Then Equation (1) can be viewed as a linear map from a K-dimensional space of trainable variables to an $N^2+N$-dimensional space of weights and biases, where K can be much smaller than $N^2+N$.

The neural septuple $(\mathbf{x},\widehat{P},p_{\partial },\widehat{w},\mathbf{b},\mathbf{f},H)$ defines the three relevant types of dynamics and their relevant time-scales:

• Activation dynamics describes the updating of neurons on the smallest time-scale, which can be set to one, due to their connection with each other and the forward propagation of a signal from the boundary. Bulk neurons are usually subject to such dynamics,

  $${\mathbf{x}}_{\cancel{\partial }}(t+1)=(\widehat{I}-\widehat{P})\mathbf{x}(t+1)=(\widehat{I}-\widehat{P})\mathbf{f}(\widehat{w}\mathbf{x}\left(t\right)+\mathbf{b}),$$

  (2)

  where $\widehat{I}$ is the identity matrix. However, in the general case of a fully unconstrained neural network, all neurons can undergo activation dynamics.
• Boundary dynamics describes updates of boundary neurons ${\mathbf{x}}_{\partial }=\widehat{P}\mathbf{x}$ on the intermediate time-scales, e.g., once per L unit time steps, e.g., drawn from probability distribution $p_{\partial }\left({\mathbf{x}}_{\partial }\right)$ which describes the dataset.
• Learning dynamics describes changes in trainable variables $\mathbf{q}$ on the largest time-scale $ML$ where M is the so-called mini-batch size. For example, for the stochastic gradient descent method,

  $${\dot{q}}_i=q_i(t+ML)-q_i\left(t\right)=-\gamma {\delta }_{ij}{\displaystyle \frac{d{\langle H\rangle }_M}{d{q}_j}}$$

  (3)

  where ${\langle \dots \rangle }_M$ is the averaging over mini-batch and $\gamma$ is the learning rate. Please note that $ML$ is a unit on the scale of $\mathbf{q}$ changes; in further analytical reasoning, for simplicity, we set $M=1$. We emphasize that the method (3) implies that the space of trainable parameters is flat globally and coordinates $q_i$ are orthonormal, i.e., in the case under consideration $g_{ij}={\delta }_{ij}$, where ${\delta }_{ij}$ is the Kronecker delta.

For example, in the case of the feedforward neural network the weight matrix $\widehat{w}$ must be nilpotent and if its degree is L, i.e., ${\widehat{w}}^L=0$, then the deep neural network has L layers and the state of boundary neurons can be updated once every L time steps.

In what follows, we will be interested in the duality mapping from the boundary space to the tangent space of trainable variables. As we shall see, the map is a composition of the activation and learning passes.

3. Dataset-Learning Duality

The main objective of this section is to establish a duality mapping between the tangent space of trainable variables and the boundary subspace of non-trainable variables. For starters, we consider a large neural network, defined in Section 2, in a local learning equilibrium, i.e., when the mean value of the trainable parameters remains nearly constant and the evolution is dominated by stochastic fluctuations [36]. In other words, the subject of the study will be a high-dimensional, yet local, problem that allows for significant simplifications. In particular, this allows us to reduce the effective dimensionality K of the space of trainable variables to the dimensionality of the space of non-trainable variables N, or even to the dimensionality of the boundary subspace of non-trainable variables $N_{\partial }=\mathrm{Tr}\left(\widehat{P}\right)$. However, we will not consider possible symmetries in the dataset that could potentially reduce the dimensionality further.

The learning dynamics, described by Equation (3), can be viewed as a map from non-trainable degrees of freedom $\mathbf{x}$ to changes in trainable degrees of freedom $\dot{\mathbf{q}}$, i.e.,

$$(\mathbf{f},{\mathbf{x}}_{\cancel{\partial }}\left(t_{L-1}\right),\dots ,{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right),{\mathbf{x}}_{\partial }|\mathbf{q})\to \dot{\mathbf{q}},$$

(4)

where, after the vertical bar, there is a set of parameters of the mapping, which are fixed. Therefore, in Equation (4), and further vector $\mathbf{q}$ is the mean equilibrium value of the trainable variables vector. Similar notations containing a vertical bar cutting off fixed parameters of mapping are used further. Please note that vector $\mathbf{f}$ makes sense of bulk neuron values after the final L-th activation step, which will be used to form the prediction of the neural network. The loss function will explicitly depend only on these values of bulk neurons. In Equation (4) and further notation ${\mathbf{x}}_{\cancel{\partial }}\left(t_i\right)$ denotes the vector of bulk neurons after i steps of activation dynamics.

For example, if the loss function H of trainable and non-trainable variables is separable, i.e.,

$$H=H_x({\mathbf{x}}_{\partial },\mathbf{f})+H_q\left(\mathbf{q}\right),$$

(5)

then, the evolution of weights and biases is given by

$$\begin{array}{ccc}\hfill {\dot{b}}_j& =& -\gamma {\delta }_{ji}{\displaystyle \frac{\partial H_q}{\partial b_i}}-\gamma {\displaystyle \frac{\partial f_j\left(y_j\left(t_{L-1}\right)\right)}{\partial y_j\left(t_{L-1}\right)}}{\displaystyle \frac{\partial H_x}{\partial f_j}}+\dots ,\hfill \\ \mathrm{(6)}& \hfill {\dot{w}}_{jk}& =& -\gamma {\delta }_{ji}{\displaystyle \frac{\partial H_q}{\partial w_{im}}}{\delta }_{mk}-\gamma {\delta }_{kl}{x}^l\left(t_{L-1}\right){\displaystyle \frac{\partial f_j\left(y_j\left(t_{L-1}\right)\right)}{\partial y_j\left(t_{L-1}\right)}}{\displaystyle \frac{\partial H_x}{\partial f_j}}+\dots .\hfill \end{array}$$

where there is no summation over j, $y_j\left(t_{L-1}\right)=w_{ji}{x}^i\left(t_{L-1}\right)+b_j$ is the total argument of the activation function $f_j$. Please note that for simplicity of notation, $\mathbf{f}$ denotes both the argument of the loss function, formed by bulk neurons after L activation steps, and the function itself if the argument is specified in parentheses after it. Therefore, expressions (6) describe the first step in back-propagation algorithms, ellipses denote all other back-propagation steps.

The activation dynamics, in turn, makes it possible to establish a connection between the initial state of the vector of non-trainable variables $({\mathbf{x}}_{\partial },{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right))$ and its state at later time $({\mathbf{x}}_{\partial },{\mathbf{x}}_{\cancel{\partial }}\left(t_i\right))$, i.e.,

$$\begin{array}{cc}\hfill {\mathbf{x}}_{\cancel{\partial }}\left(t_1\right)& =\mathbf{f}\left(\mathbf{y}\left(t_0\right)\right),\hfill \\ \hfill {\mathbf{x}}_{\cancel{\partial }}\left(t_2\right)& =\mathbf{f}\left(\mathbf{y}\left(t_1\right)\right),\hfill \\ \hfill \dots \\ \hfill {\mathbf{x}}_{\cancel{\partial }}\left(t_L\right)& =\mathbf{f}\left(\mathbf{y}\left(t_{L-1}\right)\right)=\mathbf{f}\left(\mathbf{x}\left(t_0\right)\right|\mathbf{q}).\hfill \end{array}$$

(7)

Please note that during the activation pass, the input neurons ${\mathbf{x}}_{\partial }$ remain fixed and act as a source for the bulk neurons ${\mathbf{x}}_{\cancel{\partial }}\left(t_i\right)$, which change with time $t_i$. As a result, the vector of bulk neurons in any activation step ${\mathbf{x}}_{\cancel{\partial }}\left(t_i\right)$ can be expressed through the vector $\mathbf{x}\left(t_0\right)$ before the activation starts by recursively applying the activation function. The arguments in parentheses after $\mathbf{f}$ specify the functional form of the mapping; its specific form depends on the variables on which it depends explicitly, and they are specified as arguments. This notation is valid in Equation (7) and in similar cases throughout the paper.

Composition of the activation (7) and learning (4) maps is a map from non-trainable degrees of freedom $({\mathbf{x}}_{\partial },{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right))$ at time $t_0$ to changes in trainable degrees of freedom $\dot{\mathbf{q}}$ at time $t_L$, i.e.,

$$\begin{array}{c}\hfill ({\mathbf{x}}_{\partial },{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right)|\mathbf{q})\to \dot{\mathbf{q}}.\end{array}$$

(8)

For example, if the learning dynamics is described by stochastic gradient descent, then the map is given by

$$\begin{array}{ccc}\hfill {\dot{q}}_k({\mathbf{x}}_{\partial },{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right)|\mathbf{q})& =& -\gamma {\delta }_{km}\left({\displaystyle \frac{\partial f_j(\mathbf{x}\left(t_0\right),\mathbf{q})}{\partial q_m}}{\displaystyle \frac{\partial }{\partial f_j}}+{\displaystyle \frac{\partial }{\partial q_m}}\right)H({\mathbf{x}}_{\partial },\mathbf{f}|\mathbf{q}).\hfill \end{array}$$

(9)

which is a map from N-dimensional space of non-trainable variables to K-dimensional space of fluctuations of trainable variables. Therefore, the probability distribution $p_{\dot{q}}\left(\dot{\mathbf{q}}\right)$ can be expressed as

$$p_{\dot{q}}\left(\dot{\mathbf{q}}\right|\mathbf{q})=\int p_{x_{\partial }{x}_{\cancel{\partial }}}({\mathbf{x}}_{\partial },{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right)){\delta }^{\left(K\right)}\left(\dot{\mathbf{q}}-\dot{\mathbf{q}}({\mathbf{x}}_{\partial },{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right)|\mathbf{q})\right)d^N\mathbf{x}\left(t_0\right).$$

(10)

where the different subscripts are used to emphasize that these are different probability distribution functions, e.g., $p_{\dot{q}}\left(\right)$ and $p_{x_{\partial }{x}_{\cancel{\partial }}}\left(\right)$ (which is also apparent from the arguments of these functions). The vertical bar is used for conditional distribution, e.g., $p_{\dot{q}}\left(\dot{\mathbf{q}}\right|\mathbf{q})$ and for emphasizing the fixed parameterization of functions, e.g., $\dot{\mathbf{q}}({\mathbf{x}}_{\partial },{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right)|\mathbf{q})$. Also, ${\delta }^K$ denotes the K-dimensional Dirac delta function. Please note that we also abuse notation and denote variables, $\dot{\mathbf{q}}$, and function, $\dot{\mathbf{q}}({\mathbf{x}}_{\partial },{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right)|\mathbf{q})$, using the same symbols.

If the bulk neurons are initialized to zeros (or other constant values) at $t_0$ time moment before starting of activation for every dataset element, then $p_{x_{\partial }{x}_{\cancel{\partial }}}({\mathbf{x}}_{\partial },{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right))=p_{x_{\partial }}\left({\mathbf{x}}_{\partial }\right){\delta }^{\left(N_{\cancel{\partial }}\right)}\left({\mathbf{x}}_{\cancel{\partial }}\left(t_0\right)\right)$, and by integrating (10) over ${\mathbf{x}}_{\cancel{\partial }}\left(t_0\right)$ we obtain

$$p_{\dot{q}}\left(\dot{\mathbf{q}}\right|\mathbf{q})=\int p_{x_{\partial }}\left({\mathbf{x}}_{\partial }\right){\delta }^{\left(K\right)}\left(\dot{\mathbf{q}}-\dot{\mathbf{q}}\left({\mathbf{x}}_{\partial }\right|{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right),\mathbf{q})\right)d^{N_{\partial }}{\mathbf{x}}_{\partial }.$$

(11)

If $K>N_{\partial }$, then we should be able to perform a local coordinate transformation

$$q_i^{\prime }={\Lambda }_i^j{q}_j.$$

(12)

so that $K-N_{\partial }$ direction become constraints, i.e., ${\dot{q}}_i^{\prime }=0$ for $i=N_{\partial }+1,\dots ,K$. Please note that there is more than one way to do it.

The tangent vector ${\dot{\mathbf{q}}}^{\prime }$ can be projected onto $N_{\partial }$-dimensional dynamical subspace with a projection matrix $\widehat{R}$ of size $N_{\partial }\times K$,

$${\dot{q}}_i^{\prime }=R_i^r{\Lambda }_r^j{\dot{q}}_j$$

(13)

where all components are dynamical. We believe that the region, where this linear transformation of the trainable variables provides fluctuations along new $N_{\partial }$ coordinate axes in Euclidean space, is quite wide. Let us emphasize that although we are considering a problem local to $\mathbf{q}$, non-locality in the boundary space of neurons ${\mathbf{x}}_{\partial }$ in the general case leads us to the need to introduce curvilinear coordinate system ${\mathbf{q}}^{\prime }$ instead of (12) to satisfy the requirement of fluctuations only along the $N_{\partial }$ coordinate axes according to considering duality.

Writing down expression similar to Equation (11) in transformed variables and integrating it over $K-N_{\partial }$ constrained directions we obtain

$$p_{{\dot{q}}^{\prime }}\left({\dot{\mathbf{q}}}^{\prime }\right|{\mathbf{q}}^{\prime })=\int p_{x_{\partial }}\left({\mathbf{x}}_{\partial }\right){\delta }^{\left(N_{\partial }\right)}\left({\dot{\mathbf{q}}}^{\prime }-{\dot{\mathbf{q}}}^{\prime }\left({\mathbf{x}}_{\partial }\right|{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right),{\mathbf{q}}^{\prime })\right)d^{N_{\partial }}{\mathbf{x}}_{\partial }.$$

(14)

If the map ${\dot{\mathbf{q}}}^{\prime }\left({\mathbf{x}}_{\partial }\right|{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right),{\mathbf{q}}^{\prime })$ is invertible, then it can be considered to be a true duality, and then the probability distributions are related through Jacobian matrix

$$p_{{\dot{q}}^{\prime }}\left({\dot{\mathbf{q}}}^{\prime }\right)=p_{x_{\partial }}\left({\mathbf{x}}_{\partial }\left({\dot{\mathbf{q}}}^{\prime }\right)\right){\left|det\left({\displaystyle \frac{\partial {\dot{q}}_i^{\prime }}{\partial x_{\partial }^j}}\right)\right|}^{-1}.$$

(15)

We shall refer to this map as the dataset-learning duality. For non-invertible maps ${\dot{\mathbf{q}}}^{\prime }\left({\mathbf{x}}_{\partial }\right|{\mathbf{x}}_{\cancel{\partial }}\left(t_0\right),{\mathbf{q}}^{\prime })$ we can write

$$p_{{\dot{q}}^{\prime }}\left({\dot{\mathbf{q}}}^{\prime }\right)\mathbf{d}{\dot{\mathbf{q}}}^{\prime }=\sum p_{x_{\partial }}\left({\mathbf{x}}_{\partial }\right)\mathbf{d}{\mathbf{x}}_{\partial },$$

(16)

where there is summation over different ${\mathbf{x}}_{\partial }$ that are mapped to the same ${\dot{\mathbf{q}}}^{\prime }$. However, even in this more general case, the contribution from a single term in the summation might dominate (e.g., if $p_{x_{\partial }}\left({\mathbf{x}}_{\partial }\right)$ dominates for some ${\mathbf{x}}_{\partial }$), and then Equation (15) would still be approximately satisfied.

In summary, to achieve true local dataset-learning duality via the linear transformation (Equation (12)) of the trainable variables, the range over which this transformation can accurately identify trainable directions in the entire space $\mathbf{q}$ must be sufficiently large, and the Jacobian of the transition in Equation (15) must be invertible.

4. Distribution of Fluctuations

To obtain an expression for the Jacobian in Equation (15), the gradient descent Equation (9) must be rewritten in the transformed variables (12) and (13):

$$\begin{array}{ccc}\mathrm{(17)}& \hfill {\dot{q}}_i^{\prime }\left({\mathbf{x}}_{\partial }|\mathbf{q}\right)& =& -\gamma R_i^r\left({\Lambda }_r^k{\delta }_{km}{\Lambda }_l^m\right){\displaystyle \frac{dH({\mathbf{x}}_{\partial },\mathbf{f}|{\mathbf{q}}^{\prime })}{d{q}_m^{\prime }}}\hfill & =& -\gamma R_i^r{g}_{rl}\left({\displaystyle \frac{\partial f_j\left({\mathbf{x}}_{\partial }|{\mathbf{q}}^{\prime }\right)}{\partial q_l^{\prime }}}{\displaystyle \frac{\partial H_x}{\partial f_j}}+{\displaystyle \frac{\partial H_q}{\partial q_l^{\prime }}}\right),\hfill \end{array}$$

where $g_{rl}={\Lambda }_r^k{\Lambda }_{lk}={\left(\Lambda {\Lambda }^T\right)}_{rl}$. (Please note that if $\Lambda$ is an orthogonal matrix, i.e., the transition to the new trainable variables is carried out through a rotation transformation, then the metric in the new variable space would remain unchanged, i.e., $g_{rl}={\delta }_{rl}$.) The Jacobian matrix can be expressed as

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial {\dot{q}}_i^{\prime }}{\partial x_{\partial }^k}}=-\gamma R_i^r{g}_{rl}\left({\displaystyle \frac{{\partial }^2{f}_j\left({\mathbf{x}}_{\partial }|{\mathbf{q}}^{\prime }\right)}{\partial x_{\partial }^k\partial q_l^{\prime }}}{\displaystyle \frac{\partial }{\partial f_j}}+{\displaystyle \frac{\partial f_j\left({\mathbf{x}}_{\partial }|{\mathbf{q}}^{\prime }\right)}{\partial q_l^{\prime }}}{\displaystyle \frac{{\partial }^2}{\partial x_{\partial }^k\partial f_j}}\right)H_x({\mathbf{x}}_{\partial },\mathbf{f}).\end{array}$$

(18)

By substituting it back to (15), we obtain an expression for the probability distribution of fluctuations of trainable variables

$$p_{{\dot{q}}^{\prime }}\left({\dot{\mathbf{q}}}^{\prime }\right)={\gamma }^{-N}{p}_{x_{\partial }}\left({\mathbf{x}}_{\partial }\left({\dot{\mathbf{q}}}^{\prime }\right)\right){\left|det\left(R_i^r{g}_{rl}\left({\displaystyle \frac{{\partial }^2{f}_j\left({\mathbf{x}}_{\partial }|{\mathbf{q}}^{\prime }\right)}{\partial x_{\partial }^k\partial q_l^{\prime }}}{\displaystyle \frac{\partial }{\partial f_j}}+{\displaystyle \frac{\partial f_j\left({\mathbf{x}}_{\partial }|{\mathbf{q}}^{\prime }\right)}{\partial q_l^{\prime }}}{\displaystyle \frac{{\partial }^2}{\partial x_{\partial }^k\partial f_j}}\right)H_x({\mathbf{x}}_{\partial },\mathbf{f})\right)\right|}^{-1}$$

(19)

with three factors:

• $p_{x_{\partial }}\left({\mathbf{x}}_{\partial }\right)$—distribution of non-trainable input neurons
• ${\displaystyle \frac{\partial H_x}{\partial f_j}}$; ${\displaystyle \frac{{\partial }^2{H}_x}{\partial x_{\partial }^k\partial f_j}}$—Jacobian and Hessian of the loss function
• ${\displaystyle \frac{{\partial }^2{f}_j\left({\mathbf{x}}_{\partial }|{\mathbf{q}}^{\prime }\right)}{\partial x_{\partial }^k\partial q_l^{\prime }}}$; ${\displaystyle \frac{\partial f_j\left({\mathbf{x}}_{\partial }|{\mathbf{q}}^{\prime }\right)}{\partial q_l^{\prime }}}$—dependence of the neural network (result or prediction) $\mathbf{f}$ on the dataset/boundary variables ${\mathbf{x}}_{\partial }$ and trainable variables ${\mathbf{q}}^{\prime }$.

The first factor depends directly on the boundary dynamics, i.e., training dataset, the second factor depends on the learning dynamics, i.e., the loss function, and the third factor depends on the activation dynamics, i.e., activation function.

We recall that the transformation to primed variables (12), only $N_{\partial }$ of which fluctuate, can be carried out in different ways, i.e., the transformation matrix $\Lambda$ is not unique. In fact, we have at our disposal the entire subspace in which the nonzero vector ${\dot{\mathbf{q}}}^{\prime }$ lies, to introduce an arbitrary affine coordinate system. We can carry out linear transformations in this subspace, and they will not change the form of Equations (17)–(19). At the same time, we have no reason to prefer one coordinate system to another in this subspace for the requirement of power-law distributions along its axes. This freedom can be used to choose the transformed (or primed) trainable variables in which the probability distribution function approximately factorizes, i.e.,

$$p_{{\dot{q}}^{\prime }}\left({\dot{\mathbf{q}}}^{\prime }\right)\approx \prod _{i=1}^{N_{\partial }}{p}_{{\dot{q}}_i^{\prime }}\left({\dot{q}}_i^{\prime }\right).$$

(20)

Then the m-th statistical moment for some component of the original (or unprimed) variables over some range of scales is given by

$$\begin{array}{}\mathrm{(21)}& \hfill \langle {\dot{q}}_i^m\rangle =\int {\dot{q}}_i^m\left({\dot{\mathbf{q}}}^{\prime }\right)p_{\dot{\mathbf{q}}}\left(\dot{\mathbf{q}}\left({\dot{\mathbf{q}}}^{\prime }\right)\right)\left|{\displaystyle \frac{\partial \dot{\mathbf{q}}}{\partial {\dot{\mathbf{q}}}^{\prime }}}\right|d{\dot{\mathbf{q}}}^{\prime }& =\int {\left({\left({\Lambda }^{-1}\right)}_i^j{\dot{q}}_j^{\prime }\right)}^m{p}_{{\dot{\mathbf{q}}}^{\prime }}\left({\dot{\mathbf{q}}}^{\prime }\right)d{\dot{\mathbf{q}}}^{\prime }\approx \hfill & & \approx \int {\left({\left({\Lambda }^{-1}\right)}_i^j{\dot{q}}_j^{\prime }\right)}^m\prod _{i=1}^{N_{\partial }}{p}_{{\dot{q}}_i^{\prime }}\left({\dot{q}}_i^{\prime }\right)d{q}_i^{\prime }.\hfill \end{array}$$

where the relationship between primed and unprimed variables (12) was used.

We shall also assume that each component of fluctuation in the unprimed variables can be determined by only one single (possibly not the same for all) component in the primed variables, i.e.,

$${\dot{q}}_i\approx {\Lambda }_{ij}^{-1}{\dot{q}}_j^{\prime },$$

(22)

where summation over j is absent. Then continuing to calculate m-th statistical moment for ${\dot{q}}_i$ we obtain

$$\begin{array}{cc}\mathrm{(23)}& \hfill \langle {\dot{q}}_i^m\rangle & =\int {\left({\left({\Lambda }^{-1}\right)}_{ij}{\dot{q}}_j^{\prime }\right)}^m{A}_j{\dot{q}}_j^{\prime k_j}d{\dot{q}}_j^{\prime }=\hfill & =\int {\left({\left({\Lambda }^{-1}\right)}_{ij}{\dot{q}}_j^{\prime }\right)}^m{A}_j{\displaystyle \frac{{\left({\left({\Lambda }^{-1}\right)}_{ij}{\dot{q}}_j^{\prime }\right)}^{k_j}}{{\left({\Lambda }^{-1}\right)}_{ij}^{k_j+1}}}d\left({\left({\Lambda }^{-1}\right)}_{ij}{\dot{q}}_j^{\prime }\right)=\int {\dot{q}}_i^m{p}_{{\dot{q}}_i}\left({\dot{q}}_i\right)d{\dot{q}}_i,\hfill \end{array}$$

where the probability distribution of the corresponding primed variables is given by

$$p_{{\dot{q}}_j^{\prime }}\left({\dot{q}}_j^{\prime }\right)=A_j{\dot{q}}_j^{\prime k_j},{\dot{q}}_j^{\prime }\in [a,b].$$

(24)

Thus, a power-law distribution of fluctuations for the original trainable variable $q_i$ has the same power-law,

$$p_{{\dot{q}}_i}\left({\dot{q}}_i\right)={\displaystyle \frac{A_j{\dot{q}}_i^{k_j}}{{\left({\Lambda }^{-1}\right)}_{ij}^{k_j+1}}},{\dot{q}}_i\in \left[{\left({\Lambda }^{-1}\right)}_{ij}a,{\left({\Lambda }^{-1}\right)}_{ij}b\right],$$

(25)

where the change in normalization is associated with a change in the range over which the statistical moment in unprimed variables is calculated. Thus, assuming scale-invariance of the distribution in the transformed variables ${\dot{\mathbf{q}}}^{\prime }$, we concluded that, under the condition of distribution $p_{{\dot{q}}^{\prime }}\left({\dot{\mathbf{q}}}^{\prime }\right)$ factorization, its power-law form remains unchanged even for the directly trainable weights and biases, as was observed in the previous research [35].

To summarize this section, we have directly demonstrated that the distributions of fluctuations of the trainable variables depend on all three types of dynamics: boundary, activation, and learning. In the following sections, we will analyze the contribution to the distribution $p_{\dot{q}}\left(\dot{\mathbf{q}}\right)$ that comes directly from the Jacobian in Equation (19), based on the toy model. We will also discuss how the power-law distribution $p_{\dot{q}}\left(\dot{\mathbf{q}}\right)$ can be achieved by choosing certain compositions of the activation and loss functions. Furthermore, we will experimentally demonstrate that it is precisely the contribution from the Jacobian that is responsible for the emergence of criticality in fluctuations, both in the toy model and in large models, even when the input data distributions are Gaussian.

5. Toy Model

In the previous section, we considered a general multidimensional problem and applied the so-called dataset-learning duality to identify trainable variables in which the scale-invariance is expected. In this section, we will consider a simple example of a two-dimensional problem with both continuous and discrete degrees of freedom.

Consider a neural network consisting of only two neurons: input $x^1$ and output $x^2$ connected with a single trainable weight $w=w_{21}$ and a bias $b=b_2$. In addition, we assume that the output neuron can take only two possible values, i.e., its marginal distribution is a sum of two delta functions,

$$p_{x^2}\left(x^2\right)={\displaystyle \frac{1}{2}}\delta (x^2-X^{+})+{\displaystyle \frac{1}{2}}\delta (x^2-X^{-}).$$

(26)

In this case, we can reduce the two-dimensional problem (i.e., two trainable variables $q_1=w$ and $q_2=b$ and two non-trainable variables $x^1$ and $x^2$) to two one-dimensional ones, corresponding to two different values, $x^2=X^{+}$ and $x^2=X^{-}$. Then, for each one-dimensional problem, we can define a single trainable variable $q_1^{\prime }$ that is a linear function of $q_1$ and $q_2$, and along which fluctuations will occur. At the same time, there will be no fluctuations along the orthogonal direction $q_2^{\prime }$ according to dataset-learning duality. Then the transformation matrices in Equations (12) and (13) are given by

$$\begin{array}{ccc}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\Lambda }_{\pm }& =& \left(\begin{array}{cc}cos{\theta }_{\pm }& sin{\theta }_{\pm }\\ -sin{\theta }_{\pm }& cos{\theta }_{\pm }\end{array}\right),\hfill \end{array}$$

(27)

$$\begin{array}{ccc}\hfill R& =& \left(\begin{array}{cc}1& 0\end{array}\right),\hfill \end{array}$$

(28)

where the rotation angle ${\theta }_{\pm }$ corresponds to the state of the output neuron $x^2=X^{\pm }$.

Let us assume that the loss function H depends on the output of the neural network f after only a single step of the activation dynamics, but does not depend explicitly on ${\mathbf{q}}^{\prime }$, i.e.,

$$H=H_x\left(f\right|x^2),$$

(29)

and then Equation (17) for the one-dimensional case can be written as

$${\dot{q}}_1^{\prime }=-\gamma {\displaystyle \frac{\partial f\left(x^1\right|{\mathbf{q}}^{\prime })}{\partial q_1^{\prime }}}{\displaystyle \frac{\partial H}{\partial f}}.$$

(30)

If we rewrite the loss function as a function of the argument y of the activation function $f\left(y\right)$, i.e.,

$$y(x^1,{\mathbf{q}}^{\prime })=(x^{1}cos{\theta }_{\pm }+sin{\theta }_{\pm })q_1^{\prime }+(-x^{1}sin{\theta }_{\pm }+cos{\theta }_{\pm })q_2^{\prime }$$

(31)

then

$${\dot{q}}_1^{\prime }=-\gamma {\displaystyle \frac{\partial y}{\partial q_1^{\prime }}}{\displaystyle \frac{\partial H\left(y\right|x^2)}{\partial y}}=-\gamma (x^{1}cos{\theta }_{\pm }+sin{\theta }_{\pm }){\displaystyle \frac{\partial H\left(y\right|x^2)}{\partial y}}.$$

(32)

and the expression (18) is given by

$${\displaystyle \frac{\partial {\dot{q}}_1^{\prime }}{\partial x^1}}=-\gamma cos{\theta }_{\pm }{\displaystyle \frac{\partial H\left(y\right|x^2)}{\partial y}}.$$

(33)

Finally, we arrive at the expression for the probability distribution of fluctuations of the trainable variable in our toy model,

$$p_{{\dot{q}}_1^{\prime }}\left({\dot{q}}_1^{\prime }\right)={\displaystyle \frac{p_{x_1}\left(x_1\left({\dot{q}}_1^{\prime }\right)\right)}{\left|\gamma cos{\theta }_{\pm }\right|}}{\left|{\displaystyle \frac{\partial H\left(y\right|x_2)}{\partial y}}\right|}^{-1},$$

(34)

which is similar to the general expression (19).

6. Emergent Criticality

In this section, we shall utilize the dataset-learning duality (see Section 3) to investigate the potential emergence of criticality arising mainly from the Jacobian matrix (see Section 4) within the context of the toy model (see Section 5). The idea is to determine conditions under which the criticality might emerge in the toy model and then verify the results numerically for a toy-model classification problem. Specifically, our aim is to identify compositions of activation and loss functions that give us a power-law dependence of the Jacobian leading to a power-law distribution of fluctuations in the trainable variables.

From conservation of probability in terms of ${\dot{q}}^{\prime }$ (here and below we omit the subscript, so there is only one direction of fluctuations) and y, we obtain

$$p_{{\dot{q}}^{\prime }}\left({\dot{q}}^{\prime }\right)=p_y\left(y\left({\dot{q}}^{\prime }\right)\right)\left|{\displaystyle \frac{\partial y}{\partial {\dot{q}}^{\prime }}}\right|,$$

(35)

where the function $y\left({\dot{q}}^{\prime }\right)$ is assumed to be invertible. On one hand, the power-law dependence of the Jacobian, i.e.,

$$\left|{\displaystyle \frac{dy}{d{\dot{q}}^{\prime }}}\right|={\displaystyle \frac{1}{A|{\dot{q}}^{\prime }{|}^k}},A=A(w,b)>0,k\ge 0,$$

(36)

implies two possible differential equations

$$\begin{array}{c}\hfill {\dot{q}}^{\prime }=\left\{\begin{array}{cc}\mathrm{sign}\left({\dot{q}}^{\prime }\right)exp(Ay+B)\hfill & \mathrm{for}k=1\hfill \\ \mathrm{sign}\left({\dot{q}}^{\prime }\right){(Ay+B)}^{{\displaystyle \frac{1}{1-k}}}\hfill & \mathrm{for}k\ne 1\hfill \end{array}\right.\end{array}$$

(37)

for some new $A=A(w,b)$, $B=B(w,b)$, and where the form of expressions for fluctuations depends on their sign. On the other hand, the gradient descent Equation (32) expressed through y implies another differential equation

$${\dot{q}}^{\prime }=-(Dy-C){\displaystyle \frac{\partial H}{\partial q^{\prime }}},$$

(38)

where

$$\begin{array}{cc}\hfill C& ={\displaystyle \frac{\gamma cos\left({\theta }_{\pm }\right)}{w}}(b-wtan\left({\theta }_{\pm }\right)),\hfill \\ \hfill D& ={\displaystyle \frac{\gamma cos\left({\theta }_{\pm }\right)}{w}}.\hfill \end{array}$$

(39)

and for the gradient descent

$$Dy-C>0.$$

(40)

In this section, we shall study different compositions of loss and activation functions, i.e., $H\left(f\right(y\left)\right)$, for which Equations (37) and (38) are satisfied and thus the emergence of criticality is expected.

For $k=1$ Equations (37) and (38) can be combined together as

$${\displaystyle \frac{dH\left(f\right(y\left)\right)}{dy}}=-\mathrm{sign}\left({\dot{q}}^{\prime }\right){\displaystyle \frac{exp(Ay+B)}{Dy-C}},$$

(41)

By changing variable $z=Ay-AC/D$ in (41) and integrating in some region with respect to z we obtain

$$H\left(f\left(z\right)\right)=-\mathrm{sign}\left({\dot{q}}^{\prime }\right){\displaystyle \frac{1}{D}}exp\left({\displaystyle \frac{AC}{D}}+B\right){\int }_{z_0}^{z}d{z}^{\prime }{\displaystyle \frac{exp\left(z^{\prime }\right)}{z^{\prime }}},$$

(42)

or

$$H\left(f\left(y\right)\right)=-\mathrm{sign}\left({\dot{q}}^{\prime }\right){\displaystyle \frac{1}{D}}exp\left({\displaystyle \frac{AC}{D}}+B\right){\int }_{z_0}^{Ay-AC/D}d{z}^{\prime }{\displaystyle \frac{exp\left(z^{\prime }\right)}{z^{\prime }}},$$

(43)

One can show that for $\left|z\right|\gg 1$ the integral $\int d{z}^{\prime }{\displaystyle \frac{exp\left(z^{\prime }\right)}{z^{\prime }}}$ can be approximated by the integrand, i.e.,

$$H\left(f\left(y\right)\right)\approx -\mathrm{sign}\left({\dot{q}}^{\prime }\right){\displaystyle \frac{exp(Ay+B)}{A(Dy-C)}}.$$

(44)

To obtain this result, let us consider the following integral:

$$I\left(x\right)={\int }^x{\displaystyle \frac{exp\left(z\right)}{z}}dz.$$

(45)

In the limit of $x\gg 1$, we can integrate by parts (iteratively) to obtain

$$I\left(x\right)={\displaystyle \frac{exp\left(x\right)}{x}}+{\displaystyle \frac{exp\left(x\right)}{x}}\sum _{n=1}^{\infty }{\displaystyle \frac{n!}{x^n}}\approx {\displaystyle \frac{exp\left(x\right)}{x}}+{\displaystyle \frac{exp(x+1)}{x^2}}\approx {\displaystyle \frac{exp\left(x\right)}{x}},$$

(46)

where we took into account that ${\sum }_{n=1}^{\infty }{\displaystyle \frac{n!}{x^n}}\approx {\displaystyle \frac{e}{x\left(1-\frac{1}{x}\right)}}$.

In the opposite limit $y\equiv -x\gg 1$, we obtain

$$I\left(y\right)=-{\displaystyle \frac{exp(-y)}{y}}+{\displaystyle \frac{exp(-y)}{y}}\sum _{n=1}^{\infty }{\displaystyle \frac{{(-1)}^{n+1}n!}{y^n}}\approx -{\displaystyle \frac{exp(-y)}{y}}+{\displaystyle \frac{exp(-y-1)}{y^2}}\approx -{\displaystyle \frac{exp(-y)}{y}},$$

(47)

where we took into account that ${\sum }_{n=1}^{\infty }{\displaystyle \frac{{(-1)}^{n+1}n!}{y^n}}\approx {\displaystyle \frac{e^{-1}}{y\left(1+\frac{1}{y}\right)}}$.

By combining (46) and (47) we obtain

$$I\left(x\right)\approx {\displaystyle \frac{exp\left(x\right)}{x}}.$$

(48)

for $\left|x\right|\gg 1$.

An arbitrary constant coming from the lower limit of the integral must also be added to the expression. We omit it here and in the following cases for simplicity.

Please note that in the limit $\left|Dy\right|\ll \left|C\right|$, $\left|A\right|\gg |D/C|$ and $B=log\left|AC\right|$ we obtain an exponential function

$$\begin{array}{|c|}\hline H\left(f\right(y\left)\right)=exp\left(Ay\right)\\ \hline\end{array}.$$

(49)

For $k\in [0,1)$ Equations (37) and (38) can be combined together as

$$\begin{array}{ccc}\mathrm{(50)}& \hfill {\displaystyle \frac{dH\left(f\right(y\left)\right)}{dy}}& =& -\mathrm{sign}\left({\dot{q}}^{\prime }\right){\displaystyle \frac{{(Ay+B)}^{\alpha }}{Dy-C}}\hfill & =& -\mathrm{sign}\left({\dot{q}}^{\prime }\right){\displaystyle \frac{A}{D}}{\displaystyle \frac{{(Ay+B)}^{\alpha }}{(Ay+B)-\left(B+\frac{AC}{D}\right)}},\hfill \end{array}$$

where $\alpha \equiv 1/(1-k)>1$. If $|Ay+B|\gg \left|B+{\displaystyle \frac{AC}{D}}\right|$, then

$${\displaystyle \frac{dH\left(f\right(y\left)\right)}{dy}}=-\mathrm{sign}\left({\dot{q}}^{\prime }\right){\displaystyle \frac{A}{D}}{\displaystyle \frac{{(Ay+B)}^{\alpha }}{Ay+B}}=-\mathrm{sign}\left({\dot{q}}^{\prime }\right){\displaystyle \frac{A}{D}}{(Ay+B)}^{\alpha -1},$$

(51)

whose solution is given by

$$H\left(f\left(y\right)\right)=-\mathrm{sign}\left({\dot{q}}^{\prime }\right){\displaystyle \frac{A^{\alpha }}{\alpha D}}{\left(y+{\displaystyle \frac{B}{A}}\right)}^{\alpha }.$$

(52)

Please note that for $\left|{\displaystyle \frac{A^{\alpha }}{\alpha D}}\right|=1$, $\Delta ={\displaystyle \frac{B}{A}}$ we obtain the following power-law dependence

$$\begin{array}{|c|}\hline H\left(f\left(y\right)\right)={(y+\Delta )}^{\alpha }\\ \hline\end{array}.$$

(53)

In the opposite limit, i.e., $|Ay+B|\ll \left|B+{\displaystyle \frac{AC}{D}}\right|$, we obtain

$${\displaystyle \frac{dH\left(f\right(y\left)\right)}{dy}}=\mathrm{sign}\left({\dot{q}}^{\prime }\right){\displaystyle \frac{A}{D}}{\displaystyle \frac{{(Ay+B)}^{\alpha }}{B+\frac{AC}{D}}},$$

(54)

or

$$H\left(f\left(y\right)\right)=\mathrm{sign}\left({\dot{q}}^{\prime }\right){\displaystyle \frac{A^{\alpha +1}{\left(y+\frac{B}{A}\right)}^{\alpha +1}}{(\alpha +1)(AC+BD)}}.$$

(55)

For $\left|{\displaystyle \frac{A^{\alpha +1}}{(\alpha +1)(AC+BD)}}\right|=1$ and $\Delta ={\displaystyle \frac{B}{A}}$ the desired composition function takes the form

$$\begin{array}{|c|}\hline H\left(f\left(y\right)\right)={(y+\Delta )}^{\alpha +1}\\ \hline\end{array}.$$

(56)

For $k=2$ or $\alpha =1/(1-k)=-1$ Equations (37) and (38) can be combined together as

$${\displaystyle \frac{dH\left(f\right(y\left)\right)}{dy}}={\displaystyle \frac{\mathrm{sign}\left({\dot{q}}^{\prime }\right)}{(Ay+B)(C-Dy)}}={\displaystyle \frac{\mathrm{sign}\left({\dot{q}}^{\prime }\right)}{-AD{y}^2+(AC-BD)y+BC}},$$

(57)

If the quadratic term in the denominator is small, i.e., $\left|ADy\right|\ll |AC-BD|$, then upon integration we obtain

$$H\left(f\left(y\right)\right)=\mathrm{sign}\left({\dot{q}}^{\prime }\right){\displaystyle \frac{log\left|\right(AC-BD)y+BC|}{AC-BD}}.$$

(58)

For $|AC-BD|=1$, $\Delta =BC$ we obtain the following form of composition function

$$\begin{array}{|c|}\hline H\left(f\right(y\left)\right)=-log|\Delta -y|\\ \hline\end{array}.$$

(59)

As can be seen from the formulas given in the boxes, exponential, power-law and logarithmic composition of the activation and loss functions are possible, and all of them, when certain above-mentioned conditions are met, can lead to a power-law contribution from the Jacobian (see Equation (35)) to the distribution of fluctuations of the trainable variable $q^{\prime }$ under consideration.

7. Numerical Results

7.1. Toy Model

In this section, we shall present numerical results for the toy model described in Section 5, and compare them with analytical results obtained in Section 6. In particular, we are interested in studying the emergence of critical behavior, or when the distribution of changes in trainable rotated variable $q^{\prime }$ is described by a power-law, i.e.,

$$p\left({\dot{q}}^{\prime }\right)\propto {\left({\dot{q}}^{\prime }\right)}^{-k}.$$

(60)

The goal is to provide specific numerical examples of the critical behavior (35) on some ranges of scales for exponential (49), power-law (56), and logarithmic (59) compositions of activation and loss functions, i.e., for $H\left(f\right(y\left)\right)$.

For the numerical experiments, we consider a simple dataset with only two classes (called ‘−’ and ‘+’) with a single output neuron which takes discrete values ($x^2=X^{-}=0$ or $x^2=X^{+}=1$). The input subspace also consists of only a single neuron that takes continuous values drawn from two Gaussian probability distributions $p_{-}\left(x^1\right)$ (for $x^2=0$) and $p_{+}\left(x^1\right)$ (for $x^2=1$) with mean

$$\begin{array}{ccc}\mathrm{(61)}& \hfill {\langle x^1\rangle }_{-}& =& 0\hfill \hfill {\langle x^1\rangle }_{+}& =& 1\hfill \end{array}$$

where notations ${\langle x^1\rangle }_{-}$ and ${\langle x^1\rangle }_{+}$ mean averaging over the distributions of the values of the input neuron $x^1$ with conditions $x^2=X^{-}$ and $x^2=X^{+}$, respectively. The standard deviations are chosen as follows

$$\begin{array}{ccc}\mathrm{(62)}& \hfill \sqrt{{\langle {\left(x^1\right)}^2\rangle }_{-}-{\langle x^1\rangle }_{-}^2}& =& \sqrt{{\langle {\left(x^1\right)}^2\rangle }_{-}}=0.25\hfill \hfill \sqrt{{\langle {\left(x^1\right)}^2\rangle }_{+}-{\langle \left(x^1\right)\rangle }_{+}^2}& =& \sqrt{{\langle {\left(x^1\right)}^2\rangle }_{+}-1}=0.25.\hfill \end{array}$$

This simple dataset allows us to use the architecture of the toy model of Section 5 with one input neuron $x^1$, one output neuron $x^2$, one trainable bias b, and one trainable weight w (from input neuron to output neuron).

Please note that the main goal of this analysis is not to obtain high prediction accuracy (although it is above $97\%$ in all experiments), or to develop an architecture for a realistic classification problem (although a similar dataset would have been obtained if we considered a truncated MNIST dataset [37] with only images of ‘zeros’ and ‘ones’ and input states projected down to a single dimension). Rather, our aim is to demonstrate the emergence of criticality within a simple low-dimensional problem, but we expect the same mechanisms to be responsible for the emergence of criticality in higher-dimensional problems [35].

Also, note that in the learning equilibrium, the average value of the trainable parameters does not change significantly. Therefore, the distribution of the actual changes in trainable variables, i.e., $p\left(\dot{q}\right)$, and the distribution of potential jumps when the trainable parameters remain fixed, i.e., $p\left(-\gamma {\displaystyle \frac{\partial H}{\partial q}}\right)$, are essentially the same. However, the latter is more suitable, i.e., we will consider potential jumps caused separately by ‘−’ and ‘+’ classes, which, in the learning equilibrium, can be described as a two one-dimensional problem.

In the remainder of the section, we shall consider five experiments, two for exponential (49), two for power-law (56), and one for logarithmic (59) composition of activation and loss functions.

7.1.1. Exponential Composition, $k=1$

Consider the sigmoid activation function

$$f={(1+e^{-y})}^{-1}$$

(63)

with mean-squared loss

$$H(f,x^2)={(f-x^2)}^2$$

(64)

or cross-entropy loss

$$H(f,x^2)=-(1-x^2)log(1-f)-x^{2}log\left(f\right).$$

(65)

For composition of sigmoid activation (63) and mean-squared loss (64) functions and ‘−’ class, in the limit $y<0$ Equation (63) reduces to $f\approx e^y$ and then the composition function is

$$H\left(f\left(y\right)\right)\approx e^{2y}.$$

(66)

Likewise, for the same composition and ‘+’ class, in the limit $y>0$ Equation (63) reduces to $f\approx 1-e^{-y}$ and then the composition function is

$$H\left(f\left(y\right)\right)\approx e^{-2y}.$$

(67)

Below are the graphs showing the experimental dependencies used to compute the coefficients $A,B,C,D$ for both classes in this case. Knowing these coefficients, one can be confident that the conditions required to obtain the exponential composition (49) are satisfied. Moreover, it is clear from the graphs in Figure 1 that the mapping between y and ${\dot{q}}^{\prime }$ is invertible for chosen points, i.e., duality between these variables is observed. At the same time, from graphs in Figure 2 it is clear that the linear transformation of variables $\left(\dot{\mathbf{q}}\right)\to {\dot{q}}^{\prime }$ provides a good approximation of the curve along which fluctuations actually occur. We also carried out a similar procedure to verify the relationships between the coefficients $A,B,C,D$ in subsequent experiments, selecting for analysis only those regions where true dataset-learning duality is realized.

In Figure 3, we plot in logarithmic axes the distributions of fluctuations of $q^{\prime }$ (blue dots) and the contribution from the Jacobian (red dots) to these distributions corresponding to Equation (35). The linear behavior with slope $-1$ (or $k=1$) is in good agreement with Equation (49).

For composition of sigmoid activation (63) and cross-entropy loss (65) function we obtain, for ‘−’ class and in the limit $y<0$,

$$H\left(f\left(y\right)\right)=-log(1-f\left(y\right))\approx e^y,$$

(68)

and for ‘+’ class and in the limit $y>0$,

$$H\left(f\left(y\right)\right)=log\left(f\left(y\right)\right)\approx e^{-y}.$$

(69)

In Figure 4, we plot the distribution of fluctuations of $q^{\prime }$, which is once again in agreement with Equation (49). Please note that in this case, despite a nearly perfect power-law contribution from the Jacobian (red dots), the distribution of fluctuations as a whole (blue dots) is significantly distorted by the contribution from $p_y\left(y\left({\dot{q}}^{\prime }\right)\right)$ in Equation (35).

We conclude that for the toy-model classification problem with a sigmoid activation function and either mean-squared error or cross-entropy loss functions, the fluctuations of trainable variables can follow a power law with $k=1$, as confirmed both numerically and analytically.

7.1.2. Power-Law Composition, $k=0;{\displaystyle \frac{2}{3}}$

Consider a composition of a ReLU activation function

$$f\left(y\right)=max(0,y)$$

(70)

and the power-law loss function

$$H(f,x^2)={(f-x^2)}^n.$$

(71)

In this case, non-vanishing fluctuations can take place only when the argument of ReLU is positive, i.e.,

$$y>0,$$

(72)

then, the composition function is also a power law

$$H\left(f\left(y\right)\right)={(y-x^2)}^n.$$

(73)

For $n=2$, i.e., the standard mean-squared loss, Equation (56) implies that

$$k={\displaystyle \frac{\alpha -1}{\alpha }}={\displaystyle \frac{n-2}{n-1}}=0,$$

(74)

which is in agreement with the numerical results plotted in Figure 5.

For $n=4$, Equation (56) implies that $k={\displaystyle \frac{2}{3}}$ in agreement with the numerical results plotted in Figure 6.

Thus, we have demonstrated that the power-law composition case (56) is achieved by combining power-law loss and ReLU activation functions. More generally, we can obtain any $k={\displaystyle \frac{n-2}{n-1}}$, where n is the power that appears in (73).

7.1.3. Logarithmic Composition, $k=2$

Consider the cross-entropy loss function (65) and a piece-wise linear activation function

$$f\left(y\right)=\left\{\begin{array}{c}y,0<y<1,\hfill \\ 0,y<0\mathrm{or}y>1.\hfill \end{array}\right.$$

(75)

Within the range $0<y<1$, the composition of activation and loss function is

$$H\left(f\left(y\right)\right)=-(1-x_2)log(1-y)-x_{2}log\left(y\right).$$

(76)

which is the logarithm composition of Equation (59) for ‘−’ class (i.e., $x^2=0$) with $|\Delta |=1$, and for ‘+’ class (i.e., $x^2=1$) with $\Delta =0$.

In Figure 7, we plot the distribution of fluctuations of $q^{\prime }$, for which $k=2$, in good agreement with the analytical results in (59).

7.2. Large Model

Next, we describe a procedure that reveals the power-law distribution of fluctuations in the trainable variables of a large neural network, even in the absence of a power-law distribution in the input data.

First, let us address the input data by excluding directions in the data space along which the values of non-trainable variables vary weakly. To achieve this, we compute the covariance matrix $C_{xx}$, perform its spectral decomposition $C_{xx}=SA{S}^T$, and switch to the basis of its eigenvectors, where the covariance matrix becomes diagonal: $C_{x^{\prime }{x}^{\prime }}=A$.

$$\begin{array}{cc}& C_{xx}=\mathbb{E}\left[{\mathbf{x}\mathbf{x}}^T\right]=SA{S}^T,\hfill \\ & {\mathbf{x}}^{\prime }=S^T\mathbf{x},\hfill \\ & C_{x^{\prime }{x}^{\prime }}=\mathbb{E}\left[{{\mathbf{x}}^{\prime }{\mathbf{x}}^{\prime }}^T\right]=\mathbb{E}\left[\left(S^T\mathbf{x}\right){\left(S^T\mathbf{x}\right)}^T\right]=S^T{C}_{xx}S=A.\hfill \end{array}$$

(77)

Now, we define the transformed vector as ${\mathbf{x}}^{\prime }=S_{cut}^T\mathbf{x}$, where the matrix $S_{cut}^T$ consists of the first n rows of $S^T$. In other words, the components of ${\mathbf{x}}^{\prime }$ are the projections of the original vector $\mathbf{x}$ onto the eigenvectors of the covariance matrix $C_{xx}$ corresponding to its largest n eigenvalues. Thus, we work in the subspace of the input data most relevant for learning. In this way, one can effectively estimate the dimensionality of the input data space, which in practical problems is often much smaller than the total number of input neurons.

Next, we calculate the cross-covariance matrix $C_{x^{\prime }\dot{q}}$ of the input data in the transformed variables ${\mathbf{x}}^{\prime }$ with fluctuations of the trainable variables $\dot{\mathbf{q}}$. After that, we perform SVD decomposition $C_{x^{\prime }\dot{q}}=U\Sigma V^T$ and move to the bases of the left and right singular vectors in the input space and space of trainable variable fluctuations, respectively. In these new variables cross-covariance matrix $C_{x^{″}{\dot{q}}^{\prime }}$ has nonzero elements only on the main diagonal.

$$\begin{array}{cc}& C_{x^{\prime }\dot{q}}=\mathbb{E}\left[{{\mathbf{x}}^{\prime }\dot{\mathbf{q}}}^T\right]=U\Sigma V^T,\hfill \\ & {\mathbf{x}}^{″}=U^T{\mathbf{x}}^{\prime },\hfill \\ & {\dot{\mathbf{q}}}^{\prime }=V^T\dot{\mathbf{q}},\hfill \\ & C_{x^{″}{\dot{q}}^{\prime }}=\mathbb{E}\left[{{\mathbf{x}}^{″}\dot{\mathbf{q}}}^{\prime T}\right]=\mathbb{E}\left[\left(U^T\mathbf{x}\right){\left(V^T{\dot{\mathbf{q}}}^{\prime }\right)}^T\right]=U^T{C}_{x^{\prime }\dot{q}}V=\Sigma .\hfill \end{array}$$

(78)

In such variables, the dataset-learning duality becomes explicit due to the absence of correlations between the input data ${\mathbf{x}}^{″}$ and the fluctuations of the trainable parameters ${\dot{q}}_i^{\prime }$ for $i>dim\left(\mathbf{x}\right)=n$, i.e., now making the transformation ${\mathbf{q}}^{\prime }=V_{cut}^T\mathbf{q}$, where $V_{cut}^T$ consists of the first n rows of matrix $V^T$, we conclude that the vector ${\mathbf{q}}^{\prime }$ of size n and will consider fluctuations along its components.

Note also that the input space ${\mathbf{x}}^{″}$ is actually mapped to a manifold that is not flat. However, using the cross-covariance matrix, we can obtain a linear approximation of this generally curved subspace in $\dot{\mathbf{q}}$ and gain an understanding of where the input subspace is locally mapped.

Next, we will consider what distribution we have along the direction in input space and what distribution is obtained along the direction in the trainable variables fluctuations space correlated with the chosen direction in ${\mathbf{x}}^{″}$. For this, we will select the components ${\left(x^{″}\right)}^1$ and ${\left({\dot{q}}^{\prime }\right)}_1$ along the directions of left and right singular vectors of the cross-covariance matrix in respective spaces, which correspond to the largest singular value of $C_{x^{\prime }\dot{q}}$.

7.2.1. UNN Model

Now let us consider a large fully unconstrained neural network (See Section 2), consisting of 100 input neurons ${\mathbf{x}}_{inp}$ (hereafter, we omit the subscript and denote them simply as $\mathbf{x}$) and 10 output neurons ${\mathbf{x}}_{out}$. The network is fully connected, meaning that every neuron is connected to every other neuron. Please note that there are no bulk (hidden) neurons; instead, both the input and output neurons are updated during the activation dynamics, as they are mutually connected and do not remain fixed. After all steps of activation dynamics (in our example, we used five steps), the values of the output neurons ${\mathbf{x}}_{out}$ are compared with the correct target values for training. The input data $\mathbf{x}$ consists of the 100 pixels with the greatest variance across the entire dataset. Thus, the neural network with this architecture is trained to classify images of handwritten digits. Next, the distributions of fluctuations in the local learning equilibrium are analyzed according to the procedure described above.

It can be seen in Figure 8 and Figure 9 that the distribution along ${\left(x^{″}\right)}^1$ has a form close to a Gaussian distribution, while the distribution of fluctuations along ${\left({\dot{q}}^{\prime }\right)}_1$ has an explicit power form in a wide range of scales. In Figure 8, the hyperbolic tangent was used as the activation function for all neurons, and the cross-entropy form of the loss function was chosen, while in Figure 9, the ReLU activation function was used everywhere, and the mean-squared loss function was applied.

7.2.2. CNN Model

Now, let us consider another example of a convolutional neural network, which consists of 2 convolutional layers and two fully connected layers. The total number of trainable variables is 10,000. This neural network, as in other examples, is used to classify the MNIST dataset. The values of all available pixels for each image of a digit, $N_{\partial }={28}^2$, are fed to the boundary neurons as input data.

Distributions of input data and fluctuations of the trainable variables in the local learning equilibrium for the CNN are shown in Figure 10 and Figure 11. The results are similar to those obtained for the UNN.

It is worth noting that we have already obtained such distribution powers $k\approx 1$ and $k\approx 0$ analytically and experimentally for the toy model. Interestingly, the experimental data suggest that the presence of criticality and the value of the distribution power do not depend on the size of the neural network. When using the same loss and activation functions in both small and large neural networks, we observed similar powers for the fluctuation distributions at learning equilibrium, namely $k\approx 1$ (Figure 4, Figure 8 and Figure 10) and $k\approx 0$ (Figure 5, Figure 9 and Figure 11). In all considered models, the power-law distributions of trainable variable jumps do not originate from the input data distributions; rather, their emergence is driven by specific forms of the activation and loss functions.

8. Discussion

In this paper, we accomplished two main tasks.

Firstly, we established a duality mapping between the space of boundary neurons (i.e., the dataset) and the tangent space of trainable variables (i.e., the learning), the so-called dataset-learning duality. Both spaces (the boundary and the tangent) are generally very high-dimensional, making the analysis of the duality very non-trivial. However, by considering the problem in a local learning equilibrium and under the assumption that the probability distribution function of the tangent space variables is factorizable, the multidimensional problem can be greatly simplified.

Secondly, we applied the dataset-learning duality to study the emergence of criticality in the learning systems. We show that the observed scale-invariance of fluctuations of the trainable variables (e.g., weights and biases) is caused by the emergence of criticality in the dual tangent space of trainable variables. In particular, we analyzed different compositions of activation and loss function, which can give rise to a power-law distribution of fluctuations of trainable variables on a wide range of scales for a toy model. We showed that the power-law, exponential, and logarithmic compositions of activation and loss functions can all give rise to criticality in the learning dynamics, even if the dataset is in a non-critical state. Main results of the study of criticality are summarized in the following table.

$\mathit{k}$	$\mathit{H}\left(\mathit{f}\right(\mathit{y}\left)\right)$	Examples
1	$exp\left(Ay\right)$	Cross-entropy or mean-squared loss functions and sigmoid activation function
${\displaystyle \frac{\alpha -1}{\alpha }}$	${(y+\Delta )}^{\alpha +1}$	Power-law loss and ReLU activation functions
2	$-log|\Delta -y|$	Cross-entropy loss and piece-wise linear activation functions

In addition, we conducted an experiment with a large model classifying digit images from the MNIST dataset. We found that by using the same activation and loss functions as in the toy model, the large neural network exhibits the same power-law behaviors. Thus, we have empirically demonstrated that the results obtained from the toy model generalize to real neural networks. The emergence of criticality and the specific exponents of the trainable variables’ fluctuation distributions depend not on the network size, but on the choice of certain loss and activation functions.

Besides its theoretical significance and potential relevance in modeling critical phenomena in physical and biological systems [18,19], the emergence of criticality is expected to play a central role in machine learning applications. Power-law distributions, in particular, enable trainable variables to explore a broader range of scales without facing exponential suppression. Consequently, criticality is presumed to prevent neural networks from becoming trapped in local minima, a highly desirable property for any learning system. However, the analysis of the learning efficiency and its relation to criticality must involve considerations of non-equilibrium systems, which are beyond the scope of the current paper. Nevertheless, we expect that the established dataset-learning duality can be developed further to shed light on non-equilibrium problems. We leave these and other related questions for future research.