Deep Learning and Mean-Field Games: A Stochastic Optimal Control Perspective

Abstract

We provide a rigorous mathematical formulation of Deep Learning (DL) methodologies through an in-depth analysis of the learning procedures characterizing Neural Network (NN) models within the theoretical frameworks of Stochastic Optimal Control (SOC) and Mean-Field Games (MFGs). In particular, we show how the supervised learning approach can be translated in terms of a (stochastic) mean-field optimal control problem by applying the Hamilton–Jacobi–Bellman (HJB) approach and the mean-field Pontryagin maximum principle. Our contribution sheds new light on a possible theoretical connection between mean-field problems and DL, melting heterogeneous approaches and reporting the state-of-the-art within such fields to show how the latter different perspectives can be indeed fruitfully unified.

1. Introduction

Controlled stochastic processes, which naturally arise in a plethora of heterogeneous fields, spanning, e.g., from mathematical finance to industry, can be solved in the setting of continuous time stochastic control theory. In particular, when we have to analyse complex dynamics produced by the mutual interaction of a large set of indistinguishable players, an efficient approach to infer knowledge about the resulting behaviour, typical for example of a neuronal ensemble, is provided by Mean-Field Game (MFG) methods, as described in [1]. MFG theory generalizes classical models of interacting particle systems characterizing statistical mechanics. Intuitively, each particle is replaced by rational agents whose dynamics are represented by a Stochastic Differential Equation (SDE). The term mean-field refers to the highly symmetric form of interaction: the dynamics and the objective of each particle depend on an empirical measure capturing the global behaviour of the population. The solution of an MFG is analogous to a Nash equilibrium for a non-cooperative game [2]. The key idea is that the population limit can be effectively approximated by statistical features of the system corresponding to the behaviour of a typical group of agents, in a Wasserstein space sense [3]. On the other hand, Deep Learning (DL) is frequently used in several Machine Learning (ML) based applications, spanning from image classification and speech recognition to predictive maintenance and clustering. Therefore, it has become essential to provide a strong mathematical formulation and to analyse both the setting and the associated algorithms [4,5]. Commonly, Neural Networks (NNs) are trained through the Stochastic Gradient Descent (SGD) method. It updates the trainable parameters using gradient information computed randomly via a back-propagation algorithm with the disadvantage of being slow in the first steps of training. An alternative consists of expressing the learning procedure of an NN as a dynamical system (see [6]), which can be then analysed as an optimal control problem [7].

The present paper is structured as follows. In Section 2, we introduce the fundamentals about the Wasserstein space, Stochastic Optimal Control (SOC) and MFGs. In Section 3, the link between NNs and MFGs is deeply analysed in order to reflect the probabilistic nature of the learning process. Conclusions and prospects are outlined in Section 4.

2. Problem Formulation and Preliminaries

2.1. Wasserstein Metrics

Many of the main results contained in this paper will be stated in terms of convergence in the distribution of random variables, vectors, processes and measures. We refer to [8,9,10] concerning the basics about the theory of the weak convergence of probability measures in metric spaces equipped with the natural Borel $\sigma$-algebra; see also [11,12] and the references therein for further details. According to [8] (pp. 7, 8), let us recall the following two definitions, useful to highlight the strict connection between measure theory and probability. Given a probability measure $\mu \in \mathcal{P}\left(\mathcal{X}\right)$ over the metric space $\mathcal{X}$ and a sequence $\left({\mu }_n\right)\subset \mathcal{P}\left(\mathcal{X}\right)$ with $n\in \mathbb{N}$, we say that ${\mu }_n$ converges weakly to $\mu$ or ${\mu }_n\to \mu$ if:

$$\underset{n\to \infty }{lim}{\int }_{\mathcal{X}}fd{\mu }_n={\int }_{\mathcal{X}}fd\mu ,\forall f\in {\mathcal{C}}_b\left(\mathcal{X}\right).$$

Given a sequence of $\mathcal{X}$-valued random variables ${\left\{X_n\right\}}_{n\ge 1}$, we say that $X_n$ converges weakly (or in distribution) to a $\mathcal{X}$-valued random variable X, if:

$$\underset{n\to \infty }{lim}\mathbb{E}\left[f\left(X_n\right)\right]=\mathbb{E}\left[f\left(X\right)\right],\forall f\in {\mathcal{C}}_b\left(\mathcal{X}\right),$$

denoting this convergence by $X_n\Rightarrow X$. Focusing on the convergence of empirical measures, let $\left(X_i\right)$ be a sequence of independent and identically distributed (i.i.d.) $\mathcal{X}$-random variables and define the $\mathcal{P}\left(\mathcal{X}\right)$-valued random variable:

$${\mu }_n={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\delta }_{X_i},$$

(1)

then we have a random probability measure, usually indicated as the empirical measure. According to [8] (pp. 12–16), $\mathcal{P}\left(\mathcal{X}\right)$ is endowed with a metric (compatible with the notion of weak convergence) in order to consider $\mathcal{P}\left(\mathcal{X}\right)$ as a metric space itself. Let us recall the Wasserstein metric, defined on $\mathcal{P}\left(\mathcal{X}\right)$, based on the idea of coupling. In particular, given $\mu ,\nu \in \mathcal{P}\left(\mathcal{X}\right)$, $\Pi (\mu ,\nu )$ represents the set of Borel probability measures $\pi$ on $\mathcal{X}\times \mathcal{X}$, with first, resp. second, marginal $\mu$, resp. $\nu$. Namely, $\pi (A\times \mathcal{X})=\mu \left(A\right)$ and $\pi (\mathcal{X}\times A)=\nu \left(A\right)$ for every Borel set $A\subset \mathcal{X}$. Then, we define ${\mathcal{P}}^p\left(\mathcal{X}\right)$ for $p\ge 1$, as the set of probability measures $\mu \in \mathcal{P}\left(\mathcal{X}\right)$ satisfying:

$${\int }_{\mathcal{X}}d{(x,x_0)}^p\mu \left(dx\right)<\infty ,$$

where $x_0\in \mathcal{X}$ is an arbitrary reference point. Consequently, the p-Wasserstein metric on ${\mathcal{P}}^p\left(\mathcal{X}\right)$ is defined as:

$${\mathcal{W}}_p(\mu ,\nu )={\left(\underset{\pi \in \Pi (\mu ,\nu )}{inf}{\int }_{\mathcal{X}\times \mathcal{X}}d{(x,y)}^p\pi (dx,dy)\right)}^{1/p}=={\left(\underset{X\sim \mu ,Y\sim \nu }{inf}\mathbb{E}\left[d{(X,Y)}^p\right]\right)}^{1/p},$$

(2)

where the infimum is taken over all pairs of $\mathcal{X}$-valued random variables $X,Y$ with, respectively, given marginals $\mu$ and $\nu$.

2.2. Stochastic Optimal Control Problem

Following [13] (see also [14] and the references therein), let $\{\Omega ,\mathcal{F},\left({\mathcal{F}}_t\right),\mathbb{P}\}$ be a filtered probability space, with filtration ${\mathcal{F}}_t=\{{\mathcal{F}}_t,t\in [0,T]\}$, $T>0$, supporting:

• a controlled state variable ${\left(X_t^{\alpha }\right)}_{t\in [0,T]}$, where $X_t$ is an i.i.d. sequence of ${\mathbb{R}}^n$-valued ${\mathcal{F}}_0$-measurable random variables;
• a sequence ${\left\{W_t^i\right\}}_{i\ge 1}$ of independent and ${\mathcal{F}}_t$-adapted Brownian motions.

We consider a control as enforced on the state variable by $\alpha ={\left({\alpha }_t\right)}_{t\in [0,T]}$, that is an adapted process defined on $\mathcal{A}=\left\{{\left({\alpha }_t\right)}_{t\in [0,T]}\right\}$, containing $\mathcal{F}$-progressively measurable controls, which take values in A identified as a closed subset of ${\mathbb{R}}^d$. The objective function J is then defined as:

$$J(\alpha ,X)=\mathbb{E}\left[{\int }_0^{T}f({\alpha }_s,X_s)ds+g\left(X_T\right)\right],$$

where $f:A\times {\mathbb{R}}^n\to \mathbb{R}$ denotes the running objective function, while $g:{\mathbb{R}}^n\to \mathbb{R}$ denotes the terminal objective function, and the expectation is taken with respect to (w.r.t.) the probability measure $\mathbb{P}$.

The goal corresponds to identifying the control $\alpha$ solving the maximization problem:

$$\underset{\alpha \in \mathcal{A}}{max}J(\alpha ,X_t),$$

according to the n-dimensional stochastic controlled process $X_s$ s.t.

$$\left\{\begin{array}{c}d{X}_s=b({\alpha }_s,X_s)ds+\sigma ({\alpha }_s,X_s)d{W}_s\hfill \\ X_t=x,s\in [t,T].\hfill \end{array}\right.$$

(3)

The drift function $b:A\times {\mathbb{R}}^n\to {\mathbb{R}}^n$ and the volatility function $b:A\times {\mathbb{R}}^n\to {\mathbb{R}}^{d+n}$ are measurable, and they satisfy a uniform Lipschitz condition in x, i.e., there exists $K>0$ such that, for all $x,y\in {\mathbb{R}}^n$ and ${\alpha }_s\in A$, it holds that:

$$\left|b\right(x,\alpha )-b(y,\alpha \left)\right|+\left|\sigma \right(x,\alpha )-\sigma (y,\alpha |\le K|x-y|.$$

Previous assumptions guarantee that the SDE (3) has a unique solution, denoted by ${\left(X_s^{t,x}\right)}_{s\in [t,T]}$. Therefore, the objective function can be explicitly expressed in terms of x, t, namely:

$$J(\alpha ,x,t)=\mathbb{E}\left[{\int }_0^{T}f({\alpha }_s,X_s^{t,x})ds+g\left(X_T^{t,x}\right)\right],$$

with $(t,x)\in [0,T]\times {\mathbb{R}}^n$, and the process $\alpha \in \mathcal{A}$ takes values ${\alpha }_s\in A$. Let v be the value function:

$$v(t,x)=\underset{\alpha \in \mathcal{A}}{sup}J(t,x,\alpha ),$$

(4)

then the corresponding optimal control $\widehat{\alpha }:[0,T]\times {\mathbb{R}}^n\to \mathcal{A}$ is defined by:

$$J(t,x,\widehat{\alpha })=v(t,x),\forall (t,x),\in [0,T]\times {\mathbb{R}}^n,$$

whose solution can be found by exploiting two different and interconnected approaches respectively based on the Hamilton–Jacobi–Bellman (HJB) equation and on the stochastic Pontryagin Maximum Principle (PMP).

The first one moves from the Dynamic Programming Principle (DPP), then leading to the nonlinear second order Partial Differential Equation (PDE) known as the HJB equation (see [3]), namely:

$$\left\{\begin{array}{c}\frac{\partial v}{\partial t}(t,x)+\underset{a\in A}{sup}\left[{\mathcal{L}}^{\alpha }v(t,x)+f(x,a)\right]=0\hfill \\ v(T,x)=g\left(x\right),\hfill \end{array}\right.$$

(5)

that holds $\forall (x,t)\in \left[0,T\right]\times {\mathbb{R}}^n$, where $\mathcal{L}$ defines a second order operator called the infinitesimal generator of the controlled diffusion:

$${\mathcal{L}}^{a}v=b(x,a)·{\nabla }_{x}v\left(x\right)+\frac{1}{2}Tr\left[(\sigma (x,a)\sigma {(x,a)}^T{\nabla }_x^{2}v\left(x\right)\right].$$

It is possible to define the Hamiltonian for the SOC problem [15,16] as $H:{\mathbb{R}}^d\times {\mathbb{R}}^d\times {\mathbb{S}}_d\to \mathbb{R}\cup \{\infty \}$, written as follows:

$$H(x,y,z)=\underset{a\in A}{sup}\left(b(x,a)·y+{\displaystyle \frac{1}{2}}Tr\left[\sigma {\sigma }^T(x,a)z\right]+f(x,a)\right),$$

where ${\mathbb{S}}_d$ denotes the set of symmetric $d\times d$ matrices and the variables y and z are called the adjoint variables. The previously defined Hamiltonian allows encapsulating the nonlinearity of the HJB equation, which can be rearranged as:

$${\partial }_{t}v(t,x)+H(x,\nabla (t,x),{\nabla }^2(t,x))=0.$$

On the other hand, the stochastic PMP leads to a system of coupled Forward-Backward SDEs (FBSDEs) plus an external optimality condition in terms of the Hamiltonian function; see, e.g., [2,17] and the references therein. We define the local Hamiltonian as:

$${\mathcal{H}}_t(x,u,p,q)=f_t(x,u)+b_t(x,u)p+{\sigma }_t\left(x\right)q,$$

(6)

where $(p,q)$ are the adjoint variables.

By assuming that $f_t(x,u)$, $b_t(x,u)$ and $\sigma (x,u)$ are progressively measurable, bounded in ${\mathcal{C}}^1(x,u)$ and Lipschitz $\forall (x,u)\in {\mathbb{R}}^2$, we have that if an arbitrary control maximizes the Hamiltonian, then (necessary condition for optimality) it is the optimal one:

$${\mathcal{H}}_{\tau }({\widehat{x}}_{\tau },{\widehat{\nu }}_{\tau },{\widehat{P}}_{\tau },{\widehat{Q}}_{\tau })=\underset{\nu \in \mathbb{R}}{max}{\mathcal{H}}_t({\widehat{x}}_{\tau },\nu ,{\widehat{P}}_{\tau },{\widehat{Q}}_{\tau }).$$

Moreover, by requiring that $x\mapsto g\left(x\right)$ and $x\mapsto {\widehat{\mathcal{H}}}_t(x,{\widehat{P}}_{\tau },{\widehat{Q}}_{\tau })={sup}_{\nu }{\mathcal{H}}_{\tau }({\widehat{x}}_{\tau },\nu ,{\widehat{P}}_{\tau },{\widehat{Q}}_{\tau })$ are both concave functions, the PMP becomes also a sufficient condition to characterize the optimal control. In addition, by the envelope theorem, it follows that the optimal control $\widehat{\nu }$ maximizing $\mathcal{H}$ also maximizes its derivative ${\partial }_x\mathcal{H}$:

$${\partial }_x{\mathcal{H}}_{\tau }({\widehat{x}}_{\tau },{\widehat{\nu }}_{\tau },{\widehat{P}}_{\tau },{\widehat{Q}}_{\tau })={\partial }_x{\widehat{\mathcal{H}}}_t({\widehat{x}}_{\tau },{\widehat{P}}_{\tau },{\widehat{Q}}_{\tau }).$$

2.3. Mean-Field Games

In what follows, according to [1,3,5,18], we will exploit the theory of SOC to analyse Mean-Field Games (MFGs), to gain insights into the symmetric stochastic differential games characterized by a number of players that tends to infinity. In particular, consider n players $i=1,\dots ,n$ each of which controls a private state $X^i$ whose dynamics read as follows:

$$d{X}_t^i=\frac{1}{N}\sum _{j=1}^{N}b(t,X_t^i,X_t^j,{\alpha }_t^i)+\sigma d{W}_t^i,$$

(7)

$\sigma \in \mathbb{R}$ being a (common) constant multiplying a random noise component, i.e., $d{W}_t^i$, steered by the i-th copy of a standard Brownian motion.

Each control ${\alpha }_t^i$ represents the strategy played by the ith player, while $X_t^j$ models the influence of the state of the other j-players. The dynamics in Equation (7) can be reformulated as:

$$\frac{1}{N}\sum _{j=1}^{N}b(t,X_t^i,X_t^j,{\alpha }_t^i)=\int b(t,X_t^i,X_t^j,{\alpha }_t^i){\overline{\mu }}_t^{N}dx,$$

where the empirical distribution ${\overline{\mu }}_t^N$ of private states is defined by:

$${\overline{\mu }}_t^N=\frac{1}{N}\sum _{j=1}^N{\delta }_{X_t^j}.$$

Hence, the same dynamics for each player are obtained, since each player depends only on the global behaviour of the entire system and not on the state of the single player:

$$d{X}_t^i=\frac{1}{N}\sum _{j=1}^{N}b(t,X_t^i,{\overline{\mu }}_t^N,{\alpha }_t^i)+\sigma d{W}_t^i.$$

Each individual is minimizing a cost composed by a running cost and a terminal one. Therefore, the objective function $J^i\left(\alpha \right)$, $\forall i$, is defined by:

$$J^i\left({\alpha }^i\right)=\mathbb{E}\left[{\int }_0^{T}f(t,X_t^i,{\overline{\mu }}_t^N,{\alpha }_t^i)dt+g(X_T^i,{\overline{\mu }}_T^N)\right],$$

the goal being to select $ϵ$-Nash equilibria, the vector $({\alpha }^{1\star },\dots ,{\alpha }^{N\star })$ being a $\epsilon$-Nash equilibrium if:

$$\forall i=1,\cdots ,N\wedge \forall {\alpha }^{i}J\left({\alpha }^{\star }\right)-ϵ\le J({\alpha }^{\star -i},{\alpha }^i),$$

meaning that if the ith player is changing his/her behaviour for a small value $ϵ$, while other players follow a fixed strategy, there is no change in the profit of the game in terms of the cost J. Let us note that, if $ϵ=0$, we are dealing with the standard Nash equilibria.

In this scenario, the solution of an MFG (see [3]) corresponds to a couple $({\alpha }^{\star },{\mu }^{\star })$, where ${\mu }_t^{\star }=\mathcal{L}\left(X_t^{{\alpha }^{\star }}\right)$ models the optimal empirical distribution and ${\alpha }^{\star }={\varphi }^{\star }(t,X_t^i)$ the optimal strategies.

Due to the symmetry of the system, all the agents play the same strategy profile:

$${\varphi }^{1\star }(t,X_t^1)=\cdots ={\varphi }^{N\star }(t,X_t^N)={\varphi }^{\star }\left(t\right),\forall i=1,\cdots ,N;$$

hence, by fixing the flow of the probability measure ${\left({\mu }_t\right)}_{t\in \left[0,T\right]}$, the solution ${\varphi }^{\star }$ solves the SOC problem parametrized by the choice of the family $\left({\mu }_t\right)$ as:

$${\varphi }^{\star }=\underset{\varphi {\left(t\right)}_{t\in \left[0,T\right]}}{\mathrm{argmin}}\mathbb{E}\left[{\int }_0^{T}f(t,X_t,{\mu }_t,\varphi (t,X_t))dt+g(X_T,{\mu }_T)\right],$$

subject to:

$$d{X}_t=b(t,X_t,{\mu }_t,\varphi (t,X_t))dt+\sigma d{W}_t.$$

(8)

The solution of the latter optimization problem returns the best response of a representative player to the empirical measure $\left({\mu }_t\right)$ in a scenario where no player is profitable in changing his/her own strategy.

Concerning the choice of the flow of the probability measure ${\left({\mu }_t\right)}_{t\in \left[0,T\right]}$ (see Equation (1)), if $N\to \infty$, the asymptotic independence implied by the law of large numbers ensures that the empirical measure ${\overline{\mu }}_t^N$ tends to the statistical distribution of $X_t$ in the form of a fixed point equation, that is $\mathcal{L}\left(X^{{\alpha }^{\mu }}\right)=\mu$. Once the optimal feedback ${\varphi }^{\star }$ has been found, for each choice of the parameter ${\left({\mu }_t\right)}_{0\le t\le T}$, it is necessary to check that the optimal empirical measure ${\mu }_t^{*}$ can be recovered from the statistical distribution of the optimal states $X_t^{{\alpha }^{*}}$, i.e., $\forall t$, $\mathcal{L}\left(X_t^{{\alpha }^{\star }}\right)={\mu }_t^{\star }$ must hold.

By freezing the family $\mu$ of probability, the Hamiltonian becomes:

$$H^{{\mu }_t}(t,x,y,\alpha )=yb(t,x,{\mu }_t,\alpha )+f(t,x,{\mu }_t,\alpha ).$$

(9)

Let us assume that there exists a regular function:

$$\widehat{\alpha }(t,x,y):[0,T]\times \mathbb{R}\times \mathbb{R}\to \underset{\alpha \in A}{\mathrm{argmin}}{H}^{{\mu }_t}(t,x,y,\alpha ),$$

then we denote the infimum over controls $\alpha$ of $H^{{\mu }_t}(t,x,y,\alpha )$ by:

$${\mathcal{H}}^{{\mu }_t}(t,x,y):=\underset{\alpha \in A}{inf}{H}^{{\mu }_t}(t,x,y,\alpha ),$$

ending up with a stochastic control problem that can be solved (see, e.g., [3] (pp. 7–9)) by applying one of the following two methods:

• The PDE approach through HJB Equation (21) and the Kolmogorov equation;
• The Backward Stochastic Differential Equation (BSDE) approach based on the PMP.

We introduce the HJB value equation following [3], (p. 7), i.e.,:

$$v(t,x)=\underset{\alpha \in A_t}{inf}\mathbb{E}\left[{\int }_t^{T}f(s,X_s,{\mu }_s,{\alpha }_s)ds+g(X_T,{\mu }_T)|X_t=x\right],$$

(10)

where $A_t$ denotes the set of admissible controls over the interval $[t,T]$. It is expected that v is the solution, in the viscosity sense, of the HJB equation; see, e.g., [19,20] and the references therein. Hence:

$${\partial }_{t}v+{\displaystyle \frac{{\sigma }^2}{2}}{\partial }_{xx}^{2}v+{\mathcal{H}}^{{\mu }_t}(t,x,{\partial }_{x}v(t,x))=0,(t,x)\in [0,T]\times \mathbb{R},$$

(11)

with terminal condition $v(T,x)=g(x,{\mu }_T),x\in \mathbb{R}$. Imposing $\overline{\mu }={\left({\mu }_t\right)}_{0\le t\le T}$ as the flow of the optimally controlled state, the flow of statistical distributions should satisfy Kolmogorov’s equation, so that, introducing the notation $\beta (t,x)=b(t,x,{\mu }_t,\varphi (t,x))$ with $\varphi$ the optimal feedback, the flow ${\left({\nu }_t\right)}_{0\le t\le T}$ of measures given by ${\nu }_t=\mathcal{L}\left(X_t\right)$ satisfies the Kolmogorov’s equation:

$${\partial }_t\nu -{\displaystyle \frac{{\sigma }^2}{2}}{\partial }_{xx}^2\nu -div\left(\beta (t,x)\nu \right)=0,(t,x)\in [0,T]\times \mathbb{R},$$

(12)

with initial condition ${\nu }_0={\mu }_0$. Such a PDE holds in the sense of distributions, since ${\nu }_t$ represents a density and the derivatives involved must be considered in the Wasserstein sense. Setting ${\nu }_t=\mathcal{L}\left(X_t\right)={\mu }_t$, we end up with a system of coupled non-linear forward-backward PDEs (11) and (12), constituting the so-called MFG-PDE system. On the other hand, we can approximate the solution of the aforementioned stochastic control problem via the stochastic Pontryagin principle. In particular, for each open-loop adapted control $\alpha ={\left({\alpha }_t\right)}_{0\le t\le T}$, we denote by $X^{\alpha }={\left(X_t^{\alpha }\right)}_{0\le t\le T}$ the associated state, and we introduce the adjoint equation:

$$\left\{\begin{array}{c}d{Y}_t=-{\partial }_x{H}^{{\mu }_t}(t,X_t,Y_t,{\alpha }_t)dt+Z_{t}d{W}_t,t\in [0,T]\hfill \\ Y_T={\partial }_{x}g(X_T,{\mu }_T).\hfill \end{array}\right.$$

(13)

$H^{{\mu }_t}$ corresponds to the Hamiltonian defined in Equation (9), and the solution ${(Y_t,Z_t)}_{0\le t\le T}$ of the BSDE is called a set of adjoint processes.

The PMP necessary condition states that whenever $X^{\alpha }$ is an optimal state, it must hold that $H^{{\mu }_t}(t,X_t,Y_t,{\alpha }_t)={\mathcal{H}}^{{\mu }_t}(t,X_t,Y_t),t\in [0,T]$. Moreover, if the Hamiltonian $H^{{\mu }_t}$ is convex w.r.t. the variables $(x,\alpha )$ and the terminal g is convex w.r.t. the variable x, then the system given by Equations (13) and (8) characterizes the optimal states of the MFG problem.

3. Main Result

In this section, we generalize the approach proposed in [5], providing an application of the Mean-Field (MF) optimal control to Deep Learning (DL). In particular, we start by considering the learning process characterizing supervised learning as a population risk minimization problem, hence considering its probabilistic nature in the sense that the optimal control parameters, corresponding to the trainable weights in the associated Neural Network (NN) model, depending on the population distribution of input-target pairs constituting the randomness source.

3.1. Neural Network as a Dynamical System

In order to study DL as an optimal control problem, it is necessary to express the NN learning process as a dynamical system [6,21]. In the simplest form, the feed-forward propagation in a T layer, $T\ge 1$, network can be expressed by the following difference equation:

$$x_{t+1}=x_t+f(x_t,{\theta }_t),t=0,\cdots ,T-1,$$

(14)

where $x_0$ is the input, e.g., an image, several time-series, etc., while $x_T$ is the final output, to be compared to some target $y_T$ by means of a given loss function. By moving from a discrete time formulation to a continuous one, the forward dynamics we are interested in will be described by a differential equation that takes the role of (14). The learning aim is to tune the trainable parameters ${\theta }_0,\dots ,{\theta }_{T-1}$ to have $x_T$ as close as possible to $y_T$, according to a specified metric and knowing that the target $y_T$ is joined to the input $x_0$ by means of a probability measure ${\mu }_0$.

Following the dynamical systems approach developed in [6], the supervised learning method aims to approximate some function $\mathcal{F}$, usually called the oracle, denoted by $\mathcal{F}:\mathcal{X}\to \mathcal{Y}$.

As stated before, the set $\mathcal{X}\subset {\mathbb{R}}^d$ contains the d-dimensional array of inputs, e.g., images, financial time-series, sound recorded data, text, etc., while $\mathcal{Y}$ are the targets modelling the corresponding images, numerical forecast, or predicted texts.

In this setting, it is standard to define what is called a hypothesis space as:

$$\mathcal{H}=\{{\mathcal{F}}_{\theta }:\mathcal{X}\to \mathcal{Y}|\theta \in \Theta \}.$$

Training moves from a collection of K samples of input-target pairs ${\{x^i,y^i=\mathcal{F}\left(x^i\right)\}}_{i=1}^K,$ the goal being to approximate $\mathcal{F}$ exploiting these training data points.

Let $(\Omega ,\mathcal{F},\mathbb{P})$ be a probability space supporting random variables $x_0\in {\mathbb{R}}^d$ and $y_T\in {\mathbb{R}}^l$, jointly distributed according to ${\mu }_0:={\mathbb{P}}_{(x_0,y_T)}$ with ${\mu }_0$ modelling the distribution of the input-target pairs. The set of controls $\Theta \subseteq {\mathbb{R}}^m$ denotes the admissible training weights that are assumed to be essentially bounded, measurable $L^{\infty }([0,T],\Theta )$ functions. The network depth, i.e., the number of layers, is denoted by $T>0$. We also introduce the functions:

• the feed-forward dynamics $f:{\mathbb{R}}^d\times \Theta \to {\mathbb{R}}^d$;
• the terminal loss function $\Phi :{\mathbb{R}}^d\times {\mathbb{R}}^l\to \mathbb{R}$;
• the regularization term $L:{\mathbb{R}}^d\times \Theta \to \mathbb{R}$.

State dynamics are described by an Ordinary Differential Equation (ODE) of the form:

$${\dot{x}}_t=f(x_t,{\theta }_t),$$

(15)

representing the continuous version of Equation (14), equipped with an initial condition $x_0$, which is a random variable responsible for the randomness term characterizing Equation (15).

The population risk minimization problem in DL can be expressed by the following MF-optimal control problem (see [5] (p. 5)):

$$\underset{\theta \in L^{\infty }([0,T],\Theta )}{inf}J\left(\theta \right):={\mathbb{E}}_{{\mu }_0}\left[\Phi (x_T,y_T)+{\int }_0^{T}L(x_t,{\theta }_t)dt\right],$$

(16)

subject to the dynamics expressed by the stochastic ODE (15). Since the weights $\theta$ are shared by the distribution ${\mu }_0$ of random variable $(x_0,y_T)$ pairs, Equation (16) can be studied as an MF-optimal control problem.

On the other hand, the empirical risk minimization problem can be expressed by a sampled optimal control problem after computing i.i.d. samples $\{x_0^i,y_T^i\}{)}_{i=1}^N$ modelled by ${\mu }_0={\mathbb{P}}_{(x_0,y_T)}$:

$$\underset{\theta \in L^{\infty }([0,T],\Theta )}{inf}{J}_N\left(\theta \right):={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[\Phi (x_T^i,y_T^i)+{\int }_0^{T}L(x_t^i,{\theta }_t)dt\right],$$

(17)

subject to the dynamics:

$${\dot{x}}_t^i=f(x_t^i,{\theta }_t),i=1,\dots ,N,$$

whose solutions, moving from random initial conditions through a deterministic path, correspond to random variables.

As in classical optimal control theory, the previous problem can be solved following two inter-connected approaches: a global theory, based on the Dynamic Programming Principle (DPP) leading to the HJB equation, or considering the Pontryagin Maximum Principle (PMP) approach, hence expressing the solution by a system of Forward Backward SDEs (FBSDEs) plus a local optimality condition.

3.2. HJB Equation

The idea behind the HJB formalism is to define a value function corresponding to the optimal loss of the control problem w.r.t. the general starting time and state. For the population risk minimization formulation expressed by Equation (16), the state argument of the value function corresponds to an infinite-dimensional object that models a joint distribution of the input-target as an element of a suitable Wasserstein space.

As regards random variables and their distribution, a suitable space must be defined for the rigorous treatment of the optimal control problem. In particular, we use the shorthand notation $L^2(\Omega ,{\mathbb{R}}^{d+l})$ for $L^2((\Omega ,\mathcal{F},\mathbb{P}),{\mathbb{R}}^{d+l})$ to denote the set of ${\mathbb{R}}^{d+l}$-valued square integrable random variables w.r.t. a given probability measure $\mathbb{P}$. Then, we deal with a Hilbert space considering the norm:

$${\left|\right|X\left|\right|}_{L^2}:={\left(\mathbb{E}\left(\right|\left|X\right|{|}^2\right)}^{\frac{1}{2}},X\in L^2(\Omega ,{\mathbb{R}}^{d+l}),$$

The set ${\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)$ denotes the integrable probability measures defined on the Euclidean space ${\mathbb{R}}^{d+l}$. Let us recall that the random variable X is square integrable in $L^2(\Omega ,{\mathbb{R}}^{d+l})$ if and only if its law ${\mathbb{P}}_X\in {\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)$. The space ${\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)$ can be endowed with a metric by considering the Wasserstein distance defined in Equation (2). For $p=2$, the two-Wasserstein distance reads:

$$W_2(\mu ,\nu ):=inf\left\{{\left({\int }_{{\mathbb{R}}^{d+l}\times {\mathbb{R}}^{d+l}}\left|\right|w-{z\left|\right|}^2\pi (dw,dz)\right)}^{\frac{1}{2}}|\pi \in {\mathcal{P}}_2({\mathbb{R}}^{d+l}\times {\mathbb{R}}^{d+l})with marginals\mu and\nu \right\},$$

according to the marginals introduced in Section 2.1, or equivalently:

$$W_2(\mu ,\nu ):=inf\left\{\left|\right|X-{Y\left|\right|}_{L^2}|X,Y\in L^2(\Omega ,{\mathbb{R}}^{d+l})with{\mathbb{P}}_X=\mu ,{\mathbb{P}}_Y=\nu \right\},$$

see, e.g., [5] (p. 6). Moreover, $\forall \mu \in {\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)$, we define the associated norm:

$${\left|\right|\mu \left|\right|}_{L^2}={\left({\int }_{{\mathbb{R}}^{d+l}}{\left|\right|w\left|\right|}^2\mu \left(dw\right)\right)}^{\frac{1}{2}}.$$

Given a measurable function $\psi :{\mathbb{R}}^{d+l}\to {\mathbb{R}}^q$ that is square integrable w.r.t. the probability distribution $\mu$, the following notation is introduced:

$$\langle \psi (·),\mu \rangle :={\int }_{{\mathbb{R}}^{d+l}}\psi \left(w\right)\mu \left(dw\right).$$

(18)

Concerning the dynamical evolution of probability measures, let us fix $\xi \in L^2(\Omega ,{\mathbb{R}}^{d+l})$ and the control process $\theta \in L^{\infty }([0,T],\Theta )$. Then, the dynamics of the system can be written as:

$$W_s^{t,\xi ,\theta }=\xi +{\int }_t^s\overline{f}(W_s^{t,\xi ,\theta },{\theta }_t)ds,s\in [t,T],$$

$\mu$ being the law associated with the variable $\psi$ defined by $\mu ={\mathbb{P}}_{\xi }\in {\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)$, and we can rewrite the law of $W_s^{t,\xi ,\theta }$ as:

$${\mathbb{P}}_s^{t,\mu ,\theta }:={\mathbb{P}}_{W_s^{t,\xi ,\theta }}.$$

(19)

Indeed, the law involving the dynamics $W_s^{t,\xi ,\theta }$ depends only on the law of $\xi$ and not on the random variable itself; see, e.g., [5] (p. 7).

It turns out that, to obtain the HJB Equation (5) corresponding to the above introduced formulation, it is necessary to define the concept of the derivative w.r.t. a probability measure. To begin with, it is useful to consider probability measures on ${\mathbb{R}}^{d+l}$ as laws expressing probabilistic features of the ${\mathbb{R}}^{d+l}$-valued random variables defined over the probability space $(\Omega ,\mathcal{F},\mathbb{P})$. Then, we define the Banach space of random variables to define the derivatives. Moreover, if we define a function $u:{\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)\to \mathbb{R}$, it is possible to lift it into its extension U defined on $L^2([0,T],{\mathbb{R}}^{d+l})$, as follows:

$$U\left(X\right)=u\left({\mathbb{P}}_X\right),\forall X\in L^2(\Omega ,{\mathbb{R}}^{d+l}),$$

then the definition of the derivative w.r.t. a probability measure can be expressed in terms of U in the usual Banach space setting. In particular, we have that u is ${\mathcal{C}}^1\left({\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)\right)$, if the lifted function U is Fréchet differentiable with continuous derivatives.

Since $L^2(\Omega ,{\mathbb{R}}^{d+l})$ can be identified with its dual, if the Fréchet derivative $DU\left(X\right)$ exists, by Riesz’s theorem, it can be identified with an element of $L^2(\Omega ,{\mathbb{R}}^{d+l})$, i.e.,

$$DU\left(X\right)\left(Y\right)=\mathbb{E}[DU\left(X\right)·Y],\forall Y\in L^2(\Omega ,{\mathbb{R}}^{d+l}).$$

It is worth underlining that $DU\left(X\right)$ does not depend on X, but only on the law described by X; hence, the derivative of u at $\mu ={\mathbb{P}}_X$ is described by ${\partial }_{\mu }u\left({\mathbb{P}}_{\mathbb{X}}\right):{\mathbb{R}}^{d+l}\to {\mathbb{R}}^{d+l}$, defined as:

$$DU\left(X\right)={\partial }_{\mu }u\left({\mathbb{P}}_X\right)\left(X\right).$$

By duality, we know that ${\partial }_{\mu }u\left({\mathbb{P}}_{\mathbb{X}}\right)$ is square integrable w.r.t. $\mu$. To define a notion for the chain rule in ${\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)$, a dynamical system is described by:

$$W_t=\xi +{\int }_0^t\overline{f}\left(W_s\right)ds,\xi \in L^2(\Omega ,{\mathbb{R}}^{d+l}),$$

where $\overline{f}$ denotes the feed-forward dynamics. If a function $u\in {\mathcal{C}}^1\left({\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)\right)$, meaning that it is differentiable with a continuous derivative w.r.t. a probability measure, then, for all $t\in [0,T]$, we have:

$$u\left({\mathbb{P}}_{W_t}\right)=u\left({\mathbb{P}}_{W_0}\right)+{\int }_0^t\langle {\partial }_{\mu }u\left({\mathbb{P}}_{W_s}\right)(·)·\overline{f}(·),{\mathbb{P}}_{W_s}\rangle ds,$$

(20)

where · denotes the usual inner product between the vector in ${\mathbb{R}}^{d+l}$. Equivalently, exploiting the lifted function of u, we can state:

$$U\left(W_t\right)=U\left(W_0\right)+{\int }_0^t\mathbb{E}[DU\left(W_s\right)·\overline{f}\left(W_s\right)]ds.$$

Moreover, the variable w denotes the concatenated $(d+l)$-dimensional variable $(x,y)$, where $x\in {\mathbb{R}}^d$ and $y\in {\mathbb{R}}^l$. Correspondingly, $\overline{f}(w,\theta ):=(f(x,\theta ),0)$ is the extended $(d+l)$-dimensional Feed-Forward Function (FFF), $\overline{L}(w,\theta )=L(x,\theta )$ is the extended $(d+l)$-dimensional regularization loss and $\overline{\Phi }\left(w\right):=\Phi (x,y)$ represents the terminal loss function.

Since the state variable is identified with a probability distribution $\mu \in {\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)$, the resulting objective functional can be defined as:

$$J(t,\mu ,\theta ):={\mathbb{E}}_{(x_t,y_T)\sim \mu }\left[\Phi (x_T,y_T)+{\int }_t^{T}L(x_t,{\theta }_t)dt\right],$$

which can be written, with the concatenated variable w and the bracket notation introduced in (18), as:

$$J(t,\mu ,\theta ):=\langle \overline{\Phi }\left(w\right),{\mathbb{P}}_T^{t,\mu ,\theta }\rangle +{\int }_t^T\langle \overline{L}(w,{\theta }_s),{\mathbb{P}}_s^{t,\mu ,\theta }\rangle ds.$$

In this setting, some assumptions for the value function are needed to solve Equation (16). In particular:

• f, L and $\Phi$ are bounded;
• f, L and $\Phi$ are Lipschitz w.r.t. x, and the Lipschitz constant of f and L are independent of $\theta$;
• ${\mu }_0\in {\mathcal{P}}_2\left({\mathbb{R}}^{(d+l)}\right)$.

The value function $v^{*}(t,\mu )$ is defined as the real-valued function on $[0,T]\times {\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)$, corresponding to the infimum of the functional J over the training parameters $\theta$:

$$v^{*}(t,\mu )=\underset{\theta \in L^{\infty }([0,T],\Theta )}{inf}J(t,\mu ,\theta ).$$

It is essential to observe how the value function satisfies a recursive relation based on the Dynamic Programming Principle (DPP). This implies that, for any optimal trajectory starting from any intermediate point, the remaining part of the trajectory still has to be optimal. The latter principle can be expressed by defining the value function as:

$$v^{*}(t,\mu )=\underset{\theta \in L^{\infty }\left([0,T]\right),\Theta }{inf}\left[{\int }_t^{\widehat{t}}\langle \overline{L}(·,{\theta }_s),{\mathbb{P}}_s^{t,\mu ,\theta }\rangle ds+v^{*}(\widehat{t},{\mathbb{P}}_{\widehat{t}}^{t,\mu ,\theta })\right],$$

$\forall 0\le t\le \widehat{t}\le T$ and $\mu \in {\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)$.

Considering a small increment of time $\widehat{t}=t+\delta t$ with $\delta >0$, we can compute the Taylor expansion in the Wasserstein sense, hence obtaining:

$$0=\underset{\theta \in L^{\infty }\left([0,T]\right),\Theta }{inf}\left[v^{*}(t+\delta t,{\mathbb{P}}_{t+\delta t}^{t,\mu ,\theta })-v^{*}(t,\mu )+{\int }_t^{t+\delta t}\langle \overline{L}(w,{\theta }_s),{\mathbb{P}}_s^{t,\mu ,\theta }\rangle ds\right].$$

By the chain rule in ${\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)$, we have:

$$0\approx \underset{\theta \in L^{\infty }\left([0,T]\right),\Theta }{inf}\left[{\partial }_{t}v(t,\mu )\delta t+{\int }_t^{t+\delta t}\langle {\partial }_{\mu }v(t,\mu )\left(w\right)·\overline{f}(w,\theta )+\overline{L}(w,{\theta }_s),\mu \rangle ds\right].$$

Since the infinitesimal $\delta t$ does not affect the distribution $\mu$ and the controls $\theta$ (see [5] (p. 13)), integrating the second term, we have:

$$0\approx \delta t\underset{\theta \in L^{\infty }\left([0,T]\right),\Theta }{inf}\left[{\partial }_{t}v(t,\mu )+\langle {\partial }_{\mu }v(t,\mu )\left(w\right)·\overline{f}(w,\theta )+\overline{L}(w,\theta ),\mu \rangle \right].$$

Taking $\delta t\to 0$, we have:

$$\left\{\begin{array}{c}{\displaystyle \frac{\partial v}{\partial t}}+\underset{\theta \in \Theta }{inf}〈{\partial }_{\mu }v(t,\mu )\left(w\right)·\overline{f}(w,\theta )+\overline{L}(w,\theta ),\mu 〉=0,on[0,T)\times {\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)\hfill \\ v(T,\mu )=\langle \overline{\Phi }\left(w\right),\mu \rangle ,on{\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right).\hfill \end{array}\right.$$

(21)

Since the value function should solve the HJB equation, it is essential to find the precise link between the solution of this PDE and the value function obtained from the minimization of the functional J. To provide the result, we use a verification argument allowing the following consideration: if the solution of the HJB is smooth enough, then it corresponds to the value function $v^{*}$; moreover, it allows computing the optimal control ${\theta }^{*}$.

Theorem 1

(The verification argument). Let v be a function in ${\mathcal{C}}^{1,1}([0,T]\times {\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right))$. If v is a solution of the HJB equation in (21) and there exists $\overline{\theta }$ that is mapping $(t,\mu )\to \Theta$ attaining the infimum in the HJB equation, then $v(t,\mu )=v^{*}(t,\mu )$, and $\overline{\theta }$ is an optimal feedback control policy, i.e., $\theta ={\theta }^{*}$ is a solution of the population risk minimization problem expressed by Equation (16), where ${\theta }_t^{*}:=\overline{\theta }(t,{\mathbb{P}}_{w_t^{\star }})$ with ${\mathbb{P}}_{w_0^{\star }}={\mu }_0$ and ${\displaystyle \frac{d{w}_t^{\star }}{dt}}=\overline{f}(w_t^{\star },{\theta }_t^{\star })$.

Proof of Theorem 1. 

Given any control process $\theta$, applying Formula (20) between $s=t$ and $s=T$ with explicit time dependence gives:

$$v(T,{\mathbb{P}}_T^{t,\mu ,\theta })=v(t,\mu )+{\int }_t^T{\displaystyle \frac{\partial v}{\partial t}}(s,{\mathbb{P}}_s^{t,\mu ,\theta })+\langle {\partial }_{\mu }v(s,{\mathbb{P}}_s^{t,\mu ,\theta })(·)·\overline{f}(·,{\theta }_s),{\mathbb{P}}_s^{t,\mu ,\theta })\rangle ds,$$

Equivalently:

$$\begin{array}{ccc}\hfill v(t,\mu )& =v(T,{\mathbb{P}}_T^{t,\mu ,\theta })-{\int }_t^T{\displaystyle \frac{\partial v}{\partial t}}(s,{\mathbb{P}}_s^{t,\mu ,\theta })+\langle {\partial }_{\mu }v(s,{\mathbb{P}}_s^{t,\mu ,\theta })(·)·\overline{f}(·,{\theta }_s),{\mathbb{P}}_s^{t,\mu ,\theta }\rangle ds\hfill & \\ & \le v(T,{\mathbb{P}}_T^{t,\mu ,\theta })+{\int }_t^T\langle \overline{L}(·,{\theta }_s),{\mathbb{P}}_s^{t,\mu ,\theta }\rangle ds\hfill & \\ & =\langle \overline{\Phi }(·),{\mathbb{P}}_t^{T,\mu ,\theta }+{\int }_t^T\langle \overline{L}(·,{\theta }_s),{\mathbb{P}}_s^{t,\mu ,\theta }\rangle ds\hfill & \\ & =J(t,\mu ,\theta ),\hfill & \end{array}$$

where the first inequality comes from the infimum condition in (21).

Since the control is arbitrary, we have:

$$v(t,\mu )\le v^{*}(t,\mu ),$$

then it can be substituted with ${\theta }^{*}$ where ${\theta }_t^{*}=\overline{\theta }(t,{\mathbb{P}}_s^{t,\mu ,{\theta }^{*}})$ is computed by the optimal feedback control. Repeating the above argument, the inequality becomes an equality since the infimum is attained for $\overline{\theta }$:

$$v(t,\mu )=J(t,\mu ,{\theta }^{*})\ge v^{*}(t,\mu ).$$

Thus, $v(t,\mu )=v^{*}(t,\mu )$, and $\overline{\theta }$ defines an optimal control policy. For more details, see [5] (Proposition 3, pp. 13–14). □

The importance of Theorem 1 consists of linking smooth solutions of the parabolic PDE to the solutions of the population risk minimization problem, becoming a natural candidate for the DL problem.

Moreover, the optimal control policy $\overline{\theta }:[0,T]\times {\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)\to \Theta$ is identified by computing the infimum in (21). Hence, it turns out that the HJB equation strongly characterizes the learning problem’s solution for the feedback, or closed-loop networks: control weights are actively adjusted according to the outputs, and this is the essential feature of closed-loop control. Nevertheless, the solution comes from a PDE that is in general difficult to solve, even numerically. On the other hand, open-loop solutions can be obtained from the closed-loop control policy by sequentially setting ${\theta }_t^{\star }=\overline{\theta }(t,{\mathbb{P}}_{w_t^{\star }})$, $w_t^{\star }$ being the solution of the feed-forward ODE describing the dynamics of the state variable, with $\theta ={\theta }^{\star }$ up to time t. Usually within DL settings, open-loops are used during training or to measure the inference of a trained model, since trained weights for each neuron will have a fixed value.

The great limit of such a formulation relies in assuming that the value function $v^{\star }(t,\mu )$ is continuously differentiable. It is straightforward to study a more flexible characterization for $v^{\star }$ dealing with weak solutions, also denoted as viscosity solutions. Thus, it is worth considering a weaker formulation of the PDE to go beyond the concept of classical solutions, by introducing the notion of viscosity ones, hence allowing obtaining relevant results when dealing with weaker assumptions on the coefficients defining the (stochastic) differential problem we are interested in; see, e.g., [5] (Section 5 pp. 14–22) for more details.

The key idea relies on exploiting the lifting identification between measures and random variables and moving from the Wasserstein space ${\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)$ to the Hilbert space $L^2(\Omega ,{\mathbb{R}}^{d+l})$, using tools developed to study viscosity solutions.

We introduce a functional defined as the Hamiltonian for viscosity formulation $\mathcal{H}(\xi ,P):L^2(\Omega ,{\mathbb{R}}^{d+l})\times L^2(\Omega ,{\mathbb{R}}^{d+l})\to \mathbb{R}$ through:

$$\mathcal{H}(\xi ,P)=\underset{\theta \in \Theta }{inf}\mathbb{E}[P·\overline{f}(\xi ,\theta )+\overline{L}(\xi ,\theta )].$$

(22)

Then, the lifted Bellman equation can be written w.r.t. $V(t,\xi )=v(t,{\mathbb{P}}_{\xi })$ as follows:

$$\left\{\begin{array}{c}{\displaystyle \frac{\partial V}{\partial t}}+\mathcal{H}(\xi ,DV(t,\xi ))=0on[0,T)\times L^2(\Omega ,{\mathbb{R}}^{d+l}),\hfill \\ V(T,\xi )=\mathbb{E}\left[\overline{\Phi }\left(\xi \right)\right]on{L}^2(\Omega ,{\mathbb{R}}^{d+l});\hfill \end{array}\right.$$

(23)

hence, the PDE we are analysing is now set within a larger space corresponding to $L^2(\Omega ,{\mathbb{R}}^{d+l})$.

We say that a bounded, uniformly continuous function $u:[0,T]\times {\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)\to \mathbb{R}$ is a viscosity solution of HJB Equation (21) if its lifted function $U:[0,T]\times L^2(\Omega ,{\mathbb{R}}^{d+l})\to \mathbb{R}$ defined by:

$$U(t,\xi )=u(t,{\mathbb{P}}_{\xi }),$$

is a viscosity solution to the lifted Bellman Equation (23), namely:

• $U(T,\xi )\le \mathbb{E}\left[\overline{\Phi }\left(\xi \right)\right]$ and for any test function $\psi \in {\mathcal{C}}^{1,1}([0,T]\times L^2(\Omega ,{\mathbb{R}}^{d+l}))$ such that $(U-\psi )$ has a local maximum at $(t_0,{\xi }_0)\in [0,T)\times L^2(\Omega ,{\mathbb{R}}^{d+l})$, $\psi$ solves:

  $${\partial }_t\psi (t_0,{\xi }_0)+\mathcal{H}({\xi }_0,D\psi (t_0,{\xi }_0))\ge 0.$$
• $U(T,\xi )\ge \mathbb{E}\left[\overline{\Phi }\left(\xi \right)\right]$ and for any test function $\psi \in {\mathcal{C}}^{1,1}([0,T]\times L^2(\Omega ,{\mathbb{R}}^{d+l}))$ such that the map $(U-\psi )$ has a local minimum at $(t_0,{\xi }_0)\in [0,T)\times L^2(\Omega ,{\mathbb{R}}^{d+l})$, $\psi$ solves:

  $${\partial }_t\psi (t_0,{\xi }_0)+\mathcal{H}({\xi }_0,D\psi (t_0,{\xi }_0))\le 0.$$

Actually, the unique solution of this formulation corresponds to the value function $v^{*}$ from the minimization problem; see, e.g., [5] (Theorem 1, p. 15). Therefore, the HJB equation provides both the necessary and sufficient condition for the optimality of the learning procedure.

Adopting the MF-optimal control viewpoint implies that the population risk minimization problem of DL can be studied as a variational problem, whose solution is characterized by a suitable HJB equation, in analogy with the classical calculus of variations. In other words, the HJB equation is a global characterization of the value function to be solved over the entire space $P_2\left({\mathbb{R}}^{d+l}\right)$ of input-target distributions. From the numerical point of view, it is a hard task to get a solution for the entire space; this is why the learning problem is typically locally solved, around some (small set of) trajectories generated according to the initial condition ${\mu }_0\in {\mathcal{P}}_2\left({\mathbb{R}}^{d+l}\right)$, then applying the obtained feedback to nearby input-target distributions.

3.3. Mean-Field Pontryagin Maximum Principle

We have seen how the HJB approach provides a characterization of the optimal solution for the population risk minimization problem that holds globally in $P_2\left({\mathbb{R}}^{d+l}\right)$, at the price of being difficult to handle in practice. Moving from this consideration, the MF-PMP aims to show a local condition for optimality, expressed in terms of $\mathbb{E}\left[H\right]$, i.e., the expectation of the Hamiltonian function.

Starting from the population risk minimization problem defined in Equation (16) and given a collection of K sample input-target pairs, the single ith input sample is considered. The prediction of the network can be approximated by a deterministic transformation of the terminal state $g\left(X_T^i\right)$ for some $g:{\mathbb{R}}^d\to \mathcal{Y}$ that models a function both of the initial input $x^i$ and of the control parameters $\theta$. Moreover, we define a loss function $\Phi :\mathcal{Y}\times \mathcal{Y}\to \mathbb{R}$, which is minimized when the arguments are equal. Therefore, the goal is to minimize:

$$\sum _{i=1}^K\Phi (g\left(X_T^i\right),y^i).$$

Since g is fixed, it can be absorbed into the definition of the loss function by defining the array ${\Phi }_i(·):=\Phi (·,y_i)$.

Then, the supervised learning problem can be expressed as:

$$\underset{\theta \in \mathcal{U}}{min}\left[\sum _{i=1}^K{\Phi }_i\left(X_T^i\right)+{\int }_0^{T}L\left({\theta }_t\right)dt\right],$$

(24)

where $L:\Theta \to \mathbb{R}$ acts as a regularizer term to model a running cost.

Input variables $x=(x^1,\dots ,x^K)$ can be considered as the elements of a Euclidean space ${\mathbb{R}}^{d\times K}$, representing the initial conditions of the following ODE system:

$${\dot{X}}_t^i=f_{{\theta }_t}(t,X_t^i),X_0^i=x^i,0\le t\le T,i=1,\dots ,K,$$

(25)

where $\theta :[0,T]\to \Theta$ are the control parameters to be trained. The dynamics (25) are decoupled except for the control. A general space $\mathcal{U}$ for controls $\theta$ is then defined as: $\mathcal{U}:=\{\theta :[0,T]\to \Theta :\theta isLebesguemeasurable\}$, and we are aiming to choose $\theta$ in $\mathcal{U}$ to have $g\left(X_T^i\right)$ closer to $y^i$ for $i=1,\dots ,K$.

To formulate the PMP as a set of necessary conditions for optimal solutions, it is useful to define the Hamiltonian $H:[0,T]\times {\mathbb{R}}^d\times {\mathbb{R}}^d\times \Theta \to \mathbb{R}$ given by:

$$H(t,x,p,\theta ):=p·f(t,x,\theta )-L\left(\theta \right),$$

with p modelling an adjoint process as in Equation (6).

Let us underline that all input-target pairs $(x_0,y_0)$, connected by the distribution ${\mu }_0$, share a common control parameter, and this feature suggests the idea to develop a maximum condition that has to hold in the average sense. Indeed, the control is now enforced on the continuity equation that describes the evolution of probability densities.

The following assumptions are needed:

• f is bounded and f, L are continuous w.r.t. $\theta$;
• f, L and $\Phi$ are continuously differentiable w.r.t. x;
• the distribution ${\mu }_0$ has bounded support in ${\mathbb{R}}^d\times {\mathbb{R}}^l$, which means there exists $M>0$ such that $\mu \left(\right\{(x,y)\in {\mathbb{R}}^d\times {\mathbb{R}}^l:\left|\right|x\left|\right|+\left|\right|y\left|\right|\le 1\left\}\right)=1$.

Theorem 2 (Mean-Field Pontryagin Maximum Principle).

 Let assumptions 1–3 hold and ${\theta }^{*}\in L^{\infty }([0,T],\Theta )$ be the minimizer for $J\left(\theta \right)$ corresponding to the optimal control of the population risk minimization problem (16). Define the Hamiltonian $H:{\mathbb{R}}^d\times {\mathbb{R}}^d\times \Theta \to \mathbb{R}$ as

$$H(x,p,\theta )=p·f(x,\theta )-L(x,\theta ).$$

(26)

Then, there exist absolutely continuous stochastic processes $x^{*}$ and $p^{*}$ solving the following forward backward SDEs:

$${\dot{x}}_t^{*}=f(x_t^{*},{\theta }_t^{*}),x_t^{*}=x_0,$$

(27)

$${\dot{p}}_t^{*}=-{\nabla }_{x}H(x_t^{*},p_t^{*},{\theta }_t^{*}),p_T^{*}=-{\nabla }_x\Phi (x_T^{*},y_0),$$

(28)

and the related optimality condition expressed in terms of the expectation of the Hamiltonian function:

$${\mathbb{E}}_{{\mu }_0}H(x_t^{*},p_t^{*},{\theta }_t^{*})\ge {\mathbb{E}}_{{\mu }_0}H(x_t^{*},p_t^{*},\theta ),\forall \theta \in \Theta ,for a.e.t\in [0,T].$$

(29)

Proof of Theorem 2. 

For the sake of simplicity, let us introduce a new coordinate $x^0$ satisfying the dynamics ${\dot{x}}_t^0=L(x_t{,}^{*},{\theta }_t^{*})$ with $x_0^0=0$. Through this choice, the definition of the Hamiltonian in Equation (26) can be rewritten without running loss L by redefining:

$$x\to (x^0,x),f\to (L,f),\Phi (x_T,y_0)\to \Phi (x_T,y_0)+x_T^0.$$

Assumptions 1–3 are still preserved, but now we consider without loss of generality the case $L\equiv 0$.

Let some $\tau \in (0,T]$ be a Lebesgue point of $\widehat{f}\left(t\right):=f(x_t^{*},{\theta }_t^{*})$; in this setting, these points are dense in $[0,T]$. Now, for $\epsilon \in (0,\tau )$, define the family of perturbed controls:

$${\theta }_t^{\tau ,\epsilon }=\left\{\begin{array}{cc}\omega \hfill & t\in \left[\tau -\epsilon ,\tau \right]\hfill \\ {\theta }_t^{*}\hfill & otherwise\hfill \end{array}\right.,$$

where $\omega \in \Theta$ is an admissible control; this kind of perturbation is called needle perturbation. Accordingly, define $x_t^{\tau ,\epsilon }$ by:

$$x_t^{\tau ,\epsilon }=x_0+{\int }_0^{t}f(x_s^{\tau ,\epsilon },{\theta }_s^{\tau ,\epsilon })ds,$$

that is the solution of the forward propagation equation with the perturbed control ${\theta }^{\tau ,\epsilon }$. Clearly, $x_t^{*}=x_t^{\tau ,\epsilon }$ for every $t\le \tau -\epsilon$ and every $x_0$ since the perturbation is not present. At the limit point $t=\tau$, the following holds:

$${\displaystyle \frac{1}{\epsilon }}(x_{\tau }^{\tau ,\epsilon }-x_{\tau }^{*})={\displaystyle \frac{1}{\epsilon }}{\int }_{\tau -\epsilon }^{\tau }f(x_s^{\tau ,\epsilon },\omega )-f\left(f_s^{*}\right),{\theta }_s^{*})ds,$$

and since $\tau$ is a Lebesgue point of F:

$$v_{\tau }:=\underset{\epsilon \to 0}{lim}{\displaystyle \frac{1}{\epsilon }}(x_{\tau }^{\tau ,\epsilon }-x_{\tau }^{*})=f(x_{\tau }^{*},\omega )-f(x_{\tau }^{*},{\theta }_{\tau }^{*}).$$

It is possible to characterize $v_{\tau }$ as the leading order perturbation on the state due to the needle perturbation introduced in the infinitesimal interval $[\tau -\epsilon ,\tau ]$. In interval $(\tau ,T]$, the dynamics is the same before applying the perturbation since the controls are the same.

Now, it is necessary to consider how the perturbation $v_{\tau }$ propagates. Thus, define for $t\ge \tau$:

$$v_t^{\epsilon }:={\displaystyle \frac{1}{\epsilon }}(x_t^{\tau ,\epsilon }-x_t^{*}),$$

and:

$$v_t:=\underset{\epsilon \to 0}{lim}{v}_t^{\epsilon },$$

$v_t$ is well-defined for almost every t, which are every Lebesgue point of the map $t\mapsto x^{*}\left(t\right)$, and it satisfies the following linearised equation:

$$\left\{\begin{array}{c}{\dot{v}}_t={\nabla }_{x}f{(x_t^{*},{\theta }_t^{*})}^T{v}_t,t\in (\tau ,T]\hfill \\ v_{\tau }=f(x_{\tau }^{*},\omega )-f(x_{\tau }^{*},{\theta }_{\tau }^{*}).\hfill \end{array}\right.$$

(30)

In particular, $v\left(T\right)$ represents the perturbation of the final state introduced by this control. By the optimality assumption of ${\theta }^{*}$, it follows that:

$${\mathbb{E}}_{{\mu }_0}\Phi (x_T^{\tau ,\epsilon },y_0)\ge {\mathbb{E}}_{{\mu }_0}\Phi (x_T^{*},y_0).$$

Assumptions 1 and 2 (p. 13) imply ${\nabla }_x\Phi$ is bounded. By the dominated convergence theorem, we know that:

$$0\le \underset{\epsilon \to 0}{lim}{\displaystyle \frac{1}{\epsilon }}{\mathbb{E}}_{{\mu }_0}[\Phi (x_T^{\tau ,\epsilon },y_0)-\Phi (x_T^{*},y_0)]$$

$$={\mathbb{E}}_{{\mu }_0}{\displaystyle \frac{d}{d\epsilon }}\Phi (x_T^{\epsilon ,\tau },y_0){|}_{\epsilon =0^{+}}$$

$$={\mathbb{E}}_{{\mu }_0}{\nabla }_x\Phi (x_T^{\epsilon ,\tau },y_0)·v_T.$$

(31)

Let us define $p^{\star }$ as the solution of the adjoint of Equation (30), hence:

$${\dot{p}}_t^{*}=-{\nabla }_{x}f(x_s^{*},{\theta }_s^{*})p_t^{*},p_T^{*}=-{\nabla }_x\Phi (x_T^{*},y_0).$$

By Equation (31), it follows that ${\mathbb{E}}_{{\mu }_0}{p}_T^{*}·v_T\le 0$, and moreover, for all $t\in [\tau ,t]$:

$${\displaystyle \frac{d}{dt}}(p_t^{*}·v_t)={\dot{p}}_t^{*}·v_t+{\dot{v}}_t·p_t^{*}=0;$$

thus,

$${\mathbb{E}}_{{\mu }_0}{p}_{\tau }^{*}·v_t={\mathbb{E}}_{{\mu }_0}{p}_T^{*}·v_T\le 0,\forall t\in [\tau ,T],$$

so that taking $t=\tau$:

$${\mathbb{E}}_{{\mu }_0}{p}_{\tau }^{*}·f(x_{\tau }^{*},{\theta }_{\tau }^{*})\ge {\mathbb{E}}_{{\mu }_0}{p}_T^{*}·f(x_{\tau }^{*},\omega ).$$

Since $\omega$ is arbitrarily chosen, this completes the proof by recalling that $H(x,p,\theta )=p·f(x,\theta )$. See [5] (Theorem 3, pp. 23–24) for more details. □

MF-PMP refers only to the control of the open-loop type; Equation (27) is a feed-forward ODE, describing the state dynamics under optimal controls ${\theta }^{*}$. Equation (28) defines the evolution of the co-state variable $p_s^{*}$, characterizing the evolution of an adjoint variational condition backwards in time. It is interesting to note how the optimality condition described in Equation (29) does not involve first order partial derivatives, being expressed in terms of expectations. In particular, it requires that optimal solutions must globally maximize the Hamiltonian function. This aspect allows considering also cases of non-differentiable dynamics w.r.t. the controls weights, as well as cases characterized by optimal weights lying on the boundary of the training set $\Theta$. Moreover, the usual first order optimality condition can be derived from (29). Comparing this MF formulation to the classical PMP, we can see that the main difference lies in the fact that the maximization condition is expressed in terms of the expectation above a probability density ${\mu }_0$. The latter result is not surprising, since the mean-field-optimal control must depend on the probability distribution of input-target pairs.

Let us also note that the mean-field PMP expressed in Theorem 2 can be written more compactly as follows. For each control process $\theta \in L^{\infty }([0,T],\Theta )$, we denote by $x^{\theta }:=\{x_t^{\theta }:0\le t\le T\}$ and $p^{\theta }:=\{p_t^{\theta }:0\le t\le T\}$ the solutions of Hamilton’s Equations (27) and (28), and we enforce the control expressed by the random variables $(x_0,y_T)\sim {\mu }_0$, through:

$${\dot{x}}_t^{\theta }=f(x_t^{\theta },{\theta }_t),x_0^{\theta }=x_0,$$

$${\dot{p}}_t^{\theta }=-{\nabla }_{x}H(x^{\theta },p_t^{\theta },{\theta }_t),p^{\theta }=-{\nabla }_x\Phi (x_T^{\theta },y_t).$$

Then, ${\theta }^{*}$ satisfies the PMP if and only if:

$${\mathbb{E}}_{{\mu }_0}H(x_t^{{\theta }^{*}},p_t^{{\theta }^{*}},{\theta }_t^{*})\ge {\mathbb{E}}_{{\mu }_0}H(x_t^{{\theta }^{*}},p_t^{{\theta }^{*}},\theta ),\forall \theta \in \Theta .$$

(32)

Furthermore, observe that the mean-field PMP includes, as a special case, the necessary conditions for the optimality of the sampled optimal control problem (17). In order to point out this aspect, define the empirical measure:

$${\mu }_0^N:={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\delta }_{(x_0^i,y_T^i)},$$

and apply the mean-field PMP with ${\mu }_0^N$ in place of ${\mu }_0$ to obtain:

$${\displaystyle \frac{1}{N}}\sum _{i=1}^{N}H(x_t^{{\theta }^{*},i},p_t^{{\theta }^{*},i},{\theta }_t^{*})\ge {\displaystyle \frac{1}{N}}\sum _{i=1}^{N}H(x_t^{{\theta }^{*},i},p_t^{{\theta }^{*},i},\theta ),\forall \theta \in \Theta ,$$

(33)

where $x_t^{{\theta }^{*},i}$ and $p_t^{{\theta }^{*},i}$ are defined through the input-target pair $(x_0^i.y_T^i)$. Moreover, since ${\mu }_0^N$ is a random measure, (33) is a random equation whose solutions correspond to random variables.

Concerning possible numerical analysis of the DL algorithm based on the maximum principle, we refer to [4] (pp. 13–15), where a comparison can be found between usual gradient approaches to the discrete formulation of the Mean-Field PMP stated in Theorem 2, with a loss function based on Equation (24). The test and train losses of some variants of SGD algorithms are compared to the mean-field algorithm based on the discrete PMP. It is possible to observe that the latter algorithm is characterized by a better convergence rate, being then faster. This improvement is mainly due to the fact that it allows avoiding possibly getting stuck, caused by the flat regions, as clearly shown by the graphs reported in [4] (p. 14).

3.4. Connection between the HJB Equation and the PMP

In what follows, we provide connections between the global and local formulation of the HJB formalism via the PMP, exploiting the connection between Hamilton’s canonical equations (ODEs) and the Hamilton–Jacobi equations (PDEs). The Hamiltonian dynamics of Equations (27) and (28) describe the trajectory of a random variable that is completely determined by random variables $(x_0,y_T)$. On the other hand, the optimality condition described by Equation (29) does not depend on the particular probability measure of the initial input-target pairs. Notice that the maximum principle can be expressed in terms of a Hamiltonian flow that depends on a probability measure in a suitable Wasserstein space and where Equation (29) is the corresponding lifting version. Analogously, in order to have both solutions in the same functional space, HJB has to be lifted in the $L^2(\Omega ,{\mathbb{R}}^{d+l})$ space.

Starting from the lifted Bellman Equation (23), lying in $L^2(\Omega ,{\mathbb{R}}^{d+l})$, it is possible to apply the method of the characteristics and define the following system of equations, after introducing $P_t=DV(t,{\xi }_t)$:

$$\left\{\begin{array}{c}{\dot{\xi }}_t=D_P\mathcal{H}({\xi }_t,P_t)\hfill \\ {\dot{P}}_t=-D_{\xi }\mathcal{H}({\xi }_t,P_t).\hfill \end{array}\right.$$

(34)

Suppose Equation (34) has a solution that satisfies an initial condition given by:

$${\mathbb{P}}_{{\xi }_0}={\mu }_0,$$

and a terminal one involving Bellman equation given by:

$$P_T={\nabla }_w\overline{\Phi }\left({\xi }_T\right).$$

We also assume that $\overline{\theta }(\xi ,P)$ is the optimal control achieving the infimum in (21), as an interior point of $\Theta$, then we can explicitly write the equation of the Hamiltonian w.r.t. the optimal control as:

$$H=\mathbb{E}[P·\overline{f}(\xi ,\overline{\theta }(\xi ,P))+\overline{L}(\xi ,\overline{\theta }(\xi ,P))].$$

Therefore, by the first order condition, we have:

$$\mathbb{E}[{\nabla }_{\theta }\overline{f}(\xi ,\overline{\theta }(\xi ,P))P+{\nabla }_{\theta }\overline{L}(\xi ,\overline{\theta }(\xi ,P))]=0,$$

so that, taking into consideration Equation (34), we obtain the Hamilton-type equations:

$$\left\{\begin{array}{c}{\dot{\xi }}_t=\overline{f}({\xi }_t,\overline{\theta }({\xi }_t,P_t))\hfill \\ {\dot{P}}_t=-{\nabla }_w\overline{f}({\xi }_t,\overline{\theta }({\xi }_t,P_t))P_t-{\nabla }_w\overline{L}({\xi }_t,\overline{\theta }({\xi }_t,P_t)).\hfill \end{array}\right.$$

(35)

Use $w=(x,y)$ as concatenated variable and ${\theta }^{*}=\overline{\theta }({\xi }_t,P_t)$ to remark that the last l components of $\overline{f}$ are zero and by considering only the first d components:

$$\left\{\begin{array}{c}{\dot{x}}_t=f({\xi }_t,{\theta }^{*})\hfill \\ {\dot{p}}_t=-{\nabla }_{x}f(x_t,{\theta }^{*})p_t-{\nabla }_{x}L(x_t,{\theta }_t^{*}).\hfill \end{array}\right.$$

(36)

Summing up: Hamilton’s equation of the system (36) can be viewed as the characteristic equations of the HJB equation in its lifted formulation described by Equation (23). Essentially, the PMP gives a necessary condition for any characteristic of the HJB equation: any characteristic originating from ${\mu }_0$, that is the initial law of the random variables, must satisfy a local necessary optimal condition constituted by the mean-field PMP. This justifies the claim that the PMP constitutes a local condition if compared to the HJB equation.

3.5. Small-Time Uniqueness

A natural question is to understand when the PMP solutions also provide sufficient conditions to have optimality. We start by considering that the uniqueness of the solution implies sufficiency, and we investigate which assumptions are needed to have a unique solution of the mean-field PMP equations. Equations (27) and (28) model highly non-linear two-point boundary value problems in terms of $x^{*}$ and $p^{*}$, which are also coupled through their laws. In general, even without the coupling, this kind of PDE is known to not have a unique solution; see, e.g., Ch. 7 of [22]. In order to prove the uniqueness, strong assumptions are needed to prove the small-time case.

Theorem 3

(Small-time uniqueness). Suppose that:

• f is bounded;
• f, L and Φ are continuously differentiable w.r.t. both x and θ with bounded and Lipschitz partial derivatives;
• the distribution ${\mu }_0$ has bounded support in ${\mathbb{R}}^d\times {\mathbb{R}}^l$, which means there exists $M>0$ such that $\mu \{(x,y)\in {\mathbb{R}}^d\times {\mathbb{R}}^l:|\left|x\right||+|\left|y\right||\le 1\}=1$;
• $H(x,p,\theta )$ is strongly concave in θ and uniformly in x, $p\in {\mathbb{R}}^d$, i.e., ${\nabla }_{xx}^{2}H(x,p,\theta )+{\lambda }_0⪯0$ for some ${\lambda }_0\ge 0$.

Then, for sufficiently small T, if ${\theta }^{*1}$ and ${\theta }^{*2}$ are solutions of the mean-field PMP derived in Theorem 2, then ${\theta }^{*1}={\theta }^{*2}$.

Before proving the theorem, we report the following lemma, which provides an estimate of the difference between flow-maps driven by two different controls in terms of the small-time parameter T:

Lemma 1.

Let ${\theta }^1$, ${\theta }^2\in L^{\infty }([0,T],\Theta )$. Then, there exists a constant $T_0$ such that for all $T\in [0,T_0)$, it holds that:

$$\left|\right|x^{{\theta }^1}-x^{{\theta }^2}{\left|\right|}_{L^{\infty }}+\left|\right|p^{{\theta }^1}-p^{{\theta }^2}{\left|\right|}_{L^{\infty }}\le C\left(T\right)\left|\right|{\theta }^1-{\theta }^2{\left|\right|}_{L^{\infty }},$$

where $C\left(T\right)>0$ satisfies $C\left(T\right)\to 0$ as $T\to 0$.

Proof of Lemma 1. 

We denote $\delta \theta :={\theta }^1-{\theta }^2$, $\delta x:=x^{{\theta }^1}-x^{{\theta }^2}$ and $\delta p=p^{{\theta }^1}-p^{{\theta }^2}$, respectively. Since $x_0^{{\theta }^1}=x_2^{{\theta }^1}=x_0$, integrating the respective ODEs while considering the first two assumptions of Theorem 6 leads to:

$$\left|\right|\delta x_t\left|\right|\le {\int }_0^t\left|\right|f(x_s^{{\theta }^1},{\theta }_s^1)-f(x_s^{{\theta }^1},{\theta }_s^2)\left|\right|ds\le K_L{\int }_0^T\left|\right|\delta x_s\left|\right|ds+K_L{\int }_0^T\left|\right|\delta p_s\left|\right|ds,$$

and so:

$${\left|\right|\delta x\left|\right|}_{L^{\infty }}\le K_L{T\left|\right|\delta x\left|\right|}_{\infty }+K_L{T\left|\right|\delta \theta \left|\right|}_{\infty }.$$

Now, if $T\le T_0:={\displaystyle \frac{1}{K_L}}$. We then have:

$${\left|\right|\delta x\left|\right|}_{L^{\infty }}\le {\displaystyle \frac{K_{L}T}{1-K_{L}T}}{\left|\right|\delta \theta \left|\right|}_{L^{\infty }}.$$

(37)

Similarly:

$$\left|\right|\delta p_t\left|\right|\le K_L\left|\right|\delta x_T\left|\right|+K_L{\int }_t^T\left|\right|\delta x_s\left|\right|ds+K_L{\int }_t^T\left|\right|\delta p_s\left|\right|ds$$

$$\left|\right|\delta p\left|\right|\le (K_L+K_{L}T){\left|\right|\delta x\left|\right|}_{L^{\infty }}+K_L{T\left|\right|\delta p\left|\right|}_{L^{\infty }},$$

and hence:

$${\left|\right|\delta p\left|\right|}_{L^{\infty }}\le {\displaystyle \frac{K_L(1+T)}{1-K_{L}T}}{\left|\right|\delta x\left|\right|}_{L^{\infty }},$$

which combined with Equation (37) proves the lemma.  □

Through the above estimate, it is now possible to prove Theorem 3.

Proof of Theorem 3. 

In what follows, we exploit [5] (p. 29). By uniform strong concavity, the function $\theta \to {\mathbb{E}}_{{\mu }_0}H(x_t^{{\theta }^1},p_t^{{\theta }^1},\theta )$ is strongly concave. Thus, consider a ${\lambda }_0>0$ such that:

$${\displaystyle \frac{{\lambda }_0}{2}}\left|\right|{\theta }_t^1-{\theta }_t^2{\left|\right|}^2\le \left[{\mathbb{E}}_{{\mu }_0}\nabla H(x_t^{{\theta }^1},p_t^{{\theta }^1},{\theta }_t^2)-{\mathbb{E}}_{{\mu }_0}\nabla H(x_t^{{\theta }^1},p_t^{{\theta }^1},{\theta }_t^1)\right]·({\theta }_t^1-{\theta }_t^2).$$

A similar expression holds for $\theta \to {\mathbb{E}}_{{\mu }_0}H(x_t^{{\theta }^2},p_t^{{\theta }^2},\theta )$, and by combining the two inequalities, also exploiting the smoothness of functions involved, we have:

$$\begin{array}{cc}\hfill {\lambda }_0\left|\right|{\theta }_t^1-{\theta }_t^2{\left|\right|}^2& \le \left[{\mathbb{E}}_{{\mu }_0}\nabla H(x_t^{{\theta }^1},p_t^{{\theta }^1},{\theta }_t^2)-{\mathbb{E}}_{{\mu }_0}\nabla H(x_t^{{\theta }^1},p_t^{{\theta }^1},{\theta }_t^1)\right]·({\theta }_t^1-{\theta }_t^2)\hfill \\ & +\left[{\mathbb{E}}_{{\mu }_0}\nabla H(x_t^{{\theta }^2},p_t^{{\theta }^2},{\theta }_t^1)-{\mathbb{E}}_{{\mu }_0}\nabla H(x_t^{{\theta }^2},p_t^{{\theta }^2},{\theta }_t^2)\right]·({\theta }_t^1-{\theta }_t^2)\hfill \\ \hfill & \le {\mathbb{E}}_{{\mu }_0}\left|\right|\nabla H(x_t^{{\theta }^1},p_t^{{\theta }^1},{\theta }_t^1)-\nabla H(x_t^{{\theta }^2},p_t^{{\theta }^2},{\theta }_t^1)\left|\right|\left|\right|{\theta }_t^1-{\theta }_t^2\left|\right|\hfill \\ \hfill & +{\mathbb{E}}_{{\mu }_0}\left|\right|\nabla H(x_t^{{\theta }^1},p_t^{{\theta }^1},{\theta }_t^2)-\nabla H(x_t^{{\theta }^2},p_t^{{\theta }^2},{\theta }_t^2)\left|\right|\left|\right|\le {\theta }_t^1-{\theta }_t^2\left|\right|\hfill \\ \hfill & K_L{\left|\right|\delta \theta \left|\right|}_{L^{\infty }}{\left(\right|\left|\delta x\right||}_{L^{\infty }}+{\left|\right|\delta p\left|\right|}_{L^{\infty }}).\hfill \end{array}$$

Combining the above inequality with Lemma 1, we have:

$${\left|\right|\delta \theta \left|\right|}_{L^{\infty }}^2\le {\displaystyle \frac{K_L}{{\lambda }_0}}{C\left(T\right)\left|\right|\delta \theta \left|\right|}_{L^{\infty }}^2.$$

However, $C\left(T\right)=o\left(1\right)$, and by taking T sufficiently small, so that $K_{L}C\left(T\right)<{\lambda }_0$, this implies that ${\left|\right|\delta \theta \left|\right|}_{L^{\infty }}=0$. For more details, see [5] (Lemma 2, p. 28).  □

Within the ML setting, the small-period T roughly corresponds to the regime where the reachable set of the forward dynamics is small. This can be interpreted as the case where the model has low capacity or low expressive power. In other words, Theorem 3 states that the optimal solution is unique if the network capacity is low, even when there is a huge number of parameters. It is essential to underline that the hypothesis of the strong concavity of the Hamiltonian H does not imply convexity for the loss function J, which can also be highly non-convex due to the non-linear transformation introduced by the activation functions $\sigma$.

4. Conclusions

In this paper, typical NN structures are considered as the evolution of a dynamical system to rigorously state the population risk minimization problem related to DL. Two parallel and connected perspectives are followed. The result expressed in Theorem 2 represents the generalization of the PMP in the calculus of variations and a local characterization of optimal trajectories derived from HJB Equation (21). Deriving the necessary condition for the optimality of the PMP provides several advantages: there is no reference to the derivative w.r.t. the probability measure in the Wasserstein sense; the parameter set $\Theta$ is highly general; maximization is point-wise in t, once $x^{\star }$ and $p^{\star }$ are known. Moreover, it allows the possibility to develop a learning algorithm without referring to the classical methods of DL such as the SGD. As an interest point with a general relevance within the optimal control theory, we remark that the controls, and then the weights $\theta$, are only assumed to be measurable and essentially bounded in time. In particular, we allow for discontinuous terms. Even in the case where the number of parameters is infinite, as in the MFG setting, it is feasible to derive non-trivial stability estimates. This aspect results in contrast with generalized bounds based on classical statistical learning, where the increasing number of parameters adversely affects the generalization. On the other hand, it will be interesting to consider algorithms with weights having only discrete values, e.g., binary, values, requiring small memory amounts, with corresponding efficiency and speed improvements. A further direction of our ongoing research is based on the study of the PMP from a discrete-time perspective in order to relate the theoretical framework directly to the applications.