No Statistical-Computational Gap in Spiked Matrix Models with Generative Network Priors ^†

Abstract

We provide a non-asymptotic analysis of the spiked Wishart and Wigner matrix models with a generative neural network prior. Spiked random matrices have the form of a rank-one signal plus noise and have been used as models for high dimensional Principal Component Analysis (PCA), community detection and synchronization over groups. Depending on the prior imposed on the spike, these models can display a statistical-computational gap between the information theoretically optimal reconstruction error that can be achieved with unbounded computational resources and the sub-optimal performances of currently known polynomial time algorithms. These gaps are believed to be fundamental, as in the emblematic case of Sparse PCA. In stark contrast to such cases, we show that there is no statistical-computational gap under a generative network prior, in which the spike lies on the range of a generative neural network. Specifically, we analyze a gradient descent method for minimizing a nonlinear least squares objective over the range of an expansive-Gaussian neural network and show that it can recover in polynomial time an estimate of the underlying spike with a rate-optimal sample complexity and dependence on the noise level.

1. Introduction

One of the fundamental problems in statistical inference and signal processing is the estimation of a signal given noisy high dimensional data. A prototypical example is provided by spiked matrix models where a signal $y_{🟉}\in {\mathbb{R}}^n$ is to be estimated from a matrix Y taking one of the following forms:

• Spiked Wishart Model in which $Y\in {\mathbb{R}}^{N\times n}$ is given by:

  $$Y=u{y_{🟉}}^{\mathsf{T}}+\sigma \mathcal{Z},$$

  (1)

  where $\sigma >0$, $u\sim \mathcal{N}(0,I_N)$, ${\mathcal{Z}}_{ij}$ are i.i.d. from $\mathcal{N}(0,1)$, and u and Z are independent;
• Spiked Wigner Model in which $Y\in {\mathbb{R}}^{n\times n}$ is given by:

  $$Y=y_{🟉}{y_{🟉}}^{\mathsf{T}}+\nu \mathcal{H}$$

  (2)

  where $\nu >0$, $\mathcal{H}\in {\mathbb{R}}^{n\times n}$ is drawn from a Gaussian Orthogonal Ensemble GOE$\left(n\right)$, that is, ${\mathcal{H}}_{ii}\sim \mathcal{N}(0,2/n)$ for all $1\le i\le n$ and ${\mathcal{H}}_{ij}={\mathcal{H}}_{ji}\sim \mathcal{N}(0,1/n)$ for $1\le j<i\le n$.

In the last 20 years, spiked random matrices have been extensively studied, as they serve as a mathematical model for many signal recovery problems such as PCA [1,2,3,4], synchronization over graphs [5,6,7] and community detection [8,9,10]. Furthermore, these models are archetypal examples of the trade-off between statistical accuracy and computational efficiency. From a statistical perspective, the objective is to understand how the choice of the prior on $y_{🟉}$ determines the critical signal-to-noise ratio (SNR) and number of measurements above which it becomes information-theoretically possible to estimate the signal. From a computational perspective, the objective is to design efficient algorithms that leverage such prior information. A recent and vast body of literature has shown that depending on the chosen prior, gaps can arise between the minimum SNR required to solve the problem and the one above which known polynomial-time algorithms succeed. An emblematic example is provided the Sparse PCA problem where the signal $y_{🟉}$ in (1) is taken to be s-sparse. In this case $N=\mathcal{O}\left(s\log n\right)$ number of samples are sufficient for estimating $y_{🟉}$ [2,4], while the best known efficient algorithms require $N=\mathcal{O}\left(s^2\right)$ [3,11,12]. This gap is believed to be fundamental. This “statistical-computational gap” has been observed also for Spiked Wigner models (2) and, in general, for other structured signal recovery problems where the prior imposed has a combinatorial flavor (see the next section and [13,14] for surveys).

Motivated by the recent advances of deep generative networks in learning complex data structures, in this paper we study the spiked random matrix models (1) and (2), where the planted signal $y_{🟉}$ has a generative network prior. We assume that a generative neural network $G:{\mathbb{R}}^k\to {\mathbb{R}}^n$ with $k<n$, has been trained on a data set of spikes, and the unknown spike $y_{🟉}\in {\mathbb{R}}^n$ lies on the range of G, that is, we can write $y_{🟉}=G\left(x_{🟉}\right)$ for some $x_{🟉}\in {\mathbb{R}}^k$. As a mathematical model for the trained G, we consider a network of the form:

$$G\left(x\right)=\mathrm{relu}(W_d\dots \mathrm{relu}\left(W_2\mathrm{relu}\left(W_{1}x\right)\right)\dots ),$$

(3)

with weight matrices $W_i\in {\mathbb{R}}^{n_i\times n_{i-1}}$ and $\mathrm{relu}\left(x\right)=\max (x,0)$ is applied entrywise. We furthermore assume that the network is expansive, that is, $n=n_d>n_{d-1}>\cdots >n_0=k$, and the weights have Gaussian entries. These modeling assumptions and their variants were used in [15,16,17,18,19,20].

Enforcing generative network priors has led to substantially fewer measurements needed for signal recovery than with traditional sparsity priors for a variety of signal recovery problems [17,21,22]. In the case of phase retrieval, [17,23] have shown that under the generative prior (3), efficient compressive phase retrieval is possible with sample complexity proportional (up to log factors) to the underlying signal dimensionality k. In contrast, for a sparsity-based prior, the best known polynomial time algorithms (convex methods [24,25,26], iterative thresholding [27,28,29], etc.) require a sample complexity proportional to the square of the sparsity level for stable recovery. Given that generative priors lead to no computational-statistical gap with compressive phase retrieval, one might anticipate that they will close other computational-statistical gaps as well. Indeed, [30] analyzed the spiked models (1) and (2) under a generative network prior similar to (3) and observed no computational-statistical gap in the asymptotic limit $k,n,N\to \infty$ with $n/k=\mathcal{O}\left(1\right)$ and $N/n=\mathcal{O}\left(1\right)$. For more details on this work and on the comparison of sparsity and generative priors, see Section 2.2.

Our Contribution

In this paper we analyze the spiked matrix models (1) and (2) under a generative network prior in the nonasymptotic, finite data regime. We consider a d-layer feedforward generative network $G:{\mathbb{R}}^k\to {\mathbb{R}}^n$ with architecture (3). We furthermore assume that the planted spike $y_{🟉}\in {\mathbb{R}}^n$ lies on the range of G, that is, there exists a latent vector $x_{🟉}\in {\mathbb{R}}^k$ such that $y_{🟉}=G\left(x_{🟉}\right)$.

To estimate $y_{🟉}$, we first find an estimate $\widehat{x}$ of the latent vector $x_{🟉}$ and then use $G\left(\widehat{x}\right)$ to estimate $y_{🟉}$. We thus consider the following minimization problem (under the conditions on the generative network specified below, it was shown in [15] that G is invertible and there exists a unique $x_{🟉}$ that satisfies $y_{🟉}=G\left(x\right)$:

$$\underset{x\in {\mathbb{R}}^k}{min}f\left(x\right):=\frac{1}{4}{\Vert G\left(x\right)G{\left(x\right)}^{\mathsf{T}}-M\Vert }_F^2,$$

(4)

where:

• for the Wishart model (1) we take $M={\Sigma }_N-{\sigma }^2{I}_n$ with ${\Sigma }_N=Y^{\mathsf{T}}Y/N$
• for the Wigner model (2) we take $M=Y$.

Despite the non-convexity and non-smoothness of the problem, our preliminary work in [31] shows that when the generative network G is expansive and has Gaussian weights, (4) enjoys a favorable optimization geometry. Specifically, every nonzero point outside two small neighborhoods around $x_{🟉}$ and a negative multiple of it, has a descent direction which is given a.e. by the gradient of f. Furthermore, in [31] it is shown that the the global minimum of f lies in the neighborhoods around $x_{🟉}$ and has optimal reconstruction error. This result suggests that a first order optimization algorithm can succeed in efficiently solving (4), and no statistical-computational gap is present for the spiked matrix models with a (random) generative network prior in the finite data regime. In the current paper, we prove this conjecture by providing a polynomial-time subgradient method that minimizes the non-convex problem (4) and obtains information-theoretically optimal error rates.

Our main contribution can be summarized as follows. We analyze a subgradient method (Algorithm 1) for the minimization of (4) and show that after a polynomial number of steps $\tilde{T}$ and up to polynomials factors in the depth d of the network, the iterate $x_{\tilde{T}}$ satisfies the following reconstruction errors:

• in the Spiked Wishart Model:

  $$\Vert G\left(x_{\tilde{T}}\right)-y_{🟉}{\Vert }_2\lesssim \left(1+\frac{{\sigma }^2}{\Vert y_{🟉}{\Vert }_2^2}\right)\sqrt{\frac{k\log \left(n\right)}{N}}{\Vert y_{🟉}\Vert }_2$$

  (5)

  in the regime $N\gtrsim k\log \left(n\right)$;
• in the Spiked Wigner Model:

  $$\Vert G\left(x_{\tilde{T}}\right)-y_{🟉}{\Vert }_2\lesssim \frac{\nu }{\Vert y_{🟉}{\Vert }_2^2}\sqrt{\frac{k\log \left(n\right)}{n}}{\Vert y_{🟉}\Vert }_2.$$

  (6)

We notice that these bounds are information-theoretically optimal up to the log factors in n, and correspond to the best achievable in the case of a k-dimensional subspace prior. In particular, they imply that efficient recovery in the Wishart model is possible with a number of samples N proportional to the intrinsic dimension of the signal $y_{🟉}$. Similarly, the bound in the Spiked Wigner Model implies that imposing a generative network prior leads to a reduction of the noise by a factor of $k/n$.

Algorithm 1: Subgradient method for the minizimization problem (4)

2. Related Work

2.1. Sparse PCA and Other Computational-Statistical Gaps

A canonical problem in Statistics is finding the directions that explain most of the variance in a given cloud of data, and it is classically solved by Principal Component Analysis. Spiked covariance models were introduced in [1] to study the statistical performance of this algorithm in the high dimensional regime. Under a spiked covariance model it is assumed that the data are of the form:

$$y_i=u_i{y}_{🟉}+\sigma z_i,$$

(7)

where $\sigma >0$, $u_i\sim \mathcal{N}(0,1)$ and $z_i\sim \mathcal{N}(0,I_n)$ are independent and identically distributed, and $y_{🟉}$ is the unit-norm planted spike. Each $y_i$ is an i.i.d. sample from a centered Gaussian $\mathcal{N}(0,\Sigma )$ with spiked covariance matrix given by $\Sigma =y_{🟉}{y_{🟉}}^{\mathsf{T}}+{\sigma }^2{I}_n$, with $y_{🟉}$ being the direction that explains most of the variance. The estimate of $y_{🟉}$ provided by PCA is then given by the leading eigenvector $\widehat{y}$ of the empirical covariance matrix ${\Sigma }_N=\frac{1}{N}{\sum }_{i=1}^N{y}_i{y}_i^{\mathsf{T}}$, and standard techniques from high dimensional probability can be used to show that (we write $f\left(n\right)\gtrsim g\left(n\right)$ if $f\left(n\right)\ge Cn$ for some constant $C>0$ that might depend $\sigma$ and $\Vert y_{🟉}{\Vert }^2$. Similarly for $f\left(n\right)\lesssim g\left(n\right)$ as long as $N\gtrsim n$,

$$\underset{ϵ=\pm }{min}{\Vert ϵ\widehat{y}-y_{🟉}\Vert }_2\lesssim \sqrt{\frac{n}{N}},$$

(8)

with overwhelming probability. Note incidentally that the data matrix $Y\in {\mathbb{R}}^{N\times n}$ with rows ${\left\{y_i^{\mathsf{T}}\right\}}_i$ can be written as (1).

Bounds of the form (8), however, become uninformative in modern high dimensional regimes where the ambient dimension of the data n is much larger than, or on the order of, the number of samples N. Even worse, in the asymptotic regime $n/N\to c>0$ and for ${\sigma }^2$ large enough, the spike $y_{🟉}$ and the estimate $\widehat{y}$ become orthogonal [32], and minimax techniques show that no other estimators based solely on the data (7), can achieve better overlap with $y_{🟉}$ [33].

In order to obtain consistent estimates and lower the sample complexity of the problem, therefore, additional prior information on the spike $y_{🟉}$ has to be enforced. For this reason, in recent years various priors have been analyzed such as positivity [34], cone constraints [35] and sparsity [32,36]. In the latter case $y_{🟉}$ is assumed to be s-sparse, and it can be shown (e.g., [33]) that for $N\gtrsim s\log n$ and $n\gtrsim s$, the s-sparse largest eigenvector ${\widehat{y}}_s$ of ${\Sigma }_N$

$${\widehat{y}}_s=\underset{y\in S_2^{n-1},{\Vert y\Vert }_0\le s}{argmax}{y}^{\mathsf{T}}{\Sigma }_{N}y$$

Satisfies with high probability the condition:

$$\underset{ϵ=\pm }{min}{\Vert ϵ{\widehat{y}}_s-y_{🟉}\Vert }_2\lesssim \sqrt{\frac{s\log n}{N}}.$$

This implies, in particular, that the signal $y_{🟉}$ can be estimated with a number of samples that scales linearly with its intrinsic dimension s. These rates are also minimax optimal; see for example [4] for the mean squared error and [2] for the support recovery. Despite these encouraging results, no currently known polynomial time algorithm achieves such optimal error rates and, for example, the covariance thresholding algorithm of [37] requires $N\gtrsim s^2$ samples in order to obtain exact support recovery or estimation rate

$$\underset{ϵ=\pm }{min}{\Vert ϵ{\widehat{y}}_s-y_{🟉}\Vert }_2\lesssim \sqrt{\frac{s^2\log n}{N}},$$

as shown in [3]. In summary, only computationally intractable algorithms are known to reach the statistical limit $N=\Omega \left(s\right)$ for Sparse PCA, while polynomial time methods are only sub-optimal, requiring $N=\Omega \left(s^2\right)$. Notably, [38] provided a reduction of Sparse PCA to the planted clique problem which is conjectured to be computationally hard.

Further strong evidence for the hardness of sparse PCA have been given in a series of recent works [39,40,41,42,43]. Other computational-statistical gaps have also been found and studied in a variety of other contexts such as sparse Gaussian mixture models [44], tensor principal component analysis [45], community detection [46] and synchronization over groups [47]. These works fit in the growing and important body of literature aiming at understanding the trade-offs between statistical accuracy and computational efficiency in statistical inverse problems.

We finally note that many of the above mentioned problems can be phrased as recovery of a spike vector from a spiked random matrix. The difficulty can be viewed as arising from simultaneously imposing low-rankness and additional prior information on the signal (sparsity in case of Sparse PCA). This difficulty can be found in sparse phase retrieval as well. For example, [25] has shown that $m=\mathcal{O}\left(s\log n\right)$ number of quadratic measurements are sufficient to ensure well-posedness of the estimation of an s-sparse signal of dimension n lifted to a rank-one matrix, while $m\ge \mathcal{O}(s^2/{\log }^{2}n)$ measurements are necessary for the success of natural convex relaxations of the problem. Similarly, [48] studied the recovery of simultaneously low-rank and sparse matrices, showing the existence of a gap between what can be achieved with convex and tractable relaxations and nonconvex and intractable methods.

2.2. Inverse Problems with Generative Network Priors

Recently, in the wake of successes of deep learning, generative networks have gained popularity as a novel approach for encoding and enforcing priors in signal recovery problems. In one deep-learning-based approach, a dataset of “natural signals” is used to train a generative network in an unsupervised manner. The range of this network defines a low-dimensional set which, if successfully trained, contains or approximately contains, target signals of interest [19,21]. Non-convex optimization methods are then used for recovery by optimizing over the range of the network. We notice that allowing the algorithms the complete knowledge of the generative network architecture and of the learned weights is roughtly analogous to allowing sparsity-based algorithms the knowledge of the basis or frame in which the signal is modeled as sparse.

The use of generative network for signal recovery has been successfully demonstrated in a variety of settings such as compressed sensing [21,49,50], denoising [16,51], blind deconvolution [22], inpainting [52] and many more [53,54,55,56]. In these papers, generative networks significantly outperform sparsity based priors at signal reconstruction in the low-measurement regime. This fundamentally leverages the fact that a natural signal can be represented more concisely by a generative network than by a sparsity prior under an appropriate basis. This characteristic has been observed even in untrained generative networks where the prior information is encoded only in the network architecture and has been used to devise state-of-the-art signal recovery methods [57,58,59].

Parallel to these empirical successes, a recent line of works have investigated theoretical guarantees for various statistical estimation tasks with generative network priors. Following the work of [15,21] gave global guarantees for compressed sensing, followed then by many others for various inverse problems [19,20,50,51,55]. In particular, in [17] the authors have shown that $m=\Omega \left(k\log n\right)$ number of measurements are sufficient to recover a signal from random phaseless observations, assuming that the signal lies on the range of a generative network with latent dimension k. The same authors have then provided in [23] a polynomial time algorithm for recovery under the previous settings. Note that, contrary to the sparse phase retrieval problem, generative priors for phase retrieval allow for efficient algorithms with optimal sample complexity, up to logarithmic factors, with respect to the intrinsic dimension of the signal.

Further theoretical advances in signal recovery with generative network priors have been spurred by using techniques from statistical physics. Recently, [30] analyzed the spiked matrix models (1) and (2) with $y_{🟉}$ in the range of a generative network with random weights, in the asymptotic limit $k,n,N\to \infty$ with $n/k=\mathcal{O}\left(1\right)$ and $N/n=\mathcal{O}\left(1\right)$. The analysis is carried out mainly for networks with sign or linear activation functions in the Bayesian setting where the latent vector is drawn from a separable distribution. The authors of [30] provide an Approximate Message Passing and a spectral algorithm, and they numerically observe no statistical-computational gap as these polynomial time methods are able to asymptotically match the information-theoretic optimum. In this asymptotic regime, [60] further provided precise statistical and algorithmic thresholds for compressed sensing and phase retrieval.

3. Algorithm and Main Result

In this section we present an efficient and statistically-optimal algorithm for the estimation of the signal $y_{🟉}$ given a spiked matrix Y of the form (1) or (2). The recovery method is detailed in Algorithm 1, and it is based on the direct optimization of the nonlinear least squares problem (4).

Applied in [16] for denoising and compressed sensing under generative network priors, and later used in [23] for phase retrieval, the first order optimization method described in Algorithm 1 leverages the theory of Clarke subdifferentials (the reader is referred to [61] for more details). As the objective function f is continuous and piecewise smooth, at every point $x\in {\mathbb{R}}^k$ it has a Clarke subdifferential given by

$$\partial f\left(x\right)=\mathrm{conv}\{v_1,v_2,\dots ,v_T\},$$

(9)

where conv denotes the convex hull of the vectors $v_1,\dots ,v_T$, which are respectively the gradient of the T smooth functions adjoint at x. The vectors $v_x\in \partial f\left(x\right)$ are the subgradients of f at x, and at a point x where f is differentiable it holds that $\partial f\left(x\right)=\{\nabla f(x\left)\right\}$.

The reconstruction method presented in Algorithm 1 is motivated by the landscape analysis of the minimization problem (4) for a network G with sufficiently-expansive Gaussian weights matrices. Under this assumption, we showed in [31] that (4) has a benign optimization geometry and in particular that for any nonzero point outside a neighborhood of $x_{🟉}$ and a negative multiple of it, any subgradient of f is a direction of strict descent. Furthermore we showed that the points in the vicinity of the spurious negative multiple of $x_{🟉}$ have function values strictly larger than those close to $x_{🟉}$. Figure 1 shows the expected value of f in the noiseless case, $\nu =0$ and $N\to \infty$, for a generative network with latent dimension $k=2$. This plot highlights the global minimimum at $x_{🟉}=[1,1]$, and the flat region in near a negative multiple of $x_{🟉}$.

Figure 1. Expected value, with respect to the weights, of the objective function f in (4) in the noiseless case (see (16) for explicit formula), for a network with latent dimension $k=2$ and $x_{🟉}=[1,1]$.

At each step, the subgradient method in Algorithm 1 checks if the current iterate $x_i$ has a larger loss value than its negative multiple, and if so negates $x_i$. As we show in the proof of our main result, this step will ensure that the algorithm will avoid the neighborhood around the spurious negative multiple of $x_{🟉}$ and will converge to the neighborhood around $x_{🟉}$ in a polynomial number of steps.

Below we make the following assumptions on the weight matrices of G.

Assumption 1.

The generative network G defined in (3), has weights $W_i\in {\mathbb{R}}^{n_i\times n_{i-1}}$ with i.i.d. entries from $\mathcal{N}(0,1/n_i)$ and satisfying the expansivity condition with constant $ϵ>0$:

$$n_{i+1}\ge c{n}_i{ϵ}^{-2}\log (1/ϵ)$$

(10)

for all i and a universal constant $c>0$.

We note that in [31] the expansivity condition was more stringent, requiring an additional log factor. Since the publication of our paper, [62] has shown that the more relaxed assumption (10) suffices for ensuring a benign optimization geometry. Under Assumption 1, our main theorem below shows that the subgradient method in Algorithm 1 can estimate the spike $y_{🟉}$ with optimal sample complexity and in a polynomial number of steps.

Theorem 1.

Let $x_{🟉}\in {\mathbb{R}}^k$ nonzero and $y_{🟉}=G\left(x_{🟉}\right)$ where G is a generative network satisfying Assumption 1 with $ϵ\le K_1/d^{96}$. Consider the minimization problem (4) and assume that the noise level ω satisfies $\omega \le K_2{\Vert x_{🟉}\Vert }_2^2{2}^{-d}/d^{44}$ where:

• for theSpiked Wishart Model(1) take $M={\Sigma }_N-{\sigma }^2{I}_n$, and

  $$\omega :=(\Vert y_{🟉}{\Vert }_2^2+{\sigma }^2)\max \left\{\sqrt{\frac{338k\log (3n_1^d{n}_2^{d-1}\dots n_{d-1}^{2}n)}{N}},\frac{156k\log (3n_1^d{n}_2^{d-1}\dots n_{d-1}^{2}n)}{N}\right\};$$
• for theSpiked Wigner Model(2) take $M=Y$, and

  $$\omega :=\nu \sqrt{\frac{169k\log (3n_1^d{n}_2^{d-1}\dots n_{d-1}^{2}n)}{n}}.$$

Consider Algorithm 1 with nonzero $x_0$ and $\Vert x_0{\Vert }_2<R_{🟉}$ where $R_{🟉}\ge 5{\Vert x_{🟉}\Vert }_2/\left(2\sqrt{2}\right)$, and stepsize $\mu =2^{2d}{K}_3/(8d^4{R}_{🟉}^2)$. Then with probability at least $1-2e^{-k\log n}-{\sum }_{i=1}^d{e}^{-C{n}_{i-1}}$, $0<\Vert x_i{\Vert }_2<R_{🟉}$ for any $i\ge 1$, there exists an integer $T\le K_{4}f\left(x_0\right)2^{2d}/(R_{🟉}^4{d}^4ϵ)$ such that for any $i\ge T$:

$$\begin{array}{cc}\hfill \Vert x_{i+1}-x_{🟉}{\Vert }_2& \le {\mathsf{\rho }}_1^{i+1-T}{\Vert x_T-x_{🟉}\Vert }_2+{\mathsf{\rho }}_2\frac{2^d}{\Vert x_{🟉}{\Vert }_2}\omega \hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill \Vert G\left(x_{i+1}\right)-y_{🟉}{\Vert }_2& \le \frac{1.2}{2^{d/2}}{\mathsf{\rho }}_1^{i+1-T}{\Vert x_T-x_{🟉}\Vert }_2+1.3{\mathsf{\rho }}_2\frac{\omega }{\Vert y_{🟉}{\Vert }_2}\hfill \end{array}$$

(12)

where $C>0$, $K_1,\dots ,K_4>0$, ${\mathsf{\rho }}_1\in (0,1)$ and ${\mathsf{\rho }}_2>0$ are universal constants.

Note that the quantity $2^{2d}$ in the hypotheses and conclusions of the theorem is an artifact of the scaling of the network and it should not be taken as requiring exponentially small noise or number of steps. Indeed under Assumption 1, the ReLU activation zeros out roughly half of the entries of its argument leading to an “effective” operator norm of approximately $1/2$. We furthermore notice that the dependence of the depth d is likely quite conservative and it was not optimized in the proof as the main objective was to obtain tight dependence on the intrinsic dimension of the signal k. As shown in the numerical experiments, the actual dependence on the depth is much better in practice. Finally, observe that despite the nonconvex nature of the objective function in (4) we obtain a rate of convergence which is not directly dependent on the dimension of the signal, reminiscent of what happens in the convex case.

The quantity $\omega$ in Theorem 1 can be interpreted as the intrinsic noise level of the problem (inverse SNR). The theorem guarantees that in a polynomial number of steps the iterates of the subgradient method will converge to $x_{🟉}$ up to $\omega$. For $\tilde{T}$ large enough $G\left(x_{\tilde{T}}\right)$ will satisfy the rate-optimal error bounds (5) and (6).

Numerical Experiments

We illustrate the predictions of our theory by providing results of Algorithm 1 on a set of synthetic experiments. We consider 2-layer generative networks with ReLU activation functions, hidden layer of dimension $n_1=500$, output dimension $n_2=n=1500$ and varying number of latent dimension $k\in [40,60,100]$. We randomly sample the weights of the matrix independently from $\mathcal{N}(0,2/n_i)$ (this scaling removes that $2^d$ dependence in Theorem 1). We then consider data Y according the spiked models (1) and (2), where $x_{🟉}\in {\mathbb{R}}^k$ is chosen so that $y_{🟉}=G\left(x_{🟉}\right)$ has unit norm. For the Wishart model, we vary the number of samples N; and for the Wigner model, we vary the noise level $\nu$ so that the following quantities remain constant for the different networks with latent dimension k:

$${\theta }_{\mathrm{WS}}:=\sqrt{k\log \left(n_1^{2}n\right)/N},{\theta }_{\mathrm{WG}}:=\nu \sqrt{k\log \left(n_1^{2}n\right)/n}.$$

In Figure 2 we plot the reconstruction error given by $\Vert G\left(x\right)-y_{🟉}{\Vert }_2$ against ${\theta }_{\mathrm{WS}}$ and ${\theta }_{\mathrm{WG}}$. As predicted by Theorem 1, the errors scale linearly with respect to these control parameters, and moreover the overlap of these plots confirms that these rates are tight with respect to the order of k.

Figure 2. Reconstruction error for the recovery of a spike $y_{🟉}=G\left(x_{🟉}\right)$ in the Wishart and Wigner models with random generative network priors. Each point corresponds to the average over 50 random drawing of the network weights and samples. These plots demonstrate that the reconstruction errors follow the scalings established by Theorem 1.

4. Recovery Under Deterministic Conditions

We will derive Theorem 1 from Theorem 3, below, which is based on a set of deterministic conditions on the weights of the matrix and the noise. Specifically, we consider the minimization problem (4) with

$$M=G\left(x_{🟉}\right)G{\left(x_{🟉}\right)}^{\mathsf{T}}+H$$

for an unknown symmetric matrix $H\in {\mathbb{R}}^{n\times n}$, nonzero $x_{🟉}\in {\mathbb{R}}^k$, and a given d-layer feed forward generative network G as in (3).

In order to describe the main deterministic conditions on the generative network G, we begin by introducing some notation. For $W\in {\mathbb{R}}^{n\times k}$ and $x\in {\mathbb{R}}^k$, we define the operator $W_{+,x}:=\mathrm{diag}(Wx>0)W$ such that $\mathrm{relu}\left(Wx\right)=W_{+,x}x$. Moreover, we let $W_{1,+,x}={\left(W_1\right)}_{+,x}=\mathrm{diag}(W_{1}x>0)W_1$, and for $2\le i\le d$ we define recursively

$$W_{i,+,x}=\mathrm{diag}(W_i,{\Pi }_{j=i-1}^1{W}_{j,+,x}x>0)W_i,$$

where ${\Pi }_{i=d}^1{W}_i=W_d{W}_{d-1}\dots W_1$. Finally we let ${\Lambda }_x={\Pi }_{j=d}^1{W}_{j,+,x}$ and note that $G\left(x\right)={\Lambda }_{x}x$. With this notation we next recall the following deterministic condition on the layers of the generative network.

Definition 2

(Weight Distribution Condition [15]). We say that $W\in {\mathbb{R}}^{n\times k}$ satisfies the Weight Distribution Condition (WDC) with constant $ϵ>0$ if for all nonzero $x_1,x_2\in {\mathbb{R}}^k$:

$$\Vert W_{+,x_1}^{\mathsf{T}}{W}_{+,x_2}-Q_{x_1,x_2}{\Vert }_2\le ϵ,$$

where

$$Q_{x_1,x_2}=\frac{\pi -{\theta }_{x_1,x_2}}{2\pi }{I}_k+\frac{\sin {\theta }_{x_1,x_2}}{2\pi }{M}_{{\widehat{x}}_2\leftrightarrow {\widehat{x}}_2}$$

and ${\theta }_{x_1,x_2}=\angle (x_1,x_2)$, ${\widehat{x}}_1=x_1/{\Vert x_1\Vert }_2$, ${\widehat{x}}_2=x_2/{\Vert x_2\Vert }_2$, $I_k$ is the $k\times k$ identity matrix and $M_{{\widehat{x}}_1\leftrightarrow {\widehat{x}}_2}$ is the matrix that sends ${\widehat{x}}_1\mapsto {\widehat{x}}_2$, ${\widehat{x}}_2\mapsto {\widehat{x}}_1$, and with kernel span ${\left(\{x_1,x_2\}\right)}^{\perp }$.

Note that $Q_{x_1,x_2}$ is the expected value of $W_{+,x_1}^{\mathsf{T}}{W}_{+,x_2}$ when W has rows $w_i\sim \mathcal{N}(0,I_k/n)$, and if $x_1=x_2$ then $Q_{x_1,y_2}$ is an isometry up to the scaling factor $1/2$. Below we will say that a d-layer generative network G of the form (3), satisfies the WDC with constant $ϵ>0$ if every weight matrix $W_i$ has the WDC with constant $ϵ$ for all $i=1,\dots d$.

The WDC was originally introduced in [15], and ensures that the angle between two vectors in the latent space is approximately preserved at the output layer and, in turn, it guarantees the invertibility of the network. Assumption 1 will guarantees that the generative network G satisfies the WDC with high probability.

We are now able to state our recovery guarantees for a spike $y_{🟉}$ under deterministic conditions on the network G and noise H.

Theorem 3.

Let $d\ge 2$ and assume the generative network (3) has weights $W_i\in {\mathbb{R}}^{n_i\times n_{i-1}}$ satisfying the WDC with constant $0<ϵ\le K_1/d^{96}$. Consider Algorithm 1 with $M=G\left(x_{🟉}\right)G{\left(x_{🟉}\right)}^{\mathsf{T}}+H$, $x_{🟉}\in {\mathbb{R}}^k\backslash \left\{0\right\}$ and H a symmetric matrix satisfying:

$$\parallel {\Lambda }_x^{\mathsf{T}}H{\Lambda }_x{\parallel }_2\le \frac{\omega }{2^d},\mathit{and}\omega \le K_2\frac{\parallel x_{🟉}{\parallel }_2^2{2}^{-d}}{d^{44}}.$$

(13)

Take $x_0$ nonzero and with $\Vert x_0{\Vert }_2<R_{🟉}$ where $R_{🟉}\ge 5{\Vert x_{🟉}\Vert }_2/\left(2\sqrt{2}\right)$, $\mu =2^{2d}{K}_3/(8d^4{R}_{🟉}^2)$. Then the iterates ${\left\{x_i\right\}}_{i\ge 0}$ generated by the Algorithm 1 satisfy $0<\Vert x_i\Vert <R_{🟉}$ and obey to the following:

(A) 

there exists an integer $T\le K_4\frac{f\left(x_0\right)2^{2d}}{R_{🟉}^4{d}^4ϵ}$ such that

$$\parallel x_T-x_{🟉}{\parallel }_2\le K_5{d}^{14}\sqrt{ϵ}\parallel x_{🟉}{\parallel }_2+K_6{2}^d{d}^{10}\omega {\parallel x_{🟉}\parallel }_2^{-1}$$

(B) 

for any $i\ge T$:

$$\begin{array}{cc}\hfill \parallel x_{i+1}-x_{🟉}{\parallel }_2& \le {\mathsf{\rho }}_1^{i+1-T}{\parallel x_T-x_{🟉}\parallel }_2+{\mathsf{\rho }}_2\frac{2^d}{\parallel x_{🟉}{\parallel }_2}\omega \hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill \Vert G\left(x_{i+1}\right)-G\left(x_{🟉}\right){\Vert }_2& \le \frac{1.2}{2^{d/2}}{\mathsf{\rho }}_1^{i+1-T}{\Vert x_T-x_{🟉}\Vert }_2+1.3{\mathsf{\rho }}_2\frac{\omega }{\Vert y_{🟉}{\Vert }_2}\hfill \end{array}$$

(15)

where $K_1,\dots ,K_6>0$, ${\mathsf{\rho }}_1\in (0,1)$ and ${\mathsf{\rho }}_2>0$ are universal constants.

Theorem 1 follows then from Theorem 3 after proving that with high probability the spectral norm of ${\Lambda }_x^{\mathsf{T}}H{\Lambda }_x$, where $H=M-y_{🟉}{y_{🟉}}^{\mathsf{T}}$, can be upper bounded by $\omega /2^d$, and the weights of the network G satisfy with high probability the WDC.

In the rest of this section section we will describe the main steps and tools needed to prove Theorem 3.

4.1. Technical Tools and Outline of the Proofs

Our proof strategy for Theorem 3 can be summarized as follows:

• In Proposition A1 (Appendix A.3) we show that the iterates ${\left\{x_i\right\}}_{i=1}$ of the Algorithm 1 stay inside the Euclidean ball of radius $R_{🟉}$ and remain nonzero for all $i\ge 1$.
• We then identify two small Euclidean balls ${\mathcal{B}}_{+}$ and ${\mathcal{B}}_{-}$ around respectively $x_{🟉}$ and $-{\rho }_d{x}_{🟉}$, where ${\rho }_d\in (0,1)$ only depends on the depth of the network. In Proposition A2 we show that after a polynomial number of steps, the iterates $\left\{x_i\right\}$ of the Algorithm 1 enter the region ${\mathcal{B}}_{+}\cup {\mathcal{B}}_{-}$ (Appendix A.4).
• We show, in Proposition A3, that the negation step causes the iterates of the algorithm to avoid the spurious point $-{\rho }_d{x}_{🟉}$ and actually enter ${\mathcal{B}}_{+}$ within a polynomial number of steps (Appendix A.5).
• We finally show in Proposition A4, that in ${\mathcal{B}}_{+}$ the loss function f enjoys a favorable convexity-like property, which implies that the iterates $\left\{x_i\right\}$ will remain in ${\mathcal{B}}_{+}$ and eventually converge to $x_{🟉}$ up to the noise level (Appendix A.6).

One of the main difficulties in the analysis of a subgradient method in Algorithm 1 is the lack of smoothness of the loss function f. We show that the WDC allows us to overcome this issue by showing that the subgradients of f are uniformly close, up to the noise level, to the vector field $h_x\in {\mathbb{R}}^k$:

$$h_x:=\frac{1}{2^{2d}}\left(x{x}^{\mathsf{T}}-{\tilde{h}}_x{\tilde{h}}_x^{\mathsf{T}}\right)x,$$

where ${\tilde{h}}_x$ is continuous for nonzero x (see Appendix A.2). We show furthermore that $h_x$ is locally Lipschitz, which allows us to conclude that the gradient method decreases the value of the loss function until eventually reaching ${\mathcal{B}}_{+}\cup {\mathcal{B}}_{-}$ (Appendix A.4).

Using the WDC, we show that the loss function f is uniformly close to

$$f_E\left(x\right)=\frac{1}{2^{2d+2}}\left({\Vert x\Vert }_2^4+{\Vert x_{🟉}\Vert }_2^4-2{⟨x,{\tilde{h}}_x⟩}^2\right).$$

(16)

A direct analysis of $f_E$ reveals that its values inside ${\mathcal{B}}_{-}$ are strictly larger then those inside ${\mathcal{B}}_{+}$. This property extends to f as well, and guarantees that the gradient method will not converge to the spurious point $-{\rho }_d{x}_{🟉}$ (Appendix A.5).

Appendix A.1. Notation

We collect the notation that is used throughout the paper. For any real number a, let $\mathrm{relu}\left(a\right)=\max (a,0)$ and for any vector $v\in {\mathbb{R}}^n$, denote the entrywise application of relu as $\mathrm{relu}\left(v\right)$. Let $\mathrm{diag}(Wx>0)$ be the diagonal matrix with i-th diagonal element equal to 1 if ${\left(Wx\right)}_i>0$ and 0 otherwise. For any vector x we denote with $\Vert x\Vert$ its Euclidean norm and for any matrix A we denote with $\Vert A\Vert$ its spectral norm and with ${\Vert A\Vert }_F$ its Frobenius norm. The euclidean inner product between two vectors a and b is $⟨a,b⟩$, while for two matrices A and B their Frobenius inner product will be denoted by ${⟨A,B⟩}_F$. For any nonzero vector $x\in {\mathbb{R}}^n$, let $\widehat{x}=x/\Vert x\Vert$. For a set S we will write $\left|S\right|$ for its cardinality and $S^c$ for its complement. Let $\mathcal{B}(x,r)$ be the Euclidean ball of radius r centered at x, and $S^{k-1}$ be the unit sphere in ${\mathbb{R}}^k$. Let ${\theta }_0=\angle (x,x_{🟉})$ and for $i\ge 0$ let ${\theta }_{i+1}=g\left({\theta }_i\right)$ where g is defined in (A1). We will write $\gamma =\Omega \left(\delta \right)$ to mean that there exists a positive constant C such that $\gamma \ge C\delta$ and similarly $\gamma =\mathcal{O}\left(\delta \right)$ if $\gamma \le C\delta$. Additionally we will use $a=b+O_1\left(\delta \right)$ when $\Vert a-b\Vert \le \delta$, where the norm is understood to be the absolute value for scalars, the Euclidean norm for vectors and the spectral norm for matrices.

Appendix A.2. Preliminaries

For later convenience we will define the following vectors:

$$\begin{array}{cc}\hfill p_x& :={\Lambda }_x^{\mathsf{T}}{\Lambda }_{x}x,\hfill \\ \hfill q_x& :={\Lambda }_x^{\mathsf{T}}{\Lambda }_{x_{🟉}}{x}_{🟉},\hfill \\ \hfill {\overline{v}}_x& :=[p_x{p}_x^{\mathsf{T}}-q_x{q}_x^{\mathsf{T}}]x,\hfill \\ \hfill {\eta }_x& :={\Lambda }_x^{\mathsf{T}}H{\Lambda }_{x}x.\hfill \end{array}$$

Note then that when f is differentiable at x, then ${\tilde{v}}_x:=\nabla f\left(x\right)={\overline{v}}_x-{\eta }_x$ and in particular when $H=0$ then we have ${\tilde{v}}_x={\overline{v}}_x$.

The following function controls how the angles are contracted by a ReLU layer:

$$g\left(\theta \right):={\cos }^{-1}\left(\frac{(\pi -\theta )\cos \theta +\sin \theta }{\pi }\right).$$

(A1)

As we mentioned in Section 4.1, our analysis is based on showing that the subgradients of f are uniformly close to the vector field $h_x$ given by

$$h_x:=\frac{1}{2^{2d}}\left(x{x}^{\mathsf{T}}-{\tilde{h}}_x{\tilde{h}}_x^{\mathsf{T}}\right)x,$$

(A2)

where

$${\tilde{h}}_x:=(\prod _{i=0}^{d-1}\frac{\pi -{\overline{\theta }}_i}{\pi })x_{🟉}+\sum _{i=1}^{d-1}\frac{\sin {\theta }_i}{\pi }(\prod _{j=i+1}^{d-1}\frac{\pi -{\overline{\theta }}_j}{\pi })\frac{\Vert x_{🟉}\Vert }{\Vert x\Vert }x,$$

(A3)

and ${\theta }_i:=g\left({\overline{\theta }}_{i-1}\right)$ for g given by (A1) and ${\theta }_0=\angle (x,y)$.

Lemma A1

(Lemma 8 in [15]). Suppose that $d\ge 2$ and the WDC holds with $ϵ<1/{\left(16\pi d^2\right)}^2$, then for all nonzero $x,x_{🟉}\in {\mathbb{R}}^k$,

$$\begin{array}{cc}\hfill ⟨{\Lambda }_{x}x,{\Lambda }_{x_{🟉}}{x}_{🟉}⟩& \ge \frac{1}{4\pi }\frac{1}{2^d}{\Vert x\Vert }_2\Vert x_{🟉}\Vert ,\hfill \end{array}$$

(A4)

$$\begin{array}{cc}\hfill \parallel {\Lambda }_x^{\mathsf{T}}{\Lambda }_{x_{🟉}}{x}_{🟉}-\frac{{\tilde{h}}_x}{2^d}\parallel & \le 24\frac{d^3\sqrt{ϵ}}{2^d}\parallel x_{🟉}\parallel ,\mathrm{and}\hfill \end{array}$$

(A5)

$$\begin{array}{cc}\hfill \parallel {\Lambda }_x{\parallel }^2\le \frac{1}{2^d}{(1+2ϵ)}^d\le \frac{1+4ϵd}{2^d}& \le \frac{13}{12}\frac{1}{2^d}.\hfill \end{array}$$

(A6)

Proof. 

The first two bounds can be found in [15] (Lemma 8). The third bound follows by noticing that the WDC implies:

$$\parallel {\Lambda }_x{\parallel }^2\le {\Pi }_{i=d}^1{\Vert W_{i,+,x}\Vert }^2\le \frac{1}{2^d}{(1+2ϵ)}^d\le \frac{1+4ϵd}{2^d}\le \frac{13}{12}\frac{1}{2^d},$$

where we used $\log (1+z)\le z$ and $e^z\le (1+2z)$ for all $0\le z\le 1$. □

The next lemma shows that the noiseless gradient ${\overline{v}}_x$ concentrates around $h_x$.

Lemma A2.

Suppose $d\ge 2$ and the WDC holds with $ϵ<1/{\left(16\pi d^2\right)}^2$, then for all nonzero $x,x_{🟉}\in {\mathbb{R}}^k$:

$$\parallel {\overline{v}}_x-h_x\parallel \le 86\frac{d^4\sqrt{ϵ}}{2^{2d}}\max (\parallel x_{🟉}{\parallel }^2{,\parallel x\parallel }^2)\parallel x\parallel .$$

We now use the characterization of the Clarke subdifferential given in (9) to derive a bound on the concentration of $v_x\in \partial f\left(x\right)$ around $h_x$ up to the noise level.

Lemma A3.

Under the assumptions of Lemma A2, and assuming $\parallel {\Lambda }_x^{\mathsf{T}}H{\Lambda }_x\Vert \le \omega /2^d$, for any $v_x\in \partial f\left(x\right)$:

$$\parallel v_x-h_x\Vert \le 86\frac{d^4\sqrt{ϵ}}{2^{2d}}\max (\Vert x_{🟉}{\Vert }^2{,\Vert x\Vert }^2)\Vert x\Vert +\frac{\omega }{2^d}\Vert x\Vert$$

From the above and the bound on the noise level $\omega$ we can bound the norm of the step $v_x$ in the Algorithm 1.

Lemma A4.

Under the assumptions of Lemma A3, and assuming that ω satisfies (13), for any $v_x\in \partial f\left(x\right)$:

$$\parallel v_x\Vert \le \frac{4}{2^{2d}}{d}^2{\max (\Vert x\Vert }^2,\Vert x_{🟉}{\Vert }^2)\Vert x\Vert .$$

Appendix A.3. Iterates Stay Bounded

In this section we prove that all the iterates $\left\{x_i\right\}$ generated by Algorithm 1 remain inside the Euclidean ball $\mathcal{B}(0,R_{🟉})$ where $R_{🟉}\ge \sqrt{2}{C}_{🟉}\Vert x_{🟉}\Vert$ and $C_{🟉}=5/4$.

Lemma A5.

Let the assumptions of Lemma A4 be satisfied, $\mu =K_3{2}^{2d}/(8d^4{R}_{🟉}^2)$ with $512{\pi }^2{K}_3<1$. Then for any $x\in {\mathbb{R}}^k$ with $C_{🟉}\Vert x_{🟉}\Vert <\Vert x\Vert \le R_{🟉}$ and any $\lambda \in [0,1]$, it holds that $\Vert x-\lambda \mu v_x\Vert \le \Vert x\Vert$.

From the previous lemma we can now derive the boundedness of the iterates of the Algorithm 1.

Proposition A1.

Under the assumptions of Theorem 3, if $x\in \mathcal{B}(0,R_{🟉})\backslash \left\{0\right\}$ it follows that $x-\lambda \mu v_x\in \mathcal{B}(0,R_{🟉})\backslash \left\{0\right\}$. Furthermore if $x_0\in \mathcal{B}(0,R_{🟉})\backslash \left\{0\right\}$, the iterates ${\left\{x_i\right\}}_{i=1}$ of the Algorithm 1 satisfy $x_i-\lambda \mu v_{x_i}\in \mathcal{B}(0,R_{🟉})\backslash \left\{0\right\}$ for all $i\ge 1$ and $\lambda \in [0,1]$.

Proof. 

Assume $C_{🟉}\Vert x_{🟉}\Vert <\Vert x\Vert \le R_{🟉}$, then the conclusions follows from Lemma A5. Assuming instead that $\Vert x\Vert \le C_{🟉}\Vert x_{🟉}\Vert$, note that

$$\Vert x-\lambda \mu v_x\Vert \le \Vert x\Vert +\mu \Vert v_x\Vert \le (1+\frac{1}{2d^2})\Vert x\Vert \le R_{🟉}$$

using Lemma A4, $d\ge 2$ and the assumptions on $\mu$ and $R_{🟉}$. Finally observe that if $x_i\in \mathcal{B}(0,R_{🟉})$ then the same holds for ${\tilde{x}}_i$.

Finally if for some $i\ge 0$ it was the case that $0={\tilde{x}}_i-\lambda \mu v_{{\tilde{x}}_i}$, then this would imply that $\Vert {\tilde{x}}_i\Vert =\lambda \mu \Vert v_{{\tilde{x}}_i}\Vert$ which cannot happen because by Lemma A4 and the choice of the step size it holds that $\mu \Vert v_{{\tilde{x}}_i}\Vert \le \Vert {\tilde{x}}_i\Vert /8$. □

Appendix A.4. Convergence to $S_{\beta }$

We define the set $S_{\beta }$ outside which we can lower bound the norm of $\Vert h_x\Vert$ as

$$S_{\beta }:=\{x\in {\mathbb{R}}^k|\Vert h_x\Vert \le \frac{\beta }{2^{2d}}{\max (\Vert x\Vert }^2,\Vert {x_{🟉}\Vert }^2)\Vert x\Vert \},$$

where we take

$$\beta =7·\left(86d^4\sqrt{ϵ}+2^d\omega {\Vert x_{🟉}\Vert }^{-2}\right).$$

(A7)

Outside the set $S_{\beta }$ the sub-gradients of f are bounded below and the landscape has favorable optimization geometry.

Lemma A6.

Let $x\in \mathcal{B}(0,R_{🟉})\backslash \left\{0\right\}\cap S_{\beta }^c$, then for all $v_x\in \partial f\left(x\right)$

$$\parallel v_x\Vert \ge 6\left(\frac{\max (\Vert x{\Vert }^2,\Vert x_{🟉}{\Vert }^2)\Vert x\Vert }{2^{2d}}86d^4\sqrt{ϵ}+\frac{\omega }{2^d}\Vert x\Vert \right).$$

(A8)

Moreover let $\lambda \in [0,1]$ and $x_{\lambda }=x-\lambda \mu v_{x_{\lambda }}$ then

$$\parallel v_x-v_{x_{\lambda }}\Vert \le \frac{15}{16}\Vert v_x\Vert ,$$

(A9)

for all $v_x\in \partial f\left(x\right)$ and $v_{x_{\lambda }}\in \partial f\left(x_{\lambda }\right)$.

Based on the previous lemma we can prove the main result of this section.

Proposition A2.

Under the assumptions of Theorem 3, if $x_i\in \mathcal{B}(0,R_{🟉})\backslash \left\{0\right\}\cap S_{\beta }^c$ then

$$f\left(x_{i+1}\right)-f\left(x_i\right)\le -\frac{K{R}_{🟉}^4{d}^4ϵ}{2^{2d}}$$

(A10)

for some numerical constant $K>0$. Moreover there exists an integer $T\le \frac{f\left(x_0\right)2^{2d}}{K{R}_{🟉}^4{d}^4ϵ}$ such $x_{i+T}\in S_{\beta }$.

Proof. 

Let $x_i\notin S_{\beta }$ and assume that $f\left(x_i\right)\le f(-x_i)$, then ${\tilde{x}}_i=x_i$. By the mean value theorem for Clarke subdifferentials [61] (Theorem 8.13), there exists $\lambda \in (0,1)$ such that for $x_{i,\lambda }=x_i-\mu \lambda v_{x_i}$ and a $v_{x_{i,\lambda }}\in \partial f\left(x_{i,\lambda }\right)$ it holds that

$$\begin{array}{cc}\hfill f\left(x_{i+1}\right)-f\left({\tilde{x}}_i\right)& =⟨v_{x_{i,\lambda }},-\mu v_{x_i}⟩\hfill \\ \hfill & =⟨v_{x_i},-\mu v_{x_i}⟩+⟨v_{x_{i,\lambda }}-v_{x_i},-\mu v_{x_i}⟩\hfill \\ \hfill & \le -\mu \Vert v_{x_i}\Vert (\Vert v_{x_i}\Vert -\Vert v_{x_{i,\lambda }}-v_{x_i}\Vert )\hfill \\ \hfill & \le -\frac{\mu }{16}{\Vert v_{x_i}\Vert }^2\hfill \end{array}$$

(A11)

where the first inequality follows from the triangle inequality and the second from Equation (A9). Next observe that by (A8)

$$\parallel v_x{\parallel }^2\ge 36\frac{{\max (\Vert x\Vert }^2,\Vert x_{🟉}{\Vert }^2{)}^2{\Vert x\Vert }^2}{2^{4d}}{86}^2{d}^8ϵ$$

which together with the definition of $\mu$ and (A11) gives (A10).

Next take $x_i\notin S_{\beta }$ and assume $f\left(x_i\right)>f(-x_i)$, so that ${\tilde{x}}_i=-x_i$. Observe that

$$f\left(x_{i+1}\right)-f\left(x_i\right)<f\left(x_{i+1}\right)-f(-x_i)=f\left(x_{i+1}\right)-f\left({\tilde{x}}_i\right),$$

we obtain then (A10) proceeding as before.

Finally the claim on the maximum number of iterations directly follows by a telescopic sum on (A10) and $f(·)\ge 0$. □

Appendix A.5. Convergence to a Neighborhood Around $x_{🟉}$

In the previous section we have shown that after a finite number of steps the iterates $\left\{x_i\right\}$ of the Algorithm 1 will enter in the region $S_{\beta }$. In this section we show that, thanks to the negation step in the descent algorithm, they will eventually be confined in a neighborhood of $x_{🟉}$.

The following lemma shows that $S_{\beta }$ is contained inside two balls around $x_{🟉}$ and $-x_{🟉}$.

Lemma A7.

Suppose $8\pi d^6\sqrt{\beta }\le 1$, then we have $S_{\beta }\subset {\mathcal{B}}^{+}\cup {\mathcal{B}}^{-}$ where

$${\mathcal{B}}^{+}:=\mathcal{B}(x_{🟉},R_1\beta d^{10}\Vert x_{🟉}\Vert )\mathrm{and}{\mathcal{B}}^{-}:=\mathcal{B}(-{\rho }_d{x}_{🟉},R_2\sqrt{\beta }{d}^{10}\Vert x_{🟉}\Vert ),$$

where $R_1,R_2$ are numerical constants and $0<{\rho }_d<1$ such that ${\rho }_d\to 1$ as $d\to \infty$.

We furthermore observe that the ball around $-x_{🟉}$ has values strictly higher that the one around $x_{🟉}$.

Lemma A8.

Suppose that $d\ge 2$, the WDC holds with $ϵ<1/{\left(16\pi d^2\right)}^2$ and H satisfies (13). Then for any ${\varphi }_d\in [{\rho }_d,1]$, it holds that

$$f\left(x\right)<f\left(y\right),$$

(A12)

for all $x\in \mathcal{B}({\varphi }_d{x}_{🟉},\varrho \Vert x_{🟉}\Vert d^{-12})$ and $y\in \mathcal{B}(-{\varphi }_d{x}_{🟉},\varrho \Vert x_{🟉}\Vert d^{-12})$ where $\varrho <1$ is a universal constant.

The main result of this section is about the convergence to a neighborhood of $x_{🟉}$ of the iterates $\left\{x_i\right\}$.

Proposition A3.

Under the assumptions of Theorem 3, if $x_i\in \mathcal{B}(0,R_{🟉})\backslash \left\{0\right\}$, then there exists a finite number of steps $T\le K\frac{f\left(x_i\right)2^{2d}}{R_{🟉}^4{d}^4ϵ}$ such that $x_{i+T}\in \mathcal{B}(x_{🟉},R_1\beta d^{10}\Vert x_{🟉}\Vert )$. In particular it holds that

$$\parallel x_{i+T}-x_{🟉}\Vert \le C_1{d}^{14}\sqrt{ϵ}\Vert x_{🟉}\Vert +C_2{2}^d{d}^{10}\omega {\Vert x_{🟉}\Vert }^{-1}.$$

(A13)

Proof. 

Either $x_i\in S_{\beta }$ or by Proposition A2 there exist $T^{\prime }$ such that $x_{i+T^{\prime }}\in S_{\beta }$. By the choice of $ϵ>0$, the definition of $\beta$ in (A7), and the assumption on the noise level (13), it follows that the hypotheses of Lemma A7 are satisfied. We therefore define the two neighborhoods $S_{\beta }^{+}:=S_{\beta }\cap {\mathcal{B}}^{+}$ and $S_{\beta }^{-}:=S_{\beta }\cap {\mathcal{B}}^{-}$ and conclude that that either $x_{i+T^{\prime }}\in S_{\beta }^{+}$ or $x_{i+T^{\prime }}\in S_{\beta }^{-}$.

We next notice that $S_{\beta }^{+}\subseteq \mathcal{B}(x_{🟉},\varrho \Vert x_{🟉}\Vert d^{-12})$ and $S_{\beta }^{-}\subseteq \mathcal{B}(-{\rho }_d{x}_{🟉},\varrho \Vert x_{🟉}\Vert d^{-12})$. We can then use Lemma A8 to conclude that by the negation step, if $x_{i+T^{\prime }}\in S_{\beta }^{+}$ then ${\tilde{x}}_{i+T^{\prime }}\in S_{\beta }^{+}$, otherwise we will have ${\tilde{x}}_{i+T^{\prime }}\in -S_{\beta }^{-}$.

We now analyze the case $x_{i+T^{\prime }}\in -S_{\beta }^{-}\cap S_{\beta }^c$. Applying again Proposition A2, we have that there exists and integer $T^{″}\le K\frac{f\left(x_{i+N}\right)2^{2d}}{R_{🟉}^4{d}^4ϵ}$ such that $x_{i+T^{\prime }+T^{″}}\in S_{\beta }$. Furthermore Proposition A2 implies that $f\left(x_{i+T^{\prime }+T^{″}}\right)<f\left(x_{i+T^{\prime }}\right)$, while from Lemma A8 we know that $f\left(x_{i+T^{\prime }}\right)<f\left(y\right)$ for all $y\in S_{\beta }^{-}$. We conclude therefore that $x_{i+T^{\prime }+T^{″}}$ must be in $S_{\beta }^{+}$.

In summary we obtained that there exists an integer $T\le K\frac{f\left(x_0\right)2^{2d}}{R_{🟉}^4{d}^4ϵ}$ such that $x_{i+T}\in S_{\beta }^{+}\subset {\mathcal{B}}^{+}$. Finally Equation (A13) follows from the definition of $\beta$ in (A7) and ${\mathcal{B}}^{+}$. □

Appendix A.6. Convergence to $x_{🟉}$ up to Noise

Lemma A9.

Suppose the WDC holds with $ϵ<1/\left({200}^4{d}^6\right)$, then for all $x\in \mathcal{B}(x_{🟉},d\sqrt{ϵ}\Vert x_{🟉}\Vert )$ and $v_x\in {\partial }_{x}f\left(x\right)$, it holds that

$$\parallel v_x-\frac{{\tau }_1}{2^{2d}}{\Vert x_{🟉}\Vert }^2(x-x_{🟉})\Vert \le {\tau }_2\frac{\Vert x_{🟉}{\Vert }^2}{2^{2d}}\Vert x-x_{🟉}\Vert +{\tau }_3\frac{\omega }{2^d}\Vert x_{🟉}\Vert$$

where ${\tau }_1=21/5$, ${\tau }_2=17/5$ and ${\tau }_3=(1+1/{(400)}^2)$.

Based on the previous condition on the direction of the sub-gradients we can then prove that the iterates of the algorithm converge to $x_{🟉}$ up to noise.

Proposition A4.

Under the assumptions of Theorem (3), if $x_T\in {\mathcal{B}}^{+}$ then for any $i\ge T$ it holds that $x_i\in {\mathcal{B}}^{+}$ and furthermore

$$\begin{array}{cc}\hfill \Vert x_{i+1}-x_{🟉}\Vert & \le {\mathsf{\rho }}_1^{i+1-T}\Vert x_T-x_{🟉}\Vert +{\mathsf{\rho }}_2\frac{2^d}{\Vert x_{🟉}\Vert }\omega \hfill \end{array}$$

(A14)

where ${\mathsf{\rho }}_1\in (0,1)$ and ${\mathsf{\rho }}_2>0$ are numerical constants.

Proof. 

If $i=T$, by Proposition A2, it follows that $x_i={\tilde{x}}_i\in {\mathcal{B}}^{+}$. Furthermore, the assumptions of Lemma A9 are satisfied and we can write:

$$\begin{array}{cc}\hfill \Vert x_{i+1}-x_{🟉}\Vert & =\Vert {\tilde{x}}_i-\mu v_{{\tilde{x}}_i}-x_{🟉}\Vert \hfill \\ \hfill & \le (1-\frac{\mu {\tau }_1{\Vert x_{🟉}\Vert }^2}{2^{2d}})\Vert {\tilde{x}}_i-x_{🟉}\Vert +\mu \Vert v_{{\tilde{x}}_i}-\frac{{\tau }_1}{2^{2d}}\Vert x_{🟉}{\Vert }^2({\tilde{x}}_i-x_{🟉})\Vert \hfill \\ \hfill & \le (1-\frac{\mu ({\tau }_1-{\tau }_2){\Vert x_{🟉}\Vert }^2}{2^{2d}})\Vert {\tilde{x}}_i-x_{🟉}\Vert +\mu {\tau }_3\frac{\omega }{2^d}\Vert x_{🟉}\Vert .\hfill \end{array}$$

(A15)

Next recall that $\omega$ satisfies (13), $\mu =K_3{2}^{2d}/(8d^4{R}_{🟉}^2)$ and $R_{🟉}^2\ge 25{\Vert x_{🟉}\Vert }^2/8$, then

$$\begin{array}{cc}\hfill \Vert x_{i+1}-x_{🟉}\Vert & \le (1-\frac{\mu ({\tau }_1-{\tau }_2){\Vert x_{🟉}\Vert }^2}{2^{2d}})\Vert {\tilde{x}}_i-x_{🟉}\Vert +\mu \frac{K_2}{2^{2d}}\frac{{\tau }_3}{d^{44}}{\Vert x_{🟉}\Vert }^3\hfill \\ & \le [1-({\tau }_1-{\tau }_2)\frac{K_3}{25d^4}+K_2\frac{K_3}{25d^{48}}]\Vert x_{🟉}\Vert \hfill \end{array}$$

Therefore for $K_2$ and $K_3$ small enough we obtain that $x_{i+1}\in \mathcal{B}(x_{🟉},R_1\beta d^{10}\Vert x_{🟉}\Vert )$ and by induction this holds for all $i\ge T$. Finally we obtain (A14) by letting ${\mathsf{\rho }}_1=(1-\mu ({\tau }_1-{\tau }_2){\Vert x_{🟉}\Vert }^2/2^{2d})$ and ${\mathsf{\rho }}_2=K_3{\tau }_3/\left(25d^4\right)$ in (A15). □

Appendix A.7. Proof of Theorem 3

We begin recalling the following fact on the local Lipschitz property of the generative network G under the WDC.

Lemma A10

(Lemma 21 in [16]). Suppose $x\in \mathcal{B}(x_{🟉},d\sqrt{ϵ}\Vert x_{🟉}\Vert )$, and the WDC holds with $ϵ<1/\left({200}^4{d}^6\right)$. Then it holds that:

$$\Vert G\left(x\right)-G\left(x_{🟉}\right)\Vert \le \frac{1.2}{2^{d/2}}\Vert x-x_{🟉}\Vert .$$

We then conclude the proof of Theorem 3 by using the above lemma and the results in the previous sections.

(I)

By assumption $x_0\in \mathcal{B}(0,R_{🟉})\backslash \left\{0\right\}$ so that according to Proposition A1 for any $i\ge 1$ it holds that $x_i\in \mathcal{B}(0,R_{🟉})\backslash \left\{0\right\}$.

(II)

By Proposition A3, there exists an integer T such that $x_T\in {\mathcal{B}}^{+}$ and therefore it satisfies the conclusions of Theorem 3.A

$$\Vert x_T-x_{🟉}\Vert \le C_1{d}^{14}\sqrt{ϵ}\Vert x_{🟉}\Vert +C_2{2}^d{d}^{10}\omega {\Vert x_{🟉}\Vert }^{-1}.$$

(III)

Once in ${\mathcal{B}}^{+}$ the iterates of Algorithm 1 converge to $x_{🟉}$ up to the noise level, as shown by Proposition A4 and Equation (A14)

$$\parallel x_{i+1}-x_{🟉}\Vert \le {\mathsf{\rho }}_1^{i+1-T}\Vert x_T-x_{🟉}\Vert +{\mathsf{\rho }}_2\frac{2^d}{\Vert x_{🟉}\Vert }\omega$$

which corresponds to (11) in Theorem 3.B.

(IV)

The reconstruction error (12) in Theorem 3.B, follows then from (11) by applying Lemma A10 and the lower bound (A4).

Appendix B.1. Supplementary Proofs for Appendix A.2

Below we prove Lemma A2 on the concentration of the gradient of f at a differentiable point.

Proof of Lemma A2. 

We begin by noticing that:

$${\overline{v}}_x-h_x=[⟨p_x,x⟩p_x-⟨x,x⟩\frac{x}{2^{2d}}]+[⟨{\tilde{h}}_x,x⟩\frac{{\tilde{h}}_x}{2^{2d}}-⟨q_x,x⟩x].$$

Below we show that:

$$\parallel ⟨p_x,x⟩p_x-⟨x,x⟩\frac{x}{2^{2d}}\parallel \le \frac{50}{2^{2d}}{d}^3\sqrt{ϵ}{\max \{\Vert x\Vert }^2,\Vert {x_{🟉}\Vert }^2\}\Vert x\Vert .$$

(A16)

and

$$\parallel ⟨q_x,x⟩p_x-⟨{\tilde{h}}_x,x⟩\frac{{\tilde{h}}_x}{2^{2d}}|\le \frac{36}{2^{2d}}{d}^4\sqrt{ϵ}{\max \{\Vert x\Vert }^2,\Vert {x_{🟉}\Vert }^2\}\Vert x\Vert .$$

(A17)

from which the thesis follows.

Regarding Equation (A16) observe that:

$$\begin{array}{cc}\hfill \parallel ⟨p_x,x⟩p_x-⟨x,x⟩\frac{x}{2^{2d}}\parallel & =\Vert ⟨p_x,x⟩[p_x-\frac{x}{2^d}]+⟨p_x-\frac{x}{2^d},\frac{x}{2^d}⟩x\parallel \hfill \\ & \le (\parallel {\Lambda }_x{x\Vert }^2+\frac{{\Vert x\Vert }^2}{2^d})\parallel p_x-\frac{x}{2^d}\parallel \hfill \\ & \le \frac{50}{2^{2d}}{d}^3\sqrt{ϵ}{\Vert x\Vert }^3\hfill \end{array}$$

where in the first inequality we used $⟨p_x,x⟩={\Vert {\Lambda }_{x}x\Vert }^2$ and in the second we used Equations (A5) and (A6) of Lemma A1.

Next note that:

$$\begin{array}{cc}\hfill \parallel ⟨q_x,x⟩q_x-⟨{\tilde{h}}_x,x⟩\frac{{\tilde{h}}_x}{2^{2d}}\parallel & =\Vert ⟨q_x,x⟩(q_x-\frac{{\tilde{h}}_x}{2^d})+⟨q_x-\frac{{\tilde{h}}_x}{2^d},x⟩\frac{{\tilde{h}}_x}{2^d}\parallel \hfill \\ & \le (\Vert q_x\Vert +\frac{\Vert {\tilde{h}}_x\Vert }{2^d})\Vert x\Vert \Vert q_x-\frac{{\tilde{h}}_x}{2^d}\parallel \hfill \\ & \le (\frac{13}{12}+1+\frac{d}{\pi })\frac{\Vert x\Vert \Vert x_{🟉}\Vert }{2^d}\Vert q_x-{\tilde{h}}_x\Vert \hfill \\ & \le \frac{3}{2}d\frac{\Vert x\Vert \Vert x_{🟉}\Vert }{2^d}\Vert q_x-{\tilde{h}}_x\Vert \hfill \end{array}$$

where in the second inequality we have the bound (A6) and the definition of ${\tilde{h}}_x$. Equation (A17) is then found by appealing to Equation (A5) in Lemma A1. □

The previous lemma is now used to control the concentration of the subgradients $v_x$ of f around $h_x$.

Proof of Lemma A3. 

When f is differentiable at x, $\nabla f\left(x\right)={\tilde{v}}_x={\overline{v}}_x+{\eta }_x$, so that by Lemma A2 and the assumption on the noise:

$$\begin{array}{cc}\hfill \Vert {\tilde{v}}_x-h_x\Vert & \le \Vert {\overline{v}}_x-h_x\Vert +\Vert {\eta }_x\Vert \hfill \\ \hfill & \le 86\frac{d^4\sqrt{ϵ}}{2^{2d}}\max (\Vert {x_{🟉}\Vert }^2,{\Vert x\Vert }^2)\Vert x\Vert +\frac{\omega }{2^d}\Vert x\Vert .\hfill \end{array}$$

(A18)

Observe, now, that by (9), for any $x\in {\mathbb{R}}^k$, $v_x\in \partial f\left(x\right)=$ conv$(v_1,\dots ,v_t)$, and therefore $v_x=a_1{v}_1+\dots +a_T{v}_T$ for some $a_1,\dots ,a_T\ge 0$, ${\sum }_i{a}_i=1$. Moreover for each $v_i$ there exist a $w_i$ such that $v_i={lim}_{\delta \to 0^{+}}{\tilde{v}}_{x+\delta w_i}$. Therefore using Equation (A18), the continuity of $h_x$ with respect to nonzero x and ${\sum }_i{a}_i=1$:

$$\begin{array}{cc}\hfill \Vert v_x-h_x\Vert & \le \sum _{i=1}^T{a}_i\Vert v_i-h_x\Vert \hfill \\ & \le \sum _{i=1}^T{a}_i\underset{\delta \to 0}{lim}\Vert {\tilde{v}}_{x+\delta w_i}-h_{x+\delta w_i}\Vert \hfill \\ & \le 86\frac{d^4\sqrt{ϵ}}{2^{2d}}\max (\Vert {x_{🟉}\Vert }^2,{\Vert x\Vert }^2)\Vert x\Vert +\frac{\omega }{2^d}\Vert x\Vert .\hfill \end{array}$$

□

The above results are now used to bound the norm of $v_x\in \partial f\left(x\right)$.

Proof of Lemma A4. 

Since $ϵ<1/{\left(16\pi d^2\right)}^2$ observe that $86d^4\sqrt{ϵ}\le 2d^2$, therefore by the assumption on the noise level and Lemma A3 it follows that for any $v_x\in \partial f\left(x\right)$ and $K_2\le 1$

$$\parallel v_x-h_x\Vert \le \frac{1}{2^{2d}}\frac{5}{2}{d}^2{\max (\Vert x\Vert }^2,\Vert {x_{🟉}\Vert }^2)\Vert x\Vert .$$

Next observe that that since $\Vert {\tilde{h}}_x\Vert \le d\Vert x_{🟉}\Vert$, we have:

$$\parallel h_x\Vert \le \frac{1}{2^{2d}}\frac{5}{4}{d}^2{\max (\Vert x\Vert }^2,\Vert x_{🟉}{\Vert }^2)\Vert x\Vert$$

and from $\Vert v_x\Vert \le \Vert v_x-h_x\Vert +\Vert h_x\Vert$ we obtain the thesis. □

Appendix B.2. Supplementary Proofs for Appendix A.3

In this section we prove Lemma A5 which implies that the norm of the iterates $x_i$ does not increase in the region $\mathcal{B}(0,R_{🟉})\backslash \mathcal{B}(0,C_{🟉}\Vert x_{🟉}\Vert )$.

Proof of Lemma A5. 

Note that the thesis is equivalent to $2⟨x,v_x⟩\ge \lambda \mu {\Vert v_x\Vert }^2$. Next recall that by the WDC for any $x\in {\mathbb{R}}^k$ and $2dϵ<1$:

$$\frac{(1-2ϵd)}{2^d}{\Vert x\Vert }^2\le \frac{{(1-2ϵ)}^d}{2^d}{\Vert x\Vert }^2\le {\Vert G\left(x\right)\Vert }^2\le \frac{{(1+2ϵ)}^d}{2^d}{\Vert x\Vert }^2\le \frac{(1+4ϵd)}{2^d}{\Vert x\Vert }^2.$$

At a nonzero differentiable point $x\in {\mathbb{R}}^k\backslash \left\{0\right\}$ with $C_{🟉}\Vert x_{🟉}\Vert <\Vert x\Vert <R_{🟉}$, then

$$\begin{array}{cc}\hfill ⟨{\tilde{v}}_x,x⟩& \ge {\Vert G\left(x\right)\Vert }^2{(\Vert G\left(x\right)\Vert }^2-\Vert G\left(x_{🟉}\right){\Vert }^2)-\Vert {\Lambda }_x^{\mathsf{T}}H{\Lambda }_x{\Vert \Vert x\Vert }^2\hfill \\ \hfill & \ge \frac{{\Vert x\Vert }^2}{2^{2d}}\left[{(1-2ϵd)}^2{\Vert x\Vert }^2-[(1-2ϵd)(1+4ϵd)+K_2/d^{44}]{\Vert x_{🟉}\Vert }^2\right]\hfill \\ \hfill & \ge \frac{{\Vert x\Vert }^2}{2^{2d}}\left[{(1-4dϵ)\Vert x\Vert }^2-[(1+2dϵ)+K_2/d^{44}]{\Vert x_{🟉}\Vert }^2\right].\hfill \end{array}$$

(A19)

Next, by Lemma A4, the definition of the step length $\mu$ and ${\max (\Vert x\Vert }^2,\Vert x_{🟉}{\Vert }^2)\le R_{🟉}^2$, we have:

$$\frac{\lambda \mu }{2}\Vert {\tilde{v}}_x{\Vert }^2\le \frac{K_3}{2^{2d}}{\Vert x\Vert }^4,$$

(A20)

which, using $\Vert x\Vert >C_{🟉}\Vert x_{🟉}\Vert$, gives

$$\begin{array}{cc}\hfill ⟨{\tilde{v}}_x,x⟩-\frac{\lambda \mu }{2}{\Vert {\tilde{v}}_x\Vert }^2& \ge \frac{{\Vert x\Vert }^2}{2^{2d}}\left[(1-4dϵ-K_3){\Vert x\Vert }^2-(1+2dϵ+K_2){\Vert x_{🟉}\Vert }^2\right]\hfill \\ & \ge \frac{{\Vert x\Vert }^2{\Vert x_{🟉}\Vert }^2}{2^{2d}}\left[(1-4dϵ-K_3)C_{🟉}-(1+2dϵ+K_2)\right].\hfill \end{array}$$

We can then conclude by observing that by the assumptions and for small enough constants $(1-4dϵ-K_3)C_{🟉}-(1+2dϵ+K_2/d^{44})>0$.

At a non-differentiable point x, by the characterization of the Clarke subdifferential, we can write $v_x={\sum }_{\ell =1}^m{c}_{\ell }{v}_{\ell }$ where $v_{\ell }={lim}_{{\delta }_{\ell }\nabla f(x+{\delta }_{\ell }{w}_{\ell })}$ then

$$⟨x,v_x⟩=⟨x,\sum _{\ell =1}^m{c}_{\ell }{v}_{\ell }⟩=\sum _{\ell =1}^m{c}_{\ell }\underset{{\delta }_{\ell }\to 0^{+}}{lim}⟨x+{\delta }_{\ell }{w}_{\ell },{\tilde{v}}_{x+{\delta }_{\ell }{w}_{\ell }}⟩$$

which implies that the lower bound (A19) also holds for $⟨x,v_x⟩$. Similarly Lemma A4 leads to the upper bound (A20) also for $v_x$, which then leads to the thesis $2⟨x,v_x⟩\ge \lambda \mu {\Vert v_x\Vert }^2$. □

Appendix B.3. Supplementary Proofs for Appendix A.4

In the next lemmas we show that $h_x$ is locally Lipschitz.

Lemma A11.

For all $x,y\ne 0$

$$\parallel h_x-h_y\Vert \le \frac{1}{2^{2d}}\left[{2(\Vert x\Vert }^2+{\Vert y\Vert }^2)+\left(d^2+5d^3\max (\frac{\Vert x\Vert }{\Vert y\Vert },\frac{\Vert y\Vert }{\Vert x\Vert })\right){\Vert x_{🟉}\Vert }^2\right]\Vert x-y\Vert$$

Proof. 

Note that ${\tilde{h}}_x=\xi \Vert x_{🟉}\Vert +\zeta \Vert x_{🟉}\Vert \widehat{x}$ where $\xi$ and $\zeta$ are defined in (A23). Then observe that by (A28) and (A29) it follows that $\parallel {\tilde{h}}_x\Vert \le d\Vert x_{🟉}\parallel$. Furthermore by [16] (Lemma 18) for any nonzero $x,y\in {\mathbb{R}}^k$ we have

$$\parallel {\tilde{h}}_x-{\tilde{h}}_y\Vert \le \frac{9}{4}{d}^2\max \left(\frac{1}{\Vert x\Vert },\frac{1}{\Vert y\Vert }\right)\Vert x_{🟉}\Vert \Vert x-y\Vert$$

(A21)

Next notice that

$$\parallel h_x-h_y\Vert \le \frac{1}{2^{2d}}\Vert ⟨x,x⟩x-⟨y,y⟩y\Vert +\frac{1}{2^{2d}}\parallel ⟨{\tilde{h}}_y,y⟩{\tilde{h}}_y-⟨{\tilde{h}}_x,x⟩{\tilde{h}}_x\parallel ,$$

where by triangle inequality the first term on the left hand side can be bounded as

$$\begin{array}{cc}\hfill \Vert ⟨x,x⟩x-⟨y,y⟩y\Vert & \le \left({\Vert x\Vert }^2\Vert x-y\Vert +\left|{\Vert x\Vert }^2-{\Vert y\Vert }^2\right|\Vert y\Vert \right)\hfill \\ & \le \left({\Vert x\Vert }^2+\Vert x\Vert \Vert y\Vert +{\Vert y\Vert }^2\right)\Vert x-y\Vert \hfill \\ & \le 2\left({\Vert x\Vert }^2+{\Vert y\Vert }^2\right)\Vert x-y\Vert \hfill \end{array}$$

Finally note that by from the bound (A21) we obtain:

$$\begin{array}{cc}\hfill \parallel ⟨{\tilde{h}}_y,y⟩{\tilde{h}}_y-⟨{\tilde{h}}_x,x⟩{\tilde{h}}_x\parallel & \le \Vert {\tilde{h}}_x\Vert \Vert x\Vert \Vert {\tilde{h}}_x-{\tilde{h}}_y\Vert +\Vert {\tilde{h}}_y\Vert \left|⟨{\tilde{h}}_x,x⟩-⟨{\tilde{h}}_y,y⟩\right|\hfill \\ & \le \Vert {\tilde{h}}_x\Vert \Vert x\Vert \Vert {\tilde{h}}_x-{\tilde{h}}_y\Vert +\Vert {\tilde{h}}_y\Vert \Vert {\tilde{h}}_x\Vert \Vert x-y\Vert +\Vert {\tilde{h}}_y\Vert \Vert y\Vert \Vert {\tilde{h}}_x-{\tilde{h}}_y\Vert \hfill \\ & \le d^2\Vert x_{🟉}{\Vert }^2\Vert x-y\Vert +d(\Vert x\Vert +\Vert y\Vert )\Vert x_{🟉}\Vert \Vert {\tilde{h}}_x-{\tilde{h}}_y\Vert \hfill \\ & \le \left(2d^2+5d^3\max \left(\frac{\Vert x\Vert }{\Vert y\Vert },\frac{\Vert y\Vert }{\Vert x\Vert }\right)\right)\Vert x_{🟉}{\Vert }^2\Vert x-y\Vert .\hfill \end{array}$$

where we used

$$(\Vert x\Vert +\Vert y\Vert )\max \left(\frac{1}{\Vert x\Vert ,\Vert y\Vert }\right)\le 2\frac{\max \left(\Vert x\Vert ,\Vert y\Vert \right)}{min\left(\Vert x\Vert ,\Vert y\Vert \right)}=2\max \left(\frac{\Vert x\Vert }{\Vert y\Vert },\frac{\Vert y\Vert }{\Vert x\Vert }\right).$$

□

Based on the previous lemma we can now prove that $h_x$ is locally Lipschitz.

Lemma A12.

Let $x\in \mathcal{B}(0,R_{🟉})\backslash \left\{0\right\}$, $\lambda \in (0,1)$, $\mu =K_3{2}^{2d}/(8d^4{R}_{🟉}^2)$ with $512{\pi }^2{K}_3<1$ and $v_x\in \partial f\left(x\right)$, then for $x_{\lambda }=x-\lambda \mu v_x$ it holds that:

$$\parallel h_x-h_{x_{\lambda }}\Vert \le \frac{7}{16}\Vert v_x\Vert$$

Proof. 

Consider $x\in \mathcal{B}(0,R_{🟉})$ and observe that by Lemma A4, $d\ge 2$ and the choice of $\mu$, for any $v_x\in \partial f\left(x\right)$ and any $\lambda \in (0,1)$ we have $\mu \Vert v_x\Vert \le \Vert x\Vert /8$. It follows that

$$\frac{7}{8}\Vert x\Vert \le \Vert x_{\lambda }\Vert \le \frac{9}{8}\Vert x\Vert ,$$

(A22)

and in particular

$$\max \left(\frac{\Vert x\Vert }{\Vert x_{\lambda }\Vert },\frac{\Vert x_{\lambda }\Vert }{\Vert x\Vert }\right)\le \frac{8}{7}.$$

Therefore by Lemma A11 we deduce that

$$\begin{array}{cc}\hfill \Vert h_x-h_{x_{\lambda }}\Vert & \le \frac{\lambda \mu }{2^{2d}}\left[{2(\Vert x\Vert }^2+\Vert x_{\lambda }{\Vert }^2)+(d^2+6d^3){\Vert x_{🟉}\Vert }^2\right]\Vert v_x\Vert \hfill \\ & \le \frac{\mu }{2^{2d}}\left(4R_{🟉}^2+(d^2+6d^3){\Vert x_{🟉}\Vert }^2\right)\Vert v_x\Vert \hfill \end{array}$$

where in the second inequality we used ${\max (\Vert x\Vert }^2,\Vert x_{\lambda }{\Vert }^2)\le R_{🟉}^2$ by Proposition A1. The thesis is obtained by substituting the definition of $\mu$ and using $K_3\le 1$. □

Based on the previous result we can now prove Lemma A6.

Proof of Lemma A6. 

Let $x\in \mathcal{B}(0,R_{🟉})\cap S_{\beta }^c$,

$$\begin{array}{cc}\hfill \Vert v_x\Vert & \ge \Vert h_x\Vert -\Vert v_x-h_x\Vert \hfill \\ & \ge \frac{\beta }{2^d}\max (\Vert x_{🟉}{\Vert }^2{,\Vert x\Vert }^2)\Vert x\Vert -86\frac{d^4\sqrt{ϵ}}{2^{2d}}\max (\Vert x_{🟉}{\Vert }^2{,\Vert x\Vert }^2)\Vert x\Vert -\frac{\omega }{2^d}\Vert x\Vert \hfill \\ & \ge 6\frac{\max (\Vert x{\Vert }^2,\Vert x_{🟉}{\Vert }^2)\Vert x\Vert }{2^{2d}}(86d^4\sqrt{ϵ}+2^d\omega \Vert x_{🟉}{\Vert }^{-2})\hfill \\ & \ge 6\left(\frac{\max (\Vert x{\Vert }^2,\Vert x_{🟉}{\Vert }^2)\Vert x\Vert }{2^{2d}}86d^4\sqrt{ϵ}+\frac{\omega }{2^d}\Vert x\Vert \right)\hfill \end{array}$$

where we used the definition $\beta$ in Equation (A7) and Lemma A3.

Next take $\lambda \in [0,1]$ and $x_{\lambda }=x-\lambda \mu v_x$ with $v_x\in \partial f\left(x\right)$. Then for any $v_{x_{\lambda }}\in \partial f\left(x_{\lambda }\right)$

$$\begin{array}{cc}\hfill \Vert v_x-v_{x_{\lambda }}\Vert & \le \Vert v_x-h_x\Vert +\Vert h_x-h_{x_{\lambda }}\Vert +\Vert h_{x_{\lambda }}-v_{x_{\lambda }}\Vert \hfill \\ \hfill & \le 86\frac{d^4\sqrt{ϵ}}{2^{2d}}(\max (\Vert x_{🟉}{\Vert }^2{,\Vert x\Vert }^2)\Vert x\Vert +\max (\Vert x_{🟉}{\Vert }^2,\Vert x_{\lambda }{\Vert }^2)\Vert x_{\lambda }\Vert )\hfill \\ \hfill & +\frac{\omega }{2^d}(\Vert x\Vert +\Vert x_{\lambda }\Vert )+\frac{7}{16}\Vert v_x\Vert \hfill \\ \hfill & \le 86\frac{d^4\sqrt{ϵ}}{2^{2d}}\left(1+{\left(\frac{9}{8}\right)}^3\right)\max (\Vert x_{🟉}{\Vert }^2{,\Vert x\Vert }^2)\Vert x\Vert +\left(1+\frac{9}{8}\right)\frac{\omega }{2^d}\Vert x\Vert +\frac{7}{16}\Vert v_x\Vert |\hfill \\ \hfill & \le 3\left(86\frac{d^4\sqrt{ϵ}}{2^{2d}}\max (\Vert x_{🟉}{\Vert }^2{,\Vert x\Vert }^2)\Vert x\Vert +\frac{\omega }{2^d}\Vert x\Vert \right)+\frac{7}{16}\Vert v_x\Vert \hfill \\ \hfill & \le \left(\frac{1}{2}+\frac{7}{16}\right)\Vert v_x\Vert \hfill \end{array}$$

where in the first inequality we used Lemma A3 and Lemma A12, in the second inequality (A22) and in the last one (A8). □

Appendix B.4. Supplementary Proofs for Appendix A.5

Below we prove that the region of ${\mathbb{R}}^k$ where we cannot control the norm of the vector field $h_x$ is contained in two balls around $x_{🟉}$ and $-{\rho }_d{x}_{🟉}$.

We prove Lemma A7 by showing the following.

Lemma A13.

Suppose $8\pi d^6\sqrt{\beta }\le 1$. Define:

$${\rho }_d:=\sum _{i=0}^{d-1}\frac{\sin {\check{\theta }}_i}{\pi }(\prod _{j=i+1}^{d-1}\frac{\pi -{\check{\theta }}_j}{\pi })$$

where ${\check{\theta }}_0=\pi$ and ${\check{\theta }}_i=g({\check{\theta }}_{i-1})$. If $x\in S_{\beta }$, then we have either:

$$|{\overline{\theta }}_0|\le 32d^4{\pi \beta \mathrm{and}|\Vert x\Vert }^2-\Vert x_{🟉}{\Vert }^2|\le 258\pi \beta d^6\Vert x_{🟉}\Vert$$

or

$$|{\overline{\theta }}_0-\pi |\le 8\pi d^4\sqrt{\beta }{\mathrm{and}|\Vert x\Vert }^2-{\rho }_d^2\Vert x_{🟉}{\Vert }^2|\le 281{\pi }^2\sqrt{\beta }{d}^{10}\Vert x_{🟉}\Vert .$$

In particular, we have:

$$S_{\beta }\subset \mathcal{B}(x_{🟉},R_1\beta d^{10}\Vert x_{🟉}\Vert )\cup \mathcal{B}(-{\rho }_d{x}_{🟉},R_2\sqrt{\beta }{d}^{10}\Vert x_{🟉}\Vert )$$

where $R_1,R_2$ are numerical constants and ${\rho }_d\to 1$ as $d\to \infty$.

Proof of Lemma A13. 

Without loss of generality, let $x_{🟉}=e_1$ and $x=r\cos {\overline{\theta }}_0·e_1+r\sin {\overline{\theta }}_0·e_2$, for some ${\overline{\theta }}_0\in [0,\pi ]$, and $r\ge 0$. Recall that we call $\widehat{x}=x/\Vert x\Vert$ and ${\widehat{x}}_{🟉}=x_{🟉}/\Vert x_{🟉}\Vert$. We then introduce the following notation:

$$\xi =\prod _{i=0}^{d-1}\frac{\pi -{\overline{\theta }}_i}{\pi },\zeta =\sum _{i=0}^{d-1}\frac{\sin {\overline{\theta }}_i}{\pi }\prod _{j=i+1}^{d-1}\frac{\pi -{\overline{\theta }}_j}{\pi },r=\Vert x\Vert ,R=\max (r^2,1),$$

(A23)

where ${\theta }_i=g\left({\overline{\theta }}_{i-1}\right)$ with g as in (A1), and observe that ${\tilde{h}}_x=(\xi {\widehat{x}}_{🟉}+\zeta \widehat{x})$. Let $\alpha :=⟨{\tilde{h}}_x,\widehat{x}⟩$, then we can write:

$$\begin{array}{c}\hfill h_x=\frac{1}{2^{2d}}\left[⟨x,x⟩x-⟨{\tilde{h}}_x,x⟩{\tilde{h}}_x\right]=\frac{r}{2^{2d}}\left[r^2\widehat{x}-\alpha (\xi {\widehat{x}}_{🟉}+\zeta \widehat{x})\right].\end{array}$$

Using the definition of $\widehat{x}$ and ${\widehat{x}}_{🟉}$ we obtain:

$$\frac{2^{2d}{h}_x}{r}=\left[(r^2-\alpha \zeta )\cos {\overline{\theta }}_0-\alpha \xi \right]·e_1+[r^2-\alpha \zeta ]\sin {\overline{\theta }}_0·e_2,$$

and conclude that since $x\in S_{\beta }$, then:

$$\begin{array}{cc}\hfill |(r^2-\alpha \zeta )\cos {\overline{\theta }}_0-\alpha \xi |& \le \beta R\hfill \end{array}$$

(A24)

$$\begin{array}{c}\hfill |[r^2-\alpha \zeta ]\sin {\overline{\theta }}_0|\le \beta R.\end{array}$$

(A25)

We now list some bounds that will be useful in the subsequent analysis. We have:

$$\begin{array}{cc}\hfill {\overline{\theta }}_i& \le {\overline{\theta }}_{i-1}\mathrm{for}i\ge 1\hfill \end{array}$$

(A26)

$$\begin{array}{cc}\hfill {\overline{\theta }}_i& \le {\cos }^{-1}(1/\pi )\mathrm{for}i\ge 2\hfill \end{array}$$

(A27)

$$\begin{array}{cc}\hfill \left|\xi \right|& \le 1\hfill \end{array}$$

(A28)

$$\begin{array}{cc}\hfill \left|\zeta \right|& \le \frac{d}{\pi }\sin {\overline{\theta }}_0\hfill \end{array}$$

(A29)

$$\begin{array}{cc}\hfill \xi & \ge \frac{\pi -{\overline{\theta }}_0}{\pi }{d}^{-3}\hfill \end{array}$$

(A30)

$$\begin{array}{cc}\hfill {\check{\theta }}_i& \le \frac{3\pi }{i+3}\mathrm{for}i\ge 0\hfill \end{array}$$

(A31)

$$\begin{array}{cc}\hfill {\check{\theta }}_i& \ge \frac{\pi }{i+1}\mathrm{for}i\ge 0\hfill \end{array}$$

(A32)

$$\begin{array}{cc}\hfill {\overline{\theta }}_0& =\pi +O_1\left(\delta \right)\Rightarrow \left|\xi \right|\le \frac{\delta }{\pi }\hfill \end{array}$$

(A33)

$$\begin{array}{cc}\hfill {\overline{\theta }}_0& =\pi +O_1\left(\delta \right)\Rightarrow \zeta ={\rho }_d+O_1\left(3d^3\delta \right)\mathrm{if}\frac{d^2\delta }{\pi }\le 1\hfill \end{array}$$

(A34)

$$\begin{array}{cc}\hfill 1/\pi & \le \alpha \le 1.\hfill \end{array}$$

(A35)

The identities (A26) through (A34) can be found in Lemma 16 of [16], while the identity (A35) follows by noticing that $\alpha =\xi \cos {\overline{\theta }}_0+\zeta =\cos {\theta }_d$ and using (A27) together with $d\ge 2$.

Bound on R. 

We now show that if $x\in S_{\beta }$, then $r^2\le 4d$ and therefore $R\le 4d$.

If $r^2\le 1$, then the claim is trivial. Take $r^2>1$, then note that either $|\sin {\overline{\theta }}_0|\ge 1/\sqrt{2}$ or $|\cos {\overline{\theta }}_0|\ge 1/\sqrt{2}$ must hold. If $|\sin {\overline{\theta }}_0|\ge 1/\sqrt{2}$ then from (A25) it follows that $r^2-\alpha \zeta \le \sqrt{2}\beta R=\sqrt{2}\beta r^2$ which implies:

$$r^2\le \frac{\alpha \zeta }{1-\sqrt{2}\beta }\le \frac{1}{(1-\sqrt{2}\beta )}\frac{d}{\pi }\le \frac{d}{2}$$

using (A29) and (A35) in the second inequality and $\beta <1/4$ in the third. Next take $|\cos {\overline{\theta }}_0|\ge 1/\sqrt{2}$, then (A24) implies $|r^2-\alpha \zeta |\le \sqrt{2}(\beta r^2+\alpha \xi )$ which in turn results in:

$$r^2\le \frac{\alpha (\zeta +\sqrt{2}\xi )}{1-\sqrt{2}\beta }\le 4d$$

using (A28), (A29), (A35) and $\beta <1/4$. In conclusion if $x\in S_{\beta }$ then $r^2\le 4d\Rightarrow R\le 4d$.

Bounds on ${\overline{\mathbf{\theta }}}_{\mathbf{0}}$. 

We proceed by showing that we only have to analyze the small angle case ${\overline{\theta }}_0\approx 0$ and the large angle case ${\overline{\theta }}_0\approx \pi$. At least one of the following three cases must hold:

(1)

$\sin {\overline{\theta }}_0\le 16\beta \pi d^4$: Then we have $\overline{\theta }=O_1\left(32\pi \beta \pi d^4\right)$ or $\overline{\theta }=\pi +O_1\left(32\pi \beta \pi d^4\right)$ as $32\pi \beta \pi d^4<1$.

(2)

$|r^2-\alpha \zeta |<\sqrt{\beta }R$: Then (A24), (A35) and $\beta <1$ yield $\left|\xi \right|\le 2\sqrt{\beta }\pi R$. Using (A30), we then get $\overline{\theta }=\pi +O_1\left(2\sqrt{\beta }{\pi }^2{d}^{3}R\right)$.

(3)

$\sin {\overline{\theta }}_0>16\beta \pi d^4$ and $|r^2-\alpha \zeta |\ge \sqrt{\beta }R$: Then (A27) gives $|r^2-\alpha \zeta |\le \beta M/\sin {\overline{\theta }}_0$ which used with (A24) leads to:

$$\left|\alpha \xi \right|\le \beta R+|r^2-\alpha \zeta |\le \beta R+\frac{\beta R}{\sin {\overline{\theta }}_0}\le 2\frac{\beta R}{\sin {\overline{\theta }}_0}.$$

Then using (A35), the assumption on $\sin {\overline{\theta }}_0$ and $R\le 4d$ we obtain $\xi \le d^{-3}/2$. The latter together with (A30) leads to ${\overline{\theta }}_0\ge \pi /2$. Finally as $|r^2-\alpha \zeta |\ge \sqrt{\beta }R$ then (A25) leads to $|\sin {\overline{\theta }}_0|\le \sqrt{\beta }$. Therefore as ${\overline{\theta }}_0\ge \pi /2$ and $\beta <1$, we can conclude that ${\overline{\theta }}_0=\pi +O_1\left(2\sqrt{\beta }\right)$.

Inspecting the three cases, and recalling that $R\le 4d$, we can see that it suffices to analyze the small angle case ${\overline{\theta }}_0=O_1\left(32d^4\pi \beta \right)$ and the large angle case $\overline{\theta }=\pi +O_1\left(8\sqrt{\beta }{\pi }^2{d}^4\right)$.

Small angle case. 

We assume ${\overline{\theta }}_0=O_1\left(\delta \right)$ with $\delta =32d^4\pi \beta$ and show that ${\Vert x\Vert }^2\approx {\Vert x_{🟉}\Vert }^2$. We begin collecting some bounds. Since ${\overline{\theta }}_i\le {\overline{\theta }}_0\le \delta$, then $1\ge \xi \ge {(1-\delta /\pi )}^d\ge 1+O_1(2d\delta /\pi )$ assuming $\delta d/\pi \le 1/2$, which holds true since $64d^5\beta <1$. Moreover from (A29) we have $\zeta =O_1(d\delta /\pi )$. Finally observe that $\cos {\overline{\theta }}_0=1+O_1({\overline{\theta }}_0^2/2)=1+O_1(\delta /2)$ for $\delta <1$. We then have $\alpha =1+O_1\left(2d\delta \right)$ so that $\alpha \zeta =O_1\left(d^2\delta \right)$ and $\alpha \xi =1+O_1\left(4d^2\delta \right)$. We can therefore rewrite (A24) as:

$$(r^2+O_1\left(d^2\delta \right))(1+O_1(\delta /2))-(1+O_1\left(4d^2\delta \right))=O_1\left(\beta R\right).$$

Using the bound $r^2\le R\le 4d$ and the definition of $\delta$, we obtain:

$$\begin{array}{cc}\hfill r^2-1& =O_1\left(\frac{\delta r^2}{2}+d^2\delta +\frac{d^2{\delta }^2}{2}+4d^2\delta +4d\beta \right)\hfill \\ \hfill & =O_1(8d^2\delta +4d\beta )\hfill \\ \hfill & =O_1\left(258\pi d^6\beta \right)\hfill \end{array}$$

(A36)

Large angle case. 

Here we assume $\overline{\theta }=\pi +O_1\left(\delta \right)$ with $\delta =8\sqrt{\beta }{\pi }^2{d}^4$ and show that it must be ${\Vert x\Vert }^2\approx {\rho }_d^2{\Vert x_{🟉}\Vert }^2$.

From (A33) we know that $\xi =O_1(\delta /\pi )$, while from (A34) we know that $\zeta ={\rho }_d+O_1\left(3d^3\delta \right)$ as long as $8\sqrt{\beta }\pi d^6\le 1$. Moreover for large angles and $\delta <1$, it holds $\cos {\overline{\theta }}_0=-1+O_1({({\overline{\theta }}_0-\pi )}^2/2)=-1+O_1({\delta }^2/2)$. These bounds lead to:

$$\begin{array}{cc}\hfill \alpha & =\xi \cos {\overline{\theta }}_0+\zeta \hfill \\ & ={\rho }_d+O_1\left(\frac{\delta }{\pi }+\frac{{\delta }^3}{2\pi }+3d^3\delta \right)\hfill \\ & ={\rho }_d+O_1\left(4d^3\delta \right),\hfill \end{array}$$

and using ${\rho }_d\le d$:

$$\begin{array}{cc}\hfill \alpha \zeta & ={\rho }_d^2+O_1(4d^3\delta {\rho }_d+3d^3\delta {\rho }_d+12d^6\delta )={\rho }_d^2+O_1\left(20d^6\delta \right),\hfill \\ \hfill \alpha \xi & =O_1(\frac{\delta }{\pi }{\rho }_d+4\frac{d^3{\delta }^2}{\pi })=O_1\left(2d^3\delta \right).\hfill \end{array}$$

Then recall that (A24) is equivalent to $(r^2-\alpha \zeta )\cos {\overline{\theta }}_0-\alpha \xi =O_1\left(4\beta d\right)$, that is:

$$\left(r^2-{\rho }_d^2+O_1\left(20d^6\delta \right)\right)\left(1+O_1({\delta }^2/2)\right)+O_1\left(2d^3\delta \right)=O_1\left(4\beta d\right)$$

and in particular:

$$\begin{array}{cc}\hfill r^2-{\rho }_d^2& =O_1\left(20d^6\delta +10d^6{\delta }^3+\frac{{\rho }_d{\delta }^2}{2}+\frac{r^2{\delta }^2}{2}+2d^3\delta +4\beta d\right)\hfill \\ \hfill & =O_1\left(35d^6\delta +4\beta d\right)\hfill \\ \hfill & =O_1\left(281{\pi }^2\sqrt{\beta }{d}^{10}\right)\hfill \end{array}$$

(A37)

where we used ${\rho }_d\le d$, the definition of $\delta$ and $\delta <1$.

Controlling the distance. 

We have shown that it is either ${\overline{\theta }}_0\approx 0$ and ${\Vert x\Vert }^2\approx {\Vert x_{🟉}\Vert }^2$ or ${\overline{\theta }}_0\approx \pi$ and ${\Vert x\Vert }^2\approx {\rho }_d^2{\Vert x_{🟉}\Vert }^2$. We can therefore conclude that it must be either $x\approx x_{🟉}$ or $x\approx -{\rho }_d{x}_{🟉}$.

Observe that if a two dimensional point is known to have magnitude within $\Delta r$ of some r and is known to be within an angle $\Delta \theta$ from 0, then its Euclidean distance to the point of coordinates $(r,0)$ is no more that $\Delta r+(r+\Delta r)\Delta \theta$. Similarly we can write:

$$\Vert x-x_{🟉}\Vert \le |\Vert x\Vert -\Vert x_{🟉}\Vert |+(\Vert x_{🟉}\Vert +|\Vert x\Vert -\Vert x_{🟉}\Vert \left|\right){\overline{\theta }}_0.$$

(A38)

In the small angle case, by (A36), (A38), and $\Vert x_{🟉}\Vert |\Vert x\Vert -\Vert x_{🟉}\Vert |\le {|\Vert x\Vert }^2-\Vert x_{🟉}{\Vert }^2|$, we have:

$$\Vert x-x_{🟉}\Vert \le 258\pi d^6\beta +(1+258\pi d^6\beta )32d^4\pi \beta \le 550\pi d^{10}\beta .$$

Next we notice that ${\rho }_2=1/\pi$ and ${\rho }_d\ge {\rho }_{d-1}$ as follows from the definition and (A31), (A32). Then considering the large angle case and using (A37) we have:

$$|\Vert x\Vert -{\rho }_d|\le \frac{281{\pi }^2\sqrt{\beta }{d}^{10}}{\Vert x\Vert +{\rho }_d}\le 281{\pi }^3\sqrt{\beta }{d}^{10}.$$

The latter, together with (A38), yields:

$$\begin{array}{cc}\hfill \Vert x+{\rho }_d{x}_{🟉}\Vert & \le |\Vert x\Vert -{\rho }_d|+({\rho }_d+|\Vert x\Vert -{\rho }_d\left|\right)(\pi -{\overline{\theta }}_0)\hfill \\ & \le 281{\pi }^3\sqrt{\beta }{d}^{10}+(d+281{\pi }^3\sqrt{\beta }{d}^{10})8\sqrt{\beta }{\pi }^2{d}^4\hfill \\ & \le 284{\pi }^3\sqrt{\beta }{d}^{10}\hfill \end{array}$$

where in the second inequality we have used ${\rho }_d\le d$ and in the third $8\sqrt{\beta }\pi d^6\le 1$.

We conclude by noticing that ${\rho }_d\to 1$ as $d\to 1$ as shown in [16] (Lemma 16). □

We next will show that the values of the loss function in a neighborhood of $x_{🟉}$ are strictly smaller that those in a neighborhood of $-{\rho }_d{x}_{🟉}$.

Recall that ${f\left(x\right):=1/4\Vert G\left(x\right)G\left(x\right)}^{\mathsf{T}}-G\left(x_{🟉}\right)G{\left(x_{🟉}\right)}^{\mathsf{T}}{-H\Vert }_F^2$, we next define the following loss functions:

$$\begin{array}{cc}\hfill f_0\left(x\right)& :=\frac{1}{4}{\Vert G\left(x\right)G{\left(x\right)}^{\mathsf{T}}-G\left(x_{🟉}\right)G{\left(x_{🟉}\right)}^{\mathsf{T}}\Vert }_F^2,\hfill \\ \hfill f_H\left(x\right)& :=f_0\left(x\right)-\frac{1}{2}{⟨G\left(x\right)G{\left(x\right)}^{\mathsf{T}}-G\left(x_{🟉}\right)G{\left(x_{🟉}\right)}^{\mathsf{T}},H⟩}_F,\hfill \\ \hfill f_E\left(x\right)& :=\frac{1}{2^{2d+2}}\left({\Vert x\Vert }^4+{\Vert x_{🟉}\Vert }^4-2{⟨x,{\tilde{h}}_x⟩}^2\right).\hfill \end{array}$$

In particular notice that $f\left(x\right)=f_H\left(x\right)+1/4{\Vert H\Vert }_F^2$. Below we show that assuming the WDC is satisfied, $f_0\left(x\right)$ concentrates around $f_E\left(x\right)$.

Lemma A14.

Suppose that $d\ge 2$ and the WDC holds with $ϵ<1/{\left(16\pi d^2\right)}^2$, then for all nonzero $x,x_{🟉}\in {\mathbb{R}}^k$

$$|f_0\left(x\right)-f_E\left(x\right)|\le \frac{16}{2^{2d}}{(\Vert x\Vert }^4+\Vert x_{🟉}{\Vert }^4)d^4\sqrt{ϵ}$$

Proof. 

Observe that:

$$\begin{array}{cc}\hfill |f_0\left(x\right)-f_E\left(x\right)|& \le \frac{1}{4}{|\Vert G\left(x\right)\Vert }^4-\frac{1}{2^{2d}}{\Vert x\Vert }^4|\hfill \\ & +\frac{1}{4}|\Vert G\left(x_{🟉}\right){\Vert }^4-\frac{1}{2^{2d}}\Vert x_{🟉}{\Vert }^4|\hfill \\ & +\frac{1}{2}|{⟨G\left(x\right),G\left(x_{🟉}\right)⟩}^2-\frac{1}{2^{2d}}{⟨x,{\tilde{h}}_x⟩}^2|.\hfill \end{array}$$

We analyze each term separately. The first term can be bounded as:

$$\begin{array}{cc}\hfill \frac{1}{4}{|\Vert G\left(x\right)\Vert }^4-\frac{1}{2^{2d}}{\Vert x\Vert }^4|& =\frac{1}{4}{|\Vert G\left(x\right)\Vert }^2+\frac{1}{2^d}{\Vert x\Vert }^2{\left|\right|\Vert G\left(x\right)\Vert }^2-\frac{1}{2^d}{\Vert x\Vert }^2|\hfill \\ \hfill & \le \frac{1}{4}\frac{1}{2^d}(\frac{13}{12}+1){\Vert x\Vert }^2{|\Vert G\left(x\right)\Vert }^2-\frac{1}{2^d}{\Vert x\Vert }^2|\hfill \\ \hfill & \le \frac{1}{4}\frac{1}{2^d}(\frac{13}{12}+1){\Vert x\Vert }^{2}24\frac{d^3\sqrt{ϵ}}{2^d}{\Vert x\Vert }^2\hfill \\ \hfill & \le \frac{1}{2^{2d}}13d^3\sqrt{ϵ}{\Vert x\Vert }^4\hfill \end{array}$$

where in the first inequality we used (A6) and in the second inequality (A5). Similarly we can bound the second term:

$$\frac{1}{4}|\Vert G\left(x_{🟉}\right){\Vert }^4-\frac{1}{2^{2d}}\Vert x_{🟉}{\Vert }^4|\le \frac{1}{2^{2d}}13d^3\sqrt{ϵ}{\Vert x_{🟉}\Vert }^4.$$

We next note that $\Vert {\tilde{h}}_x\Vert \le (1+d/\pi )\Vert x_{🟉}\Vert$ and therefore from (A6) and $d\ge 2$ we obtain:

$$|\Vert G\left(x\right)\Vert \Vert G\left(x_{🟉}\right)\Vert +\Vert x\Vert \frac{\Vert {\tilde{h}}_x\Vert }{2^d}|\le \frac{1}{2^d}(\frac{13}{12}+1+\frac{d}{\pi })\Vert x\Vert \Vert x_{🟉}\Vert \le \frac{1}{2^d}\frac{3}{2}d\Vert x\Vert \Vert x_{🟉}\parallel$$

We can then conclude that:

$$\begin{array}{cc}\hfill \frac{1}{2}|{⟨G\left(x\right),G\left(x_{🟉}\right)⟩}^2-{⟨x,\frac{{\tilde{h}}_x}{2^d}⟩}^2|& \le \frac{1}{2}|⟨x,{\Lambda }_x^T{\Lambda }_{x_{🟉}}{x}_{🟉}-{\tilde{h}}_x⟩\left|\right|\Vert G\left(x\right)\Vert \Vert G\left(x_{🟉}\right)\Vert +\Vert x\Vert \Vert {\tilde{h}}_x\Vert |\hfill \\ & \le \frac{1}{2}\Vert x\Vert 24\frac{d^3\sqrt{ϵ}}{2^d}\Vert x_{🟉}\Vert \frac{1}{2^d}\frac{3}{2}d\Vert x\Vert \Vert x_{🟉}\parallel \hfill \\ & \le \frac{9}{2^{2d}}{d}^4\sqrt{ϵ}(\Vert x_{🟉}{\Vert }^4+{\Vert x\Vert }^4)\hfill \end{array}.$$

□

We next consider the loss $f_E$ and show that in a neighborhood $-{\rho }_d{x}_{🟉}$, this loss function has larger values than in a neighborhood of $x_{🟉}$.

Lemma A15.

Fix $0<a\le 1/\left(2{\pi }^3{d}^3\right)$ and ${\varphi }_d\in [{\rho }_d,1]$ then:

$$\begin{array}{cc}\hfill f_E\left(x\right)& \le \frac{1}{2^{2d+2}}\Vert x_{🟉}{\Vert }^4+\frac{1}{2^{2d+2}}[{(a+{\varphi }_d)}^4-2{\varphi }_d^2+2\pi da]\Vert x_{🟉}{\Vert }^4\forall x\in \mathcal{B}({\varphi }_d{x}_{🟉},a\Vert x_{🟉}\Vert )\mathrm{and}\hfill \\ \hfill f_E\left(x\right)& \ge \frac{1}{2^{2d+2}}\Vert x_{🟉}{\Vert }^4+\frac{1}{2^{2d+2}}[{(a-{\varphi }_d)}^4-2{\rho }_d^2{\varphi }_d^2-40\pi d^{3}a]\Vert x_{🟉}{\Vert }^4\forall x\in \mathcal{B}(-{\varphi }_d{x}_{🟉},a\Vert x_{🟉}\Vert ).\hfill \end{array}$$

Proof. 

Let $x\in \mathcal{B}({\varphi }_d{x}_{🟉},a\Vert x_{🟉}\Vert )$ then observe that $0\le {\overline{\theta }}_i\le {\overline{\theta }}_0\le \pi a/2{\varphi }_d$ and $({\varphi }_d-a)\Vert x_{🟉}\Vert \le \Vert x\Vert \le (a+{\varphi }_d)\Vert x_{🟉}\Vert$. Then observe that:

$$\begin{array}{cc}\hfill ⟨x,\frac{{\tilde{h}}_d}{2^d}⟩& =\frac{1}{2^d}(\prod _{i=0}^{d-1}\frac{\pi -{\overline{\theta }}_i}{\pi })\Vert x_{🟉}\Vert \Vert x\Vert \cos {\overline{\theta }}_0+\frac{1}{2^d}\sum _{i=0}^{d-1}\frac{\sin {\overline{\theta }}_i}{\pi }\prod _{j=i+1}^{d-1}\frac{\pi -{\overline{\theta }}_j}{\pi }\Vert x_{🟉}\Vert \Vert x\Vert \hfill \\ & \ge \frac{1}{2^d}(\prod _{i=0}^{d-1}\frac{\pi -\frac{\pi a}{2{\varphi }_d}}{\pi })({\varphi }_d-a){\Vert x_{🟉}\Vert }^2(1-\frac{{\pi }^2{a}^2}{8{\varphi }_d^2})\hfill \\ & \ge \frac{1}{2^d}(1-\frac{da}{{\varphi }_d})({\varphi }_d-a)(1-\frac{{\pi }^2{a}^2}{8{\varphi }_d^2}){\Vert x_{🟉}\Vert }^2.\hfill \end{array}$$

using $\cos \theta \ge 1-{\theta }^2/2$ and ${(1-x)}^d\ge (1-2dx)$ as long as $0\le x\le 1$. We can therefore write:

$$\begin{array}{cc}\hfill f_E\left(x\right)-\frac{\Vert x_{🟉}{\Vert }^4}{2^{2d+2}}& \le \frac{1}{2^{2d+2}}{\Vert x\Vert }^4-\frac{1}{2^{2d+1}}{(1-\frac{da}{{\varphi }_d})}^2{({\varphi }_d-a)}^2{(1-\frac{{\pi }^2{a}^2}{8{\varphi }_d^2})}^2{\Vert x_{🟉}\Vert }^4\hfill \\ & \le \frac{1}{2^{2d+2}}[{({\varphi }_d+a)}^4-2(1-2\frac{da}{{\varphi }_d}){({\varphi }_d-a)}^2(1-\frac{{\pi }^2{a}^2}{4{\varphi }_d^2})]{\Vert x_{🟉}\Vert }^4\hfill \end{array}$$

where in the second inequality we used ${(1-x)}^2\ge 1-2x$ for all $x\in \mathbb{R}$. We then observe that:

$$\begin{array}{cc}\hfill (1-2\frac{da}{{\varphi }_d}){({\varphi }_d-a)}^2(1-\frac{{\pi }^2{a}^2}{4{\varphi }_d^2})& \ge (1-\frac{{\pi }^2{a}^2}{4{\varphi }_d^2}-\frac{2ad}{{\varphi }_d}){\varphi }_d^2+a(a-2{\varphi }_d)(1-2\frac{da}{{\varphi }_d})(1-\frac{{\pi }^2{a}^2}{4{\varphi }_d^2})\hfill \\ & \ge {\varphi }_d^2-a(\frac{1}{2\pi d^3}+2d{\varphi }_d)+a(a-2{\varphi }_d)(1-2\frac{da}{{\varphi }_d})(1-\frac{{\pi }^2{a}^2}{4{\varphi }_d^2})\hfill \\ & \ge {\varphi }_d^2-a(\frac{1}{2\pi d^3}+2d{\varphi }_d+2{\varphi }_d)\hfill \\ & \ge {\varphi }_d^2-\pi da,\hfill \end{array}$$

where in the second inequality we have used ${\pi }^3{d}^{3}a\le 2$ and in the last one $d\ge 2$ and ${\varphi }_d\le 1$. We can then conclude that:

$$f_E\left(x\right)-\frac{\Vert x_{🟉}{\Vert }^4}{2^{2d+2}}\le \frac{1}{2^{2d+2}}\left[{({\varphi }_d+a)}^4-2({\varphi }_d^2-\pi da)\right]{\Vert x_{🟉}\Vert }^4$$

We next take $x\in \mathcal{B}(-{\varphi }_d{x}_{🟉},a\Vert x_{🟉}\Vert )$ which implies $0\le \pi -{\overline{\theta }}_0\le {\pi }^{2}a/2=:\delta$ and $\Vert x\Vert \le (a+{\varphi }_d)\Vert x_{🟉}\Vert$. We then note that for $\xi$ and $\zeta$ as defined in (A23) we have:

$$\begin{array}{cc}\hfill |x^{\mathsf{T}}{\tilde{h}}_x{|}^2& \le \left(\right|\xi |+{\left|\zeta \right|)}^2{(a+{\varphi }_d)}^2{\Vert x_{🟉}\Vert }^4\hfill \\ & \le {(\frac{\delta }{\pi }+3d^3\delta +{\rho }_d)}^2{(a+{\varphi }_d)}^2{\Vert x_{🟉}\Vert }^4\hfill \\ & \le {(\frac{{\pi }^3{d}^3}{2}a+{\rho }_d)}^2{(a+\varphi )}^2{\Vert x_{🟉}\Vert }^4\hfill \\ & \le (2{\pi }^3{d}^{3}a+{\rho }_d^2){(a+{\varphi }_d)}^2{\Vert x_{🟉}\Vert }^4\hfill \\ & \le 20\pi d^{3}a+{\rho }_d^2{\varphi }_d^2\hfill \end{array}$$

where the second inequality is due to (A33) and (A34), the rest from $d\ge 2$, ${\rho }_d\le {\varphi }_d\le 1$ and $2{\pi }^3{d}^{3}a\le 1$. Finally using $({\varphi }_d-a)\Vert x_{🟉}\Vert \le \Vert x\Vert$, we can then conclude that:

$$f_E\left(x\right)-\frac{\Vert x_{🟉}{\Vert }^4}{2^{2d+2}}\ge \frac{1}{2^{2d+2}}\left[{({\varphi }_d-a)}^4-2(20\pi d^{3}a+{\rho }_d^2{\varphi }_d^2)\right]{\Vert x_{🟉}\Vert }^4.$$

□

The above two lemmas are now used to prove Lemma A8.

Proof of Proposition A8. 

Let $x\in \mathcal{B}(\pm {\varphi }_d{x}_{🟉},\phi \Vert x_{🟉}\Vert )$ for a $0<\phi <1$ that will be specified below, and observe that by the assumptions on the noise:

$$\begin{array}{cc}\hfill |{⟨G\left(x\right)G{\left(x\right)}^{\mathsf{T}}-G\left(x_{🟉}\right)G{\left(x_{🟉}\right)}^{\mathsf{T}},H⟩}_F|& {\le |G\left(x\right)}^{\mathsf{T}}HG\left(x\right)|+|G{\left(x_{🟉}\right)}^{\mathsf{T}}HG\left(x_{🟉}\right)|\hfill \\ & \le \frac{\omega }{2^d}{(\Vert x\Vert }^2+\Vert x_{🟉}{\Vert }^2)\hfill \\ & \le \frac{\omega }{2^d}({({\varphi }_d+\phi )}^2+1){\Vert x_{🟉}\Vert }^2,\hfill \end{array}$$

and therefore by Lemma A14:

$$\begin{array}{cc}\hfill |f_0\left(x\right)-f_E\left(x\right)|& +\frac{1}{2}\left|{⟨G\left(x\right)G{\left(x\right)}^{\mathsf{T}}-G\left(x_{🟉}\right)G{\left(x_{🟉}\right)}^{\mathsf{T}},H⟩}_F\right|\le \hfill \\ & \le \frac{16}{2^{2d}}({({\varphi }_d+\phi )}^4+1)\Vert x_{🟉}{\Vert }^4{d}^4\sqrt{ϵ}+\frac{\omega }{2^d}({({\varphi }_d+\phi )}^2+1){\Vert x_{🟉}\Vert }^2\hfill \\ & \le \frac{272}{2^{2d}}\Vert x_{🟉}{\Vert }^4{d}^4\sqrt{ϵ}+\frac{\omega }{2^d}({({\varphi }_d+\phi )}^2+1){\Vert x_{🟉}\Vert }^2\hfill \end{array}$$

We next take $\phi =ϵ$ and $x\in \mathcal{B}({\varphi }_d{x}_{🟉},\phi \Vert x_{🟉}\Vert )$, so that by Lemma A15 and the assumption $2^d{d}^{44}w\le K_2{\Vert x_{🟉}\Vert }^2$, we have:

$$\begin{array}{cc}\hfill f_H\left(x\right)& \le f_E\left(x\right)+|f_0\left(x\right)-f_E\left(x\right)|+\frac{1}{2}\left|{⟨G\left(x\right)G{\left(x\right)}^{\mathsf{T}}-G\left(x_{🟉}\right)G{\left(x_{🟉}\right)}^{\mathsf{T}},H⟩}_F\right|\hfill \\ & \le \frac{1}{2^{2d+2}}\left[1+{(ϵ+{\varphi }_d)}^4-2{\varphi }_d^2+2\pi dϵ\right]\Vert x_{🟉}{\Vert }^4+272d^4\sqrt{ϵ}\Vert x_{🟉}{\Vert }^4+\frac{\omega }{2^{d+1}}(2+2ϵ+{ϵ}^2){\Vert x_{🟉}\Vert }^2\hfill \\ & \le \frac{1}{2^{2d+2}}\left[1-2{\varphi }_d^2+{(ϵ+{\varphi }_d)}^4\right]\Vert x_{🟉}{\Vert }^4+\frac{1}{2^{2d}}\left(\frac{3}{2}{2}^d{\Vert x_{🟉}\Vert }^{-2}\omega +\frac{\pi d}{2}+272d^4\right)\sqrt{ϵ}\Vert x_{🟉}{\Vert }^4+\frac{\omega }{2^d}{\Vert x_{🟉}\Vert }^2\hfill \\ & \le \frac{1}{2^{2d+2}}\left[1-2{\varphi }_d^2+{(ϵ+{\varphi }_d)}^4\right]\Vert x_{🟉}{\Vert }^4+\frac{1}{2^{2d}}\left(\frac{3}{2}{K}_2{d}^{-44}+\frac{\pi d}{2}+272d^4\right)\sqrt{ϵ}{\Vert x_{🟉}\Vert }^4+K_2\frac{\Vert x_{🟉}{\Vert }^4}{2^{2d}}{d}^{-44}.\hfill \end{array}$$

Similarly if $y\in \mathcal{B}(-{\varphi }_d{x}_{🟉},\phi \Vert x_{🟉}\Vert )$, and $\phi =ϵ$ we obtain:

$$\begin{array}{cc}\hfill f_H\left(y\right)& \ge f_E\left(y\right)-|f_0\left(y\right)-f_E\left(y\right)|-\frac{1}{2}\left|⟨G\left(y\right)G{\left(y\right)}^{\mathsf{T}}-G\left(x_{🟉}\right)G{\left(x_{🟉}\right)}^{\mathsf{T}},H⟩\right|\hfill \\ & \ge \frac{1}{2^{2d+2}}\left[1-2{\varphi }_d^2{\rho }_d^2+{(ϵ-{\varphi }_d)}^4\right]\Vert x_{🟉}{\Vert }^4-\frac{1}{2^{2d}}\left(\frac{3}{2}{2}^d{\Vert x_{🟉}\Vert }^{-2}\omega +10\pi d^3+272d^4\right)\sqrt{ϵ}\Vert x_{🟉}{\Vert }^4-\frac{\omega }{2^d}{\Vert x_{🟉}\Vert }^2\hfill \\ & \ge \frac{1}{2^{2d+2}}\left[1-2{\varphi }_d^2{\rho }_d^2+{(ϵ-{\varphi }_d)}^4\right]\Vert x_{🟉}{\Vert }^4-\frac{1}{2^{2d}}\left(\frac{3}{2}{K}_2{d}^{-44}+10\pi d^3+272d^4\right)\sqrt{ϵ}{\Vert x_{🟉}\Vert }^4-K_2\frac{\Vert x_{🟉}{\Vert }^4}{2^{2d}}{d}^{-44}.\hfill \end{array}$$

In order to guarantee that $f\left(y\right)>f\left(x\right)$, it suffices to have:

$$2(1-{\rho }_d^2){\varphi }_d^2-8K_2{d}^{-44}>4C_d\sqrt{ϵ}$$

with $C_d:=(544d^4+10\pi d^3\pi +3K_2{d}^{-44}+\pi d/2+1/100)$, that is to require:

$$\phi =ϵ<{\left(\frac{(1-{\rho }_d^2){\varphi }_d^2-4K_2{d}^{-44}}{2C_d}\right)}^2.$$

Finally notice that by Lemma 17 in [16] it holds that $1-{\rho }_d\ge {\left(K(d+2)\right)}^{-2}$ for some numerical constant K, we therefore choose $\varphi =\varrho /d^{12}$ for some $\varrho >0$ small enough. □

Appendix B.5. Supplementary Proofs for Appendix A.6

In this section we use strong convexity and smoothness to prove convergence to $x_{🟉}$ up to the noise variance $\omega$. The idea is to show that every vector in the subgradient points in the direction $(x-x_{🟉})$. Recall that the gradient in the noiseless case was:

$${\overline{v}}_x={\Lambda }_x^{\mathsf{T}}[{\Lambda }_{x}x{x}^{\mathsf{T}}{\Lambda }_x^{\mathsf{T}}-{\Lambda }_{x_{🟉}}{x}_{🟉}{x_{🟉}}^{\mathsf{T}}{\Lambda }_{x_{🟉}}^{\mathsf{T}}]{\Lambda }_{x}x.$$

We show that by continuity of ${\Lambda }_x$, when x is close to $x_{🟉}$, then ${\overline{v}}_x$ it is close to:

$${\overline{\overline{v}}}_x:={\Lambda }_x^{\mathsf{T}}{\Lambda }_x[x{x}^{\mathsf{T}}-x_{🟉}{x_{🟉}}^{\mathsf{T}}]{\Lambda }_x^{\mathsf{T}}{\Lambda }_{x}x.$$

which in turn concentrates around:

$${\check{v}}_x:=\frac{1}{2^{2d}}[x{x}^{\mathsf{T}}-x_{🟉}{x_{🟉}}^{\mathsf{T}}]x$$

by the WDC.

We begin by recalling the following result which can be found in the proof of Lemma 22 of [16].

Lemma A16.

Suppose $x\in \mathcal{B}(x_{🟉},d\sqrt{ϵ}\Vert x_{🟉}\Vert )$ and the WDC holds with $ϵ<1/\left({200}^4{d}^6\right)$. Then it holds that:

$$\parallel {\Lambda }_x^{\mathsf{T}}{\Lambda }_x{x}_{🟉}-{\Lambda }_x^{\mathsf{T}}{\Lambda }_{x_{🟉}}{x}_{🟉}\Vert \le \frac{1}{16}\frac{1}{2^d}\Vert x-x_{🟉}\Vert .$$

We now prove that ${\overline{v}}_x\approx {\overline{\overline{v}}}_x$ for x close to $x_{🟉}$.

Lemma A17.

Suppose $x\in \mathcal{B}(x_{🟉},d\sqrt{ϵ}\Vert x_{🟉}\Vert )$ and the WDC holds with $ϵ<1/\left({200}^4{d}^6\right)$. Then it holds that:

$$\parallel {\overline{v}}_x-{\overline{\overline{v}}}_x\Vert \le \frac{13}{96}\frac{1}{2^{2d}}\Vert x\Vert \Vert x_{🟉}\Vert \Vert x-x_{🟉}\parallel$$

Proof. 

Let $g_{x,🟉}:={\Lambda }_x^{\mathsf{T}}{\Lambda }_x{x}_{🟉}$ and $q_{x,🟉}:={\Lambda }_x^{\mathsf{T}}{\Lambda }_{x_{🟉}}{x}_{🟉}$. Then observe that:

$$\begin{array}{cc}\hfill \Vert {\overline{v}}_x-{\overline{\overline{v}}}_x\Vert & =\Vert ⟨g_{x,🟉},x⟩g_{x,🟉}-⟨q_{x,🟉},x⟩q_{x,🟉}\Vert \hfill \\ & \le (\Vert x\Vert \Vert g_{x,🟉}\Vert +\Vert {\Lambda }_{x}x\Vert \Vert {\Lambda }_{x_{🟉}}{x}_{🟉}\Vert )\Vert g_{x,🟉}-q_{x,🟉}\parallel \hfill \\ & \le \frac{13}{6}\frac{1}{2^d}\Vert x\Vert \Vert x_{🟉}\Vert \Vert g_{x,🟉}-q_{x,🟉}\parallel \hfill \\ & \le \frac{13}{96}\frac{1}{2^{2d}}\Vert x\Vert \Vert x_{🟉}\Vert \Vert x-x_{🟉}\parallel \hfill \end{array}$$

where the second inequality follows from Lemma A1, and the third from Lemma A16. □

We next prove that by the WDC ${\overline{\overline{v}}}_x\approx {\check{v}}_x$.

Lemma A18.

Suppose the WDC holds with $ϵ<1/{\left(16\pi d^2\right)}^2$. Then for all nonzero $x,x_{🟉}$:

$$\parallel {\overline{\overline{v}}}_x-{\check{v}}_x\Vert \le \frac{25}{12}\frac{4ϵd}{2^{2d}}\Vert x\Vert \Vert x-x_{🟉}\Vert \Vert x+x_{🟉}\parallel$$

Proof. 

For notational convenience we define $E_d:=I_k/2^d$ the scaled identity in ${\mathbb{R}}^d$, $Q_x:={\Lambda }_x^{\mathsf{T}}{\Lambda }_x$ and $M_{x,🟉}=x{x}^{\mathsf{T}}-x_{🟉}{x_{🟉}}^{\mathsf{T}}$. Next observe that:

$$\begin{array}{cc}\hfill \Vert {\overline{\overline{v}}}_x-{\check{v}}_x\Vert & \le \Vert (Q_x-E_d)M_{x,🟉}{Q}_{x}x\Vert +\Vert E_d{M}_{x,🟉}(Q_x-E_d)x\Vert ,\hfill \\ & \le \frac{25}{12}\frac{1}{2^d}\Vert Q_x-E_d\Vert \Vert M_{x,🟉}\Vert x\Vert |\hfill \\ & \le \frac{25}{12}\frac{4ϵd}{2^{2d}}\Vert M_{x,🟉}\Vert \Vert x\Vert \hfill \end{array}$$

where the second inequality follows from Lemma A1 and the third from (17) in [15]. We conclude by noticing that $\Vert M_{x,🟉}\Vert \le \Vert x-x_{🟉}\Vert \Vert x+x_{🟉}\Vert$. □

Next consider the quartic function $\check{f}\left(x\right):=1/2^{2d+2}{\Vert x{x}^{\mathsf{T}}-x_{🟉}{x_{🟉}}^{\mathsf{T}}\Vert }_F^2$ and observe that:

$$\begin{array}{cc}\hfill \nabla \check{f}\left(x\right)& =\frac{1}{2^{2d}}(x{x}^{\mathsf{T}}-x_{🟉}{x_{🟉}}^{\mathsf{T}})x={\check{v}}_x,\hfill \\ \hfill {\nabla }^2\check{f}\left(x\right)& =\frac{1}{2^{2d}}{(\Vert x\Vert }^2{I}_n+2x{x}^{\mathsf{T}}-x_{🟉}{x_{🟉}}^{\mathsf{T}}).\hfill \end{array}$$

Following [63] we show that ${\check{v}}_x$ is $\beta$-smooth and strongly convex, in turn deriving the result in Lemma A19.

Lemma A19.

Assume $\Vert x-x_{🟉}\Vert \le \gamma \Vert x_{🟉}\Vert$ with $\gamma <1/5$. Take $\tau \ge 3{(1+\gamma )}^2+1$, then:

$$\parallel {\check{v}}_x-\frac{\tau }{2^{2d}}\Vert x_{🟉}{\Vert }^2(x-x_{🟉})\Vert \le \sqrt{\tau (\tau -\alpha )}\Vert x-x_{🟉}\Vert \frac{\Vert x_{🟉}{\Vert }^2}{2^{2d}}$$

where $\alpha =2-9\gamma$.

Proof. 

Let $\Vert x-x_{🟉}\Vert \le \gamma \Vert x_{🟉}\Vert$ with $\gamma <1/5$, then:

$$\begin{array}{c}\hfill (1-\gamma )\Vert x_{🟉}\Vert \le \Vert x\Vert \le (1+\gamma )\Vert x_{🟉}\Vert \end{array}$$

(A39)

$$\begin{array}{c}\hfill \Vert \Delta \Vert \Vert x\Vert \le \gamma (1+\gamma )\Vert x_{🟉}{\Vert }^2\end{array}$$

(A40)

since $\Delta =x-x_{🟉}$ satisfies $\Vert \Delta \Vert \le \Vert x\Vert$. Using (A40) we then obtain

$$\begin{array}{cc}\hfill x{x}^T& =x_{🟉}{x_{🟉}}^T+\Delta x^{\mathsf{T}}+x{\Delta }^T-\Delta {\Delta }^{\mathsf{T}},\hfill \\ & ⪰x_{🟉}{x_{🟉}}^{\mathsf{T}}-3\Vert \Delta \Vert \Vert x\Vert I_k,\hfill \\ & ⪰x_{🟉}{x_{🟉}}^{\mathsf{T}}-3\gamma (1+\gamma ){\Vert x_{🟉}\Vert }^2{I}_k.\hfill \end{array}$$

We therefore have:

$$\begin{array}{cc}\hfill 2^{2d}{\nabla }^2\check{f}\left(x\right)& ={\Vert x\Vert }^2{I}_k+2x{x}^{\mathsf{T}}-x_{🟉}{x_{🟉}}^{\mathsf{T}}\hfill \\ & ⪰{\Vert x\Vert }^2{I}_k+x_{🟉}{x_{🟉}}^{\mathsf{T}}-6\gamma (1+\gamma ){\Vert x_{🟉}\Vert }^2{I}_k,\hfill \\ & ⪰({(1-\gamma )}^2+1-6\gamma (1+\gamma )){\Vert x_{🟉}\Vert }^2{I}_k,\hfill \\ & ⪰(2-9\gamma )\Vert x_{🟉}{\Vert }^2{I}_k,\hfill \end{array}$$

where in the second line we have used (A39) and in the third $0<\gamma <1/5$.

Next notice that using (A39) we have

$$2^{2d}\Vert {\nabla }^2\check{f}{\left(x\right)\Vert \le 3\Vert x\Vert }^2+\Vert x_{🟉}{\Vert }^2\le (3{(1+\gamma )}^2+1){\Vert x_{🟉}\Vert }^2$$

and therefore, for all $x\in \mathcal{B}(x_{🟉},\gamma \Vert x_{🟉}\Vert )$

$$(2-9\gamma )\Vert x_{🟉}{\Vert }^2{I}_k⪯2^{2d}{\nabla }^2\check{f}\left(x\right)⪯(3{(1+\gamma )}^2+1){\Vert x_{🟉}\Vert }^2.$$

The above bounds imply in particular that for all $x\in \mathcal{B}(x_{🟉},\gamma \Vert x_{🟉}{\Vert }^2)$ the gradient of $\check{f}$ satisfies the regularity condition (see [63]):

$$2⟨\nabla \check{f}\left(x\right),x-x_{🟉}⟩\ge \mu \Vert \nabla \check{f}{\left(x\right)\Vert }^2+\lambda {\Vert x-x_{🟉}\Vert }^2,$$

(A41)

where $\lambda =(2-9\gamma )\Vert x_{🟉}{\Vert }^2/2^{2d}$ and $1/\mu =(3{(1+\gamma )}^2+1){\Vert x_{🟉}\Vert }^2/2^{2d}$. By (A41) we can then conclude that for $\sigma \ge 1/\mu$:

$$\begin{array}{cc}\hfill \Vert \nabla \check{f}\left(x\right)-\sigma (x-x_{🟉}){\Vert }^2& \le \Vert \nabla \check{f}{\left(x\right)\Vert }^2+{\sigma }^2{\Vert x-x_{🟉}\Vert }^2-2\sigma ⟨\nabla \check{f}\left(x\right),x-x_{🟉}⟩\hfill \\ & \le (1-\sigma \mu )\Vert \nabla \check{f}{\left(x\right)\Vert }^2+\sigma (\sigma -\lambda ){\Vert x-x_{🟉}\Vert }^2\hfill \\ & \le \sigma (\sigma -\lambda )\Vert x-x_{🟉}{\Vert }^2\hfill \end{array}$$

Finally letting $\sigma =\tau \Vert x_{🟉}{\Vert }^2/2^d$, $\alpha =(2-9\gamma )$, and by the assumptions on $\gamma$ and $\tau$ the thesis follows. □

We finally can prove Lemma A9.

Proof of Lemma A9. 

Let $x\in \mathcal{B}(x_{🟉},d\sqrt{ϵ}\Vert x_{🟉}\Vert )$, then $\Vert x+x_{🟉}\Vert \le (2+d\sqrt{ϵ})\Vert x_{🟉}\Vert$ and since $ϵ<1/\left({200}^4{d}^6\right)$ we have from Lemma A17 and Lemma A18

$$\parallel {\overline{v}}_x-{\check{v}}_x\Vert \le \Vert {\overline{v}}_x-{\overline{\overline{v}}}_x\Vert +\Vert {\overline{\overline{v}}}_x-{\check{v}}_x\Vert \le \frac{1}{2^{2d}}\frac{13}{48}\Vert x\Vert \Vert x_{🟉}\Vert \Vert x-x_{🟉}\Vert .$$

If $x\in \mathcal{B}(x_{🟉},d\sqrt{ϵ}\Vert x_{🟉}\Vert )$ is a differentiable point of f, then by Lemma A19 and the assumption (13) on the noise H, it holds that:

$$\parallel {\tilde{v}}_x-\frac{\tau }{2^{2d}}\Vert x_{🟉}{\Vert }^2(x-x_{🟉})\Vert \le \left[\frac{13}{48}(1+\gamma )+\sqrt{\tau (\tau -\alpha )}\right]\frac{\Vert x_{🟉}{\Vert }^2}{2^{2d}}\Vert x-x_{🟉}\Vert +\frac{\omega }{2^d}(1+\gamma )\Vert x_{🟉}\Vert .$$

where $\gamma =d\sqrt{ϵ}$, $\alpha =2-9\gamma$ and $\tau \ge 3{(1+\gamma )}^2+1$.

In general, if $x\in \mathcal{B}(x_{🟉},\gamma \Vert x_{🟉}\Vert )$ and $v_x\in \partial f\left(x\right)$, by the characterization of Clarke subdifferential (9) and the previous results:

$$\begin{array}{cc}\hfill \parallel v_x-\frac{\tau }{2^{2d}}\Vert x_{🟉}{\Vert }^2(x-x_{🟉})\Vert & \le \sum _{t=1}^T{c}_t\Vert v_t-\frac{\tau }{2^{2d}}\Vert x_{🟉}{\Vert }^2(x-x_{🟉})\Vert \hfill \\ & \le \left[\frac{13}{48}(1+\gamma )+\sqrt{\tau (\tau -\alpha )}\right]\frac{\Vert x_{🟉}{\Vert }^2}{2^{2d}}\Vert x-x_{🟉}\Vert +\frac{\omega }{2^d}(1+\gamma )\Vert x_{🟉}\Vert .\hfill \end{array}$$

We finally obtain the thesis by noting that by the assumptions on $ϵ$ it holds that $\gamma =d\sqrt{ϵ}<1/{\left(400\right)}^2$ and taking $\tau ={\tau }_1=21/5$, ${\tau }_2=17/5$ and ${\tau }_3=(1+1/{\left(400\right)}^2)$. □

Appendix C.1. Spiked Wigner Model

Recall that in the Wigner model $Y=G\left(x_{🟉}\right)G{\left(x_{🟉}\right)}^{\mathsf{T}}+\nu \mathcal{H}$ and the symmetric noise matrix $\mathcal{H}$ follows a Gaussian Orthogonal Ensemble GOE$\left(n\right)$, that is ${\mathcal{H}}_{ii}\sim \mathcal{N}(0,2/n)$ for all $1\le i\le n$ and ${\mathcal{H}}_{ji}={\mathcal{H}}_{ij}\sim \mathcal{N}(0,1/n)$ for $1\le j<i\le n$. Our goal is to bound $\Vert {\Lambda }_x^{\mathsf{T}}\mathcal{H}{\Lambda }_x\Vert$ uniformly over x with high probability.

Fix $x\in {\mathbb{R}}^k$, and let ${\mathcal{N}}_{1/4}$ be a $1/4$-net on the sphere $S^{k-1}$ such that (see for example [64]) $|{\mathcal{N}}_{1/4}|\le 9^k$ and

$$\parallel {\Lambda }_x^{\mathsf{T}}\mathcal{H}{\Lambda }_x\Vert \le 2\underset{z\in {\mathcal{N}}_{1/4}}{\max }\left|⟨{\Lambda }_x^{\mathsf{T}}\mathcal{H}{\Lambda }_{x}z,z⟩\right|.$$

Observe next that for any $v\in {\mathbb{R}}^n$ by the definition of GOE(n) it holds that

$$v^{\mathsf{T}}Hv=\sum _i^n{H}_{ii}{v}_i^2+2\sum _{i<j}^n{H}_{ij}{v}_i{v}_j\sim \mathcal{N}(0,2(\sum _i{v}_i^4+2\sum _{i<j}{v}_i^2{v}_j^2)/n)=\mathcal{N}({0,2\Vert v\Vert }^4/n).$$

Therefore for any $z\in {\mathcal{N}}_{1/4}$ let ${\ell }_{x,z}:={\Lambda }_{x}z\in {\mathbb{R}}^n$, then ${\ell }_{x,z}^{\mathsf{T}}\mathcal{H}{\ell }_{x,z}\sim \mathcal{N}(0,2\Vert {\ell }_{x,z}{\Vert }^4/n)$. In particular by (A6), the quadratic form ${\ell }_{x,z}^{\mathsf{T}}\mathcal{H}{\ell }_{x,z}$ is sub-Gaussian with parameter ${\gamma }^2$ given by

$${\gamma }^2:=\frac{2}{n}{\left(\frac{13}{12}\right)}^2\frac{1}{2^{2d}}.$$

Then for fixed $x\in {\mathbb{R}}^k$, standard sub-Gaussian tail bounds (e.g., [64]) and a union bound over ${\mathcal{N}}_{1/4}$ give for any $u\ge 0$

$$\begin{array}{cc}\hfill \mathbb{P}\left[\Vert {\Lambda }_x^{\mathsf{T}}\mathcal{H}{\Lambda }_x\Vert \ge 2u\right]& \le \mathbb{P}[\underset{z\in {\mathcal{N}}_{1/4}}{\max }\Vert {\ell }_{x,z}^{\mathsf{T}}\mathcal{H}{\ell }_{x,z}\Vert \ge u]\hfill \\ & \le \sum _{z\in {\mathcal{N}}_{1/4}}\mathbb{P}\left[\Vert {\ell }_{x,z}^{\mathsf{T}}\mathcal{H}{\ell }_{x,z}\Vert \ge u\right]\le 2·9^k{e}^{-\frac{u^2}{2{\gamma }^2}}.\hfill \end{array}$$

Lemma A21, then ensures that the number of possible ${\Lambda }_x$ is at most ${(n_1^d{n}_2^{d-1}\cdots n_d)}^{8k}$, so a union bound over this set allows us to conclude that

$$\begin{array}{cc}\hfill \mathbb{P}\left[\Vert {\Lambda }_x^{\mathsf{T}}\mathcal{H}{\Lambda }_x\Vert \le 2u,\mathrm{for}\mathrm{all}x\right]& \ge 1-{(n_1^d{n}_2^{d-1}\cdots n_d)}^{8k}\mathbb{P}\left[\Vert {\Lambda }_x^{\mathsf{T}}\mathcal{H}{\Lambda }_x\Vert \ge 2u\right]\hfill \\ & \ge 1-2exp\left(8k\log (2n_1^d{n}_2^{d-1}\cdots n_d)-u^2/\left(2{\gamma }^2\right)\right)\hfill \end{array}$$

Choosing then $u=\sqrt{2{\gamma }^2·9k\log (2n_1^d{n}_2^{d-1}\cdots n_d)}$ and substituting the definition of ${\gamma }^2$, we obtain

$$\mathbb{P}[\Vert {\Lambda }_x^{\mathsf{T}}\mathcal{H}{\Lambda }_x\Vert \le \frac{1}{2^d}\sqrt{\frac{169k\log (2n_1^d{n}_2^{d-1}\cdots n_d)}{n}},\mathrm{for}\mathrm{all}x]\ge 1-2e^{-k\log (2n_1^d{n}_2^{d-1}\cdots n_d)}$$

which implies the thesis as $n=n_d$ and $\log \left(n\right)\le \log (2n_1^d{n}_2^{d-1}\cdots n_d)$.

Appendix C.2. Spiked Wishart Model

Each row ${\left\{y_i\right\}}_{i=1}^N$ of the matrix Y in (1) can be seen as i.i.d. samples from $\mathcal{N}(0,\Sigma )$ where $\Sigma =y_{🟉}{y_{🟉}}^{\mathsf{T}}+{\sigma }^2{I}_n$. In the minimization problem (4) we take $M={\Sigma }_N-{\sigma }^2{I}_n$ where ${\Sigma }_N$ is the empirical covariance matrix $Y^{\mathsf{T}}Y/N$. The symmetric noise matrix H is then given by $H={\Sigma }_N-\Sigma$ and by the Law of Large Numbers $H\to 0$ as $N\to \infty$. We bound $\Vert {\Lambda }_x^{\mathsf{T}}H{\Lambda }_x\Vert$ with high probability uniformly over $x\in {\mathbb{R}}^k$.

Fix $x\in {\mathbb{R}}^k$, let ${\mathcal{N}}_{1/4}$ be a $1/4$-net on the sphere $S^{k-1}$ such that $|{\mathcal{N}}_{1/4}|\le 9^k$, and notice that:

$$\Vert {\Lambda }_x^{\mathsf{T}}H{\Lambda }_x\Vert \le 2\underset{z\in {\mathcal{N}}_{1/4}}{\max }\left|z^{\mathsf{T}}{\Lambda }_x^{\mathsf{T}}H{\Lambda }_{x}z\right|.$$

By a union bound on ${\mathcal{N}}_{1/4}$ we obtain for any fixed $z\in {\mathcal{N}}_{1/4}$:

$$\mathbb{P}\left[\Vert {\Lambda }_x^{\mathsf{T}}H{\Lambda }_x\Vert \ge 2u\right]\le 9^k\mathbb{P}\left[|z^T{\Lambda }_x^{\mathsf{T}}H{\Lambda }_{x}z|\ge u\right].$$

Let ${\ell }_{x,z}:={\Lambda }_{x}z$ and $s_i:={\ell }_{x,z}^{\mathsf{T}}{y}_i$, so that we can write

$$\begin{array}{cc}\hfill z^T{\Lambda }_x^{\mathsf{T}}H{\Lambda }_{x}z& =\frac{1}{N}\sum _{i=1}^N\left({\left({\ell }_x^{\mathsf{T}}{y}_i\right)}^2-\mathbb{E}\left[{\left({\ell }_x^{\mathsf{T}}{y}_i\right)}^2\right]\right)=\frac{1}{N}\sum _{i=1}^N\left(s_i^2-\mathbb{E}\left[s_i^2\right]\right)\hfill \end{array}$$

and in particular

$$\mathbb{P}\left[\Vert {\Lambda }_x^{\mathsf{T}}H{\Lambda }_x\Vert \ge 2u\right]\le 9^k\mathbb{P}\left[|z^T{\Lambda }_x^{\mathsf{T}}H{\Lambda }_{x}z|\ge u\right]=9^k\mathbb{P}\left[\frac{1}{N}|\sum _{i=1}^N{s}_i^2-\mathbb{E}\left[s_i^2\right]|\ge u\right]$$

Observe then that $s_i\sim \mathcal{N}(0,{\gamma }^2)$ with ${\gamma }^2={\ell }_{x,z}^{\mathsf{T}}\Sigma {\ell }_{x,z}$. It follows for $u\in [0,{\gamma }^2]$ by the small deviation bound for ${\chi }^2$ random variables (e.g., [33] (Example 2.11))

$$\mathbb{P}\left[\Vert {\Lambda }_x^{\mathsf{T}}H{\Lambda }_x\Vert \ge 2u\right]\le 2exp\left(2k\log 3-\frac{N{u}^2}{8{\gamma }^4}\right)$$

Recall now that $|\left\{{\Lambda }_x\right|x\ne 0\}|\le {(n_1^d{n}_2^d\cdots n_d)}^{8k}$, then proceeding as for the Wigner case by a union bound over all possible ${\Lambda }_x$:

$$\begin{array}{cc}\hfill \mathbb{P}\left[\Vert {\Lambda }_x^{\mathsf{T}}H{\Lambda }_x\Vert \le 2u,\mathrm{for}\mathrm{all}x\right]& \ge 1-{(n_1^d{n}_2^{d-1}\cdots n_d)}^{8k}\mathbb{P}\left[\Vert {\Lambda }_x^{\mathsf{T}}H{\Lambda }_x\Vert \ge 2u\right]\hfill \\ & \ge 1-2exp(8k\log (2n_1^d{n}_2^{d-1}\cdots n_d)-\frac{N{u}^2}{8{\gamma }^4}).\hfill \end{array}$$

Substituting $u=\sqrt{8{\gamma }^4·9k\log (2n_1^d{n}_2^{d-1}\cdots n_d)/N}$ we find that:

$$\mathbb{P}[\Vert {\Lambda }_x^{\mathsf{T}}H{\Lambda }_x\Vert \le 2\sqrt{\frac{72k\log (2n_1^d{n}_2^{d-1}\cdots n_d)}{N}}{\gamma }^2,\mathrm{for}\mathrm{all}x]\ge 1-2e^{-k\log \left(n\right)}$$

(A42)

since $\log \left(n\right)\le \log (2n_1^d{n}_2^{d-1}\cdots n_d)$.

Similarly if $u>{\gamma }^2$ by large deviation bounds for sub-exponential variables

$$\mathbb{P}\left[\Vert {\Lambda }_x^{\mathsf{T}}H{\Lambda }_x\Vert \le 2u,\mathrm{for}\mathrm{all}x\right]\ge 1-2exp(8k\log (2n_1^d{n}_2^{d-1}\cdots n_d)-\frac{Nu}{8{\gamma }^2}).$$

Substituting $u=8{\gamma }^2·9k\log (2n_1^d{n}_2^{d-1}\cdots n_d)/N$ we find that:

$$\mathbb{P}[\Vert {\Lambda }_x^{\mathsf{T}}H{\Lambda }_x\Vert \le 2\frac{72k\log (3n_1^d{n}_2^{d-1}\cdots n_d)}{N}{\gamma }^2,\mathrm{for}\mathrm{all}x]\ge 1-2e^{-k\log \left(n\right)}$$

(A43)

Finally observe that using (A6) for bounding $\Vert {\Lambda }_x{\Vert }^2$ (by the WDC) and ${\sigma }^2+{\Vert y_{🟉}\Vert }^2$ for bounding $\Vert \Sigma \Vert$, we have

$${\gamma }^2\le \frac{13}{12}\frac{1}{2^d}(\Vert y_{🟉}{\Vert }^2+{\sigma }^2),$$

which combined with (A42) and (A43) implies the thesis.