Sharp Guarantees and Optimal Performance for Inference in Binary and Gaussian-Mixture Models ^†

Abstract

We study convex empirical risk minimization for high-dimensional inference in binary linear classification under both discriminative binary linear models, as well as generative Gaussian-mixture models. Our first result sharply predicts the statistical performance of such estimators in the proportional asymptotic regime under isotropic Gaussian features. Importantly, the predictions hold for a wide class of convex loss functions, which we exploit to prove bounds on the best achievable performance. Notably, we show that the proposed bounds are tight for popular binary models (such as signed and logistic) and for the Gaussian-mixture model by constructing appropriate loss functions that achieve it. Our numerical simulations suggest that the theory is accurate even for relatively small problem dimensions and that it enjoys a certain universality property.

1. Introduction

1.1. Motivation

Classical estimation theory studies problems in which the number of unknown parameters n is small compared to the number of observations m. In contrast, modern inference problems are typically high-dimensional, that is n can be of the same order as m. Examples are abundant in a wide range of signal processing and machine learning applications such as medical imaging, wireless communications, recommendation systems, etc. Classical tools and theories are not applicable in these modern inference problems [1]. As such, over the last two decades or so, the study of high-dimensional estimation problems has received significant attention.

Perhaps the most well-studied setting is that of noisy linear observations (namely, linear regression). The literature on the topic is vast with remarkable contributions from the statistics, signal processing and machine learning communities. Several recent works focus on the proportional/linear asymptotic regime and derive sharp results on the inference performance of appropriate convex optimization methods (e.g., [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]). These works show that, albeit challenging, sharp results are advantageous over loose order-wise bounds. Not only do they allow for accurate comparisons between different choices of the optimization parameters, but they also form the basis for establishing optimal such choices as well as fundamental performance limitations (e.g., [12,14,15,16,24,25,26]).

This paper takes this recent line of work a step further by demonstrating that results of this nature can be achieved in binary observation models. While we depart from the previously studied linear regression model, we remain faithful to the requirement and promise of sharp results. Binary models are popularly applicable in a wide range of signal-processing (e.g., highly quantized measurements) and machine learning (e.g., binary classification) problems. We derive sharp asymptotics for a rich class of convex optimization estimators, which include least-squares, logistic regression and hinge loss as special cases. Perhaps more interestingly, we use these results to derive fundamental performance limitations and design optimal loss functions that provably outperform existing choices. Our results hold both for discriminative and generative data models.

In Section 1.2, we formally introduce the problem setup. The paper’s main contributions and organization are presented in Section 1.4. A detailed discussion of prior art follows in Section 1.5.

Notation 1.

The symbols $\mathbb{P}(·)$, $\mathbb{E}\left[·\right]$ and $\mathit{Var}[·]$ denote probability, expectation and variance, respectively. We use boldface notation for vectors. ${\parallel \mathbf{v}\parallel }_2$ denotes the Euclidean norm of a vector $\mathbf{v}$. We write $i\in \left[m\right]$ for $i=1,2,\dots ,m$. When writing $x_{*}=arg{min}_{x}f\left(x\right),$ we let the operator $argmin$ return any one of the possible minimizers of f. For all $x\in \mathbb{R}$, $\Phi \left(x\right)$ is the cumulative distribution function of standard normal and Gaussian Q-function at x is defined as $Q\left(x\right)=1-\Phi \left(x\right).$

1.2. Data Models

Consider m data pairs ${(y_i,{\mathbf{a}}_i)}_{i=1}^m$ generated i.i.d from one of the following two models such that $y_i\in \{-1,+1\}$ and ${\mathbf{a}}_i\in {\mathbb{R}}^n$ for all $i\in \left[m\right]$.

Binary models with Gaussian features: Here, the feature/measurement vectors ${\mathbf{a}}_i,i\in \left[n\right]$ have i.i.d Gaussian entries, i.e., ${\mathbf{a}}_i\sim \mathcal{N}(0,{\mathbf{I}}_n).$ Given the feature vector ${\mathbf{a}}_i$, the corresponding label takes the form

$$\begin{array}{c}\hfill y_i=f\left({\mathbf{a}}_i^T{\mathbf{x}}_0\right),i\in \left[m\right],\end{array}$$

(1)

for some unknown true signal ${\mathbf{x}}_0\in {\mathbb{R}}^n$ and a label/link function $f:\mathbb{R}\to \{-1,+1\}$ a (possibly random) binary function. Some popular examples for the label function f include the following:

• (Noisy) Signed: $\left\{\begin{array}{cc}\mathrm{sign}\left({\mathbf{a}}_i^T{\mathbf{x}}_0\right)\hfill & ,\mathrm{w}.\mathrm{p}.1-\epsilon ,\hfill \\ -\mathrm{sign}\left({\mathbf{a}}_i^T{\mathbf{x}}_0\right)\hfill & ,\mathrm{w}.\mathrm{p}.\epsilon ,\hfill \end{array}\right.\mathrm{where}\epsilon \in [0,1/2].$
• Logistic: $y_i=\left\{\begin{array}{cc}+1\hfill & ,\mathrm{w}.\mathrm{p}.\frac{1}{1+exp\left(-{\mathbf{a}}_i^T{\mathbf{x}}_0\right)},\hfill \\ -1\hfill & ,\mathrm{w}.\mathrm{p}.1-\frac{1}{1+exp\left(-{\mathbf{a}}_i^T{\mathbf{x}}_0\right)}.\hfill \end{array}\right.$
• Probit: $y_i=\left\{\begin{array}{cc}+1\hfill & ,\mathrm{w}.\mathrm{p}.\Phi \left({\mathbf{a}}_i^T{\mathbf{x}}_0\right),\hfill \\ -1\hfill & ,\mathrm{w}.\mathrm{p}.1-\Phi \left({\mathbf{a}}_i^T{\mathbf{x}}_0\right).\hfill \end{array}\right.$

We remark that when the signal strength $\parallel {\mathbf{x}}_0{\parallel }_2\to +\infty$, logistic and Probit label functions approach the signed model (i.e., noisy-signed function with $\epsilon =0$).

Throughout, we assume that $\parallel {\mathbf{x}}_0{\parallel }_2=1$. This assumption is without loss of generality since the norm of ${\mathbf{x}}_0$ can always be absorbed in the link function. Indeed, letting $\parallel {\mathbf{x}}_0{\parallel }_2=r$, we can always write the measurements as $f\left({\mathbf{a}}^T{\mathbf{x}}_0\right)=\tilde{f}\left({\mathbf{a}}^T{\tilde{\mathbf{x}}}_0\right)$, where ${\tilde{\mathbf{x}}}_0={\mathbf{x}}_0/r$ (hence, $\parallel {\tilde{\mathbf{x}}}_0{\parallel }_2=1$) and $\tilde{f}\left(t\right)=f\left(rt\right)$. We make no further assumptions on the distribution of the true vector ${\mathbf{x}}_0$.

Gaussian-mixture model: In Section 5, we also study the following generative Gaussian-mixture model (GMM):

$$\begin{array}{c}\hfill y_i=\left\{\begin{array}{cc}+1\hfill & ,\mathrm{w}.\mathrm{p}.\pi ,\hfill \\ -1\hfill & ,\mathrm{w}.\mathrm{p}.1-\pi ,\hfill \end{array}\right.,{\mathbf{a}}_i|y_i\sim \mathcal{N}(y_i{\mathbf{x}}_0,{\mathbf{I}}_n),i\in \left[m\right].\end{array}$$

(2)

Above, $\pi \in [0,1]$ is the prior of class $+1$ and ${\mathbf{x}}_0\in {\mathbb{R}}^n$ is the true signal, which here represents the mean of the features.

1.3. Empirical Risk Minimization

We study the performance of empirical-risk minimization (ERM) estimators ${\widehat{\mathbf{x}}}_{\ell }$ of ${\mathbf{x}}_0$ that solve the following optimization problem for some convex loss function $\ell :\mathbb{R}\to \mathbb{R}$

$$\begin{array}{c}\hfill {\widehat{\mathbf{x}}}_{\ell }:=arg\underset{\mathbf{x}}{min}\frac{1}{m}\sum _{i=1}^m\ell \left(y_i{\mathbf{a}}_i^T\mathbf{x}\right).\end{array}$$

(3)

Loss function. Different choices for ℓ lead to popular specific estimators including the following:

• Least Squares (LS):$\ell \left(t\right)={(t-1)}^2$,
• Least-Absolute Deviations (LAD):$\ell \left(t\right)=|t-1|$,
• Logistic Loss:$\ell \left(t\right)=log(1+exp(-t\left)\right)$,
• Exponential Loss:$\ell \left(t\right)=exp\left(-t\right)$,
• Hinge Loss:$\ell \left(t\right)=max\{1-t,0\}$.

Performance Measure. We measure performance of the estimator ${\widehat{\mathbf{x}}}_{\ell }$ by the value of its correlation to ${\mathbf{x}}_0$, i.e.,

$$\begin{array}{c}\hfill \mathrm{corr}\left({\widehat{\mathbf{x}}}_{\ell };{\mathbf{x}}_0\right):=\frac{⟨{\widehat{\mathbf{x}}}_{\ell },{\mathbf{x}}_0⟩}{\parallel {\widehat{\mathbf{x}}}_{\ell }{\parallel }_2{\parallel {\mathbf{x}}_0\parallel }_2}\in [-1,1].\end{array}$$

(4)

Obviously, we seek estimates that maximize correlation. While correlation is the measure of primal interest, our results extend rather naturally to other prediction metrics, such as classification error given by (e.g., see [27] (Section D.2.)),

$$\begin{array}{c}\hfill {\mathcal{E}}_{\ell }:={\mathbb{E}}_{\mathbf{a},y}\left[{\mathbb{1}}_{\left\{y\ne \mathrm{sign}\left(⟨{\widehat{\mathbf{x}}}_{\ell },\mathbf{a}⟩\right)\right\}}\right].\end{array}$$

(5)

Expectation in (5) is derived based on a test sample $(\mathbf{a},y)$ from the same distribution of the training set.

1.4. Contributions and Organization

As mentioned, our techniques naturally apply to both binary Gaussian and Gaussian-mixture models. For concreteness, we focus our presentation on the former models (see Section 2, Section 3 and Section 4.1). Then, we extend our results to Gaussian mixtures in Section 5. Numerical simulations corroborating our theoretical findings for both models are presented in Section 6.

Now, we state the paper’s main contributions:

• Precise Asymptotics: We show that the absolute value of correlation of ${\widehat{\mathbf{x}}}_{\ell }$ to the true vector ${\mathbf{x}}_0$ is sharply predicted by $\sqrt{1/(1+{\sigma }_{\ell }^2)}$ where the “effective noise” parameter ${\sigma }_{\ell }$ can be explicitly computed by solving a system of three non-linear equations in three unknowns. We find that the system of equations (and, thus, the value of ${\sigma }_{\ell }$) depends on the loss function ℓ through its Moreau envelope function. Our prediction holds in the linear asymptotic regime in which $m,n\to \infty$ and $m/n\to \delta >1$ (see Section 2).
• Fundamental Limits: We establish fundamental limits on the performance of convex optimization-based estimators by computing an upper bound on the best possible correlation performance among all convex loss functions. We compute the upper bound by solving a certain nonlinear equation and we show that such a solution exists for all $\delta >1$ (see Section 3.1).
• Optimal Performance and (sub)-optimality of LS for binary models: For certain binary models including signed and logistic, we find the loss functions that achieve the optimal performance, i.e., they attain the previously derived upper bound (see Section 3.2). Interestingly, for logistic and Probit models with $\parallel {\mathbf{x}}_0{\parallel }_2=1$, we prove that the correlation performance of least-squares (LS) is at least as good 0.9972 and 0.9804 times the optimal performance. However, as $\parallel {\mathbf{x}}_0{\parallel }_2$ grows large, logistic and Probit models approach the signed model, in which case LS becomes sub-optimal (see Section 4.1).
• Extension to the Gaussian-Mixture Model: In Section 5, we extend the fundamental limits and the system of equations to the Gaussian-mixture model. Interestingly, our results indicate that, for this model, LS is optimal among all convex loss functions for all $\delta >1$.
• Numerical Simulations: We do numerous experiments to specialize our results to popular models and loss functions, for which we provide simulation results that demonstrate the accuracy of the theoretical predictions (see Section 6 and Appendix E).

Figure 1 contains a pictorial preview of our results described above for the special case of signed measurements. First, Figure 1a depicts the correlation performance of LS and LAD estimators as a function of the aspect ratio $\delta$. Both theoretical predictions and numerical results are shown; note the close match between theory and empirical results for both i.i.d. Gaussian (shown by circles) and i.i.d. Rademacher (shown by squares) distributions of the feature vectors for even small dimensions. Second, the red line on the same figure shows the upper bound derived in this paper—there is no convex loss function that results in correlation exceeding this line. Third, we show that the upper bound can be achieved by the loss functions depicted in Figure 1b for several values of $\delta$. We solve (3) for this choice of loss functions using gradient descent and numerically evaluate the achieved correlation performance. The recorded values are compared in Table 1 to the corresponding values of the upper bound; again, note the close agreement between the values as predicted by the findings of this paper, which suggests that the fundamental limits derived in this paper hold for sub-Gaussian features. We present corresponding results for the logistic and Probit models in Section 6 and for the noisy-signed model in Appendix E.

Figure 1. (a) Comparison between theoretical (solid lines) and empirical (markers) performance for least-squares (LS) and least-absolute deviations (LAD), as predicted by Theorem 1, and the optimal performance, as predicted by the upper bound of Theorem 2, for the signed model. The squares and circles denote the empirical performance for Gaussian and Rademacher features, respectively. (b) Illustrations of optimal loss functions for the signed model for different values of $\delta$ according to Theorem 3.

Table 1. Theoretical predictions and empirical performance of the optimal loss function for signed model. Empirical results are averaged over 20 experiments for $n=128$.

$\mathit{\delta }$	2	3	4	5	6	7	8	9
Predicted Performance	$0.8168$	$0.9101$	$0.9457$	$0.9645$	$0.9748$	$0.9813$	$0.9855$	$0.9885$
Empirical (Gaussian)	$0.8213$	$0.9045$	$0.9504$	$0.9669$	$0.9734$	$0.9801$	$0.9834$	$0.9873$
Empirical (Rademacher)	$0.8096$	$0.9158$	$0.9490$	$0.9633$	$0.9644$	$0.9768$	$0.9808$	$0.9829$

A remark on the Gaussianity assumption. Our results on precise asymptotics (to which our study of fundamental limits rely upon) hold rigorously for the two data models in Section 1.2, in which the feature vectors have entries i.i.d. standard Gaussian. However, we conjecture that the Gaussianity assumption can be relaxed. As partial numerical evidence, note in Figure 1a the perfect match of our theory with the empirical performance over data in which the feature vectors ${\mathbf{a}}_i,i\in \left[m\right]$ have entries i.i.d. Rademacher (i.e., centered Bernoulli with probability 1/2). Figure 2 shows corresponding results for the Gaussian-mixture model. Our conjecture that the so-called universality property holds in our setting is also in line with similar numerical observations and partial theoretical evidence previously made for linear regression settings [7,28,29,30,31]. A formal proof of universality of our results is beyond the scope of this paper. However, we remark that, as long as the asymptotic predictions of Section 2 enjoy this property, then all our results on fundamental performance limits and optimal functions automatically hold under the same relaxed assumptions.

Figure 2. Theoretical (solid lines) and empirical (markers) results of classification risk in GMM as in Theorem 4 and (39) for LS, LAD and logistic loss functions as a function of $\delta$ for $r=1$. The vertical line represents the threshold ${\delta }^{☆}\approx 3.7$ as evaluated by (36). Logistic loss gives unbounded solution if and only if $\delta <{\delta }^{☆}$.

1.5. Related Works

Over the past two decades, there has been a long list of works that derive statistical guarantees for high-dimensional estimation problems. Many of these are concerned with convex optimization-based inference methods. Our work is most closely related to the following three lines of research.

(a)

Sharp asymptotics for linear measurements.

Most of the results in the literature of high-dimensional statistics are order-wise in nature. Sharp asymptotic predictions have only more recently appeared in the literature for the case of noisy linear measurements with Gaussian measurement vectors. There are by now three different approaches that have been used towards asymptotic analysis of convex regularized estimators: (i) the one that is based on the approximate message passing (AMP) algorithm and its state-evolution analysis (e.g., [5,8,14,20,32,33,34]); (ii) the one that is based on Gaussian process (GP) inequalities, specifically on the convex Gaussian min-max Theorem (CGMT) (e.g., [9,10,13,15,18,19]); and (iii) the “leave-one-out” approach [11,35]. The three approaches are quite different to each other and each comes with its unique distinguishing features and disadvantages. A detailed comparison is beyond our scope.

Our results in Theorems 2 and 3 for achieving the best performance across all loss functions is complementary to [12] (Theorem 1) and the work of Advani and Ganguli [16], who proposed a method for deriving optimal loss function and measuring its performance, albeit for linear models. Instead, we study binary models. The optimality of regularization for linear measurements is recently studied in [22].

In terms of analysis, we follow the GP approach and build upon the CGMT. Since the previous works are concerned with linear measurements, they consider estimators that solve minimization problems of the form

$$\begin{array}{c}\hfill \widehat{\mathbf{x}}:=arg\underset{\mathbf{x}}{min}\sum _{i=1}^m\tilde{\ell }(y_i-{\mathbf{a}}_i^T\mathbf{x})+rR\left(\mathbf{x}\right)\end{array}$$

(6)

Specifically, the loss function $\tilde{\ell }$ penalizes the residual. In this paper, we show that the CGMT is applicable to optimization problems in the form of (3). For our case of binary observations, (3) is more general than (6). To see this, note that, for $y_i\in \pm 1$ and popular symmetric loss functions $\tilde{\ell }\left(t\right)=\tilde{\ell }(-t)$, e.g., least-squares (LS), (3) results in (6) by choosing $\ell \left(t\right)=\tilde{\ell }(t-1)$ in the former. Moreover, (3) includes several other popular loss functions such as the logistic loss and the hinge loss which cannot be expressed by (6).

(b)

One-bit compressed sensing.

Our work naturally relates to the literature on one-bit compressed sensing (CS) [36]. The vast majority of performance guarantees for one-bit CS are order-wise in nature (e.g., [37,38,39,40,41,42]). To the best of our knowledge, the only existing sharp results are presented in [43] for Gaussian measurement vectors, which studies the asymptotic performance of regularized LS. Our work can be seen as a direct extension of the work in [43] to loss functions beyond least-squares (see Section 4.1 for details).

Similar to the generality of our paper, Genzel [41] also studied the high-dimensional performance of general loss functions. However, in contrast to our results, their performance bounds are loose (order-wise); as such, they are not informative about the question of optimal performance which we also address here.

(c)

Classification in high-dimensions.

In [44,45], the authors studied the high-dimensional performance of maximum-likelihood (ML) estimation for the logistic model. The ML estimator is a special case of (3) and we consider general binary models. In addition, their analysis is based on the AMP framework. The asymptotics of logistic loss under different classification models is also recently studied in [46]. In yet another closely related recent work [47], the authors extended the results of Sur and Candes [45] to regularized ML by using the CGMT. Instead, we present results for general convex loss functions and for binary linear models. Importantly, we also study performance bounds and optimal loss functions.

We also remark on the following closely related parallel works. While the conference version of this paper was being reviewed, the CGMT was applied by Montanari et al. [48] and Deng et al. [49] to determine the generalization performance of max-margin linear classifiers in a binary classification setting. In essence, these results are complementary to the results of our paper in the following sense. Consider a binary classification setting under the logistic model and Gaussian regressors. As discussed in Section 4.2, the optimal set of (3) is bounded with probability approaching one if and only if $\delta >{\delta }_f^{☆}$, for appropriate threshold ${\delta }_f^{☆}$ determined for first time in [44] (see also Figure 3a). Our results hold in this regime. In contrast, the papers by Montanari et al. [48] and Deng et al. [49] study the regime $\delta <{\delta }_f^{☆}$.

Figure 3. (a) Comparison between analytical and empirical results for the performance of LS, logistic loss, hinge loss and optimal loss function for logistic model. The vertical dashed line represents ${\delta }_f^{☆}\approx 2.275$, as evaluated by (35). (b) Illustrations of optimal loss functions for different values of $\delta$, derived according to Theorem 3 for logistic model. To signify the similarity of optimal loss function to the LS loss, the optimal loss functions (hardly visible) are scaled such that $\ell \left(1\right)=0$ and $\ell \left(2\right)=1$.

We close this section by mentioning works that build on our results and appeared after the initial submission of this paper. The paper by Mignacco et al. [50] studies sharp asymptotics of ridge-regularized ERM with an intercept for Gaussian-mixture models. In [27], we extend the results of this paper on fundamental limits and optimality to the case of ridge-regularized ERM (see also the concurrent work by Aubin et al. [51]).

2. Sharp Performance Guarantees

2.1. Definitions

Moreau Envelopes. Before stating the first result, we need a definition. We write

$${\mathcal{M}}_{\ell }\left(x;\lambda \right):=\underset{v}{min}\frac{1}{2\lambda }{(x-v)}^2+\ell \left(v\right),$$

for the Moreau envelope function of the loss $\ell :\mathbb{R}\to \mathbb{R}$ at x with parameter $\lambda >0$. The minimizer (which is unique by strong convexity) is known as the proximal operator of ℓ at x with parameter $\lambda$ and we denote it as ${\mathrm{prox}}_{\ell }\left(x;\lambda \right)$. A useful property of the Moreau envelope function is that it is continuously differentiable with respect to both x and $\lambda$ [52]. We denote these derivatives as follows

$$\begin{array}{cc}\hfill {\mathcal{M}}_{\ell ,1}^{\prime }\left(x;\lambda \right)& :=\frac{\partial {\mathcal{M}}_{\ell }\left(x;\lambda \right)}{\partial x},\hfill \\ \hfill {\mathcal{M}}_{\ell ,2}^{\prime }\left(x;\lambda \right)& :=\frac{\partial {\mathcal{M}}_{\ell }\left(x;\lambda \right)}{\partial \lambda }.\hfill \end{array}$$

2.2. A System of Equations

As we show shortly the asymptotic performance of the optimization in (3) is tightly connected to the solution of a certain system of nonlinear equations, which we introduce here. Specifically, define random variables $G,S$ and Y as follows:

$$\begin{array}{c}\hfill G,S\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\mathcal{N}(0,1)\mathrm{and}Y=f\left(S\right),\end{array}$$

(7)

and consider the following system of non-linear equations in three unknowns $(\mu ,\alpha \ge 0,\lambda \ge 0)$:

$$\begin{array}{cc}\hfill \mathbb{E}\left[YS·{\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha G+\mu SY;\lambda \right)\right]& =0,\hfill \end{array}$$

(8a)

$$\begin{array}{cc}\hfill {\lambda }^2\delta \mathbb{E}\left[{\left({\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha G+\mu SY;\lambda \right)\right)}^2\right]& ={\alpha }^2,\hfill \end{array}$$

(8b)

$$\begin{array}{cc}\hfill \lambda \delta \mathbb{E}\left[G·{\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha G+\mu SY;\lambda \right)\right]& =\alpha .\hfill \end{array}$$

(8c)

The expectations are with respect to the randomness of the random variables G, S and Y. We remark that the equations are well defined even if the loss function ℓ is not differentiable. In Appendix A, we summarize some well-known properties of the Moreau envelope function and use them to simplify (8) for differentiable loss functions.

2.3. Asymptotic Prediction

We are now ready to state our first main result.

Theorem 1.

(Sharp Asymptotics). Assume data generated from the binary model with Gaussian features and assume $\delta >1$ such that the set of minimizers in (3) is bounded and the system of Equation (8) has a unique solution $(\mu ,\alpha \ge 0,\lambda \ge 0)$, such that $\mu \ne 0$. Let ${\widehat{\mathbf{x}}}_{\ell }$ be as in (3). Then, in the limit of $m,n\to +\infty$, $m/n\to \delta$, it holds with probability one that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\mathrm{corr}\left({\widehat{\mathbf{x}}}_{\ell };{\mathbf{x}}_0\right)=\frac{\mu }{\sqrt{{\mu }^2+{\alpha }^2}}.\end{array}$$

(9)

Moreover,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{∥{\widehat{\mathbf{x}}}_{\ell }-\mu ·\frac{{\mathbf{x}}_0}{\parallel {\mathbf{x}}_0{\parallel }_2}∥}_2^2={\alpha }^2.\end{array}$$

(10)

Theorem 1 holds for any convex loss function. In Section 4, we specialize the result to specific popular choices and also present numerical simulations that confirm the validity of the predictions (see Figure 1a, Figure 3a, Figure 4a and Figure A4a,b). Before that, we include a few remarks on the conditions, interpretation and implications of the theorem. The proof is deferred to Appendix B and uses the convex Gaussian min-max theorem (CGMT) [13,15].

Figure 4. (a) Comparison between analytical and empirical results for the performance of LS, hinge loss and optimal loss function for Probit model. The vertical dashed line represents ${\delta }_f^{☆}\approx 2.699$, as evaluated by (35). (b) Illustrations of optimal loss functions for different values of $\delta$ derived according to Theorem 3 for Probit model. To signify the similarity of optimal loss function to the LS loss, the optimal loss functions (hardly visible) are scaled such that $\ell \left(1\right)=0$ and $\ell \left(2\right)=1$.

Remark 1.

(The Role of $\mu$ and $\alpha$). According to (9), the prediction for the limiting behavior of the correlation value is given in terms of an effective noise parameter ${\sigma }_{\ell }:=\alpha /\mu$, where μ and α are unique solutions of (8). The smaller is the value of ${\sigma }_{\ell }$ is, the larger the correlation value becomes. While the correlation value is fully determined by the ratio of α and μ, their individual role is clarified in (10). Specifically, according to (10), ${\widehat{\mathbf{x}}}_{\ell }$ is a biased estimate of the true ${\mathbf{x}}_0$ and μ represents exactly the correlation bias term. In other words, solving (3) returns an estimator that is close to a μ-scaled version of ${\mathbf{x}}_0$. When ${\mathbf{x}}_0$ and ${\widehat{\mathbf{x}}}_{\ell }$ are scaled appropriately, the ${\ell }_2$-norm of their difference converges to α.

Remark 2.

(Why $\delta >1$). The theorem requires that $\delta >1$ (equivalently, $m>n$ asymptotically). Here, we show that this condition is necessary for Equations (8) to have a bounded solution. To see this, take squares in both sides of (8c) and divide by (8b) to find that

$$\delta =\frac{\mathbb{E}\left[{\left({\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha G+\mu SY;\lambda \right)\right)}^2\right]}{{\left(\mathbb{E}\left[G·{\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha G+\mu SY;\lambda \right)\right]\right)}^2}\ge 1.$$

The inequality follows by applying Cauchy–Schwarz and using the fact that $\mathbb{E}\left[G^2\right]=1$.

Remark 3.

(On the Existence of a Solution to (8)). While $\delta >1$ is a necessary condition for the equations in (8) to have a solution, it is not sufficient in general. This depends on the specific choice of the loss function. For example, in Section 4.1, we show that, for the squared loss $\ell \left(t\right)={(t-1)}^2$, the equations have a unique solution iff $\delta >1$. On the other hand, for logistic loss and hinge loss, it is argued in Section 4.2 that there exists a threshold value ${\delta }_f^{☆}>2$ such that the set of minimizers in (3) is unbounded if $\delta <{\delta }_f^{☆}$. In this case, the assumptions of Theorem 1 do not hold. We conjecture that, for these choices of loss, Equations (8) are solvable iff $\delta >{\delta }_f^{☆}$. Justifying this conjecture and further studying more general sufficient and necessary conditions under which the Equation (8) admit a solution is left to future work. However, in what follows, given such a solution, we prove that it is unique for a wide class of convex loss functions of interest.

Remark 4.

(On the Uniqueness of Solutions to (8)). We show that, if the system of equations in (8) has a solution, then it is unique provided that ℓ is strictly convex, continuously differentiable and its derivative satisfies ${\ell }^{\prime }\left(0\right)\ne 0$. For instance, this class includes the square, the logistic and the exponential losses. However, it excludes non-differentiable functions such as the LAD and hinge loss. We believe that the differentiability assumption can be relaxed without major modification in our proof, but we leave this for future work. Our result is summarized in Proposition 1 below.

Proposition 1.

(Uniqueness). Assume that the loss function $\ell :\mathbb{R}\to \mathbb{R}$ has the following properties: (i) it is proper strictly convex; and (ii) it is continuously differentiable and its derivative ${\ell }^{\prime }$ is such that ${\ell }^{\prime }\left(0\right)\ne 0$. Further, assume that the (possibly random) link function f is such that $SY=Sf\left(S\right),S\sim \mathcal{N}(0,1)$ has strictly positive density on the real line. The following statement is true. For any $\delta >1$, if the system of equations in (8) has a bounded solution, then it is unique.

The detailed proof of Proposition 1 is deferred to Appendix B.5. Here, we highlight some key ideas. The CGMT relates—in a rather natural way—the original ERM optimization (3) to the following deterministic min-max optimization on four variables

$$\begin{array}{c}\hfill \underset{\alpha >0,\mu ,\tau >0}{min}\underset{\gamma >0}{max}F(\alpha ,\mu ,\tau ,\gamma ):=\frac{\gamma \tau }{2}-\frac{\alpha \gamma }{\sqrt{\delta }}+\mathbb{E}\left[{\mathcal{M}}_{\ell }\left(\alpha G+\mu YS;\frac{\tau }{\gamma }\right)\right].\end{array}$$

(11)

In Appendix B.4, we show that the optimization above is convex-concave for any lower semi-continuous, proper and convex function $\ell :\mathbb{R}\to \mathbb{R}$. Moreover, it is shown that one arrives at the system of equations in (8) by simplifying the first-order optimality conditions of the min-max optimization in (11). This connection is key to the proof of Proposition 1. Indeed, we prove uniqueness of the solution (if such a solution exists) to (8), by proving instead that the function $F(\alpha ,\mu ,\tau ,\gamma )$ above is (jointly) strictly convex in $(\alpha ,\mu ,\tau )$ and strictly concave in γ, provided that ℓ satisfies the conditions of the proposition. Next, let us briefly discuss how strict convex-concavity of (11) can be shown. For concreteness, we only discuss strict convexity here; the ideas are similar for strict concavity. At the heart of the proof of strict convexity of F is understanding the properties of the expected Moreau envelope function $\Omega :{\mathbb{R}}_{+}\times \mathbb{R}\times {\mathbb{R}}_{+}\times {\mathbb{R}}_{+}\to \mathbb{R}$ defined as follows:

$$\Omega (\alpha ,\mu ,\tau ,\gamma ):=\mathbb{E}\left[{\mathcal{M}}_{\ell }\left(\alpha G+\mu YS;\frac{\tau }{\gamma }\right)\right].$$

Specifically, we prove in Proposition A7 in Appendix A.6 that if ℓ is strictly convex, differentiable and does not attain its minimum at 0, then Ω is strictly convex in $(\alpha ,\mu ,\tau )$ and strictly concave in γ. It is worth noting that the Moreau envelope function ${\mathcal{M}}_{\ell }\left(\alpha g+\mu ys;\tau \right)$ for fixed $g,s$ and $y=f\left(s\right)$ is not necessarily strictly convex. Interestingly, we show that the expected Moreau envelope has this desired feature. We refer the reader to Appendix A.6 and Appendix B.5 for more details.

3. On Optimal Performance

3.1. Fundamental Limits

In this section, we establish fundamental limits on the performance of (3) by deriving an upper bound on the absolute value of correlation $\mathrm{corr}\left({\widehat{\mathbf{x}}}_{\ell };{\mathbf{x}}_0\right)$ that holds for all choices of loss functions satisfying Theorem 1. The result builds on the prediction of Theorem 1. In view of (9), upper bounding correlation is equivalent to lower bounding the effective noise parameter ${\sigma }_{\ell }=\alpha /\mu$. Theorem 2 derives such a lower bound.

Before stating the theorem, we need a definition. For a random variable H with density $p_H\left(h\right)$ that has a derivative $p_H^{\prime }\left(h\right),\forall h\in \mathbb{R}$, we denote its score function ${\xi }_H\left(h\right):=\frac{\partial }{\partial h}log{p}_H\left(h\right)=\frac{p_H^{\prime }\left(h\right)}{p_H\left(h\right)}$. Then, the Fisher information of H, denoted by $\mathcal{I}\left(H\right)\in {\mathbb{R}}_{+}$, is defined as follows (e.g., [53] (Sec. 2)):

$$\mathcal{I}\left(H\right):=\mathbb{E}\left[{\left({\xi }_H\left(H\right)\right)}^2\right].$$

Theorem 2.

(Best Achievable Performance). Let the assumptions and notation of Theorem 1 hold and recall the definition of random variables $G,S$ and Y in (7). For $\sigma >0$, define a new random variable $W_{\sigma }:=\sigma G+SY,$ and the function $\kappa :(0,\infty ]\to [0,1]$ as follows,

$$\begin{array}{c}\hfill \kappa \left(\sigma \right):=\frac{{\sigma }^2\left({\sigma }^2\mathcal{I}\left(W_{\sigma }\right)+\mathcal{I}\left(W_{\sigma }\right)-1\right)}{1+{\sigma }^2\left({\sigma }^2\mathcal{I}\left(W_{\sigma }\right)-1\right)}.\end{array}$$

Further, define ${\sigma }_{\mathrm{opt}}$ as follows,

$$\begin{array}{c}\hfill {\sigma }_{\mathrm{opt}}:=min\left\{\sigma \ge 0:\kappa \left(\sigma \right)=\frac{1}{\delta }\right\}.\end{array}$$

(12)

Then, for ${\sigma }_{\ell }:=\frac{\alpha }{\mu }$, it holds that ${\sigma }_{\ell }\ge {\sigma }_{\mathrm{opt}}$.

The theorem above establishes an upper bound on the best possible correlation performance among all convex loss functions. In Section 3.2, we show that this bound is often tight, i.e., there exists a loss function that achieves the specified best possible performance.

Remark 5.

Theorem 2 complements the results in [12,14] (Lem. 3.4) and [15] (Rem. 5.3.3), in which the authors considered only linear regression. In particular, Theorem 2 shows that it is possible to achieve results of this nature for the more challenging setting of binary classification considered here.

Proof of Theorem 2.

Fix a loss function ℓ and let $(\mu \ne 0,\alpha >0,\lambda \ge 0)$ be a solution to (8), which by assumptions of Theorem 1 is unique. The first important observation is that the error of a loss function is unique up to a multiplicative constant. To see this, consider an arbitrary loss function $\ell \left(t\right)$ and let ${\widehat{\mathbf{x}}}_{\ell }$ be a minimizer in (3). Now, consider (3) with the following loss function instead, for some arbitrary constants $C_1>0,C_2\ne 0$:

$$\begin{array}{c}\hfill \widehat{\ell }\left(t\right):=\frac{1}{C_1}\ell \left(C_{2}t\right).\end{array}$$

(13)

It is not hard to see that $\frac{1}{C_2}{\widehat{\mathbf{x}}}_{\ell }$ is the minimizer for $\widehat{\ell }$. Clearly, $\frac{1}{C_2}{\widehat{\mathbf{x}}}_{\ell }$ has the same correlation value with ${\mathbf{x}}_0$ as ${\widehat{\mathbf{x}}}_{\ell }$, showing that the two loss functions ℓ and $\widehat{\ell }$ perform the same. With this observation in mind, consider the function $\widehat{\ell }:\mathbb{R}\to \mathbb{R}$ such that $\widehat{\ell }\left(t\right)=\frac{\lambda }{{\mu }^2}\ell \left(\mu t\right)$. Then, notice that

$${\mathcal{M}}_{\ell ,1}^{\prime }\left(x;\lambda \right)=\frac{1}{\lambda }{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(x/\mu ;1\right).$$

Using this relation in (8) and setting $\sigma :={\sigma }_{\ell }=\alpha /\mu$, the system of equations in (8) can be equivalently rewritten in the following convenient form,

$$\begin{array}{cc}& \mathbb{E}\left[YS·{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right)\right]=0,\hfill \end{array}$$

(14a)

$$\begin{array}{cc}& \mathbb{E}\left[{\left({\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right)\right)}^2\right]={\sigma }^2/\delta ,\hfill \end{array}$$

(14b)

$$\begin{array}{cc}& \mathbb{E}\left[G·{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right)\right]=\sigma /\delta .\hfill \end{array}$$

(14c)

Next, we show how to use (14) to derive an equivalent system of equations based on $W_{\sigma }$. Starting with (14c), we have

$$\begin{array}{c}\hfill \mathbb{E}\left[G·{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right)\right]=\frac{1}{\sigma }\int \int u{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(u+z;1\right){\varphi }_{\sigma }\left(u\right)p_{SY}\left(z\right)\mathrm{d}u\mathrm{d}z,\end{array}$$

(15)

where ${\varphi }_{\sigma }\left(u\right):=p_{\sigma G}\left(u\right)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{u^2}{2{\sigma }^2}}$. Since it holds that ${\varphi }_{\sigma }\left(u\right)=\frac{-{\sigma }^2}{u}{\varphi }_{\sigma }^{\prime }\left(u\right)$, using (A74), it follows that

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathbb{E}\left[G·{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right)\right]& =-\sigma \int \int {\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(u+z;1\right){\varphi }_{\sigma }^{\prime }\left(u\right)p_{SY}\left(z\right)\mathrm{d}u\mathrm{d}z\hfill \\ \hfill & =-\sigma \int \int {\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(w;1\right){\varphi }_{\sigma }^{\prime }\left(u\right)p_{SY}(w-u)\mathrm{d}u\mathrm{d}w\hfill \\ & =-\sigma \int {\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(w;1\right)p_{W_{\sigma }}^{\prime }\left(w\right)\mathrm{d}w,\hfill \end{array}\end{array}$$

(16)

where in the last step we use

$$p_{W_{\sigma }}^{\prime }\left(w\right)=\int {\varphi }_{\sigma }^{\prime }\left(u\right)p_{SY}(w-u)\mathrm{d}u.$$

Therefore, we have by (16) that

$$\begin{array}{c}\hfill \mathbb{E}\left[G·{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right)\right]=-\sigma \mathbb{E}\left[{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right){\xi }_{W_{\sigma }}\left(W_{\sigma }\right)\right].\end{array}$$

(17)

This combined with (14c) gives $\mathbb{E}\left[{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right){\xi }_{W_{\sigma }}\left(W_{\sigma }\right)\right]=-1/\delta .$ Second, multiplying (14c) with ${\sigma }^2$ and adding it to (14a) yields that,

$$\begin{array}{cc}\hfill \mathbb{E}\left[W_{\sigma }·{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right)\right]& ={\sigma }^2/\delta ,\hfill \end{array}$$

(18)

Putting these together, we conclude with the following system of equations which is equivalent to (14),

$$\begin{array}{cc}& \mathbb{E}\left[W_{\sigma }·{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right)\right]={\sigma }^2/\delta ,\hfill \end{array}$$

(19a)

$$\begin{array}{cc}& \mathbb{E}\left[{\left({\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right)\right)}^2\right]={\sigma }^2/\delta ,\hfill \end{array}$$

(19b)

$$\begin{array}{cc}& \mathbb{E}\left[{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right){\xi }_{W_{\sigma }}\left(W_{\sigma }\right)\right]=-1/\delta .\hfill \end{array}$$

(19c)

Note that, for $\sigma >0$, ${\xi }_{W_{\sigma }}=p_{W_{\sigma }}^{\prime }/p_{W_{\sigma }}$ exists everywhere. This is because for all $w\in \mathbb{R}$: $p_{W_{\sigma }}\left(w\right)>0$ and $p_{W_{\sigma }}(·)$ is continuously differentiable. Combining (19a) and (19c), we derive the following equation which holds for ${\alpha }_1,{\alpha }_2\in \mathbb{R}$,

$$\begin{array}{cc}\hfill \mathbb{E}\left[({\alpha }_1{W}_{\sigma }+{\alpha }_2{\xi }_{W_{\sigma }}\left(W_{\sigma }\right))·{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right)\right]& ={\alpha }_1{\sigma }^2/\delta -{\alpha }_2/\delta .\hfill \end{array}$$

By Cauchy–Schwarz inequality, we have that

$$\begin{array}{cc}\hfill & {\left(\mathbb{E}\left[({\alpha }_1{W}_{\sigma }+{\alpha }_2{\xi }_{W_{\sigma }}\left(W_{\sigma }\right))·{\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right)\right]\right)}^2\le \hfill \\ & \mathbb{E}\left[{({\alpha }_1{W}_{\sigma }+{\alpha }_2{\xi }_{W_{\sigma }}\left(W_{\sigma }\right))}^2\right]\mathbb{E}\left[{\left({\mathcal{M}}_{\widehat{\ell },1}^{\prime }\left(W_{\sigma };1\right)\right)}^2\right].\hfill \end{array}$$

(20)

Using the fact that $\mathbb{E}\left[W_{\sigma }{\xi }_{W_{\sigma }}\left(W_{\sigma }\right)\right]=-1$ (by integration by parts), $\mathbb{E}\left[{\left({\xi }_{W_{\sigma }}\left(W_{\sigma }\right)\right)}^2\right]=\mathcal{I}\left(W_{\sigma }\right)$, $\mathbb{E}\left[W_{\sigma }^2\right]={\sigma }^2+1$ and (19b), the right hand side of (20) is equal to

$$\begin{array}{c}\hfill \left({\alpha }_1^2({\sigma }^2+1)+{\alpha }_2^2\mathcal{I}\left(W_{\sigma }\right)-2{\alpha }_1{\alpha }_2\right){\sigma }^2/\delta .\end{array}$$

Therefore, we conclude with the following inequality for $\sigma$,

$$\begin{array}{c}\hfill \delta {\sigma }^2\left({\alpha }_1^2({\sigma }^2+1)+{\alpha }_2^2\mathcal{I}\left(W_{\sigma }\right)-2{\alpha }_1{\alpha }_2\right)\ge {({\alpha }_1{\sigma }^2-{\alpha }_2)}^2,\end{array}$$

(21)

which holds for all ${\alpha }_1,{\alpha }_2\in \mathbb{R}$. In particular, (21) holds for the following choice of values for ${\alpha }_1$ and ${\alpha }_2$:

$$\begin{array}{c}\hfill {\alpha }_1=\frac{1-{\sigma }^2\mathcal{I}\left(W_{\sigma }\right)}{\delta ({\sigma }^2\mathcal{I}\left(W_{\sigma }\right)+\mathcal{I}\left(W_{\sigma }\right)-1)},{\alpha }_2=\frac{1}{\delta ({\sigma }^2\mathcal{I}\left(W_{\sigma }\right)+\mathcal{I}\left(W_{\sigma }\right)-1)}.\end{array}$$

(The choice above is motivated by the result of Section 3.2; see Theorem 3). Rewriting (21) with the chosen values of ${\alpha }_1$ and ${\alpha }_2$ yields the following inequality,

$$\begin{array}{c}\hfill \frac{1}{\delta }\le \frac{{\sigma }^2({\sigma }^2\mathcal{I}\left(W_{\sigma }\right)+\mathcal{I}\left(W_{\sigma }\right)-1)}{1+{\sigma }^2({\sigma }^2\mathcal{I}\left(W_{\sigma }\right)-1)}=\kappa \left(\sigma \right),\end{array}$$

(22)

where on the right-hand side above, we recognize the function $\kappa$ defined in the theorem.

Next, we use (22) to show that ${\sigma }_{\mathrm{opt}}$ defined in (12) yields a lower bound on the achievable value of $\sigma$. For the sake of contradiction, assume that $\sigma <{\sigma }_{\mathrm{opt}}$. By the above, $1/\delta \le \kappa \left(\sigma \right)$. Moreover, by the definition of ${\sigma }_{\mathrm{opt}}$, we must have that $1/\delta <\kappa \left(\sigma \right).$ Since $\kappa \left(0\right)=0$ and $\kappa (·)$ is a continuous function we conclude that for some ${\sigma }_1\in (0,\sigma )$, it holds that $\kappa \left({\sigma }_1\right)=1/\delta$. Therefore, for ${\sigma }_1<{\sigma }_{\mathrm{opt}}$, we have $\kappa \left({\sigma }_1\right)=1/\delta$, which contradicts the definition of ${\sigma }_{\mathrm{opt}}$. This proves that $\sigma \ge {\sigma }_{\mathrm{opt}}$, as desired.

To complete the proof, it remains to show that the equation $\kappa \left(\sigma \right)=1/\delta$ admits a solution for all $\delta >1$. For this purpose, we use the continuous mapping theorem and the fact that the Fisher information is a continuous function [54]. Recall that, for two independent and non-constant random variables, it holds that $\mathcal{I}(X+Y)<\mathcal{I}\left(X\right)$ [53] (Eq. 2.18). Since G and $SY$ are independent random variables, we find that $\mathcal{I}(\sigma G+SY)<\mathcal{I}\left(SY\right)$ which implies that $\mathcal{I}(\sigma G+SY)$ is uniformly bounded for all values of $\sigma$. Therefore,

$$\underset{\sigma \to 0}{lim}\kappa \left(\sigma \right)=\underset{\sigma \to 0}{lim}\frac{{\sigma }^2\left({\sigma }^2\mathcal{I}\left(W_{\sigma }\right)+\mathcal{I}\left(W_{\sigma }\right)-1\right)}{1+{\sigma }^2\left({\sigma }^2\mathcal{I}\left(W_{\sigma }\right)-1\right)}=0.$$

Furthermore, ${\sigma }^2\mathcal{I}(\sigma G+SY)=\mathcal{I}(G+\frac{1}{\sigma }SY)\to \mathcal{I}\left(G\right)=1$ when $\sigma \to \infty$. Hence,

$$\underset{\sigma \to \infty }{lim}\kappa \left(\sigma \right)=\underset{\sigma \to \infty }{lim}\frac{{\sigma }^2\left({\sigma }^2\mathcal{I}\left(W_{\sigma }\right)+\mathcal{I}\left(W_{\sigma }\right)-1\right)}{1+{\sigma }^2\left({\sigma }^2\mathcal{I}\left(W_{\sigma }\right)-1\right)}=1.$$

Note that ${\sigma }^2\mathcal{I}(\sigma G+SY)<{\sigma }^2\mathcal{I}\left(\sigma G\right)=1$, which further yields that $\kappa \left(\sigma \right)<1$ for all $\sigma \ge 0$. Finally, since $\mathcal{I}(·)$ is a continuous function, we deduce that range of $\kappa :{\mathbb{R}}^{+}\cup 0\to \mathbb{R}$ is $[0,1)$, implying the existence of a solution to (12) for all $\delta >1$. This completes the proof of Theorem 2. □

A useful closed-form bound on the best achievable performance: In general, determining ${\sigma }_{\mathrm{opt}}$ requires computing the Fisher information of the random variable $\sigma G+SY$ for $\sigma >0$. If the probability distribution of $SY$ is continuously differentiable (e.g., logistic model; see Appendix C.1), then we obtain the following simplified bound.

Corollary 1.

(Closed-form Lower Bound on ${\sigma }_{\mathrm{opt}}$). Let $p_{SY}:\mathbb{R}\to \mathbb{R}$ be the probability distribution of $SY$. If $p_{SY}\left(x\right)$ is differentiable for all $x\in \mathbb{R}$, then,

$$\begin{array}{c}\hfill {\sigma }_{\mathrm{opt}}^2\ge \frac{1}{(\delta -1)\left(\mathcal{I}\right(SY)-1)}.\end{array}$$

(23)

Proof. 

Based on Theorem 2, the following equation holds for $\sigma ={\sigma }_{\mathrm{opt}}$

$$\frac{1}{\delta }=\kappa \left(\sigma \right)$$

or, equivalently, by rewriting the right-hand side,

$$\begin{array}{c}\hfill \frac{1}{\delta }=1-\frac{1}{\frac{1}{1-{\sigma }^2\mathcal{I}\left(W_{\sigma }\right)}-{\sigma }^2}.\end{array}$$

(24)

Define the following function

$$\begin{array}{c}\hfill h\left(x\right):=1-\frac{1}{\frac{1}{1-{\sigma }^{2}x}-{\sigma }^2}.\end{array}$$

The function h is increasing in the region ${\mathcal{R}}_{\sigma }=\{z:z>{\sigma }^{-2}-{\sigma }^{-4}\}.$ According to Stam’s inequality [55], for two independent random variables X and Y with continuously differentiable $p_X$ and $p_Y$, it holds that

$$\mathcal{I}(X+Y)\le \frac{\mathcal{I}\left(X\right)·\mathcal{I}\left(Y\right)}{\mathcal{I}\left(X\right)+\mathcal{I}\left(Y\right)},$$

where equality is achieved if and only if X and Y are independent Gaussian random variables. Therefore, since by assumption $p_{SY}$ is differentiable on the real line, Stam’s inequality yields

$$\begin{array}{c}\hfill \mathcal{I}\left(W_{\sigma }\right)=\mathcal{I}(\sigma G+SY)\le \frac{\mathcal{I}\left(\sigma G\right)·\mathcal{I}\left(SY\right)}{\mathcal{I}\left(\sigma G\right)+\mathcal{I}\left(SY\right)}.\end{array}$$

(25)

Next, we prove that for all $\sigma >0$, both sides of (25) are in the region ${\mathcal{R}}_{\sigma }$. First, we prove that $\mathcal{I}\left(W_{\sigma }\right)\in {\mathcal{R}}_{\sigma }$. By Cramer–Rao bound (e.g., see [53] (Eq. 2.15)) for Fisher information of a random variable X, we have that $\mathcal{I}\left(X\right)\ge 1/\left(\mathrm{Var}\left[X\right]\right)$. In addition, for the random variable $W_{\sigma }$, we know that $\mathrm{Var}\left[W_{\sigma }\right]=1+{\sigma }^2-{\left(\mathbb{E}\left[SY\right]\right)}^2$, thus

$$\begin{array}{c}\hfill \mathcal{I}\left(W_{\sigma }\right)\ge \frac{1}{1+{\sigma }^2-{\left(\mathbb{E}\left[SY\right]\right)}^2}.\end{array}$$

(26)

Using the relation ${\left(\mathbb{E}\left[SY\right]\right)}^2\le \mathbb{E}\left[S^2\right]\mathbb{E}\left[Y^2\right]=1$, one can check that the following inequality holds:

$$\begin{array}{c}\hfill \frac{1}{1+{\sigma }^2-{\left(\mathbb{E}\left[SY\right]\right)}^2}\ge {\sigma }^{-2}-{\sigma }^{-4}.\end{array}$$

(27)

Therefore, from (26) and (27), we derive that $\mathcal{I}\left(W_{\sigma }\right)\in {\mathcal{R}}_{\sigma }$ for all $\sigma >0$. Furthermore, by the inequality in (25) and the definition of ${\mathcal{R}}_{\sigma }$ it directly follows that for all $\sigma >0$

$$\frac{\mathcal{I}\left(\sigma G\right)\mathcal{I}\left(SY\right)}{\mathcal{I}\left(\sigma G\right)+\mathcal{I}\left(SY\right)}\in {\mathcal{R}}_{\sigma }.$$

Finally, noting that $h(·)$ is increasing in ${\mathcal{R}}_{\sigma }$, combined with (25), we have

$$\frac{1}{\delta }=h\left(\mathcal{I}\left(W_{\sigma }\right)\right)\le h\left(\frac{\mathcal{I}\left(\sigma G\right)·\mathcal{I}\left(SY\right)}{\mathcal{I}\left(\sigma G\right)+\mathcal{I}\left(SY\right)}\right),$$

which after using the relation $\mathcal{I}\left(\sigma G\right)={\sigma }^{-2}$ and further simplification yields the inequality in the statement of the corollary. □

The proof of the corollary reveals that (23) holds with equality when $SY$ is Gaussian. In Appendix C.1, we compute $p_{SY}$ for the logistic and the Probit models with $\parallel {\mathbf{x}}_0{\parallel }_2=1$ and numerically show that it is close to the density of a Gaussian random variable. Consequently, the lower bound of Corollary 1 is almost exact when measurements are obtained according to the logistic and Probit models (see Figure A2 in the Appendix C).

3.2. On the Optimal Loss Function

It is natural to ask whether there exists a loss function that attains the bound of Theorem 2. If such a loss function exists, then we say it is optimal in the sense that it maximizes the correlation performance among all convex loss functions in (3).

Our next theorem derives a candidate for the optimal loss function, which we denote ${\ell }_{\mathrm{opt}}$. Before stating the result, we provide some intuition about the proof which builds on Theorem 2. The critical observation in the proof of Theorem 2 is that the effective noise ${\sigma }_{\widehat{\ell }}$ of $\widehat{\ell }$ is minimized (i.e., it attains the value ${\sigma }_{\mathrm{opt}}$) if the Cauchy–Schwartz inequality in (20) holds with equality. Hence, we seek $\widehat{\ell }={\ell }_{\mathrm{opt}}$ so that for some $c\in \mathbb{R}$,

$$\begin{array}{c}\hfill {\mathcal{M}}_{{\ell }_{\mathrm{opt}},1}^{\prime }\left(w;1\right)=c({\alpha }_{1}w+{\alpha }_2·{\xi }_{W_{\mathrm{opt}}}\left(w\right)).\end{array}$$

(28)

By choosing $c=-1$, integrating and ignoring constants irrelevant to the minimization of the loss function, the previous condition is equivalent to the following ${\mathcal{M}}_{{\ell }_{\mathrm{opt}}}\left(w;1\right)=-{\alpha }_1{w}^2/2-{\alpha }_{2}log\left(p_{W_{\mathrm{opt}}}\left(w\right)\right).$ It turns out that this condition can be “inverted” to yield the explicit formula for ${\ell }_{\mathrm{opt}}$ as, ${\ell }_{\mathrm{opt}}\left(w\right)=-{\mathcal{M}}_{{\alpha }_{1}q+{\alpha }_{2}log\left(p_{W_{\mathrm{opt}}}\right)}\left(w;1\right).$ Of course, one has to properly choose ${\alpha }_1$ and ${\alpha }_2$ to make sure that this function satisfies the system of equations in (19) with $\sigma ={\sigma }_{\mathrm{opt}}$. The correct choice is specified in the theorem below. The proof is deferred to Appendix D.1.

Theorem 3.

(Optimal Loss Function). Recall the definition of ${\sigma }_{\mathrm{opt}}$ in (12). Define the random variable $W_{\mathrm{opt}}:={\sigma }_{\mathrm{opt}}G+SY$ and let $p_{W_{\mathrm{opt}}}$ denote its density. Consider the following loss function ${\ell }_{\mathrm{opt}}:\mathbb{R}\to \mathbb{R}$

$$\begin{array}{c}\hfill {\ell }_{\mathrm{opt}}\left(w\right)=-{\mathcal{M}}_{{\alpha }_{1}q+{\alpha }_{2}log\left(p_{W_{\mathrm{opt}}}\right)}\left(w;1\right),\end{array}$$

(29)

where $q\left(x\right)=x^2/2$ and

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\alpha }_1& =\frac{1-{\sigma }_{\mathrm{opt}}^2\mathcal{I}\left(W_{\mathrm{opt}}\right)}{\delta ({\sigma }_{\mathrm{opt}}^2\mathcal{I}\left(W_{\mathrm{opt}}\right)+\mathcal{I}\left(W_{\mathrm{opt}}\right)-1)},\hfill \\ \hfill {\alpha }_2& =\frac{1}{\delta ({\sigma }_{\mathrm{opt}}^2\mathcal{I}\left(W_{\mathrm{opt}}\right)+\mathcal{I}\left(W_{\mathrm{opt}}\right)-1)}.\hfill \end{array}\end{array}$$

(30)

If ${\ell }_{\mathrm{opt}}$ defined as in (29) is convex and the equation $\kappa \left(\sigma \right)=1/\delta$ has a unique solution, then ${\sigma }_{{\ell }_{\mathrm{opt}}}={\sigma }_{\mathrm{opt}}$.

In general, there is no guarantee that the function ${\ell }_{\mathrm{opt}}(·)$ as defined in (29) is convex. However, if this is the case, the theorem above guarantees that it is optimal (Strictly speaking, the performance is optimal among all convex loss functions ℓ for which (8) has a unique solution as required by Theorem 2.). A sufficient condition for ${\ell }_{\mathrm{opt}}\left(w\right)$ to be convex is provided in Appendix D.2. Importantly, in Appendix D.2.1, we show that this condition holds for observations following the signed model. Thus, for this case, the resulting function is convex. Although we do not prove the convexity of optimal loss function for the logistic and Probit models, our numerical results (e.g., see Figure 3b) suggest that this is the case. Concretely, we conjecture that the loss function ${\ell }_{\mathrm{opt}}$ is convex for logistic and Probit models, and therefore by Theorem 3 its performance is optimal.

4. Special Cases

4.1. Least-Squares

By choosing $\ell \left(t\right)={(t-1)}^2$ in (3), we obtain the standard least-squares estimate. To see this, note that since $y_i=\pm 1$, it holds for all i that ${(y_i{\mathbf{a}}_i^T\mathbf{x}-1)}^2={(y_i-{\mathbf{a}}_i^T\mathbf{x})}^2.$ Thus, $\widehat{\mathbf{x}}$ is minimizing the sum of squares of the residuals:

$$\begin{array}{c}\hfill \widehat{\mathbf{x}}=arg\underset{\mathbf{x}}{min}\sum {(y_i-{\mathbf{a}}_i^T\mathbf{x})}^2.\end{array}$$

(31)

For this choice of a loss function, we can solve the equations in (8) in closed form. Furthermore, the equations have a (unique, bounded) solution for any $\delta >1$ provided that $\mathbb{E}\left[SY\right]>0$. The final result is summarized in the corollary below (see Appendix F.1 for the proof).

Corollary 2.

(Least-squares). Assume data generated from the binary model and $\delta >1$. For the label function assume that $\mathbb{E}\left[SY\right]>0$ in the notation of (7). Let $\widehat{\mathbf{x}}$ be as in (41). Then, in the limit of $m,n\to +\infty$, $m/n\to \delta$, Equations (9) and (10) hold with probability one with α and μ given as follows:

$$\begin{array}{cc}\hfill \mu & =\mathbb{E}\left[SY\right],\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill \alpha & =\sqrt{1-{\left(\mathbb{E}\left[SY\right]\right)}^2}·\sqrt{\frac{1}{\delta -1}}.\hfill \end{array}$$

(33)

Corollary 2 appears in [43] (see also [40,41,56] and Appendix F for an interpretation of the result). However, these previous works obtain results that are limited to least-squares loss. In contrast, our results are general and LS prediction is obtained as a simple corollary of our general Theorem 1. Moreover, our study of fundamental limits allows us to quantify the sub-optimality gap of least-square (LS) as follows.

On the Optimality of LS. On the one hand, Corollary 2 derives an explicit formula for the effective noise variance ${\sigma }_{\mathrm{LS}}=\alpha /\mu$ of LS in terms of $E\left[YS\right]$ and $\delta$. On the other hand, Corollary 1 provides an explicit lower bound on the optimal value ${\sigma }_{\mathrm{opt}}$ in terms of $\mathcal{I}\left(SY\right)$ and $\delta$. Combining the two, we conclude that

$$\frac{{\sigma }_{\mathrm{LS}}^2}{{\sigma }_{\mathrm{opt}}^2}\le \xi :=(\mathcal{I}\left(SY\right)-1)\frac{1-{\left(\mathbb{E}\left[SY\right]\right)}^2}{{\left(\mathbb{E}\left[SY\right]\right)}^2}.$$

In terms of correlation,

$$\begin{array}{c}\hfill \frac{{\mathrm{corr}}_{\mathrm{opt}}}{{\mathrm{corr}}_{\mathrm{LS}}}=\sqrt{\frac{1+{\sigma }_{\mathrm{LS}}^2}{1+{\sigma }_{\mathrm{opt}}^2}}\le \frac{{\sigma }_{\mathrm{LS}}}{{\sigma }_{\mathrm{opt}}}\le \sqrt{\xi },\end{array}$$

where the first inequality follows from the fact that ${\sigma }_{\mathrm{LS}}\ge {\sigma }_{\mathrm{opt}}.$ Therefore, the performance of LS is at least as good as $\frac{1}{\sqrt{\xi }}$ times the optimal one. In particular, assuming $\parallel {\mathbf{x}}_0\parallel =1$ and for logistic and Probit models (for which Corollary 1 holds), we can explicitly compute $\frac{1}{\sqrt{\xi }}=0.9972\mathrm{and}0.9804$, respectively. However, we recall that for large $\parallel {\mathbf{x}}_0\parallel$ logistic and Probit models approach the signed model, and, as Figure 1a demonstrates, LS becomes suboptimal.

Another interesting consequence of combining Corollaries 1 and 2 is that LS would be optimal if $SY$ were a Gaussian random variable. To see this, recall from Corollary 1 that, if $SY$ is Gaussian, then:

$${\sigma }_{\mathrm{opt}}^2=\frac{1}{(\delta -1)\left(\mathcal{I}\right(SY)-1)}.$$

However, for $SY$ Gaussian, we can explicitly compute $\mathcal{I}\left(SY\right)=1/\mathrm{Var}\left[SY\right]$, which leads to

$${\sigma }_{\mathrm{opt}}^2=\frac{1-{\left(\mathbb{E}\left[SY\right]\right)}^2}{{\left(\mathbb{E}\left[SY\right]\right)}^2(\delta -1)}.$$

The right hand side is exactly ${\sigma }_{\mathrm{LS}}^2$. Therefore, the optimal performance is achieved by the square loss function if $SY$ is a Gaussian random variable. Remarkably, for logistic and Probit models with small SNR (i.e., small $\parallel {\mathbf{x}}_0\parallel$), density of $SY$ is close to the density of a normal random variable (see Figure A2 in the Appendix C), implying the optimality of LS for these models.

4.2. Logistic and Hinge Loss

Theorem 1 only holds in regimes for which the set of minimizers of (3) is bounded. As we show here, this is $not$ always the case. Specifically, consider non-negative loss functions $\ell \left(t\right)\ge 0$ with the property ${lim}_{t\to +\infty }\ell \left(t\right)=0$. For example, the hinge, exponential and logistic loss functions all satisfy this property. Now, we show that for such loss functions the set of minimizers is unbounded if $\delta <{\delta }_f^{☆}$ for some appropriate ${\delta }_f^{☆}>2$. First, note that the set of minimizers is unbounded if the following condition holds:

$$\begin{array}{c}\hfill \exists {\mathbf{x}}_s\ne \mathbf{0}\mathrm{such}\mathrm{that}{y}_i{\mathbf{a}}_i^T{\mathbf{x}}_s\ge 0,\forall i\in \left[m\right].\end{array}$$

(34)

Indeed, if (34) holds then $\mathbf{x}=c·{\mathbf{x}}_s$ with $c\to +\infty$, attains zero cost in (3); thus, it is optimal and the set of minimizers is unbounded. To proceed, we rely on a recent result by Candes and Sur [44] who proved that (34) holds iff (To be precise, Candes and Sur [44] proved the statement for measurements $y_i,i\in \left[m\right]$ that follow a logistic model. Close inspection of their proof shows that this requirement can be relaxed by appropriately defining the random variable Y in (7) (see also [48,49]).)

$$\begin{array}{c}\hfill \delta \le {\delta }_f^{☆}:={\left(\underset{c\in \mathbb{R}}{min}\mathbb{E}\left[{\left(G+cSY\right)}_{-}^2\right]\right)}^{-1},\end{array}$$

(35)

where $G,S$ and Y are random variables as in (7) and ${\left(t\right)}_{-}:=min\{0,t\}$. We highlight that logistic and hinge losses give unbounded solutions in the noisy-signed model with $\epsilon =0$, since the condition (34) holds for ${\mathbf{x}}_s={\mathbf{x}}_0$. However, their performances are comparable to the optimal performance in both logistic and Probit models (see Figure 3a and Figure 4a).

5. Extensions to Gaussian-Mixture Models

In this section, we show that our results on sharp asymptotics and lower bounds on error can be extended to include the Gaussian-Mixture model (GMM) presented in Section 1.2. The discussions on the phase transition for the existence of a bounded solution in Section 4.2 applies here as well. We rely on a phase-transition result [49] (Prop. 3.1), which proves that (34) holds if and only if

$$\begin{array}{c}\hfill \delta \le {\delta }^{☆}:={\left(\underset{t\in \mathbb{R}}{min}\mathbb{E}\left[{\left(W_1+t{W}_2\right)}_{-}^2\right]\right)}^{-1},\end{array}$$

(36)

where $W_1$ and $W_2$ are random variables defined in (7) and ${\left(x\right)}_{-}^2:={\left(min\{x,0\}\right)}^2$. Therefore, for loss functions satisfying this property, e.g., hinge loss and logistic loss, the solution to (3) is unbounded if and only if $\delta \le {\delta }^{☆}.$

5.1. System of Equations for GMM

It turns out that, similar to the generative models, the asymptotic performance of (3) for GMM depends on the loss function ℓ via its Moreau envelope. Specifically, let $W_1$ and $W_2$ be independent Gaussian random variables such that

$$\begin{array}{c}\hfill W_1\sim \mathcal{N}(0,1),W_2\sim \mathcal{N}(r,1),\end{array}$$

(37)

where $r:=\parallel {\mathbf{x}}_{\mathbf{0}}{\parallel }_2>0.$ Consider the following system of non-linear equations in three unknowns $(\mu ,\alpha \ge 0,\lambda \ge 0)$:

$$\begin{array}{cc}\hfill 0& =\mathbb{E}\left[W_2·{\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha W_1+\mu W_2;\lambda \right)\right],\hfill \end{array}$$

(38a)

$$\begin{array}{cc}\hfill {\alpha }^2& ={\lambda }^2\delta \mathbb{E}\left[{\left({\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha W_1+\mu W_2;\lambda \right)\right)}^2\right],\hfill \end{array}$$

(38b)

$$\begin{array}{cc}\hfill \alpha & =\lambda \delta \mathbb{E}\left[W_1·{\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha W_1+\mu W_2;\lambda \right)\right].\hfill \end{array}$$

(38c)

The expectations above are with respect to the randomness of the random variables $W_1$ and $W_2$.

As we show shortly, the solution to these equations is tightly connected to the asymptotic behavior of the optimization in (3).

5.2. Theoretical Prediction of Error for Convex Loss Functions

Theorem 4.

(Asymptotic Prediction). Assume data generated from the Gaussian-mixture model and assume $\delta >1$ such that the set of minimizers in (3) is bounded and the system of Equation (38) has a unique solution $(\mu ,\alpha ,\lambda )$, such that $\mu \ne 0$. Let ${\widehat{\mathbf{x}}}_{\ell }$ be as in (3) and ${\sigma }_{\ell }=\alpha /\mu$. Then, in the limit of $m,n\to +\infty$, $m/n\to \delta$, it holds with probability one that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\mathrm{corr}\left({\widehat{\mathbf{x}}}_{\ell };{\mathbf{x}}_0\right)=\frac{\mu }{\sqrt{{\mu }^2+{\alpha }^2}},\underset{n\to \infty }{lim}{\mathcal{E}}_{\ell }=Q\left(\frac{r}{\sqrt{1+{\sigma }_{\ell }^2}}\right),\end{array}$$

(39)

where ${\mathcal{E}}_{\ell }$ denotes the classification test error defined in (5).

Remark 6

(Proof of Theorem 4). The high-level steps of the proof of Theorem 4 follow closely the proof of Theorem 1. Particularly, for GMM one can show the correlation of the ERM estimate with the true vector ${\mathbf{x}}_0$ is predicted by a system of Equations as in (38), only with $W_2$ replaced by a non-gaussian random variable (denoted as $SY$ in Theorem 1). Specifically, by rotational invariance of the Gaussian feature vectors ${\mathbf{a}}_i$, we can assume, without loss of generality, that ${\mathbf{x}}_0={[r,0,0,\dots ,0]}^T$. Then, we can can guarantee that with probability one it holds that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{\widehat{\mathbf{x}}}_{\ell }\left(1\right)=\mu ,\underset{n\to \infty }{lim}\sum _{j=2}^n{\widehat{\mathbf{x}}}_{\ell }^2\left(j\right)={\alpha }^2,\end{array}$$

(40)

where μ and α are specified by (38). To see how this implies (39), we argue as follows. Recalling that $\mathbf{x}|y\sim \mathcal{N}(y{\mathbf{x}}_0,\mathbf{I})$, we have

$$y⟨{\widehat{\mathbf{x}}}_{\ell },\mathbf{a}⟩\sim \mathcal{N}\left(r{\widehat{\mathbf{x}}}_{\ell }\left(1\right),{∥{\widehat{\mathbf{x}}}_{\ell }∥}_2^2\right).$$

Using this and (40) leads to the asymptotic value of correlation and classification error as presented in (39).

Remark 7.

(On the Uniqueness of Solutions to Equation (38)) Our results in proving the uniqueness of solutions to the equations for generative models (8) in Proposition 1, extend to GMM. Noting that $W_2\sim \mathcal{N}(r,1)$ in (38) plays the role of $SY$ in (8), we straightforwardly deduce the following result for uniqueness of solutions to (38).

Proposition 2.

Assume that the loss function $\ell :\mathbb{R}\to \mathbb{R}$ has the following properties: (i) it is proper strictly convex; and (ii) it is continuously differentiable and its derivative ${\ell }^{\prime }$ is such that ${\ell }^{\prime }\left(0\right)\ne 0$. The following statement is true. For any $\delta >1$, if the system of equations in (38) has a bounded solution, then it is unique.

5.3. Special Case: Least-Squares

By choosing $\ell \left(t\right)={(t-1)}^2$ in (3), we obtain the standard least-squares estimate. To see this, note that since $y_i=\pm 1$, it holds for all i that ${(y_i{\mathbf{a}}_i^T\mathbf{x}-1)}^2={(y_i-{\mathbf{a}}_i^T\mathbf{x})}^2.$

Thus, the estimator ${\widehat{\mathbf{x}}}_{LS}$ is minimizing the sum of squares of the residuals:

$$\begin{array}{c}\hfill {\widehat{\mathbf{x}}}_{LS}=arg\underset{\mathbf{x}}{min}\sum {(y_i-{\mathbf{a}}_i^T\mathbf{x})}^2.\end{array}$$

(41)

For the choice $\ell \left(t\right)={(t-1)}^2$, it turns out that we can solve the equations in (38) in closed form. The final result is summarized in the corollary below and proved in Appendix G.1.

Corollary 3.

(Least-Squares). Let ${\widehat{\mathbf{x}}}_{LS}$ be as in (31) and $\delta >1$. Then, in the limit of $m,n\to +\infty$, $m/n\to \delta$, Equation (39) holds with probability one with ${\sigma }_{LS}^2$ given as follows:

$$\begin{array}{c}\hfill {\sigma }_{LS}^2=\frac{1+r^2}{r^2}·\frac{1}{(\delta -1)}.\end{array}$$

(42)

5.4. Optimal Risk for GMM

Next, we characterize the best achievable classification error by different choices of loss function. Considering (39), we see that an optimal choice of ℓ is the one that minimizes ${\sigma }_{\ell }^2$. The next theorem characterizes the best achievable ${\sigma }_{\ell }$ among convex loss functions by deriving an equivalent set of equations to (38) and combining them with proper coefficients. Similar to the proof of Theorem 2, a key step in the proof is properly setting up a Cauchy–Schwarz inequality that exploits the structure of the new set of equations. The proof is deferred to Appendix G.2.

Theorem 5.

(Lower Bound on Risk). Under the assumptions of Theorem 4, the following inequality holds for the effective risk parameter (${\sigma }_{\ell }$) of a loss function ℓ:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{\sigma }_{\ell }^2\ge {\sigma }_{☆}^2:=\frac{1+r^2}{r^2}·\frac{1}{\delta -1}\end{array}$$

(43)

Remark 8.

(Optimality of Least-squares for GMM). Theorem 5 provides a lower bound for the asymptotic value of ${\sigma }_{\ell }$ which holds for all $\delta >1$ and $r>0$. This result together with Corollary 3 implies that least-squares achieves the least value of risk (i.e., ${\sigma }_{\ell }$ and ${\mathcal{E}}_{\ell }$) for all $\delta >1$ and $r>0$ among all convex loss functions ℓ for which the set of minimizers in (3) is bounded.

6. Numerical Experiments

In this section, we present numerical simulations that validate the predictions of Theorems 1–5. To begin, we use the following three popular models as our case study: signed, logistic and Probit. We generate random measurements according to (1). Without loss of generality (due to rotational invariance of the Gaussian measure), we set ${\mathbf{x}}_0={[1,0,\dots ,0]}^T$. We then obtain estimates ${\widehat{\mathbf{x}}}_{\ell }$ of ${\mathbf{x}}_0$ by numerically solving (3) and measure performance by the correlation value $\mathrm{corr}\left({\widehat{\mathbf{x}}}_{\ell };{\mathbf{x}}_0\right)$. Throughout the experiments, we set $n=128$ and the recorded values of correlation are averages over 25 independent realizations. For each label function, we first provide plots that compare results of Monte Carlo simulations to the asymptotic predictions for loss functions discussed in Section 4, as well as to the optimal performance of Theorem 2. We next present numerical results on optimal loss functions. To empirically derive the correlation of optimal loss function, we run gradient descent-based optimization with 1000 iterations. As a general comment, we note that, despite being asymptotic, our predictions appear accurate even for relatively small problem dimensions. For the analytical predictions, we apply Theorem 1. In particular, for solving the system of non-linear equations in (3), we empirically observe (see also [15,47] for similar observation) that, if a solution exists, then it can be efficiently found by the following fixed-point iteration method. Let $\mathbf{v}:={[\mu ,\alpha ,\lambda ]}^T$ and $\mathcal{F}:{\mathbb{R}}^3\to {\mathbb{R}}^3$ be such that (3) is equivalent to $\mathbf{v}=\mathcal{F}\left(\mathbf{v}\right)$. With this notation, we initialize $\mathbf{v}={\mathbf{v}}_0$ and for $k\ge 1$ repeat the iterations ${\mathbf{v}}_{k+1}=\mathcal{F}\left({\mathbf{v}}_k\right)$ until convergence.

Logistic model. For the logistic model, comparison between the predicted values and the numerical results is illustrated in Figure 3a. Results are shown for LS, logistic and hinge loss functions. Note that minimizing the logistic loss corresponds to the maximum-likelihood estimator (MLE) for logistic model. An interesting observation in Figure 3a is that in the high-dimensional setting (finite $\delta$) LS has comparable (if not slightly better) performance to MLE. Additionally, we observe that in this model, performance of LS is almost the same as the best possible performance derived according to Theorem 2. This confirms the analytical conclusion of Section 4.1. The comparison between the optimal loss function as in Theorem 3 and other loss functions is illustrated in Figure 3b. We note the obvious similarity between the shapes of optimal loss functions and LS which further explains the similarity between their performance.

Probit model. Theoretical predictions for the performance of hinge and LS loss functions are compared with the empirical results and optimal performance of Theorem 2 in Figure 4a. Similar to the logistic model, in this model, LS also outperforms hinge loss and its performance resembles the performance of optimal loss function derived according to Theorem 3. Figure 4b illustrates the shapes of LS, hinge loss and the optimal loss functions for the Probit model. The obvious similarity between the shape of LS and optimal loss functions for all values of $\delta$ explains the close similarity of their performance.

Additionally, by comparing the LS performance for the three models in Figure 1a, Figure 3a and Figure 4a, it is clear that higher (respectively, lower) correlation values are achieved for signed (respectively, logistic) measurements. This behavior is indeed predicted by Corollary 2: correlation performance is higher for higher values of $\mu =\mathbb{E}\left[SY\right]$. It can be shown that, for the signed, probit and logistic models (with $\parallel {\mathbf{x}}_0{\parallel }_2=1$), we have $\mu =\sqrt{2/\pi },\sqrt{1/\pi }\mathrm{and}0.4132$, respectively.

Optimal loss function. By putting together Theorems 2 and 3, we obtain a method on deriving the optimal loss function for generative binary models. This requires the following steps.

• Find ${\sigma }_{\mathrm{opt}}$ by solving (12).
• Compute the density of $W_{\mathrm{opt}}={\sigma }_{\mathrm{opt}}G+SY$.
• Compute ${\ell }_{\mathrm{opt}}$ according to (29).

Note that computing ${\sigma }_{\mathrm{opt}}$ needs the density function $p_W$ of the random variable $W=\sigma G+SY$. In principle $p_W$ can be calculated as the convolution of the Gaussian density with the pdf $p_{SY}$ of $SY$. Moreover, it follows from the recipe above that the optimal loss function depends on $\delta$ in general. This is because ${\sigma }_{\mathrm{opt}}$ itself depends on $\delta$ via (12).

Numerical Experiments for GMM

Theorem 5 implies the optimality of least-squares among convex loss functions in the under-parameterized regime $\delta >1$. In Figure 2, we demonstrate the classification risk of least-squares alongside other well-known loss functions LAD and logistic, for $r=1$. Solid lines correspond to the theoretical predictions of Theorem 4. For least-squares we rely on the result of Corollary 3 and for LAD and logistic loss, the system of equations are solved by iterating over the equations, where we observe that after relatively small number of iterations the triple $(\mu ,\alpha ,\lambda )$ converges to $({\mu }^{☆},{\alpha }^{☆},{\lambda }^{☆})$. We use ${10}^5$ and ${10}^3$ samples to compute the expectations in (38) for LAD and logistic loss, respectively. After deriving ${\sigma }_{\ell }=\alpha /\mu$, the classification risk ${\mathcal{E}}_{\ell }$ is obtained according to the formula in (39). Dots correspond to the empirical evaluations of the classification risk of loss functions for $n=60$ and for different values of $\delta =m/n>1.$ The resulting numbers are averaged over 30 independent experiments. As is observed, the empirical results closely follow the theoretical predictions of Theorem 4. Furthermore, as predicted by Theorem 5, least-squares has the minimum expected classification risk among other convex loss functions and for all $\delta >1$.

7. Conclusions

We derive theoretical predictions for the generalization error of estimators obtained by ERM for generative binary models and a Gaussian Mixture model. Furthermore, we use this theoretical characterizations to find the optimal performance and optimal loss function among all convex losses. Although our analysis is true for Gaussian matrices, we empirically show they hold for sub-Gaussian matrices as well. As an exciting future direction, we plan to extend our analysis on sharp asymptotics and optimal loss function to non-isotropic (Gaussian) features with arbitrary covariance. A more challenging, albeit interesting, direction is going beyond (binary) linear models studied in this paper, by considering asymptotics and optimal error for kernel models and neural networks (see [48,57] for partial progress in this direction).

Appendix A.1. Derivatives

Recall the definition of the Moreau envelope ${\mathcal{M}}_{\ell }\left(x;\lambda \right)$ and proximal operator ${\mathrm{prox}}_{\ell }\left(x;\lambda \right)$ of a function ℓ:

$$\begin{array}{c}\hfill {\mathcal{M}}_{\ell }\left(x;\lambda \right)=\underset{y}{min}\frac{1}{2\lambda }{(x-y)}^2+\ell \left(y\right),\end{array}$$

(A1)

and ${\mathrm{prox}}_{\ell }\left(x;\lambda \right)=arg{min}_y\frac{1}{2\lambda }{(x-y)}^2+\ell \left(y\right)$.

Proposition A1

(Basic properties of ${\mathcal{M}}_{\ell }$ and ${\mathrm{prox}}_{\ell }$ [52]). Let $\ell :\mathbb{R}\to \mathbb{R}$ be lower semi-continuous (lsc), proper and convex. The following statements hold for any $\lambda >0$.

(a)

The proximal operator ${\mathrm{prox}}_{\ell }\left(x;\lambda \right)$ is unique and continuous. In fact, ${\mathrm{prox}}_{\ell }\left(x;\lambda \right)\to {\mathrm{prox}}_{\ell }\left(x^{\prime };{\lambda }^{\prime }\right)$ whenever $(x,\lambda )\to (x^{\prime },{\lambda }^{\prime })$ with ${\lambda }^{\prime }>0$.

(b)

The value ${\mathcal{M}}_{\ell }\left(x;\lambda \right)$ is finite and depends continuously on $(\lambda ,x)$, with ${\mathcal{M}}_{\ell }\left(x;\lambda \right)\to f\left(x\right)$ for all x as $\lambda \to 0_{+}$.

(c)

The Moreau envelope function is differentiable with respect to both arguments. Specifically, for all $x\in \mathbb{R}$, the following properties are true:

$$\begin{array}{cc}\hfill {\mathcal{M}}_{\ell ,1}^{\prime }\left(x;\lambda \right)& =\frac{1}{\lambda }(x-{\mathrm{prox}}_{\ell }\left(x;\lambda \right)),\hfill \end{array}$$

(A2)

$$\begin{array}{cc}\hfill {\mathcal{M}}_{\ell ,2}^{\prime }\left(x;\lambda \right)& =-\frac{1}{2{\lambda }^2}{(x-{\mathrm{prox}}_{\ell }\left(x;\lambda \right))}^2.\hfill \end{array}$$

(A3)

If in addition ℓ is differentiable and ${\ell }^{{}^{\prime }}$ denotes its derivative, then

$$\begin{array}{cc}\hfill {\mathcal{M}}_{\ell ,1}^{\prime }\left(x;\lambda \right)& ={\ell }^{\prime }\left({\mathrm{prox}}_{\ell }\left(x;\lambda \right)\right),\hfill \end{array}$$

(A4)

$$\begin{array}{cc}\hfill {\mathcal{M}}_{\ell ,2}^{\prime }\left(x;\lambda \right)& =-\frac{1}{2}({\ell }^{\prime }{\left({\mathrm{prox}}_{\ell }\left(x;\lambda \right)\right)}^2.\hfill \end{array}$$

(A5)

Appendix A.2. Alternative Representations of (8)

Replacing the above relations for derivative of ${\mathcal{M}}_{\ell }$ in (8), we can write the equations in terms of the proximal operator. If ℓ is differentiable, then Equations (8) can be equivalently written as follows:

$$\begin{array}{cc}\hfill \mathbb{E}\left[YS·{\ell }^{\prime }\left({\mathrm{prox}}_{\ell }\left(\alpha G+\mu SY;\lambda \right)\right)\right]& =0,\hfill \end{array}$$

(A6a)

$$\begin{array}{cc}\hfill {\lambda }^2\delta \mathbb{E}\left[{\left({\ell }^{\prime }\left({\mathrm{prox}}_{\ell }\left(\alpha G+\mu SY;\lambda \right)\right)\right)}^2\right]& ={\alpha }^2,\hfill \end{array}$$

(A6b)

$$\begin{array}{cc}\hfill \lambda \delta \mathbb{E}\left[G·{\ell }^{\prime }\left({\mathrm{prox}}_{\ell }\left(\alpha G+\mu SY;\lambda \right)\right)\right]& =\alpha .\hfill \end{array}$$

(A6c)

Finally, if ℓ is two times differentiable, then applying integration by parts in Equation (14c) results in the following reformulation of (8c):

$$\begin{array}{cc}\hfill 1& =\lambda \delta \mathbb{E}\left[\frac{{\ell }^{\prime \prime }\left({\mathrm{prox}}_{\ell }\left(\alpha G+\mu SY;\lambda \right)\right)}{1+\lambda {\ell }^{\prime \prime }\left({\mathrm{prox}}_{\ell }\left(\alpha G+\mu SY;\lambda \right)\right)}\right].\hfill \end{array}$$

(A7)

Appendix A.3. Examples of Proximal Operators

LAD.

For $\ell \left(t\right)=|t-1|$, the proximal operator admits a simple expression, as follows:

$$\begin{array}{c}\hfill {\mathrm{prox}}_{\ell }\left(x;\lambda \right)=1+\mathcal{H}\left(x-1;\lambda \right),\end{array}$$

(A8)

where

$$\mathcal{H}\left(x;\lambda \right)=\left\{\begin{array}{cc}x-\lambda ,\hfill & \mathrm{if}x>\lambda ,\hfill \\ x+\lambda ,\hfill & \mathrm{if}x<-\lambda ,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

is the standard soft-thresholding function.

Hinge Loss.

When $\ell \left(t\right)=max\{0,1-t\}$, the proximal operator can be expressed in terms of the soft-thresholding function as follows:

$$\begin{array}{c}\hfill {\mathrm{prox}}_{\ell }\left(x;\lambda \right)=1+\mathcal{H}\left(x+\frac{\lambda }{2}-1;\frac{\lambda }{2}\right).\end{array}$$

Appendix A.4. Fenchel–Legendre Conjugate Representation

For a function $h:\mathbb{R}\to \mathbb{R}$, its Fenchel–Legendre conjugate, $h^{☆}:\mathbb{R}\to \mathbb{R}$ is defined as:

$$\begin{array}{c}\hfill h^{☆}\left(x\right)=\underset{y}{max}\left[xy-h\left(y\right)\right].\end{array}$$

The following proposition relates Moreau Envelope of a function to its Fenchel–Legendre conjugate.

Proposition A2.

For $\lambda >0$ and a function h, we have:

$$\begin{array}{c}\hfill {\mathcal{M}}_h\left(x;\lambda \right)=\frac{q\left(x\right)}{\lambda }-\frac{1}{\lambda }{\left(q+\lambda h\right)}^{☆}\left(x\right),\end{array}$$

(A9)

where $q\left(x\right)=x^2/2.$

Proof. 

$$\begin{array}{cc}{\mathcal{M}}_h\left(x;\lambda \right)\hfill & =\frac{1}{2\lambda }\underset{y}{min}\left[{(x-y)}^2+2\lambda h\left(y\right)\right]\hfill \\ \hfill & =\frac{x^2}{2\lambda }+\frac{1}{2\lambda }\underset{y}{min}\left[y^2-2xy+2\lambda h\left(y\right)\right]\hfill \\ \hfill & =\frac{x^2}{2\lambda }-\frac{1}{\lambda }\underset{y}{max}\left[xy-\left(y^2/2+\lambda h\left(y\right)\right)\right]\hfill \\ \hfill & =\frac{q\left(x\right)}{\lambda }-\frac{1}{\lambda }{\left(q+\lambda h\right)}^{☆}\left(x\right).\hfill \end{array}$$

□

Appendix A.5. Convexity of the Moreau Envelope

Lemma A1.

The function $H:{\mathbb{R}}^3\to \mathbb{R}$ defined as follows

$$\begin{array}{c}\hfill H(x,v,\lambda )=\frac{1}{2\lambda }{(x-v)}^2,\end{array}$$

(A10)

is jointly convex in its arguments.

Proof. 

Note that the function $h(x,v)={(x-v)}^2$ is jointly convex in $(x,v)$. Thus, its perspective function

$$\lambda h(x/\lambda ,v/\lambda )={(x-v)}^2/\lambda =2H(x,v,\lambda )$$

is jointly convex in $(x,v,\lambda )$ [58] (Sec. 2.3.3), which completes the proof. □

Proposition A3.

(a) Ref. [52] (Prop. 2.22) Let $f(x,y)$ be jointly convex in its arguments. Then, the function $g\left(x\right)={min}_{y}f(x,y)$ is convex.

(b) Ref. [58] (Sec. 3.2.3) Suppose $f_i:\mathbb{R}\to \mathbb{R}$ is a set of concave functions, with $i\in A$ an index set. Then, the function $f:\mathbb{R}\to \mathbb{R}$ defined as $f\left(x\right):={inf}_{i\in A}{f}_i\left(x\right)$ is concave.

Lemma A2.

Let $\ell :\mathbb{R}\to \mathbb{R}$ be a lsc, proper, convex function. Then, ${\mathcal{M}}_{\ell }\left(x;\lambda \right)$ is jointly convex in $(x,\lambda )$.

Proof. 

Recall that

$$\begin{array}{c}\hfill {\mathcal{M}}_{\ell }\left(x;\lambda \right)=\underset{v}{min}G\left(\mathbf{a}\right):=\frac{1}{2\lambda }{(x-v)}^2+\ell \left(v\right),\end{array}$$

(A11)

where, for compactness, we let $\mathbf{a}\in {\mathbb{R}}^3$ denote the triplet $(x,v,\lambda )$. Now, let ${\mathbf{a}}_i=(x_i,v_i,{\lambda }_i),i=1,2$, $\theta \in (0,1)$ and $\overline{\theta }:=1-\theta$. With this notation, we may write

$$\begin{array}{cc}\hfill G(\theta {\mathbf{a}}_1+\overline{\theta }{\mathbf{a}}_2)& =H\left(\theta x_1+\overline{\theta }{x}_2,\theta {\lambda }_1+\overline{\theta }{\lambda }_2,\theta v_1+\overline{\theta }{v}_2\right)+\ell (\theta v_1+\overline{\theta }{v}_2)\hfill \\ & \le \theta H(x_1,v_1,{\lambda }_1)+\overline{\theta }H(x_2,v_2,{\lambda }_2)+\theta \ell \left(v_1\right)+\overline{\theta }\ell \left(v_2\right)\hfill \\ & =\theta G\left({\mathbf{a}}_1\right)+\overline{\theta }G\left({\mathbf{a}}_2\right).\hfill \end{array}$$

For the first equality above, we recall the definition of $H:{\mathbb{R}}^3\to \mathbb{R}$ in (A10) and the inequality right after follows from Lemma A1 and convexity of ℓ. Thus, the function G is jointly convex in its arguments. Using this fact, as well as (A11), and applying Proposition A3(a) completes the proof. □

Appendix A.6. The Expected Moreau-Envelope (EME) Function and its Properties

The performance of the ERM estimator (3) is governed by the system of equations (8) in which the Moreau envelope function ${\mathcal{M}}_{\ell }\left(x;\lambda \right)$ of the loss function ℓ plays a central role. More precisely, as already hinted by (8) and becomes clear in Appendix B, what governs the behavior is the function

$$\begin{array}{c}\hfill (\alpha >0,\mu ,\tau >0,\gamma >0)\mapsto \mathbb{E}\left[{\mathcal{M}}_{\ell }\left(\alpha G+\mu SY;\tau /\gamma \right)\right],\end{array}$$

(A12)

which we call the expected Moreau envelope (EME). Recall here that $Y=f\left(S\right)$. Hence, the EME is the key summary parameter that captures the role of both the loss function $\ell :\mathbb{R}\to \mathbb{R}$ and of the link function $f:\mathbb{R}\to \{\pm 1\}$ on the statistical performance of (3).

In this section, we study several favorable properties of the EME. In (A12), the expectation is over $G,S\stackrel{\mathrm{iid}}{\sim }\mathcal{N}(0,1)$. We first study the EME under more general distribution assumptions in Appendix A.6.1, Appendix A.6.2 and Appendix A.6.3 and we then specialize our results to Gaussian random variables G and S in Appendix A.6.4.

Appendix A.6.1. Derivatives

Proposition A4.

Let $\ell :\mathbb{R}\to \mathbb{R}$ be a lsc, proper and convex function. Further, let $X,Z$ be independent random variables with bounded second moments $\mathbb{E}\left[X^2\right]<\infty$, $\mathbb{E}\left[Z^2\right]<\infty$. Then, the expected Moreau envelope function $\mathbb{E}\left[{\mathcal{M}}_{\ell }\left(cX+Z;\lambda \right)\right]$ is differentiable with respect to both c and λ and the derivatives are given as follows:

$$\begin{array}{cc}\hfill \frac{\partial }{\partial c}\mathbb{E}\left[{\mathcal{M}}_{\ell }\left(cX+Z;\lambda \right)\right]& =\mathbb{E}\left[X{\mathcal{M}}_{\ell ,1}^{\prime }\left(cX+Z;\lambda \right)\right],\hfill \end{array}$$

(A13)

$$\begin{array}{cc}\hfill \frac{\partial }{\partial \lambda }\mathbb{E}\left[{\mathcal{M}}_{\ell }\left(cX+Z;\lambda \right)\right]& =\mathbb{E}\left[{\mathcal{M}}_{\ell ,2}^{\prime }\left(cX+Z;\lambda \right)\right].\hfill \end{array}$$

(A14)

Proof. 

The proof is an application of the Dominated Convergence Theorem (DCT). First, by Proposition A1(b), for every $c\in \mathbb{R}$ and any $\lambda >0$, the function $\mathbb{E}\left[{\mathcal{M}}_{\ell }\left(cX+Z;\lambda \right)\right]$ takes a finite value. Second, by Proposition A1(c), ${\mathcal{M}}_{\ell }\left(cx+z;\lambda \right)$ is continuously differentiable with respect to both c and $\lambda$:

$$\begin{array}{cc}\hfill \frac{\partial }{\partial c}{\mathcal{M}}_{\ell }\left(cX+Z;\lambda \right)& =X{\mathcal{M}}_{\ell ,1}^{\prime }\left(cX+Z;\lambda \right)=X\frac{1}{\lambda }\left(cX+Z-{\mathrm{prox}}_{\ell }\left(cX+Z;\lambda \right)\right),\hfill \\ \hfill \frac{\partial }{\partial \lambda }{\mathcal{M}}_{\ell }\left(cX+Z;\lambda \right)& ={\mathcal{M}}_{\ell ,2}^{\prime }\left(cX+Z;\lambda \right)=-\frac{1}{2{\lambda }^2}{\left(cX+Z-{\mathrm{prox}}_{\ell }\left(cX+Z;\lambda \right)\right)}^2.\hfill \end{array}$$

From this, note that the Cauchy–Schwarz inequality gives

$$\mathbb{E}\left[\frac{\partial }{\partial c}{\mathcal{M}}_{\ell }\left(cX+Z;\lambda \right)\right]\le \left(\mathbb{E}\left[X^2\right]{)}^{1/2}\right){\left(\mathbb{E}\left[\frac{1}{{\lambda }^2}\underset{:=A}{\underbrace{{\left(cX+Z-{\mathrm{prox}}_{\ell }\left(cX+Z;\lambda \right)\right)}^2}}\right]\right)}^{1/2},$$

Therefore, the remaining condition to check so that DCT can be applied is that the term $A/{\lambda }^2$ above is integrable. To begin with, we can easily bound A as: $A\le 2{(cX+Z)}^2+2{\left({\mathrm{prox}}_{\ell }\left(cX+Z;\lambda \right)\right)}^2.$ Next, by non-expansiveness (Lipschitz property) of the proximal operator [52] (Prop. 12.19), we have that $|{\mathrm{prox}}_{\ell }\left(cX+Z;\lambda \right)|\le |cX+Z|+|{\mathrm{prox}}_{\ell }\left(0;\lambda \right)|.$ Putting together, we find that

$$A\le 6{(cX+Z)}^2+2|{\mathrm{prox}}_{\ell }\left(0;\lambda \right){|}^2\le 12c^2{X}^2+12Z^2+2{\left|{\mathrm{prox}}_{\ell }\left(0;\lambda \right)\right|}^2.$$

We consider two cases. First, for fixed $\lambda >0$ and any compact interval $\mathcal{I}$, we have that

$$\mathbb{E}\underset{c\in \mathcal{I}}{sup}\left[A\right]\le 12\left(\underset{c\in \mathcal{I}}{sup}{c}^2\right)\mathbb{E}\left[X^2\right]+12\mathbb{E}{\left[Z\right]}^2+2{\left|{\mathrm{prox}}_{\ell }\left(0;\lambda \right)\right|}^2<\infty .$$

Similarly, for fixed c and any compact interval $\mathcal{J}$ on the positive real line, we have that

$$\mathbb{E}\underset{\lambda \in \mathcal{J}}{sup}[A/{\lambda }^2]\le 12\underset{\lambda \in \mathcal{J}}{sup}\frac{c^2\mathbb{E}\left[X^2\right]+\mathbb{E}{\left[Z\right]}^2}{{\lambda }^2}+2\underset{\lambda \in \mathcal{J}}{sup}\frac{|{\mathrm{prox}}_{\ell }\left(0;\lambda \right){|}^2}{{\lambda }^2}<\infty ,$$

where we also used boundedness of the proximal operator (cf. Proposition A1(a)). This completes the proof. □

Appendix A.6.2. Strict Convexity

We study convexity properties of the expected Moreau envelope function $\Psi :{\mathbb{R}}^3\to \mathbb{R}$:

$$\begin{array}{c}\hfill \Psi \left(\mathbf{v}\right):=\Psi (\alpha ,\mu ,\lambda ):=\mathbb{E}\left[{\mathcal{M}}_{\ell }\left(\alpha X+\mu Z;\lambda \right)\right],\end{array}$$

(A15)

for a lsc, proper, convex function ℓ and independent random variables X and Z with positive densities. Here, and onwards, we let $\mathbf{v}\in {\mathbb{R}}^3$ denote a triplet $(\alpha ,\mu ,\lambda )$ and the expectation is over the randomness of X and Z. From Lemma A2, it is easy to see that $\Psi \left(\mathbf{v}\right)$ is convex. In this section, we prove a stronger claim:

“ If ℓ is strictly convex and does not attain its minimum at 0, then $\Psi \left(\mathbf{v}\right)$ is also strictly convex. ”

This is summarized in Proposition A5 below.

Proposition A5.

(Strict Convexity). Let $\ell :\mathbb{R}\to \mathbb{R}$ be a function with the following properties: (i) it is proper strictly convex; and (ii) it is continuously differentiable and its derivative ${\ell }^{\prime }$ is such that ${\ell }^{\prime }\left(0\right)\ne 0$. Further, let $X,Z$ be independent random variables with strictly positive densities. Then, the function $\Psi :{\mathbb{R}}^3\to \mathbb{R}$ in (A15) is jointly strictly convex in its arguments.

Proof. 

Let ${\mathbf{v}}_i=({\alpha }_i,{\mu }_i,{\lambda }_i),i=1,2$, $\theta \in (0,1)$ and $\overline{\theta }=1-\theta$. Further, assume that ${\mathbf{v}}_1\ne {\mathbf{v}}_2$ and define the proximal operators

$$p_i\left(X,Z\right):={\mathrm{prox}}_{\ell }\left({\alpha }_{i}X+{\mu }_{i}Z;{\lambda }_i\right)=arg\underset{v}{min}\frac{1}{2{\lambda }_i}{\left({\alpha }_{i}X+{\mu }_{i}Z-v\right)}^2+\ell \left(v\right),$$

for $i=1,2$. Finally, denote ${\lambda }_{\theta }:=\theta {\lambda }_1+\overline{\theta }{\lambda }_2,{\alpha }_{\theta }:=\theta {\alpha }_1+\overline{\theta }{\alpha }_2$ and ${\mu }_{\theta }:=\theta {\mu }_1+\overline{\theta }{\mu }_2$. With this notation, -4.6cm0cm

$$\begin{array}{cc}\hfill & \Psi (\theta {\mathbf{v}}_1+\overline{\theta }{\mathbf{v}}_2)\hfill \\ & \le \mathbb{E}\left[\frac{1}{2{\lambda }_{\theta }}{\left({\alpha }_{\theta }X+{\mu }_{\theta }Z-(\theta p_1\left(X,Z\right)+\overline{\theta }{p}_2\left(X,Z\right))\right)}^2+\ell \left(\theta p_1\left(X,Z\right)+\theta p_2\left(X,Z\right)\right)\right]\hfill \\ & =\mathbb{E}\left[H\left({\alpha }_{\theta }X+{\mu }_{\theta }Z,\theta p_1\left(X,Z\right)+\theta p_2\left(X,Z\right),{\lambda }_{\theta }\right)+\ell \left(\theta p_1\left(X,Z\right)+\overline{\theta }{p}_2\left(X,Z\right)\right)\right]\hfill \\ & \le \mathbb{E}[\theta H\left({\alpha }_{1}X+{\mu }_{1}Z,p_1\left(X,Z\right),{\lambda }_1\right)+\overline{\theta }H\left({\alpha }_{2}X+{\mu }_{2}Z,p_2\left(X,Z\right),{\lambda }_2\right)\hfill \\ & +\ell \left(\theta p_1\left(X,Z\right)+\overline{\theta }{p}_2\left(X,Z\right)\right)].\hfill \end{array}$$

(A16)

The first inequality above follows by the definition of the Moreau envelope in (A1). The equality in the second line uses the definition of the function $H:{\mathbb{R}}^3\to \mathbb{R}$ in (A10). Finally, the last inequality follows from convexity of H as proved in Lemma A1.

Continuing from (59), we may use convexity of ℓ to find that

$$\begin{array}{cc}\hfill & \Psi (\theta {\mathbf{v}}_1+\overline{\theta }{\mathbf{v}}_2)\hfill \\ & \le \mathbb{E}[\theta H({\alpha }_{1}X+{\mu }_{1}Z,{\lambda }_1,p_1\left(X,Z\right))+\overline{\theta }H({\alpha }_{2}X+{\mu }_{2}Z,{\lambda }_2,p_2\left(X,Z\right))\hfill \\ & +\theta \ell \left(p_1\left(X,Z\right)\right)+\overline{\theta }\ell \left(p_2\left(X,Z\right)\right)]\hfill \\ & =\theta \Psi \left({\mathbf{v}}_1\right)+\overline{\theta }\Psi \left({\mathbf{v}}_2\right).\hfill \end{array}$$

(A17)

This already proves convexity of (A15). In what follows, we argue that the inequality in (A17) is in fact strict under the assumption of the lemma.

Specifically, in Lemma A3, we prove that, under the assumptions of the proposition, for ${\mathbf{v}}_1\ne {\mathbf{v}}_2$, it holds that

$$\mathbb{E}\left[\ell \left(\theta p_1\left(X,Z\right)+\overline{\theta }{p}_2\left(X,Z\right)\right)\right]<\theta \mathbb{E}\left[\ell \left(p_1\left(X,Z\right)\right)\right]+\overline{\theta }\mathbb{E}\left[\ell \left(p_2\left(X,Z\right)\right)\right].$$

Using this in (A16) completes the proof of the proposition. The idea behind the proof of Lemma A3 is as follows. First, we use the fact that ${\mathbf{v}}_1\ne {\mathbf{v}}_2$ and ${\ell }^{\prime }\left(0\right)\ne 0$ to argue that there exists a non-zero measure set of $(x,z)\in {\mathbb{R}}^2$ such that $p_1\left(x,z\right)\ne p_2\left(x,z\right)$. Then, the desired claim follows by strict convexity of ℓ. □

Lemma A3.

Let $\ell :\mathbb{R}\to \mathbb{R}$ be a proper strictly convex function that is continuously differentiable with ${\ell }^{\prime }\left(0\right)\ne 0$. Further, assume independent continuous random variables $X,Z$ with strictly positive densities. Fix arbitrary triplets ${\mathbf{v}}_i=({\alpha }_i,{\mu }_i,{\lambda }_i),i=1,2$ such that ${\mathbf{v}}_1\ne {\mathbf{v}}_2$. Further, denote

$$\begin{array}{c}\hfill p_i\left(X,Z\right):={\mathrm{prox}}_{\ell }\left({\alpha }_{i}X+{\mu }_{i}Z;{\lambda }_i\right),i=1,2.\end{array}$$

(A18)

Then, there exists a ball $\mathcal{S}\subset {\mathbb{R}}^2$ of nonzero measure, i.e., $\mathbb{P}\left((X,Z)\in \mathcal{S}\right)>0$, such that $p_1\left(x,z\right)\ne p_2\left(x,z\right)$, for all $(x,z)\in \mathcal{S}$. Consequently, for any $\theta \in (0,1)$ and $\overline{\theta }=1-\theta$, the following strict inequality holds,

$$\begin{array}{c}\hfill \mathbb{E}\left[\ell \left(\theta p_1\left(X,Z\right)+\overline{\theta }{p}_2\left(X,Z\right)\right)\right]<\theta \mathbb{E}\left[\ell \left(p_1\left(X,Z\right)\right)\right]+\overline{\theta }\mathbb{E}\left[\ell \left(p_2\left(X,Z\right)\right)\right].\end{array}$$

(A19)

Proof. 

Note that (A19) holds trivially with “$<"$ replaced by “$\le "$ due to the convexity of ℓ. To prove that the inequality is strict, it suffices, by strict convexity of ℓ, that there exists subset $\mathcal{S}\subset {\mathbb{R}}^2$ that satisfies the following two properties:

• $p_1\left(x,z\right)\ne p_2\left(x,z\right)$, for all $(x,z)\in \mathcal{S}$.
• $\mathbb{P}\left((X,Z)\in \mathcal{S}\right)>0$.

Consider the following function $f:{\mathbb{R}}^2\to \mathbb{R}$:

$$\begin{array}{c}\hfill f(x,z):=p_1\left(x,z\right)-p_2\left(x,z\right).\end{array}$$

(A20)

By Lemma A4, there exists $(x_0,z_0)$ such that

$$\begin{array}{c}\hfill f(x_0,z_0)\ne 0.\end{array}$$

(A21)

Moreover, by continuity of the proximal operator (cf. Proposition A1(a)), it follows that f is continuous. From this and (A21), we conclude that for sufficiently small $\zeta >0$ there exists a $\zeta$-ball $\mathcal{S}$ centered at $(x_0,z_0)$, such that property 1 holds. Property 2 is also guaranteed to hold for $\mathcal{S}$, since both $X,Z$ have strictly positive densities and are independent. □

Lemma A4.

Let $\ell :\mathbb{R}\to \mathbb{R}$ be a proper, convex function. Further, assume that $\ell :\mathbb{R}\to \mathbb{R}$ is continuously differentiable and ${\ell }^{\prime }\left(0\right)\ne 0$. Let ${\alpha }_1,{\alpha }_2>0$, ${\lambda }_1,{\lambda }_2>0$. Then, the following statement is true

-4.6cm0cm

$$\begin{array}{c}\hfill ({\alpha }_1,{\mu }_1,{\lambda }_1)\ne ({\alpha }_2,{\mu }_2,{\lambda }_2)\to \exists (x,z)\in {\mathbb{R}}^2:{\mathrm{prox}}_{\ell }\left({\alpha }_{1}x+{\mu }_{1}z;{\lambda }_1\right)\ne {\mathrm{prox}}_{\ell }\left({\alpha }_{2}x+{\mu }_{2}z;{\lambda }_2\right).\end{array}$$

(A22)

Proof. 

We prove the claim by contradiction, but first, let us set up some useful notation. Let $\mathbf{v}\in {\mathbb{R}}^3$ denote triplets $(\alpha ,\mu ,\lambda )$ and further define

$$p_{\alpha ,\mu ,\lambda }\left(x,z\right):={\mathrm{prox}}_{\ell }\left(\alpha x+\mu z;\lambda \right),$$

and

$$L_{\alpha ,\mu ,\lambda }\left(x,z\right):={\ell }^{\prime }\left({\mathrm{prox}}_{\ell }\left(\alpha x+\mu z;\lambda \right)\right).$$

By Proposition A1, the following is true:

$$\begin{array}{c}\hfill L_{\alpha ,\mu ,\lambda }\left(x,z\right)=\frac{1}{\lambda }\left(\alpha x+\mu z-p_{\alpha ,\mu ,\lambda }\left(x,z\right)\right).\end{array}$$

(A23)

For the sake of contradiction, assume that the claim of the lemma is false. Then,

$$\begin{array}{c}\hfill p_{{\alpha }_1,{\mu }_1,{\lambda }_1}\left(x,z\right)=p_{{\alpha }_2,{\mu }_2,{\lambda }_2}\left(x,z\right),\forall (x,z)\in {\mathbb{R}}^2.\end{array}$$

(A24)

From this, it also holds that

$$\begin{array}{c}\hfill L_{{\alpha }_1,{\mu }_1,{\lambda }_1}\left(x,z\right)=L_{{\alpha }_2,{\mu }_2,{\lambda }_2}\left(x,z\right),\forall (x,z)\in {\mathbb{R}}^2.\end{array}$$

(A25)

Recalling (A23) and applying (A24), we derive the following from (A25):

$$\begin{array}{c}\hfill ({\lambda }_2-{\lambda }_1)p_{{\alpha }_1,{\mu }_1,{\lambda }_1}\left(x,z\right)=({\lambda }_2{\alpha }_1-{\lambda }_1{\alpha }_2)x+({\lambda }_2{\mu }_1-{\lambda }_1{\mu }_2)z,\forall (x,z)\in {\mathbb{R}}^2.\end{array}$$

(A26)

We consider the following two cases separately.

Case 1: ${\lambda }_1={\lambda }_2$: Since ${\mathbf{v}}_1\ne {\mathbf{v}}_2$, it holds that

$$\begin{array}{c}\hfill \exists (x,z)\in {\mathbb{R}}^2:{\alpha }_{1}x+{\mu }_{1}z\ne {\alpha }_{2}x+{\mu }_{2}z.\end{array}$$

(A27)

However, from (A26) we have that $({\alpha }_1-{\alpha }_2)x+({\mu }_1-{\mu }_2)z=0$ for all $(x,z)\in {\mathbb{R}}^2$. This contradicts (A27) and completes the proof for this case.

Case 2: ${\lambda }_1\ne {\lambda }_2$: Continuing from (A26), we can compute that for all $(x,z)\in {\mathbb{R}}^2$

$$\begin{array}{cc}\hfill {\ell }^{\prime }\left(p_{{\alpha }_1,{\mu }_1,{\lambda }_1}\left(x,z\right)\right)& =\frac{1}{{\lambda }_1}({\alpha }_{1}x+{\mu }_{1}z-p_{{\alpha }_1,{\mu }_1,{\lambda }_1}\left(x,z\right))\hfill \\ & =\frac{{\alpha }_2-{\alpha }_1}{{\lambda }_2-{\lambda }_1}x+\frac{{\mu }_2-{\mu }_1}{{\lambda }_2-{\lambda }_1}z.\hfill \end{array}$$

(A28)

By replacing $p_{{\alpha }_1,{\mu }_1,{\lambda }_1}\left(x,z\right)$ from (A26), we derive that:

$$\begin{array}{c}\hfill {\ell }^{\prime }({\epsilon }_{1}x+{\epsilon }_{2}z)={\epsilon }_{3}x+{\epsilon }_{4}z,\forall (x,z)\in {\mathbb{R}}^2,\end{array}$$

(A29)

where

$$\begin{array}{cc}\hfill {\epsilon }_1& =\frac{{\lambda }_2{\alpha }_1-{\lambda }_1{\alpha }_2}{{\lambda }_2-{\lambda }_1},{\epsilon }_2=\frac{{\lambda }_2{\mu }_1-{\lambda }_1{\mu }_2}{{\lambda }_2-{\lambda }_1},\hfill \\ \hfill {\epsilon }_3& =\frac{{\alpha }_2-{\alpha }_1}{{\lambda }_2-{\lambda }_1},{\epsilon }_4=\frac{{\mu }_2-{\mu }_1}{{\lambda }_2-{\lambda }_1}.\hfill \end{array}$$

By replacing $x=z=0$ in (A29), we find that ${\ell }^{\prime }\left(0\right)=0$. This contradicts the assumption of the lemma and completes the proof. □

Appendix A.6.3. Strict Concavity

In this section, we study the following variant $\Gamma :{\mathbb{R}}_{+}\to \mathbb{R}$ of the expected Moreau envelope:

$$\begin{array}{c}\hfill \Gamma \left(\gamma \right):=\mathbb{E}\left[{\mathcal{M}}_{\ell }\left(X;1/\gamma \right)\right],\end{array}$$

(A30)

for a lower semi-continuous, proper, convex function ℓ and continuous random variable X. The expectation above is over the randomness of X. In Appendix B.4, we show that the function $\Gamma$ is concave in $\gamma$. Here, we prove the following statement regarding strict-concavity of $\Gamma$:

“ If ℓ is convex, continuously differentiable and ${\ell }^{\prime }\left(0\right)\ne 0$, then $\Gamma$ is strictly concave. ”

This is summarized in Proposition A6 below.

Proposition A6.

(Strict concavity). Let $\ell :\mathbb{R}\to \mathbb{R}$ be a convex, continuously differentiable function for which ${\ell }^{\prime }\left(0\right)\ne 0$. Further, let X be a continuous random variable in $\mathbb{R}$ with strictly positive density in the real line. Then, the function Γ in (A23) is strictly concave in ${\mathbb{R}}_{+}$.

Proof. 

Before everything, we introduce the following convenient notation:

$${\tilde{\Gamma }}_x\left(\gamma \right):={\mathcal{M}}_{\ell }\left(x;1/\gamma \right)\mathrm{and}{p}_{\gamma }^x:={\mathrm{prox}}_{\ell }\left(x;1/\gamma \right).$$

Note from Proposition A1 that ${\tilde{\Gamma }}_x$ is differentiable with derivative

$$\begin{array}{c}\hfill {\tilde{\Gamma }}_x^{\prime }\left(\gamma \right)=\frac{1}{2}{\left(x-{\mathrm{prox}}_{\ell }\left(x;1/\gamma \right)\right)}^2.\end{array}$$

(A31)

We proceed in two steps as follows. First, for fixed $x\in \mathbb{R}$ and ${\gamma }_2>{\gamma }_1$, we prove in Lemma A5 that

$$\begin{array}{c}\hfill {(x-p_{{\gamma }_2}^x)}^2-{(x-p_{{\gamma }_1}^x)}^2\le -\frac{{\gamma }_1}{{\gamma }_2-{\gamma }_1}{(p_{{\gamma }_1}^x-p_{{\gamma }_2}^x)}^2,\end{array}$$

(A32)

This shows that for all $x\in \mathbb{R}$

$$\begin{array}{c}\hfill {\tilde{\Gamma }}_x^{\prime }\left({\gamma }_2\right)-{\tilde{\Gamma }}_x^{\prime }\left({\gamma }_1\right)\le 0.\end{array}$$

(A33)

Second, we use Lemma A3 to argue that the inequality is in fact strict for all $x\in \mathcal{S}$ where $\mathcal{S}\subset \mathbb{R}$ and $\mathbb{P}(X\in \mathcal{S})>0$. To be concrete, apply Lemma A3 for ${\mathbf{v}}_i=(1,0,1/{\gamma }_i),i=1,2$. Notice that all the assumptions of the lemma are satisfied, hence there exists interval $\mathcal{S}\subset \mathbb{R}$ for which $\mathbb{P}(X\in \mathcal{S})>0$ and

$$p_{{\gamma }_1}^x\ne p_{{\gamma }_2}^x\Rightarrow {(p_{{\gamma }_1}^x-p_{{\gamma }_2}^x)}^2>0,\forall x\in \mathcal{S}.$$

Hence, from (A32), it follows that

$${(x-p_{{\gamma }_2}^x)}^2-{(x-p_{{\gamma }_1}^x)}^2<0,\forall x\in \mathcal{S}.$$

From this, and (A31) we conclude that

$$\begin{array}{c}\hfill {\tilde{\Gamma }}_x^{\prime }\left({\gamma }_2\right)-{\tilde{\Gamma }}_x^{\prime }\left({\gamma }_1\right)<0,\forall x\in \mathcal{S}.\end{array}$$

(A34)

Thus, from (A33) and (A34), as well as the facts that $\Gamma \left(\gamma \right)=\mathbb{E}\left[{\tilde{\Gamma }}_X\left(\gamma \right)\right]$ and $\mathbb{P}(X\in \mathcal{S})>0$, we conclude that $\Gamma$ is strictly concave in ${\mathbb{R}}_{+}$. □

Lemma A5.

Let $\ell :\mathbb{R}\to \mathbb{R}$ be a convex, continuously differentiable function. Fix $x\in \mathbb{R}$ and denote $p_{\gamma }:={\mathrm{prox}}_{\ell }\left(x;1/\gamma \right)$. Then, for any $\gamma ,\tilde{\gamma }>0$, it holds that

$$\begin{array}{c}\hfill (\tilde{\gamma }-\gamma )(p_{\tilde{\gamma }}-p_{\gamma })(p_{\gamma }-x)+\tilde{\gamma }{(p_{\tilde{\gamma }}-p_{\gamma })}^2\le 0.\end{array}$$

(A35)

Moreover, for ${\gamma }_2>{\gamma }_1$, the following statement is true:

$$\begin{array}{c}\hfill {(x-p_{{\gamma }_2})}^2-{(x-p_{{\gamma }_1})}^2\le -\frac{{\gamma }_1}{{\gamma }_2-{\gamma }_1}{(p_{{\gamma }_1}-p_{{\gamma }_2})}^2.\end{array}$$

(A36)

Proof. 

First, we prove (A35). Then, we use it to prove (A36).

Proof of (A35): Consider function $g:\mathbb{R}\to \mathbb{R}$ defined as follows $g\left(p\right)=\frac{\tilde{\gamma }}{2}{(x-p)}^2+\ell \left(p\right)$. By assumption, g is differentiable with derivative $g^{\prime }\left(p\right)=\tilde{\gamma }(p-x)+{\ell }^{\prime }\left(p\right)$. Moreover, g is ${\gamma }_2$-strongly convex. Finally, by optimality of the proximal operator (cf. Proposition A1), it holds that $\gamma (x-p_{\gamma })={\ell }^{\prime }\left(p_{\gamma }\right)$ and $\tilde{\gamma }(x-p_{\tilde{\gamma }})={\ell }^{\prime }\left(p_{\tilde{\gamma }}\right)$. Using these, it can be computed that $g^{\prime }\left(p_{\tilde{\gamma }}\right)=0$ and $g^{\prime }\left(p_{\gamma }\right)=(\tilde{\gamma }-\gamma )(p_{\gamma }-x)$.

In the following inequalities, we combine all the aforementioned properties of the function g to find that

$$\begin{array}{cc}\hfill g\left(p_{\gamma }\right)& \ge g\left(p_{\tilde{\gamma }}\right)+\frac{\tilde{\gamma }}{2}{(p_{\gamma }-p_{\tilde{\gamma }})}^2\ge g\left(p_{\gamma }\right)+(\tilde{\gamma }-\gamma )(p_{\gamma }-x)(p_{\tilde{\gamma }}-p_{\gamma })+\tilde{\gamma }{(p_{\gamma }-p_{\tilde{\gamma }})}^2.\hfill \end{array}$$

This leads to the desired statement and completes the proof of (A35).

Proof of (A36): We fix ${\gamma }_2>{\gamma }_1$ and apply (A35) two times as follows. First, applying (A35) for $(\tilde{\gamma },\gamma )=({\gamma }_2,{\gamma }_1)$ and using the fact that ${\gamma }_2>{\gamma }_1$, we find that

$$\begin{array}{c}\hfill (p_{{\gamma }_2}-p_{{\gamma }_1})(p_{{\gamma }_1}-x)\le -\frac{{\gamma }_2}{{\gamma }_2-{\gamma }_1}{(p_{{\gamma }_2}-p_{{\gamma }_1})}^2.\end{array}$$

(A37)

Second, applying (A35) for $(\tilde{\gamma },\gamma )=({\gamma }_1,{\gamma }_2)$ and using again the fact that ${\gamma }_2>{\gamma }_1$, we find that

$$\begin{array}{cc}& ({\gamma }_1-{\gamma }_2)(p_{{\gamma }_1}-p_{{\gamma }_2})(p_{{\gamma }_2}-x)+{\gamma }_1{(p_{{\gamma }_1}-p_{{\gamma }_2})}^2\le 0\hfill \\ \hfill \Rightarrow & (p_{{\gamma }_2}-p_{{\gamma }_1})(p_{{\gamma }_2}-x)\le -\frac{{\gamma }_1}{{\gamma }_2-{\gamma }_1}{(p_{{\gamma }_1}-p_{{\gamma }_2})}^2.\hfill \end{array}$$

(A38)

Adding (A37) and (A38), we show the desired property as follows:

$$(p_{{\gamma }_2}-p_{{\gamma }_1})(p_{{\gamma }_2}-x)+(p_{{\gamma }_2}-p_{{\gamma }_1})(p_{{\gamma }_1}-x)\le -\frac{{\gamma }_2+{\gamma }_1}{{\gamma }_2-{\gamma }_1}{(p_{{\gamma }_1}-p_{{\gamma }_2})}^2.$$

□

Appendix A.6.4. Summary of Properties of (uid135)

Proposition A7.

Let $\ell :\mathbb{R}\to \mathbb{R}$ be a lsc, proper, convex function. Let $G,S\stackrel{\mathrm{iid}}{\sim }\mathcal{N}(0,1)$ and function $f:\mathbb{R}\to \{\pm 1\}$ such that the random variable $YS=f\left(S\right)S$ has a continuous strictly positive density on the real line. Then, the following properties are true for the expected Moreau envelope function

$$\begin{array}{c}\hfill \Omega :(\alpha >0,\mu ,\tau >0,\gamma >0)\mapsto \mathbb{E}\left[{\mathcal{M}}_{\ell }\left(\alpha G+\mu SY;\tau /\gamma \right)\right]:\end{array}$$

(A39)

(a)

The function Ω is differentiable and its derivatives are given as follows:

$$\begin{array}{cc}\hfill \frac{\partial }{\partial \alpha }\Omega (\alpha ,\mu ,\tau ,\gamma )& =\mathbb{E}\left[G{\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha G+\mu SY;\tau /\gamma \right)\right],\hfill \\ \hfill \frac{\partial }{\partial \mu }\Omega (\alpha ,\mu ,\tau ,\gamma )& =\mathbb{E}\left[SY{\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha G+\mu SY;\tau /\gamma \right)\right],\hfill \\ \hfill \frac{\partial }{\partial \tau }\Omega (\alpha ,\mu ,\tau ,\gamma )& =\frac{1}{\gamma }\mathbb{E}\left[{\mathcal{M}}_{\ell ,2}^{\prime }\left(\alpha G+\mu SY;\tau /\gamma \right)\right],\hfill \\ \hfill \frac{\partial }{\partial \gamma }\Omega (\alpha ,\mu ,\tau ,\gamma )& =-\frac{\tau }{{\gamma }^2}\mathbb{E}\left[{\mathcal{M}}_{\ell ,2}^{\prime }\left(\alpha G+\mu SY;\tau /\gamma \right)\right].\hfill \end{array}$$

(b)

The function Ω is jointly convex $(\alpha ,\mu ,\tau )$ and concave on γ.

(c)

The function Ω is increasing in α.

For the statements below, further assume that ℓ is strictly convex and continuously differentiable with ${\ell }^{\prime }\left(0\right)\ne 0$.

(d)

The function Ω is strictly convex in $(\alpha ,\mu ,\tau )$ and strictly concave in λ.

(e)

The function Ω is strictly increasing in α.

Proof. 

Statements (a), (b) and (d) follow directly by Propositions A4–A6. It remains to prove Statements (c) and (e). Let ${\alpha }_2>{\alpha }_1$. Then, there exist independent copies $G^{\prime },G^{\prime \prime }$ of G and $\tilde{\alpha }>0$ such that ${\alpha }_{2}G={\alpha }_1{G}^{\prime }+\tilde{\alpha }{G}^{\prime \prime }$. Hence, we have the following chain of inequalities:

$$\begin{array}{cc}\hfill \mathrm{\Omega }({\alpha }_2,\mu ,\tau ,\gamma )& =\mathbb{E}\left[{\mathcal{M}}_{\ell }\left({\alpha }_1{G}^{\prime }+\tilde{\alpha }{G}^{\prime \prime }+\mu SY;\tau /\gamma \right)\right]\ge \mathbb{E}\left[{\mathcal{M}}_{\ell }\left({\alpha }_1{G}^{\prime }+\tilde{\alpha }\mathbb{E}\left[G^{\prime \prime }\right]+\mu SY;\tau /\gamma \right)\right]\hfill \\ \hfill & =\mathbb{E}\left[{\mathcal{M}}_{\ell }\left({\alpha }_1{G}^{\prime }+\mu SY;\tau /\gamma \right)\right]=\Omega ({\alpha }_1,\mu ,\tau ,\gamma ),\hfill \end{array}$$

where the inequality follows from Jensen and convexity of $\mathrm{\Omega }$ with respect to $\alpha$ (see Statement (b) of the Proposition). This proves Statement (c). For Statement (e), note that the inequality is strict provided that $\mathrm{\Omega }$ is strictly convex (see Statement (d) of the Proposition). □

Appendix B.1. Technical Tool: CGMT

Appendix B.1.1. Gordon’s Min-Max Theorem (GMT)

The Gordon’s Gaussian comparison inequality [59] compares the min-max value of two doubly indexed Gaussian processes based on how their autocorrelation functions compare. The inequality is quite general (see [59]), but for our purposes we only need its application to the following two Gaussian processes:

$$\begin{array}{cc}\hfill X_{\mathbf{w},\mathbf{u}}& :={\mathbf{u}}^T\mathbf{G}\mathbf{w}+\psi (\mathbf{w},\mathbf{u}),\hfill \end{array}$$

(A40a)

$$\begin{array}{cc}\hfill Y_{\mathbf{w},\mathbf{u}}& :={\parallel \mathbf{w}\parallel }_2{\mathbf{g}}^T\mathbf{u}+{\parallel \mathbf{u}\parallel }_{2}T\mathbf{w}+\psi (\mathbf{w},\mathbf{u}),\hfill \end{array}$$

(A40b)

where $\mathbf{G}\in {\mathbb{R}}^{m\times n}$, $\mathbf{g}\in {\mathbb{R}}^m$, ${\mathbb{R}}^n$, they all have entries iid Gaussian; the sets ${\mathcal{S}}_{\mathbf{w}}\subset {\mathbb{R}}^n$ and ${\mathcal{S}}_{\mathbf{u}}\subset {\mathbb{R}}^m$ are compact; and $\psi :{\mathbb{R}}^n\times {\mathbb{R}}^m\to \mathbb{R}$. For these two processes, define the following (random) min-max optimization programs, which we refer to as the primary optimization (PO) problem and the auxiliary optimization (AO).

$$\begin{array}{cc}\hfill \tilde{\Phi }\left(\mathbf{G}\right)& =\underset{\mathbf{w}\in {\mathcal{S}}_{\mathbf{w}}}{min}\underset{\mathbf{u}\in {\mathcal{S}}_{\mathbf{u}}}{max}{X}_{\mathbf{w},\mathbf{u}},\hfill \end{array}$$

(A41a)

$$\begin{array}{cc}\hfill \varphi (\mathbf{g},& =\underset{\mathbf{w}\in {\mathcal{S}}_{\mathbf{w}}}{min}\underset{\mathbf{u}\in {\mathcal{S}}_{\mathbf{u}}}{max}{Y}_{\mathbf{w},\mathbf{u}}.\hfill \end{array}$$

(A41b)

According to Gordon’s comparison inequality (To be precise, the formulation in (A42), which is due to [13], is slightly different from the original statement in Gordon’s paper (see [13] for details).), for any $c\in \mathbb{R}$, it holds:

$$\mathbb{P}\left(\tilde{\Phi }\left(\mathbf{G}\right)<c\right)\le 2\mathbb{P}\left(\varphi (\mathbf{g},<c\right).$$

(A42)

In other words, a high-probability lower bound on the AO is a high-probability lower bound on the PO. The premise is that it is often much simpler to lower bound the AO rather than the PO. To be precise, (A42) is a slight reformulation of Gordon’s original result proved in [13].

Appendix B.1.2. Convex Gaussian Min-Max Theorem (CGMT)

The proof of Theorem 1 builds on the CGMT [13]. For ease of reference, we summarize here the essential ideas of the framework following the presentation in [15] (please see [15] (Section 6) for the formal statement of the theorem and further details). The CGMT is an extension of the GMT and it asserts that the AO in (41b) can be used to tightly infer properties of the original (PO) in (41a), including the optimal cost and the optimal solution. According to the CGMT [15] (Theorem 6.1), if the sets ${\mathcal{S}}_{\mathbf{w}}$ and ${\mathcal{S}}_{\mathbf{u}}$ are convex and $\psi$ is continuous convex-concave on ${\mathcal{S}}_{\mathbf{w}}\times {\mathcal{S}}_{\mathbf{u}}$, then, for any $\nu \in \mathbb{R}$ and $t>0$, it holds that

$$\mathbb{P}\left(|\tilde{\Phi }\left(\mathbf{G}\right)-\nu |>t\right)\le 2\mathbb{P}\left(\left|\varphi (\mathbf{g},-\nu \right|>t\right).$$

(A43)

In words, concentration of the optimal cost of the AO problem around $\mu$ implies concentration of the optimal cost of the corresponding PO problem around the same value $\mu$. Moreover, starting from (A43) and under strict convexity conditions, the CGMT shows that concentration of the optimal solution of the AO problem implies concentration of the optimal solution of the PO to the same value. For example, if minimizers of (A41b) satisfy $\parallel {\mathbf{w}}^{*}{(\mathbf{g},\parallel }_2\to {\zeta }^{*}$ for some ${\zeta }^{*}>0$, then the same holds true for the minimizers of (A41a): ${\parallel {\mathbf{w}}^{*}\left(\mathbf{G}\right)\parallel }_2\to {\zeta }^{*}$ [15] ([Theorem 6.1(iii)). Thus, one can analyze the AO to infer corresponding properties of the PO, the premise being of course that the former is simpler to handle than the latter.

Appendix B.2. Applying the CGMT to ERM for Binary Classification

In this section, we show how to apply the CGMT to (3). For convenience, we drop the subscript ℓ from ${\widehat{\mathbf{x}}}_{\ell }$ and simply write

$$\widehat{\mathbf{x}}=arg\underset{\mathbf{x}}{min}\frac{1}{m}\sum _{i=1}^m\ell \left(y_i{\mathbf{a}}_i^T\mathbf{x}\right),$$

(A44)

where the measurements $y_i,i\in \left[m\right]$ follow (1). By rotational invariance of the Gaussian distribution of the measurement vectors ${\mathbf{a}}_i,i\in \left[m\right]$, we assume without loss of generality that ${\mathbf{x}}_0={[1,0,\dots ,0]}^T$. We can rewrite (A44) as a constrained optimization problem by introducing n variables $u_i$ as follows:

$$\widehat{\mathbf{x}}=arg\underset{\mathbf{x},\mathbf{u}}{min}\frac{1}{m}\sum _{i=1}^m\ell \left(u_i\right)\mathrm{subject}\mathrm{to}{u}_i=y_i{\mathbf{a}}_i^T\mathbf{x},i\in \left[n\right].$$

This problem is now equivalent to the following min-max formulation:

$$\begin{array}{c}\hfill \underset{\mathbf{u},\mathbf{x}}{min}\underset{\beta }{max}\frac{1}{m}\sum _{i=1}^m\ell \left(u_i\right)+\frac{1}{m}\sum _{i=1}^m{\beta }_i{u}_i-\frac{1}{m}\sum _{i=1}^m{\beta }_i{y}_i{\mathbf{a}}_i^T\mathbf{x}.\end{array}$$

(A45)

Now, let us define

$${\mathbf{a}}_i=[s_i;{\tilde{\mathbf{a}}}_i],i\in \left[m\right]\mathrm{and}\mathbf{x}=[x_1;\tilde{\mathbf{x}}],$$

such that $s_i$ and $x_1$ are the first entries of ${\mathbf{a}}_i$ and $\mathbf{x}$, respectively. Note that in this new notation (1) becomes:

$$\begin{array}{c}\hfill y_i=f\left(s_i\right),\end{array}$$

(A46)

and

$$\begin{array}{c}\hfill \mathrm{corr}\left(\widehat{\mathbf{x}};{\mathbf{x}}_0\right)=\frac{{\widehat{x}}_1}{\sqrt{{\widehat{x}}_1^2+{\parallel \widetilde{\widehat{\mathbf{x}}}\parallel }_2^2}},\end{array}$$

(A47)

where we decompose $\widehat{\mathbf{x}}=[{\widehat{x}}_1;\widetilde{\widehat{\mathbf{x}}}]$. In addition, (A45) is written as

$$\underset{\mathbf{u},\mathbf{x}}{min}\underset{\beta }{max}\frac{1}{m}\sum _{i=1}^m\ell \left(u_i\right)+\frac{1}{m}\sum _{i=1}^m{\beta }_i{u}_i+\frac{1}{m}\sum _{i=1}^m{\beta }_i{y}_i{\tilde{\mathbf{a}}}_i^T\tilde{\mathbf{x}}-\frac{1}{m}\sum _{i=1}^m{\beta }_i{y}_i{s}_i{x}_1$$

or, in matrix form, as

$$\underset{\mathbf{u},\mathbf{x}}{min}\underset{\beta }{max}\frac{1}{m}{\beta }^T{\mathbf{D}}_y\tilde{\mathbf{A}}\tilde{\mathbf{x}}+\frac{1}{m}{x}_1{\beta }^T{\mathbf{D}}_y\mathbf{s}+\frac{1}{m}{\beta }^T\mathbf{u}+\frac{1}{m}\sum _{i=1}^m\ell \left(u_i\right).$$

(A48)

where ${\mathbf{D}}_{\mathbf{y}}:=\mathrm{diag}(y_1,y_2,\dots ,y_m)$ is a diagonal matrix with $y_1,y_2,\dots y_m$ on the diagonal, $\mathbf{s}={[s_1,\dots ,s_m]}^T$ and $\tilde{\mathbf{A}}$ is an $m\times (n-1)$ matrix with rows ${\tilde{\mathbf{a}}}_i^T,i\in \left[m\right]$.

In (A48), we recognize that the first term has the bilinear form required by the GMT in (A41a). The rest of the terms form the function $\psi$ in (A41a): they are independent of $\tilde{\mathbf{A}}$ and convex-concave as desired by the CGMT. Therefore, we express (A44) in the desired form of a PO and for the rest of the proof we analyze the probabilistically equivalent AO problem. In view of (A41b), this is given as follows,

$$\underset{\mathbf{u},\mathbf{x}}{min}\underset{\beta }{max}\frac{1}{m}{∥\tilde{\mathbf{x}}∥}_2{\mathbf{g}}^T{\mathbf{D}}_y\beta +\frac{1}{m}{∥{\mathbf{D}}_y\beta ∥}_2{\mathbf{h}}^T\tilde{\mathbf{x}}-\frac{1}{m}{x}_1{\beta }^T{\mathbf{D}}_y\mathbf{s}+\frac{1}{m}{\beta }^T\mathbf{u}+\frac{1}{m}\sum _{i=1}^m\ell \left(u_i\right),$$

(A49)

where as in (A41b) $\mathbf{g}\sim \mathcal{N}(0,I_m)$ and $\mathbf{h}\sim \mathcal{N}(0,I_{n-1})$.

Appendix B.3. Analysis of the Auxiliary Optimization

Here, we show how to analyze the AO in (A49). To begin with, note that $y_i\in \{\pm 1\}$, therefore ${\mathbf{D}}_{\mathbf{y}}\mathbf{g}\sim \mathcal{N}(0,I_m)$ and ${∥{\mathbf{D}}_{\mathbf{y}}\beta ∥}_2={∥\beta ∥}_2$. In addition, let us denote the first entry $x_1$ of $\mathbf{x}$ as

$$\mu :=x_1.$$

The first step is to optimize over the direction of $\tilde{\mathbf{x}}$. For this, we express the AO as: -4.6cm0cm

$$\underset{\mathbf{u},\mu ,\alpha \ge 0}{min}\underset{\parallel \tilde{\mathbf{x}}{\parallel }_2=\alpha }{min}\underset{\beta }{max}\frac{1}{m}{∥\tilde{\mathbf{x}}∥}_2{\mathbf{g}}^T{\mathbf{D}}_y\mathbf{\beta }+\frac{1}{m}{∥{\mathbf{D}}_y\mathbf{\beta }∥}_2{\mathbf{h}}^T\tilde{\mathbf{x}}-\frac{1}{m}\mu {\mathbf{\beta }}^T{\mathbf{D}}_y\mathbf{s}+\frac{1}{m}{\mathbf{\beta }}^T\mathbf{u}+\frac{1}{m}\sum _{i=1}^m\ell \left(u_i\right),$$

(A50)

Now, denote ${\tilde{\mathbf{x}}}_{☆}{=-\alpha \parallel }_2$ and observe that for every $\mathbf{\beta }$ the objective above is minimized (with respect to to $\tilde{\mathbf{x}}$) at ${\tilde{\mathbf{x}}}_{☆}$. Thus, it follows by [23] (Lem. 8) that (A50) simplifies to

$$\underset{\alpha \ge 0,\mu ,\mathbf{u}}{min}\underset{\mathbf{\beta }}{max}\frac{1}{m}\alpha {\mathbf{g}}^T\mathbf{\beta }-\frac{\alpha }{m}{∥\mathbf{\beta }∥}_2{∥\mathbf{h}∥}_2-\frac{1}{m}\mu {\mathbf{s}}^T{\mathbf{D}}_{\mathbf{y}}\mathbf{\beta }+\frac{1}{m}{\mathbf{\beta }}^T\mathbf{u}+\frac{1}{m}\sum _{i=1}^m\ell \left(u_i\right).$$

(A51)

Next, let $\gamma :=\frac{{∥\mathbf{\beta }∥}_2}{\sqrt{m}}$ and optimize over the direction of $\mathbf{\beta }$ to yield

$$\underset{\alpha \ge 0,\mathbf{u},\mu }{min}\underset{\gamma \ge 0}{max}\frac{\gamma }{\sqrt{m}}{∥\alpha \mathbf{g}-\mu {\mathbf{D}}_{\mathbf{y}}\mathbf{s}+\mathbf{u}∥}_2-\frac{\alpha }{\sqrt{m}}\gamma {∥\mathbf{h}∥}_2+\frac{1}{m}\sum _{i=1}^m\ell \left(u_i\right).$$

(A52)

To continue, we utilize the fact that for all $x\in \mathbb{R}$, ${min}_{\tau >0}\frac{\tau }{2}+\frac{x^2}{2\tau m}=\frac{x}{\sqrt{m}}$. Hence,

$$\begin{array}{c}\hfill \frac{\gamma }{\sqrt{m}}{∥\alpha \mathbf{g}-\mu {\mathbf{D}}_{\mathbf{y}}\mathbf{s}+\mathbf{u}∥}_2=\underset{\tau >0}{min}\frac{\gamma \tau }{2}+\frac{\gamma }{2\tau m}{∥-\alpha \mathbf{g}+\mu {\mathbf{D}}_{\mathbf{y}}\mathbf{s}-\mathbf{u}∥}_2^2.\end{array}$$

With this trick, the optimization over $\mathbf{u}$ becomes separable over its coordinates $u_i,i\in \left[m\right]$. By inserting this in (A42), we have

$$\underset{\alpha \ge 0,\mathbf{u},\mu }{min}\underset{\gamma \ge 0}{max}\underset{\tau >0}{min}\frac{\gamma \tau }{2}-\frac{\alpha }{\sqrt{m}}\gamma {∥\mathbf{h}∥}_2+\frac{\gamma }{2\tau m}\sum _{i=1}^m{(-\alpha g_i+\mu y_i{s}_i-u_i)}^2+\frac{1}{m}\sum _{i=1}^m\ell \left(u_i\right),$$

Now, we show that the objective function above is convex-concave. Clearly, the function is linear (thus, concave in $\gamma$). Moreover, from Lemma A1, the function $\frac{1}{2\tau }{(\alpha g_i+\mu y_i{s}_i-u_i)}^2$ is jointly convex in $(\alpha ,\mu ,u_i,\tau )$. The rest of the terms are clearly convex and this completes the argument. Hence, with a permissible change in the order of min-max, we arrive at the following convenient form (Here, we skip certain technical details in this argument regarding boundedness of the constraint sets in (A49). While they are not trivial, they can be handled with the same techniques used in [15,60].):

$$\underset{\mu ,\alpha \ge 0,\tau >0}{min}\underset{\gamma \ge 0}{max}\frac{\gamma \tau }{2}-\frac{\alpha }{\sqrt{m}}\gamma {∥\mathbf{h}∥}_2+\frac{1}{m}\sum _{i=1}^m{\mathcal{M}}_{\ell }\left(-\alpha g_i+\mu s_i{y}_i;\frac{\tau }{\gamma }\right),$$

(A53)

where recall the definition of the Moreau envelope in (A1). As to now, we have reduced the AO into a random min-max optimization over only four scalar variables in (A53). For fixed $\mu ,\alpha ,\tau ,\gamma$, direct application of the weak law of large numbers shows that the objective function of (A53) converges in probability to the following as $m,n\to \infty$ and $\frac{m}{n}=\delta$:

$$\gamma \frac{\tau }{2}-\frac{\alpha \gamma }{\sqrt{\delta }}+\mathbb{E}\left[{\mathcal{M}}_{\ell }\left(\alpha G+\mu YS;\frac{\tau }{\gamma }\right)\right],$$

where $G,S\sim \mathcal{N}(0,1)$ and $Y\sim f\left(S\right)$ (in view of (A46)). Based on that, it can be shown (similar arguments are developed in [15,60]) that the random optimizers ${\alpha }_n$ and ${\mu }_n$ of (A53) converge to the deterministic optimizers $\alpha$ and $\mu$ of the following (deterministic) optimization problem (whenever these are bounded as the statement of the theorem requires):

$$\underset{\alpha \ge 0,\mu ,\tau >0}{min}\underset{\gamma \ge 0}{max}\gamma \frac{\tau }{2}-\frac{\alpha \gamma }{\sqrt{\delta }}+\mathbb{E}\left[{\mathcal{M}}_{\ell }\left(\alpha G+\mu YS;\frac{\tau }{\gamma }\right)\right].$$

(A54)

At this point, recall that $\alpha$ represents the norm of $\tilde{\mathbf{x}}$ and $\mu$ the value of $x_1$. Thus, in view of (i) (A47); (ii) the equivalence between the PO and the AO; and (iii) our derivations thus far, we have that with probability approaching 1,

$$\begin{array}{c}\hfill \underset{n\to +\infty }{lim}\mathrm{corr}\left(\widehat{\mathbf{x}};{\mathbf{x}}_0\right)=\frac{\mu }{\sqrt{{\mu }^2+{\alpha }^2}},\end{array}$$

where $\mu$ and $\alpha$ are the minimizers in (A54). The three equations in (8) are derived by the first-order optimality conditions of the optimization in (A54). We show this next.

Appendix B.4. Convex-Concavity and First-Order Optimality Conditions

First, we prove that the objective function in (A54) is convex–concave. For convenience define the function $F:{\mathbb{R}}^4\to \mathbb{R}$ as follows

$$\begin{array}{c}\hfill F(\alpha ,\mu ,\tau ,\gamma ):=\frac{\gamma \tau }{2}-\frac{\alpha \gamma }{\sqrt{\delta }}+\mathbb{E}\left[{\mathcal{M}}_{\ell }\left(\alpha G+\mu YS;\frac{\tau }{\gamma }\right)\right].\end{array}$$

(A55)

Based on Lemma A2, it immediately follows that, if ℓ is convex, F is jointly convex in $(\alpha ,\mu ,\tau )$. To prove concavity of F based on $\gamma$, it suffices to show that ${\mathcal{M}}_{\ell }\left(x;1/\gamma \right)$ is concave in $\gamma$ for all $x\in \mathbb{R}$. To show this, we note that

$$\begin{array}{c}\hfill {\mathcal{M}}_{\ell }\left(x;1/\gamma \right)=\underset{u}{min}\frac{\gamma }{2}{(x-u)}^2+\ell \left(u\right),\end{array}$$

which is the point-wise minimum of linear functions of $\gamma$. Thus, using Proposition A3(b), we conclude that ${\mathcal{M}}_{\ell }\left(x;1/\gamma \right)$ is concave in $\gamma$. This completes the proof of convex-concavity of the function F in (A55) when ℓ is convex. By direct differentiation and applying Proposition A7(a), the first-order optimality conditions of the min–max optimization in (A54) are as follows:

$$\begin{array}{cc}\hfill \mathbb{E}\left[SY·{\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha G+\mu SY;\frac{\tau }{\gamma }\right)\right]& =0,\hfill \end{array}$$

(A56a)

$$\begin{array}{cc}\hfill \mathbb{E}\left[G·{\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha G+\mu SY;\frac{\tau }{\gamma }\right)\right]& =\frac{\gamma }{\sqrt{\delta }},\hfill \end{array}$$

(A56b)

$$\begin{array}{cc}\hfill \frac{\gamma }{2}+\frac{1}{\gamma }\mathbb{E}\left[{\mathcal{M}}_{\ell ,2}^{\prime }\left(\alpha G+\mu SY;\frac{\tau }{\gamma }\right)\right]& =0,\hfill \end{array}$$

(A56c)

$$\begin{array}{cc}\hfill -\frac{\alpha }{\sqrt{\delta }}-\frac{\tau }{{\gamma }^2}\mathbb{E}\left[{\mathcal{M}}_{\ell ,2}^{\prime }\left(\alpha G+\mu SY;\frac{\tau }{\gamma }\right)\right]+\frac{\tau }{2}& =0.\hfill \end{array}$$

(A56d)

Next, we show how these equations simplify to the following system of equations (same as (8):

$$\begin{array}{cc}\hfill \mathbb{E}\left[YS·{\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha G+\mu SY;\lambda \right)\right]& =0,\hfill \end{array}$$

(A57a)

$$\begin{array}{cc}\hfill {\lambda }^2\delta \mathbb{E}\left[{\left({\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha G+\mu SY;\lambda \right)\right)}^2\right]& ={\alpha }^2,\hfill \end{array}$$

(A57b)

$$\begin{array}{cc}\hfill \lambda \delta \mathbb{E}\left[G·{\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha G+\mu SY;\lambda \right)\right]& =\alpha .\hfill \end{array}$$

(A57c)

Let $\lambda :=\frac{\tau }{\gamma }$. First, (A57a) is immediate from equation (A56a). Second, substituting $\gamma$ from (A56c) in (A56d) yields $\tau =\frac{\alpha }{\sqrt{\delta }}$ or $\gamma =\frac{\alpha }{\lambda \sqrt{\delta }}$, which together with (A56b) leads to (A57c). Finally, (A57b) can be obtained by substituting $\gamma =\frac{\alpha }{\lambda \sqrt{\delta }}$ in (A56c) and using the fact that (see Proposition A1):

$${\mathcal{M}}_{\ell ,2}^{\prime }\left(\alpha G+\mu SY;\lambda \right)=-\frac{1}{2}{\left({\mathcal{M}}_{\ell ,1}^{\prime }\left(\alpha G+\mu SY;\lambda \right)\right)}^2.$$

Appendix B.5. On the Uniqueness of Solutions to (A57): Proof of Proposition 1

Here, we prove the claim of Proposition 1 through the following lemmas. As discussed in Remark 4, the main part of the proof is showing strict convex-concavity of F in (11). Lemma A6 proves that this is the case, and Lemmas A7 and A8 show that this is sufficient for the uniqueness of solutions to (A57). When put together, these complete the proof of Proposition 1.

Lemma A6.

(Strict Convex-Concavity of (A55)). Let $\ell :\mathbb{R}\to \mathbb{R}$ be proper and strictly convex function. Further, assume that ℓ is continuously differentiable with ${\ell }^{\prime }\left(0\right)\ne 0$. In addition, assume that $SY$ has positive density in the real line. Then, the function $F:{\mathbb{R}}^4\to \mathbb{R}$ defined in (A55) is strictly convex in $(\alpha ,\mu ,\tau )$ and strictly concave in γ.

Proof. 

The claim follows directly from the strict convexity-concavity properties of the expected Moreau-envelope proved in Propositions A5 and A6. Specifically, we apply Proposition A7. □

Lemma A7.

If the objective function in (A55) is strictly convex in $(\alpha ,\mu ,\tau )$ and strictly concave in γ, then (A56) has a unique solution $(\alpha ,\mu ,\tau ,\gamma )$.

Proof. 

Let $({\alpha }_i,{\mu }_i,{\tau }_i,{\gamma }_i),i=1,2,$ be two different saddle points of (A55). For convenience, let ${\mathbf{x}}_i:=({\alpha }_i,{\mu }_i,{\tau }_i)$ for $i=1,2$. By strict-concavity in $\gamma$, for fixed values of $\mathbf{x}:=(\alpha ,\mu ,\tau )$, the value of $\gamma$ maximizing $F(\mathbf{x},\gamma )$ is unique. Thus, if ${\mathbf{x}}_1={\mathbf{x}}_2$, then it must hold that ${\gamma }_1={\gamma }_2$, which is a contraction to our assumption of $({\mathbf{x}}_1,{\gamma }_1)\ne ({\mathbf{x}}_2,{\gamma }_2)$. Similarly, we can use strict-convexity to derive that ${\gamma }_1\ne {\gamma }_2$. Then, based on the definition of the saddle point and strict convexity-concavity, the following two relations hold for $i=1,2$:

$$\begin{array}{c}\hfill F({\mathbf{x}}_i,\gamma )<F({\mathbf{x}}_i,{\gamma }_i)<F(\mathbf{x},{\gamma }_i),\mathrm{for}\mathrm{all}\mathbf{x}\ne {\mathbf{x}}_i,\gamma \ne {\gamma }_i.\end{array}$$

We choose $\mathbf{x}={\mathbf{x}}_2,\gamma ={\gamma }_2$ for $i=1$ and $\mathbf{x}={\mathbf{x}}_1,\gamma ={\gamma }_1$ for $i=2$ to find

$$\begin{array}{c}\hfill F({\mathbf{x}}_1,{\gamma }_2)<F({\mathbf{x}}_1,{\gamma }_1)<F({\mathbf{x}}_2,{\gamma }_1),\\ \hfill F({\mathbf{x}}_2,{\gamma }_1)<F({\mathbf{x}}_2,{\gamma }_2)<F({\mathbf{x}}_1,{\gamma }_2).\end{array}$$

From the above, it follows that $F({\mathbf{x}}_1,{\gamma }_1)<F({\mathbf{x}}_2,{\gamma }_2)$ and $F({\mathbf{x}}_1,{\gamma }_1)>F({\mathbf{x}}_2,{\gamma }_2)$, which is a contradiction. This completes the proof. □

Lemma A8.

If (A56) has a unique solution $({\alpha }^{☆},{\mu }^{☆},{\tau }^{☆},{\gamma }^{☆})$, then (A57) has a unique solution $({\alpha }^{☆},{\mu }^{☆},{\lambda }^{☆})$.

Proof. 

First, following the same approach of deriving Equations (A57) from (A56) in (A56), it is easy to see that existence of solution $({\alpha }_1,{\mu }_1,{\tau }_1,{\gamma }_1)$ to (A57) implies existence of solution $({\alpha }_1,{\mu }_1,{\lambda }_1:=\frac{{\tau }_1}{{\gamma }_1})$ to (A57). Now, for the sake of contradiction to the statement of the lemma, assume that there are two different triplets ${\mathbf{v}}_1:=({\alpha }_1,{\mu }_1,{\lambda }_1)$ and ${\mathbf{v}}_2:=({\alpha }_2,{\mu }_2,{\lambda }_2)$ with ${\alpha }_1,{\alpha }_2,{\lambda }_1,{\lambda }_2>0$ and satisfying (Appendix B.4). Then, we can show that both ${\mathbf{w}}_i:=({\alpha }_i,{\mu }_i,{\tau }_i,{\gamma }_i)i=1,2,$ such that:

$$\begin{array}{c}\hfill {\tau }_i:=\frac{{\alpha }_i}{\sqrt{\delta }},{\gamma }_i=\frac{{\alpha }_i}{{\lambda }_i\sqrt{\delta }},i=1,2,\end{array}$$

satisfy the system of equations in (A56). However, since ${\mathbf{v}}_1\ne {\mathbf{v}}_2$, it must be that ${\mathbf{w}}_1\ne {\mathbf{w}}_2$. This contradicts the assumption of uniqueness of solutions to (A56) and completes the proof. □

Appendix C.1. Distribution of SY in Special Cases

We derive the following densities for $SY$ for the special cases ($\parallel {\mathbf{x}}_0{\parallel }_2=1$):

• Signed: $p_{SY}\left(w\right)=\sqrt{\frac{2}{\pi }}exp(-w^2/2){\mathbb{1}}_{\{w\ge 0\}}.$
• Logistic: $p_{SY}\left(w\right)=\sqrt{\frac{2}{\pi }}\frac{exp\left(-w^2/2\right)}{1+exp(-w)}.$
• Probit: $p_{SY}\left(w\right)=\sqrt{\frac{2}{\pi }}\Phi \left(w\right)exp\left(-w^2/2\right).$

In particular, we numerically observe that for logistic and Probit models; the resulting densities are similar to the density of a gaussian distribution derived according to $\mathcal{N}\left(\mathbb{E}\right[SY],\mathrm{Var}[SY\left]\right)$. Figure A2 illustrates this similarity for these two models. As discussed in Corollary 1, this similarity results in the tightness of the lower bound achieved for ${\sigma }_{\mathrm{opt}}$ in Equation (23).

Figure A1. The value of $\kappa \left(\sigma \right)$ as in Theorem 2 for various measurement models. Since $\kappa \left(\sigma \right)$ is a monotonic function of $\sigma$, the solution to $\kappa \left(\sigma \right)=1/\delta$ determines the minimum possible value of $\sigma$.

Figure A2. Probability distribution function of $SY$ for the logistic and Probit models ($\parallel {\mathbf{x}}_0{\parallel }_2=1$) compared with the probability distribution function of the Gaussian random variable (dashed lines) with the same mean and variance i.e., $\mathcal{N}\left(\mathbb{E}\right[SY],\mathrm{Var}[SY\left]\right)$.

Appendix D.1. Proof of Theorem 3

We show that the triplet $(\mu =1,\alpha ={\sigma }_{\mathrm{opt}},\lambda =1)$ is a solution to Equations (8) for ℓ chosen as in (29). Using Proposition A2 in the Appendix, we rewrite ${\ell }_{\mathrm{opt}}$ using the Fenchel–Legendre conjugate as follows:

$$\begin{array}{c}\hfill {\ell }_{\mathrm{opt}}\left(w\right)={\left(q+{\alpha }_{1}q+{\alpha }_{2}log{p}_{W_{\mathrm{opt}}}\right)}^{☆}\left(w\right)-q\left(w\right),\end{array}$$

(A58)

where $q\left(w\right)=w^2/2$. For a function f, its Fenchel–Legendre conjugate is defined as:

$$f^{☆}\left(x\right)=\underset{y}{max}xy-f\left(y\right).$$

Next, we use the fact that, for any proper, closed and convex function f, it holds that ${\left(f^{☆}\right)}^{☆}=f$ [61] (theorem 12.2). Therefore, noting that $q+{\alpha }_{1}q+{\alpha }_{2}log{p}_{W_{\mathrm{opt}}}$ is a convex function (see the proof of Lemma A9 in the Appendix), combined with (A58), it yields that

$$\begin{array}{c}\hfill {({\ell }_{\mathrm{opt}}+q)}^{☆}=q+{\alpha }_{1}q+{\alpha }_{2}log{p}_{W_{\mathrm{opt}}}.\end{array}$$

(A59)

Additionally, using Proposition A2, we find that ${\mathcal{M}}_{{\ell }_{\mathrm{opt}}}\left(w;1\right)=q\left(w\right)-{(q+{\ell }_{\mathrm{opt}})}^{☆}\left(w\right),$ which by (A49) reduces to:

$${\mathcal{M}}_{{\ell }_{\mathrm{opt}}}\left(w;1\right)=-{\alpha }_{1}q\left(w\right)-{\alpha }_{2}log{p}_{W_{\mathrm{opt}}}\left(w\right).$$

Thus, by differentiation, we find that ${\ell }_{\mathrm{opt}}$ satisfies (28) with $c=-1$, i.e.,

$$\begin{array}{c}\hfill {\mathcal{M}}_{{\ell }_{\mathrm{opt}},1}^{\prime }\left(w;1\right)=-{\alpha }_{1}w-{\alpha }_2·{\xi }_{W_{\mathrm{opt}}}\left(w\right).\end{array}$$

(A60)

Next, we establish the desired by directly substituting (A60) into the system of equations in (19). First, using the values of ${\alpha }_1$ and ${\alpha }_2$ in (30), as well as the fact that $\kappa \left({\sigma }_{\mathrm{opt}}\right)=1/\delta$, we have the following chain of equations:

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\left({\mathcal{M}}_{{\ell }_{\mathrm{opt}},1}^{\prime }\left(W_{\mathrm{opt}};1\right)\right)}^2\right]& =\mathbb{E}\left[{({\alpha }_1{W}_{\mathrm{opt}}+{\alpha }_2{\xi }_{W_{\mathrm{opt}}}\left(W_{\mathrm{opt}}\right))}^2\right]\hfill \\ \hfill & ={\alpha }_1^2({\sigma }_{\mathrm{opt}}^2+1)+{\alpha }_2^2\mathcal{I}\left(W_{\mathrm{opt}}\right)+2{\alpha }_1{\alpha }_2\mathbb{E}\left[W_{\mathrm{opt}}·{\xi }_{W_{\mathrm{opt}}}\left(W_{\mathrm{opt}}\right)\right]\hfill \\ \hfill & =\frac{1+{\sigma }_{\mathrm{opt}}^2\left({\sigma }_{\mathrm{opt}}^2\mathcal{I}\left(W_{\mathrm{opt}}\right)-1\right)}{{\delta }^2\left({\sigma }_{\mathrm{opt}}^2\mathcal{I}\left(W_{\mathrm{opt}}\right)+\mathcal{I}\left(W_{\mathrm{opt}}\right)-1\right)}=\frac{{\sigma }_{\mathrm{opt}}^2}{{\delta }^2\kappa \left({\sigma }_{\mathrm{opt}}\right)}\hfill \\ \hfill & ={\sigma }_{\mathrm{opt}}^2/\delta .\hfill \end{array}$$

(A61)

This shows (8b). Second, using again the specified values of ${\alpha }_1$ and ${\alpha }_2$, a similar calculation yields

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\mathcal{M}}_{{\ell }_{\mathrm{opt}},1}^{\prime }\left(W_{\mathrm{opt}};1\right){\xi }_{W_{\mathrm{opt}}}\left(W_{\mathrm{opt}}\right)\right]& =-\mathbb{E}\left[\left({\alpha }_1{W}_{\mathrm{opt}}+{\alpha }_2{\xi }_{W_{\mathrm{opt}}}\left(W_{\mathrm{opt}}\right)\right){\xi }_{W_{\mathrm{opt}}}\left(W_{\mathrm{opt}}\right)\right]\hfill \\ \hfill & ={\alpha }_1-{\alpha }_2\mathcal{I}\left(W_{\mathrm{opt}}\right)\hfill \\ \hfill & =-1/\delta .\hfill \end{array}$$

(A62)

Recall from (17) that $\mathbb{E}\left[G·{\mathcal{M}}_{{\ell }_{\mathrm{opt}},1}^{\prime }\left(W_{\mathrm{opt}};1\right)\right]=-{\sigma }_{\mathrm{opt}}\mathbb{E}\left[{\mathcal{M}}_{{\ell }_{\mathrm{opt}},1}^{\prime }\left(W_{\mathrm{opt}};1\right){\xi }_{W_{\mathrm{opt}}}\left(W_{\mathrm{opt}}\right)\right].$ This combined with (A62) yields (8c). Finally, we use again (A60) and the specified values of ${\alpha }_1$ and ${\alpha }_2$ to find that

$$\begin{array}{cc}\hfill \mathbb{E}\left[W_{\mathrm{opt}}·{\mathcal{M}}_{{\ell }_{\mathrm{opt}},1}^{\prime }\left(W_{\mathrm{opt}};1\right)\right]& =\mathbb{E}\left[W_{\mathrm{opt}}·(-{\alpha }_1{W}_{\mathrm{opt}}-{\alpha }_2{\xi }_{W_{\mathrm{opt}}}\left(W_{\mathrm{opt}}\right))\right]\hfill \\ & =-{\alpha }_1\mathbb{E}\left[W_{\mathrm{opt}}^2\right]-{\alpha }_2\mathbb{E}\left[W_{\mathrm{opt}}{\xi }_{W_{\mathrm{opt}}}\left(W_{\mathrm{opt}}\right)\right]\hfill \\ \hfill & =-{\alpha }_1({\sigma }_{\mathrm{opt}}^2+1)-{\alpha }_2{\int }_{-\infty }^{\infty }w{p}_{W_{\mathrm{opt}}}^{\prime }\left(w\right)\mathrm{d}w\hfill \\ & =-{\alpha }_1({\sigma }_{\mathrm{opt}}^2+1)+{\alpha }_2\hfill \\ & ={\sigma }_{\mathrm{opt}}^2/\delta .\hfill \end{array}$$

(A63)

However, using (17), it holds that

$$\begin{array}{cc}\hfill \mathbb{E}& \left[W_{\mathrm{opt}}·{\mathcal{M}}_{{\ell }_{\mathrm{opt}},1}^{\prime }\left(W_{\mathrm{opt}};1\right)\right]=\hfill \\ & -{\sigma }_{\mathrm{opt}}^2\mathbb{E}\left[{\mathcal{M}}_{{\ell }_{\mathrm{opt}},1}^{\prime }\left(W_{\mathrm{opt}};1\right){\xi }_{W_{\mathrm{opt}}}\left(W_{\mathrm{opt}}\right)\right]+\mathbb{E}\left[YS·{\mathcal{M}}_{{\ell }_{\mathrm{opt}},1}^{\prime }\left(W_{\mathrm{opt}};\lambda \right)\right].\hfill \end{array}$$

This combined with (A63) and (A62) shows that $\mathbb{E}\left[YS·{\mathcal{M}}_{{\ell }_{\mathrm{opt}},1}^{\prime }\left(W_{\mathrm{opt}};\lambda \right)\right]=0$, as desired to satisfy (8a). This completes the proof of the theorem.

Appendix D.2. On the Convexity of Optimal Loss Function

Here, we provide a sufficient condition for ${\ell }_{\mathrm{opt}}\left(w\right)$ to be convex.

Lemma A9.

The optimal loss function as defined in Theorem 3 is convex if

$${(log\left(p_{W_{\sigma }}\right))}^{\prime \prime }\left(w\right)\le -\frac{1}{{\sigma }^2+1},\mathrm{for}\mathrm{all}w\in \mathbb{R}\mathrm{and}\sigma \ge 0.$$

Proof. 

Using (A9) optimal loss function is written in the following form

$$\begin{array}{c}\hfill {\ell }_{\mathrm{opt}}\left(w\right)={\left(q+{\alpha }_{1}q+{\alpha }_{2}log\left(p_{W_{\mathrm{opt}}}\right)\right)}^{☆}\left(w\right)-q\left(w\right).\end{array}$$

(A64)

Next, we prove that $q+{\alpha }_{1}q+{\alpha }_{2}log\left(p_{W_{\mathrm{opt}}}\right)$ is a convex function. We first show that both ${\alpha }_1$ and ${\alpha }_2$ are positive numbers for all values of ${\sigma }_{\mathrm{opt}}$. We first note that, since G and $SY$ are independent random variables, ${\sigma }_{\mathrm{opt}}^2\mathcal{I}\left(W_{\mathrm{opt}}\right)<{\sigma }_{\mathrm{opt}}^2\mathcal{I}\left({\sigma }_{\mathrm{opt}}G\right)=1$. Therefore,

$$\begin{array}{c}\hfill 1-{\sigma }_{\mathrm{opt}}^2\mathcal{I}\left(W_{\mathrm{opt}}\right)>0.\end{array}$$

(A65)

Additionally, following the Cramer–Rao bound [53] for Fisher information yields that:

$$\begin{array}{cc}\hfill \mathcal{I}\left(W_{\mathrm{opt}}\right)& >\frac{1}{\mathbb{E}\left[{(W_{\mathrm{opt}}-\mathbb{E}\left[W_{\mathrm{opt}}\right])}^2\right]}\hfill \\ \hfill & =\frac{1}{1+{\sigma }_{\mathrm{opt}}^2-{\left(\mathbb{E}\left[SY\right]\right)}^2}.\hfill \end{array}$$

Using this inequality for $\mathcal{I}\left(W_{\mathrm{opt}}\right)$, we derive that

$$\begin{array}{c}\hfill {\sigma }_{\mathrm{opt}}^2\mathcal{I}\left(W_{\mathrm{opt}}\right)+\mathcal{I}\left(W_{\mathrm{opt}}\right)-1>0.\end{array}$$

(A66)

From (A65) and (A66), it follows that ${\alpha }_1,{\alpha }_2>0$.

Based on the definition of the random variable $W_{\mathrm{opt}}$:

$$\begin{array}{c}\hfill log{p}_{W_{\mathrm{opt}}}\left(w\right)=-w^2/\left(2{\sigma }_{\mathrm{opt}}^2\right)+log{\int }_{-\infty }^{\infty }exp\left((2wz-z^2)/2{\sigma }_{\mathrm{opt}}^2\right)p_{SY}\left(z\right)\mathrm{d}z+c,\end{array}$$

where c is a constant independent of w. By differentiating twice, we see that

$$log{\int }_{-\infty }^{\infty }exp\left((2wz-z^2)/2{\sigma }_{\mathrm{opt}}^2\right)p_{SY}\left(z\right)\mathrm{d}z$$

is a convex function of w. Therefore, to prove that $q+{\alpha }_{1}q+{\alpha }_{2}log\left(p_{W_{\mathrm{opt}}}\right)$ is a convex function, it is sufficient to prove that $(1+{\alpha }_1-{\alpha }_2/{\sigma }_{\mathrm{opt}}^2)q$ is a convex function or equivalently $1+{\alpha }_1-{\alpha }_2/{\sigma }_{\mathrm{opt}}^2\ge 0$. Replacing values of ${\alpha }_1,{\alpha }_2$ and recalling the equation for ${\sigma }_{\mathrm{opt}}$ yields that

$$1+{\alpha }_1-{\alpha }_2/{\sigma }_{\mathrm{opt}}^2=0,$$

which implies the convexity of $q+{\alpha }_{1}q+{\alpha }_{2}log\left(p_{W_{\mathrm{opt}}}\right)$. To obtain the derivative of ${\ell }_{opt}$, we use the result in [61] (Cor. 23.5.1), which states that, for a convex function f,

$${\left(f^{☆}\right)}^{\prime }={\left(f^{\prime }\right)}^{-1}.$$

Therefore, following (A64),

$$\begin{array}{c}\hfill {\ell }_{\mathrm{opt}}^{\prime }\left(w\right)={(q^{\prime }+{\alpha }_1{q}^{\prime }+{\alpha }_2{(log\left(p_{W_{\mathrm{opt}}}\right))}^{\prime })}^{-1}\left(w\right)-w.\end{array}$$

(A67)

Differentiating again and using the properties of inverse function yields that

$$\begin{array}{c}\hfill {\ell }_{\mathrm{opt}}^{\prime \prime }\left(w\right)=\frac{1}{1+{\alpha }_1+{\alpha }_2{(log\left(p_{W_{\mathrm{opt}}}\right))}^{\prime \prime }\left(g\left(w\right)\right)}-1,\end{array}$$

(A68)

where

$$g\left(w\right):={(q^{\prime }+{\alpha }_1{q}^{\prime }+{\alpha }_2{(log\left(p_{W_{\mathrm{opt}}}\right))}^{\prime })}^{-1}\left(w\right).$$

Note that the denominator of (A68) is nonnegative since it is second derivative of a convex function. Therefore, it is evident from (A68) that a sufficient condition for the convexity of ${\ell }_{\mathrm{opt}}$ is that

$${\alpha }_1+{\alpha }_2{(log\left(p_{W_{\mathrm{opt}}}\right))}^{\prime \prime }\left(w\right)\le 0,\mathrm{for}\mathrm{all}w\in \mathbb{R},$$

or

$$1-{\sigma }_{\mathrm{opt}}^2\mathcal{I}\left(W_{\mathrm{opt}}\right)+{(log\left(p_{W_{\mathrm{opt}}}\right))}^{\prime \prime }\left(w\right)\le 0.$$

This condition is satisfied if the statement of the lemma holds for $\sigma ={\sigma }_{\mathrm{opt}}$:

$$\begin{array}{cc}\hfill & 1-{\sigma }_{\mathrm{opt}}^2\mathcal{I}\left(W_{\mathrm{opt}}\right)+{(log\left(p_{W_{\mathrm{opt}}}\right))}^{\prime \prime }\left(w\right)\le 1-{\sigma }_{\mathrm{opt}}^2\mathcal{I}\left(W_{\mathrm{opt}}\right)-\frac{1}{1+{\sigma }_{\mathrm{opt}}^2}<0,\hfill \end{array}$$

where we use (A66) in the last inequality. This concludes the proof. □

Appendix D.2.1. Provable Convexity of the Optimal Loss Function for Signed Model

In the case of signed model, it can be proved that the conditions of Lemma A9 is satisfied. Since $W_{\sigma }=\sigma G+SY$, we derive the probability density of $W_{\sigma }$ as follows:

$$\begin{array}{c}\hfill p_{W_{\sigma }}\left(w\right)=p_{{}_{\sigma G}}\left(w\right)*p_{SY}\left(w\right)=\frac{\exp (-w^2/(2+2{\sigma }^2))}{\sqrt{2\pi (1+{\sigma }^2)}}·f\left(w\right),\end{array}$$

where

$$f\left(w\right)=2-2\mathrm{Q}(w/\left(\sigma \sqrt{2+2{\sigma }^2}\right)).$$

Direct calculation shows that f is a log-concave function for all $w\in \mathbb{R}$. Therefore,

$$\begin{array}{cc}\hfill {(log\left(p_{W_{\sigma }}\right))}^{\prime \prime }\left(w\right)& =-\frac{1}{{\sigma }^2+1}+{(log\left(f\right))}^{\prime \prime }\left(w\right)\hfill \\ \hfill & \le -\frac{1}{{\sigma }^2+1}.\hfill \end{array}$$

This proves the convexity of optimal loss function derived according to Theorem 3 when measurements follow the signed model.

Appendix F.1. Proof of Corollary 2

To get the values of $\alpha$ and $\mu$ as in the statement of the corollary, we show how to simplify Equations (8) for $\ell \left(t\right)={(t-1)}^2$. In this case, the proximal operator admits a simple expression:

$${\mathrm{prox}}_{\ell }\left(x;\lambda \right)=\left(x+2\lambda \right)/\left(1+2\lambda \right).$$

In addition, ${\ell }^{\prime }\left(t\right)=2(t-1)$. Substituting these in (14a) gives the formula for $\mu$ as follows:

$$\begin{array}{cc}\hfill 0& =\mathbb{E}\left[YS(\alpha G+\mu SY-1)\right]=\mu \mathbb{E}\left[S^2\right]-\mathbb{E}\left[YS\right]\hfill \\ & ⟹\mu =\mathbb{E}\left[YS\right],\hfill \end{array}$$

where we have also used from (7) that $\mathbb{E}\left[S^2\right]=1$ and G is independent of S. In addition, since ${\ell }^{\prime \prime }\left(t\right)=2$, direct application of (A7) gives

$$\begin{array}{c}\hfill 1=\lambda \delta \frac{2}{1+2\lambda }⟹\lambda =\frac{1}{2(\delta -1)}.\end{array}$$

Finally, substituting the value of $\lambda$ into (14b), we obtain the desired value for $\alpha$ as follows:

$$\begin{array}{cc}\hfill {\alpha }^2& =4{\lambda }^2\delta \mathbb{E}\left[{({\mathrm{prox}}_{\ell }\left(\alpha G+\mu SY;\lambda \right)-1)}^2\right]\hfill \\ \hfill & =\frac{4{\lambda }^2}{{(1+2\lambda )}^2}\delta \mathbb{E}\left[{(\alpha G+\mu SY-1)}^2\right]\hfill \\ \hfill & =\frac{4{\lambda }^2\delta }{{(1+2\lambda )}^2}({\alpha }^2+{\mu }^2+1-2\mu \mathbb{E}\left[SY\right])\hfill \\ \hfill & =\frac{1}{\delta }({\alpha }^2+1-{\left(\mathbb{E}\left[SY\right]\right)}^2)\hfill \\ \hfill & ⟹\alpha =\sqrt{1-{\left(\mathbb{E}\left[SY\right]\right)}^2}·\sqrt{\frac{1}{\delta -1}}.\hfill \end{array}$$

Appendix F.2. Discussion

Linear vs. Binary

On the one hand, Corollary 2 shows that least-squares performance for binary measurements satisfies

$$\underset{n\to \infty }{lim}\parallel \widehat{\mathbf{x}}-\frac{\mu }{\parallel {\mathbf{x}}_0{\parallel }_2}·{\mathbf{x}}_0{\parallel }_2^2={\tau }^2·\frac{1}{\delta -1},$$

(A70)

where $\mu$ is as in (32) and ${\tau }^2:=1-{\left(\mathbb{E}\left[SY\right]\right)}^2$. On the other hand, it is well-known (e.g., see references in [15] (Sec. 5.1)) that least-squares for (scaled) linear measurements with additive Gaussian noise (i.e., $y_i=\rho {\mathbf{a}}_i^T{\mathbf{x}}_0+\sigma z_i$, $z_i\sim \mathcal{N}(0,1)$) leads to an estimator that satisfies

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{∥\widehat{\mathbf{x}}-\rho ·{\mathbf{x}}_0∥}_2^2={\sigma }^2·\frac{1}{\delta -1}.\end{array}$$

(A71)

Direct comparison of (A70) to (A71) suggests that least-squares with binary measurements performs the same as if measurements were linear with scaling factor $\rho =\mu /\parallel {\mathbf{x}}_0{\parallel }_2$ and noise variance ${\sigma }^2={\tau }^2={\alpha }^2(\delta -1)$. This worth-mentioning conclusion is not new, as it is proved in [40,43,56,62]. We include a short discussion on the relation to this prior work in the following paragraph. We highlight that all these existing results are limited to a least-squares loss unlike our general analysis.

Prior work. There is a lot of recent work on the use of least-squares-type estimators for recovering signals from nonlinear measurements of the form $y_i=h\left({\mathbf{a}}_i^T{\mathbf{x}}_0\right)$ with Gaussian vectors ${\mathbf{a}}_i$. The original work that suggests least-squares as a reasonable estimator in this setting is due to Brillinger [56]. In his 1982 paper, Brillinger studied the problem in the classical statistics regime (namely, n is fixed not scaling with $m\to +\infty$) and he proved for the least-squares solution satisfies

$$\underset{m\to +\infty }{lim}\frac{1}{m}{∥\widehat{\mathbf{x}}-\frac{\mu }{\parallel {\mathbf{x}}_0{\parallel }_2}·{\mathbf{x}}_0∥}_2^2={\tau }^2,$$

where

$$\begin{array}{cc}\hfill \mu & =\mathbb{E}\left[SY\right],S\sim \mathcal{N}(0,1),\hfill \\ \hfill {\tau }^2& =\mathbb{E}\left[{(Y-\mu S)}^2\right].\hfill \end{array}$$

(A72)

and the expectations are with respect to S and possible randomness of f. Evaluating (A72) for $Y=f_{\epsilon }\left(S\right)$ leads to the same values for $\mu$ and ${\tau }^2$ in (A70). In other works, (A70) for $\delta \to +\infty$ indeed recovers Brillinger’s result. The extension of Brillinger’s original work to the high-dimensional setting (both $m,n$ large) was first studied by Plan and Vershynin [40], who derived (non-sharp) non-asymptotic upper bounds on the performance of constrained least-squares (such as the Lasso). Shortly after, Thrampoulidis et al. [43] extended this result to sharp asymtpotic predictions and to regularized least-squares. In particular, Corollary 2 is a special case of the main theorem in [43]. Several other interesting extensions of the result by Plan and Vershynin have recently appeared in the literature (e.g., [41,62,63,64]). However, the one in [43] is the only one to give results that are sharp in the flavor of this paper. Our work, extends the result of Thrampoulidis et al. [43] to general loss functions beyond least-squares. The techniques of Thrampoulidis et al. [43] that have guided the use of the CGMT in our context were also recently applied by Dhifallah et al. [60] in the context of phase-retrieval.

Appendix G.1. Proof of Corollary 3

The proof follows directly by noting that, when $\ell \left(t\right)={(t-1)}^2$, it holds that ${\mathcal{M}}_{\ell }\left(x;\lambda \right)=\frac{{(x-1)}^2}{2\lambda +1}.$ By inserting this into (38a) and simplifying the equations, we find the value of $\mu$:

$$\mu =\frac{r}{1+r^2}.$$

Similarly, we derive $\lambda$ using Equation (38c):

$$\lambda =\frac{1}{2(\delta -1)}.$$

Substituting these values of $\mu$ and $\lambda$ into (38b) yields that

$${\alpha }^2=\frac{1}{\delta -1}·\frac{1}{r^2+1}.$$

Recalling that ${\sigma }_{LS}=\alpha /\mu$ concludes the proof.

Appendix G.2. Proof of Theorem 5

The high-level steps of the proof follow the proof of Theorem 2. First, we note that by scaling the loss function ℓ the value of ${\sigma }_{\ell }$ does not change. In particular, if $\tilde{\ell }\left(t\right):=C_1\ell \left(C_{2}t\right)$ for arbitrary constants $C_1>0,C_2\ne 0$, it is not hard to see that ${\widehat{\mathbf{x}}}_{\tilde{\ell }}=1/C_2{\widehat{\mathbf{x}}}_{\ell }$ is the minimizer of (3). Thus, we conclude from (40) that ${\sigma }_{\tilde{\ell }}={\sigma }_{\ell }$. With this observation, consider the function $\tilde{\ell }:\mathbb{R}\to \mathbb{R}$ such that $\tilde{\ell }\left(t\right)=\frac{\lambda }{{\mu }^2}\ell \left(\mu t\right)$. Then, notice that

$${\mathcal{M}}_{\ell ,1}^{\prime }\left(x;\lambda \right)=\frac{1}{\lambda }{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(x/\mu ;1\right).$$

Using this relation in (38) and setting $\sigma :={\sigma }_{\ell }=\alpha /\mu$, the system of equations in (38) can be equivalently rewritten in the following convenient form, where $Z_{\sigma }=\sigma W_1+W_2$:

$$\begin{array}{cc}& \mathbb{E}\left[W_2·{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right)\right]=0,\hfill \end{array}$$

(A73a)

$$\begin{array}{cc}& \mathbb{E}\left[{\left({\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right)\right)}^2\right]={\sigma }^2/\delta ,\hfill \end{array}$$

(A73b)

$$\begin{array}{cc}& \mathbb{E}\left[W_1·{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right)\right]=\sigma /\delta .\hfill \end{array}$$

(A73c)

Next, we show how to use (A73) to derive an equivalent system of equations in terms of only $Z_{\sigma }$. Starting with (A73c), we have

$$\mathbb{E}\left[W_1·{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right)\right]=\frac{1}{\sigma }\int \int x{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(x+y;1\right)p_{\sigma W_1}\left(x\right)p_{W_2}\left(y\right)\mathrm{d}x\mathrm{d}y,$$

(A74)

where recall that $p_{\sigma W_1}\left(x\right)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{x^2}{2{\sigma }^2}}$. Since it holds that $p_{\sigma W_1}\left(x\right)=\frac{-{\sigma }^2}{x}{p}_{\sigma W_1}^{\prime }\left(x\right)$, using (A74) yields that

$$\begin{array}{cc}\hfill \mathbb{E}\left[W_1·{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right)\right]& =-\sigma \int \int {\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(x+y;1\right)p_{\sigma W_1}^{\prime }\left(x\right)p_{W_2}\left(y\right)\mathrm{d}x\mathrm{d}y\hfill \\ & =-\sigma \int \int {\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(z;1\right)p_{\sigma W_1}^{\prime }\left(x\right)p_{W_2}(z-x)\mathrm{d}x\mathrm{d}z=-\sigma \int {\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(z;1\right)p_{Z_{\sigma }}^{\prime }\left(z\right)\mathrm{d}z,\hfill \end{array}$$

where in the last step we use

$$p_{Z_{\sigma }}^{\prime }\left(w\right)=\int p_{\sigma W_1}^{\prime }\left(x\right)p_{W_2}(z-x)\mathrm{d}x.$$

Therefore,

$$\begin{array}{c}\hfill \mathbb{E}\left[W_1·{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right)\right]=-\sigma \mathbb{E}\left[{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right){\xi }_{Z_{\sigma }}\left(Z_{\sigma }\right)\right].\end{array}$$

This combined with (A73c) gives

$$\mathbb{E}\left[{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right){\xi }_{Z_{\sigma }}\left(Z_{\sigma }\right)\right]=-1/\delta .$$

Second, multiplying (A73c) with ${\sigma }^2$ and adding it to (A73a) yields

$$\begin{array}{cc}\hfill \mathbb{E}\left[Z_{\sigma }·{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right)\right]& ={\sigma }^2/\delta .\hfill \end{array}$$

(A75)

Putting these together, we conclude with the following system of equations which is equivalent to (A73),

$$\begin{array}{cc}& \mathbb{E}\left[Z_{\sigma }·{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right)\right]={\sigma }^2/\delta ,\hfill \end{array}$$

(A76a)

$$\begin{array}{cc}& \mathbb{E}\left[{\left({\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right)\right)}^2\right]={\sigma }^2/\delta ,\hfill \end{array}$$

(A76b)

$$\begin{array}{cc}& \mathbb{E}\left[{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right){\xi }_{Z_{\sigma }}\left(Z_{\sigma }\right)\right]=-1/\delta .\hfill \end{array}$$

(A76c)

Next, considering (A76a) and (A76c), the following holds for any $c_1,c_2\in \mathbb{R}$,

$$\mathbb{E}\left[\left(c_1{Z}_{\sigma }+c_2{\xi }_{Z_{\sigma }}\left(Z_{\sigma }\right)\right)·{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right)\right]=c_1{\sigma }^2/\delta -c_2/\delta .$$

(A77)

Applying Cauchy–Schwarz inequality to the LHS of (A77) gives

$$\begin{array}{cc}\hfill & {\left(c_1{\sigma }^2/\delta -c_2/\delta \right)}^2={\left(\mathbb{E}\left[\left(c_1{Z}_{\sigma }+c_2{\xi }_{Z_{\sigma }}\left(Z_{\sigma }\right)\right)·{\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right)\right]\right)}^2\hfill \end{array}$$

$$\begin{array}{c}\hfill \le \mathbb{E}\left[{\left(c_1{Z}_{\sigma }+c_2{\xi }_{Z_{\sigma }}\left(Z_{\sigma }\right)\right)}^2\right]\mathbb{E}\left[{\left({\mathcal{M}}_{\tilde{\ell },1}^{\prime }\left(Z_{\sigma };1\right)\right)}^2\right].\end{array}$$

(A78)

By considering (A76b), $\mathbb{E}\left[Z_{\sigma }{\xi }_{Z_{\sigma }}\left(Z_{\sigma }\right)\right]=-1$ (follows from integration by parts) and $\mathbb{E}\left[{\left({\xi }_{Z_{\sigma }}\left(Z_{\sigma }\right)\right)}^2\right]=\mathcal{I}\left(Z_{\sigma }\right)={({\sigma }^2+1)}^{-1}$, we simplify (A78) to the following:

$${(c_1{\sigma }^2/\delta -c_2/\delta )}^2\le \left(c_1^2({\sigma }^2+1+r^2)+c_2^2/({\sigma }^2+1)-2c_1{c}_2)\right){\sigma }^2/\delta .$$

Choosing $c_1=1$ and $c_2=(1+r^2)(1+{\sigma }^2)$ and simplifying both sides, we derive the lower bound for ${\sigma }^2$:

$${\sigma }^2\ge \frac{1+r^2}{r^2}·\frac{1}{(\delta -1)}.$$

This completes the proof of theorem.