On a Low-Rank Matrix Single-Index Model

Abstract

In this paper, we conduct a theoretical examination of a low-rank matrix single-index model. This model has recently been introduced in the field of biostatistics, but its theoretical properties for jointly estimating the link function and the coefficient matrix have not yet been fully explored. In this paper, we make use of the PAC-Bayesian bounds technique to provide a thorough theoretical understanding of the joint estimation of the link function and the coefficient matrix. This allows us to gain a deeper insight into the properties of this model and its potential applications in different fields.

1. Introduction

In this study, we investigate a particular type of single-index model, where the response variable, denoted by Y, is a real number and the covariate matrix, represented by X, is a matrix of real numbers with dimensions of $d\times d$. The model is defined in Equation (1) as

$$Y=f^{\star }\left(⟨X,B^{\star }⟩\right)+ϵ$$

(1)

In this equation, $⟨X,B^{\star }⟩=\mathrm{trace}\left(X^{\top }{B}^{\star }\right)$ represents the inner product between matrices X and $B^{\star }$, where $B^{\star }$ is an unknown coefficient matrix with dimensions of $d\times d$. The link function $f^{\star }$ is an unknown univariate measurable function. The noise term, represented by $ϵ$, is assumed to have a mean of 0 and is independent of the covariate X.

In line with the recent research presented in [1,2], we make the assumption that the coefficient matrix $B^{\star }$ is a symmetric, low-rank matrix with $\mathrm{rank}\left(B^{\star }\right)<d$. Additionally, in order to ensure the uniqueness of the model, we impose the condition that the Frobenius norm of $B^{\star }$ is equal to 1, i.e., $\parallel B^{\star }{\parallel }_F=1$.

Previous studies have been conducted on a similar model to the one presented in this paper, where the unknown coefficient matrix $B^{\star }$ is assumed to have sparse elements. In particular, the work of [1] in the field of biostatistics has been used to examine the correlation between a response variable and the functional connectivity associated with a certain brain region. Additionally, recent research by [2] has focused on developing methods for estimating the unknown low-rank matrix $B^{\star }$ by using implicit regularization techniques.

The model discussed in this paper can be thought of as a nonparametric version of the trace regression model that has been previously proposed in the literature, specifically in the works in [3,4,5]. This trace regression model utilizes the identity function as the link function, and encompasses a diverse array of statistical models, including but not limited to reduced rank regression, matrix completion, and linear regression.

The single-index model is a versatile extension of the linear model, which offers a natural interpretation. This model only changes in the direction of the parameter (vector/matrix), and the nature of this change is depicted by the link function $f^{\star }$. This has been the subject of extensive research in the literature, with various studies exploring its applications and extensions in various fields. Examples of such works include [1,6,7,8,9,10,11,12,13,14]. These studies have demonstrated the versatility and utility of the single-index model in a wide range of contexts, making it a valuable tool for researchers in various fields.

Definition 1.

Let $S_1^d$ denote the set of all symmetric matrix $B\in {\mathbb{R}}^{d\times d}$ such that ${\parallel B\parallel }_F=1$.

Given the covariates ${\left\{X_i\right\}}_{i=1}^n$, the response variables ${\left\{Y_i\right\}}_{i=1}^n$ are i.i.d. generated from model (1). We define the expected risk for any measurable $f:\mathbb{R}\to \mathbb{R}$ and $B\in S_1^d$ as

$$R(B,f)=\mathbb{E}\left[(Y-f{\left(⟨X,B⟩\right)}^2)\right]$$

and denote the empirical counterpart of $R(f,B)$ by

$$r_n(B,f)=\frac{1}{n}\sum _{i=1}^n(Y_i-f{\left(⟨X_i,B⟩\right)}^2).$$

In this research, we examine the forecasting abilities of the model. More specifically, we consider a pair $(f,B)$ to have comparable predictive performance to $(f^{\star },B^{\star })$ if the difference between $R(B,f)$ and $R(B^{\star },f^{\star })$ is minimal.

Our approach in this work is built on the PAC-Bayesian bound technique, which is a powerful tool for obtaining oracle inequalities bounds [15]. Similar to Bayesian analysis, one important aspect of a PAC-Bayesian bound is specifying a prior distribution over the parameter space. In our approach, we adopt the prior distribution for the link function from the reference [11], while the prior distribution for the matrix parameter B is inspired by the eigen decomposition of the matrix. The specifics of our approach and the details of the prior distributions we chose are discussed in the next section. The use of the PAC-Bayesian bound technique in combination with carefully chosen prior distributions allows us to obtain reliable and accurate estimates of the unknown parameters in our model.

2. Main Result

2.1. Method

We make an additional assumption in our model (1) that $\mathbb{E}\left[ϵ\right|X]=0$, and the following conditional moment assumptions on the noise $ϵ$ are assumed.

Assumption 1.

We assume that there exist two constants $\sigma >0$ and $L>0$, such that for all integers $s\ge 2$,

$${\mathbb{E}\left[\right|ϵ|}^s\left|X\right]\le \frac{s!}{2}{\sigma }^2{L}^{s-2}.$$

Remark 1.

The assumption stated above implies that the noise term in our model follows a subexponential distribution. This class of distributions includes, for example, Gaussian noise or bounded noise, as discussed in [16]. In simpler terms, this means that the noise term in our model is characterized by a rate of decay that is slower than that of an exponential distribution. This assumption is critical for the application of our approach, as it allows us to obtain accurate and reliable estimates of the unknown parameters under a wide range of noise conditions. This is an important consideration, as the presence of noise can have a significant impact on the accuracy of the estimates obtained from our model. By assuming that the noise follows a subexponential distribution, we can be confident that our estimates are robust to the presence of noise.

In addition to the assumptions stated previously, it is also necessary to assume that the covariate matrix X is almost surely bounded by a constant. Additionally, the unknown link function $f^{\star }$ is also assumed to be bounded by some known positive constant. To make this more precise, we use the notation ${\parallel X\parallel }_{\infty }$ to represent its supremum norm and $\parallel f^{\star }{\parallel }_{\infty }$ to denote its functional supremum norm over the interval $[-1,1]$. Based on these definitions, we make the following assumption:

Assumption 2.

We assume that ${\parallel X\parallel }_{\infty }\le 1$ a.s. and $\exists C\ge 1$, such that $\parallel f^{\star }{\parallel }_{\infty }\le C.$

In order to present the technical proofs in the clearest and simplest manner, we did not attempt to find the best constant used in the proofs. Specifically, the condition that $C\ge 1$ is just convenient for the proofs in nature, and it could be eliminated by using $max[C,1]$ in the proofs.

The link function $f^{\star }$ is approximately estimated through a given specific countable set of measurable functions (dictionary) ${\left\{{\phi }_k\right\}}_{k=1}^{\infty }$. For this purpose, the set of finite linear combinations of functions from the dictionary is utilized, and we denote this vector space by $\mathcal{F}$. We assume that each element ${\phi }_k$ in the dictionary is defined on the interval $[-1,1]$ and takes values within the range $[-1,1]$.

Assumption 3.

For the sake of simplicity, we assume that the basic functions are differentiable and there exists some constant $C_{\varphi }>0$, such that

$$\parallel {\phi }_k^{\prime }{\parallel }_{\infty }\le k{C}_{\varphi }.$$

An example of such a collection of functions is the system of non-normalized trigonometric functions, where

$${\phi }_1\left(t\right)=1,{\phi }_{2k}\left(t\right)=cos(\pi kt),{\phi }_{2k+1}\left(t\right)=sin(\pi kt),k=1,2,\dots$$

satisfy this assumption. This assumption on the dictionary functions enables us to approximate the unknown link function $f^{\star }$ with a finite linear combination of these functions.

Our approach is inspired by the work of [11], where the authors explored the PAC-Bayesian approach in [15] for a sparse-vector single-index model. The method needs to first specify a distribution $\pi$ on ${\mathcal{S}}_1^d\times \mathcal{F}$, similar to the prior distribution in Bayesian analysis. This prior distribution in our framework should enforce the characteristics of the underlying link function and the parameter matrix. In this work, we consider the following prior distribution:

$$\mathrm{d}\pi (B,f)=\mathrm{d}\mu \left(B\right)\mathrm{d}\nu \left(f\right),$$

in other words, it means that the prior distribution of the index matrix and the prior distribution over the link functions are assumed to be independent.

In this study, the matrix B is treated as a symmetric matrix and can be expressed in its eigen-decomposition form $B=U\mathrm{\Lambda }{U}^{\top }$. The matrix U is an orthogonal matrix with $U{U}^{\top }=U{U}^{-1}=I_d$ (identity matrix of dimension $d\times d$), and the diagonal matrix $\mathrm{\Lambda }$ holds the corresponding eigenvalues ${\lambda }_1,\dots ,{\lambda }_d$. To enforce that ${\parallel B\parallel }_F=1$, the sum of the squares of the eigenvalues ${\lambda }_i$ must equal 1, as ${\parallel B\parallel }_F=\sqrt{\mathrm{trace}\left(B^2\right)}$ and $\mathrm{trace}\left(B^2\right)={\sum }_{j=1}^d{\lambda }_i^2$. Additionally, the requirement of low-rankness on B means that most of the eigenvalues ${\lambda }_1,\dots ,{\lambda }_d$ are close to zero, with only a few being significantly larger.

With the goal of obtaining an appropriate low-rank-promoting prior for B, we propose the following approach. We simulate an orthogonal matrix V and simulate $({\gamma }_1,\dots ,{\gamma }_d)$ from a Dirichlet distribution $Dir({\alpha }_1,\dots ,{\alpha }_d)$. Put

$$B=V\mathrm{diag}({\gamma }_1^{1/2},\dots ,{\gamma }_d^{1/2})V^{\top }.$$

To obtain an approximate low-rank matrix, we take all parameters of the Dirichlet distribution to be very close to 0, for example, by setting ${\alpha }_1=\dots ={\alpha }_d=1/d$. It is worth noting that a typical drawing of the Dirichlet distribution leads to one of the ${\gamma }_i$s being close to 1 and the others being close to 0. For more detailed discussions on how to choose the parameters for the Dirichlet distribution, one can refer to [17].

Now, we present a prior distribution on $\mathcal{F}$. We opted to use the prior introduced in [11]. With any integer M that $0<M\le n$, let us put

$${\mathcal{B}}_M\left(c_{\mathrm{\Lambda }}\right)=\left\{({\beta }_1,\cdots ,{\beta }_M)\in {\mathbb{R}}^M:\sum _{s=1}^{M}s\left|{\beta }_s\right|\le c_{\mathrm{\Lambda }}\mathrm{and}{\beta }_M\ne 0\right\},\forall c_{\mathrm{\Lambda }}>0.$$

Now, we define ${\mathcal{F}}_M\left(c_{\mathrm{\Lambda }}\right)\subset \mathcal{F}$ the image of ${\mathcal{B}}_M\left(c_{\mathrm{\Lambda }}\right)$ by the function

$$\begin{array}{cccc}{G}_M& :{\mathbb{R}}^M\hfill & \to & \mathcal{F}\hfill \\ & ({\beta }_1,\cdots ,{\beta }_M)\hfill & \mapsto & \sum _{j=1}^M{\beta }_j{\phi }_j.\hfill \end{array}$$

Remark 2.

Corollary 1 (below) provides a discussion regarding the approximation of Sobolev spaces (see [18] by the set ${\mathcal{F}}_M\left(c_{\mathrm{\Lambda }}\right)$), which become more accurate as M increases.

Now, a prior distribution ${\nu }_M\left(\mathrm{d}f\right)$ is defined on the set ${\mathcal{F}}_M(C+1)$. This is performed by considering the image of the uniform measure on ${\mathcal{B}}_M(C+1)$ obtained through the function $G_M$. We consider the following choice for the prior distribution $\nu$ on $\mathcal{F}$

$$\mathrm{d}\nu \left(f\right)={\displaystyle \frac{{\displaystyle \sum _{M=1}^n{10}^{-M}{\nu }_M\left(\mathrm{d}f\right)}}{1-{\left(\frac{1}{10}\right)}^n}}.$$

(2)

The reason for choosing $C+1$ rather than C in the above definition of the prior distribution support is essentially for technical proof. This is to ensure that as soon as the underlying link function $f^{\star }$ belongs to ${\mathcal{F}}_n\left(C\right)$, there then exists a small ball around it that is contained in ${\mathcal{F}}_n(C+1)$. One could safely replace it by $C+a_n$, where ${\left\{a_n\right\}}_{n=1}^{\infty }$ is any positive sequence vanishing sufficiently slowly as $n\to \infty$.

Remark 3.

The integer M can be viewed as a measure of the “dimension” of the function f—the larger the M, the more complex the function—and the prior ν adapts again to the sparsity idea by penalizing large-dimensional functions f. The coefficient ${10}^{-M}$, which appears in (2), shows that more complex models have a geometrically decreasing influence. Inspired from the practical results in [11], the value 10 is a random choice. This choice could be in general changed by another positive constant, but it requires more technical attention.

2.2. The Proposed Estimator

Definition 2.

The Gibbs posterior distribution over ${\mathcal{S}}_1^d\times {\mathcal{F}}_n(C+1)$ is defined as

$${\widehat{\rho }}_{\lambda }(B,f)=\frac{exp\left[-\lambda r_n(B,f)\right]d\pi (B,f)}{{\displaystyle \int exp\left[-\lambda r_n(B,f)\right]d\pi (B,f)}}.$$

Now, we define an estimator as follows. Let $\lambda >0$ be a tuning parameter, or sometime called the inverse temperature parameter. Let $({\hat{B}}_{\lambda },{\hat{f}}_{\lambda })$ be an estimator of $(B^{\star },f^{\star })$. It is simply achieved by a random draw from ${\widehat{\rho }}_{\lambda }$, the Gibbs posterior distribution above.

2.3. Theoretical Results

As $\mathbb{E}\left[Y\right|X]=f^{\star }\left(⟨X,B^{\star }⟩\right)$ almost surely, it is noted that for all $(B,f)\in {\mathcal{S}}_1^d\times {\mathcal{F}}_n(C+1)$,

$$\begin{array}{cc}\hfill R(B,f)-R(B^{\star },f^{\star })& =\mathbb{E}{\left[Y-f\left(⟨X,B⟩\right)\right]}^2-\mathbb{E}{\left[Y-f^{\star }\left(⟨X,B^{\star }⟩\right)\right]}^2\hfill \\ \hfill & =\mathbb{E}{\left[f\left(⟨X,B⟩\right)-f^{\star }\left(⟨X,B^{\star }⟩\right)\right]}^2\hfill \end{array}$$

(Pythagoras theorem).

Definition 3.

For any positive integer $M\le n$, we set

$$\left(B_M^{\star },f_M^{\star }\right)\in arg\underset{(B,f)\in {\mathcal{S}}_1^d\times {\mathcal{F}}_M\left(C\right)}{min}R(B,f).$$

Remark 4.

It is noted here that the infimum $f_M^{\star }$ is defined on ${\mathcal{F}}_M\left(C\right)$ for each value of M. However, the prior distribution is defined on a slightly larger set, that is, ${\mathcal{F}}_M(C+1)$.

Let us define

$$w:=64(C+1)max[L,C+1],C_1:=8[{(C+1)}^2+{\sigma }^2].$$

The theoretical results in this work mainly come from the following theorem, the proof of which is provided in Section 3. It should be noted that throughout the paper, the phrase “with probability $1-\delta$” refers to the probability calculated with respect to both the distribution ${\mathbf{P}}^{\otimes n}$ of the data and the conditional Gibbs distribution ${\widehat{\rho }}_{\lambda }$.

Theorem 1.

Assume that Assumptions 1 and 2 hold, with

$$\lambda =\frac{n}{w+2C_1}.$$

(3)

We have that, for all $\delta \in (0,1)$, with a probability of at least $1-\delta$,

$$\begin{array}{cc}R({\hat{B}}_{\lambda },{\hat{f}}_{\lambda })-R(B^{\star },f^{\star })\le \mathfrak{C}\underset{1\le M\le n}{inf}\hfill & \{R(B_M^{\star },f_M^{\star })-R(B^{\star },f^{\star })+\hfill \\ & \hfill \hspace{1em}\hspace{1em}\frac{log\left(n\right)(M+d\mathrm{rank}\left(B^{\star }\right)+dlog\left(d\right))+log\left(\frac{2}{\delta }\right)}{n}\},\end{array}$$

where $\mathfrak{C}>0$ is a constant depending only on $L,\sigma ,C,C_{\varphi }$.

Remark 5.

As in practice, the value of w and $C_1$ are not known, and the theoretical value of λ cannot be used. However, it provides a good order to tune this parameter, for example, using cross-validation.

Remark 6.

Theorem 1 can be interpreted in a straightforward manner. Essentially, it states that if there exists a “small” M and $\mathrm{rank}(B^{\star })$ is small, such that the difference between $R(B_M^{\star },f_M^{\star })$ and $R(B^{\star },f^{\star })$ is minimal, then the difference between $R({\hat{B}}_{\lambda },{\hat{f}}_{\lambda })$ and $R(B^{\star },f^{\star })$ will also be small in the order of $log(n)/n$. On the other hand, if neither of these conditions are met, then the rate $Mlog(n)/n$ or $\mathrm{rank}(B^{\star })dlog(n)/n$ (or either) will start to dominate, thus resulting in a decrease in the general quality of the convergence rate.

We can obtain a good convergence rate as soon as a low-rank assumption is considered. This is typically achievable when $B^{\star }$ is already low-rank or can be well approximated by a low-rank matrix. In the case that $f^{\star }$ is sufficiently regular, we can obtain a good approximation with a “small” M.

As shown in [11], when $f^{\star }$ belongs to a Sobolev space, we can derive a more specific nonparametric rate for the above theorem. For example, assume that ${\left\{{\phi }_k\right\}}_{k=1}^{\infty }$ is the system of trigonometric functions and in addition that the link function $f^{\star }$ is in the following Sobolev ellipsoid space [18],

$$\mathcal{W}\left(k,\frac{6C^2}{{\pi }^2}\right)=\left\{f\in L_2\left([-1,1]\right):f=\sum _{j=1}^{\infty }{\beta }_j{\phi }_j \mathrm{and} \sum _{j=1}^{\infty }{j}^{2k}{\beta }_j^2\le \frac{6C^2}{{\pi }^2}\right\}$$

where $k\ge 2$ is an unknown regularity parameter. In this context, the approximation set ${\mathcal{F}}_M(C+1)$ is in the following form:

$$\begin{array}{c}\hfill {\mathcal{F}}_M(C+1)=\left\{f\in L_2\left([-1,1]\right):f=\sum _{s=1}^M{\beta }_s{\phi }_s,\sum _{s=1}^{M}s\left|{\beta }_s\right|\le C+1 \mathrm{and} {\beta }_M\ne 0\right\}.\end{array}$$

It should be noted that the results presented in this paper are in the so-called adaptive setting, where the regularity parameter k is not assumed to be known. However, in order to obtain these results, it is necessary to make an additional assumption.

Assumption 4.

We assume that the probability density of the random variable $⟨X,B^{\star }⟩$ is defined on $[-1,1]$, and it is upper-bounded by a constant $A>0$.

Corollary 1.

Assume that Theorem 1 and additional Assumption 4 hold. Moreover, assume that $f^{\star }$ is in the Sobolev ellipsoid space $\mathcal{W}(k,6C^2/{\pi }^2)$, where the regularity parameter $k\ge 2$ is unknown. The tuning parameter λ is as in (3). We have that for all $\delta \in (0,1)$ with a probability of at least $1-\delta$,

$$\begin{array}{c}R({\hat{B}}_{\lambda },{\hat{f}}_{\lambda })-R(B^{\star },f^{\star })\le \hfill \\ & \hfill {\mathfrak{C}}^{\prime }\left\{{\left(\frac{log\left(n\right)}{n}\right)}^{\frac{2k}{2k+1}}+\frac{log\left(n\right)(d\mathrm{rank}\left(B^{\star }\right)+dlogd)+log\left(\frac{2}{\delta }\right)}{n}\right\},\end{array}$$

(4)

where ${\mathfrak{C}}^{\prime }>0$ is a constant depending only on L, C, σ, $C_{\varphi },A$.

The proof for Corollary 1 follows a similar approach to that of Corollary 4 in [11], and thus, it is not included in this paper.

Remark 7.

From an asymptotic point of view, that d is fixed and $n\to \infty$, the leading rate on the right-hand side in the above Corollary is ${(log\left(n\right)/n)}^{\frac{2k}{2k+1}}$. This is known to be the minimax rate of convergence up to a $log\left(n\right)$ factor over a Sobolev class; see [18]. On the other hand, in a nonasymptotic setting where n is “small”, we obtain the estimation rate $\mathrm{rank}\left(B^{\star }\right)dlog\left(n\right)/n$, which was also obtained by [2], and it is minimax optimal up to a logarithmic term, as in [3].

From Theorem 1, it is actually possible to derive that the Gibbs posterior ${\widehat{\rho }}_{\lambda }$ contracts around $(B^{\star },f^{\star })$ at the optimal rate.

Theorem 2.

Under the same assumptions for Theorem 1 and the same definition for λ, let ${\epsilon }_n$ be any sequence in $(0,1)$, such that ${\epsilon }_n\to 0$ when $n\to \infty$. Define

$$\begin{array}{c}{\mathcal{E}}_n=\{(B,f)\in {\mathcal{S}}_1^d\times {\mathcal{F}}_n(C+1):R(B,f)-R(B^{\star },f^{\star })\hfill \\ \hfill \le \mathfrak{C}\underset{1\le M\le n}{inf}\{R(B_M^{\star },f_M^{\star })-R(B^{\star },f^{\star })+\\ \hfill & & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\frac{log\left(n\right)(M+\mathrm{rank}\left(B^{\star }\right)d+dlogd)+log\left(\frac{2}{{\epsilon }_n}\right)}{n}\}.\end{array}$$

Then,

$$\mathbb{E}\left[{\mathbb{P}}_{(B,f)\sim {\widehat{\rho }}_{\lambda }}((B,f)\in {\mathcal{E}}_n)\right]\ge 1-{\epsilon }_n\underset{n\to \infty }{\to }1.$$

3. Proofs

For the sake of simplicity in the proofs, we put

$$R^{\star }:=R(B^{\star },f^{\star }),r_n^{\star }:=r_n(B^{\star },f^{\star }).$$

We have that for each $f={\sum }_{j=1}^M{\beta }_j{\phi }_j\in {\mathcal{F}}_M(C+1)$, ${\parallel f\parallel }_{\infty }\le {\sum }_{j=1}^M\left|{\beta }_j\right|\le C+1.$

The following lemma, Lemma 1, is a Bernstein-type inequality [16] that is useful for our proofs. We denote by ${\left(Z\right)}_{+}$ the positive part of a random variable Z.

Lemma 1.

Let $Z_1,\cdots ,Z_n$ be independent real-valued random variables. It is assumed that there exist two constants $v>0,w>0$ that for all integers $r\ge 2$, ${\sum }_{s=1}^n\mathbb{E}\left[{\left(Z_s\right)}_{+}^r\right]\le \frac{r!}{2}v{w}^{r-2}.$ We have that with $\zeta \in (0,1/w)$,

$$\mathbb{E}{e}^{\zeta {\sum }_{s=1}^n(Z_s-\mathbb{E}{Z}_s)}\le e^{\frac{v{\zeta }^2}{2(1-w\zeta )}}.$$

Let $(A,\mathcal{A})$ be a measurable space and ${\gamma }_1$ and ${\gamma }_2$ be two probability measures on $(A,\mathcal{A})$. Denote by $\mathcal{K}({\gamma }_1,{\gamma }_2)$ the Kullback–Leibler divergence of ${\gamma }_1$ with respect to ${\gamma }_2$. Lemma 2 is a classical result, and its proof can be found, for example, in [15], (page 4).

Lemma 2.

Let $(A,\mathcal{A})$ be a measurable space. For any probability measure ν on $(A,\mathcal{A})$ and any measurable function $g:A\to \mathbb{R}$, such that $\int (exp\circ g)\mathrm{d}\nu <\infty$, we have

$$log\int (exp\circ g)\mathrm{d}\nu =\underset{\kappa }{sup}\left(\int g\mathrm{d}\kappa -\mathcal{K}(\kappa ,\nu )\right),$$

(5)

where κ is a probability measure on $(A,\mathcal{A})$ and $\infty -\infty =-\infty$. In addition, when g is upper-bounded on the support of ν, the supremum in (5) is obtained by the Gibbs distribution g, given by

$$\frac{\mathrm{d}\rho }{\mathrm{d}\nu }\left(a\right)=\frac{exp\left(g\right(a\left)\right)}{{\displaystyle \int (exp\circ g)\mathrm{d}\nu }},a\in A.$$

Lemma 3.

We assume that Assumption 1 is satisfied. Put $w=16(C+1)max[L,2(C+1)],C_1:=8[{(C+1)}^2+{\sigma }^2]$ and take $\lambda \in \left(0,\frac{n}{w+C_1}\right)$ and put

$$\alpha =\left(\lambda -\frac{{\lambda }^2{C}_1}{2n(1-\frac{C_2\lambda }{n})}\right)\mathrm{and}\beta =\left(\lambda +\frac{{\lambda }^2{C}_1}{2n(1-\frac{C_2\lambda }{n})}\right).$$

(6)

With $\delta \in (0,1)$ and any distribution ${\widehat{\rho }}_{\lambda }\ll \pi$, we have that

$$\begin{array}{c}\mathbb{E}\int exp[\alpha \left(R(B,f)-R^{\star }\right)+\lambda \left(-r_n(B,f)+r_n^{\star }\right)-log\left(\frac{d{\widehat{\rho }}_{\lambda }}{d\pi }(B,f)\right)-\hfill \\ & \hfill log\frac{2}{\delta }]d{\widehat{\rho }}_{\lambda }(B,f)\le \delta /2,\end{array}$$

(7)

$$\begin{array}{c}\mathbb{E}\underset{\rho }{sup}exp[\beta \left(-\int R(B,f)d\rho -R^{\star }\right)+\lambda \left(\int r_n(B,f)d\rho -r_n^{\star }\right)-\hfill \\ & \hfill \mathcal{K}(\rho ,\pi )-log\frac{2}{\delta }]\le \delta /2,\end{array}$$

(8)

Proof. 

Fix $B\in {\mathcal{S}}_1^d$ and $f\in {\mathcal{F}}_n(C+1)$. We start by using Lemma 1 with the following random variables:

$$T_i=-{\left(Y_i-f\left(⟨X_i,B⟩\right)\right)}^2+{\left(Y_i-f^{\star }\left(⟨X_i,B^{\star }⟩\right)\right)}^2,i=1,\dots ,n.$$

Note that $T_i,i=1,\dots ,n$ are independent, and we have that

$$\begin{array}{cc}\hfill \sum _{i=1}^n\mathbb{E}{T}_i^2& =\sum _{i=1}^n\mathbb{E}\left\{{\left[2Y_i-f\left(⟨X_i,B⟩\right)-f^{\star }\left(⟨X_i,B^{\star }⟩\right)\right]}^2{\left[f\left(⟨X_i,B⟩\right)-f^{\star }\left(⟨X_i,B^{\star }⟩\right)\right]}^2\right\}\hfill \\ \hfill & =\sum _{i=1}^n\mathbb{E}\left\{{\left[2{ϵ}_i+f^{\star }\left(⟨X_i,B^{\star }⟩\right)-f\left(⟨X_i,B⟩\right)\right]}^2{\left[f\left(⟨X_i,B⟩\right)-f^{\star }\left(⟨X_i,B^{\star }⟩\right)\right]}^2\right\}\hfill \\ \hfill & \le \sum _{i=1}^n\mathbb{E}\left\{\left[8{ϵ}_i^2+8{(C+1)}^2\right]{\left[f\left(⟨X_i,B⟩\right)-f^{\star }\left(⟨X_i,B^{\star }⟩\right)\right]}^2\right\}.\hfill \\ \hfill & \le 8\left[{(C+1)}^2+{\sigma }^2\right]\sum _{i=1}^n\mathbb{E}{\left[f\left(⟨X_i,B⟩\right)-f^{\star }\left(⟨X_i,B^{\star }⟩\right)\right]}^2:=v,\hfill \end{array}$$

where we set $C_1:=8[{(C+1)}^2+{\sigma }^2];$ and $v=n{C}_1\left[R(B,f)-R^{\star }\right].$

Now, for all integers k greater than 3, we have that

$$\begin{array}{cc}\hfill & \sum _{i=1}^n\mathbb{E}\left[{\left(T_i\right)}_{+}^k\right]\hfill \\ \hfill & \le \sum _{i=1}^n\mathbb{E}\left\{|2Y_i-f\left(⟨X_i,B⟩\right)-f^{\star }\left(⟨X_i,B^{\star }⟩\right){|}^k{|f\left(⟨X_i,B⟩\right)-f^{\star }\left(⟨X_i,B^{\star }⟩\right)|}^k\right\}\hfill \\ \hfill & =\sum _{i=1}^n\mathbb{E}\left\{|2{ϵ}_i+f^{\star }\left(⟨X_i,B^{\star }⟩\right)-f\left(⟨X_i,B⟩\right){|}^k{|f\left(⟨X_i,B⟩\right)-f^{\star }\left(⟨X_i,B^{\star }⟩\right)|}^k\right\}\hfill \\ \hfill & \le 2^{k-1}\sum _{i=1}^n\mathbb{E}\left\{\left[2^k{\left|{ϵ}_i\right|}^k+2^k{(C+1)}^k\right]2^{k-2}{(C+1)}^{k-2}{|f\left(⟨X_i,B⟩\right)-f^{\star }\left(⟨X_i,B^{\star }⟩\right)|}^2\right\}.\hfill \end{array}$$

In the last inequality, we used the fact that ${|q+w|}^k\le 2^{k-1}{\left(\right|q|}^k+{\left|w\right|}^k)$. We obtain that

$$\begin{array}{cc}\hfill \sum _{i=1}^n\mathbb{E}\left[{\left(T_i\right)}_{+}^k\right]& \le \sum _{i=1}^n\left[2^{2k-2}k!{\sigma }^2{L}^{k-2}+2^{2k-1}{(C+1)}^k\right]2^{k-2}{(C+1)}^{k-2}\left[R(B,f)-R^{\star }\right]\hfill \\ \hfill & =v\times \frac{\left[2^{2k-2}k!{\sigma }^2{L}^{k-2}+2^{2k-1}{(C+1)}^k\right]2^{k-2}{(C+1)}^{k-2}}{[2{(C+1)}^2+4{\sigma }^2]}\hfill \\ \hfill & \le v\times \frac{k!8^{k-2}max\left[L^{k-2},2^{k-2}{(C+1)}^{k-2}\right]2^{k-2}{(C+1)}^{k-2}}{2}:=\frac{k!}{2}v{w}^{k-2},\hfill \end{array}$$

with $w=64(C+1)max[L,C+1]$.

Thus, for any $\lambda \in (0,n/w)$, taking $\zeta =\lambda /n$, we apply Lemma 1 to obtain

$$\begin{array}{cc}\hfill \mathbb{E}exp\left[\lambda \left(R(B,f)-R^{\star }-r_n(B,f)+r_n^{\star }\right)\right]& \le exp\left(\frac{v{\lambda }^2}{2n^2(1-\frac{w\lambda }{n})}\right)\hfill \\ \hfill & =exp\left(\frac{C_1\left[R(B,f)-R^{\star }\right]{\lambda }^2}{2n(1-\frac{w\lambda }{n})}\right).\hfill \end{array}$$

Therefore, we obtain, with the $\alpha$ given in (6),

$$\begin{array}{c}\hfill \mathbb{E}{e}^{\alpha \left(R(B,f)-R^{\star }\right)+\lambda \left(-r_n(B,f)+r_n^{\star }\right)-log\left(\frac{2}{\delta }\right)}\le \delta /2.\end{array}$$

Next, integrating with respect to $\pi$ and consequently using Fubini’s theorem, we obtain

$$\begin{array}{c}\hfill \mathbb{E}\int exp[\alpha \left(R(B,f)-R^{\star }\right)+\lambda \left(-r_n(B,f)+r_n^{\star }\right)-log\left(2/\delta \right)]\mathrm{d}\pi (B,f)\le \delta /2.\end{array}$$

To obtain (7), it is noted that for any measurable function h,

$$\begin{array}{c}\hfill \int exp\left[h(B,f)\right]d\pi =\int exp\left[h(B,f)-log\frac{d{\widehat{\rho }}_{\lambda }}{d\pi }(B,f)\right]d{\widehat{\rho }}_{\lambda }.\end{array}$$

The proof for (8) is similar. More precisely, we apply Lemma 1 with $T_i={(Y_i-f\left(⟨X,B⟩\right))}^2-{(Y_i-f^{\star }\left(⟨X,B^{\star }⟩\right))}^2$. We obtain, for any $\lambda \in (0,n/w)$,

$$\begin{array}{c}\hfill \mathbb{E}exp\left[\lambda \left(r_n(B,f)+r_n^{\star }-R(B,f)+R^{\star }\right)\right]\le exp\left(\frac{v{\lambda }^2}{2n^2(1-\frac{w\lambda }{n})}\right).\end{array}$$

By rearranging terms, using definition of $\beta$ in (6), and multiplying both sides by $\delta /2$, we obtain

$$\begin{array}{c}\hfill \mathbb{E}exp\left[\beta \left(-R(B,f)+R^{\star }\right)+\lambda \left(r_n(B,f)-r_n^{\star }\right)-log\frac{2}{\delta }\right]\le \delta /2.\end{array}$$

Integrating with respect to $\pi$ and using Fubini’s theorem, we obtain

$$\begin{array}{c}\hfill \mathbb{E}\int exp\left[\beta \left(-R(B,f)+R^{\star }\right)+\lambda \left(r_n(B,f)-r_n^{\star }\right)-log\frac{2}{\delta }\right]\mathrm{d}\pi \le \delta /2.\end{array}$$

Now, Lemma 2 is applied to the integral, and this directly yields (8). □

Proof of Theorem 1.

Recall that ${\mathbf{P}}^{\otimes n}$ stands for the distribution of the sample ${\mathcal{D}}_n$; the Equation (7) can be written conveniently as

$$\begin{array}{c}{\mathbb{E}}_{{\mathcal{D}}_n\sim {\mathbf{P}}^{\otimes n}}{\mathbb{E}}_{(\hat{B},\hat{f})\sim {\widehat{\rho }}_{\lambda }}exp[\alpha \left(R(\hat{B},\hat{f})-R^{\star }\right)+\lambda \left(-r_n(\hat{B},\hat{f})+r_n^{\star }\right)-\hfill \\ \hfill log\left(\frac{\mathrm{d}{\widehat{\rho }}_{\lambda }}{\mathrm{d}\pi }(\hat{B},\hat{f})\right)-log\frac{2}{\delta }]\le \delta /2,\end{array}$$

Now, we use the standard Chernoff trick to transform an exponential moment inequality into a deviation inequality, i.e., using $exp(\lambda x)\ge {\mathbf{1}}_{{\mathbb{R}}_{+}}(x)$. We obtain, with a probability of at least $1-\delta /2$ for any $\delta \in (0,1)$ and any distribution ${\widehat{\rho }}_{\lambda }$,

$$\begin{array}{c}\hfill R(\hat{B},\hat{f})-R^{\star }\le \frac{\lambda }{\alpha }(r_n(\hat{B},\hat{f})-r_n^{\star }+\frac{log\left(\frac{\mathrm{d}{\widehat{\rho }}_{\lambda }}{\mathrm{d}\pi }(\hat{B},\hat{f})\right)+log\left(\frac{2}{\delta }\right)}{\lambda }).\end{array}$$

It is noted that we have

$$\begin{array}{cc}\hfill log\left(\frac{\mathrm{d}{\widehat{\rho }}_{\lambda }}{\mathrm{d}\pi }(\hat{B},\hat{f})\right)& =log\left(\frac{exp(-\lambda r_n(\hat{B},\hat{f}))}{\int exp(-\lambda r_n(B,f))\mathrm{d}\pi }\right)\hfill \\ \hfill & =-\lambda r_n(\hat{B},\hat{f})-log\int e^{-\lambda r_n(B,f)}\mathrm{d}\pi ;\hfill \end{array}$$

thus, we obtain, with a probability larger than $1-\delta /2$,

$$\begin{array}{c}\hfill R(\hat{B},\hat{f})-R^{\star }\le \frac{1}{\alpha }(log\int exp(-\lambda r_n(B,f))\mathrm{d}\pi -\lambda r_n^{\star }+log\left(\frac{2}{\delta }\right)).\end{array}$$

Now, using Lemma 2, it yields that with a probability larger than $1-\delta /2$,

$$\begin{array}{c}\hfill R(\hat{B},\hat{f})-R^{\star }\le \frac{\lambda }{\alpha }(\int r_n(B,f)\mathrm{d}{\widehat{\rho }}_{\lambda }-r_n^{\star }+\frac{\mathcal{K}({\widehat{\rho }}_{\lambda },\pi )+log\left(\frac{2}{\delta }\right)}{\lambda }).\end{array}$$

(9)

Now, from (8) with an application of the standard Chernoff trick, we obtain, with a probability larger than $1-\delta /2$ for any $\delta \in (0,1)$ and any distribution ${\widehat{\rho }}_{\lambda }\ll \pi$,

$$\begin{array}{c}\hfill \int r_n(B,f)\mathrm{d}{\widehat{\rho }}_{\lambda }-r_n^{\star }\le \frac{\beta }{\lambda }(\int R(B,f)\mathrm{d}{\widehat{\rho }}_{\lambda }-R^{\star })+\frac{\mathcal{K}({\widehat{\rho }}_{\lambda },\pi )+log\left(\frac{2}{\delta }\right)}{\lambda }.\end{array}$$

(10)

Combining (9) and (10) with a union bound argument gives the bound, with a probability larger than $1-\delta$,

$$\begin{array}{c}\hfill R(\hat{B},\hat{f})-R^{\star }\le \underset{\rho }{inf}\left\{\frac{\beta }{\alpha }\left(\int R(B,f)\mathrm{d}\rho -R^{\star }\right)+2\frac{\mathcal{K}(\rho ,\pi )+log\left(\frac{2}{\delta }\right)}{\alpha }\right\}.\end{array}$$

(11)

The final steps of the proof involve making the right-hand side of the inequality more explicit. To achieve this, we limit the infimum bound to a specific distribution. This allows us to have a more concrete understanding of the result and to explicitly obtain the error rate.

Put $B^{\star }=U\mathrm{\Lambda }{U}^{\top }$ and let $r=\#\{i:{\mathrm{\Lambda }}_i>\epsilon \},$ with small $\epsilon \in (0,1)$. Take

$$\mathrm{d}{\rho }_{\eta }^1\propto \mathbf{1}(\forall i:|v_i-{\mathrm{\Lambda }}_i|\le \epsilon ;\forall i=1,\dots ,r:\parallel u_i-U_i{\parallel }_F\le \eta )\pi (du,dv)$$

For any positive integer $M\le n$ and any $\eta ,\gamma \in (0,1/n)$, let the probability measure ${\rho }_{M,\eta ,\gamma }$ be defined by

$$\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f)=\mathrm{d}{\rho }_{\eta }^1\left(B\right)\mathrm{d}{\rho }_{M,\gamma }^2\left(f\right),$$

with

$${\rho }_{M,\gamma }^2\left(f\right)\propto {\mathbf{1}}_{[\parallel f-f_M^{\star }{\parallel }_M\le \gamma ]}{\nu }_M\left(f\right).$$

We denote for $f={\sum }_{s=1}^M{\beta }_s{\phi }_s\in {\mathcal{F}}_M(C+1)$, ${\parallel f\parallel }_M={\sum }_{j=1}^{M}j\left|{\beta }_j\right|.$

Inequality (11) leads to

$$\begin{array}{c}R(\hat{B},\hat{f})-R^{\star }\le \underset{1\le M\le n}{inf}\underset{\eta ,\gamma >0}{inf}\{\frac{\beta }{\alpha }\left(\int R(B,f)\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f)-R^{\star }\right)+\hfill \\ \hfill 2\frac{\mathcal{K}({\rho }_{M,\eta ,\gamma },\pi )+log\left(\frac{2}{\delta }\right)}{\alpha }\}.\end{array}$$

(12)

To finish the proof, we have to control the different terms in (12). Note first that

$$\begin{array}{cc}\hfill \mathcal{K}({\rho }_{M,\eta ,\gamma },\pi )& =\mathcal{K}({\rho }_{\eta }^1\otimes {\rho }_{M,\gamma }^2,\mu \otimes {\nu }_M)\hfill \\ \hfill & =\mathcal{K}({\rho }_{\eta }^1,\mu )+\mathcal{K}({\rho }_{M,\gamma }^2,{\nu }_M)+log\frac{1-{(1/10)}^n}{{10}^{-M}}.\hfill \end{array}$$

By technical Lemma 4, we know that

$$\mathcal{K}({\rho }_{\eta }^1,\mu )\le rdlog(16/\eta )+C_{D_1}dlogd(1+log(2/\epsilon )).$$

Additionally, by technical Lemma 10 in [11], we have that

$$\mathcal{K}({\rho }_{M,\gamma }^2,{\nu }_M)=Mlog\left(\frac{C+1}{\gamma }\right).$$

Bringing together all the parts, it arrives at

$$\mathcal{K}({\rho }_{M,\eta ,\gamma },\pi )\le rdlog(1/c)+C_{D_1}dlogd(1+log(2/\delta ))+Mlog\left(\frac{C+1}{\gamma }\right)+log\frac{1}{{10}^{-M}}.$$

(13)

Finally, it remains to control the term $\int R(B,f)\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f).$ To this aim, we write

$$\begin{array}{cc}\hfill & \int R(B,f)\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f)\hfill \\ & =\int \mathbb{E}\left[{\left(Y-f\left(⟨X,B⟩\right)\right)}^2\right]\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f)\hfill \\ \hfill & =\int \mathbb{E}\left[\right(Y-f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)+f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B_M^{\star }⟩\right)+\hfill \\ & f\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B⟩\right){)}^2]\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f)\hfill \\ \hfill & =R(B_M^{\star },f_M^{\star })+\int \mathbb{E}[{\left(f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B_M^{\star }⟩\right)\right)}^2+{\left(f\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B⟩\right)\right)}^2\hfill \\ \hfill & +2\left(Y-f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)\right)\left(f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B_M^{\star }⟩\right)\right)+\hfill \\ \hfill & 2\left(Y-f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)\right)\left(f\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B⟩\right)\right)\hfill \\ \hfill & +2\left(f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B_M^{\star }⟩\right)\right)\left(f\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B⟩\right)\right)]\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f)\hfill \\ \hfill & :=R(B_M^{\star },f_M^{\star })+\mathbf{A}+\mathbf{B}+\mathbf{C}+\mathbf{D}+\mathbf{E}.\hfill \end{array}$$

Computation of C by Fubini’s theorem:

$$\begin{array}{cc}\hfill & \mathbf{C}\hfill \\ \hfill =& \mathbb{E}\left[\int 2\left(Y-f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)\right)\left(f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B_M^{\star }⟩\right)\right)\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f)\right]\hfill \\ \hfill =& \mathbb{E}\left\{\int \left[2\left(Y-f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)\right)\int \left(f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B_M^{\star }⟩\right)\right)\mathrm{d}{\rho }_{M,\gamma }^2\left(f\right)\right]\mathrm{d}{\rho }_{\eta }^1\left(B\right)\right\}.\hfill \end{array}$$

Using the triangle inequality, we obtain that for $f={\sum }_{s=1}^M{\beta }_s{\phi }_s$ and $f_M^{\star }={\sum }_{s=1}^M{\left({\beta }_M^{\star }\right)}_s{\phi }_s$,

$$\sum _{j=1}^{M}j|{\beta }_j|\le \sum _{j=1}^{M}j|{\beta }_j-{\left({\beta }_M^{\star }\right)}_j|+\sum _{j=1}^{M}j|{\left({\beta }_M^{\star }\right)}_j|.$$

Since $f_M^{\star }\in {\mathcal{F}}_M\left(C\right)$, and thus ${\sum }_{s=1}^{M}s\left|{\left({\beta }_M^{\star }\right)}_s\right|\le C,$ as a consequence, ${\sum }_{s=1}^{M}s\left|{\beta }_s\right|\le C+1$ as soon as $\parallel f-f_M^{\star }{\parallel }_M\le 1$. This shows that the set

$$\left\{f=\sum _{j=1}^M{\beta }_j{\phi }_j:{\parallel f-f_M^{\star }\parallel }_M\le \gamma \right\}$$

is contained in the support of ${\nu }_M$. In particular, this implies that ${\rho }_{M,\gamma }^2$ is centered at $f_M^{\star }$ and, consequently,

$$\int \left(f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B_M^{\star }⟩\right)\right)\mathrm{d}{\rho }_{M,\gamma }^2\left(f\right)=0.$$

This proves that $\mathbf{C}=0$.

Control of A: Clearly,

$$\mathbf{A}\le \int \underset{y\in \mathbb{R}}{sup}{\left((f_M^{\star }\left(y\right)-f\left(y\right)\right)}^2\mathrm{d}{\rho }_{M,\gamma }^2\left(f\right)\le {\gamma }^2.$$

Control of B: We have

$$\begin{array}{cc}\hfill & \mathbf{B}=\int \mathbb{E}\left[{\left(f\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B⟩\right)\right)}^2\right]\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f)\hfill \\ \hfill & \le \int \mathbb{E}\left[{\left(C_{\varphi }(C+1)(B_M^{\star }-B)X\right)}^2\right]\mathrm{d}{\rho }_{\eta }^1\left(B\right)\left(\mathrm{using\; the\; mean\; value\; theorem}\right)\hfill \\ \hfill & \le C_{\varphi }^2{(C+1)}^2\mathbb{E}\left[{\parallel X\parallel }_{\infty }^2\right]\int {\parallel B_M^{\star }-B\parallel }_F^2\mathrm{d}{\rho }_{\eta }^1\left(B\right)\left(\mathrm{by\; Assumption\; 4}\right).\hfill \end{array}$$

Using Lemma 6 from [19], we have that

$$\int \parallel B_M^{\star }{-B\parallel }_F^2\mathrm{d}{\rho }_{\eta }^1\left(B\right)\le {(3dc+2r\eta )}^2.$$

Thus,

$$\mathbf{B}\le C_{\varphi }^2{(C+1)}^2{(3dc+2r\eta )}^2.$$

Control of E: We have that

$$\begin{array}{cc}\hfill \left|\mathbf{E}\right|& \le 2\int \mathbb{E}\left[|f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B_M^{\star }⟩\right)\left|\right|f\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B⟩\right)|\right]\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f)\hfill \\ \hfill & \le 2\int \mathbb{E}\left[|f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B_M^{\star }⟩\right)|C_{\varphi }(C+1)\left|(B_M^{\star }-B)X\right|\right]\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f)\hfill \\ \hfill & \le 2{\left(\int \mathbb{E}{\left(f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B_M^{\star }⟩\right)\right)}^2\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f)\right)}^{\frac{1}{2}}\hfill \\ \hfill & {\left(\int \mathbb{E}{\left(C_{\varphi }(C+1)(B_M^{\star }-B)X\right)}^2\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f)\right)}^{\frac{1}{2}}\hfill \\ \hfill & \le 2{\left({\gamma }^2\right)}^{\frac{1}{2}}{\left(C_{\varphi }^2{(C+1)}^2{(3dc+2r\eta )}^2\right)}^{\frac{1}{2}}=2C_{\varphi }(C+1)\gamma (3d\epsilon +2r\eta ).\hfill \end{array}$$

Control of D: Finally,

$$\begin{array}{cc}\hfill \mathbf{D}& =2\int \mathbb{E}\left[\left(Y-f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)\right)\left(f\left(⟨X,B_M^{\star }⟩\right)-f\left(⟨X,B⟩\right)\right)\right]\mathrm{d}{\rho }_{M,\eta ,\gamma }(B,f)\hfill \\ \hfill & =2\int \mathbb{E}\left[\left(Y-f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)\right)\left(f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f_M^{\star }\left(⟨X,B⟩\right)\right)\right]\mathrm{d}{\rho }_{\eta }^1\left(B\right)\hfill \\ \hfill & (\mathrm{since}\int f\mathrm{d}{\rho }_{M,\gamma }^2\left(f\right)=f_M^{\star })\hfill \\ \hfill & =2\mathbb{E}\left[\left(Y-f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)\right)\int \left(f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f_M^{\star }\left(⟨X,B⟩\right)\right)\mathrm{d}{\rho }_{\eta }^1\left(B\right)\right]\hfill \\ \hfill & \le 2\sqrt{\mathbb{E}\left[{\left(Y-f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)\right)}^2\right]}\sqrt{\mathbb{E}{\left[\int \left(f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f_M^{\star }\left(⟨X,B⟩\right)\right)\mathrm{d}{\rho }_{\eta }^1\left(B\right)\right]}^2}\hfill \\ \hfill & =2\sqrt{R(B_M^{\star },f_M^{\star })}\sqrt{\mathbb{E}{\left[\int \left(f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f_M^{\star }\left(⟨X,B⟩\right)\right)\mathrm{d}{\rho }_{\eta }^1\left(B\right)\right]}^2}.\hfill \end{array}$$

As we have that

$$\begin{array}{c}\hfill |f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f_M^{\star }\left(⟨X,B⟩\right)|\le C_{\varphi }(C+1)|⟨(B_M^{\star }-B)X⟩|\le C_{\varphi }(C+1){\parallel B_M^{\star }-B\parallel }_F,\end{array}$$

it leads to

$$\begin{array}{cc}\hfill {\left[\int \left(f_M^{\star }\left(⟨X,B_M^{\star }⟩\right)-f_M^{\star }\left(⟨X,B⟩\right)\right)\mathrm{d}{\rho }_{\eta }^1\left(B\right)\right]}^2& \le C_{\varphi }^2{(C+1)}^2{\left[\int \parallel B_M^{\star }{-B\parallel }_F\mathrm{d}{\rho }_{\eta }^1\left(B\right)\right]}^2\hfill \\ \hfill & \le C_{\varphi }^2{(C+1)}^2{(3dc+2r\eta )}^2,\hfill \end{array}$$

and therefore,

$$\begin{array}{c}\hfill \mathbf{D}\le 2C_{\varphi }(C+1)(3dc+2r\eta )\sqrt{R(0,0)/2}\le \sqrt{2}{C}_{\varphi }(C+1)(3d\epsilon +2r\eta )\sqrt{C^2+{\sigma }^2}.\end{array}$$

Thus, taking $\eta =\gamma =\epsilon =1/n$ and assembling all the components, we obtain that

$$\mathbf{A}+\mathbf{B}+\mathbf{C}+\mathbf{D}+\mathbf{E}\le \frac{{\mathfrak{C}}_1}{n},$$

where ${\mathfrak{C}}_1$ is a positive constant function of C, $\sigma$, and $C_{\varphi }$. Combining this inequality with (12) and (13) yields, with a probability larger than $1-\delta$,

$$\begin{array}{cc}\hfill R({\hat{B}}_{\lambda },{\hat{f}}_{\lambda })-R^{\star }& \le \underset{1\le M\le n}{inf}\{\frac{\beta }{\alpha }\left(R(B_M^{\star },f_M^{\star })-R^{\star }+\frac{{\mathfrak{C}}_1}{n}\right)\hfill \\ \hfill & +2\frac{Mlog\left((C+1)10n\right)+rdlog\left(16n\right)+C_{D_1}dlogdlog\left(2ne\right)+log\left(\frac{2}{\delta }\right)}{\lambda }\}.\hfill \end{array}$$

Finally, choosing $\lambda =\frac{n}{w+2C_1},$ it yields that there exists a constant ${\mathfrak{C}}_2>0$ depending only on $L,\sigma ,C,C_{\varphi }$ with a probability of at least $1-\delta$, such that

$$\begin{array}{cc}\hfill R({\hat{B}}_{\lambda },{\hat{f}}_{\lambda })-R^{\star }& \le {\mathfrak{C}}_2\underset{1\le M\le n}{inf}\{R(B_M^{\star },f_M^{\star })-R^{\star }+\hfill \\ \hfill & \frac{Mlog\left(10Cn\right)+rdlog\left(16n\right)+{\mathfrak{C}}_{3}dlogdlog\left(2ne\right)+log\left(\frac{2}{\delta }\right)}{n}\}.\hfill \end{array}$$

This concludes the proof of Theorem 1. □

Lemma 4.

Let $r=\#\{i:{\mathrm{\Lambda }}_i>\epsilon \}$ with small $\epsilon \in [0,1)$. Take

$$d{\rho }_{\eta }^1\propto \mathbf{1}(\forall i:|v_i-{\mathrm{\Lambda }}_i|\le \epsilon ;\forall i=1,\dots ,r:\parallel u_i-U_i{\parallel }_F\le \eta )\mu (du,dv)$$

Then,

$$\mathcal{K}({\rho }_{\eta }^1,\mu )\le rdlog(16/\eta )+{\mathfrak{C}}_{3}dlogdlog(2e/\epsilon )$$

where ${\mathfrak{C}}_3$ is a universal constant.

Proof. 

We have that

$$\begin{array}{cc}\hfill \mathcal{K}({\rho }_{\eta }^1,\mu )=& log\frac{1}{\mu \left(\right\{u,v:\forall i:|v_i-{\mathrm{\Lambda }}_i|\le \epsilon ;\forall i=1,r:\parallel u_i-U_i{\parallel }_F\le \eta \left\}\right)}\hfill \\ \hfill =& log\frac{1}{\mu \left(\left\{\forall i=1,r:\parallel u_{i.}-U_{i.}{\parallel }_F\le \eta \right\}\right)}+log\frac{1}{\mu \left(\right\{\forall i:|v_i-{\mathrm{\Lambda }}_i|\le \epsilon \})}.\hfill \end{array}$$

The first log term

$$\begin{array}{cc}\hfill \pi \left(\left\{\forall i=1,r:\parallel u_{i.}-U_{i.}{\parallel }_F\le \eta \right\}\right)& \ge \prod _{i=1}^r\left[{\displaystyle \frac{{\pi }^{(d-1)/2}{(\eta /2)}^{d-1}}{\Gamma (\frac{d-1}{2}+1)}}/{\displaystyle \frac{2{\pi }^{(d+1)/2}}{\Gamma \left(\frac{d+1}{2}\right)}}\right]\hfill \\ \hfill & \ge {\left[{\displaystyle \frac{{\eta }^{d-1}}{2^d\pi }}\right]}^r\ge {\displaystyle \frac{{\eta }^{r(d-1)}}{2^{4rd}}}.\hfill \end{array}$$

Note the following for the above calculation: firstly, the distribution of the orthogonal vector is approximated by the uniform distribution on the sphere [20], and secondly, the probability is greater or equal to the volume of the (d-1)-“circle” with radius $c/2$ over the surface area of the d-“unit sphere”.

It is noted that if $\gamma \sim Beta(a,b)$ (beta distribution), then ${\gamma }^{1/2}$ has the pdf as $f\left(\gamma \right)=2\frac{{\gamma }^{2a-1}{(1-{\gamma }^2)}^{b-1}}{Be(a,b)},0<\gamma <1$ where $Be(a,b)$ is the beta function. The second log term in the Kullback–Leibler term with $a={\alpha }_i,b={\sum }_{i=1}^d{\alpha }_i-{\alpha }_i,{\alpha }_i=1/d$ is

$$\begin{array}{cc}\hfill \pi \left(\right\{\forall i:|v_i-{\mathrm{\Lambda }}_i|\le \epsilon \})& =\prod _{i=1}^d{\int }_{max({\mathrm{\Lambda }}_i-\epsilon ,0)}^{min({\mathrm{\Lambda }}_i+\epsilon ,1)}\frac{v_i^{2a-1}{(1-v_i^2)}^{b-1}}{2Be(a,b)}d{v}_i\hfill \\ \hfill & \ge \prod _{i=1}^d{\int }_0^{\epsilon }\frac{v_i^{2a-1}{(1-v_i^2)}^{b-1}}{2Be(a,b)}d{v}_i\ge {\mathfrak{C}}_3{(\epsilon /2d)}^d{e}^{-d log d}.\hfill \end{array}$$

The interval of integration contains at least an interval of length $\epsilon$. Thus, we obtain

$$\begin{array}{c}\hfill \mathcal{K}({\rho }_{\eta }^1,\mu )\le log{\displaystyle \frac{2^{4rd}}{{\eta }^{r(d-1)}}}+log\left(\frac{{\left(2d\right)}^d{e}^{d log d}}{{\mathfrak{C}}_3{\epsilon }^d}\right)\le rdlog(\frac{16}{\eta })+{\mathfrak{C}}_{3}dlogdlog(\frac{e2}{\epsilon })\end{array}$$

for some absolute numerical constant ${\mathfrak{C}}_3$ that does not depend on $r,n$ or d. □

Proof of Theorem 2.

We also apply Lemma 3, and focus on (7), applied to $\delta :={\epsilon }_n$, that is

$$\begin{array}{c}\hfill \mathbb{E}\int exp[\alpha \left(R(B,f)-R^{\star }\right)+\lambda \left(-r_n(B,f)+r_n^{\star }\right)-log\left(\frac{\mathrm{d}{\widehat{\rho }}_{\lambda }}{\mathrm{d}\pi }(B,f)\right)-\\ \hfill log\frac{2}{{\epsilon }_n}]\mathrm{d}{\widehat{\rho }}_{\lambda }(B,f)\le {\epsilon }_n/2\end{array}$$

Using Chernoff’s inequality, this leads to

$$\mathbb{E}\left[{\mathbb{P}}_{(B,f)\sim {\widehat{\rho }}_{\lambda }}((B,f)\in {\mathcal{A}}_n)\right]\ge 1-\frac{{\epsilon }_n}{2}$$

where

$${\mathcal{A}}_n=\left\{(B,f):\alpha \left(R(B,f)-R^{\star }\right)+\lambda \left(-r_n(B,f)+r_n^{\star }\right)\le log\left[\frac{d{\widehat{\rho }}_{\lambda }}{d\pi }(B,f)\right]+log\frac{2}{{\epsilon }_n}\right\}.$$

From the definition of ${\widehat{\rho }}_{\lambda }$, for $(B,f)\in {\mathcal{A}}_n$, we obtain that

$$\begin{array}{cc}\hfill \alpha \left(R(B,f)-R^{\star }\right)& \le \lambda \left(r_n(B,f)-r_n^{\star }\right)+log\left[\frac{d{\widehat{\rho }}_{\lambda }}{d\pi }(B,f)\right]+log\frac{2}{{\epsilon }_n}\hfill \\ \hfill & \le -log\int exp\left[-\lambda r_n(B,f)\right]\pi \left(\mathrm{d}(B,f)\right)-\lambda r_n^{\star }+log\frac{2}{{\epsilon }_n}\hfill \\ \hfill & =\lambda \left(\int r_n(B,f){\widehat{\rho }}_{\lambda }\left(\mathrm{d}(B,f)\right)-r_n^{\star }\right)+\mathcal{K}({\widehat{\rho }}_{\lambda },\pi )+log\frac{2}{{\epsilon }_n}\hfill \\ \hfill & =\underset{\rho }{inf}\left\{\lambda \left(\int r_n(B,f)\rho \left(\mathrm{d}(B,f)\right)-r_n^{\star }\right)+\mathcal{K}(\rho ,\pi )+log\frac{2}{{\epsilon }_n}\right\}.\hfill \end{array}$$

Now, put

$${\mathcal{B}}_n:=\left\{\forall \rho ,\beta \left(-\int R(B,f)d\rho +R^{\star }\right)+\lambda \left(\int r_{n}d\rho -r_n^{\star }\right)\le \mathcal{K}(\rho ,\pi )+log\frac{2}{{\epsilon }_n}\right\}.$$

Using (8), we have that

$$\mathbb{E}\left[{\mathbf{1}}_{{\mathcal{B}}_n}\right]\ge 1-\frac{{\epsilon }_n}{2}.$$

We now prove that if $\lambda$ is such that $\alpha >0$,

$$\mathbb{E}\left[{\mathbb{P}}_{(B,f)\sim {\widehat{\rho }}_{\lambda }}((B,f)\in {\mathcal{E}}_n)\right]\ge \mathbb{E}\left[{\mathbb{P}}_{(B,f)\sim {\widehat{\rho }}_{\lambda }}((B,f)\in {\mathcal{A}}_n){\mathbf{1}}_{{\mathcal{B}}_n}\right]$$

and, together with,

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\mathbb{P}}_{(B,f)\sim {\widehat{\rho }}_{\lambda }}((B,f)\in {\mathcal{A}}_n){\mathbf{1}}_{{\mathcal{B}}_n}\right]& =\mathbb{E}\left[(1-{\mathbb{P}}_{(B,f)\sim {\widehat{\rho }}_{\lambda }}((B,f)\notin {\mathcal{A}}_n))(1-{\mathbf{1}}_{{\mathcal{B}}_n^c})\right]\hfill \\ \hfill & \ge \mathbb{E}\left[1-{\mathbb{P}}_{(B,f)\sim {\widehat{\rho }}_{\lambda }}((B,f)\notin {\mathcal{A}}_n)-{\mathbf{1}}_{{\mathcal{B}}_n^c}\right]\hfill \\ \hfill & \ge 1-{\epsilon }_n\hfill \end{array}$$

leads to

$$\mathbb{E}\left[{\mathbb{P}}_{(B,f)\sim {\widehat{\rho }}_{\lambda }}((B,f)\in {\mathcal{E}}_n)\right]\ge 1-{\epsilon }_n.$$

To obtain that, assume that we are on the set ${\mathcal{B}}_n$, and let $(B,f)\in {\mathcal{A}}_n$. Then,

$$\begin{array}{cc}\hfill \alpha \left(R(B,f)-R^{\star }\right)& \le \underset{\rho }{inf}\left\{\lambda \left(\int r_n(B,f)\rho \left(\mathrm{d}(B,f)\right)-r_n^{\star }\right)+\mathcal{K}(\rho ,\pi )+log\frac{2}{{\epsilon }_n}\right\}\hfill \\ \hfill & \le \underset{\rho }{inf}\left\{\beta \left(\int R(B,f)\rho \left(\mathrm{d}(B,f)\right)-R^{\star }\right)+2\mathcal{K}(\rho ,\pi )+2log\frac{2}{{\epsilon }_n}\right\}\hfill \end{array}$$

that is,

$$R(B,f)-R^{\star }\le \underset{\rho }{inf}\frac{\beta \left[\int Rd\rho -R^{\star }\right]+2\left[\mathcal{K}(\rho ,\pi )+log\frac{2}{\epsilon }\right]}{\alpha }$$

We upper-bound the right-hand side similarly as in the proof of Theorem 1, which leads to $(B,f)\in {\mathcal{E}}_n$. □

4. Conclusions

In this paper, we conduct a theoretical study of a low-rank matrix single-index model. The model is used to estimate the link function and the coefficient matrix jointly. We leverage the PAC-Bayesian bounds technique to gain a deeper insight into the properties of this model and its potential applications. The study extends previous work in the field by considering a low-rank matrix, rather than a sparse vector, as the coefficient matrix. We also provide a detailed explanation of the choice of prior distributions for the link function and the coefficient matrix, which allows to obtain accurate and reliable estimates of the unknown parameters. Overall, this study provides a thorough theoretical understanding of the low-rank matrix single-index model.

The focus of future research would center on executing the proposed approach. There are various possible avenues to explore. One of the promising approaches is to use the reversible jump Markov chain Monte Carlo method, which was successfully applied in the past to address the sparse vector single-index model, as documented in [11].