A Log-Det Heuristics for Covariance Matrix Estimation: The Analytic Setup

Abstract

This paper studies a new nonconvex optimization problem aimed at recovering high-dimensional covariance matrices with a low rank plus sparse structure. The objective is composed of a smooth nonconvex loss and a nonsmooth composite penalty. A number of structural analytic properties of the new heuristics are presented and proven, thus providing the necessary framework for further investigating the statistical applications. In particular, the first and the second derivative of the smooth loss are obtained, its local convexity range is derived, and the Lipschitzianity of its gradient is shown. This opens the path to solve the described problem via a proximal gradient algorithm.

1. Introduction

The estimation of large covariance or precision matrices is a relevant challenge nowadays, due to the increasing availability of datasets composed of a large number of variables p compared to the sample size n in many fields. The urgency of this topic is testified by several recent books [1,2,3], and comprehensive reviews [4,5,6]. In this paper, we assume for the $p\times p$ covariance matrix ${\Sigma }^{*}$ a low rank plus sparse decomposition, that is

$${\Sigma }^{*}={\mathbf{L}}^{*}+{\mathbf{S}}^{*}=\mathbf{B}{\mathbf{B}}^{\prime }+{\mathbf{S}}^{*},$$

(1)

where ${\mathbf{L}}^{*}=\mathbf{B}{\mathbf{B}}^{\prime }={\mathbf{U}}_L{\mathsf{\Lambda }}_L{\mathbf{U}}_L^{\prime }$, ${\mathbf{U}}_L$ is a $p\times r$ matrix such that ${\mathbf{U}}_L^{\prime }{\mathbf{U}}_L={\mathbf{I}}_{\mathbf{r}}$, ${\mathsf{\Lambda }}_L$ is a $r\times r$ diagonal matrix, and ${\mathbf{S}}^{*}$ is element-wise sparse, i.e. it contains only $s\ll \frac{p(p-1)}{2}$ off-diagonal non-zero elements. Since [7] proposed their approximate factor model, structure (1) has become the reference model for many high-dimensional covariance matrix estimators, like POET [8].

The recovery of structure (1) is a statistical problem of primary relevance. Ref. [7] proposed to consistently estimate ${\mathbf{L}}^{*}$ (as $p\to \infty$) by means of principal component analysis (PCA, see [9]), assuming that the eigenvalues of ${\mathbf{L}}^{*}$ diverge with the dimension p while the eigenvalues of ${\mathbf{S}}^{*}$ remain bounded. [8] proposes to estimate ${\mathbf{L}}^{*}$ by the top r principal components of the sample covariance matrix ${\Sigma }_n$ (as $p\to \infty$) and to estimate ${\mathbf{S}}^{*}$ by thresholding their orthogonal complement. In [10], ${\mathbf{L}}^{*}$ and ${\mathbf{S}}^{*}$ are recovered by nuclear norm plus $l_1$ penalization, that is by computing

$$\left(\widehat{\mathbf{L}},\widehat{\mathbf{S}}\right)=arg\underset{\mathbf{L}⪰0,\mathbf{S}\succ 0}{min}\mathfrak{L}(\mathbf{L},\mathbf{S})+\mathfrak{P}(\mathbf{L},\mathbf{S}),$$

(2)

where $\mathfrak{L}(\mathbf{L},\mathbf{S})$ is a smooth loss function, $\mathfrak{P}(\mathbf{L},\mathbf{S})$ is a nonsmooth penalty function, $\mathbf{L}⪰0$ denotes positive semidefiniteness for $\mathbf{L}$ and $\mathbf{S}\succ 0$ denotes positive definiteness for $\mathbf{S}$. In particular, denoting by ${\lambda }_i\left(\mathbf{M}\right)$, $i=1,\dots ,p$, the eigenvalues of a $p\times p$ matrix $\mathbf{M}$ sorted in descending order, $\mathfrak{L}(\mathbf{L},\mathbf{S})=\frac{1}{2}{\parallel {\Sigma }_n-(\mathbf{L}+\mathbf{S})\parallel }_F^2$, $\mathfrak{P}(\mathbf{L},\mathbf{S})={\psi \parallel \mathbf{L}\parallel }_{*}+\rho {\parallel \mathbf{S}\parallel }_1$, where ${\parallel \mathbf{L}\parallel }_{*}={\sum }_{i=1}^p{\lambda }_i\left(\mathbf{L}\right)$ (the nuclear norm of $\mathbf{L}$), ${\parallel \mathbf{S}\parallel }_1={\sum }_{i=1}^p{\sum }_{j=i}^p\left|{\mathbf{S}}_{ij}\right|$ (the $l_1$ norm of $\mathbf{S}$), and $\psi$ and $\rho$ are non-negative threshold parameters.

The nuclear norm was first proposed in [11] as an alternative to PCA. Ref. [12] furnishes a proof that ${\psi \parallel \mathbf{L}\parallel }_{*}+\rho {\parallel \mathbf{S}\parallel }_1$ is the tightest convex relaxation of the original non-convex penalty $\psi \mathrm{rk}\left(\mathbf{L}\right)+{\rho \parallel \mathbf{S}\parallel }_0$. Ref. [13] proves that the $l_1$ norm minimization provides the sparsest solution to most large underdetermined linear systems, while [14] proves that the nuclear norm minimization provides guaranteed rank minimization under a set of linear equality constraints. Ref. [15] shows that $l_1$ norm minimization selects the best linear model in a wide range of situations. The nuclear norm has instead been used to solve large matrix completion problems, like in [16,17,18], and [19]. Nuclear norm plus $l_1$ norm minimization was first exploited in [20] to provide a robust version of PCA under grossly corrupted or missing data.

The pair of estimators (2) derived in [10] is named ALCE (ALgebraic Covariance Estimator). Although ALCE has many desirable statistical properties, there is room to further improve it by replacing $\frac{1}{2}{\parallel {\Sigma }_n-(\mathbf{L}+\mathbf{S})\parallel }_F^2$ by a different loss. The Frobenius loss optimizes in fact the entry by entry performance of $\widehat{\Sigma }$, while a loss able to explicitly control the spectrum estimation quality may be desirable. In this paper, we consider the loss

$$\mathfrak{L}(\mathbf{L},\mathbf{S})=\frac{1}{2}logdet({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime }),$$

(3)

where ${\Delta }_n=\Sigma -{\Sigma }_n$, and $\Sigma =\mathbf{L}+\mathbf{S}$. Heuristics (3) is controlled by the individual singular values of ${\Delta }_n$, because

$$logdet({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })=log\prod _{i=1}^p\left({\lambda }_i({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })\right)\le \sum _{i=1}^p(1+{\lambda }_i{\left({\Delta }_n\right)}^2)=p+\sum _{i=1}^p{\lambda }_i{\left({\Delta }_n\right)}^2,$$

(4)

and, therefore, it is better suited for the estimation of the underlying spectrum.

To the best of our knowledge, the mathematical properties of (3) have not been extensively studied. Analogously to the univariate context ($p=1$), (3) is not a convex function. According to ongoing works like [21], nonconvex problems may be approached either by searching for approximate solutions instead of global solutions, or by exploiting the geometric structure of the objective function. Furthermore, in this case, the idea of restricting the analysis to the convexity region of the objective, a region that may be indefinitely extended (see the concept of Extendable Local Strong Convexity in [22]), is the key to apply, for instance, existing proximal gradient algorithms for convex functions (see [23]). For this reason, in this paper we calculate the first and second derivatives of (3), we derive its range of local convexity, and the Lipschitzianity of (3) and of its gradient. This opens the path to using the usual proximal gradient algorithms (see [23]) to solve problem (2) with $\mathfrak{L}(\mathbf{L},\mathbf{S})$ as in (3).

2. Analytic Setup

We consider the objective function

$$\varphi (\mathbf{L},\mathbf{S})=\mathfrak{L}(\mathbf{L},\mathbf{S})+\mathfrak{P}(\mathbf{L},\mathbf{S}),$$

(5)

where $\mathfrak{L}(\mathbf{L},\mathbf{S})=\frac{1}{2}logdet({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })$ is the smooth part of $\varphi (\mathbf{L},\mathbf{S})$ and $\mathfrak{P}(\mathbf{L},\mathbf{S})={\psi \parallel \mathbf{L}\parallel }_{*}+\rho {\parallel \mathbf{S}\parallel }_1$ is the non-smooth (but convex) part of $\varphi (\mathbf{L},\mathbf{S})$. First, we calculate the derivative of the smooth component $\mathfrak{L}(\mathbf{L},\mathbf{S})$ wrt $\mathbf{L}$ and $\mathbf{S}$, which is

$$\frac{\delta \frac{1}{2}logdet({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })}{\delta \mathbf{L}}=\frac{\delta \frac{1}{2}logdet({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })}{\delta \mathbf{S}}={({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })}^{-1}{\Delta }_n.$$

(6)

Proof. 

Let us consider two generic $p\times p$ matrices $\mathbf{L}$ and $\mathbf{S}$, their sum $\Sigma =\mathbf{L}+\mathbf{S}$, and the matrix ${\Delta }_n=\Sigma -{\Sigma }_n$. Let us define the matrix function $\phi (\Sigma )={\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime }$ and the function $\varphi (\Sigma )=logdet\phi (\Sigma )$. We denote by ${\mathbf{e}}_i$ the i-th canonical basis vector, by ${\mathbf{e}}_i^l={\delta }_{il}$ its l-th element, and by ${\sigma }_{ij}$ the $ij$ entry of $\Sigma$. Then, following [24], for each $i,j=1,\dots ,p$, we can write

$$\begin{array}{c}\hfill \frac{\partial \Sigma }{\partial {\sigma }_{ij}}=m_{ij}={\mathbf{e}}_i^{\prime }{\mathbf{e}}_j\\ \hfill \frac{\partial }{\partial {\sigma }_{ij}}logdet\phi (\Sigma )=\mathrm{Tr}\left({\phi }^{-1}(\Sigma )\frac{\partial \phi }{\partial {\sigma }_{ij}}\right)\\ \hfill \frac{\partial \phi }{\partial {\sigma }_{ij}}={\Delta }_n\frac{\partial {\Delta }_n^{\prime }}{\partial {\sigma }_{ij}}+\frac{\partial {\Delta }_n}{\partial {\sigma }_{ij}}{\Delta }_n^{\prime }\\ \hfill \frac{\partial {\Delta }_n}{\partial {\sigma }_{ij}}=m_{ij}.\end{array}$$

Therefore,

$$\begin{array}{cc}\hfill \frac{\partial }{\partial {\sigma }_{ij}}logdet\phi (\Sigma )=& \mathrm{Tr}\left({\phi }^{-1}(\Sigma )\left[{\Delta }_n{m}_{ji}+m_{ij}{\Delta }_n^{\prime }\right]\right)\hfill \\ \hfill =& \mathrm{Tr}\left({\phi }^{-1}(\Sigma ){\Delta }_n{m}_{ji}\right)+\mathrm{Tr}\left({\phi }^{-1}(\Sigma )m_{ij}{\Delta }_n^{\prime }\right).\hfill \end{array}$$

Since for $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$ conformable matrices

$$\mathrm{Tr}\left(\mathbf{A}\mathbf{B}\mathbf{C}\right)=\mathrm{Tr}\left({\left(\mathbf{A}\mathbf{B}\mathbf{C}\right)}^{\prime }\right)=\mathrm{Tr}\left({\mathbf{C}}^{\prime }{\mathbf{B}}^{\prime }{\mathbf{A}}^{\prime }\right)=\mathrm{Tr}\left({\mathbf{A}}^{\prime }{\mathbf{B}}^{\prime }{\mathbf{C}}^{\prime }\right),$$

we get

$$\begin{array}{cc}\frac{\partial }{\partial {\sigma }_{ij}}logdet\phi (\Sigma )\hfill & =2\mathrm{Tr}\left({\phi }^{-1}(\Sigma ){\Delta }_n{m}_{ji}\right)\hfill \\ \hfill & =2\sum _{\nu }{\left({\phi }^{-1}(\Sigma ){\Delta }_n{m}_{ji}\right)}_{\nu \nu }\hfill \\ \hfill & =2\sum _{\nu ,\rho }{\left({\phi }^{-1}(\Sigma )\right)}_{\nu \rho }{\left({\Delta }_n{m}_{ji}\right)}_{\rho \nu }\hfill \\ \hfill & =2\sum _{\nu ,\rho ,\sigma }{\left({\phi }^{-1}(\Sigma )\right)}_{\nu \rho }{\Delta }_{n,\rho \sigma }{\left(m_{ji}\right)}_{\sigma \nu }.\hfill \end{array}$$

(7)

Finally, considering that

$${\left(m_{ji}\right)}_{\sigma \nu }={({\mathbf{e}}_j\otimes {\mathbf{e}}_i)}_{\sigma \nu }={\delta }_{j\sigma }{\delta }_{i\nu },$$

we get

$$\begin{array}{ccc}\hfill \frac{\partial }{\partial {\sigma }_{ij}}logdet\phi (\Sigma )& =& 2\sum _{\nu ,\rho ,\sigma }{\left({\phi }^{-1}(\Sigma )\right)}_{\nu \rho }{\Delta }_{n,\rho \sigma }{\delta }_{j\sigma }{\delta }_{i\nu }\hfill \\ & =& 2\sum _{\rho }{\left({\phi }^{-1}(\Sigma )\right)}_{i\rho }{\Delta }_{n,\rho j}\hfill \\ & =& 2{\left({\phi }^{-1}(\Sigma ){\Delta }_n\right)}_{ij}.\hfill \end{array}$$

To sum up,

$$\frac{\partial }{\partial \Sigma }\left[\frac{1}{2}logdet\phi (\Sigma )\right]=\frac{\partial }{\partial \mathbf{L}}\left[\frac{1}{2}logdet\phi (\Sigma )\right]=\frac{\partial }{\partial \mathbf{S}}\left[\frac{1}{2}logdet\phi (\Sigma )\right]={\phi }^{-1}(\Sigma ){\Delta }_n.$$

(8)

   □

In the following, we explicit the second derivative of $\mathfrak{L}(\mathbf{L},\mathbf{S})=\frac{1}{2}logdet\phi (\Sigma )$, with $\phi (\Sigma )=({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })$ and $\Sigma =\mathbf{L}+\mathbf{S}$:

$$\begin{array}{cc}\frac{{\partial }^2}{\partial {\sigma }_{ij}\partial {\sigma }_{hk}}\frac{1}{2}logdet\phi (\Sigma )\hfill & ={\left(\frac{1}{2}\mathrm{Hess}logdet\phi (\Sigma )\right)}_{ijhk}\hfill \\ \hfill & ={\delta }_{jk}{\left({\phi }^{-1}(\Sigma )\right)}_{ih}-\sum _{\mu ,\sigma }{\left({\phi }^{-1}(\Sigma )\right)}_{h\mu }{\Delta }_{\mu j}{\left({\phi }^{-1}(\Sigma )\right)}_{i\sigma }{\Delta }_{\sigma k}\hfill \\ \hfill & -{\left({\phi }^{-1}(\Sigma )\right)}_{ih}\sum _{\mu ,\lambda }{\left({\phi }^{-1}(\Sigma )\right)}_{\lambda \mu }{\Delta }_{\mu j}{\Delta }_{\lambda k}.\hfill \end{array}$$

(9)

More, if $\Sigma ={\Sigma }_n$, we get

$${\left(\frac{1}{2}\mathrm{Hess}logdet\phi (\Sigma )\right)}_{ijhk}={\delta }_{jk}\otimes {\delta }_{ih}={\left({\mathbf{I}}_p\otimes {\mathbf{I}}_p\right)}_{ijhk},$$

(10)

that is,

$$\frac{1}{2}\mathrm{Hess}logdet\phi (\Sigma )={\mathbf{I}}_p\otimes {\mathbf{I}}_p.$$

(11)

Proof. 

From [24], we write

$$\frac{{\partial }^2}{\partial {\sigma }_{ij}\partial {\sigma }_{hk}}\frac{1}{2}logdet\phi (\Sigma )=\frac{1}{2}\left(\mathrm{Tr}\left({\phi }^{-1}(\Sigma )\frac{{\partial }^2\phi (\Sigma )}{\partial {\sigma }_{ij}\partial {\sigma }_{hk}}\right)-\mathrm{Tr}\left({\phi }^{-1}(\Sigma )\frac{\partial \phi (\Sigma )}{\partial {\sigma }_{ij}}{\phi }^{-1}(\Sigma )\frac{\partial \phi (\Sigma )}{\partial {\sigma }_{hk}}\right)\right),$$

and we recall that

$$\frac{{\partial }^2\phi (\Sigma )}{\partial {\sigma }_{ij}\partial {\sigma }_{hk}}={\sigma }_{hk}{\sigma }_{ji}+{\sigma }_{ij}{\sigma }_{kh}.$$

Then, we can calculate

$$\begin{array}{cc}\hfill \mathrm{Tr}\left({\phi }^{-1}(\Sigma )\frac{{\partial }^2\phi (\Sigma )}{\partial {\sigma }_{ij}\partial {\sigma }_{hk}}\right)& =\mathrm{Tr}\left({\phi }^{-1}(\Sigma ){\sigma }_{hk}{\sigma }_{ji}+{\phi }^{-1}(\Sigma ){\sigma }_{ij}{\sigma }_{kh}\right)\\ & =\mathrm{Tr}\left({\phi }^{-1}(\Sigma ){\sigma }_{hk}{\sigma }_{ji}\right)+\mathrm{Tr}\left({\phi }^{-1}(\Sigma ){\sigma }_{ij}{\sigma }_{hk}\right)\\ & =2\mathrm{Tr}\left({\phi }^{-1}(\Sigma ){\sigma }_{hk}{\sigma }_{ji}\right)\\ & =2{\sum }_{\nu }\left({\phi }^{-1}(\Sigma ){\sigma }_{hk}{\sigma }_{ji}\right)\\ & =2{\delta }_{jk}{\left({\phi }^{-1}(\Sigma )\right)}_{ih}.\end{array}$$

(12)

The second summand

$$\mathrm{Tr}\left({\phi }^{-1}(\Sigma )\frac{\partial \phi (\Sigma )}{\partial {\sigma }_{ij}}{\phi }^{-1}(\Sigma )\frac{\partial \phi (\Sigma )}{\partial {\sigma }_{hk}}\right)$$

can be derived from (12) as

$$2\mathrm{Tr}\left({\phi }^{-1}(\Sigma )\Delta {\sigma }_{ji}{\phi }^{-1}(\Sigma )\Delta {\sigma }_{kh}\right)+2\mathrm{Tr}\left({\phi }^{-1}(\Sigma )\Delta {\sigma }_{ji}{\phi }^{-1}(\Sigma ){\sigma }_{hk}\Delta \right).$$

Equation (9) is consequently proved.    □

3. Local Convexity

The aim of this section is to determine the range of convexity for $\mathfrak{L}(\mathbf{L},\mathbf{S})=\frac{1}{2}logdet({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })$, ${\Delta }_n=\Sigma -{\Sigma }_n$, wrt to the semidefinite positive matrix ${\Delta }_n{\Delta }_n^{\prime }$. In the univariate context, the function $\frac{1}{2}lndet(1+x^2)$ is convex if and only if $\left|x\right|<\frac{1}{\sqrt{2}}$. In the multivariate context, it is therefore reasonable to suppose that a similar condition on ${\Delta }_n{\Delta }_n^{\prime }$ ensures local convexity. A proof can be given by showing the positive definiteness of the Hessian of $\mathfrak{L}(\mathbf{L},\mathbf{S})$ for some range of $\parallel {\Delta }_n{\Delta }_n^{\prime }\parallel$. In other words, we need to show that there exists a positive $\delta$ such that, whenever $\parallel {\Delta }_n{\Delta }_n^{\prime }\parallel <\delta$, the function $\frac{1}{2}logdet({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })$ is convex.

Lemma 1.

Given $0<\mu \le \frac{1}{3p}$, we have that the function

$$logdet\left({\mathbf{I}}_p+\mathbf{A}{\mathbf{A}}^{*}\right)$$

(13)

is convex on the set ${\mathcal{C}}_{\mu }=\left\{\mathbf{A}\right|\mathbf{A}\mathrm{is}\mathrm{a}\mathrm{real}p\times {p\mathrm{matrix},\parallel \mathbf{A}\parallel }_2\le \mu \}$ where ${\parallel \mathbf{A}\parallel }_2$ denotes the spectral norm of $\mathbf{A}$.

Proof. 

We proceed using the criterion of convexity estimating the second derivative with respect to t of

$$\varphi \left(t\right)=logdet\left({\mathbf{I}}_p+(t\mathbf{A}+(1-t)\mathbf{B}){(t\mathbf{A}+(1-t)\mathbf{B})}^{*}\right).$$

(14)

Let us recall that

$$\frac{d}{dt}logdet\mathbf{G}\left(t\right)=\mathrm{Tr}\left(\mathbf{G}{\left(t\right)}^{-1}{\mathbf{G}}^{\prime }\left(t\right)\right),$$

(15)

where $\mathbf{G}\left(t\right)$ is a differentiable square matrix-valued function and (15) holds for those values of t for which $\mathbf{G}\left(t\right)$ is invertible.

Furthermore, we have as well

$$\frac{d}{dt}\mathrm{Tr}\left(\mathbf{A}\left(t\right)\right)=\mathrm{Tr}(\frac{d}{dt}\mathbf{A}\left(t\right)),\frac{d}{dt}{\mathbf{A}}^{-1}=-{\mathbf{A}}^{-1}\left(\frac{d}{dt}\mathbf{A}\right){\mathbf{A}}^{-1},$$

(16)

for any differentiable square matrix-valued function $\mathbf{A}\left(t\right)$. (See [25] or [26] e.g., how to prove these identities).

Calling $\mathbf{G}\left(t\right)={\mathbf{I}}_p+(t\mathbf{A}+(1-t)\mathbf{B}){(t\mathbf{A}+(1-t)\mathbf{B})}^{*}$, we see that ${\displaystyle {\mathbf{G}}^{\prime }\left(t\right)=2t\mathsf{\Lambda }{\mathsf{\Lambda }}^{*}+\mathbf{R}}$ and ${\mathbf{G}}^{″}\left(t\right)=2\mathsf{\Lambda }{\mathsf{\Lambda }}^{*}$ where

$$\mathsf{\Lambda }=\mathbf{A}-\mathbf{B},\mathbf{R}=\mathbf{B}{\mathsf{\Lambda }}^{*}+\mathsf{\Lambda }{\mathbf{B}}^{*}$$

and $\mathbf{G}\left(t\right)=\frac{1}{3p\mu }{\mathbf{I}}_p+t^2\mathsf{\Lambda }{\mathsf{\Lambda }}^{*}+t\mathbf{R}+\mathbf{B}{\mathbf{B}}^{*}$.

Thus, applying (15) and (16) we get

$${\varphi }^{\prime }\left(t\right)=\mathrm{Tr}\left(\mathbf{G}{\left(t\right)}^{-1}{\mathbf{G}}^{\prime }\left(t\right)\right)$$

and

$${\varphi }^{″}\left(t\right)=\mathrm{Tr}\left({\left(\mathbf{G}{\left(t\right)}^{-1}{\mathbf{G}}^{\prime }\left(t\right)\right)}^{\prime }\right)=\mathrm{Tr}({\mathbf{G}}^{-1}{\mathbf{G}}^{″}-{\mathbf{G}}^{-1}{\mathbf{G}}^{\prime }{\mathbf{G}}^{-1}{\mathbf{G}}^{\prime })$$

(17)

Convexity will follow once we have proven that (17) is non-negative for every $t\in [0,1]$ and every $\mathbf{A}$ and $\mathbf{B}$ in ${\mathcal{C}}_{\mu }$.

Due to the circularity of the trace function we also have

$${\varphi }^{″}\left(t\right)=\mathrm{Tr}\left({\mathbf{G}}^{-1/2}{\mathbf{G}}^{″}{\mathbf{G}}^{-1/2}-{\mathbf{G}}^{-1/2}{\mathbf{G}}^{\prime }{\mathbf{G}}^{-1}{\mathbf{G}}^{\prime }{\mathbf{G}}^{-1/2}\right),$$

(18)

that is

$${\varphi }^{″}\left(t\right)=\mathrm{Tr}\left({\mathbf{G}}^{-1/2}{\mathbf{G}}^{″}{\mathbf{G}}^{-1/2}\right)-\mathrm{Tr}\left({\mathbf{G}}^{-1/2}{\mathbf{G}}^{\prime }{\mathbf{G}}^{-1/2}{\mathbf{G}}^{-1/2}{\mathbf{G}}^{\prime }{\mathbf{G}}^{-1/2}\right).$$

(19)

This can be written as

$${\varphi }^{″}\left(t\right)=2\mathrm{Tr}\left({\mathbf{G}}^{-1/2}\mathsf{\Lambda }{\mathsf{\Lambda }}^{*}{\mathbf{G}}^{-1/2}\right)-\mathrm{Tr}\left({\mathbf{G}}^{-1/2}(2t\mathsf{\Lambda }{\mathsf{\Lambda }}^{*}+\mathbf{R}){\mathbf{G}}^{-1/2}{\mathbf{G}}^{-1/2}(2t\mathsf{\Lambda }{\mathsf{\Lambda }}^{*}+\mathbf{R}){\mathbf{G}}^{-1/2}\right).$$

(20)

We recall that $\mathbf{G}\left(t\right)$ is self-adjoint so that denoting by $\mathbf{H}$ the matrix ${\mathbf{G}}^{-1/2}\mathsf{\Lambda }$ and $\mathbf{K}$ the matrix ${\mathbf{G}}^{-1/2}{\mathbf{G}}^{\prime }{\mathbf{G}}^{-1/2}$ we get that (19) can be written as

$${\varphi }^{″}\left(t\right)=2\mathrm{Tr}\left(\mathbf{H}{\mathbf{H}}^{*}\right)-\mathrm{Tr}\left(\mathbf{K}{\mathbf{K}}^{*}\right).$$

(21)

We also recall that $\mathrm{Tr}\left(\mathbf{A}{\mathbf{B}}^{*}\right)$ induces a scalar product to which the trace norm is attached:

$${\left|\right|\mathbf{A}\left|\right|}_{\mathrm{tr}}=\sqrt{\mathrm{Tr}\left(\mathbf{A}{\mathbf{A}}^{*}\right)}=\sum _i{\sigma }_i\left(\mathbf{A}\right),$$

where ${\sigma }_i\left(\mathbf{A}\right)$ are the singular values of $\mathbf{A}$. In particular we have ${\parallel \mathbf{A}\parallel }_2\le {\left|\right|\mathbf{A}\left|\right|}_{\mathrm{tr}}\le p{\parallel \mathbf{A}\parallel }_2$ for every $\mathbf{A}$. Now from (21) convexity can be checked as

$${\varphi }^{″}\left(t\right)={2\left|\right|\mathbf{H}\left|\right|}_{\mathrm{tr}}-{\left|\right|\mathbf{K}\left|\right|}_{\mathrm{tr}}\ge 0.$$

(22)

Let us consider

$${\parallel \mathbf{K}\parallel }_2=\parallel {\mathbf{G}}^{-1/2}{\mathbf{G}}^{\prime }{\mathbf{G}}^{-1/2}{\parallel }_2={\parallel {\mathbf{G}}^{-1/2}(2t\mathsf{\Lambda }{\mathsf{\Lambda }}^{*}+\mathbf{R}){\mathbf{G}}^{-1/2}\parallel }_2.$$

We have

$${\parallel \mathbf{K}\parallel }_2\le 2t\parallel {\mathbf{G}}^{-1/2}\mathsf{\Lambda }{\mathsf{\Lambda }}^{*}{\mathbf{G}}^{-1/2}{\parallel }_2+{\parallel {\mathbf{G}}^{-1/2}\mathbf{R}{\mathbf{G}}^{-1/2}\parallel }_2.$$

Notice that the spectral norm ${\parallel •\parallel }_2$ is self-adjoint, that is $\parallel {\mathbf{M}}^{*}{\parallel }_2={\parallel \mathbf{M}\parallel }_2$ for every $\mathbf{M}$ (see e.g., [27]). Then

$${\parallel \mathbf{K}\parallel }_2\le 2t\parallel \mathbf{H}{\mathbf{H}}^{*}{\parallel }_2+{\parallel {\mathbf{G}}^{-1/2}(\mathbf{B}{\mathsf{\Lambda }}^{*}+\mathsf{\Lambda }{\mathbf{B}}^{*}){\mathbf{G}}^{-1/2}\parallel }_2,$$

that is

$${\parallel \mathbf{K}\parallel }_2\le {2\parallel \mathbf{H}\parallel }_2\parallel {\mathbf{G}}^{-1/2}{\mathsf{\Lambda }\parallel }_2+2\parallel {\mathbf{G}}^{-1/2}{\mathsf{\Lambda }\parallel }_2{\parallel {\mathbf{G}}^{-1/2}\mathbf{B}\parallel }_2.$$

Thus,

$${\parallel \mathbf{K}\parallel }_2\le 2{\parallel \mathbf{H}\parallel }_2\left(\parallel {\mathbf{G}}^{-1/2}{\mathsf{\Lambda }\parallel }_2+{\parallel {\mathbf{G}}^{-1/2}\mathbf{B}\parallel }_2\right).$$

(23)

Assume now that $\mathbf{A},\mathbf{B}\in {\mathcal{C}}_{\mu }$: ${\parallel \mathbf{A}\parallel }_2\le \mu$, ${\parallel \mathbf{B}\parallel }_2\le \mu$. We deduce that ${\parallel \mathsf{\Lambda }\parallel }_2\le 2\mu$ and due to the structure of $\mathbf{G}\left(t\right)={\mathbf{I}}_p+\mathbf{Q}\left(t\right)\mathbf{Q}{\left(t\right)}^{*}$ we also have

$$\parallel {\mathbf{G}}^{-1/2}{\mathsf{\Lambda }\parallel }_2\le 2\mu ,{\parallel {\mathbf{G}}^{-1/2}\mathbf{B}\parallel }_2\le \mu .$$

(24)

Finally, we have

$${\parallel \mathbf{K}\parallel }_2\le 6\mu {\parallel \mathbf{H}\parallel }_2.$$

Going back to (22) we have

$${\varphi }^{″}\left(t\right)={2\left|\right|\mathbf{H}\left|\right|}_{\mathrm{tr}}-{\left|\right|\mathbf{K}\left|\right|}_{\mathrm{tr}}\ge {2\parallel \mathbf{H}\parallel }_2-{p\parallel \mathbf{K}\parallel }_2\ge 2(1-3p\mu ){\parallel \mathbf{H}\parallel }_2\ge 0,$$

(25)

since $0<\mu \le \frac{1}{3p}$.    □

By means of a simple change of variable, the following result can be proven.

Lemma 2.

For any $\delta >0$ the function

$$logdet\left({\delta }^{-2}{\mathbf{I}}_p+\mathbf{A}{\mathbf{A}}^{*}\right)$$

(26)

is convex on the closed ball ${\mathcal{C}}_{\delta }=\left\{\mathbf{A}\right|\mathbf{A}\mathrm{is}\mathrm{a}\mathrm{real}p\times {p\mathrm{matrix},\parallel \mathbf{A}\parallel }_2\le \frac{1}{3\delta p}\}$.

In conclusion, even though the function $logdet({\mathbf{I}}_p+\mathbf{A})$ is always concave, Lemma 2 shows that the function $logdet\left({\delta }^{-2}{\mathbf{I}}_p+\mathbf{A}{\mathbf{A}}^{*}\right)$ can be made locally convex into any ball centered in 0, just choosing a suitable $\delta$.

4. Lipschitz-Continuity

In this section, we prove the Lipschitzianity of the smooth function $\mathfrak{L}(\mathbf{L},\mathbf{S})=\frac{1}{2}lndet({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })$, and of its gradient function, $\frac{\delta \mathfrak{L}(\mathbf{L},\mathbf{S})}{\delta \mathbf{L}}=\frac{\delta \mathfrak{L}(\mathbf{L},\mathbf{S})}{\delta \mathbf{S}}={({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })}^{-1}{\Delta }_n$ (see (6)).

Lemma 3.

The function $\mathfrak{L}(\mathbf{L},\mathbf{S})=\frac{1}{2}lndet({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })$ is Lipschitz continuous in Euclidean norm with Lipschitz constant equal to 1:

$$|logdet\phi \left({\Sigma }_1\right)-logdet\phi \left({\Sigma }_2\right)|\le \parallel {\Sigma }_1-{\Sigma }_2{\parallel }_2.$$

(27)

Proof. 

Let us recall that $\mathbf{L}$ and $\mathbf{S}$ are two generic $p\times p$ matrices, $\Sigma =\mathbf{L}+\mathbf{S}$ is their sum, and ${\Delta }_n=\Sigma -{\Sigma }_n$. We reconsider the matrix function $\phi (\Sigma )={\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime }$ and the function $\varphi (\Sigma )=logdet\phi (\Sigma )$. We recall from (6) that

$$\frac{\partial }{\partial \Sigma }\left[\frac{1}{2}logdet\phi (\Sigma )\right]={\phi }^{-1}(\Sigma ){\Delta }_n.$$

Given two vectors $\mathbf{u},\mathbf{v}\in {\mathbb{R}}^p$, let us define the Euclidean inner product $\langle \mathbf{u},\mathbf{v}\rangle ={\mathbf{u}}^{\prime }\mathbf{v}$. We consider

$$\langle \phi (\Sigma )\mathbf{v},\mathbf{v}\rangle =\langle ({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })\mathbf{v},\mathbf{v}\rangle ={\left|\mathbf{v}\right|}^2+{\left|{\Delta }_n^{\prime }\mathbf{v}\right|}^2,$$

where $\left|\mathbf{v}\right|$ is the Euclidean norm of $\mathbf{v}\in {\mathbb{R}}^p$. Then we have

$${\left|\mathbf{v}\right|}^2+{\left|{\Delta }_n^{\prime }\mathbf{v}\right|}^2\le \frac{1}{2}\left(\frac{1}{{\delta }^2}{\left|\phi (\Sigma )\mathbf{v}\right|}^2+{\delta }^2{\left|\mathbf{v}\right|}^2\right),$$

via Cauchy-Schwarz, for any $\delta \in \mathbb{R}$. Now choose $\delta =\sqrt{2}$, then we have

$$4|{\Delta }_n^{\prime }{\mathbf{v}|}^2\le {\left|\phi (\Sigma )\mathbf{v}\right|}^2$$

for every $\mathbf{v}\in {\mathbb{R}}^p$. Noticing that $\phi (\Sigma )$ is invertible and plugging in the previous inequality $\mathbf{v}=\phi {(\Sigma )}^{-1}\mathbf{w}$, $\mathbf{w}\in {\mathbb{R}}^p$, we obtain

$$\underset{\mathbf{w}\ne \mathbf{0}}{max}\frac{|{\Delta }_n^{\prime }\phi {(\Sigma )}^{-1}\mathbf{w}|}{{\left|\mathbf{w}\right|}^2}\le \frac{1}{2}.$$

Now recall that (see [25] p. 312) that the spectral norm of a matrix $\mathbf{A}$, ${\parallel \mathbf{A}\parallel }_2$, can be computed also via the equality

$${\parallel \mathbf{A}\parallel }_2=\underset{\mathbf{x}\ne \mathbf{0}}{max}\frac{\left|\mathbf{A}\mathbf{x}\right|}{\left|\mathbf{x}\right|},$$

and that the spectral norm is self-adjoint (again see [25] p. 309), that is $\parallel {\mathbf{A}}^{\prime }{\parallel }_2={\parallel \mathbf{A}\parallel }_2$. Summing up, we have proved that

$${\parallel \phi (\Sigma )}^{-1}{\Delta }_n{\parallel }_2\le \frac{1}{2}.$$

(28)

This means that the gradient of $logdet\phi (\Sigma )$ is uniformly bounded and since $\Sigma \mapsto logdet\phi (\Sigma )$ is a smooth function we have that the Lipschitz condition is satisfied with Lipschitz constant equal to 1:

$$|logdet\phi \left({\Sigma }_1\right)-logdet\phi \left({\Sigma }_2\right)|\le \parallel {\Sigma }_1-{\Sigma }_2{\parallel }_2.$$

(29)

   □

We have proven that the function

$$\begin{array}{cc}\hfill \frac{{\partial }^2}{\partial {\sigma }_{ij}\partial {\sigma }_{hk}}\frac{1}{2}logdet\phi (\Sigma )& ={\delta }_{jk}{\left({\phi }^{-1}(\Sigma )\right)}_{ih}-\sum _{\mu ,\sigma }{\left({\phi }^{-1}(\Sigma )\right)}_{h\mu }{\Delta }_{n,\mu j}{\left({\phi }^{-1}(\Sigma )\right)}_{i\sigma }{\Delta }_{n,\sigma k}\hfill \\ \hfill & -{\left({\phi }^{-1}(\Sigma )\right)}_{ih}\sum _{\mu ,\lambda }{\left({\phi }^{-1}(\Sigma )\right)}_{\lambda \mu }{\Delta }_{n,\mu j}{\Delta }_{n,\lambda k}\hfill \\ \hfill & ={\delta }_{jk}{\left({\phi }^{-1}(\Sigma )\right)}_{ih}-{\left({\phi }^{-1}(\Sigma ){\Delta }_n\right)}_{hj}{\left({\phi }^{-1}(\Sigma ){\Delta }_n\right)}_{ik}\hfill \\ \hfill & -{\left({\phi }^{-1}(\Sigma )\right)}_{ih}\langle {\phi }^{-1}(\Sigma ){\Delta }_{n,j},{\Delta }_{n,k}\rangle \hfill \end{array}$$

(30)

is Lipschitz continuous.

Now, we prove that the function $\frac{\delta \mathfrak{L}(\mathbf{L},\mathbf{S})}{\delta \mathbf{L}}=\frac{\delta \mathfrak{L}(\mathbf{L},\mathbf{S})}{\delta \mathbf{S}}={({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })}^{-1}{\Delta }_n$ is Lipschitz continuous.

Lemma 4.

The function $\frac{\delta \mathfrak{L}(\mathbf{L},\mathbf{S})}{\delta \mathbf{L}}=\frac{\delta \mathfrak{L}(\mathbf{L},\mathbf{S})}{\delta \mathbf{S}}={({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })}^{-1}{\Delta }_n$ is Lipschitz continuous with Lipschitz constant equal to $\frac{5}{4}$:

$$\parallel F({\Delta }_n+ϵ\mathbf{H})-F\left({\Delta }_n\right){\parallel }_2\le \frac{5}{4}ϵ{\parallel \mathbf{H}\parallel }_2+O\left({ϵ}^2\right),$$

(31)

with $F\left({\Delta }_n\right)={\phi }^{-1}(\Sigma ){\Delta }_n={({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })}^{-1}{\Delta }_n$ and fix $ϵ>0$, for any $ϵ>0$.

Proof. 

Let us call $F\left({\Delta }_n\right)={\phi }^{-1}(\Sigma ){\Delta }_n={({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })}^{-1}{\Delta }_n$ and fix $ϵ>0$.

Let us compute

$$F({\Delta }_n+ϵ\mathbf{H})-F\left({\Delta }_n\right)={({\mathbf{I}}_p+({\Delta }_n+ϵ\mathbf{H}){({\Delta }_n+ϵ\mathbf{H})}^{\prime })}^{-1}({\Delta }_n+ϵ\mathbf{H})-{({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })}^{-1}{\Delta }_n.$$

We have

$${({\mathbf{I}}_p+({\Delta }_n+ϵ\mathbf{H}){({\Delta }_n+ϵ\mathbf{H})}^{\prime })}^{-1}({\Delta }_n+ϵ\mathbf{H})={({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime }+\mathsf{\Lambda })}^{-1}({\Delta }_n+ϵ\mathbf{H}),$$

with $\mathsf{\Lambda }=ϵ{\mathsf{\Lambda }}_0+{ϵ}^2{\mathsf{\Lambda }}_1$ and

$${\mathsf{\Lambda }}_0=\mathbf{H}{\Delta }_n^{\prime }+{\Delta }_n{\mathbf{H}}^{\prime },{\mathsf{\Lambda }}_1=\mathbf{H}{\mathbf{H}}^{\prime }.$$

Calling $\mathsf{\Psi }={\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime }$ we have

$${(\mathsf{\Psi }+\mathsf{\Lambda })}^{-1}={\left(\mathsf{\Psi }({\mathbf{I}}_p+{\mathsf{\Psi }}^{-1}\mathsf{\Lambda })\right)}^{-1}={({\mathbf{I}}_p+{\mathsf{\Psi }}^{-1}\mathsf{\Lambda })}^{-1}{\mathsf{\Psi }}^{-1},$$

so that we have

$$F({\Delta }_n+ϵ\mathbf{H})-F\left({\Delta }_n\right)={({\mathbf{I}}_p+{\mathsf{\Psi }}^{-1}\mathsf{\Lambda })}^{-1}{\mathsf{\Psi }}^{-1}({\Delta }_n+ϵ\mathbf{H})-{\mathsf{\Psi }}^{-1}{\Delta }_n.$$

Recalling that

$${({\mathbf{I}}_p+\mathbf{G})}^{-1}={\mathbf{I}}_p-\mathbf{G}+{({\mathbf{I}}_p+\mathbf{G})}^{-1}{\mathbf{G}}^2,$$

whenever ${\mathbf{I}}_p+\mathbf{G}$ is invertible, we have

$$F({\Delta }_n+ϵ\mathbf{H})-F\left({\Delta }_n\right)=({\mathbf{I}}_p-{\mathsf{\Psi }}^{-1}\mathsf{\Lambda }+{({\mathbf{I}}_p+{\mathsf{\Psi }}^{-1}\mathsf{\Lambda })}^{-1}{\left({\mathsf{\Psi }}^{-1}\mathsf{\Lambda }\right)}^2){\mathsf{\Psi }}^{-1}({\Delta }_n+ϵ\mathbf{H})-{\mathsf{\Psi }}^{-1}{\Delta }_n.$$

We develop in the powers of $ϵ$:

$$\begin{array}{ccc}\hfill F({\Delta }_n+ϵ\mathbf{H})-F\left({\Delta }_n\right)& =& ({\mathbf{I}}_p-{\mathsf{\Psi }}^{-1}(ϵ{\mathsf{\Lambda }}_0+{ϵ}^2{\mathsf{\Lambda }}_1)+({\mathbf{I}}_p+{\mathsf{\Psi }}^{-1}{(ϵ{\mathsf{\Lambda }}_0+{ϵ}^2{\mathsf{\Lambda }}_1)}^{-1})\hfill \\ & & {\left({\mathsf{\Psi }}^{-1}(ϵ{\mathsf{\Lambda }}_0+{ϵ}^2{\mathsf{\Lambda }}_1)\right)}^2){\mathsf{\Psi }}^{-1}({\Delta }_n+ϵ\mathbf{H})-{\mathsf{\Psi }}^{-1}{\Delta }_n\hfill \end{array}$$

A tedious but simple computation yields to

$$F({\Delta }_n+ϵ\mathbf{H})-F\left({\Delta }_n\right)=ϵ{\mathbf{T}}_1+O\left({ϵ}^2\right),$$

with

$${\mathbf{T}}_1={\mathsf{\Psi }}^{-1}\mathbf{H}-{\mathsf{\Psi }}^{-1}{\mathsf{\Lambda }}_0{\mathsf{\Psi }}^{-1}{\Delta }_n,$$

that is

$${\mathbf{T}}_1={\mathsf{\Psi }}^{-1}\mathbf{H}({\mathbf{I}}_p-{\Delta }_n^{\prime }{\mathsf{\Psi }}^{-1}{\Delta }_n)-{\mathsf{\Psi }}^{-1}{\Delta }_n{\mathbf{H}}^{\prime }{\mathsf{\Psi }}^{-1}{\Delta }_n.$$

The previous computations for the Lipschizianity of $logdet({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })$ gave us (see (28)) that

$$\parallel {\mathsf{\Psi }}^{-1}{\Delta }_n{\parallel }_2\le \frac{1}{2}.$$

It is also easy to check that

$$\parallel {\mathsf{\Psi }}^{-1}{\parallel }_2\le 1,$$

and that

$$\parallel {\mathbf{I}}_p-{\Delta }_n^{\prime }{\mathsf{\Psi }}^{-1}{\Delta }_n{\parallel }_2\le 1.$$

Putting all together we get

$$\parallel F({\Delta }_n+ϵ\mathbf{H})-F\left({\Delta }_n\right){\parallel }_2\le \frac{5}{4}ϵ{\parallel \mathbf{H}\parallel }_2+O\left({ϵ}^2\right),$$

(32)

such that we have proven that the directional derivative of the gradient is bounded in every direction by $5/4$, i.e., the gradient is Lipschitz as a function from $\mathcal{M}\left(p\right)$ to $\mathcal{M}\left(p\right)$, the vector space of $p\times p$ real matrices.    □

5. Discussion

In this paper, we have proved that the loss $\frac{1}{2}logdet({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })$ has good analytic properties for the purpose of optimization, provided that the matrix ${\Delta }_n$ fulfills certain conditions. As a consequence, by [23,28] and the supplement of [10], it follows that our analytic setup can provide a numerical solution to the problem

$$\underset{\mathbf{L}⪰0,\mathbf{S}\succ 0}{min}\frac{1}{2}logdet({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })+{\psi \parallel \mathbf{L}\parallel }_{*}+\rho {\parallel \mathbf{S}\parallel }_1,$$

(33)

by using proximal gradient algorithms. The local convexity of $logdet({\mathbf{I}}_p+{\Delta }_n{\Delta }_n^{\prime })$ is the key to apply first-order methods to solve (33). Following [23,28] and the supplement of [10], we derive the following solution Algorithm 1.

Such algorithm may be applied in many fields, like economics, finance, biology, genetics, health, climatology, social science, among others. In future research, we plan to properly develop the selection of threshold parameters, to study how local convexity may cope with the random nature of the sample error matrix ${\Delta }_n$, and to establish the consistency of the solution pair of (33).

Algorithm 1 Pseudocode to solve problem (33) given any input covariance matrix ${\Sigma }_n$.

(1) Set $({\mathbf{L}}_0,{\mathbf{S}}_0)=\frac{1}{2}(\mathrm{diag}\left({\Sigma }_n\right),\mathrm{diag}\left({\Sigma }_n\right))$, ${\eta }_0=1$. (2) Initialize ${\mathbf{Y}}_0={\mathbf{L}}_0$ and ${\mathbf{Z}}_0={\mathbf{S}}_0$. Set $t=1$. (3) For $t\ge 1$, repeat: (i)  calculate calculate ${\Delta }_{t,n}={\mathbf{Y}}_{t-1}+{\mathbf{Z}}_{t-1}-{\Sigma }_n$; (ii)  compute $\frac{\partial \frac{1}{2}logdet\left({\mathbf{I}}_p+{\Delta }_{t,n}{\Delta }_{t,n}^{\prime }\right)}{\partial {\mathbf{Y}}_{t-1}}=\frac{\partial \frac{1}{2}logdet\left({\mathbf{I}}_p+{\Delta }_{t,n}{\Delta }_{t,n}^{\prime }\right)}{\partial {\mathbf{Z}}_{t-1}}={({\mathbf{I}}_p+{\Delta }_{t,n}{\Delta }_{t,n}^{\prime })}^{-1}{\Delta }_{t,n}$. (iii)  apply the singular value thresholding (SVT, [29]) operator $T_{\psi }$ to ${\mathbf{E}}_{Y,t}={\mathbf{Y}}_{t-1}-\frac{1}{\ell }{({\mathbf{I}}_p+{\Delta }_{t,n}{\Delta }_{t,n}^{\prime })}^{-1}{\Delta }_{t,n}$, with $\ell =\frac{10}{4}$, and set ${\mathbf{L}}_t=T_{\psi }\left({\mathbf{E}}_{Y,t}\right)=\widehat{\mathbf{U}}{\widehat{\mathbf{D}}}_{\psi }{\widehat{\mathbf{U}}}^{\top }$. (v)  apply the soft-thresholding operator [30] $T_{\rho }$ to ${\mathbf{E}}_{Z,t}={\mathbf{Z}}_{t-1}-\frac{1}{\ell }{({\mathbf{I}}_p+{\Delta }_{t,n}{\Delta }_{t,n}^{\prime })}^{-1}{\Delta }_{t,n}$, with $\ell =\frac{10}{4}$, and set ${\mathbf{S}}_t=T_{\rho }\left({\mathbf{E}}_{Z,t}\right)$. (vi)  set $({\mathbf{Y}}_t,{\mathbf{Z}}_t)=({\mathbf{L}}_t,{\mathbf{S}}_t)+\left\{\frac{{\eta }_{t-1}-1}{{\eta }_t}\right\}\{({\mathbf{L}}_t,{\mathbf{S}}_t)-({\mathbf{L}}_{t-1},{\mathbf{S}}_{t-1})\}$ where ${\eta }_t=\frac{1}{2}+\frac{1}{2}\sqrt{1+4{\eta }_{t-1}^2}$. (vii)  stop if the convergence criterion $\frac{\parallel {\mathbf{L}}_t-{\mathbf{L}}_{t-1}{\parallel }_F}{1+\parallel {\mathbf{L}}_{t-1}{\parallel }_F}+\frac{\parallel {\mathbf{S}}_t-{\mathbf{S}}_{t-1}{\parallel }_F}{1+\parallel {\mathbf{S}}_{t-1}{\parallel }_F}\le \epsilon$. (4) Set ${\widehat{\mathbf{L}}}_{\mathrm{ALCE}}={\mathbf{Y}}_t$ and ${\widehat{\mathbf{S}}}_{\mathrm{ALCE}}={\mathbf{Z}}_t$.