High-Dimensional Precision Matrix Estimation through GSOS with Application in the Foreign Exchange Market

Abstract

This article studies the estimation of the precision matrix of a high-dimensional Gaussian network. We investigate the graphical selector operator with shrinkage, GSOS for short, to maximize a penalized likelihood function where the elastic net-type penalty is considered as a combination of a norm-one penalty and a targeted Frobenius norm penalty. Numerical illustrations demonstrate that our proposed methodology is a competitive candidate for high-dimensional precision matrix estimation compared to some existing alternatives. We demonstrate the relevance and efficiency of GSOS using a foreign exchange markets dataset and estimate dependency networks for 32 different currencies from 2018 to 2021.

1. Introduction

In recent years, covariance and precision matrix estimates have been studied extensively under high-dimensional scenarios. The motivation for these investigations is the argument that traditional likelihood-based techniques perform inaccurately or do not even exist. Examples of these include the graphical lasso algorithm and its extensions ([1,2,3,4,5,6,7]), and the ridge regularization-driven approach for the log-likelihood function ([8,9,10,11]). In addition to considering single $L_1$ (graphical lasso) and Frobenius (graphical ridge) penalties for the precision matrix, some studies have proposed combining both, resulting in an elastic net-type penalty. For instance, the following articles can be mentioned [10,12,13,14].

Despite the vast number of reported studies in the literature on the estimation of precision matrices, there are few reported studies concerning generalization to allow for the incorporation of prior knowledge of a precision matrix. For the Frobenius penalized case we can mentioned [9,10,11]. The only proposals for incorporating target matrices (prior knowledge) for the $L_1$ and elastic net penalty are those of [10,15]. In this paper, we propose the graphical selector operator with shrinkage (GSOS) as a combination of the generalized ridge (ridge with target matrix) and $L_1$ penalty. The main aim of this paper is to present and compare the new precision matrix estimator and propose a novel algorithm for Gaussian graphical models. The numerical study shows that the GSOS estimator is comparable with other available estimators and performs better in most cases than others.

Estimation, Literature Review

In graphical models, Gaussian graphical models are frequently used for modeling conditional dependencies in multivariate data. Dependency structures are determined by estimating the precision matrix through standard methodologies, for instance, a maximum likelihood or the regularized type of maximum likelihood in high-dimensional cases.

Consider random vector $\mathit{z}\sim {\mathcal{N}}_p(\mathbf{\mu },\mathsf{\Sigma })$; it follows a multivariate Gaussian distribution with mean vector $\mathbf{\mu }\in {\mathbb{R}}^p$ and positive definite covariance matrix $\mathsf{\Sigma }\in {\mathbb{R}}^p\times {\mathbb{R}}^p$ corresponding to conditional independence graph $\mathcal{G}$. In terms of mathematics, a pair $(\mathit{z},\mathcal{G})$ is a Gaussian graphical model. In these models, the graph $\mathcal{G}$ and the precision matrix $\mathsf{\Theta }:={\mathsf{\Sigma }}^{-1}$ are intimately related. That means that the zeroes in the precision matrix correspond to pairs of conditionally independent features, given all the other variables. In other words, there is no edge between these two variables (nodes) in a graph $\mathcal{G}$. Furthermore, two coordinates ${\mathit{z}}_k$ and ${\mathit{z}}_k^{{}^{\prime }}$ are independent if, in the graph $\mathcal{G}$, there is no edge between them. Maximum likelihood estimation is a logical technique to estimate the precision matrix for the positive definite matrix $\mathsf{\Theta }$; this estimator is expressed as

$$\widehat{\mathsf{\Theta }}=arg\underset{\mathsf{\Theta }\succ 0}{min}\{-logdet\left(\mathsf{\Theta }\right)+tr\left(\mathit{S}\mathsf{\Theta }\right)\},$$

(1)

where $\mathit{S}={\mathit{z}}^T\mathit{z}/n$ is the sample covariance matrix and the maximum likelihood estimator $\mathsf{\Theta }$ is equal to ${\mathit{S}}^{-1}$. However, two issues might emerge when utilizing this maximum likelihood technique to estimate $\mathsf{\Theta }$. First, in the high dimension case $p>n$, the empirical covariance matrix $\mathit{S}$ is singular and hence cannot be inverted to provide an estimate of $\mathsf{\Theta }$. Even if p and n are almost equal and $\mathit{S}$ is singular, the maximum likelihood estimate for $\mathsf{\Theta }$ will have a relatively high variance. Second, it is frequently valuable to identify pairs of disconnected variables in a graphical model that are conditionally independent; these correspond to zeroes in $\mathsf{\Theta }$. However, in general, (1) will produce an estimate of $\mathsf{\Theta }$ with no elements equal to zero.

The likelihood function is complemented with a penalty function in high-dimensional settings, which yields the maximum regularized likelihood estimator. Graphical lasso adds an $L_1$ penalty to log-likelihood and induces sparsity of $\mathsf{\Theta }$ due to maximizing the penalized log-likelihood:

$${\widehat{\mathsf{\Theta }}}_{Glasso}\left(\rho \right)=arg\underset{\mathsf{\Theta }\succ 0}{min}{\{-logdet\left(\mathsf{\Theta }\right)+tr\left(\mathit{S}\mathsf{\Theta }\right)+\rho |\left|\mathsf{\Theta }\right||}_1\},$$

(2)

where $\rho$ is a non-negative tuning parameter and ${\left|\right|\mathsf{\Theta }\left|\right|}_1$ denotes the sum of the absolute values of the elements of $\mathsf{\Theta }$. This approach was considered almost simultaneously by [1,16,17,18]. Due to the ensuing sparse answer, the graphical lasso estimate has gained significant interest and has become popular; it is still is an active area of research (cf. [6,19,20,21]). In contrast to circumstances where sparsity is advantageous, there are cases when more exact representations of the high-dimensional precision matrix are fundamentally desirable. Furthermore, the genuine (graphical) model does not have to be (very) sparse in terms of having many zero elements. In these situations, we may use a regularization approach that shrinks the estimated precision matrix elements instead of forcing them to be zero. The ridge precision estimator maximizes the log-likelihood augmented by a Frobenius norm, [9] in the most general form presented the ridge estimator as follows

$${\widehat{\mathsf{\Theta }}}_{Ridge}\left(\lambda \right)=arg\underset{\mathsf{\Theta }\succ 0}{min}\{-logdet\left(\mathsf{\Theta }\right)+tr\left(\mathit{S}\mathsf{\Theta }\right)+\frac{1}{2}\lambda \left|\right|\mathsf{\Theta }-{\mathit{T}\left|\right|}_F^2\},$$

(3)

where $\lambda$ is the penalty parameter, $\mathit{T}$ is a known symmetric and semi-positive definite target matrix, and ${\left|\right|\mathit{A}\left|\right|}_F^2$ denotes the Frobenius norm, the sum of the square values of the elements of the matrix $\mathit{A}$. Before estimating, the target matrix is set as an initial guess, toward which the precision estimate is shrunk. Estimator (3) is called the alternative Type I ridge precision estimator, and when the target matrix is equal to zero matrices, the alternative Type II ridge precision estimator. It should be mentioned that [11] considered the estimator mentioned in (3) independently and concurrently, and called it a ridge-type operator for precision matrix estimation (ROPE).

In a recent study in this area, Ref. [10] generalized the ridge inverse covariance estimator to allow for entry-wise penalization. Their proposed estimator shrinks toward a user-specified non-random target matrix and is shown to be positive definite and consistent. Furthermore, they obtained a generalization of the graphical lasso estimator and its elastic net counterpart. Recently, ref. [15] considered elastic net-type penalization (gelnet) for precision matrix estimation in the presence of a diagonal target matrix. They showed that it is possible to adopt the iterative procedure of [10] for the elastic net problem, but it is not computationally attractive.

Under the continuity property of the ridge estimator and apart from sparsity, estimations of the precision matrix with (3) can be far from the actual value $\mathsf{\Theta }$. Hence, the motivation of our approach to find a closer estimator. We use the Frobenius norm penalty to penalize the deviation between any primary or initial target estimator and $\mathsf{\Theta }$, and tune it to improve the accuracy of the final estimation. Furthermore, the $L_1$ penalty is added to account for sparsity, returning to the well-known elastic net with some added refinement. Therefore, in this paper, we consider combining the estimation approaches (2) and (3) to propose an elastic net-type estimator for the precision matrix: the GSOS.

2. The Proposed Method

Let $X_{n\times p}$ be the data matrix, including n observations of a $p—$dimensional Gaussian distribution with zero mean and positive definite covariance matrix $\mathsf{\Sigma }$. Consider the following type of estimation problem for estimating the unknown precision matrix $\mathsf{\Theta }={\mathsf{\Sigma }}^{-1}$ based on n observations:

$$\widehat{\mathsf{\Theta }}(\alpha ,\lambda ,\mathit{T})=arg\underset{\mathsf{\Theta }\succ 0}{min}{\{-logdet\left(\mathsf{\Theta }\right)+tr\left(\mathit{S}\mathsf{\Theta }\right)+\lambda (\alpha \left|\right|\mathsf{\Theta }\left|\right|}_1+\frac{(1-\alpha )}{2}\left|\right|\mathsf{\Theta }-{\mathit{T}\left|\right|}_F^2\left)\right\},$$

(4)

where ${\mathit{T}}_{p\times p}$ is a known symmetric and semi-positive definite target matrix. Additionally, $\lambda \ge 0$, $\alpha \in [0,1]$ are tuning parameters. Note that if $\alpha =0$, the precision matrix estimator (4) becomes the ridge estimator that is mentioned in (3).

Using sub-gradient notation and rules from [22], we obtain the following optimal conditions for (4)

$$\begin{array}{cc}\hfill & -{\mathsf{\Theta }}^{-1}+\mathit{S}+\lambda \alpha \mathsf{\Gamma }-\lambda (1-\alpha )(\mathit{T}-\mathsf{\Theta })=\mathbf{0}\hfill \\ \hfill i.e.,& -{\mathsf{\Theta }}^{-1}+\mathit{S}+\lambda \alpha \mathsf{\Gamma }-\lambda (1-\alpha )\mathit{T}+\lambda (1-\alpha )\mathsf{\Theta }=\mathbf{0}\hfill \\ \hfill i.e.,& -{\mathsf{\Theta }}^{-1}+\mathit{A}+\lambda \alpha \mathsf{\Gamma }+\lambda (1-\alpha )\mathsf{\Theta }=\mathbf{0},\hfill \end{array}$$

(5)

where $\mathit{A}:=\mathit{A}(\mathit{S},\alpha ,\lambda ,\mathit{T})=\mathit{S}-\lambda (1-\alpha )\mathit{T}$, and the matrix $\mathsf{\Gamma }=\left[{\gamma }_{k{k}^{{}^{\prime }}}\right]$ denotes the component-wise signs of $\mathsf{\Theta }$ with ${\gamma }_{k{k}^{{}^{\prime }}}\in [-1,1]$ for ${\mathsf{\Theta }}_{k{k}^{{}^{\prime }}}=0$. The solution of the normal Equation (5) is found by iteratively running over the columns/rows, considering the remaining ones fixed. The update requires that each matrix in (5) is partitioned as follows:

$$\mathsf{\Theta }=\left(\begin{array}{cc}{\mathsf{\Theta }}_{11}& {\mathit{\theta }}_{12}\\ {\mathit{\theta }}_{21}& {\theta }_{22}\end{array}\right),\mathsf{\Gamma }=\left(\begin{array}{cc}{\mathsf{\Gamma }}_{11}& {\mathbf{\gamma }}_{12}\\ {\mathbf{\gamma }}_{21}& {\gamma }_{22}\end{array}\right),\mathit{A}=\left(\begin{array}{cc}{\mathit{A}}_{11}& {\mathit{a}}_{12}\\ {\mathit{a}}_{21}& a_{22}\end{array}\right),$$

(6)

where ${\mathsf{\Theta }}_{11}$ is a square matrix, ${\mathit{\theta }}_{12}$ is a vector, and ${\theta }_{22}$ is a scalar. $\mathsf{\Gamma }$ and $\mathit{A}$ are partitioned similarly. Consider $\mathit{W}={\mathsf{\Theta }}^{-1}$; using the properties of inverses of block-partitioned matrices, we have that

$$\mathit{W}=\left(\begin{array}{cc}{\mathit{W}}_{11}& {\mathit{w}}_{12}\\ {\mathit{w}}_{21}& w_{22}\end{array}\right)=\left(\begin{array}{cc}{\mathsf{\Theta }}_{11}^{-1}+\frac{{\mathsf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}{\mathit{\theta }}_{21}{\mathsf{\Theta }}_{11}^{-1}}{({\theta }_{22}-{\mathit{\theta }}_{21}{\mathsf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12})}& -\frac{{\mathsf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}}{{\theta }_{22}-{\mathit{\theta }}_{21}{\mathsf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}}\\ -\frac{{\mathit{\theta }}_{21}{\mathsf{\Theta }}_{11}^{-1}}{{\theta }_{22}-{\mathsf{\Theta }}_{21}{\mathsf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}}& \frac{1}{{\theta }_{22}-{\mathit{\theta }}_{21}{\mathsf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}}\end{array}\right).$$

(7)

Considering the pth column of (5), we obtain

$$-{\mathit{w}}_{12}+{\mathit{a}}_{12}+\lambda \alpha {\mathbf{\gamma }}_{12}+\lambda (1-\alpha ){\mathit{\theta }}_{12}=\mathbf{0}.$$

(8)

By substituting ${\mathit{w}}_{12}$ and $w_{22}$ into (7), we have that

$$\begin{array}{cc}\hfill & \frac{{\mathsf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}}{{\theta }_{22}-{\mathit{\theta }}_{21}{\mathsf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}}+{\mathit{a}}_{12}+\lambda \alpha {\mathbf{\gamma }}_{12}+\lambda (1-\alpha ){\mathit{\theta }}_{12}=\mathbf{0}\hfill \\ \hfill i.e.,& {\mathsf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}{w}_{22}+{\mathit{a}}_{12}+\lambda \alpha {\mathbf{\gamma }}_{12}+\lambda (1-\alpha ){\mathit{\theta }}_{12}=\mathbf{0}\hfill \\ \hfill i.e.,& ({\mathsf{\Theta }}_{11}^{-1}{w}_{22}+\lambda (1-\alpha ){\mathit{I}}_{p-1}){\mathit{\theta }}_{12}+{\mathit{a}}_{12}+\lambda \alpha {\mathbf{\gamma }}_{12}=\mathbf{0}.\hfill \end{array}$$

(9)

Consider ${\tilde{\mathit{q}}}_{12}:=abs\left({\mathit{\theta }}_{12}\right)$ and $\tilde{\mathbf{\gamma }}:=\lambda \alpha {\mathbf{\gamma }}_{12}$; then, (9) is equivalent to

$$\begin{array}{cc}\hfill & ({\mathsf{\Theta }}_{11}^{-1}\frac{w_{22}}{\alpha \lambda }+\frac{1-\alpha }{\alpha }{\mathit{I}}_{p-1}){\tilde{q}}_{12}\ast \tilde{\mathbf{\gamma }}+{\mathit{a}}_{12}+\tilde{\mathbf{\gamma }}=\mathbf{0},\hfill \\ \hfill & {\tilde{\mathit{q}}}_{12}*(abs\left(\tilde{\mathbf{\gamma }}\right)-\lambda \alpha {\mathbf{1}}_{p-1})=\mathbf{0},\hfill \\ \hfill & \left|\right|\tilde{\mathbf{\gamma }}{\left|\right|}_{\infty }\le \lambda \alpha ,\hfill \end{array}$$

(10)

where ∗ denotes the element-wise multiplication of two vectors. The conditions mentioned in (10) are KKT optimality conditions for the following box-constrained problem for $\mathbf{\gamma }\in R^{p-1}$

$$\underset{\left|\right|\tilde{\mathbf{\gamma }}{\left|\right|}_{\infty }\le \lambda \alpha }{minimize}\frac{1}{2}{({\mathit{a}}_{12}+\tilde{\mathbf{\gamma }})}^T{({\mathsf{\Theta }}_{11}^{-1}\frac{w_{22}}{\alpha \lambda }+\frac{1-\alpha }{\alpha }{\mathit{I}}_{p-1})}^{-1}({\mathit{a}}_{12}+\tilde{\mathbf{\gamma }}).$$

(11)

By solving the optimization problem (11) and finding the optimum point ${\mathbf{\lambda }}^{*}$, ${\mathit{\theta }}_{12}$ can be updated as follows for all $\alpha \in (0,1]$

$$\begin{array}{c}\hfill {\widehat{\mathit{\theta }}}_{12}=-{({\mathsf{\Theta }}_{11}^{-1}\frac{w_{22}}{\alpha \lambda }+\frac{1-\alpha }{\alpha }{\mathit{I}}_{p-1})}^{-1}({\mathit{a}}_{12}+{\mathbf{\gamma }}^{*}).\end{array}$$

(12)

For the diagonal elements of the precision matrix ${\theta }_{22}$, we consider (5) for diagonal elements

$$-w_{22}+{\mathit{a}}_{22}+\lambda \alpha +\lambda (1-\alpha ){\theta }_{22}=0.$$

(13)

From (7), we have that $w_{22}=1/({\theta }_{22}-{\mathit{\theta }}_{21}{\mathsf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12})$; this implies that (13) is equivalent to

$$1+({\mathit{a}}_{22}+\lambda \alpha +\lambda (1-\alpha ){\theta }_{22})({\mathit{\theta }}_{21}{\mathsf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}-{\theta }_{22})=0.$$

(14)

Let $C:={\mathit{\theta }}_{21}{\mathsf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}$; therefore, we have this quadratic equation

$$-\lambda (1-\alpha ){\theta }_{22}^2+(\lambda (1-\alpha )C-a_{22}-\lambda \alpha ){\theta }_{22}+C(a_{22}+\lambda \alpha )+1=0.$$

(15)

Quadratic Equation (15) has two distinct real roots: positive (acceptable) and negative (unacceptable). Hence, we can update the diagonal elements of the precision matrix with the positive root. Finally, ${\mathit{w}}_{12}$ and $w_{22}$ are updated using normal Equation (5). Our method here is similar to the dpglasso estimator proposed by [5]. Furthermore, we can follow the glasso approach to solve the problem. The fundamental distinction between glasso and dpglasso is that in glasso, $\mathit{W}$ is not equal to the inverse of $\mathsf{\Theta }$. Furthermore, glasso deals with $\mathit{W}$, whereas dpglasso considers its inverse $\mathsf{\Theta }$. Finally, Algorithm 1 shows the procedure for $\alpha \in (0,1]$.

Algorithm 1 GSOS based on dpglasso approach

Initialize $\mathsf{\Theta }$, a diagonal matrix with the diagonal elements as follows, for $k=1,\dots ,p$ $${\theta }_{kk}=\frac{-(a_{kk}+\lambda \alpha )+\sqrt{{(a_{kk}+\lambda \alpha )}^2+4\lambda (1-\alpha )}}{2\lambda (1-\alpha )}$$ and $\mathit{W}=\mathit{A}+\lambda \alpha {\mathit{I}}_p+\lambda (1-\alpha )\mathsf{\Theta }$, Focus on the columns and repeatedly perform the following steps till convergence. (A)  Rearrange the rows/columns so that the target column is last (according to (6)); (B)  Solve (11) and update $$\begin{array}{c}\hfill {\widehat{\mathit{\theta }}}_{12}=-{({\mathsf{\Theta }}_{11}^{-1}\frac{w_{22}}{\alpha \lambda }+\frac{1-\alpha }{\alpha }{\mathit{I}}_{p-1})}^{-1}({\mathit{a}}_{12}+{\mathbf{\gamma }}^{*});\end{array}$$ (C)  Solve (15) to obtain ${\widehat{\theta }}_{22}$; (D)  Update ${\widehat{\mathit{w}}}_{12}={\mathit{a}}_{12}+{\mathbf{\gamma }}^{*}$; (E)  Update ${\widehat{w}}_{22}=a_{22}+\lambda \alpha +\lambda (1-\alpha ){\widehat{\theta }}_{22}$.

Glasso Scenario

The following relationship between $\mathit{W}$ and $\mathsf{\Theta }$ is established in glasso

$$\mathit{W}=\left(\begin{array}{cc}{\mathit{W}}_{11}& {\mathit{w}}_{12}\\ {\mathit{w}}_{21}& w_{22}\end{array}\right)=\left(\begin{array}{cc}{({\mathsf{\Theta }}_{11}-\frac{{\mathit{\theta }}_{12}{\mathit{\theta }}_{21}}{{\theta }_{22}})}^{-1}& -{\mathit{W}}_{11}\frac{{\mathit{\theta }}_{12}}{{\theta }_{22}}\\ -\frac{{\mathit{\theta }}_{21}}{{\theta }_{22}}{\mathit{W}}_{11}& \frac{1}{{\theta }_{22}}+\frac{{\mathit{\theta }}_{21}{\mathit{W}}_{11}{\mathit{\theta }}_{12}}{{\theta }_{22}^2}\end{array}\right).$$

(16)

Consider the pth column of (5) and substitute ${\mathit{w}}_{12}$ from (16):

$$\begin{array}{c}\hfill -{\mathit{w}}_{12}+{\mathit{a}}_{12}+\lambda \alpha {\mathbf{\gamma }}_{12}+\lambda (1-\alpha ){\mathit{\theta }}_{12}=\mathbf{0},\\ \hfill {\mathit{W}}_{11}\frac{{\mathit{\theta }}_{12}}{{\theta }_{22}}+{\mathit{a}}_{12}+\lambda \alpha {\mathbf{\gamma }}_{12}+\lambda (1-\alpha ){\mathit{\theta }}_{12}=\mathbf{0}.\end{array}$$

Define $\mathbf{\beta }=-\frac{{\mathit{\theta }}_{12}}{{\theta }_{22}}$; therefore

$${\mathit{W}}_{11}\mathbf{\beta }-{\mathit{a}}_{12}-\lambda \alpha {\mathbf{\gamma }}_{12}+\lambda (1-\alpha ){\theta }_{22}\mathbf{\beta }=\mathbf{0}.$$

(17)

Letting ${\mathit{W}}_{11}=\mathbf{V}$, ${\mathit{a}}_{12}=\mathbf{u}$ and ${\mathbf{\gamma }}_{12}=\mathbf{\rho }$, then, for every $i=1,\dots ,p$, we have that

$$\begin{array}{c}\hfill V_{ii}{\beta }_i+\sum _{k\ne i}^{p-1}{V}_{ki}{\beta }_k-u_i-\lambda \alpha {\rho }_i+\lambda (1-\alpha ){\theta }_{22}{\beta }_i=0.\end{array}$$

(18)

Subsequently, the update has the following form

$${\widehat{\beta }}_i\leftarrow \frac{S(u_i-{\sum }_{k\ne i}^{p-1}{V}_{ki}{\beta }_k,\alpha \lambda )}{V_{ii}+\lambda (1-\alpha ){\theta }_{22}},$$

(19)

where S is the soft-threshold operator $S(x,t)=sign\left(x\right){\left(\right|x|-t)}_{+}$. We cycle through the predictors until convergence. $\widehat{\beta }$ is the optimum point of the following quadratic problem

$$\underset{\mathbf{\beta }\in R^{p-1}}{minimize}\frac{1}{2}{\mathbf{\beta }}^T{\mathit{W}}_{11}\mathbf{\beta }-{\mathbf{\beta }}^T{\mathit{a}}_{12}-{\lambda \alpha \left|\right|\mathbf{\beta }\left|\right|}_1+\frac{\lambda (1-\alpha )}{2}{\theta }_{22}{\mathbf{\beta }}^T\mathbf{\beta },$$

(20)

which corresponds to the normal equations (17). After solving this quadratic problem and finding $\widehat{\mathbf{\beta }}$, we can update ${\widehat{\mathit{w}}}_{12}$. From (5) for the diagonal elements of ${\theta }_{22}$ of the precision matrix, we have that

$$-w_{22}+a_{22}+\lambda \alpha +\lambda (1-\alpha ){\theta }_{22}=0.$$

(21)

From (16) we have that $w_{22}=\frac{1}{{\theta }_{22}}+\frac{{\mathit{\theta }}_{21}{\mathit{W}}_{11}{\mathit{\theta }}_{12}}{{\theta }_{22}^2}=\frac{1}{{\theta }_{22}}+{\mathbf{\beta }}^T{\mathit{W}}_{11}\mathbf{\beta }$; hence, (21) is equivalent to

$$\lambda (1-\alpha ){\theta }_{22}^2-({\mathbf{\beta }}^T{\mathit{W}}_{11}\mathbf{\beta }-a_{22}-\alpha \lambda ){\theta }_{22}-1=0.$$

(22)

Equation (22) has two distinct real roots with positive (acceptable) and negative (unacceptable) signs. Hence, we can update the diagonal elements of the precision matrix with the positive root. Finally, ${\widehat{\mathit{\theta }}}_{12}$ and ${\widehat{w}}_{22}$ are updated by (5). Algorithm 2 briefly outlines these steps.

Algorithm 2 GSOS based on glasso approach

Initialize $\mathsf{\Theta }$, a diagonal matrix with the diagonal elements as follows, for $k=1,\dots ,p$ $${\theta }_{kk}=\frac{-(a_{kk}+\lambda \alpha )+\sqrt{{(a_{kk}+\lambda \alpha )}^2+4\lambda (1-\alpha )}}{2\lambda (1-\alpha )}$$ and $\mathit{W}=\mathit{A}+\lambda \alpha {\mathit{I}}_p+\lambda (1-\alpha )\mathsf{\Theta }$. Focus on the columns and repeatedly perform the following steps till convergence. (A)  Rearrange the rows/columns so that the target column is last; (B)  Solve (20) and update ${\widehat{\mathit{w}}}_{12}={\mathit{W}}_{11}\widehat{\mathbf{\beta }}$; (C)  Solve (22) to update ${\widehat{\theta }}_{22}$; (D)  Update ${\widehat{\mathit{\theta }}}_{12}=-{\widehat{\theta }}_{22}\widehat{\mathbf{\beta }}$; (E)  Update ${\widehat{w}}_{22}=a_{22}+\lambda \alpha +\lambda (1-\alpha ){\widehat{\theta }}_{22}$.

3. Simulation

In this section, the statistical performance of GSOS is compared to some other popular estimators (glasso [1], ROPE or alternative ridge [9,11], and graphical elastic net [15]).

We simulate the data from a multivariate Gaussian distribution ${\mathcal{N}}_p(\mathbf{0},\mathsf{\Sigma })$, where $\mathsf{\Sigma }=\left[{\sigma }_{k,k^{{}^{\prime }}}\right]$ and $\mathsf{\Theta }={\mathsf{\Sigma }}^{-1}=\left[{\theta }_{k,k^{{}^{\prime }}}\right]$ are $p\times p$ positive definite matrices. Six different models are used to compare the methods (see Appendix A for precision matrices). We evaluate these estimators on the six network structures with $p=20,50,100$ nodes and sample size of $n=50$ by implementing Algorithm 2.

• Network 1: A model with compound symmetry structure where ${\sigma }_{k,k}=1$ and ${\sigma }_{k,k^{{}^{\prime }}}={0.6}^2$ for $k\ne k^{{}^{\prime }}$. In this model, the covariance matrix is structured and non-sparse;
• Network 2. The prototype matrix ${\mathsf{\Theta }}_0$ is used to standardize the precision matrix $\mathsf{\Theta }$ to possess a unit diagonal. Let ${\mathsf{\Theta }}_0=\mathit{A}+a\mathit{I}$, where each off-diagonal entry in $\mathit{A}$ is generated independently and equals $0.5$ with probability $0.1$, or 0 with probability $0.9$. a is chosen such that the condition number of the matrix is equal to p. Here we have an unstructured and sparse precision matrix;
• Network 3. The precision matrix is defined $\mathsf{\Theta }=\frac{1}{n}{\mathbf{Y}}^T\mathbf{Y}$, where $\mathbf{Y}=\left[y_{k{k}^{{}^{\prime }}}\right]$ is an $n\times p$ matrix with $n=10,000$ and $y_{k{k}^{{}^{\prime }}}$ comes from $\mathcal{N}(0,1)$. The precision matrix of this model is unstructured and non-sparse;
• Network 4. A star model with ${\theta }_{k,k}=1$, ${\theta }_{1,k}={\theta }_{k,1}=0.1$ and ${\theta }_{k,k^{{}^{\prime }}}=0$ otherwise. This local area network has a structured and sparse precision matrix;
• Network 5. A moving average (MA) model with ${\sigma }_{k,k^{{}^{\prime }}}=1$, ${\sigma }_{k,k-1}={\sigma }_{k-1,k}=0.2$ and ${\sigma }_{k,k-2}={\sigma }_{k-2,k}={0.2}^2$. This covariance matrix is structured and sparse;
• Network 6. A diagonally dominant model. Consider $\mathit{B}=\frac{1}{2}(\mathit{A}+{\mathit{A}}^T)$, where $\mathit{A}$ is a $p\times p$ matrix with zero diagonal elements. Each off-diagonal element of $\mathit{A}$ is drawn from a standard uniform distribution. Compute a matrix $\mathit{D}=\frac{1}{\gamma }\mathit{B}$, where $\mathit{D}=\left[d_{k,k^{{}^{\prime }}}\right]$ and $\gamma$ is the largest row sum of the absolute values of the elements of the matrix $\mathit{B}$. Finally, each off-diagonal element of $\mathsf{\Sigma }$ is chosen as ${\sigma }_{k,k^{{}^{\prime }}}=d_{k,k^{{}^{\prime }}}$ and ${\sigma }_{k,k}=1+e_i$, where $e_i$ is drawn from uniform distribution with minimum 0 and maximum $0.1$. This covariance matrix is unstructured and non-sparse.

To calculate the performance measures for all the methods, 100 independent simulations for each network are performed and the average of several loss functions is calculated. The optimal tuning parameter $\lambda$ for each method is determined for each simulation run using five-fold cross-validation. For glasso, gelnet, and ROPE, we use the five-fold cross-validation found in the R-package “GLassoElnetFast” with a set value of $\lambda$ of 50 elements varying from $0.01$ to 10, and $\alpha$ with 20 elements varying from $0.01$ to $0.99$ for gelnet and our estimators. As a target matrix, assume identity matrix $\mathit{I}$ and a scalar matrix $\nu \mathit{I}$, where $\nu =p^2/tr\left(\mathit{S}\right)$.

3.1. Performance Measures

To calculate the performance of a given estimator $\widehat{\mathsf{\Theta }}$, we consider four loss functions that have been used widely in other research in this area (see, e.g., [7,11,14,15]).

Loss Functions

• The Kullback–Leibler loss: $KL=tr\left(\mathsf{\Sigma }\widehat{\mathsf{\Theta }}\right)-logdet\left(\mathsf{\Sigma }\widehat{\mathsf{\Theta }}\right)-p$;
• The $L_2$ loss: $L_2=\left|\right|\mathsf{\Theta }-\widehat{\mathsf{\Theta }}{\left|\right|}_F$;
• The quadratic loss: $QL=tr{(\mathsf{\Sigma }\widehat{\mathsf{\Theta }}-{\mathit{I}}_p)}^2$;
• The spectral norm loss: $SP=d_1$, where $d_1^2$ is the largest eigenvalue of the matrix ${(\mathsf{\Theta }-\widehat{\mathsf{\Theta }})}^2$.

To provide risk measures, the averages of these losses are determined for each method from 100 simulations. Figure 1, Figure 2, Figure 3 and Figure 4 show the final findings. The plot layout is taken from [11]; the columns that go along the little black dots represent loss means. The bars at the top of each column represent standard errors ($mean\pm SD$).

3.2. Simulation Results

The results for different networks are displayed in Figure 1, Figure 2, Figure 3 and Figure 4. We summarize some observations based on the results as follows:

• In general, as mentioned in [15], it is advantageous to include a target in the methods, since each approach works better with the appropriate target;
• Compared to other alternatives, GSOS is often a considerable contender for high-dimensional precision matrix estimation;
• The question of which target is more effective remains. However, in most cases, our simulations suggest that the identity target matrix works better.

4. Real Data

In the following section, we study market data provided by the Pacific Exchange Rate Service dataset (PERS), which is available on https://fx.sauder.ubc.ca/, accessed on 13 September 2022. PERS provides daily values of the currencies and commodities priced in various base currencies, known as numeraire. We consider 32 different currencies for the four years between 2018 and 2021. The names of these 32 currencies and their abbreviations are listed in

Table 1. List of abbreviations of the considered currencies.

Currencies
AUD	Australian dollar	MXN	Mexican peso
BRL	Brazilian real	MYR	Malaysian ringgit
CAD	Canadian dollar	NOK	Norwegian krone
CHF	Swiss franc	NZD	New Zealand dollar
CNY	Chinese renminbi	PEN	Peruvian nuevo sol
CZK	Czech koruna	PHP	Philippines peso
DKK	Danish krone	PLN	Polish zloty
EUR	Euro	RUB	Russian ruble
GBP	British pound	SAR	Saudi Arabian riyal
HKD	Hong Kong dollar	SEK	Swedish krona
HRK	Croatian kuna	SGD	Singapore dollar
HUF	Hungarian forint	THB	Thailand baht
IDR	Indonesian rupiah	TRY	Turkish lira
ISK	Icelandic krona	TWD	Taiwanese dollar
INR	Indian rupee	USD	US dollar
JPY	Japanese yen	ZAR	South African rand

.

We gather the data initially with the US dollar (USD) as numeraire and then, according to the initial data, calculate all currency exchange rates (for more detail, see [23,24]). After determining all exchange rates, using each of the 32 currencies as a numeraire, we compute the daily log returns of the remaining 31 currencies. We estimate the related covariance and precision matrices based on all possible exchange currencies (496 different exchanges). We apply our proposed estimator GSOS, with the scalar matrix as a target for four-year and annual data separately (the five-fold cross-validation is considered in selecting the tuning parameters). The estimated networks are presented in Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9; it should be mentioned that we consider 50 of the strongest partial correlations.

Real Data Results

To report the analytical results, we consider the different thresholds for partial correlation to explain the most important relations between considered currencies. We apply filtering by considering only ones that exceed those thresholds to explain how these networks are formed.

• 2018–2021 network: First, strong partial correlations appear between European currencies: DKK, HRK, CZK, PLN, HUF, CHF, and EUR. Additionally, we find a cluster that consists of HKD, SAR, CNY, TWD, SGD, and USD. Considering a smaller threshold, one notices the partial correlation between the Oceania couple of AUD and NZD and CAD, and the joining of EUR to the USD-based cluster, which connects European currencies to this cluster. Finally, a rather interesting cluster centered around a Latin American currency, MXN, appears, which consists of BRL, ZAR, and RUB;
• 2018 network: We observe strong correlations between European currencies: DKK, HRK, CZK, PLN, HUF, and EUR. The USD-based cluster here consists of HKD, PEN, TWD, SGD, CNY, THB, and MYR. The EUR currency connects these two major clusters. We find the relation between the Oceania couple AUD and NZD and CAD for a smaller threshold. In addition, a cluster between the Latin American, African, and Asian currencies MXN, ZAR, and RUB is noteworthy;
• 2019 network: Similar to the 2018 network, we have strong and connected European currencies: DKK, HRK, CZK, PLN, and HUF. The USD-based cluster is formed by HKD, SAR, TWD, SGD, CNY, and MYR at first; by considering a smaller threshold, we find other currencies such as PEN and THB. We observe that EUR connects these two clusters, and another cluster consists of AUD, NZD, NOK, and SEK;
• 2020 network: This year, we observe the USD cluster as follows: HKD, SAR, TWD, SGD, and CNY. We can see MYR, PHP, and THB as a disjointed clique (every two distinct vertices are adjacent). Regarding European currencies, EUR and CHF have a connecting role between this cluster and the USD-based cluster. The European currency cluster consists of DKK, HRK, CZK, PLN, and HUF. In addition, we observe two other clusters: the Oceania couple AUD and NZD with CAD and GBP, and MXN, BRL, ZAR, and RUB. These two clusters have a connection to European currencies by SEK;
• 2021 network: As previously, we observe that the USD-based cluster consists of HKD, SAR, TWD, CNY, and SGD. DKK, HRK, CZK, HUF, PLN, CHF, SEK, and EUR form a European currency cluster, which is connected to the USD cluster by EUR. In addition, we discover an interesting cluster centered around AUD consisting of NZD, CAD, NOK, and GBP. Considering a smaller threshold, we find more relations in the last cluster and considerable partial correlations between AUD, NZD, NOK, and SEK. This year, we see a strong partial correlation between GBP and SGD, which has never been observed before. Finally, we have a small cluster consisting of three Latin American, Asian, and African currencies: MXN, RUB, and ZAR.

5. Discussion

This section provides a summary of the paper and highlights the results. Additionally, we address some limitations of the research and gaps for future research. Finally, we review the findings and the results of the proposed method: GSOS.

5.1. Summary

In this paper, we proposed two methods to estimate a precision matrix in multivariate Gaussian settings. Estimating a precision matrix is one of the most critical tools for reconstructing an undirected graphical model expressing conditional dependencies between variables. Furthermore, obtaining an estimated network of partial correlations with a suitable rescaling of the precision matrix is possible.

We focused on the high-dimensional case to obtain regularized and sparse estimates considering the elastic net-type penalty. This penalty is a combination of the Frobenius norm when a target matrix is taken into account and the $L_1$ norm penalty to create sparsity. The first method, Algorithm 1, employs the dpglasso technique proposed by [5] to find penalized log-likelihood estimation. The second method, Algorithm 2, follows a similar approach to glasso, as studied by [1], and solves optimization problems by turning them into quadratic problems.

We conducted a simulation study to test the proposed algorithms using three different sample sizes and six common network structures reflecting various forms of conditional dependence. To calculate the performance measures for all the methods, we considered 100 independent simulations for each network; we presented the average of the Kullback–Leibler, $L_2$, quadratic, and spectral norm loss functions. The optimal tuning parameters for each method were determined using five-fold cross-validation for each simulation run.

Lastly, we presented an empirical study of the network between 32 currencies for the years between 2018 and 2021. We estimated annual and four-year networks according to these years. We evaluated the high-dimensional precision matrix in every yearly network in this real data study.

5.2. Contributions

We added sparsity to the alternative ridge estimator by considering the $L_1$ norm as an additional penalty. In this area, [15] also focused on the elastic net-type penalties for a Gaussian log-likelihood-based precision matrix estimation called gelnet. However, they simultaneously considered the target matrix for the $L_1$ and Frobenius norm. Therefore, we cannot obtain our estimator from gelnet when considering nonzero target matrices. These relations encouraged us to choose glasso, alternative ridge, and gelnet as competitor estimators to compare with our proposed estimator, GSOS.

We used the R programming language [25] and the “GLassoElnetFast” package for the simulation study; the codes are available on https://github.com/Azamkheyri/GSOS.git, accessed on 27 October 2022. In terms of simulation results, GSOS outperformed alternative ridge with most of the underlying structures in the high-dimensional case for all considered sample sizes. GSOS and gelnet, in most cases, behaved almost similarly, but in Network 4 for the Kullback–Leibler and $L_2$ risk measures, GSOS significantly outperformed gelnet.

On glasso: according to the Kullback–Leibler risk measure, GSOS and glasso performed similarly, except in Networks 1 and 2, where GSOS performed better. For the $L_2$ measure, GSOS performed better in Network 1, while for the quadratic risk measure, their behaviors are different; in three networks, GSOS outperformed glasso and for the remainder, glasso was better. Therefore, our proposed estimator is an efficient way to estimate precision matrices for the high-dimension Gaussian graphical models.

Finally, we added a real data example, the PERS dataset, and estimated four annual and one four-year dependency networks from 2018 to 2021. Since the log returns must be calculated using data from two consecutive days, we removed from our research those currencies for which there were at least ten missing values for exchange rates with the USD. Therefore, we estimated high-dimension precision matrices with GSOS for 32 currencies worldwide.

5.3. Strengths and Limitations

As we mentioned before, we considered the elastic net-type penalty for the Gaussian log-likelihood problem to estimate the precision matrix. The most important strength of our proposal is that this kind of penalty could help simultaneously obtain the advantages of the Frobenius norm and $L_1$-penalized estimations. We could not prove that our proposed estimator mathematically outperforms the known methods; this might be considered as a limitation. However, we illustrated the performance of the GSOS estimator with a simulation study under six different frequently used dependency networks in the literature.

Our simulation results are related to the considered network’s structure and performance measures; hence, they are not general. Furthermore, despite our proposed estimator’s considerable statistical performance, there is no overall winner.

Finally, because of the presence of missing data in the foreign exchange markets and the fact that log returns of the data have to be calculated from two consecutive days, only those currencies in which there were less than ten missing values for exchange rates based on USD were considered.

5.4. Future Work

For future research, we will consider studying the asymptotic behavior of our suggested estimators. In addition, while all the recommended estimators depend on the multivariate Gaussian assumption, we might also consider ways to do away with it by proposing distribution-free approaches or looking at other distributions to capture the data’s properties better. The relationship between the values of two tuning parameters—$\alpha$ and $\lambda$—might be worthwhile to investigate.

Moreover, the considered real data example has more potential for further study, for instance, considering long-term studies and interpreting the results based on geographic, economic, or political factors.

6. Conclusions

The proposed precision estimator is based on the $L_1$ penalty for sparsity on Gaussian log-likelihood and penalization with Frobenius norm shrinkage to an arbitrary non-random target value. We proposed two algorithms based on gradient descent (glasso) and the box-constrained quadratic program (dpglasso). Our approach is similar to gelnet by [15], but with some refinement; the estimator proposed here cannot be obtained from gelnet. Compared to other alternatives, the simulation study illustrated that our proposed strategy is a good competitor for high-dimensional precision matrix estimation. Additionally, we presented an empirical analysis of a network of 32 popular currencies and estimated yearly and four-year dependency networks.