Consistent Estimators of the Population Covariance Matrix and Its Reparameterizations

Abstract

For the high-dimensional covariance estimation problem, when ${lim}_{n\to \infty }p/n=c\in (0,1)$, the orthogonally equivariant estimator of the population covariance matrix proposed by Tsai and Tsai exhibits certain optimal properties. Under some regularity conditions, the authors showed that their novel estimators of eigenvalues are consistent with the eigenvalues of the population covariance matrix. In this paper, under the multinormal setup, we show that they are consistent estimators of the population covariance matrix under a high-dimensional asymptotic setup. We also show that the novel estimator is the MLE of the population covariance matrix when $c\in (0,1)$. The novel estimator is used to establish that the optimal decomposite $T_T^2$-test has been retained. A high-dimensional statistical hypothesis testing problem is used to carry out statistical inference for high-dimensional principal component analysis-related problems without the sparsity assumption. In the final section, we discuss the situation in which $p>n$, especially for high-dimensional low-sample size categorical data models in which $p>>n$.

1. Introduction

The problem of high-dimensional covariance estimation is one of the most interesting topics in statistics (Pourahmadi [1], Zagidullina [2]). Stein ([3,4]) investigated an orthogonally equivariant nonlinear shrinkage estimator for the population covariance matrix. Stein’s estimator has been considered as the gold standard, from which a significant strand of research on the orthogonally equivariant estimation of the covariance matrix was generated (see, e.g., Ledoit and Wolf [5,6,7,8]; Rajaratnam and Vincenzi [9] and the references therein).

Tsai and Tsai [10] focused their attention on rotation-equivariant estimators, showing that Stein’s estimator can be inadmissible when the dimension p is fixed. Under a high-dimensional asymptotic setup—namely, when both the sample size n and dimension p were sufficiently large with a concentration of $c={lim}_{n\to \infty }p/n,c\in (0,1)$— they re-examined the asymptotic optimal property of the estimators proposed by Stein [3] and Ledoit and Wolf [6]. Moreover, Tsai and Tsai [10] looked into the mechanism of the Mar$\check{c}$enko–Pastur equation (Silverstein [11]) to determine an explicit equality relationship of the quantiles of the limiting spectral distributions. They used the obtained equality to propose a new kind of orthogonally equivalent estimator for the population covariance matrix and showed that their novel estimators of the eigenvalues are consistent with the eigenvalues of the population covariance matrix. When $p/n\to c\in (0,1)$, they further showed that their proposed covariance estimator is the best orthogonally equivariant estimator for the population covariance matrix under the normalized Stein loss function. In contrast, both Stein’s estimator and the sample covariance matrix can be inadmissible.

In this context, the question regarding whether the consistent estimator of the population covariance matrix does or does not exist naturally arises. In this paper, we further show that the estimator proposed by Tsai and Tsai [10] is a consistent estimator of the population covariance matrix $\mathbf{\Sigma }$ when $p/n\to c\in [0,1)$. To achieve this, first, under the multinormal setup, we show that the components of spectral decomposition of the sample covariance matrix are the maximum likelihood estimators (MLEs) of the components of the population covariance matrix when the dimension p is fixed and the sample size n is large (i.e., $c=0$). This process is demonstrated in Section 3. Then, in Section 4, we extend the results of Section 3 to the case in which $p/n\to c\in (0,1)$: namely, to show that the novel estimator is not only consistent but is also the MLE of the population covariance matrix. Based on the proposed covariance estimator, the optimal decomposite $T_T^2$-test for a high-dimensional statistical hypothesis testing problem is established. This test can also be applied to make statistical inferences for high-dimensional principal component analysis (PCA)-related problems without the sparsity assumption, as shown in Section 5. In the final section, we discuss the situation in which $p>n$, even when $p>>n$.

2. Preliminary Notations

Let ${\mathbf{X}}_1,\dots ,{\mathbf{X}}_n$ be independent p-dimensional random vectors with a common multivariate normal distribution, $N_p(\mathbf{0},\mathbf{\Sigma })$. First, we assume that the dimension p is fixed in Section 2 and Section 3. A basic problem that is considered in the literature is the estimation of the $p\times p$ covariance matrix $\mathbf{\Sigma }$, which is unknown and assumed to be non-singular. It is also assumed that $n\ge p$, so that the sufficient statistic

$$\begin{array}{c}\hfill \mathbf{A}=\sum _{i=1}^n{\mathbf{X}}_i{\mathbf{X}}_i^{\top }\end{array}$$

(1)

is positive definite with a probability of one. In the literature, the estimators $\varphi \left(\mathbf{A}\right)$ of $\mathbf{\Sigma }$ are the functions of $\mathbf{A}$. The sample space $𝒮$, the parameter space $\Theta$, and the action space $\mathcal{A}$ are taken to be the set ${\mathcal{P}}_p$ of the $p\times p$ symmetric positive definite matrices. The general linear group $Gl\left(p\right)$ acts on the space ${\mathcal{P}}_p$. Note that $\mathbf{A}$ has a Wishart distribution, $W(\mathbf{\Sigma },n)$, and the maximum likelihood estimator (MLE) of $\mathbf{\Sigma }$ is expressed as follows:

$$\begin{array}{c}\hfill {\widehat{\mathbf{\Sigma }}}_{ML}=\mathbf{S},\mathrm{where}\mathbf{S}=n^{-1}\mathbf{A},\end{array}$$

(2)

which is unbiased (Anderson [12]).

We consider the invariant loss function L, i.e., L satisfies the condition that $L(g\varphi \left(\mathbf{A}\right)g^{\top },g\mathbf{\Sigma }{g}^{\top })=L(\varphi \left(\mathbf{A}\right),\mathbf{\Sigma })$ for all $g\in Gl\left(p\right)$. An estimator $\widehat{\mathbf{\Sigma }}$ is defined as $Gl\left(p\right)$-equivariant if $\widehat{\mathbf{\Sigma }}\left(\mathbf{G}\mathbf{A}{\mathbf{G}}^{\top }\right)=\mathbf{G}\widehat{\mathbf{\Sigma }}\left(\mathbf{A}\right){\mathbf{G}}^{\top },\forall \mathbf{G}\in Gl\left(p\right),\forall \mathbf{A}\in {\mathcal{P}}_p$. Suppose that $\mathbf{G}$ acts on ${\mathcal{P}}_p$, whereby the orbit through $x\in {\mathcal{P}}_p$ is the set $\mathbf{G}x=\left\{gx\right|g\in \mathbf{G}\}\subset {\mathcal{P}}_p$. This action is called transitive if $\Theta$ is one orbit, which is defined by the condition that if $\forall x,y\in \Theta$, there is some $g\in \mathbf{G}$ in which $gx=y$. It may then be easy to note the fact that if L is $Gl\left(p\right)$-invariant, $\widehat{\mathbf{\Sigma }}$ is $Gl\left(p\right)$-equivariant, and $\mathbf{G}$ acts transitively on ${\mathcal{P}}_p$; then, the risk function is constant on ${\mathcal{P}}_p$: $R(\widehat{\mathbf{\Sigma }},\mathbf{\Sigma })=R(\widehat{\mathbf{\Sigma }},\mathbf{I}),\forall \mathbf{\Sigma }\in {\mathcal{P}}_p$.

One of the most interesting loss functions was introduced by Stein [13]:

$$\begin{array}{c}\hfill L(\varphi \left(\mathbf{S}\right),\mathbf{\Sigma })=\mathrm{tr}{\mathbf{\Sigma }}^{-1}\varphi \left(\mathbf{S}\right)-\log \det {\mathbf{\Sigma }}^{-1}\varphi \left(\mathbf{S}\right)-p,\end{array}$$

(3)

where $\mathrm{tr}$ and $\det$ denote the trace and determinant of a matrix, respectively. Because $Gl\left(p\right)$ acts transitively on the space ${\mathcal{P}}_p$, the best $Gl\left(p\right)$-equivariant estimator exists. It can be easily determined that the MLE $\mathbf{S}$ of $\mathbf{\Sigma }$ is the best $Gl\left(p\right)$-equivariant estimator. The minimum risk is

$$\begin{array}{c}\hfill R_m({\widehat{\mathbf{\Sigma }}}_{ML},\mathbf{\Sigma })=\sum _{i=1}^p\{\log n-\mathcal{E}\left[\log {\chi }_{n-i+1}^2\right]\},\end{array}$$

(4)

where $\mathcal{E}\left[X\right]$ denotes the expectation of the random variable of X.

3. Optimal Estimators of $\mathbf{\Sigma }$ When It Is Reparameterized

As the general linear group $Gl\left(p\right)$ is not an amenable group, to study the minimax problem, James and Stein [14] reparameterized the parameter $\mathbf{\Sigma }\to \mathbf{\Theta }{\mathbf{\Theta }}^{\top },\mathbf{\Theta }\in G_T^{+}$, where $G_T^{+}$ denotes the group of $p\times p$ lower triangular matrices with positive diagonal elements; the loss function is also invariant under $G_T^{+}$. Using the Cholesky decomposition, we may express that $\mathbf{A}=\mathbf{T}{\mathbf{T}}^{\top }$, where $\mathbf{T}\in G_T^{+}$. As $G_T^{+}$ acts transitively on the space ${\mathcal{P}}_p$, the best $G_T^{+}$-equivariant estimator was proposed by James and Stein [14] as ${\widehat{\mathbf{\Sigma }}}_S=\mathbf{T}{\mathbf{D}}_S^{-1}{\mathbf{T}}^{\top }$, where ${\mathbf{D}}_S$ is a positive diagonal matrix with the elements $d_{Sii}=n+p-2i+1,i=1,\cdots ,p$. The minimum risk for the best $G_T^{+}$-equivariant estimator ${\widehat{\mathbf{\Sigma }}}_S$ is

$$\begin{array}{c}\hfill R_m({\widehat{\mathbf{\Sigma }}}_S,\mathbf{\Sigma })=\sum _{i=1}^p\{\log (n+p-2i+1)-\mathcal{E}\left[\log {\chi }_{n-i+1}^2\right]\}.\end{array}$$

(5)

This is because $G_T^{+}$ is the solvable group and, hence, it is amenable. Thus, Stein’s estimator ${\widehat{\mathbf{\Sigma }}}_S$ is the minimax.

3.1. The Stein Phenomenon

It is easy to see that $R({\widehat{\mathbf{\Sigma }}}_S,\mathbf{\Sigma })\le R({\widehat{\mathbf{\Sigma }}}_{ML},\mathbf{\Sigma })$; thus, the MLE $\mathbf{S}$ is inadmissible, and the estimator ${\widehat{\mathbf{\Sigma }}}_S$ should be used instead of $\mathbf{S}$. This is the well-known Stein phenomenon in the covariance estimation problem (for details, see Anderson [12]).

In order to determine why the Stein phenomenon occurs, which is when the MLE $\mathbf{S}$ of $\mathbf{\Sigma }$ is inadmissible, we began to think about the deeper meaning behind the Stein phenomenon. Tsai [15] extended Stein’s method to establish another minimax estimator. We explain this method briefly in the following. Let ${\mathbf{\Sigma }}_{\left(k\right)}$ and ${\mathbf{A}}_{\left(k\right)}$ be partitioned as

$$\begin{array}{c}\hfill {\mathbf{\Sigma }}_{\left(k\right)}=\left[\begin{array}{cc}{\sigma }_{\left(k\right)11}& {\mathbf{\Sigma }}_{\left(k\right)12}\\ {\mathbf{\Sigma }}_{\left(k\right)21}& {\mathbf{\Sigma }}_{\left(k\right)22}\end{array}\right]\mathrm{and}{\mathbf{A}}_{\left(k\right)}=\left[\begin{array}{cc}{a}_{\left(k\right)11}& {\mathbf{A}}_{\left(k\right)12}\\ {\mathbf{A}}_{\left(k\right)21}& {\mathbf{A}}_{\left(k\right)22}\end{array}\right],\end{array}$$

(6)

and for all $k=1,\cdots ,p$ with ${\mathbf{\Sigma }}_{\left(1\right)}=\mathbf{\Sigma }$ and ${\mathbf{A}}_{\left(1\right)}=\mathbf{A}$. Define

$$\begin{array}{c}\hfill {\mathbf{\Sigma }}_{(k+1)}={\mathbf{\Sigma }}_{\left(k\right)22:1}={\mathbf{\Sigma }}_{\left(k\right)22}-{\mathbf{\Sigma }}_{\left(k\right)21}{\mathbf{\Sigma }}_{\left(k\right)12}/{\sigma }_{\left(k\right)11}\end{array}$$

(7)

and

$$\begin{array}{c}\hfill {\mathbf{A}}_{(k+1)}={\mathbf{A}}_{\left(k\right)22:1}={\mathbf{A}}_{\left(k\right)22}-{\mathbf{A}}_{\left(k\right)21}{\mathbf{A}}_{\left(k\right)12}/a_{\left(k\right)11}.\end{array}$$

(8)

Note that the dimension of ${\mathbf{\Sigma }}_{(k+1)}$ is one less than that of ${\mathbf{\Sigma }}_{\left(k\right)}$, which denotes a process of successive diagonalization. Let

$$\begin{array}{c}\hfill {\mathbf{g}}_{\left(k\right)}=\left[\begin{array}{cc}1& \mathbf{0}\\ -{\mathbf{\Sigma }}_{\left(k\right)21}{\sigma }_{\left(k\right)11}^{-1}& \mathbf{I}\end{array}\right]\mathrm{and}{\mathbf{h}}_{\left(k\right)}=\left[\begin{array}{cc}1& \mathbf{0}\\ -{\mathbf{A}}_{\left(k\right)21}{a}_{\left(k\right)11}^{-1}& \mathbf{I}\end{array}\right],k=1,\cdots ,p.\end{array}$$

(9)

We then have the following:

$$\begin{array}{cc}\hfill {\tilde{\mathbf{\Sigma }}}_{\left(k\right)}& ={\mathbf{g}}_{\left(k\right)}{\mathbf{\Sigma }}_{\left(k\right)}{\mathbf{g}}_{\left(k\right)}^{\top }\hfill \\ \hfill & =\left[\begin{array}{cc}{\sigma }_{\left(k\right)11}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\left(k\right)22:1}\end{array}\right],\hfill \end{array}$$

(10)

and

$$\begin{array}{cc}\hfill {\tilde{\mathbf{A}}}_{\left(k\right)}& ={\mathbf{h}}_k{\mathbf{A}}_{\left(k\right)}{\mathbf{h}}_{\left(k\right)}^{\top }=\left[\begin{array}{cc}{a}_{\left(k\right)11}& \mathbf{0}\\ \mathbf{0}& {\mathbf{A}}_{\left(k\right)22:1}\end{array}\right],k=1,\cdots ,p.\hfill \end{array}$$

(11)

Let

$$\begin{array}{c}\hfill {\mathbf{\Sigma }}^{*}=\mathrm{Diag}({\sigma }_{\left(1\right)11},\cdots ,{\sigma }_{\left(p\right)11})\mathrm{and}{\mathbf{A}}^{*}=\mathrm{Diag}(a_{\left(1\right)11},\cdots ,a_{\left(p\right)11}).\end{array}$$

(12)

Consequently, $\mathbf{\Sigma }$ and $\mathbf{A}$ are individually transformed into the diagonal matrices ${\mathbf{\Sigma }}^{*}$ and ${\mathbf{A}}^{*}$ so that the one-to-one correspondences of $\mathbf{\Sigma }\leftrightarrow {\mathbf{\Sigma }}^{*}$ and $\mathbf{A}\leftrightarrow {\mathbf{A}}^{*}$ are established to allow $\varphi \left(\mathbf{A}\right)=\mathbf{D}{\mathbf{A}}^{*}$ for the Stein loss function and $\mathbf{D}\in D\left(p\right)$ to be the group of positive diagonal matrices. Based on the properties of the Wishart distribution (see Theorem 4.3.4, Theorem 7.3.4, and Theorem 7.3.6 of Anderson [12]), it is easy to note that $a_{\left(i\right)11}/{\sigma }_{\left(i\right)11},i=1,\cdots ,p,$ which are independent ${\chi }^2$ random variables with $n-i+1$ degrees of freedom. Let ${\mathbf{D}}_0$ be the diagonal matrix with elements $d_{0ii}=n-i+1,i=1,\cdots ,p$. We may then conclude that ${\mathbf{A}}^{*}$ is Wishart distribution with a mean matrix ${\mathbf{D}}_0{\mathbf{\Sigma }}^{*}$. Furthermore, it should be noted that all the p Jacobins of the transformation of $\mathbf{A}\to {\mathbf{A}}^{*}$ are one, and the Wishart density of $\mathbf{A}$ is equivalent to the Wishart density of ${\mathbf{A}}^{*}$. Thus, the Stein loss function is

$$\begin{array}{c}\hfill L(\varphi \left({\mathbf{A}}^{*}\right),{\mathbf{\Sigma }}^{*})=\mathrm{tr}{\mathbf{\Sigma }}^{*-1}\mathbf{D}{\mathbf{A}}^{*}-\log \det {\mathbf{\Sigma }}^{*-1}\mathbf{D}{\mathbf{A}}^{*}-p.\end{array}$$

(13)

As ${\mathbf{A}}^{*}$ also acts transitively on the space ${\mathcal{P}}_p$, the best $D\left(p\right)$-equivariant estimator can be expressed in the following form:

$$\begin{array}{c}\hfill {\widehat{\mathbf{\Sigma }}}^{*}={\mathbf{D}}_0^{-1}{\mathbf{A}}^{*}.\end{array}$$

(14)

Thus, the minimum risk for the estimator ${\widehat{\mathbf{\Sigma }}}_I^{*}$ is

$$\begin{array}{c}\hfill R_m({\widehat{\mathbf{\Sigma }}}^{*},{\mathbf{\Sigma }}^{*})=\sum _{i=1}^p\{\log (n-i+1)-\mathcal{E}\left[\log {\chi }_{n-i+1}^2\right]\}.\end{array}$$

(15)

As the group $D\left(p\right)$ is also solvable, we may conclude that ${\widehat{\mathbf{\Sigma }}}^{*}$ is a minimax. Based on (5) and (15), it is easy to see that $R_m({\widehat{\mathbf{\Sigma }}}^{*},{\mathbf{\Sigma }}^{*})\le R_m({\widehat{\mathbf{\Sigma }}}_S,\mathbf{\Sigma })$; hence, similarly to the Stein phenomenon, we may conclude that Stein’s estimator ${\widehat{\mathbf{\Sigma }}}_S$ is inadmissible, while the estimator ${\widehat{\mathbf{\Sigma }}}^{*}$ is admissible.

3.2. The Optimal Properties of the MLE

We may note that the MLE $\mathbf{S}$ of $\mathbf{\Sigma }$ is the best $Gl\left(p\right)$-equivariant estimator. James and Stein [14] used the Cholesky decomposition to parameterize the parameter $\mathbf{\Sigma }$ to obtain the Stein estimator ${\widehat{\mathbf{\Sigma }}}_S$, which is the best $G_T^{+}$-equivariant estimator. Tsai [15] used the full Iwasawa decomposition to obtain the best $D\left(p\right)$-equivariant estimator ${\widehat{\mathbf{\Sigma }}}^{*}$. It is important to note that the inequality $R_m({\widehat{\mathbf{\Sigma }}}^{*},{\mathbf{\Sigma }}^{*})\le R_m({\widehat{\mathbf{\Sigma }}}_S,\mathbf{\Sigma })\le R_m({\widehat{\mathbf{\Sigma }}}_{ML},\mathbf{\Sigma })$ holds. As $D\left(p\right)\subseteq G_T^{+}\subseteq Gl\left(p\right)$, we can easily see that the above inequality holds. The minimum risk of the estimator is larger for the larger group and smaller for the smaller group.

Tsai [15] showed that the minimum risks of the MLEs under the Cholesky decomposition and the full Iwasawa decomposition are the same when the geodesic distance loss function on a non-Euclidean space $P_p$ is adopted. Comparing the minimum risks of estimators for different groups does not make much statistical sense. The comparison of different estimators may make sense when they are compared under the same parameterized decomposition. For the spectral decomposition and when the dimension p is fixed, Tsai and Tsai [10] claimed that the sample covariance matrix $\mathbf{S}$ is the best orthogonally equivariant estimator of spectral decomposition under the Stein loss function, Stein ([3,4]) another orthogonally equivariant estimator can be inadmissible under spectral decomposition. These results are different from the Stein phenomenon, in which $\mathbf{S}$ is inadmissible. Hence, we cannot help but suspect that the Stein phenomenon is due to the parameterized decompositions and does not hold significant statistical meaning. We hope that this paper may impact those statisticians who have been constantly warned not to use an MLE for the covariance matrix ever since the Stein phenomenon occurred, making them reconsider the employment of an MLE for the covariance matrix.

When the dimension p is fixed, each of the three estimators possess their optimal properties for their respective parameterized decompositions. All three estimators—$\mathbf{S}$, ${\widehat{\mathbf{\Sigma }}}_S$, and ${\widehat{\mathbf{\Sigma }}}^{*}$—are the best $Gl\left(p\right)$-equivariant, $G_T^{+}$-equivariant, and $D\left(p\right)$-equivariant estimators, respectively. The sample covariance matrix $\mathbf{S}$ is not only the best $Gl\left(p\right)$-equivariant estimator but also the best $\mathcal{O}\left(p\right)$-equivariant estimator. They are the MLEs for $Gl\left(p\right)$, $G_T^{+}$, and $D\left(p\right)$ decompositions, respectively. The optimal property of the MLE is essentially not affected.

Note that the Stein loss function is essentially equivalent to the entropy loss function under the multinormal setup. When the dimension p is fixed and the sample size $n\to \infty$, the literature agrees that $\mathbf{S}$ and ${\widehat{\mathbf{\Sigma }}}^{*}$ will converge to $\mathbf{\Sigma }$ and ${\mathbf{\Sigma }}^{*}$, respectively, almost surely $(a.s.)$ when $n\to \infty$ (Anderson [12]). We extend Tsai’s [15] approach to the case of spectral decomposition conditions, i.e., to determine whether the sample covariance matrix $\mathbf{S}$ is the best orthogonally ($\mathcal{O}\left(p\right))$-equivariant estimator of the population covariance matrix $\mathbf{\Sigma }$ when the dimension p is fixed. In other words, we investigate whether $\mathbf{S}$ is the MLE of $\mathbf{\Sigma }$ under spectral decomposition conditions, so that the sample components converge to the corresponding population components $a.s.$ as $n\to \infty$.

3.3. The Best Orthogonally Equivariant Estimator

For the application to the statistical inference of principal component analysis, we need the notation of the so-called spectral decomposition of the population covariance matrix, which can be understood as another type of reparametrization of $\mathbf{\Sigma }$. Stein ([3,4]) considered the orthogonally equivariant estimator for the population covariance matrix, which has been considered the gold standard. Consider the spectral decomposition of the population covariance matrix: namely, $\mathbf{\Sigma }=\mathbf{V}\mathbf{\Gamma }{\mathbf{V}}^{\top }$, where $\mathbf{\Gamma }$ is a diagonal matrix with the eigenvalues ${\gamma }_{i,p},i=1,\dots ,p$, and $\mathbf{V}={({\mathbf{v}}_1,\dots ,{\mathbf{v}}_p)}^{\top }$ is the corresponding orthogonal matrix with ${\mathbf{v}}_i$ being the eigenvector that is associated to the ith largest eigenvalue ${\gamma }_{i,p},v_{i1}\ge 0,i=1,\dots ,p$. Similar dynamics apply to the sample spectral decomposition, i.e., $\mathbf{S}=\mathbf{U}\mathbf{L}{\mathbf{U}}^{\top }$, where $\mathbf{L}$ is a diagonal matrix with the eigenvalues $l_{i,p}$, and $\mathbf{U}={({\mathbf{u}}_1,\dots ,{\mathbf{u}}_p)}^{\top }$ is the corresponding orthogonal matrix, with ${\mathbf{u}}_i$ being the eigenvector corresponding to $l_{i,p},u_{i1}\ge 0,i=1,\dots ,p$. Write $\mathbf{L}=\mathrm{diag}(l_{1,p},\dots ,l_{p,p})$ and $\mathbf{\Gamma }=\mathrm{diag}({\gamma }_{1,p},\dots ,{\gamma }_{p,p})$. Note that the matrices $\mathbf{U}$ and $\mathbf{L}$ are the consistent estimators of $\mathbf{V}$ and $\mathbf{\Gamma }$, respectively, when the dimension p is fixed and the sample size n is large (for details, see Anderson [12]). Hence, we may conclude that there are two situations in which the dimension p is fixed: (i) When $\mathbf{\Sigma }$ is not reparameterized, the sample covariance matrix $\mathbf{S}$ is unbiased, and hence, it is consistent. (ii) When $\mathbf{\Sigma }$ is reparameterized via spectral decomposition, the components $\mathbf{U}$ and $\mathbf{L}$ are the consistent estimators of $\mathbf{V}$ and $\mathbf{\Gamma }$, respectively. Then, the sample covariance matrix $\mathbf{S}$ is still consistent.

 Remark 1. 

We aim to study the consistency property with the help of the optimal properties of MLEs. The main reason is that based on the general theory of estimation, the maximum likelihood estimator is consistent; that is, it tends toward the true value with a probability of one as the sample size increases under certain regularity conditions, which are satisfied by the non-degenerated Wishart distribution.

We may note that when $\mathbf{\Sigma }$ is not reparameterized, it is easy to see that the sample covariance matrix $\mathbf{S}$ is the MLE of $\mathbf{\Sigma }$. When the spectral decomposition for $\mathbf{\Sigma }$ is adopted, it is expected that the sample components $\mathbf{U}$ and $\mathbf{L}$ are the MLEs of the corresponding population components $\mathbf{V}$ and $\mathbf{\Gamma }$, respectively.

First, when the dimension p is fixed, $n\mathbf{S}$ is Wishart-distributed when $n>p$. Under the spectral decompositions for $\mathbf{\Sigma }$ and $\mathbf{S}$, we find the MLEs of $\mathbf{V}$ and $\mathbf{\Gamma }$ in the following. Note that $\mathbf{V},\mathbf{U}\in \mathcal{O}\left(p\right)$ constitutes the set of orthogonal matrices. Let $\mathbf{H}={\mathbf{V}}^{\top }\mathbf{U}$; then, $\mathbf{H}\in \mathcal{O}\left(p\right)$. Assume that $n\ge p+1$, and then, the $-{\displaystyle \frac{2}{n}}$ log-likelihood function of $\mathbf{S}$ is

$$\begin{array}{cc}\hfill l\left(\mathbf{S}\right|\mathbf{\Sigma })& =tr{\mathbf{\Sigma }}^{-1}\mathbf{S}-\log \det {\mathbf{\Sigma }}^{-1}\mathbf{S}-{\displaystyle \frac{2}{n}}\log c_n\left(\mathbf{S}\right)\hfill \\ \hfill & =tr\mathbf{V}{\mathbf{\Gamma }}^{-1}{\mathbf{V}}^{\top }\mathbf{U}\mathbf{L}{\mathbf{U}}^{\top }-\log \det \mathbf{V}{\mathbf{\Gamma }}^{-1}{\mathbf{V}}^{\top }\mathbf{U}\mathbf{L}{\mathbf{U}}^{\top }-{\displaystyle \frac{2}{n}}\log c_n\left(\mathbf{L}\right)\hfill \\ \hfill & =tr{\mathbf{\Gamma }}^{-1}\mathbf{H}\mathbf{L}{\mathbf{H}}^{\top }-\log \det {\mathbf{\Gamma }}^{-1}\mathbf{L}-{\displaystyle \frac{2}{n}}\log c_n\left(\mathbf{L}\right),\hfill \end{array}$$

(16)

where $c_n\left(\mathbf{S}\right)={\displaystyle \frac{n^{(n-p-1)/2}{\left|\mathbf{S}\right|}^{-(p+1)/2}}{2^{np/2}{\pi }^{p(p-1)/4}{\prod }_{i=1}^p\mathbf{\Gamma }\left[\frac{1}{2}(n-i+1)\right]}}=c_n\left(\mathbf{L}\right)$, which is independent of $\mathbf{\Sigma }$ (i.e., $\mathbf{V},\mathbf{\Gamma }$). Equation (16) is essentially equivalent to the Stein loss function.

 Theorem 1 

(von Neumann [16]). For $\mathbf{H}$ orthogonal and ${\mathbf{D}}_{\mathit{\gamma }}$ and ${\mathbf{D}}_{\mathbf{l}}$ diagonal (${\gamma }_1\ge \dots \ge {\gamma }_p>0,l_1>\dots >l_p>0$),

$$\begin{array}{c}\hfill {min}_{\mathbf{H}\in \mathcal{O}\left(p\right)}tr{\mathbf{D}}_{\mathit{\gamma }}^{-1}\mathbf{H}{\mathbf{D}}_{\mathbf{l}}{\mathbf{H}}^{\top }=tr{\mathbf{D}}_{\mathit{\gamma }}^{-1}{\mathbf{D}}_{\mathbf{l}},\end{array}$$

(17)

and a minimizing value of $\mathbf{H}$ is $\widehat{\mathbf{H}}=\mathbf{I}$. For the detailed proofs, see Theorem A.4.7 and Lemma A.4.6 by Anderson [12].

Based on the result of the von Neumann Theorem, we can determinate that the MLE of V is $\widehat{\mathbf{V}}=\mathbf{U}$ and, hence,

$$\begin{array}{cc}\hfill & {\min }_{\mathbf{V}\in \mathcal{O}\left(p\right)}\mathrm{tr}\mathbf{V}{\mathbf{\Gamma }}^{-1}{\mathbf{V}}^{\top }\mathbf{U}\mathbf{L}{\mathbf{U}}^{\top }\hfill \\ \hfill & =\mathrm{tr}{\mathbf{\Gamma }}^{-1}\mathbf{L}.\hfill \end{array}$$

(18)

Thus, we may state that

$$\begin{array}{c}\hfill {\min }_{\mathbf{V}\in \mathcal{O}\left(p\right)}l\left(\mathbf{S}\right|\mathbf{\Sigma })=\mathrm{tr}{\mathbf{\Gamma }}^{-1}\mathbf{L}-\log \det {\mathbf{\Gamma }}^{-1}\mathbf{L}-{\displaystyle \frac{2}{n}}\log c_n\left(\mathbf{L}\right).\end{array}$$

(19)

After some calculations, the function ${\min }_{\mathbf{V}\in \mathcal{O}\left(p\right)}l\left(\mathbf{S}\right|\mathbf{\Sigma })$ in (19) is further minimized with respect to $\mathbf{\Gamma }$ at $\widehat{\mathbf{\Gamma }}=\mathbf{L}$. As such, when p is fixed, we have that $\mathbf{U}$ is the MLE of $\mathbf{V}$, and $l_{i,p}$ is the MLE of ${\gamma }_{i,p},i=1,\dots ,p$. Thus, when p is fixed and the sample size n is large, according to the property of the MLE, we have $\mathbf{U}\to \mathbf{V}$ and $\mathbf{L}\to \mathbf{\Gamma }$ almost surely (a.s.); therefore, $\mathbf{U}$ and $\mathbf{L}$ are the consistent estimators of $\mathbf{V}$ and $\mathbf{\Gamma }$, respectively. Hence, $\mathbf{S}=\mathbf{U}\mathbf{L}{\mathbf{U}}^{\top }\to \mathbf{V}\mathbf{\Gamma }{\mathbf{V}}^{\top }=\mathbf{\Sigma }$ a.s. Therefore, in terms of the spectral decompositions, when the dimension p is fixed, the sample covariance matrix $\mathbf{S}$ is the consistent estimator of the population covariance matrix $\mathbf{\Sigma }$. Based on the above arguments, when the dimension p is fixed, we may note that the MLEs play an important role in achieving optimal conditions whether the dimension is reparameterized or not. When it is not reparameterized, the MLE $\mathbf{S}$ of $\mathbf{\Sigma }$ is unbiased and consistent, while when it is reparameterized, the MLEs of the component parameters for spectral decomposition are consistent.

However, the situation may be different when the dimension p is large, so that $c\in (0,1)$, because the sample covariance matrix $\mathbf{S}$ is no longer the MLE of the population covariance matrix $\mathbf{\Sigma }$. Hence, the question naturally arises as to whether the consistent estimator of $\mathbf{\Sigma }$ exists or not under the large dimensional asymptotic setup. Under the spectral decomposition, Tsai and Tsai [10] proved the consistency of their proposed estimators of population eigenvalues with the help of random matrix theory. Some notations of this are presented below.

4. High-Dimensional Case

For a large $(n,p)$ setup, a large dimensional asymptotic framework is set up when $(n,p)\to \infty$, so that $c={lim}_{n\to \infty }p/n$ is fixed, with $0\le c<1$. In this section, we extend the class of orthogonally equivariant estimators to the realm of large dimensional asymptotics with a concentration of $c\in (0,1)$.

4.1. The Mar$\check{c}$enko–Pastur Equation

In accordance with Ledoit and P$\acute{e}$ch$\acute{e}$ [17], we make the following assumptions:

A1. Note that ${\mathbf{x}}_i={\mathbf{\Sigma }}^{1/2}{\mathbf{z}}_i,i=1,\dots ,n$, where ${\mathbf{z}}_i$ values are independent and identically distributed with a mean of $\mathbf{0}$ and covariance matrix $\mathbf{I}$. Assume that the 12th absolute central moment of each variable $z_{ij}$ is bounded by a constant.

A2. The population covariance matrix $\mathbf{\Sigma }$ is nonrandom and positive definite. ${\lim \inf }_{p\to \infty }{\gamma }_{p,p}>0$, and ${\lim \sup }_{p\to \infty }{\gamma }_{1,p}<\infty$.

A3. For a large $(n,p)$ setup, the large dimensional asymptotic framework is established when $(n,p)\to \infty$, so that $c=p/n$ is fixed as $0\le c<1$ in this paper.

A4. Let $0<{\gamma }_{p,p}<\cdots <{\gamma }_{1,p}$. The empirical spectral distribution of $\mathbf{\Sigma }$, defined as $H_n\left(\gamma \right)={\displaystyle \frac{1}{p}}{\sum }_{i=1}^p{1}_{[{\gamma }_{i,p},\infty )}\left(\gamma \right)$, converges as $p\to \infty$ to a probability distribution function $H\left(\gamma \right)$ at every point of continuity of H. The support of H, $\mathrm{Supp}\left(H\right)$, is included in a compact set, $[h_1,h_2]$, with $0<h_1\le h_2<\infty$.

Let $F_n\left(\lambda \right)={\displaystyle \frac{1}{p}}{\sum }_{i=1}^p{1}_{[l_{i,p},\infty )}\left(\lambda \right)$ be the sample spectral distribution and F be its limiting factor. It is proved that $F_n$ converges to F a.s. as $n\to \infty$ (Mar$\check{c}$enko–Pastur [18]).

The Stieltjes transformation of distribution function F is defined as follows:

$$\begin{array}{c}\hfill m_F\left(z\right)={\int }_{-\infty }^{\infty }{\displaystyle \frac{1}{l-z}}dF\left(l\right),\forall z\in C^{+},\end{array}$$

(20)

where $C^{+}$ is the half-plane of complex numbers with a strictly positive imaginary part. Let

$$\begin{array}{c}\hfill m_{F_n}\left(z\right)=p^{-1}\mathrm{tr}\left[{(\mathbf{S}-z\mathbf{I})}^{-1}\right].\end{array}$$

(21)

Then, based on the results of random matrix theory, $F_n\left(z\right)$ converges to $F\left(z\right)$ if and only if $m_{F_n}\left(z\right)$ converges to $m_F\left(z\right)$. Subsequently, the well-known Mar$\check{c}$enko–Pastur equation (Silverstein [19]) can be expressed in the following form:

$$\begin{array}{c}\hfill m_F\left(z\right)={\int }_{-\infty }^{\infty }{\displaystyle \frac{1}{\gamma [1-c-cz{m}_F\left(z\right)]-z}}dH\left(\gamma \right),\forall z\in C^{+},\end{array}$$

(22)

where H denotes the limiting behavior of the population spectral distribution. Based on the Mar$\check{c}$enko–Pastur equation, meaningful information regarding the population spectral distribution can be retrieved under the large dimensional asymptotic framework. Choi and Silverstein [20] further showed that

$$\begin{array}{c}\hfill \underset{z\in C^{+}\to ł}{lim}{m}_F\left(z\right)={\check{m}}_F\left(l\right)\end{array}$$

(23)

exists for any $l\in R/\left\{0\right\}$.

Using the Sokhotski–Plemelj formula, the term ${\check{m}}_F\left(l\right)$ can be separated into the real part, which becomes a principal value integral (the so-called Hilbert transform), while the imaginary part becomes $\pi$ times the limiting sample spectral density function $f\left(l\right)$. Namely,

$$\begin{array}{c}\hfill {\check{m}}_F\left(l\right)=\mathrm{Re}\left[{\check{m}}_F\left(l\right)\right]+i\pi f\left(l\right),\end{array}$$

(24)

where the Hilbert transform denotes

$$\begin{array}{c}\hfill \mathrm{Re}\left[{\check{m}}_F\left(x\right)\right]=Pr\int {\displaystyle \frac{dF\left(t\right)}{t-x}}.\end{array}$$

(25)

In some special cases, ${\check{m}}_F\left(x\right)$ can be expressed explicitly. For example, let ${\lambda }_{+}={(1+\sqrt{c})}^2$ and ${\lambda }_{-}={(1-\sqrt{c})}^2$. When $\mathbf{\Sigma }=\mathbf{I}$, the Mar$\check{c}$enko–Pastur density function is of the following form:

$$\begin{array}{c}\hfill f_{MP}\left(x\right)={\displaystyle \frac{\sqrt{(x-{\lambda }_{-})({\lambda }_{+}-x)}}{2\pi cx}},x\in ({\lambda }_{-},{\lambda }_{+}).\end{array}$$

(26)

Using the resolvent method, we then have that

$$\begin{array}{c}\hfill {\check{m}}_F\left(x\right)={\displaystyle \frac{1-c-x+i\sqrt{(x-{\lambda }_{-})({\lambda }_{+}-x)}}{2cx}},\end{array}$$

(27)

where the real part is the Cauchy principal value, i.e.,

$$\begin{array}{cc}\hfill \mathrm{Re}\left[{\check{m}}_F\left(x\right)\right]& =Pr\int f_{MP}\left(t\right){\displaystyle \frac{dt}{t-x}}\hfill \\ \hfill & ={\displaystyle \frac{1-c-x}{2cx}}.\hfill \end{array}$$

(28)

Generally, $\mathbf{\Sigma }$ is unknown, and the form of $\mathrm{Re}\left[{\check{m}}_F\left(x\right)\right]$ will not be explicit.

Stein [3] used the naive empirical counterpart ${\check{m}}_{F_n}\left(l_{i,p}\right)(={\displaystyle \frac{1}{p}}{\sum }_{j\ne i}{\displaystyle \frac{1}{l_{j,p}-l_{i,p}}})$ to estimate the Hilbert transformation, $\mathrm{Re}\left[{\check{m}}_F\left(l_i\right)\right]$, where $l_i$ denotes the $(1-\alpha )$ quantile of the limiting sample spectral distribution F such that $\left[p\right(1-\alpha \left)\right]=i,i=1,\dots ,p$ with $\left[x\right]$ denoting the largest integer of x. As $F_n\left(z\right)$ converges to $F\left(z\right)$ a.s., meaning that $m_{F_n}\left(z\right)$ converges to $m_F\left(z\right)$ a.s., Stein concluded that $l_{i,p}$ converges to $l_{i}a.s.,withi=1,\dots ,p$. Then, the empirical counterpart ${\check{m}}_{F_n}\left(l_{i,p}\right)$ is a consistent estimator of the Hilbert transform.

4.2. The Consistent Estimators of Population Eigenvalues

The Mar$\check{c}$enko–Pastur equation in (22) shows the implicit relationship between F and H. Tsai and Tsai [10] further established the following explicit equality relationship:

$$\begin{array}{c}\hfill {\gamma }_i={\displaystyle \frac{l_i}{1-c-c{l}_i\mathrm{Re}\left[{\check{m}}_F\left(l_i\right)\right]}},i=1,\dots ,p,\end{array}$$

(29)

where ${\gamma }_i$ and $l_i$ denote the $(1-\alpha )$ quantiles of the limiting population and sample spectral distributions H and F, respectively, such that $\left[p\right(1-\alpha \left)\right]=i,i=1,\dots ,p$, with $\left[x\right]$ denoting the largest integer of x. Let $\mathrm{Supp}\left(F\right)$ be the support of F. Using Theorem 2 of Choi and Silverstein [19], Ledoit and P$\acute{e}$ch$\acute{e}$ [17] pointed out that if $l_i\notin \mathrm{Supp}\left(F\right)$, $l_i/1-c-c{l}_i\mathrm{Re}\left[{\check{m}}_F\left(l_i\right)\right]\notin \mathrm{Supp}\left(H\right)$ for $l_i\in R/\left\{0\right\},i=1,\dots ,p.$

Based on ${\gamma }_i={\psi }_i\left(\mathbf{L}\right),i=1,\dots ,p$ and the results of (29) and the empirical counterpart of $\mathrm{Re}\left[{\check{m}}_F\left(l_i\right)\right]$, Tsai and Tsai [10] proposed a new kind of orthogonally equivariant estimator, ${\widehat{\mathbf{\Sigma }}}_T$ of $\mathbf{\Sigma }$, which is of the following form:

$$\begin{array}{cc}\hfill {\widehat{\mathbf{\Sigma }}}_T=\mathbf{U}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right){\mathbf{U}}^{\top },& \mathrm{where}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right)=\mathrm{diag}({\widehat{\psi }}_1\left(\mathbf{L}\right),\cdots ,{\widehat{\psi }}_p\left(\mathbf{L}\right))\mathrm{with}\hfill \\ \hfill {\widehat{\psi }}_i\left(\mathbf{L}\right)& ={\displaystyle \frac{n{l}_{i,p}}{n-p+1-p{l}_{i,p}{\check{m}}_{F_n}\left(l_{i,p}\right)}}\hfill \\ \hfill & =n{l}_{i,p}{(n-p+1-l_{i,p}\sum _{j\ne i}{\displaystyle \frac{1}{l_{j,p}-l_{i,p}}})}^{-1},i=1,\dots ,p.\hfill \end{array}$$

(30)

When the dimension p is fixed and n is large (i.e., $c=0$), as discussed in Section 3, we have that ${\gamma }_{i,p}\to {\gamma }_i$ and ${\gamma }_i=l_i,i=1,\dots ,p$. However, when $p/n\to c\in (0,1)$, ${\gamma }_{i,p}\to {\gamma }_i$, and ${\gamma }_i$ is no longer $l_i$. In contrast, by means of Equation (29), it should be of the following form: ${\displaystyle \frac{l_i}{1-c-c{l}_i\left[\mathrm{Re}{\check{m}}_F\left(l_i\right)\right]}},i=1,\dots ,p.$ Note that based on assumption A4, $H_n$ converges to H when $c\in (0,1)$, and thus, ${\gamma }_{i,p}$ converges to ${\gamma }_i,i=1,\dots ,p$. In other words, $\mathbf{\Gamma }$ converges to $\mathbf{\Psi }\left(\mathbf{L}\right)$ as defined in Proposition 1. Hence, to estimate ${\gamma }_{i,p}$ is the same as to estimate ${\gamma }_i$ under the large dimensional asymptotic setup, $i=1,\dots ,p$. Under certain regularity conditions, Tsai and Tsai [10] claimed that their proposed estimators of the population eigenvalues are consistent. We summarize the results in the following.

 Proposition 1. 

Let $\mathbf{\Psi }\left(\mathbf{L}\right)=diag({\gamma }_1,\dots ,{\gamma }_p)$ and $\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right)=diag({\widehat{\psi }}_1\left(\mathbf{L}\right),\dots ,{\widehat{\psi }}_p\left(\mathbf{L}\right))$ be defined in (29) and (30), respectively. Under the assumptions of Theorem 2 of Tsai and Tsai [10], $\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right)$ is the then consistent estimator of $\mathbf{\Psi }\left(\mathbf{L}\right)$, namely, $\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right)$ is the consistent estimator of Γ, when $p/n\to c\in (0,1)$.

 Remark 2. 

In the literature, there are methods for imposing additional structure, such as sparse methods [20], the factor model [21], or a graph model [9], on the covariance matrix estimation. Orthogonally equivariant estimators are widely adopted. Using the Mar$\check{c}$enko–Pastur equation in (22), meaningful information on the population spectral distribution can be retrieved under the large dimensional asymptotic framework. However, the relationship is tangled. Under certain regularity conditions, Tsai and Tsai [10] established an explicit relationship between the quantiles of the limiting sample spectral distribution F and the limiting population spectral distribution H in (29). Then, the consistent estimators of the population eigenvalues can easily be established. This result makes up for the deficiency of estimators in the literature, such as Stein’s [4] and Ledoit and Wolf’s [5] estimators, which are inconsistent estimators of population eigenvalues. Tsai and Tsai [10] proposed a new kind estimator, ${\widehat{\mathbf{\Sigma }}}_T$, for the population covariance matrix  Σ  and showed that the proposed estimator is the best orthogonally ($\mathcal{O}\left(p\right))$-equivariant estimator of the population covariance matrix Σ under the normalized Stein loss function when the $c\in (0,1)$. Random matrix theory provided essential support for these results. However, it remains undetermined whether the proposed estimator is consistent for  Σ, namely, whether the sample component estimators are consistent for the corresponding population components. To investigate this, we adopt the MLE approach; thus, in this paper, we assume that ${\mathbf{X}}_i,i=1,\cdots ,n$ are independent and identically multinormally distributed.

4.3. The Consistent Estimator of the Population Covariance Matrix

When the dimension p is large, the sample covariance matrix $\mathbf{S}$ is no longer the MLE of $\mathbf{\Sigma }$. It is difficult to directly determined the functional form of $\mathbf{S}$ such that it is the MLE of $\mathbf{\Sigma }$; therefore, we may follow the detour of the reparametrization of $\mathbf{\Sigma }$ via spectral decomposition. Following this, the main goal is to see whether the orthonormal matrix $\mathbf{U}$ is the MLE of $\mathbf{V}$ or not. The result of $\mathcal{E}\mathbf{U}=\mathbf{V}$ implies that the limiting distribution of $\mathbf{U}$ on $\mathcal{O}\left(p\right)$ is entirely concentrated at $\mathbf{V}$, and unbiasedness is not a useful optimal property, the role of which would be replaced by the property of equivariance. Ledoit and P$\acute{e}$ch$\acute{e}$ [17] pointed out that the projection of the sample eigenvector onto the population eigenvector, $p|{\mathbf{u}}_i^{\top }{\mathbf{v}}_j{|}^2$, will wipe out the non-rotation-equivariant behavior and the average of the quantities of $p|{\mathbf{u}}_i^{\top }{\mathbf{v}}_j{|}^2$ over the sample eigenvectors that are associated with the sample eigenvalues, demonstrating how the eigenvectors of the sample covariance matrix deviate from those of the population covariance matrix under large dimensional asymptotics. This is one of the main reasons why we prefer to restrict it to the class of rotation-equivariant estimators. Tsai and Tsai [10] established the best orthogonally equivariant estimator ${\widehat{\mathbf{\Sigma }}}_T$ for $\mathbf{\Sigma }$. We continue to study whether the proposed estimator ${\widehat{\mathbf{\Sigma }}}_T$ is the consistent estimator of the population covariance matrix $\mathbf{\Sigma }$ or not when $p/n\to c\in (0,1)$. Based on Proposition 1, it only needs to be determined whether $\mathbf{U}$ is the consistent estimator of $\mathbf{V}$.

The orthogonal matrix $\mathbf{U}$ may not generally be a consistent estimator of $\mathbf{V}$ when the dimension p is large (see Bai et al. [22] and references therein). Hence, we may process it under the restricted model, namely, under the Wishart distribution setup, when $p/n\to c\in (0,1)$.

When $\mathbf{\Sigma }$ is reparameterized via the spectral decomposition, we aim to study the consistency property of the component parameters when the dimension p is large. Under the multivariate normal setup, when the dimension p is fixed, $n\mathbf{S}$ is Wishart-distributed with the mean matrix $\mathbf{\Sigma }$. However, when ${lim}_{n\to \infty }p/n=c\in (0,1)$, $n\mathbf{S}$ is not Wishart-distributed with the mean matrix $\mathbf{\Sigma }$, and nor is $\mathbf{S}$ the MLE of $\mathbf{\Sigma }$. Instead, we may notice that $n{\widehat{\mathbf{\Sigma }}}_T$ is Wishart-type-distributed with the mean matrix $\mathbf{\Sigma }$ when $p/n\to c\in (0,1)$. It is easy to note that the $-{\displaystyle \frac{2}{n}}$ log-likelihood function $l\left({\widehat{\mathbf{\Sigma }}}_T\right|\mathbf{\Sigma })$ of ${\widehat{\mathbf{\Sigma }}}_T$ is similar to (16) with ${\widehat{\mathbf{\Sigma }}}_T$ in (30) replacing $\mathbf{S}$ in (16) (i.e., with $\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right)$ replacing $\mathbf{L}$), which still satisfies the regularity conditions, and it does not degenerate. Based on $l\left({\widehat{\mathbf{\Sigma }}}_T\right|\mathbf{\Sigma })$, our goal is to show that ${\widehat{\mathbf{\Sigma }}}_T$ is the MLE of $\mathbf{\Sigma }$ when ${lim}_{n\to \infty }p/n=c\in (0,1)$.

First, we want to show that $\mathbf{U}$ is the MLE of $\mathbf{V}$ when $p/n\to c\in (0,1)$, namely, to extend the von Neumann Theorem to the case in which $p/n\to c\in (0,1)$. Note that $\mathbf{H}={\mathbf{V}}^{\top }\mathbf{U}\in \mathcal{O}\left(p\right)$. Thus, ${\mathrm{m}in}_{\mathbf{V}\in \mathcal{O}\left(p\right)}l\left({\widehat{\mathbf{\Sigma }}}_T\right|\mathbf{\Sigma })$ is equal to ${\mathrm{m}in}_{\mathbf{H}\in \mathcal{O}\left(p\right)}l\left({\widehat{\mathbf{\Sigma }}}_T\right|\mathbf{\Sigma })$. Note that $\mathbf{H}{\mathbf{H}}^{\top }=\mathbf{I}$ implies that $d\mathbf{H}{\mathbf{H}}^{\top }+\mathbf{H}d{\mathbf{H}}^{\top }=\mathbf{0}$. Moreover,

$$\begin{array}{cc}\hfill d\mathrm{tr}{\mathbf{\Gamma }}^{-1}\mathbf{H}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right){\mathbf{H}}^{\top }& =\mathrm{tr}[{\mathbf{\Gamma }}^{-1}d\mathbf{H}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right){\mathbf{H}}^{\top }+{\mathbf{\Gamma }}^{-1}\mathbf{H}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right)d{\mathbf{H}}^{\top }]\hfill \\ \hfill & =\mathrm{tr}[{\mathbf{\Gamma }}^{-1}d\mathbf{H}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right){\mathbf{H}}^{\top }-{\mathbf{\Gamma }}^{-1}\mathbf{H}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right){\mathbf{H}}^{\top }d\mathbf{H}{\mathbf{H}}^{\top }].\hfill \end{array}$$

(31)

Then, the derivative becomes ${\displaystyle \frac{d\mathrm{tr}{\mathbf{\Gamma }}^{-1}\mathbf{H}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right){\mathbf{H}}^{\top }}{d\mathbf{H}}}={\mathbf{\Gamma }}^{-1}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right){\mathbf{H}}^{\top }-{\mathbf{\Gamma }}^{-1}\mathbf{H}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right){\mathbf{H}}^{\top }{\mathbf{H}}^{\top }$. Thus, ${\displaystyle \frac{d\mathrm{tr}{\mathbf{\Gamma }}^{-1}\mathbf{H}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right){\mathbf{H}}^{\top }}{d\mathbf{H}}}=\mathbf{0}$ implies that $\mathbf{H}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right){\mathbf{H}}^{\top }=\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right)$. Similarly to the above arguments, we can also show that $\mathbf{H}{\mathbf{\Gamma }}^{-1}{\mathbf{H}}^{\top }={\mathbf{\Gamma }}^{-1}$. Hence, we may have that ${\min }_{\mathbf{H}\in \mathcal{O}\left(p\right)}\mathrm{t}r{\mathbf{\Gamma }}^{-1}\mathbf{H}\widehat{\Psi }\left(\mathbf{L}\right){\mathbf{H}}^{\top }=\mathrm{t}r{\mathbf{\Gamma }}^{-1}\widehat{\Psi }\left(\mathbf{L}\right)$; namely, the minimum of $tr{\mathbf{\Gamma }}^{-1}\mathbf{H}\widehat{\Psi }\left(\mathbf{L}\right){\mathbf{H}}^{\top }$, with respect to $\mathbf{H}\in \mathcal{O}\left(p\right)$, occurs at $\widehat{\mathbf{H}}=\mathbf{I}(i.e.,\widehat{\mathbf{V}}=\mathbf{U})$. Thus, the von Neumann Theorem still holds for the case in which $p/n\to c\in (0,1)$. As such, in terms of the spectral decompositions, we may have that $\mathbf{U}$ is also the MLE of $\mathbf{V}$ when $p/n\to c\in (0,1)$. Hence, based on the property of the MLE, we may summarize it as follows:

 Theorem 2. 

Let ${\mathbf{X}}_1,\dots ,{\mathbf{X}}_n$ be independent p-dimensional random vectors with a common multivariate normal distribution, $N_p(\mathbf{0},\mathbf{\Sigma })$. Consider the spectral decompositions $\mathbf{\Sigma }=\mathbf{V}\mathbf{\Gamma }{\mathbf{V}}^{\top }$ and $\mathbf{S}=\mathbf{U}\mathbf{L}{\mathbf{U}}^{\top }$ and let ${\widehat{\mathbf{\Sigma }}}_T$ be defined as in (30). Under the assumptions of Proposition 1, when ${lim}_{n\to \infty }p/n=c\in (0,1)$, $\mathbf{U}$ is the MLE of $\mathbf{V}$. Hence, it is the consistent estimator of $\mathbf{V}$.

Based on Proposition 1 and Theorem 2, we may then conclude that the proposed novel estimator ${\widehat{\mathbf{\Sigma }}}_T$ is consistent for the population covariance matrix $\mathbf{\Sigma }$ when the dimension p is large. Next, we continue to investigate whether the proposed estimator ${\widehat{\mathbf{\Sigma }}}_T$ is the MLE of $\mathbf{\Sigma }$ or not. Note that

$$\begin{array}{c}\hfill {\min }_{\mathbf{V}\in \mathcal{O}\left(p\right)}l\left({\widehat{\mathbf{\Sigma }}}_T\right|\mathbf{\Sigma })=\mathrm{tr}{\mathbf{\Gamma }}^{-1}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right)-\log \det {\mathbf{\Gamma }}^{-1}\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right)-{\displaystyle \frac{2}{n}}\log c_n\left(\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right)\right).\end{array}$$

(32)

After a number of calculations, the function ${\min }_{\mathbf{V}\in \mathcal{O}\left(p\right)}l\left({\widehat{\mathbf{\Sigma }}}_T\right|\mathbf{\Sigma })$ in (32) is further minimized with respect to $\mathbf{\Gamma }$ at $\widehat{\mathbf{\Gamma }}=\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right)$. As such, when $p/n\to c\in (0,1)$, we may conclude that $\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right)$ is the MLE of $\mathbf{\Gamma }$. Thus, based on Theorem 2, we have that ${\widehat{\mathbf{\Sigma }}}_T$ is the MLE of $\mathbf{\Sigma }$; however, the sample covariance matrix $\mathbf{S}$ is not. According to the property of the MLE, we have that $\mathbf{U}$, $\widehat{\mathbf{\Psi }}\left(\mathbf{L}\right)$, and ${\widehat{\mathbf{\Sigma }}}_T$ are the consistent estimators of $\mathbf{V}$, $\mathbf{\Gamma }$, and $\mathbf{\Sigma }$, respectively. Therefore, the following can be stated:

 Theorem 3. 

Under the assumptions of Theorem 2, when $p/n\to c\in (0,1)$, ${\widehat{\mathbf{\Sigma }}}_T$ is the MLE of Σ. Hence, it is consistent.

 Remark 3. 

We may make the following three conclusions: (i) The sample covariance matrix $\mathbf{S}$ is the MLE of the population covariance matrix  Σ  when the dimension p is fixed. (ii) The estimator ${\widehat{\mathbf{\Sigma }}}_T$ is the MLE of  Σ  when the dimension p is large so that ${lim}_{n\to \infty }p/n=c\in (0,1)$. (iii) It is easy to see that ${\widehat{\mathbf{\Sigma }}}_T$ reduces to the sample covariance matrix $\mathbf{S}$ when the dimension p is fixed and the sample size n is large (i.e., $c=0$). These are insightful parallels. Hence, for simplicity, we may integrate the above results into a unified one: when p is fixed or ${lim}_{n\to \infty }p/n=c\in (0,1)$ (i.e., $c\in [0,1))$, $n{\widehat{\mathbf{\Sigma }}}_T$ is Wishart-distributed with a mean matrix  Σ, and ${\widehat{\mathbf{\Sigma }}}_T$ is the MLE of  Σ. Thus, ${\widehat{\mathbf{\Sigma }}}_T$ is the consistent estimator of  Σ, and hence, ${\widehat{\mathbf{\Sigma }}}_T$ converges to $\mathbf{\Sigma }a.s.$ as $n\to \infty .$ Therefore, we may use ${\widehat{\mathbf{\Sigma }}}_T$ to replace $\mathbf{S}$ when making statistical inferences when the dimension p is fixed or $c\in (0,1)$.

 Remark 4. 

Tsai and Tsai [10] used the fundamental statistical concept to determine the quantile equality relationship of the limiting sample and population spectral distributions so that the consistent problems between the sample eigenvalues and the population eigenvalues can be easily handled. Then, we can use the likelihood function to progress. As long as the density function does not degenerate, the statistical inference can be performed similarly to the traditional one. The key point based on the conclusion is to find a consistent estimator of the population covariance matrix, namely, to directly determine the MLE of  Σ  when $n>p$.

When the dimension p is fixed, we have that $l_{i,p}$ is the MLE of ${\gamma }_{i,p},i=1,\dots ,p$. However, it is not true that $l_{i,p}$ is the MLE of ${\gamma }_{i,p},i=1,\dots ,p$ when $p/n\to c\in (0,1)$.

Johnstone and Paul [23] provided a detailed discussion of sample eigenvalue bias and eigenvector inconsistency under the spiked covariance model and for high-dimension PCA-related phenomena.

 Remark 5. 

We may note that $n{\widehat{\mathbf{\Sigma }}}_T$ is Wishart-distributed with the mean matrix  Σ, and ${\widehat{\mathbf{\Sigma }}}_T$ reduces to the sample covariance matrix $\mathbf{S}$ when the dimension p is fixed. With the proposed novel estimator, ${\widehat{\mathbf{\Sigma }}}_T$, replacing the sample covariance matrix, $\mathbf{S}$, the case for the traditional fixed dimension p and that for the new high-dimensional dimensions can be integrated into one. Hence, we may suggest using the proposed consistent estimator ${\widehat{\mathbf{\Sigma }}}_T$ to replace the sample covariance matrix $\mathbf{S}$ in order to carry out the multivariate statistical inference, including PCA-related problems for both cases in which (i) p is fixed and n is large (i.e., $c=0$) and ones in which (ii) $p/n\to c\in (0,1)$.

We provide an outline for the likelihood ratio test (LRT) of the hypothesis testing problem in the next section.

5. The Decomposite ${\mathit{T}}_{\mathit{T}}^{\mathit{2}}$-Test When the Dimension $\mathit{p}$ Is Large

Let ${\mathbf{X}}_i,i=1,\dots ,n,$ be a n$i.i.d$ random vector with a p-dimensional multinormal distribution, a mean vector $\mathit{\mu }$, and an unknown positive definite covariance matrix $\mathbf{\Sigma }$. Consider the hypothesis testing problem of

$$\begin{array}{c}\hfill H_0:\mathit{\mu }=\mathbf{0}\mathrm{versus}{H}_1:\mathit{\mu }\ne \mathbf{0}\end{array}$$

(33)

when both the dimension p and sample size n are large. Let

$$\begin{array}{c}\hfill \overline{\mathbf{X}}={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\mathbf{X}}_i\mathrm{and}\mathbf{S}={\displaystyle \frac{1}{n-1}}\sum _{i=1}^n({\mathbf{X}}_i-\overline{\mathbf{X}}){({\mathbf{X}}_i-\overline{\mathbf{X}})}^{\top }.\end{array}$$

(34)

Then, the well-known Hotelling’s $T^2$-test statistic, which can be found in the literature, is denoted as follows:

$$\begin{array}{c}\hfill T^2=n{\overline{\mathbf{X}}}^{\top }{\mathbf{S}}^{-1}\overline{\mathbf{X}}.\end{array}$$

(35)

When the dimension p is fixed, Hotelling’s $T^2$-test is optimal for the problem (33).

However, when the dimension p is large, the performance of Hotelling’s $T^2$-test is not optimal, because the sample covariance matrix $\mathbf{S}$ is no longer a consistent estimator of $\mathbf{\Sigma }$. To overcome this difficulty, we can adopt the novel estimator ${\widehat{\mathbf{\Sigma }}}_T$ to replace the sample covariance matrix $\mathbf{S}$ and then consider the following decomposite $T_T^2$-test statistic:

$$\begin{array}{c}\hfill T_T^2=n{\overline{\mathbf{X}}}^{\top }{\widehat{\mathbf{\Sigma }}}_T^{-1}\overline{\mathbf{X}},\end{array}$$

(36)

where ${\widehat{\mathbf{\Sigma }}}_T$ is defined in (30), with the $\mathbf{S}$ that is defined in (34) replacing the one defined in (2). It is evident that the $T_T^2$-test is the LRT statistic for the problem (33).

To avoid the issue that the power of any reasonable test moves toward one as $n\to \infty$, Le Cam’s contiguity concept was adopted to study the asymptotically local distribution when the dimension p was fixed. Note that the traditional local alternatives do not depend on the dimension p. Based on a large dimension p, in her Ph.D. thesis, Chia-Hsuan incorporated the dimension p into the consideration to study an asymptotical distribution under local alternatives.

$$\begin{array}{c}\hfill H_0:\mathit{\mu }=\mathbf{0}\mathrm{versus}{H}_{1n}:\mathit{\mu }=n^{-1/2}{p}^{1/4}\mathit{\delta },\end{array}$$

(37)

where $\mathit{\delta }$ is a fixed p-dimensional vector, which leads to the assumption that ${\mathit{\delta }}^{\top }{\mathbf{\Sigma }}^{-1}\mathit{\delta }<\infty$ when p is large. Compared with the traditional one, the local alternatives also depend on the dimension p with a slight change in the converge rate. Let

$$\begin{array}{c}\hfill T_0^2=n{\overline{\mathbf{X}}}^{\top }{\mathbf{\Sigma }}^{-1}\overline{\mathbf{X}}.\end{array}$$

(38)

Similarly to the arguments presented in Chia-Hsuan’s thesis, we can show that $T_T^2$ does not converge to $T_0^2$ in terms of probability; however, it is true that $T_T^2$ converges to $T_0^2$ in its local distribution. Note that in the traditional case of the fixed dimension p, the proposed decomposite $T_T^2$-test is reduced to Hotelling’s $T^2$-test, and Hotelling’s $T^2$-test statistic converges to $T_0^2$ in terms of probability, which implies a convergence in distribution. It is not difficult to ascertain that the $T_T^2$-test statistic reduces asymptotically and locally (under $H_{1n}$ with a rate of $n^{-1/2}{p}^{1/4}$) to a non-central chi-square ${\chi }_p^2\left({\mathit{\delta }}^{\top }{\mathbf{\Sigma }}^{-1}\mathit{\delta }\right)$ distribution. This asymptotically local power function is still the monotone function of non-centrality ${\mathit{\delta }}^{\top }{\mathbf{\Sigma }}^{-1}\mathit{\delta }$. Hence, when $p/n\to c\in [0,1)$, it is easy to see that the proposed decomposite $T_T^2$-test is optimal for the problem (33).

 Remark 6. 

The high-dimensional PCA problem has mainly been studied in spiked covariance models, and there is a need to assume sparsity in the population eigenvectors for the consistent problem (Johnstone and Lu [24] and the references therein). In contrast, with the proposed novel estimator ${\widehat{\mathbf{\Sigma }}}_T$ replacing the sample covariance matrix $\mathbf{S}$ for statistical inference, when $p/n\to c\in [0,1)$, the results of Theorem 3 can be applied to make multivariate statistical inferences and solve PCA-related problems without the sparsity assumption. When $c\in [0,1)$, our approach unifies the traditional ($c=0$) and modern high-dimensional $(c\in (0,1\left)\right)$ cases for multivariate statistical methods and high-dimensional PCA-related problems. The proposed novel estimator is incorporated to establish the optimal decomposite $T_T^2$-test for a high-dimensional statistical hypothesis testing problem and can be directly applied to high-dimensional PCA-related problems without the sparsity assumption.

In the final section, we discuss the case in which $p>n$, especially for high-dimensional low-sample size categorical data models in which $p>>n$.

6. General Remarks When $\mathit{p}>\mathit{n}$

6.1. When $p>n$, Both n and p Are Fixed

Under the multinormal setup, we may note that when $p>n$, p and n are fixed. Then, the density function of $\mathbf{S}$ becomes a singular Wishart distribution (Uhlig [25]), which it degenerates. In this situation, assume that rank ($\mathbf{S})=n$; we then may use the following notations: $\mathbf{L}$ is a $n\times n$ diagonal matrix, and the reparametrization $\mathbf{S}={\mathbf{U}}_{\mathbf{1}}\mathbf{L}{\mathbf{U}}_{\mathbf{1}}^{\top }$, where ${\mathbf{U}}_{\mathbf{1}}\in {\mathcal{V}}_{n,p}$, and the $np-n(n+1)/2$-dimensional Stiefel manifold of the $p\times n$ matrix ${\mathbf{U}}_1$ with orthonormal columns ${\mathbf{U}}_{\mathbf{1}}^{\mathbf{T}}{\mathbf{U}}_{\mathbf{1}}=\mathbf{I}$. Note that $\mathbf{V}\in \mathcal{O}\left(p\right)$; thus, $tr{\mathbf{\Sigma }}^{-1}\mathbf{S}=tr\mathbf{V}{\mathbf{\Gamma }}^{-1}{\mathbf{V}}^{\top }{\mathbf{U}}_1\mathbf{L}{\mathbf{U}}_1^{\top }=tr{\mathbf{\Gamma }}^{-1}{\mathbf{H}}_1\mathbf{L}{\mathbf{H}}_1^{\top }$, where ${\mathbf{H}}_1={\mathbf{V}}^{\top }{\mathbf{U}}_1\in {\mathcal{V}}_{n,p}$. Note that ${\mathbf{H}}_1\notin \mathcal{O}\left(p\right)$, which means that the von Neumann Theorem may fail to be true in general.

6.2. When $p>n$, Both n and p Are Large, So That $c\in (1,\infty )$

When $p>n$, since the population covariance matrix $\mathbf{\Sigma }$ is assumed to be a positive definite symmetric matrix, its p eigenvalues are all positive; however, rank($\mathbf{S})=n$. Thus, it has $p-n$ sample eigenvalues with 0 in probability. As such, it seems difficult to obtain all the consistent eigenvalue estimators of the population eigenvalues. When the sample size n and the dimension p are both large, we may only need the n largest eigenvalue estimators to be consistent for the first n largest population eigenvalues. If this is the case, the method developed in this note is still applicable.

6.3. Use in High-Dimensional Low-Sample-Size (HDLSS) Categorical Data Models

When $p>>n$ (i.e., in HDLSS categorical models), our method might still be used in some situations. HDLSS categorical models are abundant in genomics and bioinformatics, with relatively smaller sample sizes n, but also often $p>>n$. Motivated by the 2002 severe acute respiratory syndrome coronavirus (SARSCoV) epidemic model, a general model of comparing $G(\ge 2)$ groups is considered. Each sequence has P positions, each one relating to a categorical response, indexed as $1,\cdots ,C$, and there are $n_g$ sequences in the gth group, with $g=1,\cdots ,G$. For the gth group, pth position, and cth category, let $n_{gpc}$ be the number of sequences, and let $n_{gp}={\sum }_{c=1}^C{n}_{gpc}$ for $p=1,\cdots ,P,g=1,\dots ,G$. Note that if there is no missing value, each sequence, at each position, takes on one of the C responses $1,\dots ,C$, so that $n_{gp}=n_g$, for all $p=1,\cdots ,P$. The combined group sample size is $n={\sum }_{g=1}^G{n}_g$. For geographically separated sequences, the assumption of independence of the G groups may be reasonable, but the sequences within a group may not be independent due to their shared ancestry. For SARSCoV or HIV genome sequences, because of the rapid evolution of the virus, the independence assumption may not be very stringent. Further, for each sequence, the responses at the P positions are generally not independent nor necessarily identically distributed.

For SARSCoV genome sequences, the scientific focus is the statistical comparison of different strata to coordinate plausible differences in response to pertinent environmental factors. In many fields of applications, particularly in genomic studies, not only do we have $p>>n$, but also, n is often small, leading to a range with dimensional problems. We encounter conceptual and operational roadblocks due to there being too many unknown parameters. For such genomic sequences, any single position (gene) yields very little statistical information. Hence, a composite measure of the qualitative variation over the entire sequence is thought to be a better way of gauging the statistical group discrimination. In this specific context, some molecular epidemiologic studies have advocated for a suitable external sequence analysis such as multivariate analysis of variance (MANOVA), although there are impasses of various types. Genomic research is a prime illustration of the need for an appropriate statistical methodology for comprehending the genomic variation in such high-dimensional categorical data models. Variation (diversity) in such large P small n models cannot be properly statistically studied using standard discrete multivariate analysis tools or the full likelihood approach. For qualitative data models, the Gini–Simpson (GS) index (Gini [26]; Simpson [27]) and Shannon entropy (Shannon [28]) are commonly used for statistical analysis in a range of fields, including genetic variation studies (Chakraborty and Rao [29]). The Hamming distance provides an average measure that does not ignore dependence or possible heterogeneity. The U-statistics methodology (Hoeffding [30]) is incorporated to obtain optimal nonparametric estimators and their jackknife variance estimators. In the genome sequence context, we are confronted with the $P>>n$ environment. Within this framework, we encounter two scenarios: (i) $P>>n$, in which n is at least moderately large, and (ii) $P>>n$, in which n is small. In (i), the sample estimates of the Hamming distances are all U-statistics, to which standard asymptotics (Sen [31]) apply: the estimators are asymptotically (with $n\to \infty$) normal, and their jackknifed variance estimators are consistent. Hence, we shall not enter into a detailed discussion of (i). Case (ii), which is more commonly encountered in genomic studies, entails different perspectives. We must use the appropriate central limit theory (CLT) for dependent sequences of bounded random variables.

Let ${\pi }_{gpc}$ denote the cth cell probability for the pth marginal law ${\mathit{\pi }}_{gp}={({\pi }_{gp1,},\cdots ,{\pi }_{gpC})}^{\prime }$ of group $g(1\le c\le C,1\le g\le G,1\le p\le P)$, and let $n_{gpc}$ be the cell frequencies for the pth marginal table corresponding to the gth group, so that the MLE of ${\pi }_{gpc}$ is ${\widehat{\pi }}_{gpc}=n_{gpc}/n_g,1\le c\le C$, where $n_g={\sum }_{c=1}^C{n}_{gpc}$, with the same process being applied to every $p(=1,\cdots ,P)$. We incorporate the jackknife methodology to obtain the nonparametric estimators. The jackknife estimator, a plug-in estimator that is based on the MLE of ${\mathit{\pi }}_{gp}$, of the Hamming–Shannon measure is considered. The difficulties associated with the HDLSS asymptotics in the HDLSS genomic context are assessed, and suitable permutation procedures are appraised. Under the null hypothesis, i.e., the homogeneity of the G groups, the advantage of the resulting permutation invariance structure is used. Therefore, we proceed with this extended permutation–jackknife methodology.

Consider all possible equally likely permutations of the observations for each p, each with the same conditional probability ${\displaystyle \frac{1}{N}}$, where $N=n!/{\prod }_{g=1}^G{n}_g!,g=1,\cdots ,G$. Let ${\mathbf{Y}}_1={(T_{2:1},\cdots ,T_{G:G-1})}^t$ with $T_{\{i:i-1\}}$ being well-defined statistics. In practice, to overcome the difficulty that N is too large, we may choose $N_1$, which is sufficiently large, instead of $N_1<<N$. Next, generate a set of $(N_1-1)$ permutations. For this construction, we use the permutation distribution that has been generated by the set of all possible permutations among themselves. Let ${\mathbf{Y}}_i$ be the $(i-1)$th corresponding permutation of ${\mathbf{Y}}_1$, $i=2,\cdots ,N_1$ and the corresponding covariance matrix ${\mathbf{S}}_{N_1}={(N_1-1)}^{-1}{\sum }_{i=1}^{N_1}({\mathbf{Y}}_i-\overline{\mathbf{Y}}){({\mathbf{Y}}_i-\overline{\mathbf{Y}})}^t$, where $\overline{\mathbf{Y}}={N_1}^{-1}{\sum }_{i=1}^{N_1}{\mathbf{Y}}_i$. In practice, $N_1$ is taken to be sufficiently large so that $li{m}_{N_1\to \infty }P/N_1=c\in (0,1)$. Carry out the spectral decomposition for the matrix ${\mathbf{S}}_{N_1}$, with the eigenvalues of ${\mathbf{S}}_{N_1}$ being replaced by the new corresponding eigenvalues, which are obtained based on Equation (30) in the same way that the sample covariance matrix $\mathbf{S}$ was replaced by ${\widehat{\mathbf{\Sigma }}}_T$, to obtain the new and improved jackknife covariance matrix replacing ${\mathbf{S}}_{N_1}$ for the statistical inference. Thus, under these circumstances, the procedure that we proposed in Section 4 works well for HDLSS categorical data models.

When the sample size n is larger than the dimension p, so that ${lim}_{n\to \infty }p/n=c\in (0,1)$, as shown in Section 4, we can note that $n{\widehat{\mathbf{\Sigma }}}_T$ is Wishart-distributed with a mean matrix $\mathbf{\Sigma }$. It is demonstrated that ${\widehat{\mathbf{\Sigma }}}_T$ is the MLE of the population covariance matrix $\mathbf{\Sigma }$; hence, it is consistent. Moreover, it is easy to ascertain that ${\widehat{\mathbf{\Sigma }}}_T$ reduces to the sample covariance matrix $\mathbf{S}$ when the dimension p is fixed. Hence, when $n>P$, with the proposed novel estimator ${\widehat{\mathbf{\Sigma }}}_T$ replacing the sample covariance matrix $\mathbf{S}$, the traditional case of a fixed dimension p and the modern case of a high-dimensional setup can be integrated into a unified theory. We may therefore also conclude that ${\widehat{\mathbf{\Sigma }}}_T$ is the best $\mathcal{O}\left(p\right)$-equivariant estimator. Thus, when $n>P$, the proposed novel estimator ${\widehat{\mathbf{\Sigma }}}_T$ of the population covariance matrix $\mathbf{\Sigma }$ plays a fundamental role in the further theoretical development of statistical inference. Practically, it has applications in certain HDLSS categorical data models. When the sample size n is moderate, the optimal nonparametric methods for the genomic data may be proposed. When $P>>n$ and the sample n is small, we may incorporate the permutation and jackknife methodologies to make statistical inferences for the genomic data. Hopefully, the optimal statistical methods can be of help for scientific breakthroughs as well as for real-world applications within gene science.