When It Counts—Econometric Identification of the Basic Factor Model Based on GLT Structures

Abstract

Despite the popularity of factor models with simple loading matrices, little attention has been given to formally address the identifiability of these models beyond standard rotation-based identification such as the positive lower triangular (PLT) constraint. To fill this gap, we review the advantages of variance identification in simple factor analysis and introduce the generalized lower triangular (GLT) structures. We show that the GLT assumption is an improvement over PLT without compromise: GLT is also unique but, unlike PLT, a non-restrictive assumption. Furthermore, we provide a simple counting rule for variance identification under GLT structures, and we demonstrate that within this model class, the unknown number of common factors can be recovered in an exploratory factor analysis. Our methodology is illustrated for simulated data in the context of post-processing posterior draws in sparse Bayesian factor analysis.

1. Introduction

Ever since the pioneering work of Thurstone (1935, 1947), factor analysis has been a popular method to model the covariance matrix $\mathbf{\Omega }$ of correlated, multivariate observations ${\mathbf{y}}_t$ of dimension m (see, e.g., Anderson (2003) for a comprehensive review). Assuming r uncorrelated factors in a basic factor model, for instance, yields the representation $\mathbf{\Omega }=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }+{\mathbf{\Sigma }}_0$, with a $m\times r$ factor loading matrix $\mathbf{\Lambda }$ and a diagonal matrix ${\mathbf{\Sigma }}_0$. This considerable reduction in the number of parameters compared to the $m(m+1)/2$ parameters of an unconstrained covariance matrix is the main motivation for applying factor models to covariance estimation, especially if m is large (see, among many others, Fan et al. (2008)). In addition, shrinkage estimation has been shown to lead to very efficient covariance estimation (see, for example, Kastner (2019) in Bayesian factor analysis and Ledoit and Wolf (2020) in a non-Bayesian context).

In numerous applications, factor analysis reaches beyond covariance modeling (see, among many others, Forni et al. (2009) in the context of structural factor models). From the very beginning, the goal of factor analysis has been to extract the underlying loading matrix $\mathbf{\Lambda }$ to understand the driving forces behind the observed correlation between the measurements (see, e.g., Owen and Wang (2016) for a recent review). However, also in this setting, the only source of information is the observed covariance of the data, making the decomposition of the covariance matrix $\mathbf{\Omega }$ into the cross-covariance matrix $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ and the variance ${\mathbf{\Sigma }}_0$ of the idiosyncratic errors more challenging than estimating only $\mathbf{\Omega }$ itself.

A huge amount of literature, dating back to Koopmans and Reiersøl (1950) and Reiersøl (1950), has addressed this problem of identification which can be resolved only by imposing additional structure on the factor model. Anderson and Rubin (1956) consider two kinds of conditions for identification. The first problem, also known as solving rotational invariance, aims at identifying $\mathbf{\Lambda }$, assuming that $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ is determined uniquely. This problem has received considerable attention in econometrics, statistics, and machine learning. The most popular condition for solving rotational invariance is to consider positive lower triangular (PLT) loading matrices (see, e.g., Geweke and Zhou (1996); Lopes and West (2004); West (2003)), although other strategies have been put forward (see, e.g., Neudecker (1990), Bai and Ng (2013), Aßmann et al. (2016), Chan et al. (2018), and Williams (2020)). In related strands, Anderson et al. (2016) examine the generic identifiability of dense vector autoregressive systems with mixed frequencies, and their theory is also applicable to static factor models.

In the second problem, Anderson and Rubin (1956) consider conditions for variance identification, i.e., unique identification of $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ and ${\mathbf{\Sigma }}_0$ assuming that the covariance matrix $\mathbf{\Omega }=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }+{\mathbf{\Sigma }}_0$ arises from a basic factor model. Examples in Bartholomew (1987), for instance, show that two different models could imply the same covariance matrix. Sufficient conditions for ensuring variance identification have received much less attention than the first problem. Conditions include the row-deletion property (Anderson and Rubin 1956) and a simple counting rule for the number of non-zero loading in each column of the factor loading matrix in the context of dedicated factor models (Conti et al. 2014); see also Bekker (1989) for related work.

In this work, we discuss conditions for identification based on ordered and unordered generalized lower triangular (GLT) structures which relax the PLT condition, see Figure 1 for illustration. This concept was first introduced in the unpublished work by Frühwirth-Schnatter and Lopes (2018) as part of an MCMC sampler for sparse Bayesian factor analysis where the number of factors is unknown. In the present paper, GLT structures are given a full and comprehensive mathematical treatment. It will be proven that GLT structures simultaneously address rotational invariance and variance identification in factor models. Variance identification relies on a counting rule for the number of non-zero elements in the loading matrix $\mathbf{\Lambda }$, which is a sufficient condition that extends the previous work by Sato (1992) and Conti et al. (2014).

In addition, we show that unordered GLT structures are useful in exploratory factor analysis where the factor dimension r is unknown. Identification of the number of factors in applied factor analysis is a notoriously difficult problem for many latent factor models, with considerable ambiguity about which method works best, be it BIC-type criteria for approximate factor models (Bai and Ng 2002), marginal likelihoods for basic factor models (Lopes and West 2004), techniques from Bayesian non-parametrics involving infinite-dimensional factor models (Bhattacharya and Dunson 2011; Legramanti et al. 2020; Ročková and George 2017) or more heuristic procedures for dynamic factor models (Kaufmann and Schumacher 2019). Imposing an unordered GLT structure in exploratory factor analysis allows us to identify the true factor dimension by spotting spurious columns in a possibly overfitting model and to identify the true loading matrix $\mathbf{\Lambda }$ and the matrix ${\mathbf{\Sigma }}_0$.

The theoretical results of the present paper are exploited in related work. Relying on ordered and unordered GLT structures, Frühwirth-Schnatter et al. (2023) develop an efficient reversible jump MCMC sampler for sparse Bayesian factor analysis under very general shrinkage priors when the number of factors is unknown and use the counting rules introduced in this paper for postprocessing the posterior draws.

The current paper is structured as follows. Section 2 reviews the role of identification in a basic factor model using illustrative examples. Section 3 introduces ordered and unordered GLT structures, proves identification for simple GLT structures, and shows that any full column-rank unconstrained loading matrix has a unique representation as a GLT matrix. Section 4 addresses variance identification under GLT structures. Section 5 discusses exploratory factor analysis under unordered GLT structures and addresses additional identification problems that arise in a basic factor model when the number of factors is unknown. Section 7 presents illustrative applications to simulated and empirical data. Section 8 concludes.

2. The Basic Factor Model

2.1. Model Definition

Let $({\mathbf{y}}_1,\dots ,{\mathbf{y}}_T)$ be a sequence of observations, where ${\mathbf{y}}_t={(y_{1t},\dots ,y_{mt})}^{\top }$ for $t=1,\dots ,T$ is a vector of m measurements assumed to arise from a multivariate normal distribution, ${\mathbf{y}}_t\sim {\mathcal{N}}_m\left(\mathbf{0},\mathbf{\Omega }\right)$, with zero mean and covariance matrix $\mathbf{\Omega }$. In factor analysis, the correlation among the measurements in ${\mathbf{y}}_t$ is assumed to be driven by a latent r-variate random variable ${\mathbf{f}}_t={(f_{1t},\dots ,f_{rt})}^{\top }$, the so-called common factors, through the following observation equation:

$$\begin{array}{c}\hfill {\mathbf{y}}_t=\mathbf{\Lambda }{\mathbf{f}}_t+{\mathit{ϵ}}_t,\end{array}$$

(1)

where the $m\times r$ matrix $\mathbf{\Lambda }$ containing the factor loadings ${\mathsf{\Lambda }}_{ij}$ is of full column rank, $\mathrm{rk}\left(\mathbf{\Lambda }\right)=r$, equal to the factor dimension r.

In the present paper, we focus on the so-called basic factor model where the observations $({\mathbf{y}}_1,\dots ,{\mathbf{y}}_T)$ are assumed to be iid. Furthermore, the vector ${\mathit{ϵ}}_t={({ϵ}_{1t},\dots ,{ϵ}_{mt})}^{\top }$ accounts for the independent, idiosyncratic variation in each measurement and is distributed as ${\mathit{ϵ}}_t\sim {\mathcal{N}}_m\left(\mathbf{0},{\mathbf{\Sigma }}_0\right)$, with ${\mathbf{\Sigma }}_0=\mathrm{Diag}\left({\sigma }_1^2,\dots ,{\sigma }_m^2\right)$ being a positive definite diagonal matrix. The common factors are orthogonal, meaning that ${\mathbf{f}}_t\sim {\mathcal{N}}_r\left(\mathbf{0},{\mathbf{I}}_r\right)$, and independent of ${\mathit{ϵ}}_t$. In this case, the observation Equation (1) implies the following covariance matrix $\mathbf{\Omega }$, when we integrate with respect to the latent common factors ${\mathbf{f}}_t$:

$$\begin{array}{c}\hfill \mathbf{\Omega }=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }+{\mathbf{\Sigma }}_0.\end{array}$$

(2)

Finally, the model assumes independence of ${\mathbf{f}}_t$ and ${\mathbf{f}}_s$, ${\mathbf{f}}_t$ and ${\mathit{ϵ}}_s$, and ${\mathit{ϵ}}_t$ and ${\mathit{ϵ}}_s$ for all $s\ne t$. All cross-sectional dependence among the m measurements in ${\mathbf{y}}_t$ is explained through the latent common factors and the off-diagonal elements of $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ define the marginal covariance between any two measurements $y_{i_1,t}$ and $y_{i_2,t}$:

$$\begin{array}{c}\hfill \mathrm{Cov}(y_{i_1,t},y_{i_2,t})={\mathbf{\Lambda }}_{i_1,•}{\mathbf{\Lambda }}_{i_2,•}^{\top },\end{array}$$

(3)

where ${\mathbf{\Lambda }}_{i,•}$ is the ith row of $\mathbf{\Lambda }$. Consequently, we will refer to $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ as the cross-covariance matrix. Since the number of factors, r, is often considerably smaller than the number of measurements, m, (2) can be seen as a parsimonious representation of the dependence between the measurements, with considerably fewer parameters in the factor loading matrix $\mathbf{\Lambda }$ than the $m(m-1)/2$ off-diagonal elements in an unconstrained covariance matrix $\mathbf{\Omega }$.

2.2. Loading Matrices with a Simple Structure

The factor loading matrices given in Figure 1 are examples of simple structures in the sense specified by Thurstone (1947), where each variable loads only on specific factors and factors affect only a subset of variables. In traditional factor analysis, after estimating an identified matrix of loadings, factors are rotated so as to enhance a simple structure, the most popular procedure being Varimax (see, e.g., Magnus and Neudecker (2019, sct. 17.14)). In sparse Bayesian factor analysis, which will be discussed in more detail in Section 6.1, priors are chosen that encourage automatic rotation to a simple structure (see, e.g., Ročková and George (2017)).

Subsequently, we will make use of a representation of a factor loading matrix $\mathbf{\Lambda }$ with a simple structure called the sparsity matrix. The sparsity matrix $\mathit{\delta }$ is a binary indicator matrix of 0 s and 1 s of the same dimension as $\mathbf{\Lambda }$ and contains the information which elements of a factor loading matrix with a simple structure are equal to 0 and which elements are unconstrained, i.e., if ${\delta }_{ij}=0$, then ${\mathsf{\Lambda }}_{ij}=0$, while ${\mathsf{\Lambda }}_{ij}\in \mathbb{R}$ if ${\delta }_{ij}=1$.

The sparsity matrix $\mathit{\delta }$ contains a lot of information about the structure of $\mathbf{\Lambda }$ (see the left-hand matrix in Figure 1 for illustration). In total, six factors are needed to fully explain the variation in these 22 measurements. The fifth row contains only zeros, which tells us that measurement $y_{5t}$ is uncorrelated with the remaining measurements, since $\mathrm{Cov}(y_{it},y_{5t})=0$ for all $i\ne 5$. The measurements 1, 3, 10, 11, 14, and 17 each constitute a new factor. Hence, the variation in the first nine measurements can be explained by at most two factors which also load on some (but not all) of the remaining measurements. The rows which constitute a new factor will play an instrumental role for identification in the present paper. More specifically, the sparsity matrix on the left hand side tells us that the underlying loading matrix $\mathbf{\Lambda }$ has an ordered GLT structure, while the matrix in the middle is one of many unordered GLT structures that can be derived from $\mathbf{\Lambda }$. The sparsity matrix on the right hand side indicates that the loading matrix $\mathbf{\Lambda }$ has the commonly applied PLT structure where the first six measurements lead the six factors. Ordered and unordered GLT structures will be discussed in full details in Section 3.

2.3. A Brief Review of Identification When the Number of Factors Is Known

Since the factors ${\mathbf{f}}_t$ are unobserved, the only information available to estimate $\mathbf{\Lambda }$ and ${\mathbf{\Sigma }}_0$ is the covariance matrix $\mathbf{\Omega }$, which creates well-known identification issues for the basic factor model. Consider, for example, the following factor loading matrix $\mathbf{\Lambda }$ and all loading matrices $\mathit{\beta }=\mathbf{\Lambda }{\mathbf{P}}_{\alpha b}$ defined as a rotation of $\mathbf{\Lambda }$:

$$\begin{array}{c}\hfill \mathbf{\Lambda }=\left(\begin{array}{cc}{\lambda }_{11}& 0\\ {\lambda }_{21}& 0\\ {\lambda }_{31}& 0\\ {\lambda }_{41}& {\lambda }_{42}\\ 0& {\lambda }_{52}\\ 0& {\lambda }_{62}\end{array}\right),{\mathbf{P}}_{\alpha b}=\left(\begin{array}{cc}\hfill cos\alpha & \hfill {(-1)}^{b}sin\alpha \\ \hfill -sin\alpha & \hfill {(-1)}^{b}cos\alpha \end{array}\right),\mathit{\beta }=\left(\begin{array}{cc}{\beta }_{11}& {\beta }_{21}\\ {\beta }_{21}& {\beta }_{22}\\ {\beta }_{31}& {\beta }_{32}\\ {\beta }_{41}& {\beta }_{42}\\ {\beta }_{51}& {\beta }_{52}\\ {\beta }_{61}& {\beta }_{62}\end{array}\right).\end{array}$$

(4)

Then, for any $\alpha \in [0,2\pi )$ and $b\in \{0,1\}$, the factor loading matrix $\mathit{\beta }$ yields the same cross-covariance matrix for ${\mathbf{y}}_t$ as $\mathbf{\Lambda }$:

$$\begin{array}{c}\hfill \mathit{\beta }{\mathit{\beta }}^{\top }=\mathbf{\Lambda }{\mathbf{P}}_{\alpha b}{\mathbf{P}}_{\alpha b}^{\top }{\mathbf{\Lambda }}^{\top }=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }.\end{array}$$

(5)

The rotational invariance apparent in (5) holds for any basic factor model (1), as is easily verified. Take any $r\times r$ rotation matrix $\mathbf{P}$ (i.e., $\mathbf{P}{\mathbf{P}}^{\top }={\mathbf{I}}_r$) and define the basic factor model

$$\begin{array}{c}\hfill {\mathbf{f}}_t^{★}\sim {\mathcal{N}}_r\left(\mathbf{0},{\mathbf{I}}_r\right),{\mathbf{y}}_t=\mathit{\beta }{\mathbf{f}}_t^{★}+{\mathit{ϵ}}_t,{\mathit{ϵ}}_t\sim {\mathcal{N}}_m\left(\mathbf{0},{\mathbf{\Sigma }}_0\right),\end{array}$$

(6)

where $\mathit{\beta }=\mathbf{\Lambda }\mathbf{P}$ and ${\mathbf{f}}_t^{★}={\mathbf{P}}^{\top }{\mathbf{f}}_t$. Obviously, both models imply the same covariance $\mathbf{\Omega }$, given by (2). Hence, without imposing further conditions, $\mathbf{\Lambda }$ is in general not identified from the cross-covariance matrix $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$. The usual way of dealing with this rotational invariance is to constrain $\mathbf{\Lambda }$ in such a way that the only possible rotation is the identity $\mathbf{P}={\mathbf{I}}_r$. For orthogonal factors, at least $r(r-1)/2$ restrictions on the elements of $\mathbf{\Lambda }$ are needed to eliminate rotational indeterminacy (Anderson and Rubin 1956).

As Anderson and Rubin (1956) point out, additional insight can be gained by comparing the number of parameters in $\mathbf{\Omega }$, namely $m(m+1)/2$, with the number of parameters in the pair $(\mathbf{\Lambda },{\mathbf{\Sigma }}_0)$, namely $m(r+1)$. If $r(r-1)/2$ restrictions are imposed to eliminate rotational indeterminacy, this yields the necessary condition ${(m-r)}^2>m+r$ (see also Anderson (2003, sct. 14.2.2)). However, this condition is by no means sufficient for identification, as will be illustrated in Example (9) below, where $m=5$ and $r=2$ satisfies this condition, but identifiability fails.

A rigorous approach toward the identification of factor models was first offered by Anderson and Rubin (1956). Assume that a pair of parameters $(\mathbf{\Lambda },{\mathbf{\Sigma }}_0)$, where $\mathbf{\Lambda }$ is an $m\times r$ matrix and ${\mathbf{\Sigma }}_0$ is a positive definite diagonal matrix, satisfying (2) is given. Let $(\mathit{\beta },\mathbf{\Sigma })$, where $\mathit{\beta }$ is an $m\times r$ matrix and $\mathbf{\Sigma }$ is a positive definite diagonal matrix, be another pair of parameters satisfying (2). Then,

$$\begin{array}{c}\hfill \mathbf{\Omega }=\mathit{\beta }{\mathit{\beta }}^{\top }+\mathbf{\Sigma }=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }+{\mathbf{\Sigma }}_0,\end{array}$$

and both pairs imply the same Gaussian distribution ${\mathbf{y}}_t\sim {\mathcal{N}}_m\left(\mathbf{0},\mathbf{\Omega }\right)$ for every possible realization ${\mathbf{y}}_t$. For an identified model, it would follow that the two pairs of parameters are identical, i.e., $\mathit{\beta }=\mathbf{\Lambda }$ and $\mathbf{\Sigma }={\mathbf{\Sigma }}_0$. However, as discussed above, identification can be achieved for a basic factor model only by imposing conditions on $\mathbf{\Lambda }$ and ${\mathbf{\Sigma }}_0$.

Anderson and Rubin (1956) consider two kinds of conditions for identification. First, they consider conditions assuming that $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ and ${\Sigma }_0$ are determined uniquely. According to their Lemma 5.1, any alternative loading matrix $\mathit{\beta }$ which satisfies $\mathit{\beta }{\mathit{\beta }}^{\top }=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ (while $\mathbf{\Sigma }={\mathbf{\Sigma }}_0$) and, consequently, implies the same covariance matrix $\mathbf{\Omega }$ as $\mathbf{\Lambda }$, is an orthogonal rotation $\mathbf{P}$ of $\mathbf{\Lambda }$, i.e., $\mathit{\beta }=\mathbf{\Lambda }\mathbf{P}$. Conditions are then imposed on the structure of $\mathbf{\Lambda }$ to solve this rotational invariance problem (see Section 2.4 for details). Second, they consider conditions that ensure that $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ and ${\mathsf{\Sigma }}_0$ are, indeed, determined uniquely. In their Theorem 5.1, they formulate a row-deletion property as a sufficient condition to resolve the variance identification problem and ensure unique identification of the variance decomposition in (2) (see Section 2.5 for details). The literature on factor analysis often reduces the identification of factor models to the first problem; however, as we will argue in the present paper, variance identification is equally important, in particular for loading matrices with a simple structure.

2.4. Conditions Resolving Rotational Invariance

Let us assume that variance identification holds and $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ and ${\mathsf{\Sigma }}_0$ are determined uniquely from $\mathbf{\Omega }$. By far, the most popular constraints to deal with rotational invariance are positive lower triangular (PLT) loading matrices, where the upper triangular part is constrained to be zero and the main diagonal elements ${\mathsf{\Lambda }}_{11},\dots ,{\mathsf{\Lambda }}_{rr}$ of $\mathbf{\Lambda }$ are strictly positive.1 The corresponding sparsity matrix $\mathit{\delta }$ also exhibits a PLT structure (see the right-hand side of Figure 1 for illustration).

Despite its popularity, the PLT structure is restrictive, as outlined already by Jöreskog (1969). Let $\mathit{\beta }{\mathit{\beta }}^{\top }$ be an arbitrary cross-covariance matrix with factor loading matrix $\mathit{\beta }$. A PLT representation for $\mathit{\beta }{\mathit{\beta }}^{\top }$ is possible if the top r rows of $\mathit{\beta }$ are linearly independent, or, equivalently, if a rotation matrix $\mathbf{P}$ exists such that $\mathit{\beta }$ can be rotated into a PLT matrix $\mathbf{\Lambda }=\mathit{\beta }\mathbf{P}$. However, as example (4) illustrates, this is not necessarily the case. Obviously, $\mathbf{\Lambda }$ is not a PLT matrix, since ${\mathsf{\Lambda }}_{22}=0$. Any of the possible rotations $\mathit{\beta }=\mathbf{\Lambda }{\mathbf{P}}_{\alpha b}$ have non-zero elements above the main diagonal and are not PLT matrices either. This example demonstrates that the PLT representation is restrictive. To circumvent this problem in Example (4), one could reorder the measurements in an appropriate manner. However, in applied factor analysis, such an appropriate ordering is typically not known in advance and the choice of the first r measurements is an important modeling decision under PLT constraints (see, e.g., Lopes and West (2004) and Carvalho et al. (2008)).

An alternative identification condition, also dating back to Anderson and Rubin (1956), is assuming diagonality of ${\mathbf{\Lambda }}^{\top }{\mathbf{\Sigma }}_0^{-1}\mathbf{\Lambda }$ and requires the corresponding diagonal elements $d_1,\dots ,d_r$ to be different. In practice, however, weak identifiability might occur if some of the elements $d_1,\dots ,d_r$ are similar, even if not equal. Furthermore, in an attempt to formalize the criteria for simplicity of Thurstone (1947), Reiersøl (1950) put forward a condition which requires a minimum of r zero factor loadings in each column. However, this condition rules out the existence of a “market” factor that loads on all measurements as for the financial returns in our empirical case study in Section 7.2.

In Section 3, we discuss a new identification strategy to resolve rotational invariance in factor models based on the concept of generalized lower triangular (GLT) structures. Loosely speaking, GLT structures generalize PLT structures by freeing the position of the first non-zero factor loading in each column (see the examples in Figure 1 and the loading matrix $\mathbf{\Lambda }$ in (4)). Consequently, the use of GLT structures for rotational identification does not imply that the first r rows of the loading matrix are linearly independent, as the use of PLT structures would, but only that there exist r linearly independent rows in $\mathbf{\Lambda }$. This weakened condition hints at a wider applicability of GLT structures compared to PLT structures. In particular, we show in Section 3.2 that a unique GLT structure $\mathbf{\Lambda }$ can be identified from any cross-covariance matrix $\mathit{\beta }{\mathit{\beta }}^{\top }$, provided that variance identification holds and, consequently, $\mathit{\beta }{\mathit{\beta }}^{\top }$ itself is identified. Even if $\mathit{\beta }{\mathit{\beta }}^{\top }$ is obtained from a loading matrix $\mathit{\beta }$ that does not take the form of a GLT structure, such as the matrix $\mathit{\beta }$ in (4), we show that an orthogonal matrix $\mathbf{G}$ exists which represents $\mathit{\beta }=\mathbf{\Lambda }{\mathbf{G}}^{\top }$ as a rotation of an ordered GLT structure $\mathbf{\Lambda }$ and defines the unique rotation $\mathbf{\Lambda }=\mathit{\beta }\mathbf{G}$ of $\mathit{\beta }$ into GLT. Hence, the GLT representation is unrestrictive in the sense of Anderson and Rubin (1956) and Jöreskog (1969), and is a generic way to resolve rotational invariance for any factor loading matrix.

2.5. Conditions for Variance Identification

Conditions that resolve rotational invariance typically take variance identification, i.e., identification of $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$, for granted (see, e.g., Geweke and Zhou (1996)). Variance identification refers to the problem that the idiosyncratic variances ${\sigma }_1^2,\dots ,{\sigma }_m^2$ in ${\mathbf{\Sigma }}_0$ are identified only from the diagonal elements of $\mathbf{\Omega }$, as all elements in the cross-variance matrix $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ are independent of the ${\sigma }_i^2$s (see again (3)). To achieve variance identification of ${\sigma }_i^2$ from ${\mathsf{\Omega }}_{ii}={\mathbf{\Lambda }}_{i,•}{\mathbf{\Lambda }}_{i,•}^{\top }+{\sigma }_i^2$, all factor loadings have to be identified solely from the off-diagonal elements of $\mathbf{\Omega }$. Variance identification, however, is easily violated, as the following examples illustrate.

Let us consider a factor model with the following loading matrix (also known as a dedicated factor model; see, e.g., Conti et al. 2014):

$$\begin{array}{c}\hfill \mathbf{\Lambda }=\left(\begin{array}{cc}{\lambda }_{11}& 0\\ {\lambda }_{21}& 0\\ {\lambda }_{31}& 0\\ 0& {\lambda }_{42}\\ 0& {\lambda }_{52}\\ 0& {\lambda }_{62}\end{array}\right).\end{array}$$

(7)

The corresponding covariance matrix $\mathbf{\Omega }$ is given by:

$$\begin{array}{c}\hfill \mathbf{\Omega }=\left(\begin{array}{cccccc}{\lambda }_{11}^2+{\sigma }_1^2& {\lambda }_{11}{\lambda }_{21}& {\lambda }_{11}{\lambda }_{31}& & & \\ {\lambda }_{11}{\lambda }_{21}& {\lambda }_{21}^2+{\sigma }_2^2& {\lambda }_{21}{\lambda }_{31}& & \mathbf{0}& \\ {\lambda }_{11}{\lambda }_{31}& {\lambda }_{21}{\lambda }_{31}& {\lambda }_{31}^2+{\sigma }_3^2& & & \\ & & & {\lambda }_{42}^2+{\sigma }_4^2& {\lambda }_{42}{\lambda }_{52}& {\lambda }_{42}{\lambda }_{62}\\ & \mathbf{0}& & {\lambda }_{42}{\lambda }_{52}& {\lambda }_{52}^2+{\sigma }_5^2& {\lambda }_{52}{\lambda }_{62}\\ & & & {\lambda }_{42}{\lambda }_{62}& {\lambda }_{52}{\lambda }_{62}& {\lambda }_{62}^2+{\sigma }_6^2\end{array}\right).\end{array}$$

(8)

Let us assume that the sparsity matrix $\mathit{\delta }$ of $\mathbf{\Lambda }$ is given (i.e., we know which elements in (7) are zero), but the specific values of the unconstrained loadings $({\lambda }_{11},\dots ,{\lambda }_{62})$ are unknown. An interesting question is the following. Knowing $\mathbf{\Omega }$ and $\mathit{\delta }$, can the unconstrained loadings ${\lambda }_{11},\dots ,{\lambda }_{62}$ and the idiosyncratic variances ${\sigma }_1^2,\dots ,{\sigma }_m^2$ be identified uniquely? Given $\mathbf{\Omega }$, the three nonzero covariances $\mathrm{Cov}(y_{1t},y_{2t})={\lambda }_{11}{\lambda }_{21}$, $\mathrm{Cov}(y_{1t},y_{3t})={\lambda }_{11}{\lambda }_{31}$ and $\mathrm{Cov}(y_{2t},y_{3t})={\lambda }_{21}{\lambda }_{31}$ are available to identify the three factor loadings $({\lambda }_{11},{\lambda }_{21},{\lambda }_{31})$. Similarly, the three nonzero covariances $\mathrm{Cov}(y_{4t},y_{5t})={\lambda }_{42}{\lambda }_{52}$, $\mathrm{Cov}(y_{4t},y_{6t})={\lambda }_{42}{\lambda }_{62}$ and $\mathrm{Cov}(y_{5t},y_{6t})={\lambda }_{52}{\lambda }_{62}$ are available to identify the factor loadings $({\lambda }_{42},{\lambda }_{52},{\lambda }_{62})$. Hence, variance identification holds. However, when we remove the last measurement from the loading factor matrix defined in (7), we obtain

$$\begin{array}{c}\hfill \mathbf{\Lambda }=\left(\begin{array}{cc}{\lambda }_{11}& 0\\ {\lambda }_{21}& 0\\ {\lambda }_{31}& 0\\ 0& {\lambda }_{42}\\ 0& {\lambda }_{52}\end{array}\right),\end{array}$$

(9)

and the corresponding covariance matrix $\mathbf{\Omega }$ reads:

$$\begin{array}{c}\hfill \mathbf{\Omega }=\left(\begin{array}{ccccc}{\lambda }_{11}^2+{\sigma }_1^2& {\lambda }_{11}{\lambda }_{21}& {\lambda }_{11}{\lambda }_{31}& & \\ {\lambda }_{11}{\lambda }_{21}& {\lambda }_{21}^2+{\sigma }_2^2& {\lambda }_{21}{\lambda }_{31}& \mathbf{0}& \\ {\lambda }_{11}{\lambda }_{31}& {\lambda }_{21}{\lambda }_{31}& {\lambda }_{31}^2+{\sigma }_3^2& & \\ & & & {\lambda }_{42}^2+{\sigma }_4^2& {\lambda }_{42}{\lambda }_{52}\\ & \mathbf{0}& & {\lambda }_{42}{\lambda }_{52}& {\lambda }_{52}^2+{\sigma }_5^2\end{array}\right).\end{array}$$

While the three factor loadings $({\lambda }_{11},{\lambda }_{21},{\lambda }_{31})$ are still identified from the off-diagonal elements of $\mathbf{\Omega }$ as before, variance identification of ${\sigma }_4^2$ and ${\sigma }_5^2$ fails. Since $\mathrm{Cov}(y_{4t},y_{5t})={\lambda }_{42}{\lambda }_{52}$ is the only non-zero element that depends on the loadings ${\lambda }_{42}$ and ${\lambda }_{52}$, infinitely many different parameters $({\lambda }_{42},{\lambda }_{52},{\sigma }_4^2,{\sigma }_5^2)$ imply the same covariance matrix $\mathbf{\Omega }$.

From these considerations, it is evident that a minimum of three non-zero loadings is necessary in each column to achieve variance identification, a condition which dates back to Anderson and Rubin (1956). At the same time, this condition is not sufficient. For illustration, consider the rotation $\mathit{\beta }=\mathbf{\Lambda }{\mathbf{P}}_{\alpha b}$ of the loading matrix $\mathbf{\Lambda }$ in (9) with $\alpha \ne \{0,\frac{\pi }{2},\pi ,\frac{3\pi }{2}\}$, where all elements of the corresponding sparsity matrix are equal to 1. Although each column of $\mathit{\beta }$ has six non-zero elements, variance identification obviously does not hold.

For more general loading matrices, variance identification is not as easily verified as for these examples and we rely in the present paper on the row-deletion property introduced by Anderson and Rubin (1956).

Definition 1

(Row-deletion property AR (Anderson and Rubin 1956)). An $m\times r$ factor loading matrix Λ satisfies the row-deletion property if the following condition is satisfied: whenever an arbitrary row is deleted from Λ, two disjoint submatrices of rank r remain.

Anderson and Rubin (1956, Theorem 5.1) prove that the row-deletion property is a sufficient condition for the identification of $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ and ${\mathbf{\Sigma }}_0$ from the marginal covariance matrix $\mathbf{\Omega }$ given in (2). For any (not necessarily GLT) factor loading matrix $\mathbf{\Lambda }$, the row-deletion property AR can be trivially tested by a step-by-step analysis, where every single row of $\mathbf{\Lambda }$ is sequentially deleted and the two distinct submatrices are determined from examining the remaining matrix (Hayashi and Marcoulides 2006). However, this procedure is inefficient in higher dimensions and simpler conditions for verifying variance identification under the row-deletion property AR are warranted.

The literature provides necessary conditions for AR that are based on counting the number of non-zero factor loadings in $\mathbf{\Lambda }$, such as: for every nonsingular r-dimensional square matrix $\mathbf{G}$, the matrix $\mathit{\beta }=\mathbf{\Lambda }\mathbf{G}$ contains at least three nonzero factor loadings in each column, and at least five nonzero factor loadings in each pair of columns (Anderson and Rubin 1956). Sato (1992, Theorem 3.3) extends these necessary conditions in the following way: every subset of $1\le q\le r$ columns of $\mathit{\beta }=\mathbf{\Lambda }\mathbf{G}$ contains at least $2q+1$ nonzero factor loadings for every nonsingular matrix $\mathbf{G}$. We call this the 3579 counting rule for obvious reasons.

For illustration, apply the 3579 counting rule to all rotations $\mathit{\beta }=\mathbf{\Lambda }{\mathbf{P}}_{\alpha b}$ of the loading matrix $\mathbf{\Lambda }$ in Example (7) for which we already verified variance identification. We find that the counting rules are satisfied for all rotations $\mathit{\beta }$, and one might wonder if the 3579 counting rule can also lead to a sufficient criterion for variance identification under AR.

Sufficiency of counting rules was proven by Conti et al. (2014) in the context of a dedicated factor model, where the factor loading matrix $\mathbf{\Lambda }$ has a perfect simple structure, i.e., each measurement $y_{it}$ loads on at most one factor, as in (7) and (9); however, the sparsity matrix $\mathit{\delta }$, i.e., the exact position of the non-zero elements, is unknown. Conti et al. (2014) consider a dedicated factor model with correlated factors, ${\mathbf{f}}_t\sim {\mathcal{N}}_r\left(\mathbf{0},\mathbf{R}\right)$, and prove that the following conditions are both necessary and sufficient for uniqueness of the variance decomposition: the correlation matrix $\mathbf{R}$ is of full rank, $\mathrm{rk}\left(\mathbf{R}\right)=r$, and each column of $\mathbf{\Lambda }$ contains at least three nonzero loadings.

In the present paper, we aim for simple structures with potentially more than one non-zero loading in each row and generalize this work to basic factor models with orthogonal factors. We impose a GLT condition on the loading matrices $\mathbf{\Lambda }$ and provide sufficient conditions for variance identification in Section 4. These conditions are formulated as counting rules for the sparsity matrix $\mathit{\delta }$ of $\mathbf{\Lambda }$. More specifically, if the 3579 counting rule of Sato (1992, Theorem 3.3) holds for $\mathit{\delta }$, then this is a sufficient condition for the row-deletion property AR and, consequently, for variance identification, except for a set of measure 0.

Assuming a GLT structure also avoids the need to check the counting rule for all rotations of a given loading matrix. For illustration, we return to example (9) and use the counting rule of Sato (1992) to verify variance identification for $\mathit{\beta }$. If no condition to resolve rotational invariance is imposed on $\mathit{\beta }$, then the counting rule of Sato (1992) has to be checked not only for $\mathit{\beta }$, but for all rotations $\mathit{\beta }\mathbf{G}$ or, equivalently, for all possible rotations $\mathbf{\Lambda }{\mathbf{P}}_{\alpha b}$ of $\mathbf{\Lambda }$. Nearly all rotations have five non-zero rows and do not violate the counting rules, except for the eight rotations where $(\alpha ,b)\in \{0,\frac{\pi }{2},\pi ,\frac{3\pi }{2}\}\times \{0,1\}$ leads to trivial rotations of $\mathbf{\Lambda }$:

$$\begin{array}{cc}\hfill & \left(\begin{array}{cc}{\lambda }_{11}& 0\\ {\lambda }_{21}& 0\\ {\lambda }_{31}& 0\\ 0& {\lambda }_{42}\\ 0& {\lambda }_{52}\end{array}\right)\left(\begin{array}{cc}{\lambda }_{11}& 0\\ {\lambda }_{21}& 0\\ {\lambda }_{31}& 0\\ 0& -{\lambda }_{42}\\ 0& -{\lambda }_{52}\end{array}\right)\left(\begin{array}{cc}-{\lambda }_{11}& 0\\ -{\lambda }_{21}& 0\\ -{\lambda }_{31}& 0\\ 0& {\lambda }_{42}\\ 0& {\lambda }_{52}\end{array}\right)\left(\begin{array}{cc}-{\lambda }_{11}& 0\\ -{\lambda }_{21}& 0\\ -{\lambda }_{31}& 0\\ 0& -{\lambda }_{42}\\ 0& -{\lambda }_{52}\end{array}\right)\hfill \\ \hfill & \left(\begin{array}{cc}0& {\lambda }_{11}\\ 0& {\lambda }_{21}\\ 0& {\lambda }_{31}\\ {\lambda }_{42}& 0\\ {\lambda }_{52}& 0\end{array}\right)\left(\begin{array}{cc}0& -{\lambda }_{11}\\ 0& -{\lambda }_{21}\\ 0& -{\lambda }_{31}\\ {\lambda }_{42}& 0\\ {\lambda }_{52}& 0\end{array}\right)\left(\begin{array}{cc}0& {\lambda }_{11}\\ 0& {\lambda }_{21}\\ 0& {\lambda }_{31}\\ -{\lambda }_{42}& 0\\ -{\lambda }_{52}& 0\end{array}\right)\left(\begin{array}{cc}0& -{\lambda }_{11}\\ 0& -{\lambda }_{21}\\ 0& -{\lambda }_{31}\\ -{\lambda }_{42}& 0\\ -{\lambda }_{52}& 0\end{array}\right).\hfill \end{array}$$

(10)

On the other hand, if we impose an unordered GLT structure on $\mathit{\beta }$, then the set of all possible rotations $\mathit{\beta }\mathbf{G}$ reduces to the eight permutations in (10) and lack of variance identification can be verified by applying the counting rule of Sato (1992) to a single one of them.

3. Solving Rotational Invariance through GLT Structures

3.1. Ordered and Unordered GLT Structures

In this work, we introduce a new identification strategy to resolve rotational invariance based on the concept of generalized lower triangular (GLT) structures. Throughout this section, we assume that variance identification holds and $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ and ${\mathsf{\Sigma }}_0$ are uniquely determined from $\mathbf{\Omega }$. First, we introduce the notion of pivot rows of a factor loading matrix $\mathbf{\Lambda }$.2

Definition 2

(Pivot rows). Consider an $m\times r$ factor loading matrix Λ with r non-zero columns. For each column $j=1,\dots ,r$ of Λ, the pivot row $l_j$ is defined as the row index of the first non-zero factor loading in column j, i.e., ${\mathsf{\Lambda }}_{ij}=0,\forall i<l_j$ and ${\mathsf{\Lambda }}_{l_j,j}\ne 0$. The factor loading ${\mathsf{\Lambda }}_{l_j,j}$ is called the leading factor loading of column j.

Conditions on the pivot rows define different structures on the factor loading matrix. If the pivot rows lie on the main diagonal, i.e., $(l_1,\dots ,l_r)=(1,\dots ,r)$, then the factor loading matrix $\mathbf{\Lambda }$ exhibits a PLT structure and the leading factor loadings are equal to ${\mathsf{\Lambda }}_{jj}$ for all columns $j=1,\dots ,r$. GLT structures also require the pivot rows $(l_1,\dots ,l_r)$ to be pairwise distinct, but impose less stringent conditions on their position than a PLT structure. We will distinguish between two types of GLT structures, namely ordered and unordered GLT structures. In the first case, the pivots $l_1<\dots <l_r$ are ordered by size (see Definition 3), whereas they may take arbitrary positions $(l_1,\dots ,l_r)$ for unordered GLT structures, as long as they are pairwise distinct (a more formal definition is given below). Examples of all three structures are displayed in Figure 1 for a model with $r=6$ factors. Obviously, GLT structures contain the PLT structure as the special case where $l_j=j$ for $j=1,\dots ,r$.

Definition 3

(Ordered GLT structures). An $m\times r$ factor loading matrix Λ with full column rank r has an ordered GLT structure if the pivot rows $l_1,\dots ,l_r$ of Λ are ordered, i.e., $l_1<\dots <l_r$, and the leading factor loadings are positive, i.e., ${\mathsf{\Lambda }}_{l_j,j}>0$ for $j=1,\dots ,r$.

Since the “diagonal” loadings ${\mathsf{\Lambda }}_{jj}$ are allowed to be zero for an ordered GLT structure, measurements different from the first r ones may lead the factors. For each factor j, the leading variable is the response variable $y_{l_j,t}$ corresponding to the pivot row $l_j$. From a mathematical viewpoint, one could argue that the m measurements $y_{1t},\dots ,y_{mt}$ can be rearranged such that a PLT structure holds and the diagonal of the first r rows of $\mathbf{\Lambda }$ has non-zero elements. However, in practice, it is not obvious which measurements are able to constitute a new factor; in particular, if we expect the loading matrix $\mathbf{\Lambda }$ to exhibit a simple structure, but the measurements have no natural grouping. Even if such a grouping exists, rearranging the measurements might be challenging, if more than one factor is needed to explain the group-specific covariance.

As opposed to PLT, under GLT structures, we may learn from the data whether the ith row of the loading matrix is linearly independent of the previous rows 1 to $i-1$. Only in this case, the measurement $y_{it}$ constitutes an additional factor and row i defines the next pivot. In the empirical case study in Section 7.2, for instance, the measurements are grouped by industry and the pivot rows learned from the data show that this ordering is in conflict with the PLT assumption (see also Figure 2).

Imposing an ordered GLT structure resolves rotational invariance if the pivot rows are known. For any two ordered GLT matrices $\mathit{\beta }$ and $\mathbf{\Lambda }$ with identical pivot rows $l_1,\dots ,l_r$, the identity $\mathit{\beta }=\mathbf{\Lambda }\mathbf{P}$ evidently holds if $\mathbf{P}={\mathbf{I}}_r$. GLT structures where the pivot rows are unknown will be discussed in Section 3.3.

In some treatment, it is customary to impose conditions that resolve rotational invariance up to column and sign switching (see, e.g., Conti et al. (2014)). This trivial form of rotational invariance does not impose any additional mathematical challenges and is often convenient from a computational viewpoint, in particular for Bayesian inference (see also Frühwirth-Schnatter et al. (2023)). More formally, so-called signed permutations are introduced to permute the columns of the factor loading matrix $\mathbf{\Lambda }$ and to reverse the sign of all factor loadings in specific columns. Such a signed permutation $\mathit{\beta }$ of a loading matrix $\mathbf{\Lambda }$ is defined as

$$\begin{array}{c}\hfill \mathit{\beta }=\mathbf{\Lambda }{\mathbf{P}}_{\pm }{\mathbf{P}}_{\rho },\end{array}$$

(11)

where the permutation matrix ${\mathbf{P}}_{\rho }$ corresponds to one of the r! permutations of the r columns of $\mathbf{\Lambda }$ and introduces column switching. The reflection matrix ${\mathbf{P}}_{\pm }=\mathrm{Diag}\left(\pm 1,\dots ,\pm 1\right)$ corresponds to one of the $2^r$ possibilities to either keep or reverse the sign of each of the r columns of $\mathbf{\Lambda }$ and introduces sign switching. This generates a whole equivalence class of loading matrices given by all $2^{r}r!$ signed permutations $\mathit{\beta }=\mathbf{\Lambda }{\mathbf{P}}_{\pm }{\mathbf{P}}_{\rho }$ of $\mathbf{\Lambda }$.

It is easy to verify how identification up to signed permutations can be achieved for GLT structures. For $r=2$, for instance, all eight signed permutations of the ordered GLT structure $\mathbf{\Lambda }$ defined in (9) are depicted in (10). Applying all signed permutations to an arbitrary ordered GLT structure leads to the specification of so-called unordered GLT structures as loading matrices where the pivot rows $l_1,\dots ,l_r$ simply occupy r different rows (see Definition 4).

Definition 4

(Unordered GLT structures). An $m\times r$ factor loading matrix β with full column rank r has an unordered GLT structure if the pivot rows $l_1,\dots ,l_r$ of β are pairwise distinct.

In Definition 4, no order constraint is imposed on the pivot rows and no sign constraint is imposed on the leading factor loadings. This very general structure allows us to design highly efficient sampling schemes for Bayesian factor analysis under GLT structures (see Frühwirth-Schnatter et al. (2023)).

For unordered GLT structures with known pivots, rotational invariance is resolved only up to signed permutations; however, full identification can be easily obtained. Any unordered GLT structure $\mathit{\beta }$ has (unordered) pivot rows $l_1,\dots ,l_r$ occupying different rows. The corresponding ordered GLT structure $\mathbf{\Lambda }$ is recovered from $\mathit{\beta }$ by sorting the columns of $\mathit{\beta }$ such that the pivot rows of $\mathbf{\Lambda }$ are equal to the order statistics $l_{\left(1\right)}<\dots <l_{\left(r\right)}$ of the pivot rows $l_1,\dots ,l_r$ of $\mathit{\beta }$ (see again Figure 1). This procedure resolves rotational invariance, since the pivot rows $l_1,\dots ,l_r$ in the unordered GLT structure are distinct. Furthermore, imposing the condition ${\mathsf{\Lambda }}_{l_j,j}>0$ in each column j resolves sign switching: if ${\mathsf{\Lambda }}_{l_j,j}<0$, then the sign of all factor loadings ${\mathsf{\Lambda }}_{ij}$ in column j is reversed.

3.2. Rotation into GLT

Ordered GLT structures generalize the PLT constraint, but one might wonder how restrictive this condition still is. Theorem 1 proves that it is unrestrictive in the sense of Anderson and Rubin (1956) and Jöreskog (1969). We show that for any basic factor model with an unconstrained loading matrix $\mathit{\beta }$ there exists an equivalent representation involving an ordered GLT structure $\mathbf{\Lambda }$, which is related to $\mathit{\beta }$ by an orthogonal transformation, provided that uniqueness of the variance decomposition holds. See Appendix A for a proof.

Theorem 1

(Rotation into GLT). Let β be an arbitrary loading matrix with full column rank r and let ${\mathit{\beta }}_{l,•}$ denote the lth row of β. Then, the following statements hold:

(a) 

There exists an equivalent unique representation of β involving an ordered GLT structure Λ,

$$\begin{array}{c}\hfill \mathit{\beta }=\mathbf{\Lambda }{\mathbf{G}}^{\top },\end{array}$$

(12)

where $\mathbf{G}$ is a rotation matrix. Λ is called the GLT representation of β;

(b) 

To compute $\mathbf{G}$ from β, first find the smallest row index $l_1$ such that the $l_1$th row of β is not fully zero. Next, in an iterative manner, given indices $(l_1,\dots ,l_{i-1})$ for $2\le i\le r$, find the smallest row index $l_i$ such that ${\mathit{\beta }}_{l_1,•}$, …, ${\mathit{\beta }}_{l_{i-1},•}$ and ${\mathit{\beta }}_{l_i,•}$ together form a linearly independent set of vectors. After the last iteration, the rows ${\mathit{\beta }}_{l_1,•}$, …, ${\mathit{\beta }}_{l_r,•}$ form an $r\times r$ invertible matrix $\tilde{\mathit{\beta }}$. Then, $\mathbf{G}$ is the ‘Q’ part of the QR-decomposition of ${\tilde{\mathit{\beta }}}^{\top }$.3

Does a similar result hold for PLT structures? The answer is definitely no, as has already been established in Section 2; in (4), for example. As mentioned above, ordered GLT structures encompass PLT structures as a special case. Hence, as a consequence of Theorem 1, if a PLT representation $\mathbf{\Lambda }$ exists for a loading matrix $\mathit{\beta }=\mathbf{\Lambda }\mathbf{P}$, then the ordered GLT representation in (12) automatically reduces to the PLT structure $\mathbf{\Lambda }$ with the “rotation into GLT” being $\mathbf{G}={\mathbf{P}}^{\top }$. On the other hand, if the ordered GLT representation $\mathbf{\Lambda }$ differs from a PLT structure, then no equivalent PLT representation exists. Hence, forcing a PLT structure in the representation (1) may introduce a bias in estimating the marginal covariance matrix $\mathbf{\Omega }$.

In practice, the pivot rows $l_1,\dots ,l_r$ of a GLT structure are unknown and need to be identified from the marginal covariance matrix $\mathbf{\Omega }$ for a given number of factors r. Given variance identification, i.e., assuming that the cross-covariance matrix $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ is identified, an interesting question regarding the identification of a GLT factor model is whether $\mathbf{\Lambda }$ is uniquely identified from $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ if the pivot rows are unknown. Non-trivial rotations $\tilde{\mathbf{\Lambda }}=\mathbf{\Lambda }\mathbf{P}$ of a loading matrix $\mathbf{\Lambda }$ might exist such that $\tilde{\mathbf{\Lambda }}{\tilde{\mathbf{\Lambda }}}^{\top }=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$, while the pivot rows ${\tilde{l}}_1,\dots ,{\tilde{l}}_r$ of $\tilde{\mathbf{\Lambda }}$ are different from the pivot rows $l_1,\dots ,l_r$ of $\mathbf{\Lambda }$.

For ordered GLT structures with unknown pivots, we obtain as an immediate (and somewhat trivial) consequence of Theorem 1 that $\tilde{\mathbf{\Lambda }}$ is the unique ordered GLT representation of $\mathbf{\Lambda }$, and therefore $\mathbf{P}={\mathbf{I}}_r$. Indeed, when we compute the rotation matrix $\mathbf{G}$ in $\mathbf{\Lambda }=\tilde{\mathbf{\Lambda }}{\mathbf{G}}^{\top }$ as described in Part (b), we find that the matrix $\tilde{\mathit{\beta }}$ is equal to the pivot rows of $\mathbf{\Lambda }$. Therefore, ${\tilde{\mathit{\beta }}}^{\top }$ is an upper triangular matrix of full rank and the ‘Q’ part of its QR-decomposition is equal to the identity matrix. Since $\mathbf{\Lambda }=\tilde{\mathbf{\Lambda }}{\mathbf{P}}^{\top }$ and rotation into GLT is unique, we obtain that ${\mathbf{P}}^{\top }$ = ${\mathbf{G}}^{\top }$ = ${\mathbf{I}}_r$. This insight is formalized in Corollary 1.

Corollary 1.

An ordered GLT structure is uniquely identified, provided that uniqueness of the variance decomposition holds, i.e., if Λ and $\tilde{\mathbf{\Lambda }}$ are GLT matrices, respectively, with pivot rows $l_1<\dots <l_r$ and ${\tilde{l}}_1<\dots <{\tilde{l}}_r$ that satisfy $\tilde{\mathbf{\Lambda }}{\tilde{\mathbf{\Lambda }}}^{\top }=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$, then $\tilde{\mathbf{\Lambda }}=\mathbf{\Lambda }$ and, consequently, $({\tilde{l}}_1,\dots ,{\tilde{l}}_r)=(l_1,\dots ,l_r)$.

For unordered GLT structures with unknown pivots, the factor loading matrix $\mathbf{\Lambda }$ is identified from $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ up to signed permutations and $\tilde{\mathbf{\Lambda }}$ and $\mathbf{\Lambda }$ have the same pivot rows $l_1,\dots ,l_r$, provided that $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ is identified. This can be easily shown by extending Corollary 1 to unordered GLT structures, as any signed permutation $\tilde{\mathbf{\Lambda }}=\mathbf{\Lambda }{\mathbf{P}}_{\rho }{\mathbf{P}}_{\pm }$ of $\mathbf{\Lambda }$ is uniquely identified from $\tilde{\mathbf{\Lambda }}{\tilde{\mathbf{\Lambda }}}^{\top }=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$. To summarize, Theorem 1 together with Corollary 1 establish the existence and uniqueness of GLT structures for any basic factor model, provided that uniqueness of the variance decomposition holds.

3.3. Simple GLT Structures

In Definitions 3 and 4, “structural” zeros are introduced for a GLT structure for all factor loading above the pivot row $l_j$, while the factor loading ${\mathsf{\Lambda }}_{l_j,j}$ in the pivot row is non-zero by definition. We call $\mathbf{\Lambda }$ a dense GLT structure if all loadings below the pivot rows are unconstrained and can take any value in $\mathbb{R}$.

A simple GLT structure results if factor loadings at unspecified places below the pivot rows are zero and only the remaining loadings are unconstrained. As discussed in Section 2.2, any simple structure can be characterized by the sparsity matrix $\mathit{\delta }$, defined as a binary indicator matrix of 0/1s of the same size as $\mathbf{\Lambda }$, where $\mathit{\delta }=\mathbb{I}(\mathbf{\Lambda }\ne 0)$ and the indicator function is applied element-wise. Evidently, the sparsity matrix of a GLT structure $\mathbf{\Lambda }$ exhibits the same pivots as $\mathbf{\Lambda }$, regardless of whether the structure is ordered or unordered (see again Figure 1 for an illustration).

In sparse Bayesian factor analysis, single factor loadings take zero-values with positive probability and the corresponding sparsity matrix $\mathit{\delta }$ is a random binary matrix that has to be identified from the data (see Section 6.1 for more details). Identification in sparse Bayesian factor analysis has to provide conditions under which the entire 0/1 pattern in $\mathit{\delta }$ can be identified from a given covariance matrix $\mathbf{\Omega }$, if $\mathit{\delta }$ is unknown. Whether this is possible hinges on variance identification, i.e., whether the decomposition of $\mathbf{\Omega }$ into $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ and ${\mathbf{\Sigma }}_0$ is unique. How variance identification can be verified for simple GLT structures is investigated in detail in Section 4. Let us assume at this point that variance identification holds, i.e., the cross-covariance matrix $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ is identified. Then, an important step toward the identification of a factor model with a simple structure is to verify whether the 0/1 pattern of $\mathbf{\Lambda }$, characterized by the sparsity matrix $\mathit{\delta }$, is uniquely identified from $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$. If $\mathbf{\Lambda }$ exhibits an ordered GLT structure, then it follows immediately from Corollary 1 that the indicator matrix $\mathit{\delta }$ is, indeed, uniquely identified from $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$, since $\mathbf{\Lambda }$ is identified and ${\delta }_{ij}=0$ if ${\mathsf{\Lambda }}_{ij}=0$ for all $i,j$.

We would like to emphasize that in sparse Bayesian factor analysis with unconstrained loading matrices $\mathbf{\Lambda }$, this is not necessarily the case. The sparsity matrix $\mathit{\delta }$ is, in general, not uniquely identified from $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$, because rotations may change the zero pattern in $\mathit{\beta }=\mathbf{\Lambda }\mathbf{P}$, while $\mathit{\beta }{\mathit{\beta }}^{\top }=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$. For illustration, let us return to Example (4) and assume that the sparsity matrix $\mathit{\delta }$ is unknown. While $\mathit{\delta }$ is uniquely identified from $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ under GLT, two distinct solutions $\mathit{\delta }$ exist if the loading matrix is left unconstrained and any rotation $\mathit{\beta }=\mathbf{\Lambda }{\mathbf{P}}_{\alpha b}$ of $\mathbf{\Lambda }$ is an admissible solution. For all rotations where $(\alpha ,b)\in \{0,\frac{\pi }{2},\pi ,\frac{3\pi }{2}\}\times \{0,1\}$, $\mathit{\beta }$ corresponds to one of the eight signed permutations of $\mathbf{\Lambda }$ (similar to the rotations in (10)) and $\mathit{\delta }$ is equal to the sparsity matrix of $\mathbf{\Lambda }$ up to column switching. For all other rotations, all elements of $\mathit{\beta }$ are different from zero and $\mathit{\delta }$ is simply a matrix of ones.

4. Variance Identification for Simple GLT Structures

As mentioned in the previous sections, conditions imposed on the structure of a factor loading matrix $\mathbf{\Lambda }$ will resolve rotational invariance only if the uniqueness of the variance decomposition holds and the cross-covariance matrix $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ is identified. However, such conditions do not necessarily guarantee the uniqueness of the variance decomposition.

One exception is the popular factor analysis model where $\mathbf{\Lambda }$ takes the form of a dense PLT matrix, where all factor loadings below the main diagonal are unconstrained and may take any value in $\mathbb{R}$. For this model, condition AR and hence variance identification holds, except for a set of measure 0, provided that the condition

$$m\ge 2r+1⟺r\le \frac{m-1}{2}$$

(13)

on the number of factors is satisfied for the given number m of measurements. For simple structures variance identification is easily violated; consider, e.g., a simple PLT loading matrix where only a single factor loading below the diagonal is nonzero in some column.

In Section 4.1, we derive sufficient conditions for variance identification of simple GLT structures based on the 3579 counting rule of Sato (1992, Theorem 3.3). In Section 4.2, we discuss how to verify variance identification for simple GLT structures in practice.

4.1. Counting Rules for Variance Identification

First, of all, we need not constrain the factor loading matrix to take the form of an ordered GLT structure, since variance identification is invariant to signed permutations. If we can verify the variance identification for a single signed permutation $\mathit{\beta }=\mathbf{\Lambda }{\mathbf{P}}_{\pm }{\mathbf{P}}_{\rho }$ of a loading matrix $\mathbf{\Lambda }$, as defined in (11), then variance identification of $\mathbf{\Lambda }$ holds, since $\mathit{\beta }$ and $\mathbf{\Lambda }$ imply the same cross-covariance matrix $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$. Hence, we focus in this section on the variance identification of unordered GLT structures. We will show how to verify from the 0/1 pattern of the sparsity matrix $\mathit{\delta }$ of an unordered, possibly simple GLT structure $\mathit{\beta }$, whether the row-deletion property AR holds for $\mathit{\beta }$ and all its signed permutations. Our condition is a structural counting rule expressed solely in terms of the sparsity matrix $\mathit{\delta }$ underlying $\mathit{\beta }$, and does not involve the values of the unconstrained factor loadings in $\mathit{\beta }$, which can take any value in $\mathbb{R}$.

Next, we recall the so-called extended row-deletion property in Definition 5, introduced by Tumura and Sato (1980) which applies to arbitrary loading matrices $\mathit{\beta }$.

Definition 5

(Extended row-deletion property $\mathrm{RD}(r,s)$). An $m\times r$ factor loading matrix β satisfies the row-deletion property $RD(r,s)$ if the following condition is satisfied: whenever $s\in {\mathbb{N}}_0$ rows are deleted from β, then two disjoint submatrices of rank r remain.

The row-deletion property of Anderson and Rubin (1956) results as a special case where $s=1$. As will be shown in Section 5, the extended row-deletion properties $\mathrm{RD}(r,s)$ for $s>1$ are useful in exploratory factor analysis, when the factor dimension r is unknown. In Definition 6, we introduce a counting rule for binary matrices.

Definition 6

(Counting rule $\mathrm{CR}(r,s)$). Let δ be an $m\times r$ binary matrix. For each $q=1,\dots ,r$, consider all submatrices ${\mathit{\delta }}_{q,\ell }$, $\ell =1,\dots ,\left(\genfrac{}{}{0pt}{}{r}{q}\right)$, built from q columns of δ. δ is said to satisfy the $CR(r,s)$ counting rule for $s\in {\mathbb{N}}_0$ if the matrix ${\mathit{\delta }}_{q,\ell }$ has at least $2\ell +s$ nonzero rows for all $(q,\ell )$.

Note that the counting rule $\mathrm{CR}(r,s)$, like the extended row-deletion property $\mathrm{RD}(r,s)$, is invariant to signed permutations. Lemma A1 in Appendix A summarizes further useful properties of $\mathrm{CR}(r,s)$.

For a given binary matrix $\mathit{\delta }$ of dimension $m\times r$, let ${\mathsf{\Theta }}_{\delta }$ be the space generated by the non-zero elements of all unordered simple GLT structures $\mathit{\beta }$ with the same sparsity matrix $\mathit{\delta }$ and all their $2^{r}r!-1$ signed permutations $\mathit{\beta }{\mathbf{P}}_{\pm }{\mathbf{P}}_{\rho }$. We prove in Theorem 2 that for simple GLT structures, the counting rule $\mathrm{CR}(r,s)$ and the extended row-deletion property $\mathrm{RD}(r,s)$ are equivalent conditions for all simple loading matrices in ${\mathsf{\Theta }}_{\delta }$ with the same sparsity matrix $\mathit{\delta }$, except for a set of measure 0.

Theorem 2.

Let δ be a binary $m\times r$ matrix with an unordered GLT structure. Then, the following holds:

(a) 

If δ violates the counting rule $CR(r,s)$, then the extended row-deletion property $RD(r,s)$ is violated for all simple structures $\mathit{\beta }\in {\mathsf{\Theta }}_{\delta }$ generated by δ;

(b) 

If δ satisfies the counting rule $CR(r,s)$, then the extended row-deletion property $RD(r,s)$ holds for all simple structures $\mathit{\beta }\in {\mathsf{\Theta }}_{\delta }$ except for a set of measure 0.

See Appendix A for a proof. The special case $s=1$ is relevant for verifying the row-deletion property AR. It proves that, for unordered simple GLT structures, the 3579 counting rule of Sato (1992) is not only a necessary, but also a sufficient condition for AR to hold. In addition, this means that the counting rule needs to be verified only for the sparsity matrix $\mathit{\delta }$ of a single signed permutation $\mathit{\beta }=\mathbf{\Lambda }{\mathbf{P}}_{\pm }{\mathbf{P}}_{\rho }$ rather than for every non-singular matrix $\mathbf{G}$. This result is summarized in Corollary 2.

Corollary 2

(Variance identification rule for simple GLT structures). For any unordered simple GLT structure β of size $m\times r$, the following holds:

(a) 

If a binary matrix δ of size $m\times r$ satisfies the 3579 counting rule, i.e., every column of δ has at least three non-zero elements, every pair of columns at least five, and, more generally, every possible combination of $q=3,\dots ,r$ columns has at least $2q+1$ non-zero elements, then variance identification is given for all simple unordered GLT structures $\mathit{\beta }\in {\mathsf{\Theta }}_{\delta }$ except for a set of measure 0; i.e., for any other factor decomposition of the marginal covariance matrix $\mathbf{\Omega }=\mathit{\beta }{\mathit{\beta }}^{\top }+\mathbf{\Sigma }=\tilde{\mathit{\beta }}{\tilde{\mathit{\beta }}}^{\top }+\tilde{\mathbf{\Sigma }}$, where $\tilde{\mathit{\beta }}$ is an unordered GLT matrix, it follows that $\tilde{\mathbf{\Sigma }}=\mathbf{\Sigma }$, i.e., $\tilde{\mathit{\beta }}{\tilde{\mathit{\beta }}}^{\top }=\mathit{\beta }{\mathit{\beta }}^{\top }$, and $\tilde{\mathit{\beta }}=\mathit{\beta }{\mathbf{P}}_{\pm }{\mathbf{P}}_{\rho }$.

(b) 

If a binary matrix δ of size $m\times r$ violates the 3579 counting rule, then for all $\mathit{\beta }\in {\mathsf{\Theta }}_{\delta }$, the row-deletion property AR does not hold.

(c) 

For $r=1$, $r=2$, and $r=3$, condition $CR(r,1)$ is both sufficient and necessary for variance identification.

A few comments are in order. If $\mathit{\delta }$ satisfies $\mathrm{CR}(r,1)$, then $\mathbf{AR}$ holds for all $\mathit{\beta }\in {\mathsf{\Theta }}_{\delta }$ and a sufficient condition for variance identification is satisfied. As shown by Anderson and Rubin (1956), $\mathbf{AR}$ is a necessary condition for variance identification only for $r=1$ and $r=2$. Tumura and Sato (1980, Theorem 3) show the same for $r=3$, provided that $m\ge 7$. It follows that $\mathrm{CR}(r,1)$ is a necessary and sufficient condition for variance identification for all models summarized in (c). In all other cases, variance identification may hold for loading matrices $\mathit{\beta }\in {\mathsf{\Theta }}_{\delta }$, even if $\mathit{\delta }$ violates $\mathrm{CR}(r,1)$.

The definition of unordered GLT structures given in Section 3 imposes no condition on the position of the pivot rows $l_1,\dots ,l_r$ beyond the assumption that they are distinct. This may lead to GLT structures that can never satisfy the 3579 rule, even if all elements below the pivot rows are non-zero. Consider, for instance, a GLT matrix with the pivot row in column r being equal to $l_r=m-1$. The loading matrix has at most two nonzero elements in column r and violates the necessary condition for variance identification. This example shows that there is an upper bound for the pivot elements beyond which the 3579 rule can never hold. This insight is formalized in Definition 7.

Definition 7.

An unordered GLT structure β fulfills condition GLT-AR, if the pivots $l_1,\dots ,l_r$ satisfy the following condition, where $z_j$ is the rank of $l_j$ in the ordered sequence $l_{\left(1\right)}<\dots <l_{\left(r\right)}$:

$$\begin{array}{c}\hfill l_j\le m-2(r-z_j+1).\end{array}$$

(14)

For an ordered GLT structure Λ with pivots $l_1<\dots <l_r$, condition GLT-AR reduces to:

$$\begin{array}{c}\hfill l_j\le m-2(r-j+1).\end{array}$$

(15)

For the special case of a PLT structure where $l_j=j$, condition (15) reduces to the upper bound for the number of factors given in (13).

For dense GLT structures, condition GLT-AR is a sufficient condition for AR. However, for simple GLT structures with zeros below the pivot rows, GLT-AR is only a necessary condition for AR, as discussed above for an example, and the 3579 rule has to be verified explicitly. Very conveniently for verifying variance identification in factor analysis based on GLT structures, Theorem 2 and Corollary 2 operate solely on the sparsity matrix $\mathit{\delta }$ summarizing the simple structure in $\mathit{\beta }$.

4.2. Variance Identification in Practice

To verify $\mathrm{CR}(r,s)$ in practice, all submatrices of q columns have to be extracted from the sparsity matrix $\mathit{\delta }$ to verify if at least $2q+1$ rows of this submatrix are non-zero. For $q=1,2,r-1,r$, this condition is easily verified from simple functionals of $\mathit{\delta }$; see Corollary 3, which follows immediately from Theorem 2 (see Appendix A for details).

Corollary 3

(Simple counting rules for $\mathrm{CR}(\mathbf{r}\mathbf{,}\mathbf{s})$). Let δ be a $m\times r$ unordered GLT sparsity matrix. The following conditions on δ are necessary for $CR(r,s)$ to hold:

$$\begin{array}{ccc}& {\mathbf{1}}_{r\times m}·\mathit{\delta }+{\mathit{\delta }}^{\top }({\mathbf{1}}_{m\times r}-\mathit{\delta })\ge 4+s-2{\mathbf{I}}_r,& \hfill \end{array}$$

(16)

$$\begin{array}{ccc}& {\mathbf{1}}_{1\times m}·\mathbb{I}({\mathit{\delta }}^{★}>0)\ge 2r+s,{\mathit{\delta }}^{★}=\mathit{\delta }·{\mathbf{1}}_{r\times 1},& \hfill \end{array}$$

(17)

$$\begin{array}{ccc}& {\mathbf{1}}_{1\times m}·\mathbb{I}({\mathit{\delta }}^{★}>0)\ge 2(r-1)+s,{\mathit{\delta }}^{★}=({\mathbf{1}}_{m\times m}-{\mathbf{I}}_m)·\mathit{\delta },& \hfill \end{array}$$

(18)

where the indicator function $\mathbb{I}({\mathit{\delta }}^{★}>0)$ is applied element-wise and ${\mathbf{1}}_{n\times k}$ denotes an $n\times k$ matrix of ones. For $r\le 4$, these conditions are also sufficient for $CR(r,s)$ to hold for δ.

Using Corollary 3 for $s=1$, one can efficiently verify if the 3579 counting rule and hence the row-deletion property AR holds for simple unordered GLT factor models with up to four ($r\le 4$) factors. For models with more than four factors ($r>4$), a more elaborated strategy is needed. After checking the conditions of Corollary 3, $\mathrm{CR}(r,s)$ could be verified for a given binary matrix $\mathit{\delta }$ by iterating over all remaining $r!/(q!(r-q)!)$ subsets of $q=3,\dots ,r-2$ columns of $\mathit{\delta }$. While this is a finite task, such a naïve approach may need to visit $2^r-1$ matrices in order to make a decision and the combinatorial explosion quickly becomes an issue in practice as r increases. Recent work by Hosszejni and Frühwirth-Schnatter (2022) establishes the applicability of this framework for large models.4

5. Identification in Exploratory Factor Analysis

In this section, we discuss how the concept of GLT structures is helpful for addressing identification problems in exploratory factor analysis.

5.1. Exploratory Factor Analysis

Consider data $\{{\mathbf{y}}_1,\dots ,{\mathbf{y}}_T\}$ from a zero-mean multivariate Gaussian distribution, where an investigator wants to perform factor analysis since she expects that the covariances of m measurements are driven by common factors. In practice, the number of factors is often unknown and it may be uncertain whether all measurements are actually correlated. It is then common to employ exploratory factor analysis (EFA) by fitting the following basic factor model with an assumed maximum number of factors, $H,$ to all measurements in ${\mathbf{y}}_t$:

$$\begin{array}{c}\hfill {\mathbf{y}}_t={\mathit{\beta }}_H{\mathbf{f}}_t+{\mathit{ϵ}}_t,{\mathit{ϵ}}_t\sim {\mathcal{N}}_m\left(\mathbf{0},{\mathbf{\Sigma }}_H\right),\end{array}$$

(19)

where ${\mathit{\beta }}_H$ is an $m\times H$ loading matrix, not necessarily of full column rank, ${\mathbf{f}}_t\sim {\mathcal{N}}_H\left(\mathbf{0},{\mathbf{I}}_H\right)$, and ${\mathbf{\Sigma }}_H$ is a diagonal matrix with strictly positive entries. The EFA model (19) is potentially overfitting in two ways. First, if some measurements in ${\mathbf{y}}_t$ are uncorrelated with the remaining measurements, then ${\mathit{\beta }}_H$ allows for too many non-zero rows. However, as will be discussed in Section 5.3, such irrelevant measurements are easily identified. Second, the assumed number of factors H is possibly larger than the true number of factors r, i.e., ${\mathit{\beta }}_H$ has too many columns. The goal is then to extract the true number of factors from the non-zero columns of ${\mathit{\beta }}_H$, collected in a $m\times k$ submatrix ${\mathit{\beta }}_k$ of rank k. Before we discuss this challenging problem in more detail in Section 6.3, additional identification issues for overfitting factor models have to be addressed.

We assume that the data are generated by a basic factor model with error covariance matrix ${\mathbf{\Sigma }}_0$ and a loading matrix $\mathbf{\Lambda }$ of factor dimension equal to r. Therefore, ${\mathbf{y}}_t\sim {\mathcal{N}}_m\left(\mathbf{0},\mathbf{\Omega }\right)$, where the covariance matrix $\mathbf{\Omega }$ has a representation as in (2):

$$\begin{array}{c}\hfill \mathbf{\Omega }=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }+{\mathbf{\Sigma }}_0.\end{array}$$

(20)

Furthermore, we assume that variance identification holds for (20) and the true cross-correlation matrix $\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ as well as ${\mathbf{\Sigma }}_0$ are uniquely determined from $\mathbf{\Omega }$.

Two questions arise in this context and can be answered based on Reiersøl (1950). First, under which conditions the covariance matrix is implied by the loading matrix ${\mathit{\beta }}_k$ extracted from the EFA model (19) equivalent to the true covariance matrix of the data, i.e., when does the following hold (note that ${\mathbf{\Sigma }}_k={\mathbf{\Sigma }}_H)$:

$$\begin{array}{c}\hfill {\mathit{\beta }}_k{\mathit{\beta }}_k^{\top }+{\mathbf{\Sigma }}_k=\mathbf{\Omega }?\end{array}$$

(21)

Second, if such an equivalent representation exists, does variance identification still hold, i.e., can ${\mathit{\beta }}_k{\mathit{\beta }}_k^{\top }$ and ${\mathbf{\Sigma }}_k$ be uniquely determined from (21)?

It follows from Reiersøl (1950) that no equivalent representation exists, if $k\le H<r$. Reiersøl (1950) show that the true number of factors r is equal to the smallest value k that satisfies (21). Hence, if the assumed number of factors H in the EFA model is equal to the true number of factors r and ${\mathit{\beta }}_H$ has full column rank, then such a representation obviously exists. Since variance identification of (20) holds, we obtain that ${\mathit{\beta }}_H{\mathit{\beta }}_H^{\top }={\mathit{\beta }}_r{\mathit{\beta }}_r^{\top }=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }$ and ${\mathbf{\Sigma }}_H={\mathbf{\Sigma }}_r={\mathbf{\Sigma }}_0$. Consequently, ${\mathit{\beta }}_H={\mathit{\beta }}_r=\mathbf{\Lambda }\mathbf{P}$ is a rotation of $\mathbf{\Lambda }$.

Equivalent representations also exist for $H\ge k>r$, in which case the EFA model is overfitting. It follows from Reiersøl (1950, Theorem 3.3) that any structure $(\mathbf{\Lambda },{\mathbf{\Sigma }}_0)$ in a basic factor model of factor dimension r creates infinitely many solutions $({\mathit{\beta }}_k,{\mathbf{\Sigma }}_k)$ of dimension $k=r+1,\dots ,H$ which imply the same covariance matrix $\mathbf{\Omega }$ as $(\mathbf{\Lambda },{\mathbf{\Sigma }}_0)$, i.e.,

$$\begin{array}{c}\hfill \mathbf{\Omega }={\mathit{\beta }}_k{\mathit{\beta }}_k^{\top }+{\mathbf{\Sigma }}_k=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }+{\mathbf{\Sigma }}_0,\end{array}$$

(22)

where the rank of the loading matrix ${\mathit{\beta }}_k$ is equal to $k>r$ and ${\mathbf{\Sigma }}_k$ is a positive definite matrix different from ${\mathbf{\Sigma }}_0$ (see also Geweke and Singleton (1980)). Since infinitely many solutions can be created that differ in ${\mathbf{\Sigma }}_k$, the decomposition (22) is no longer variance identified.

For illustration, we return to Example (4), where $r=2$, and construct infinitely many solutions for $k=3$. The first two columns of ${\mathit{\beta }}_3$ are equal to $\mathbf{\Lambda }$, the third column is a so-called spurious factor with a single non-zero loading, and ${\mathbf{\Sigma }}_3$ is defined as follows:

$$\begin{array}{c}\hfill {\mathit{\beta }}_3=\left(\begin{array}{ccc}{\lambda }_{11}& 0& 0\\ {\lambda }_{21}& 0& {\beta }_{23}\\ {\lambda }_{31}& 0& 0\\ {\lambda }_{41}& {\lambda }_{42}& 0\\ 0& {\lambda }_{52}& 0\\ 0& {\lambda }_{62}& 0\end{array}\right),{\mathbf{\Sigma }}_3=\mathrm{Diag}\left({\sigma }_1^2,{\sigma }_2^2-{\beta }_{23}^2,{\sigma }_3^2,{\sigma }_4^2,{\sigma }_5^2,{\sigma }_6^2\right).\end{array}$$

(23)

We can place the spurious factor loading ${\beta }_{i3}$ in any row $i\in \{1,\dots ,m\}$ and it can take any value satisfying $0<{\beta }_{i3}^2<{\sigma }_i^2$. It is easy to verify that any such pair $({\mathit{\beta }}_3,{\mathbf{\Sigma }}_3)$ indeed implies the same covariance matrix $\mathbf{\Omega }$ as the pair $(\mathbf{\Lambda },{\mathbf{\Sigma }}_0)$. Therefore, while variance identification holds for $r=2$, it fails for $k=3$, because infinitely many solutions with different error covariance matrices ${\mathbf{\Sigma }}_3$ are available, depending on the choice of i and the spurious factor loading ${\beta }_{i3}$. On the other hand, the spurious column is easily spotted for any such solution ${\mathit{\beta }}_3$ and the true loading matrix $\mathbf{\Lambda }$ is clearly identified from the two remaining columns. One may even recover ${\mathbf{\Sigma }}_0$, by adding ${\beta }_{i3}^2$ to the ith diagonal element of ${\mathbf{\Sigma }}_k$.

So far in this section, we imposed no conditions that resolve rotational invariance, either on the true loading matrix $\mathbf{\Lambda }$ or on the overfitting loading matrix ${\mathit{\beta }}_3$. In this case, additional solutions are obtained by rotating the loading matrices ${\mathit{\beta }}_3$ in (23), e.g.,

$$\begin{array}{c}\hfill {\tilde{\mathit{\beta }}}_3=\left(\begin{array}{ccc}0& 0& {\lambda }_{11}\\ {\beta }_{23}& 0& {\lambda }_{21}\\ 0& 0& {\lambda }_{31}\\ 0& -{\lambda }_{42}& {\lambda }_{41}\\ 0& -{\lambda }_{52}& 0\\ 0& -{\lambda }_{62}& 0\end{array}\right),{\widetilde{\tilde{\mathit{\beta }}}}_3=\left(\begin{array}{ccc}-{\lambda }_{11}sin\alpha & 0& {\lambda }_{11}cos\alpha \\ {\beta }_{23}cos\alpha -{\lambda }_{21}sin\alpha & 0& {\lambda }_{21}cos\alpha \\ -{\lambda }_{31}sin\alpha & 0& {\lambda }_{31}cos\alpha \\ -{\lambda }_{41}sin\alpha & {\lambda }_{42}& {\lambda }_{41}cos\alpha \\ 0& {\lambda }_{52}& 0\\ 0& {\lambda }_{62}& 0\end{array}\right),\end{array}$$

(24)

both combined with the same ${\mathbf{\Sigma }}_3$ as in (23). The first solution ${\tilde{\mathit{\beta }}}_3$ is a signed permutation of ${\mathit{\beta }}_3$, while the second solution ${\widetilde{\tilde{\mathit{\beta }}}}_3$ combines a signed permutation of ${\mathit{\beta }}_3$ with a rotation of the spurious and $\mathbf{\Lambda }$’s first column involving ${\mathbf{P}}_{\alpha b}$. In ${\tilde{\mathit{\beta }}}_3$, the spurious column is still easily spotted and the true loading matrix $\mathbf{\Lambda }$ is identified from the two remaining columns of ${\mathit{\beta }}_3$ up to column and sign switching.5 However, for ${\widetilde{\tilde{\mathit{\beta }}}}_3$ the presence of a spurious column is no longer obvious. While the second column of $\mathbf{\Lambda }$ still pops up, the first column is disguised.

To summarize, further identifiability problems arise for $r<k\le H$ beyond the ones discussed in the previous sections for a basic factor model where r is known. We discuss these problems in a more formal manner in Section 5.2 and investigate the class of overfitting GLT structures where an unordered GLT condition is imposed on the non-zero columns ${\mathit{\beta }}_k$ of the loading matrix ${\mathit{\beta }}_H$ in the EFA model (19). We apply results by Tumura and Sato (1980) to this class and prove that under this condition, (a) spurious factors in ${\mathit{\beta }}_k$ are as easily spotted as in ${\tilde{\mathit{\beta }}}_3$, and (b) the non-spurious columns are an unordered GLT representation of the true loading matrix $\mathbf{\Lambda }$. Our strategy relies on the concept of extended variance identification and the extended row-deletion property introduced by Tumura and Sato (1980), where more than one row is deleted from the loading matrix. The extended counting rule $\mathrm{RD}(r,s)$ with $s>1$ introduced in Definition 5 in Section 4 will be useful in this context.

5.2. “Revealing the Truth” in an Overfitting EFA Model

As illustrated in Section 5.1, fundamental identifiability problems arise if an overfitting EFA model with $r<k\le H$ is fitted to data arising from a basic factor model of factor dimension r. The question arises if we could, nevertheless, recover the true loading matrix $\mathbf{\Lambda }$ from the non-zero columns ${\mathit{\beta }}_k$ of ${\mathit{\beta }}_H$. We will show how this can be achieved mathematically by combining the important work by Tumura and Sato (1980) with the framework of GLT structures.

We have demonstrated in Section 5.1, using examples (23) and (24) for a model where $k=r+1$, how to construct infinitely many solutions $({\mathit{\beta }}_k,{\mathbf{\Sigma }}_k)$ with the same covariance matrix $\mathbf{\Omega }={\mathit{\beta }}_k{\mathit{\beta }}_k^{\top }+{\mathbf{\Sigma }}_k$ as the true factor model. First, set the first r columns of ${\mathit{\beta }}_k$ to $\mathbf{\Lambda }$ and append a spurious column to its right with the single non-zero loading ${\beta }_{l_k,k}$ lying in any row $l_k\in \{1,\dots ,m\}$ taking any value that satisfies $0<{\beta }_{l_k,k}^2<{\sigma }_{l_k}^2$. Then, reduce the idiosyncratic variance in row $l_k$ to ${\sigma }_{l_k}^2-{\beta }_{l_k,k}^2$, and finally apply an arbitrary rotation $\mathbf{P}$:

$$\begin{array}{c}\hfill {\mathit{\beta }}_k=\left(\begin{array}{cc}\mathbf{\Lambda }& \left|\begin{array}{c}0\\ {\beta }_{l_k,k}\\ 0\end{array}\right.\end{array}\right)\mathbf{P},{\mathbf{\Sigma }}_k=\mathrm{Diag}\left({\sigma }_1^2,\dots ,{\sigma }_{l_k}^2-{\beta }_{l_k,k}^2,\dots ,{\sigma }_m^2\right).\end{array}$$

(25)

The following questions arise: under which conditions is (25) an exhaustive representation of all possible solutions ${\mathit{\beta }}_k$ of rank k where the degree of overfitting defined as $s=k-r$ is equal to one? How can all possible solutions ${\mathit{\beta }}_k$ be represented if $s>1$?

Such identifiability problems in overfitting EFA models have been analyzed in depth by Tumura and Sato (1980). They provide a general representation of the factor loading matrix ${\mathit{\beta }}_k$ in an overfitting representation (22) with $k>r$. In addition, they show that a stronger condition than $\mathrm{RD}(r,1)$ is needed for $\mathbf{\Lambda }$ in the underlying variance decomposition (21) to ensure that only spurious factors, and no additional common factors, are present in (21).

Theorem 3.

(Tumura and Sato 1980, Theorem 1) Suppose that Ω has a decomposition as in (20) with r factors, and that for some $S\in \mathbb{N}$ with $2r+S+1\le m$ the extended row-deletion property $RD(r,1+S)$ holds for Λ. If Ω has another decomposition such that $\mathbf{\Omega }={\mathit{\beta }}_k{\mathit{\beta }}_k^{\top }+{\mathbf{\Sigma }}_k$ where ${\mathit{\beta }}_k$ is an $m\times (r+s)$-matrix of rank $k=r+s$ with $1\le s\le S$, then there exists an orthogonal matrix ${\mathbf{T}}_k$ of rank k such that

$$\begin{array}{c}\hfill {\mathit{\beta }}_k{\mathbf{T}}_k=\left(\begin{array}{cc}\mathbf{\Lambda }& {\mathbf{M}}_s\end{array}\right),{\mathbf{\Sigma }}_k={\mathbf{\Sigma }}_0-{\mathbf{M}}_s{\mathbf{M}}_s^{\top },\end{array}$$

(26)

where the off-diagonal elements of ${\mathbf{M}}_s{\mathbf{M}}_s^{\top }$ are zero.

The $m\times s$-matrix ${\mathbf{M}}_s$ is a so-called spurious factor loading matrix that does not contribute to explaining the covariance in ${\mathbf{y}}_t$, since

$$\begin{array}{cc}& {\mathit{\beta }}_k{\mathit{\beta }}_k^{\top }+{\mathbf{\Sigma }}_k={\mathit{\beta }}_k{\mathbf{T}}_k{\mathbf{T}}_k^{\top }{\mathit{\beta }}_k^{\top }+{\mathbf{\Sigma }}_k=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }+{\mathbf{M}}_s{\mathbf{M}}_s^{\top }+({\mathbf{\Sigma }}_0-{\mathbf{M}}_s{\mathbf{M}}_s^{\top })=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }+{\mathbf{\Sigma }}_0=\mathbf{\Omega }.\end{array}$$

While this theorem is an important result, Tumura and Sato (1980) did not impose conditions that resolve rotational invariance, either on the true loading matrix $\mathbf{\Lambda }$ or on the overfitting loading matrix ${\mathit{\beta }}_k$. However, without such conditions, the factor loading matrix ${\mathit{\beta }}_k$ in an overfitting EFA model does not immediately “reveal the truth”, as the separation of ${\mathit{\beta }}_k$ into the true factor loading matrix $\mathbf{\Lambda }$ and the spurious factor loading matrix ${\mathbf{M}}_s$ is possible only up to an arbitrary rotation ${\mathbf{T}}_k$ of ${\mathit{\beta }}_k$.

However, the “truth” in an overfitting EFA model can be recovered if Tumura and Sato (1980, Theorem 1) is applied within the class of unordered GLT structures introduced in this paper. Under the condition that $\mathbf{\Lambda }$ is a GLT structure which satisfies the extended row-deletion property $\mathrm{RD}(r,1+S)$, we prove in Theorem 4 the following result. If the factor loading matrix ${\mathit{\beta }}_k$ in an overfitting EFA model satisfies the unordered GLT condition, then ${\mathit{\beta }}_k$ has a representation, where the rotation in (26) reduces to a signed permutation ${\mathbf{T}}_k={\mathbf{P}}_{\pm }{\mathbf{P}}_{\rho }$. Furthermore, the spurious factor loading matrix ${\mathbf{M}}_s$ takes the form of a spurious unordered GLT structure, introduced in Definition 8.

Definition 8

(Spurious unordered GLT structures). An $m\times s$ unordered GLT structure ${\mathbf{M}}_s$ with pivots rows $\{n_1,\dots ,n_s\}$ is a spurious unordered GLT structure if all columns exhibit a single nonzero loading in the corresponding pivot row.

Theorem 4.

Let Λ be an $m\times r$ factor loading matrix with an unordered GLT structure with pivot rows $l_1,\dots ,l_r$ and assume that Λ obeys the extended row-deletion property $RD(r,1+S)$ for some $S\in \mathbb{N}$. Assume that the $m\times k$ matrix ${\mathit{\beta }}_k$ in the EFA variance decomposition $\mathbf{\Omega }={\mathit{\beta }}_k{\mathit{\beta }}_k^{\top }+{\mathbf{\Sigma }}_k$ is of rank $\mathrm{rk}\left({\mathit{\beta }}_k\right)=k=r+s$, where $1\le s\le S$. If an unordered GLT condition is imposed on ${\mathit{\beta }}_k$, then (26) reduces to

$$\begin{array}{c}\hfill {\mathit{\beta }}_k{\mathbf{P}}_{\pm }{\mathbf{P}}_{\rho }=\left(\begin{array}{cc}\mathbf{\Lambda }& {\mathbf{M}}_s\end{array}\right),{\mathbf{\Sigma }}_k={\mathbf{\Sigma }}_0-{\mathbf{M}}_s{\left({\mathbf{M}}_s\right)}^{\top },\end{array}$$

where ${\mathbf{M}}_s$ is a spurious unordered GLT structure with pivot rows $n_1,\dots ,n_s$ which are distinct from the r pivot rows in Λ. Hence, r columns of ${\mathit{\beta }}_k$ are a signed permutation of the true loading matrix Λ, while the remaining s columns of ${\mathit{\beta }}_k$ are a spurious unordered GLT structure with pivots rows $n_1,\dots ,n_s$.

See Appendix A for a proof. Theorem 4 is employed in Section 6.3 to recover the number of factors from a Bayesian inference of the EFA model (19) using sparsity priors.

5.3. Identifying Irrelevant Variables

In applied factor analysis, the assumption that each measurement $y_{it}$ is correlated with at least one other measurement is too restrictive, because irrelevant measurements might be present that are uncorrelated with all the other measurements. As argued by Boivin and Ng (2006) and Kaufmann and Schumacher (2017) for various latent factor models, it is useful to identify such variables. Within the framework of sparse Bayesian factor analysis, irrelevant variables are identified in Kaufmann and Schumacher (2017) by exploring the sparsity matrix $\mathit{\delta }$ of a factor loading matrix $\mathbf{\Lambda }$ with respect to zero rows. Since $\mathrm{Cov}(y_{it},y_{lt})=0$ for all $l\ne i$ if the entire ith row of $\mathbf{\Lambda }$ is zero (see also (3)), the presence of $m_0$ irrelevant measurements causes the corresponding $m_0$ rows of $\mathbf{\Lambda }$ and $\mathit{\delta }$ to be zero.

Let us investigate the identification of the zero rows in $\mathbf{\Lambda }$ and the corresponding sparsity matrix $\mathit{\delta }$ for the case that the assumed and the true number of factors in the EFA model (19) are identical, i.e., $H=r$. Provided that variance identification of (20) in the underlying model holds, we obtain that ${\mathit{\beta }}_r=\mathbf{\Lambda }\mathbf{P}$ is a rotation of $\mathbf{\Lambda }$. Therefore, the position of the zero rows both in $\mathbf{\Lambda }$ and ${\mathit{\beta }}_r$ are identical and all irrelevant variables can be identified from ${\mathit{\beta }}_r$ or the corresponding sparsity matrix ${\mathit{\delta }}_r$, regardless of the conditions imposed to resolve rotational invariance.

To ensure variance identification under condition AR, the loading matrix $\mathbf{\Lambda }$ has to satisfy the row-deletion property $\mathrm{RD}(r,1)$. If $\mathbf{\Lambda }$ contains $m_0$ zero row, then a necessary condition for $\mathrm{RD}(r,1)$ is that $2r+1\le m-m_0$. This leads to a tighter bound for r than (13), namely

$$m-m_0\ge 2r+1⟺r\le \frac{m-m_0-1}{2}.$$

(27)

Hence, there is a trade-off between $m_0$ and r: the more irrelevant measurements are included among the m measuements, the smaller the maximum number of factors r can be.

6. Identifying the Number of Factors in Sparse Bayesian Factor Analysis

6.1. Sparse Bayesian Factor Analysis

Sparse Bayesian factor analysis operates in the exploratory factor analysis (EFA) model (19), which allows up to H factors, but the true factor dimension r is unknown (see, among many others, Ročková and George (2017); Frühwirth-Schnatter and Lopes (2018), and Ohn and Kim (2022)). Often, spike-and-slab priors are employed, where the elements ${\beta }_{ij}$ of the loading matrix ${\mathit{\beta }}_H$ in the EFA model a priori are allowed to be exactly zero with positive probability. This is achieved through a suitable prior on the corresponding $m\times H$ sparsity matrix ${\mathit{\delta }}_H$. In each column j of ${\mathit{\delta }}_H$, the binary indicators ${\delta }_{ij}$ are active a priori with a column-specific probability ${\tau }_j$, i.e., $\mathrm{Pr}({\delta }_{ij}=1|{\tau }_j)={\tau }_j$ for $i=1,\dots ,m$, where the probabilities ${\tau }_1,\dots ,{\tau }_H$ arise from an exchangeable shrinkage process (ESP) prior:

$$\begin{array}{c}\hfill {\tau }_j|H\sim \mathcal{B}\left(\gamma \frac{\alpha }{H},\gamma \right),j=1,\dots ,H.\end{array}$$

(28)

If $\gamma =1$, then (28) is a so-called one-parameter-beta (1PB) prior, otherwise (28) is a so-called two-parameter-beta (2PB) prior. The 1PB prior converges to the Indian buffet process prior (Teh et al. 2007) for $H\to \infty$.

This specification leads to a Dirac-spike-and-slab prior for the loadings ${\beta }_{ij}$ in ${\mathit{\beta }}_H$,

$$\begin{array}{cc}\hfill & {\beta }_{ij}|\kappa ,{\sigma }_i^2,{\tau }_j\sim (1-{\tau }_j){\Delta }_0+{\tau }_j\mathcal{N}\left(0,\kappa {\sigma }_i^2\right),\end{array}$$

(29)

where a Gaussian slab distribution is assumed and the scale of the prior depends on the idiosyncratic variance ${\sigma }_i^2$ and a random global shrinkage parameter $\kappa$. The priors ${\sigma }_i^2\sim {\mathcal{G}}^{-1}\left(c^{\sigma },b^{\sigma }\right)$ and $\kappa \sim {\mathcal{G}}^{-1}\left(c^{\kappa },b^{\kappa }\right)$ are assumed for ${\sigma }_1^2,\dots ,{\sigma }_m^2$ and $\kappa$. Other slab distributions are possible (see, e.g., Zhao et al. (2016) and Frühwirth-Schnatter et al. (2023)).

As shown by Frühwirth-Schnatter (2023), the ESP prior (28) has a representation as a cumulative shrinkage process (CUSP) prior (Legramanti et al. 2020). For the 1PB prior, e.g., the decreasing order statistics ${\tau }_{\left(1\right)}>\dots >{\tau }_{\left(H\right)}$ of the slab probabilities, ${\tau }_1,\dots ,{\tau }_H$ can be expressed by the following multiplicative (stick-breaking) representation in terms of independent beta random variables, i.e., for $j=1,\dots ,H$:

$$\begin{array}{c}\hfill {\tau }_{\left(j\right)}=\prod _{\ell =1}^j{\nu }_{\ell },{\nu }_{\ell }\sim \mathcal{B}\left(\alpha \frac{H-\ell +1}{H},1\right),\ell =1,\dots ,H.\end{array}$$

(30)

With the largest slab probability following ${\tau }_{\left(1\right)}\sim \mathcal{B}\left(\alpha ,1\right)$, the subsequent slab probabilities ${\tau }_{\left(j\right)}={\tau }_{(j-1)}{\nu }_j$ are rapidly converging to zero as j increases. Hence, the columns of the loading matrix ${\mathit{\beta }}_H$ (if permuted according to the order statistics ${\tau }_{\left(1\right)}>\dots >{\tau }_{\left(H\right)}$) are increasingly pulled toward 0 and the ESP prior induces column sparsity, with the number of non-zero columns k in ${\mathit{\beta }}_H$ being considerably smaller than H a priori.

The hyperparameters $\alpha$ and $\gamma$ of the ESP prior are instrumental in controlling prior column sparsity and retrieving the number of factors from an EFA model (see Section 6.3) and are learned from the data under the priors $\alpha \sim \mathcal{G}(a^{\alpha },b^{\alpha })$ and $\gamma \sim \mathcal{G}(a^{\gamma },b^{\gamma })$.

6.2. MCMC Estimation

For a given choice of hyperparameters $(c^{\sigma },b^{\sigma },c^{\kappa },b^{\kappa },a^{\alpha },b^{\alpha },a^{\gamma },b^{\gamma })$, Markov chain Monte Carlo (MCMC) methods are applied to sample from the posterior distribution $p({\mathit{\beta }}_H,{\mathbf{\Sigma }}_H,{\mathit{\delta }}_H|\mathbf{y})$, given T multivariate observations $\mathbf{y}=({\mathbf{y}}_1,\dots ,{\mathbf{y}}_T)$ (see, e.g., Kaufmann and Schumacher (2019) among many others). In Frühwirth-Schnatter et al. (2023), such a sampler is developed for GLT factor models. To move between factor models of different factor dimensions, Frühwirth-Schnatter et al. (2023) exploit Theorem 4 proven in the present paper to add and delete spurious columns through a reversible jump MCMC (RJMCMC) sampler. We refer to Frühwirth-Schnatter et al. (2023) for full details of this algorithm.

6.3. Identifying the Number of Factors

Identification of the number of factors is a notoriously difficult problem which is closely related to the intrinsic identifiability problems of overfitting EFA models discussed in Section 5.2. Any EFA model, where H is overfitting the true number of factors, may generate decompositions of $\mathbf{\Omega }$ which contain spurious columns. This ambiguity makes estimating the number of factors in applied factor analysis challenging.

A common procedure is to apply an incremental procedure, by increasing the maximum number of factors H step by step, and to use model selection criteria such as information criteria (Aßmann et al. 2016; Bai and Ng 2002) or Bayes factors (Lee and Song 2002; Lopes and West 2004) to choose the number of factors. Kaufmann and Schumacher (2019) estimate a sparse dynamic factor model with an increasing number H of potential factors and use so-called “extracted factor representation” during post-processing MCMC draws to select the number of factors. In sparse Bayesian analysis, the number of factors is usually estimated in one sweep jointly with all unknown parameters using shrinkage priors, such as the ESP prior (28). For instance, based on the prior multiplicative gamma process (Bhattacharya and Dunson 2011), Carvalho et al. (2008) infer r as the number columns that remain after removing columns in the loading matrix with a few nonzero elements in a fairly heuristic manner.

In the present paper, we suggest a one-sweep approach within the framework of sparse Bayesian inference of the overfitting EFA model (19). We identify the number of factors by post-processing the posterior draws of the matrix ${\mathit{\beta }}_k$ containing the k nonzero columns of the loading matrix ${\mathit{\beta }}_H$, where we rely on the insights gained in Section 5.2 regarding the role of variance identification and spurious columns.6

Analyzing the problem from the viewpoint of variance identification is helpful in understanding some of the fundamental difficulties. If the number of factors is unknown, then we need to find a decomposition of $\mathbf{\Omega }$ as in (21), where ${\mathit{\beta }}_k{\mathit{\beta }}_k^{\top }$ is identified. If variance identification holds, then (20) and (21) are equivalent and unique. Therefore, $k=r$, and we can identify the true loading matrix $\mathbf{\Lambda }={\mathit{\beta }}_r\mathbf{P}$ from ${\mathit{\beta }}_r$ up to a rotation $\mathbf{P}$. On the other hand, for any decomposition (21) which is not variance identified, we can deduce that k is bigger than the true number of factors.

In an overfitting EFA model, many posterior draws with k nonzero columns will have a representation as in Theorem 3 or Theorem 4, and contain a submatrix ${\mathbf{M}}_s$ with s spurious columns. Hence, these draws violate even the most simple conditions for variance identification. As a consequence, the number of non-zero columns k overestimates r since $k=r+s$ or, equivalently, $r=k-s$. These insights show that verifying variance identification is essential for recovering the true number of factors, and this has implications for applied factor analysis. Most importantly, methods of inferring the number of factors from the rank or the number of non-zero columns of the posterior draws ${\mathit{\beta }}_H$ in an overfitting factor model are prone to overestimate the number of factors.

In addition, we operate under the GLT condition and rely on the mathematically justified representation of ${\mathit{\beta }}_k$ in an overfitting EFA model provided by Theorem 4. Under an unordered GLT condition, spurious columns among the non-zero columns ${\mathit{\beta }}_k$ are easily spotted, as the $m\times s$ spurious factor loading matrix ${\mathbf{M}}_s$ has an extremely simple structure with a single non-zero loading in each column.7 Furthermore, the remaining columns ${\mathit{\beta }}_r$ are an unordered GLT representation of the true factor loading matrix $\mathbf{\Lambda }$, if variance identification holds and ${\mathit{\beta }}_r{\mathit{\beta }}_r^{\top }$ is identified.

These considerations imply the following strategy for postprocessing the posterior draws obtained by the RJMCMC procedure of Frühwirth-Schnatter et al. (2023). For each posterior draw of the non-zero columns ${\mathit{\beta }}_k$ of ${\mathit{\beta }}_H$, the active columns ${\mathit{\beta }}_r$ (i.e., all columns with at least two non-zero elements) and the corresponding sparsity matrix ${\mathit{\delta }}_r$ are determined. If ${\mathit{\delta }}_r$ satisfies the counting rule $\mathrm{CR}(r,1)$, then ${\mathit{\beta }}_r$ is a signed permutation of $\mathbf{\Lambda }$ by virtue of Theorem 4. The corresponding error covariance matrix is equal to ${\mathbf{\Sigma }}_r={\mathbf{\Sigma }}_k+{\mathbf{M}}_s{\left({\mathbf{M}}_s\right)}^{\top }$, where ${\mathbf{M}}_s$ contains the spurious columns of ${\mathit{\beta }}_k$. These variance identified draws are kept for further inference and the number of columns of ${\mathit{\beta }}_r$ is considered a posterior draw of the unknown factor dimension r. The posterior distribution $p\left(r\right|\mathbf{y})$ can be estimated from these draws and the posterior mode provides a point estimator of the number of factors r.

The proposed strategy of recovering the number of factors through a sparse Bayesian analysis of an overfitting EFA model is illustrated in Section 7 for simulated as well as real data (see also Frühwirth-Schnatter et al. (2023) for further applications).

7. Illustrative Applications

7.1. An Illustrative Simulation Study

For illustration, we perform a simulation study with $m=30$ and $T=150$, and consider three different scenarios for the factor loading matrix $\mathbf{\Lambda }$. In all three scenarios, $r_{\mathrm{true}}=5$ factors are assumed, but the sparsity matrix $\mathit{\delta }$ is quite different. The first setting is a dedicated factor model, where the first six variables load on factor 1, the next six variables load on factor 2, and so forth, and the final six variables load on factor 5. The second scenario is a block factor model, where the first 15 observations load only on factors 1 and 2, while the remaining 15 observations only load on factors 3, 4, and 5, and the covariance matrix has a block-diagonal structure. All factor loadings within a block are non-zero. The third scenario is a dense factor loading matrix without any zero loadings. For all three scenarios, non-zero factor loadings are drawn as ${\lambda }_{ij}={(-1)}^{b_{ij}}(1+0.1\mathcal{N}\left(0,1\right))$, where the exponent $b_{ij}$ is a binary variable with $\mathrm{Pr}(b_{ij}=1)=0.2$. With the exception of the first scenario, no GLT condition is imposed on the simulated loading matrix. In all three scenarios, ${\mathbf{\Sigma }}_0=\mathbf{I}$. A total of 21 data sets are sampled under these three scenarios from the basic factor model (1).

The overfitting exploratory factor model (19) with the maximum number of factors $H=14$ being equal to the upper bound defined in (13) is fitted to each simulated data set. Regarding rotational invariance, we compare a model where the non-zero columns of the factor loading matrix ${\mathit{\beta }}_k$ extracted from ${\mathit{\beta }}_H$ obey an unordered GLT structure with a model where ${\mathit{\beta }}_H$ is left unconstrained.

Inference is based on the Bayesian approach described in Section 6.1. We consider both a 1PB and a 2PB shrinkage prior on the sparsity matrix ${\mathit{\delta }}_H$, and select the following priors: ${\sigma }_i^2\sim {\mathcal{G}}^{-1}\left(2.5,1.5\right),i=1,\dots ,30$, $\kappa \sim {\mathcal{G}}^{-1}\left(5,25\right)$, $\alpha \sim \mathcal{G}\left(6,2\right)$ and $\gamma \sim \mathcal{G}\left(6,6\right)$. Using the RJMCMC algorithm of Frühwirth-Schnatter et al. (2023), 3000 posterior draws are generated after a burn-in of 2000 draws.8 As discussed in Section 6.3, the active columns ${\mathit{\beta }}_r$ are retrieved for each MCMC draw of the loading matrix ${\mathit{\beta }}_H$. Under the GLT condition, ${\mathit{\beta }}_r$ is checked for variance identification using the counting rule $\mathrm{CR}(r,1)$. For sparse Bayesian factor analysis with unstructured loading matrices, the draws of ${\mathit{\beta }}_r$ are not screened for variance identification and inference is based on all draws.

For each of the 21 simulated data sets, we evaluate all 12 combinations of data scenarios, structural constraints (GLT versus unconstrained) and priors on the sparsity matrix (1PB versus 2PB) through Monte Carlo estimates of the following statistics: to assess the performance in estimating the true number $r_{\mathrm{true}}$ of factors, we consider the mode $\widehat{r}$ of the posterior distribution $p\left(r\right|\mathbf{y})$ as a point estimator of r. In addition, we consider the magnitude of the posterior ordinate $p(\widehat{r}=r_{\mathrm{true}}|\mathbf{y})$ as a measure of how strongly the posterior distribution concentrates around the true value. The closer this value is to 1, the smaller is the uncertainty in estimating r. To assess the accuracy in estimating the true covariance matrix ${\mathbf{\Omega }}_0=\mathbf{\Lambda }{\mathbf{\Lambda }}^{\top }+{\mathbf{\Sigma }}_0$ of the data via the posterior draws of ${\mathbf{\Omega }}_r={\mathit{\beta }}_r{\mathit{\beta }}_r^{\top }+{\mathbf{\Sigma }}_r$, we consider the mean squared error (MSE) defined by

$$\begin{array}{ccc}& & {\mathrm{MSE}}_{\mathsf{\Omega }}=\sum _i\sum _{\ell \le i}\mathrm{E}\left({\left({\mathbf{\Omega }}_{r,i\ell }-{\mathbf{\Omega }}_{0,i\ell }\right)}^2\right|\mathbf{y})/(m(m+1)/2),\hfill \end{array}$$

which accounts both for variance and bias.

Table 1. Sparse Bayesian factor analysis under GLT and unconstrained structures (EFA) under a 1PB prior ($\alpha \sim \mathcal{G}\left(6,2\right)$) and a 2PB prior ($\alpha \sim \mathcal{G}\left(6,2\right),\gamma \sim \mathcal{G}\left(6,6\right)$). The true number of factors is equal to 5. GLT and EFA-V use only the variance identified draws ($M_V$ is the percentage of variance identified draws), EFA uses all posterior draws.

			${\mathit{M}}_{\mathit{V}}$	$\widehat{\mathit{r}}$	$\mathit{p}(\widehat{\mathit{r}}={\mathit{r}}_{\mathbf{true}}|\mathbf{y})$	${\mathbf{MSE}}_{\mathbf{\Omega }}$
Scenario		Prior	Med (QR)	Med (QR)	Med (QR)	Med (QR)
Dedic	GLT	1PB	97.0 (91.5, 98.3)	5 (5, 5)	0.90 (0.94, 0.99)	0.018 (0.014, 0.030)
		2PB	97.6 (87.7, 98.9)	5 (5, 5)	0.99 (0.83, 1.00)	0.019 (0.016, 0.027)
	EFA	1PB	-	5 (5, 6)	0.66 (0.09, 0.79)	0.020 (0.015, 0.026)
		2PB	-	5 (5, 6)	0.69 (0.36, 0.80)	0.019 (0.014, 0.024)
	EFA-V	1PB	80.3 (49.8, 87.0)	5 (5, 6)	0.81 (0.17, 0.91)	0.020 (0.015, 0.026)
		2PB	82.6 (63.4, 87.9)	5 (5, 6)	0.84 (0.53, 0.92)	0.019 (0.014, 0.024)
Block	GLT	1PB	96.5 (39.4, 98.9)	5 (5, 5)	0.99 (0.28, 0.99)	0.12 (0.08, 0.18)
		2PB	98.7 (61.9, 99.4)	5 (5, 5)	0.99 (0.54, 1.00)	0.10 (0.08, 0.14)
	EFA	1PB	-	5 (4, 5)	0.78 (0.22, 0.88)	0.14 (0.11, 0.20)
		2PB	-	5 (4, 5)	0.79 (0.08, 0.89)	0.12 (0.08, 0.24)
	EFA-V	1PB	87.0 (55.0, 91.5)	5 (4, 5)	0.89 (0.09, 0.96)	0.14 (0.11, 0.20)
		2PB	85.9 (28.3, 90.4)	5 (4, 5)	0.92 (0.03, 0.97)	0.12 (0.08, 0.24)
Dense	GLT	1PB	95.7 (84.6, 98.6)	5 (5, 5)	0.98 (0.92, 0.99)	0.67 (0.44, 1.12)
		2PB	99.4 (90.8, 99.8)	5 (5, 5)	0.99 (0.93, 1.00)	0.68 (0.51, 1.18)
	EFA	1PB	-	5 (5, 6)	0.76 (0.43, 0.85)	0.54 (0.39, 0.76)
		2PB	-	5 (5, 5)	0.80 (0.66, 0.91)	0.59 (0.43, 0.90)
	EFA-V	1PB	84.4 (76.0, 90.2)	5 (5, 6)	0.89 (0.57, 0.95)	0.54 (0.39, 0.76)
		2PB	89.7 (80.4, 93.9)	5 (5, 5)	0.93 (0.77, 0.98)	0.59 (0.43, 0.90)

Med is the median and QR are the 5% and the 95% quantile of the various statistics over the 21 simulated data sets.

reports, for all 12 combinations, the median, the 5% and the 95% quantile of these statistics across all simulated data sets. For GLT structures, the fraction of variance identified draws is also reported and is, in general, pretty high.

Several conclusions can be drawn from Table 1. First, of all, sparse Bayesian factor analysis under the GLT constraint successfully recovers the true number of factors in all three scenarios. For most of the simulated data sets, the posterior ordinate $p(\widehat{r}=r_{\mathrm{true}}|\mathbf{y})$ is larger than 0.9. Sparse Bayesian factor analysis with unstructured loading matrices is also quite successful in recovering $r_{\mathrm{true}}$, but with less confidence. Both over- and underfitting is observed, and $p(\widehat{r}=r_{\mathrm{true}}|\mathbf{y})$ is much smaller than under a GLT structure. For both structures, the 2PB prior yields higher posterior ordinates than the 1PB prior.

Recently, Hosszejni and Frühwirth-Schnatter (2022) proved that the counting rule $\mathrm{CR}(r,1)$ can also be applied to verify variance identification of ${\mathit{\beta }}_r$ for unconstrained loading matrices. As is evident from Table 1, the fraction of variance identified draws is, however, much smaller than under GLT structures. Nevertheless, inference with respect to to the number of factors can be improved also for an unconstrained EFA model by rejecting all draws of ${\mathit{\beta }}_r$ that do not obey the counting rule $\mathrm{CR}(r,1)$.

7.2. A Real Data Example

For further illustration, we consider monthly log returns from the New York Stock Exchange (NYSE) for two industry sectors. For each month $t=1,\dots ,T$ from February 1999 till August 2019 (i.e., $T=247$), we consider $m=28$ firms. The first 10 measurements in ${\mathbf{y}}_t$ correspond to all firms in the energy sector, the remaining 18 measurements to all firms in the health care sector.9 We compare sparse Bayesian factor analysis with maximum likelihood estimation and to this aim demean and standardize the data.

Bayesian inference is based on the approach described in Section 6.1. The overfitting exploratory factor model (19) is fitted to this data set under an unordered GLT structure with the maximum number of factors $H=13$ being equal to the upper bound defined in (13). MCMC estimation is run for 30,000 iterations after a burn-in of 20,000 using the RJMCMC algorithm of Frühwirth-Schnatter et al. (2023). Prior choices are exactly as in Section 7.1 and we report results for the 2PB prior.

As discussed in Section 6.3, the active columns ${\mathit{\beta }}_r$ are retrieved for each MCMC draw of the loading matrix ${\mathit{\beta }}_H$ and checked for variance identification using the counting rule $\mathrm{CR}(r,1)$. The fraction of variance identified MCMC draws for the 2PB prior is equal to 63%. The number of columns for each variance identified draw ${\mathit{\beta }}_r$ changes due to the dimension-changing nature of the RJMCMC algorithm and can be considered a posterior draw of the number of active factors r by virtue of Theorem 4. As shown in

Table 2. Posterior distribution $p\left(r\right|\mathbf{y})$ of the number of factors based on all variance identified posterior draws.

r	0–3	4	5	6	7	8	9–13
$p\left(r\right|\mathbf{y})$	0	0.28	0.45	0.25	0.02	0.001	0

, the posterior distribution $p\left(r\right|\mathbf{y})$ derived in this manner is rather disperse and models from four up to six factors receive considerable posterior probability.

We proceed with the posterior mode estimator of the number of factors and compare various estimators of the factor loading matrix $\mathbf{\Lambda }$ for a basic factor model (1) where $r=5$ is assumed. First, we estimate ${\widehat{\mathbf{\Lambda }}}_{\mathrm{ML}}$ using maximum likelihood methods and apply the Varimax procedure to rotate ${\widehat{\mathbf{\Lambda }}}_{\mathrm{ML}}$ into a simple structure ${\widehat{\mathbf{\Lambda }}}_{\mathrm{VM}}$.10

Second, we derive two Bayesian estimators where we impose, respectively, the PLT and the GLT condition. Under the GLT condition we learn the pivot rows $\mathbf{l}=(l_1,\dots ,l_5)$ from the data, while $\mathbf{l}=(1,\dots ,5)$ under the PLT condition. To identify simple PLT and GLT structures, we perform variable selection beyond the pivot rows and assume the uniform prior ${\tau }_j\sim \mathcal{B}\left(1,1\right)$ (instead of 1PB or 2PB shrinkage priors), since the number of factors is known. The prior on the factor loadings and the idiosyncratic variances is the same as before. 10,000 posterior draws are generated by adjusting the algorithm of Frühwirth-Schnatter et al. (2023) to these models. The draws are screened for variance identification using the counting rule $\mathrm{CR}(5,1)$. The average of the variance identified draws immediately yields the Bayesian estimator ${\widehat{\mathbf{\Lambda }}}_{\mathrm{PLT}}$ under the PLT condition.

Under the GLT condition, the sampler operates within an unordered GLT structure with unknown pivot rows. All variance identified posterior draws are identified up to a signed permutation. Full identification is achieved by resolving column and sign switching for each posterior draw. Column switching is resolved by ordering the pivot rows such that $l_1<\dots <l_5$ and an ordered GLT structure is imposed on $\mathbf{\Lambda }$ by reordering the columns accordingly. Among these ordered GLT posterior draws of $\mathbf{\Lambda }$, the combination of pivot rows $\widehat{\mathbf{l}}=(1,4,11,12,15)$ is visited most often. All ordered GLT draws where the pivot rows coincide with $\widehat{\mathbf{l}}$ are averaged to derive the Bayesian estimator ${\widehat{\mathbf{\Lambda }}}_{\mathrm{GLT}}$ of the GLT factor loading matrix $\mathbf{\Lambda }$. Beforehand, sign switching in the posterior draws is resolved by imposing the constraint ${\mathsf{\Lambda }}_{11}>0$, ${\mathsf{\Lambda }}_{42}>0$, ${\mathsf{\Lambda }}_{11,3}>0$, ${\mathsf{\Lambda }}_{12,4}>0$ and ${\mathsf{\Lambda }}_{15,5}>0$ on $\mathbf{\Lambda }$.

All four estimators are depicted in Figure 2. Note that the measurements are grouped by industry. The pivot rows learned under GLT from the data clearly show that this ordering is in conflict with the PLT assumption. The varimax estimator ${\widehat{\mathbf{\Lambda }}}_{\mathrm{VM}}$, as well as both Bayesian estimators ${\widehat{\mathbf{\Lambda }}}_{\mathrm{PLT}}$ and ${\widehat{\mathbf{\Lambda }}}_{\mathrm{GLT}}$, exhibit a simple structure, and sparsity is more pronounced for the two Bayesian estimators due to variable selection on the factor loadings.

Figure 2. Comparing estimators of the loading matrix $\mathbf{\Lambda }$ for the NYSE data (1–10, firms in the energy sector; 11–28, firms in the health care sector) for a basic factor model with $r=5$ factors. Heatmaps from left to right: Bayesian estimator ${\widehat{\mathbf{\Lambda }}}_{\mathrm{GLT}}$ of the GLT representation of the loading matrix; Bayesian estimator ${\widehat{\mathbf{\Lambda }}}_{\mathrm{PLT}}$ obtained by imposing a PLT condition; ML estimator ${\widehat{\mathbf{\Lambda }}}_{\mathrm{ML}}$ and the corresponding Varimax rotation ${\widehat{\mathbf{\Lambda }}}_{\mathrm{VM}}$. Red, white, and blue colors denote positive, zero, and negative values, respectively.

Figure 2. Comparing estimators of the loading matrix $\mathbf{\Lambda }$ for the NYSE data (1–10, firms in the energy sector; 11–28, firms in the health care sector) for a basic factor model with $r=5$ factors. Heatmaps from left to right: Bayesian estimator ${\widehat{\mathbf{\Lambda }}}_{\mathrm{GLT}}$ of the GLT representation of the loading matrix; Bayesian estimator ${\widehat{\mathbf{\Lambda }}}_{\mathrm{PLT}}$ obtained by imposing a PLT condition; ML estimator ${\widehat{\mathbf{\Lambda }}}_{\mathrm{ML}}$ and the corresponding Varimax rotation ${\widehat{\mathbf{\Lambda }}}_{\mathrm{VM}}$. Red, white, and blue colors denote positive, zero, and negative values, respectively.

In our opinion, the Bayesian GLT estimator ${\widehat{\mathbf{\Lambda }}}_{\mathrm{GLT}}$ allows a clearer interpretation of the four factors than the other estimators. In particular, forcing a PLT assumption leads to a less clear interpretation of the factors. The first factor is a market factor that loads on all 28 firms. The second factor captures additional correlations among the 10 firms in the energy sector, as well as cross-sectional correlations with specific firms in the health care sector. Three additional factors are present that are specific to firms in the health care sector, with factor three loading on nearly all firms in this sector.

8. Concluding Remarks

We have given a full and comprehensive mathematical treatment to generalized lower triangular (GLT) structures, an identification strategy that relaxes the popular positive lower triangular (PLT) assumption for factor loadings matrices. We have proven that GLT retains PLT’s good properties: uniqueness and rotational invariance. At the same time, and unlike PLT, a GLT structure exists for any factor loadings matrix; i.e., it is not a restrictive assumption. Furthermore, we have shown that verifying variance identification under GLT structures is simple and is based purely on the zero-nonzero pattern of the factor loadings matrix. Additionally, we have embedded the GLT model class into exploratory factor analysis with unknown factor dimension and discussed how easily spurious factors and irrelevant variables are recognized in that setup. At the end, we demonstrated our framework in a simulation study and for real data.