Exploratory Factor Analysis by Gauss-Newton

Abstract

After more than a century, factor analysis remains one of the most popular tools in applied statistics. Exploratory factor analysis tends to be driven by iterated principal axis factorization. Confirmatory factor analysis tends to rely on maximum likelihood estimation. This paper demonstrates the computational superiority of the Gauss–Newton method in minimizing the principal axis loss. On sample problems, Gauss–Newton is more than three orders of magnitude faster than popular implementations of maximum likelihood factor analysis. As part of this comparison, we derive an alternative Gauss–Newton method that leverages the MM principle of optimization. We also explore a simple perturbation correction that dramatically improves the accuracy of the factor loading matrices derived from approximate spectral decompositions. Finally, we suggest a robust version of iterated principal axis factorization that leverages the MM principle, block descent, and the Gauss–Newton method. These innovations are implemented in Julia code and applied to a sequence of representative problems.

1. Introduction

In factor analysis driven by iterated principal axis factorization [1], one minimizes the loss

$$\begin{array}{ccc}\hfill f(\mathit{F},\mathit{D})& =& {\displaystyle \frac{1}{2}}{\parallel \mathit{S}-\mathit{F}{\mathit{F}}^{\top }-\mathit{D}\parallel }_F^2\hfill \end{array}$$

by block descent. Here, $\mathit{S}$ is an $p\times p$ sample covariance matrix, $\mathit{F}$ is an $p\times r$ factor loadings matrix, $\mathit{D}$ is an $p\times p$ nonnegative diagonal matrix, and ${\parallel ·\parallel }_F$ is the Frobenius norm on $p\times p$ matrices. The rank r of $\mathit{F}$ is chosen to be much smaller than the rank of $\mathit{S}$, which cannot exceed p. In the background lurks a random sample ${\mathit{x}}_1,\dots ,{\mathit{x}}_n$ of centered vectors in ${\mathbb{R}}^p$ with mean 0 along each row. If we stack the row vectors ${\mathit{x}}_i^{\top }$ into a matrix $\mathit{X}$, then $\mathit{S}={\displaystyle \frac{1}{n}}{\mathit{X}}^{\top }\mathit{X}={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{\mathit{x}}_i{\mathit{x}}_i^{\top }$. The blocks in iterated principal axis factorization are $\mathit{F}$ and $\mathit{D}$. These are updated at iteration m by

$$\begin{array}{ccc}\hfill {\mathit{D}}_{m+1}& =& \underset{\mathit{D}}{\mathrm{argmin}}f({\mathit{F}}_m,\mathit{D})\hfill \\ \hfill {\mathit{F}}_{m+1}& =& \underset{\mathit{F}}{\mathrm{argmin}}f(\mathit{F},{\mathit{D}}_{m+1}).\hfill \end{array}$$

These two updates are alternated until convergence. In contrast, maximum likelihood estimation optimizes the loglikelihood [2,3]

$$\begin{array}{ccc}\hfill \mathcal{L}(\mathit{F},\mathit{D})& =& -{\displaystyle \frac{n}{2}}\left[\ln det\mathbf{\Omega }+\mathrm{tr}\left({\mathbf{\Omega }}^{-1}\mathit{S}\right)\right]\hfill \\ \hfill \mathbf{\Omega }& =& \mathit{F}{\mathit{F}}^{\top }+\mathit{D}.\hfill \end{array}$$

The loglikelihood can be maximized by either Newton’s method [4], a Gauss–Newton algorithm [5], or an EM algorithm [6,7], all of which perform reasonably well in low-dimensional problems. In practice, algorithms that merely reduce $f(\mathit{F},{\mathit{D}}_{m+1})$ rather than minimize it can be effective. Furthermore, any transformation $\mathit{F}\mathit{O}$ by an orthogonal matrix $\mathit{O}$ leads to the same product $\mathit{F}\mathit{O}{\left(\mathit{F}\mathit{O}\right)}^{\top }=\mathit{F}{\mathit{F}}^{\top }$. Post-processing of the estimated $\mathit{F}$ easily renders its columns orthogonal.

As summarized in the reviews [2,8,9,10], both optimization criteria have something to recommend them. Iterated principal axis factorization makes no distributional assumptions, does not require $\mathit{S}$ to be of full rank, is more robust to unequal factor loadings and small sample sizes, and is better able to recover weak factors. In contrast, maximum likelihood permits computation of multiple fit indices and of significance tests of the factor loadings. Of course, a Gaussian likelihood can be invoked nonparametrically, but this leaves significance testing questionable. In practice, the sample covariance matrix $\mathit{S}$ is often replaced by the sample correlation matrix. Because this substitution loses some information, we prefer the sample covariance formulation of factor analysis. To its credit, the Gauss–Newton strategy applies in both settings.

In the current article we argue the case for iterated principal axis factorization purely on the grounds of computation efficiency in high-dimensional problems. Given the complexity of the loglikelihood function, this conclusion is plausible. However, details matter in implementing iterated principal axis factorization. For instance, in embracing block descent, the obvious choice for updating the diagonal entry $d_{ii}$ of $\mathit{D}$ is

$$\begin{array}{ccc}\hfill d_{m+1,ii}& =& \max \{{(\mathit{S}-{\mathit{F}}_m{\mathit{F}}_m^{\top })}_{ii},0\}.\hfill \end{array}$$

(1)

Implausible Heywood estimates where some $d_i<0$ are not allowed. The updates (1) can be carried out in parallel, respect the non-negativity constraints on the $d_{ii}$, and do not require computation of the full matrix product ${\mathit{F}}_m{\mathit{F}}_m^{\top }$. Finally, compared to simultaneously updating $\mathit{D}$ and $\mathit{F}$, block descent reduces the number of parameters active in any update.

The traditional block update of $\mathit{F}=\mathit{V}{\mathbf{\Lambda }}^{1/2}$ depends on extracting the eigenvector matrix $\mathit{V}$ and the diagonal eigenvalue matrix $\mathbf{\Lambda }$ corresponding to the r largest eigenvalues of $\mathit{S}-\mathit{D}$. Ordinarily, the r largest eigenvalues are positive. If this is not the case, then $\mathit{V}$ and $\mathbf{\Lambda }$ can be pruned until it is the case. This construction is computationally intensive when a complete eigen-decomposition of $\mathit{S}-\mathit{D}$ is invoked. On the other hand, if only the top r eigen-pairs are generated by an iterative method such as the Lanczos algorithm [11], then the inaccuracy of $\mathbf{\Lambda }$ and the imperfect orthogonality of $\mathit{V}$ may seriously corrupt updates.

The current paper proposes three innovations that potentially improve either the speed or the robustness of iterated principal axis factorization. First, we derive an alternative Gauss–Newton method that leverages the MM principle of optimization [12,13]. Second, we recommend a simple perturbation correction for dramatically improving the accuracy of $\mathbf{\Lambda }$ and the orthogonality of $\mathit{V}$ delivered by the Lanczos algorithm. Third, we suggest a robust version of iterated principal axis factorization that also leverages the MM principle and block descent. These innovations are implemented in Julia code that is three to four orders of magnitude faster than popular implementations of maximum likelihood factor analysis. These advances put high-dimensional factor analysis within reach without sacrificing inference quality.

2. Background

2.1. Notation

Here are the notational conventions used throughout this article. All vectors and matrices appear in boldface. All entries of the vector $\mathbf{0}$ equal 0; $\mathit{I}$ indicates an identity matrix. The ^⊤ superscript indicates a vector transpose. The Euclidean norm and ${\ell }_1$ norm of a vector $\mathit{x}$ are denoted by $\parallel \mathit{x}\parallel$ and ${\parallel \mathit{x}\parallel }_1$. For a matrix $\mathit{M}$ the norm ${\parallel \mathit{M}\parallel }_1$ indicates ${\parallel \mathrm{vec}\left(\mathit{M}\right)\parallel }_1$. The Frobenius norm of a matrix $\mathit{M}$ is denoted by ${\parallel \mathit{M}\parallel }_F$. For a smooth real-valued function $f\left(\mathit{x}\right)$, we write its first differential (row vector of partial derivatives) as $df\left(\mathit{x}\right)$. Finally, we denote the directional derivative of $f\left(\mathit{x}\right)$ in the direction $\mathit{v}$ by $d_{\mathit{v}}f\left(\mathit{x}\right)$. When the function $f\left(\mathit{x}\right)$ is differentiable, $d_{\mathit{v}}f\left(\mathit{x}\right)=df\left(\mathit{x}\right)\mathit{v}$.

2.2. MM Principle

Two of our algorithms invoke the majorization–minimization (MM) principle [12,13,14]. MM algorithms depend on a surrogate function that majorizes the objective $f\left(\mathit{x}\right)$ around the current iterate ${\mathit{x}}_m$ in the sense of satisfying the tangency condition $g({\mathit{x}}_m\mid {\mathit{x}}_m)=f\left({\mathit{x}}_m\right)$ and the domination condition $f\left(\mathit{x}\right)\le g(\mathit{x}\mid {\mathit{x}}_m)$ for all $\mathit{x}$. The surrogate balances two goals, hugging the objective tightly and simplifying minimization. Minimizing the surrogate produces the next iterate ${\mathit{x}}_{m+1}$ and drives the objective downhill. Indeed, the descent property

$$\begin{array}{ccc}\hfill f\left({\mathit{x}}_{m+1}\right)& \le & g({\mathit{x}}_{m+1}\mid {\mathit{x}}_m)\le g({\mathit{x}}_m\mid {\mathit{x}}_m)=f\left({\mathit{x}}_m\right).\hfill \end{array}$$

follows directly from the definitions. This property depends only on $g(\mathit{x}\mid {\mathit{x}}_m)$ being reduced rather than being minimized. In maximization, the surrogate instead minorizes the objective and is maximized. The tangency condition remains the same, but now the domination condition $f\left(\mathit{x}\right)\le g(\mathit{x}\mid {\mathit{x}}_m)$ is reversed. The acronym MM also applies to minorization-maximization.

2.3. Perturbation Correction of Eigen-Decompositions

Following the reference [15], suppose $\mathit{S}$ is a symmetric matrix. Let $\mathit{V}$ be a collection of r approximate unit length eigenvectors. We want the correction $\mathit{V}({\mathit{I}}_r+\mathit{E})$ to satisfy the orthogonality relations

$$\begin{array}{ccc}\hfill ({\mathit{I}}_r+{\mathit{E}}^{\top }){\mathit{V}}^{\top }\mathit{V}({\mathit{I}}_r+\mathit{E})& =& {\mathit{I}}_r\hfill \\ \hfill ({\mathit{I}}_r+{\mathit{E}}^{\top }){\mathit{V}}^{\top }\mathit{S}\mathit{V}({\mathit{I}}_r+\mathit{E})& =& \mathbf{\Lambda },\hfill \end{array}$$

where $\mathbf{\Lambda }$ is a $r\times r$ diagonal matrix of eigenvalues of $\mathit{S}$. Assuming $\mathit{E}$ is small, the expansion ${({\mathit{I}}_r+\mathit{E})}^{-1}={\sum }_{k=0}^{\infty }{(-1)}^k{\mathit{E}}^k$ implies

$$\begin{array}{ccc}\hfill {\mathit{V}}^{\top }\mathit{V}& =& {({\mathit{I}}_r+{\mathit{E}}^{\top })}^{-1}{({\mathit{I}}_r+\mathit{E})}^{-1}\hfill \\ & =& {\mathit{I}}_r-\mathit{E}-{\mathit{E}}^{\top }+{O(\parallel \mathit{E}\parallel }^2)\hfill \end{array}$$

(2)

and

$$\begin{array}{ccc}\hfill {\mathit{V}}^{\top }\mathit{S}\mathit{V}& =& {({\mathit{I}}_r+{\mathit{E}}^{\top })}^{-1}\mathbf{\Lambda }{({\mathit{I}}_r+\mathit{E})}^{-1}\hfill \\ & =& \mathbf{\Lambda }-{\mathit{E}}^{\top }\mathbf{\Lambda }-\mathbf{\Lambda }\mathit{E}+{O(\parallel \mathit{E}\parallel }^2).\hfill \end{array}$$

(3)

The diagonal of the constraint (2) implies $e_{ii}=0$ for all i. The diagonal of the constraint (3) provides the eigenvalues ${\lambda }_i={\mathit{v}}_i^{\top }\mathit{S}{\mathit{v}}_i$. The off-diagonals of the constraint (2) imply $e_{ij}+e_{ji}=-{\mathit{v}}_i^{\top }{\mathit{v}}_j$. The off-diagonals of the constraint (3) imply ${\lambda }_i{e}_{ij}+{\lambda }_j{e}_{ji}=-{\mathit{v}}_i^{\top }\mathit{S}{\mathit{v}}_j$. In summary, it follows that

$$\begin{array}{ccc}\hfill {\lambda }_i& =& {\mathit{v}}_i^{\top }\mathit{S}{\mathit{v}}_i\hfill \\ \hfill e_{ii}& =& 0\hfill \\ \hfill e_{ij}& =& {\displaystyle \frac{{\mathit{v}}_i^{\top }\mathit{S}{\mathit{v}}_j-{\lambda }_j{\mathit{v}}_i^{\top }{\mathit{v}}_j}{{\lambda }_j-{\lambda }_i}},i\ne j,\hfill \end{array}$$

assuming the eigenvalues are distinct. If ${\lambda }_i={\lambda }_j$, then the choice

$$\begin{array}{ccc}\hfill e_{ij}& =& e_{ji}=-{\displaystyle \frac{{\mathit{v}}_i^{\top }{\mathit{v}}_j}{2}}\hfill \end{array}$$

satisfies the equation $e_{ij}+e_{ji}=-{\mathit{v}}_i^{\top }{\mathit{v}}_j$. This correction scheme enjoys a local quadratic rate of convergence when $\mathit{S}$ has simple eigenvalues [15]. We refer readers to the original paper and follow-up papers [16,17,18] for precise error bounds.

2.4. Moreau Envelope and Proximal Map

The Moreau envelope

$$\begin{array}{ccc}\hfill M_{\mu f}\left(\mathit{y}\right)& =& \underset{\mathit{x}}{inf}\left[f\left(\mathit{x}\right)+{\displaystyle \frac{1}{2\mu }}{\parallel \mathit{y}-\mathit{x}\parallel }^2\right]\hfill \end{array}$$

of a function $f\left(\mathit{x}\right)$ from ${\mathbb{R}}^p$ to $\mathbb{R}\cup \{\infty \}$ is valuable in converting nonsmooth optimization problems into smooth optimization problems. If $f\left(\mathit{x}\right)$ is convex, then the infimum is attained, and its Moreau envelope $M_{\mu f}\left(\mathit{y}\right)$ is convex and continuously differentiable [19]. The proximal operator

$$\begin{array}{ccc}\hfill \underset{\mu f}{\mathrm{prox}}\left(\mathit{y}\right)& =& \underset{\mathit{x}}{\mathrm{argmin}}\left[f\left(\mathit{x}\right)+{\displaystyle \frac{1}{2\mu }}{\parallel \mathit{y}-\mathit{x}\parallel }^2\right]\hfill \end{array}$$

(4)

encodes the solution to the Moreau envelope minimization. Fortunately, we have the obvious quadratic majorization

$$\begin{array}{ccc}\hfill M_{\mu f}\left(\mathit{y}\right)& \le & f\left({\mathit{z}}_m\right)+{\displaystyle \frac{1}{2\mu }}{\parallel \mathit{y}-{\mathit{z}}_m\parallel }^2\hfill \end{array}$$

(5)

at ${\mathit{y}}_m$ for ${\mathit{z}}_m={\mathrm{prox}}_{\mu f}\left({\mathit{y}}_m\right)$. The inequality $M_{\mu f}\left(\mathit{y}\right)\le f\left(\mathit{y}\right)$ is obvious, and the limit ${lim}_{\mu \downarrow 0}{M}_{\mu f}\left(\mathit{y}\right)=f\left(\mathit{y}\right)$ holds under mild conditions. In particular, if $f\left(\mathit{y}\right)$ is Lipschitz with constant L, then

$$\begin{array}{ccc}\hfill 0& \le & f\left(\mathit{y}\right)-M_{\mu f}\left(\mathit{y}\right) \le {\displaystyle \frac{L^2\mu }{2}}.\hfill \end{array}$$

The example most relevant to our exposition is the Moreau envelope of the nonsmooth absolute value function $\left|x\right|$. The Moreau envelope of $\left|x\right|$ is Huber’s function

$$\begin{array}{ccc}\hfill M_{\mu |·|}\left(y\right)& =& \left\{\begin{array}{cc}{\displaystyle \frac{1}{2\mu }}{y}^2,\hfill & \left|y\right|\le \mu \hfill \\ \left|y\right|-{\displaystyle \frac{\mu }{2}},\hfill & \left|y\right|>\mu \hfill \end{array}\right.\hfill \end{array}$$

with proximal map

$$\begin{array}{ccc}\hfill \underset{\mu |·|}{\mathrm{prox}}\left(y\right)& =& \left(1-{\displaystyle \frac{\mu }{\max \left\{\right|y|,\mu \}}}\right)y,\hfill \end{array}$$

(6)

which soft thresholds the scalar y. By design $M_{\mu |·|}\left(y\right)$ grows linearly as $\left|y\right|$ tends to ∞.

3. Algorithm Derivations

3.1. The Classical Gauss–Newton Algorithm

The Gauss–Newton was derived by Jennrich [5] and later rederived by Liu et al. [20]. For the reader’s convenience, we now present a condensed version of Liu et al.’s derivation. To minimize $f(\mathit{d},\mathit{F})$ with respect to $\mathit{F}$ in iterated principal axis factorization, it is helpful to linearize $\mathit{F}{\mathit{F}}^{\top }$ around the current iterate ${\mathit{F}}_m$ and minimize

$$\begin{array}{ccc}\hfill g(\mathbf{\Delta })& =& {\displaystyle \frac{1}{2}}{\parallel \mathit{S}-{\mathit{F}}_m{\mathit{F}}_m^{\top }-{\mathit{F}}_m{\mathbf{\Delta }}^{\top }-\mathbf{\Delta }{\mathit{F}}_m^{\top }-{\mathit{D}}_{m+1}\parallel }_F^2\hfill \end{array}$$

with respect to the increment $\mathbf{\Delta }=\mathit{F}-{\mathit{F}}_m$. If ${\mathit{Z}}_m=\mathit{S}-{\mathit{F}}_m{\mathit{F}}_m^{\top }-{\mathit{D}}_{m+1}$, then at a local minimum the stationary condition

$$\begin{array}{ccc}\hfill \mathbf{0}& =& d_{\mathit{V}}g(\mathbf{\Delta })=-\mathrm{tr}\left[{({\mathit{Z}}_m-{\mathit{F}}_m{\mathbf{\Delta }}^{\top }-\mathbf{\Delta }{\mathit{F}}_m^{\top })}^{\top }({\mathit{F}}_m{\mathit{V}}^{\top }+\mathit{V}{\mathit{F}}_m^{\top })\right]\hfill \end{array}$$

for all directions $\mathit{V}$ entails

$$\begin{array}{ccc}\hfill \mathbf{0}& =& 2({\mathit{Z}}_m-{\mathit{F}}_m{\mathbf{\Delta }}^{\top }-\mathbf{\Delta }{\mathit{F}}_m^{\top }){\mathit{F}}_m.\hfill \end{array}$$

If we postmultiply this stationary condition by ${\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}$, then it becomes

$$\begin{array}{ccc}\hfill {\mathit{F}}_m{\Delta }^{\top }{\mathit{F}}_m{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}+\mathbf{\Delta }& =& {\mathit{Z}}_m{\mathit{F}}_m{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}.\hfill \end{array}$$

(7)

Premultiplying the result by ${\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}{\mathit{F}}_m^{\top }$ now gives the symmetric version

$$\begin{array}{c}\hfill {\Delta }^{\top }{\mathit{F}}_m{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}+{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}{\mathit{F}}_m^{\top }\mathbf{\Delta }={\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}{\mathit{F}}_m^{\top }{\mathit{Z}}_m{\mathit{F}}_m{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}.\end{array}$$

(8)

The solution $\mathbf{\Delta }$ of the system (8) can be decomposed into a component from the range of ${\mathit{F}}_m$ plus one from its orthogonal complement. In other words, $\mathbf{\Delta }=\mathit{U}+{\mathit{F}}_m\mathit{C}$ with ${\mathit{U}}^{\top }{\mathit{F}}_m=\mathbf{0}$. Substitution in Equation (8) gives

$$\begin{array}{ccc}\hfill {\mathit{C}}^{\top }+\mathit{C}& =& {\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}{\mathit{F}}_m^{\top }{\mathit{Z}}_m{\mathit{F}}_m{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1},\hfill \end{array}$$

and assuming $\mathit{C}$ is symmetric, it follows that

$$\begin{array}{ccc}\hfill \mathit{C}& =& {\displaystyle \frac{1}{2}}{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}{\mathit{F}}_m^{\top }{\mathit{Z}}_m{\mathit{F}}_m{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}.\hfill \end{array}$$

Finally, substitution of $\mathbf{\Delta }=\mathit{U}+{\mathit{F}}_m\mathit{C}$ in Equation (7) yields

$$\begin{array}{ccc}\hfill {\mathit{F}}_m\mathit{C}+\mathit{U}+{\mathit{F}}_m\mathit{C}& =& {\mathit{Z}}_m{\mathit{F}}_m{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}\hfill \\ \hfill \mathit{U}+{\mathit{F}}_m\mathit{C}& =& {\mathit{Z}}_m{\mathit{F}}_m{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}-{\mathit{F}}_m\mathit{C}\hfill \\ \hfill {\mathit{F}}_{m+1}& =& {\mathit{F}}_m+\left[\mathit{I}-{\displaystyle \frac{1}{2}}{\mathit{F}}_m{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}{\mathit{F}}_m^{\top }\right]{\mathit{Z}}_m{\mathit{F}}_m{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}.\hfill \end{array}$$

The following Julia code implements the Liu et al. [20] Gauss–Newton algorithm. Comments, prefixed by #, are liberally sprinkled throughout the code.

This code incorporates the scale-adjusted convergence criterion

$$\begin{array}{ccc}\hfill \parallel {\mathit{d}}_{m+1}-{\mathit{d}}_m\parallel & <& {10}^{-8}\parallel {\mathit{d}}_m\parallel \hfill \end{array}$$

and is applied throughout our numerical examples. Just prior to output, the code also orthogonalizes $\mathit{F}$ by extracting its singular value decomposition. Given a small number of desired factors r, orthogonalization is cheap. The cost $O\left(p^{2}r\right)$ of forming the matrix product $\mathit{S}\mathit{Y}$ dominates the overall computational cost of the algorithm. Fortunately, matrix multiplication is very fast on modern computers.

using LinearAlgebra # Julia’s linear algebra~package

function FactorAnalysisGN(S::Matrix{T}, r::Int) where T <: Real

p = size(S, 1) # size of sample covariance matrix

(d, old_d, conv) = (zeros(T, p), zeros(T, p), 1.0e-8)

F = randn(T, p, r) # random initialization of loadings

(FtF, Y, Z) = (zeros(T, r, r), zeros(T, r, r), zeros(T, p, r))

for iter = 1:500

for i = 1:p # update specific variances

d[i] = max(S[i, i] - norm(F[i, :])^2, zero(T))

end

mul!(FtF, transpose(F), F) # prepare to update loadings

Y = F / FtF

mul!(Z, S, Y) # Z = S * Y

Z .= Z .- Diagonal(d) * Y

F .= Z .- F * ((transpose(Y) * Z - I) ./ 2) # loadings update

if norm(d - old_d) < conv * norm(old_d) && iter > 1 break end

old_d .= d

end

SVD = svd(F) # orthogonalize the columns of F before output

F = SVD.U * Diagonal(SVD.S)

return (F, d)

end

3.2. An MM Version of Gauss–Newton

It is possible to derive a different Gauss–Newton algorithm based on the MM principle. The identity ${\parallel \mathit{A}\parallel }_F={\parallel {\mathit{A}}^{\top }\parallel }_F$ and Jensen’s inequality provide the majorization

$$\begin{array}{ccc}\hfill g(\mathbf{\Delta })& =& {\displaystyle \frac{1}{2}}{∥{\displaystyle \frac{1}{2}}({\mathit{Z}}_m-2{\mathit{F}}_m{\mathbf{\Delta }}^{\top })+{\displaystyle \frac{1}{2}}({\mathit{Z}}_m-2\mathbf{\Delta }{\mathit{F}}_m^{\top })∥}_F^2\hfill \\ & \le & {\displaystyle \frac{1}{4}}{∥{\mathit{Z}}_m-2{\mathit{F}}_m{\mathbf{\Delta }}^{\top }∥}_F^2+{\displaystyle \frac{1}{4}}{∥{\mathit{Z}}_m-2\mathbf{\Delta }{\mathit{F}}_m^{\top }∥}_F^2\hfill \\ & =& {\displaystyle \frac{1}{2}}{∥{\mathit{Z}}_m-2\mathbf{\Delta }{\mathit{F}}_m^{\top }∥}_F^2\hfill \\ & =& 2{∥{\mathit{U}}_m-\mathbf{\Delta }{\mathit{F}}_m^{\top }∥}_F^2\hfill \\ & =& h(\mathbf{\Delta }\mid \mathbf{0})\hfill \end{array}$$

of $g(\mathbf{\Delta })$, where ${\mathit{U}}_m={\displaystyle \frac{1}{2}}{\mathit{Z}}_m={\displaystyle \frac{1}{2}}(\mathit{S}-{\mathit{F}}_m{\mathit{F}}_m^{\top }-{\mathit{D}}_{m+1})$ is symmetric.  This majorization is carefully designed to reduce the loss to a more symmetric sum of squares. The directional derivative

$$\begin{array}{ccc}\hfill d_{\mathit{V}}h(\mathbf{\Delta }\mid \mathbf{0})& =& -4\mathrm{tr}\left[{({\mathit{U}}_m-\mathbf{\Delta }{\mathit{F}}_m^{\top })}^{\top }\mathit{V}{\mathit{F}}_m^{\top }\right]=-4\mathrm{tr}\left[{\mathit{F}}_m^{\top }({\mathit{U}}_m-{\mathit{F}}_m{\mathbf{\Delta }}^{\top })\mathit{V}\right]\hfill \end{array}$$

vanishes for all $\mathit{V}$ if and only if ${\mathit{F}}_m^{\top }({\mathit{U}}_m-{\mathit{F}}_m{\mathbf{\Delta }}^{\top })=\mathbf{0}$. This normal equation for the surrogate $h(\mathbf{\Delta }\mid \mathbf{0})$ entails the solution

$$\begin{array}{ccc}\hfill {\mathbf{\Delta }}^{\top }& =& {\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}{\mathit{F}}_m^{\top }{\mathit{U}}_m\hfill \\ \hfill \mathbf{\Delta }& =& {\mathit{U}}_m{\mathit{F}}_m{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}\hfill \\ \hfill {\mathit{F}}_{m+1}& =& {\mathit{F}}_m+\mathbf{\Delta }\hfill \\ & =& {\mathit{F}}_m+{\mathit{U}}_m{\mathit{F}}_m{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}\hfill \\ & =& {\mathit{F}}_m+{\displaystyle \frac{1}{2}}[\mathit{S}{\mathit{F}}_m-{\mathit{F}}_m\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)-{\mathit{D}}_{m+1}{\mathit{F}}_m]{\left({\mathit{F}}_m^{\top }{\mathit{F}}_m\right)}^{-1}.\hfill \end{array}$$

(9)

The second form of the ${\mathit{F}}_{m+1}$ update is preferred because it avoids creation of the $p\times p$ product matrix ${\mathit{F}}_m{\mathit{F}}_m^{\top }$. The following Julia code implements the MM algorithm.

using LinearAlgebra # Julia’s linear algebra~package

function FactorAnalysisMM(S::Matrix{T}, r::Int) where T <: Real

(p, iters) = (size(S, 1), 0) # size of sample covariance matrix

(d, old_d, conv) = (zeros(T, p), zeros(T, p), 1.0e-8)

F = randn(p, r) # random initialization of loadings

FtF = zeros(T, r, r) # stores F transpose * F

for iter = 1:1000

iters = iters + 1

for i = 1:p # update specific variances

d[i] = max(S[i, i] - norm(F[i, :])^2, zero(T))

end

mul!(FtF, transpose(F), F) # prepare to update loadings

F .= F .+ (S * F - F * FtF - Diagonal(d) * F) / (2FtF)

if norm(d - old_d) < conv * norm(old_d) && iter > 1 break end

old_d .= d

end

SVD = svd(F) # before output orthogonalize the columns of F

F = SVD.U * Diagonal(SVD.S)

return (F, d, iters)

end

The $O\left(p^{2}r\right)$ arithmetic operations required to form the matrix product $\mathit{S}\mathit{F}$ dominates the computational cost of the MM algorithm. Although both the Gauss–Newton and the MM updates are guaranteed to reduce $g(\mathbf{\Delta })$, they are not guaranteed to reduce $f(\mathit{D},\mathit{F})$. Step-halving mitigates this problem but it is omitted from both Julia functions because the updates invariably reduce $f(\mathit{D},\mathit{F})$ after a few rounds of block descent.

3.3. Eigen Correction

As already mentioned, the iterated principal axis factorization alternates the update (1) and the update $\mathit{F}=\mathit{V}{\mathbf{\Lambda }}^{1/2}$ derived from the eigenvector matrix $\mathit{V}$ and the diagonal eigenvalue matrix $\mathbf{\Lambda }$ corresponding to the r largest eigenvalues of $\mathit{S}-{\mathit{D}}_{m+1}$. To this recipe we add the eigen correction of Ogita and Aishima [15]. Again we list actual Julia code rather than a less informative algorithm sketch. The main function appears third. Julia’s implementation of Arpack provides approximate eigenvectors.

using LinearAlgebra, StatsBase, Arpack

function LoadingsUpdate(S::Matrix{T}, r::Int) where T <: Real

(lambda, V) = eigs(S, r, which=:LR) # Arpack call

V = real(V) # convert to real

(lambda, V) = EigenRefinement(S, V)

D = Diagonal(sqrt.(max.(lambda[1:r], zero(T))))

return V[:, 1:r] * D

end

function EigenRefinement(S::Matrix{T}, V::Matrix{T}) where T <: Real

(p, r) =  size(V)

foreach(normalize!, eachcol(V)) # normalize eigenvectors

(lambda, E) = (zeros(T, r), zeros(T, r, r))

VtSV = transpose(V) * ( S * V)

VtV = transpose(V) * V

lambda = diag(VtSV) # eigenvalues

for i = 1:r # construct correction matrix

for j = 1:(i - 1)

E[i, j] = VtSV[i, j] - lambda[j] * VtV[i, j]

E[i, j] = E[i, j] /(lambda[j] - lambda[i])

E[j, i] = VtSV[j, i] - lambda[i] * VtV[j, i]

E[j, i] = E[j, i] / (lambda[i] - lambda[j])

end

end

V = V * (I + E) # improved eigenvectors

foreach(normalize!, eachcol(V)) # normalize eigenvectors

return (lambda, V)

end

function FactorAnalysisPartial(S::Matrix{T}, r::Int) where T <: Real

p = size(S, 1)

(d, old_d, conv) = (zeros(T, p), zeros(T, p), 1.0e-8)

F = randn(p, r) # random factor loadings

for iter = 1:100

for i = 1:p # update specific variances

d[i] = max(S[i, i] - norm(F[i, :])^2, zero(T))

end

for i = 1:p # adjust diagonal sample variances

S[i, i] = S[i, i] - d[i]

end

F = LoadingsUpdate(S, r) # update loadings

for i = 1:p # restore sample variances

S[i, i] = S[i, i] + d[i]

end

if norm(d - old_d) < conv * norm(old_d) && iter > 1 break end

old_d .= d

end

return (F, d)

end

3.4. Robust Factor Analysis

An appealing version of robust factor analysis minimizes the criterion

$$\begin{array}{ccc}\hfill f_{\mu }(\mathit{F},\mathit{D})& =& \sum _i\sum _j{M}_{\mu |·|}[s_{ij}-{\left(\mathit{F}{\mathit{F}}^{\top }\right)}_{ij}-d_{ij}]\hfill \\ & \approx & \sum _i\sum _j|s_{ij}-{\left(\mathit{F}{\mathit{F}}^{\top }\right)}_{ij}-d_{ij}|\hfill \\ & =& \parallel \mathit{S}-\mathit{F}{\mathit{F}}^{\top }{-\mathit{D}\parallel }_1.\hfill \end{array}$$

The minimum of $z\mapsto M_{\mu |·|}(z-d)$ subject to $d\ge 0$ occurs at $d=z$ when $z\ge 0$ and at $d=0$ when $z<0$. This determines the update of the diagonal entries $d_{ii}$ of $\mathit{D}$. In updating $\mathit{F}$, $f_{\mu }(\mathit{F},{\mathit{D}}_{m+1})$ is majorized by

$$\begin{array}{c}\hfill {\displaystyle \frac{1}{2\mu }}{\parallel \mathit{S}-\mathit{F}{\mathit{F}}^{\top }-{\mathit{D}}_{m+1}-\underset{\mu |·|}{\mathrm{prox}}(\mathit{S}-{\mathit{F}}_m{\mathit{F}}_m^{\top }-{\mathit{D}}_{m+1})\parallel }_F^2+c_m,\end{array}$$

where $c_m$ is an irrelevant constant. This is the same optimization criterion as before with $\mathit{S}-{\mathit{D}}_{m+1}$ replaced by $\mathit{S}-{\mathit{D}}_{m+1}-{\mathrm{prox}}_{\mu |·|}(\mathit{S}-{\mathit{F}}_m{\mathit{F}}_m^{\top }-{\mathit{D}}_{m+1})$. The following Julia code implements the Gauss–Newton update of $\mathit{F}$, which is orthogonalized prior to exiting the function FactorAnalysisLAD.

function ProxAbsolute(y::T, t::T) where T <: Real

return sign(y) * max(0, abs(y) - t)

end

function FactorAnalysisLAD(S::Matrix{T}, r::Int, mu::T)

where T <: Real

p = size(S, 1)

(d, old_d, conv) = (zeros(T, p), zeros(T, p), 1.0e-8)

F = randn(p, r) # factor loadings

(FtF, Y, Z, R) = (zeros(T, r, r), zeros(T, r, r), zeros(T, p, r))

R = similar(S)

for iter = 1:100

for i = 1:p # update specific variances

d[i] = max(S[i, i] - norm(F[i, :])^2, zero(T))

end

D = Diagonal(d)

mul!(R, F, transpose(F))

for j = 1:p

for i = 1:p

pr = ProxAbsolute(S[i, j] - R[i, j] - D[i, j], mu)

R[i, j] = S[i, j] - D[i, j] - pr

end

end

mul!(FtF, transpose(F), F) # prepare to update loadings

Y = F / FtF

mul!(Z, S, Y)

Z .= Z .- Diagonal(d) * Y

F .= Z .- F * ((transpose(Y) * Z - I) ./ 2) # loadings update

if norm(d - old_d) < conv * norm(old_d) && iter > 1 break end

old_d .= d

end

SVD = svd(F) # orthogonalize the columns of F

F = SVD.U * Diagonal(SVD.S)

return (F, d)

end

4. Computational Trials

To compare the performance of the algorithms, we simulated data for $r=5$ factors and various combinations of the number of cases n and the dimension p of the ambient space ${\mathbb{R}}^p$. In each simulated dataset, standard normal deviates populate the $p\times r$ factor loading matrix $\mathit{F}$, and uniform deviates on $[0,1]$ populate the specific variance vector $\mathit{d}$. Each observation $\mathit{x}$ follows a Gaussian distribution with mean $\mathbf{0}$ and covariance $\mathit{F}{\mathit{F}}^{\top }+\mathit{D}$, where $\mathit{D}$ is the diagonal matrix with diagonal $\mathit{d}$. After subtracting sample means, the sample covariance is $\mathit{S}={\displaystyle \frac{1}{n}}{\mathit{X}}^{\top }\mathit{X}$. Computational times are averaged over five independent trials for each tuple $(n,p)$.

Table 1 illustrates the consequences of an inaccurate spectral decomposition in iterated principal axis factorization. The values of $\parallel \mathit{S}-\widehat{\mathit{F}}{\widehat{\mathit{F}}}^{\top }-\widehat{\mathit{D}}{\parallel }_F$ are tabulated for a few combinations of the number of cases n and number of features p under the converged values of $\mathit{F}$ and $\mathit{D}$ for the Gauss–Newton algorithm and the partial spectral decomposition algorithm without correction. Clearly, the spectral decompositions delivered by Arpack are untrustworthy in this application. Fortunately, a good remedy is available.

Table 1. Factor analysis accuracy comparisons.

$(\mathit{n},\mathit{p})$	GN	Partial	$(\mathit{n},\mathit{p})$	GN	Partial
(500, 6)	0.0000	9.980	(500, 100)	2.1387	212.17
(500, 10)	0.0763	20.289	(500, 250)	5.3815	548.18
(500, 25)	0.5681	53.674	(500, 500)	10.8630	1044.36
(500, 50)	0.9449	115.939

Table 2 summarizes the computational efficiency of the various algorithms with $n=4000$ cases and smoothing constant $\mu =1$ under the robust method. Spectral correction is applied with the partial spectral decomposition method. The second and third columns of the table supply the average execution time in seconds and the average number of iterations until convergence under the Gauss–Newton algorithm. The remaining columns record the ratios of the average execution times of the other algorithms to the average execution times of Gauss–Newton. The column headings GN, MM, Partial, Full, Robust, EM, and CM indicate Gauss–Newton, MM Gauss–Newton, partial spectral decomposition, full spectral decomposition, approximate least absolute deviation, EM algorithm, and CM algorithm, respectively. The EM and CM algorithms [6,7] are the computational engines for the maximum likelihood factor analysis option of the Julia package MultivariateStats. Blank entries in the table indicate $(p,r)$ combinations too painful to analyze on a laptop computer. Comparison to the MultivariateStats code allows algorithm efficiency differences to emerge unaffected by computer language differences.

Table 2. Factor analysis timing comparisons.

$(\mathit{p},\mathit{r})$	GN	Iters	MM	Partial	Full	Robust	EM	CM
(250, 5)	0.003	14.2	2.267	14.3	50.7	4.14	1360	2344
(500, 5)	0.004	13.2	2.308	19.5	94.6	7.57	2393	10,344
(1000, 5)	0.010	12.0	3.147	25.4	196.2	12.13	-	-
(2000, 5)	0.030	12.0	3.046	71.3	252.7	14.88	-	-
(4000, 5)	0.110	11.8	2.812	130.4	547.3	18.52	-	-
(8000, 5)	0.413	11.6	2.773	154.0	2368.0	20.20	-	-
(250, 10)	0.008	15.8	1.717	-	-	-	-	-
(500, 10)	0.007	14.6	2.354	-	-	-	-	-
(1000, 10)	0.013	13.6	2.457	-	-	-	-	-
(2000, 10)	0.047	12.8	2.680	-	-	-	-	-
(4000, 10)	0.184	12.0	2.854	-	-	-	-	-
(8000, 10)	0.564	11.4	2.798	-	-	-	-	-
(250, 50)	0.036	29.8	1.963	-	-	-	-	-
(500, 50)	0.035	19.6	4.005	-	-	-	-	-
(1000, 50)	0.050	16.2	2.468	-	-	-	-	-
(2000, 50)	0.131	14.4	2.462	-	-	-	-	-
(4000, 50)	0.448	13.8	2.680	-	-	-	-	-
(8000, 50)	1.598	13.0	2.749	-	-	-	-	-
(250, 250)	3.304	500.0	0.952	-	-	-	-	-
(500, 250)	0.841	83.2	2.138	-	-	-	-	-
(1000, 250)	0.531	30.8	2.589	-	-	-	-	-
(2000, 250)	0.930	21.0	2.263	-	-	-	-	-
(4000, 250)	1.937	17.4	2.288	-	-	-	-	-
(8000, 250)	5.128	15.0	2.504	-	-	-	-	-

Gauss–Newton is the clear winner in these speed comparisons. Under the full rank combination $(p,r)=(250,250)$ it struggles and does not satisfy the strict convergence criterion. The MM Gauss–Newton fares better in this scenario. Gauss–Newton’s advantage stems from its avoidance of spectral decompositions and unnecessary allocations of $p\times p$ matrices. The MM version of Gauss–Newton is a close second in speed. The partial spectral decomposition method outperforms the full spectral decomposition method. Both seriously lag Gauss–Newton as the number of parameters increases. The robust method outperforms both spectral decomposition methods. To their credit, all four methods consistently converge to the same solution from random initializations of $\mathit{F}$. Contrary to the claims of the Julia documentation, the EM algorithm is faster than the CM algorithm on our simulated data.

We now turn to the genotype data featured in the software repository SNPArrays (https://github.com/OpenMendel/SnpArrays.jl (accessed on 12 July 2026)). The small European $p\times n$ genotype matrix $\mathit{X}$ deposited there records genotypes on $n=379$ people typed on $p$ = 54,051 SNPS (single nucleotide polymorphisms). Each genotype counts the number of reference alleles at an SNP and is coded as 0, 1, or 2. The genetic relationship matrix $\mathit{G}$ derived from $\mathit{X}$ has entries

$$\begin{array}{ccc}\hfill g_{jk}& =& {\displaystyle \frac{1}{p}}{\displaystyle \sum _{i=1}^p}{\displaystyle \frac{(x_{ij}-2f_i)(x_{ik}-2f_i)}{2f_i(1-f_i)}}\hfill \end{array}$$

and empirically summarizes how closely two people j and k are related. The quantity $f_i$ is the frequency of the reference allele in the population under study.

Factor analysis enters the picture via the approximation $\mathit{G}\approx \mathit{F}{\mathit{F}}^{\top }+\mathit{D}$. This approximation compresses $\mathit{G}$ and serves to reduce the computational load in estimating variance components such as additive genetic variance. On this problem with $r=5$, the EM algorithm takes $0.93$ s and yields the loss $\parallel \mathit{G}-\mathit{F}{\mathit{F}}^{\top }{-\mathit{D}\parallel }_F^2=9.8163$. The CM algorithm fails with a cryptic error message. On a typical run the MM algorithm takes $0.0745$ s, converges in 246 iterations, and achieves a loss of $1.9630$. In contrast, the Gauss–Newton algorithm takes $0.2975$ s, fails to converge in 1000 iterations, and achieves a loss of either $1.9805$ or $1.9808$. In fact, around iteration 75 the Gauss–Newton algorithm repeatedly cycles between two different points with these two losses. Started from the MM estimates, Gauss–Newton does not budge. Except for cycling, the behaviors of MM and Gauss–Newton are quite consistent across many random initializations of $\mathit{F}$. The absence of cycling by the MM algorithm speaks in its favor.

A second dataset from the same software repository records genotypes on $n=1940$ mice typed on $p$ = 10,150 SNPs. On this problem with $r=5$, the EM algorithm takes $130.09$ s and yields a loss of $95.3345$. The MM algorithm takes $0.7586$ s, converges in 207 iterations, and achieves a loss of $57.1027$. The Gauss–Newton algorithm takes $0.2704$ s, converge in 94 iterations, and achieves the same loss. No cycling occurs across 10 runs with different random initializations of $\mathit{F}$. Readers wishing to replicate our findings can avail themselves of the program GenotypeMatrix.jl in our previously mentioned software site.

5. Discussion

Jennrich [5] derives a Gauss–Newton algorithm for factor analysis and identifies it as a version of Fisher scoring. Without presenting empirical evidence, he states

We feel this algorithm is important for several reasons. Because the Fisher scoring algorithm plays a fundamental role in statistical analysis it is of natural interest to know what form it can take when applied to exploratory factor analysis. The resulting algorithm turns out to be simple, easy to implement, faster per iteration than most exploratory algorithms (it requires no eigenvectors), and a respectable algorithm in its own right.

Our numerical experiments validate each of these claims. The computational complexity $O\left(p^{2}r\right)$ of Gauss–Newton per iteration stems from a single $\mathit{S}\mathit{F}$ matrix multiplication, which is accomplished by a fast BLAS level 3 operation. Formation of the sample covariance matrix $\mathit{S}$ also benefits from fast matrix multiplication.

Our MM version of the Gauss–Newton method is tailored to the sum of squares $f(\mathit{F},\mathit{D})={\displaystyle \frac{1}{2}}{\parallel \mathit{S}-\mathit{F}{\mathit{F}}^{\top }-\mathit{D}\parallel }_F^2$. Linearization of $f(\mathit{F},\mathit{D})$ in the increment $\mathbf{\Delta }$ to $\mathit{F}$ and a subsequent majorization lead to the exact update (9). Owing to the MM principle, this update is guaranteed to reduce the linearization. Unfortunately, the additional iterations until convergence counteract the simplicity of the MM updates. The original Gauss–Newton algorithm still comes out ahead, with the caution that it occasionally suffers from parameter cycling close to convergence.

Block descent gives a particularly simple update of $\mathit{D}$ that respects the non-negativity constraints on the diagonal entries of $\mathit{D}$. In Jennrich’s simultaneous versions of Gauss–Newton [5], non-negativity is handled by quadratic programming. Although block descent often takes more iterations to converge than competing algorithms that update all parameters simultaneously, our version of Gauss–Newton factor analysis suffers little penalty in this regard. It is hard to overstate the computational attractiveness of Gauss–Newton factor analysis. We have found that it can be over a thousand-fold faster than maximum likelihood factor analysis. Even if one prefers maximum likelihood factor analysis, the Gauss–Newton estimates can serve as initial values in a maximum likelihood search over parameter space.

Other potential competitors to Gauss–Newton include MINRES (minimum residuals) [21,22] and Newton’s method [23]. MINRES operates on correlation matrices and depends on Gauss–Seidel updating of one row of $\mathit{F}$ at a time. Block updates of this form slow dramatically as the number of blocks increase. Newton’s method, as advocated by Derflinger [23], converges in very few iterations but requires a full $p\times p$ eigen-decomposition at each iteration. This requirement is nearly prohibitive for p large.

Pison et al. [24] suggest a version of factor analysis that leverages a robust estimate of the sample covariance matrix $\mathit{S}$. Although it is beyond the scope of this paper to compare this approach to our robust method of factor analysis, it is worth noting that Pison et al.’s procedure potentially benefits from the Gauss–Newton algorithm.

The perturbation correction of Ogita and Aishima [15] has applications beyond the confines of factor analysis. Correction of approximate principal components immediately comes to mind. Randomized algorithms for PCA give approximate principal components in a fraction of the time taken by full PCA extraction [25]. Single precision PCA partially achieves the same goal. Both methods can be improved by perturbation correction. Another potential application is iteratively reweighted least squares, as practiced in parameter estimation in generalized linear models. Ogita and Aishima [26] recently extended perturbation correction to singular value decompositions. These advances in numerical linear algebra are under-appreciated.