QR-FOLDA: A Fast Orthogonal Linear Discriminant Analysis Based on QR Decomposition

Abstract

Orthogonal Linear Discriminant Analysis (OLDA) has been widely studied for dimensionality reduction, as the orthogonality constraints on its projection matrix enable more effective elimination of redundant features compared to conventional Linear Discriminant Analysis (LDA). However, most existing methods for solving OLDA rely on iterative optimization to sequentially construct orthogonal components, incurring massive repeated high-cost matrix operations and thus leading to substantial computational inefficiency. To address this limitation, we propose QR decomposition-based Fast OLDA (QR-FOLDA), a method built upon a theoretical result (Theorem 1) established in this work: the optimal solutions of LDA remain valid under any full-rank linear transformation. By leveraging this property, QR-FOLDA applies QR decomposition directly to the optimal LDA solution, thereby enforcing orthogonality while avoiding the repeated high-cost matrix operations typically involved in iterative optimization procedures. Experimental evaluations conducted on nine real-world datasets across various domains show that QR-FOLDA not only achieves substantial improvements in computational efficiency compared to existing OLDA methods but also delivers superior classification performance. These findings position QR-FOLDA as a theoretically sound and practically efficient solution for orthogonal discriminant analysis.

1. Introduction

With the rapid advancement of artificial intelligence and sensor technologies, the acquisition and storage of large-scale, high-dimensional data have become increasingly feasible [1]. While such data offer substantial informational richness and potential [2,3], they also pose significant technical challenges, including the curse of dimensionality and increased computational complexity [4,5,6]. As a result, dimensionality reduction techniques, capable of extracting essential features while eliminating redundant information, have emerged as indispensable tools for the effective analysis and processing of high-dimensional data [7,8,9].

Dimensionality reduction techniques primarily fall into two categories: feature selection and feature extraction [10,11]. Feature selection preserves the physical meaning of original features, thereby offering greater interpretability [11,12]. However, this approach inevitably incurs information loss due to feature elimination. In contrast, feature extraction employs data transformation to retain maximal information content, typically yielding superior downstream task performance compared to feature selection [13]. Linear Discriminant Analysis (LDA) is a classical feature extraction technique [14] that aims to maximize intra-class compactness and inter-class separation by finding an optimal projection or transformation matrix. Due to its generalized assumptions, intuitive model, and demonstrated performance improvements, LDA has been widely applied in data analysis, pattern recognition, and related fields [15,16,17].

However, due to the non-orthogonality of the projection matrix in LDA [18,19,20], it fails to effectively eliminate redundant features during data transformation [21,22]. Therefore, researchers have proposed Orthogonal LDA (OLDA), which enforces orthogonality constraints on projection vectors to mitigate feature redundancy and enhance downstream model performance. For instance, Nanni et al. [23] employed a stepwise iterative strategy to enforce mutual orthogonality among the projection vectors in LDA. In [24], a Regularized Fisher’s Linear Discriminant (RFLD) method was introduced, which integrates orthogonality constraints and utilizes eigen decomposition to iteratively eliminate the information contained in previously computed vectors, thereby ensuring orthogonality in subsequent projections. To further improve the computational efficiency of RFLD, Ye et al. proposed the Fast OLDA (FOLDA) algorithm [25], which replaces eigen decomposition with spectral regression to rapidly generate orthogonal projection vectors in RFLD. Furthermore, Ref. [26] reformulated the Rayleigh quotient of LDA into a difference-based optimization problem under the maximum margin criteria, enabling the efficient computation of orthogonal projection vectors. Chu et al. developed a new and fast OLDA (NFOLDA) algorithm that requires neither eigenvalue decomposition nor matrix inversion, solving the problem through several QR factorizations [27].

A critical review of prior studies shows that existing OLDA methods rely on iterative frameworks to sequentially construct orthogonal components, which inevitably introduce a large number of repetitive and computationally expensive matrix operations. To address this limitation, we propose a QR decomposition-based Fast Orthogonal Linear Discriminant Analysis (QR-FOLDA) algorithm that obtains the OLDA solution in a single step by applying QR decomposition to the optimal solution of classical LDA. This approach is grounded in a theoretical result established in this work (Theorem 1 in Section 3), which demonstrates an intrinsic property of LDA: its optimal solution remains valid for the original problem after a linear transformation. By exploiting this property, QR-FOLDA computes orthogonal projection vectors without iterative optimization, thereby significantly reducing computational cost while maintaining strong discriminative performance.

The main contributions and innovations of this work are as follows:

(1)

Unlike existing OLDA methods that construct orthogonal components sequentially during the optimization process, the proposed approach establishes a theoretical relationship between LDA solutions and leverages this relationship to obtain the OLDA solution indirectly. This represents a substantial departure from the conventional OLDA solution paradigm.

(2)

A theoretical result (Theorem 1) is established and rigorously proven that an LDA solution remains valid under any linear transformation. This result enables the proposed method to orthogonalize the LDA solution via QR decomposition, thereby obtaining the OLDA solution without iterative construction.

(3)

Experimental results demonstrate that QR-FOLDA consistently outperforms existing OLDA algorithms by delivering lower computational costs and higher classification accuracy. These results not only confirm the method’s theoretical soundness but also establish it as a practically robust solution for real-world dimensionality reduction tasks.

2. Related Works

In this section, several representative OLDA solution methods, such as Maximum Margin Criterion (MMC) [26], RFLD [24], fast OLDA (FOLDA) [25] and NFOLDA [27], are introduced. Before presenting these algorithms, however, a brief overview of OLDA is first provided.

2.1. OLDA

OLDA seeks to identify a set of orthogonal projection vectors that simultaneously enhance intra-class compactness and inter-class separation of data samples [23]. The objective function can be formulated as

$$\begin{array}{l}\underset{W}{\max }{\displaystyle \frac{Tr(W^T{S}_{b}W)}{Tr(W^T{S}_{w}W)}}\\ s.t.W^{T}W=I\end{array}$$

(1)

where $S_w={\displaystyle \sum _{l=1}^c{\displaystyle \sum _{i=1}^{n_l}(x_i-{\mu }_l)}}{(x_i-{\mu }_l)}^T$ and $S_b={\displaystyle \sum _{l=1}^c{n}_l({\mu }_l-\mu )}{({\mu }_l-\mu )}^T$ denote the intra- and inter- class scatter matrices, respectively, $c$ means the number of classes, $n_l$ represents the number of l-th class samples, ${\mu }_l$ is the l-th class center, $\mu$ indicates the center of samples.

2.2. Maximum Margin Criterion (MMC)

The Maximum Margin Criterion (MMC) [26] reformulates the objective function of OLDA through a different reconstruction, with the transformed objective function expressed as

$$\underset{W^{T}W=I}{\max }Tr(W^T\overline{X}M{\overline{X}}^{T}W-W^T\overline{X}{\overline{X}}^{T}W)$$

(2)

where $\overline{X}$ denotes the centered data matrix of $X$, $M=diag[M^{(1)},M^{(2)},\cdot \cdot \cdot ,M^{(c)}]$ is an $n\times n$ block-diagonal matrix, each submatrix $M^{(l)}$ is an $n_l\times n_l$ matrix with all elements equal to $1/n_l$.

The projection vectors $W$ correspond to the set of eigenvectors associated with the top $m$ largest eigenvalues of the matrix $\overline{X}M{\overline{X}}^T-\overline{X}{\overline{X}}^T$. It is of importance to note that while MMC does not suffer from singularity issues, it shares the same limitation as classical LDA (namely, the dimensionality of the most discriminative projection vectors is constrained to $c-1$).

2.3. RFLD

To obtain the orthogonal projection vectors of LDA, the Regularized Fisher’s Linear Discriminant (RFLD) [24] reformulates the OLDA objective function based on the graph embedding framework. The modified objective function is given as

$$W^{\ast }=\arg \max {\displaystyle \frac{tr(W^{T}XM{X}^{T}W)}{tr(W^{T}X{X}^{T}W)}}$$

(3)

where $n$ represents the total number of samples.

During the computation of orthogonal vectors, the method adopts an iterative approach to determine the orthogonal projection vectors. Throughout the optimization process of Equation (3), each newly generated feature vector must consistently satisfy the following constraints

$$\begin{array}{l}{w}_k=\arg \max {\displaystyle \frac{w^T{X}^{(k)}M{X}^{(k)T}w}{w^T{X}^{(k)}{X}^{(k)T}w}}\\ s.t.w_k^T{w}_1=w_k^T{w}_2=\dots =w_k^T{w}_{k-1}=0\end{array}$$

(4)

where $X^{(k)}=X-W^{(k-1)}{W}^{(k-1)T}X$ represents the updated sample matrix, which eliminates residual information captured by previously computed orthogonal projection vectors.

To enforce orthogonality between $w_k$ and prior projection vectors, RFLD combines the constraints with the generalized eigenvalue problem $X^{(k)}M{X}^{(k)T}w=\lambda X^{(k)}{X}^{(k)T}w$. This leads to the following augmented system: $C^{(k)}w=\lambda B^{(k)}w$, where $B^{(k)}={(X^{(k)}M{X}^{(k)T},w_1,\dots ,w_{k-1})}^T\in {ℝ}^{(d+k-1)\times d}$ and $C^{(k)}={(X^{(k)}{X}^{(k)T},0,\dots ,0)}^T\in {ℝ}^{(d+k-1)\times d}$.

Given that $B^k$ is full rank, the solution $w_k$ corresponds to the eigenvector associated with the largest eigenvalue of $B^{kT}Cw=\lambda B^{kT}{B}^{k}w$.

2.4. FOLDA

Building upon the conceptual framework of RFLD, Reference [25] proposed the Fast Orthogonal Linear Discriminant Analysis (FOLDA) algorithm, which maintains the same objective function as RFLD (Equation (3)). However, FOLDA introduces a more computationally efficient approach by employing spectral regression (SR) to compute individual projection vectors, in contrast to the singular value decomposition (SVD) method used in RFLD. This methodological advancement substantially improves computational efficiency while preserving the theoretical soundness of the algorithm.

2.5. NFOLDA

In [27], the authors proposed a new method for fast solving OLDA, named NFOLDA. This approach effectively avoids eigenvalue decomposition and matrix inversion. Through rigorous mathematical proofs, it achieves the solution with several QR factorizations. The specific procedure is as follows:

• Compute the economic QR factorization of X as $X=U_{1}R$, where $U_1$ is column orthogonal and $R=[R_1 R_2 R_3]$.
• Compute $[R_2 R_3]=Q_1\left[\begin{array}{c}{R}_{1,1}\\ 0\\ 0\end{array}\begin{array}{c}{R}_{1,2}\\ R_{2,2}\\ 0\end{array}\right]$, where Q₁ is a column orthogonal matrix, $R_{1,1}$ and $R_{2,2}$ are of full row rank, and $\gamma =rank[R_2 R_3]$.
• Compute $QR(\left[\begin{array}{c}{R}_{1,2}\\ R_{2,2}\end{array}\right]R_{2,2}^T)=V\left[\begin{array}{c}P\\ 0\end{array}\right]$, where $V\in R^{\gamma \times \gamma }$ is orthogonal.
• Obtain $\widehat{W}=U_1{Q}_1(:,1:\gamma )V(:,\lambda +1-m:\gamma )$, where m indicates the number of dimensions after reduction.

3. Methodology

3.1. Proposed Method (QR Decomposition-Based FOLDA)

To address the dimensionality bottleneck and computational complexity associated with conventional orthogonal linear discriminant analysis (OLDA) in high-dimensional data processing, this paper proposes an efficient OLDA solution method. The proposed approach first leverages the intrinsic relationships among LDA solutions (the optimal solution of LDA remains valid under a linear transformation) and then applies QR matrix decomposition to derive the OLDA projection matrix for orthogonal data transformation. In contrast to computation-intensive methods such as RFLD, the proposed method substantially reduces computational time while improving algorithmic efficiency and practical applicability.

To efficiently compute the orthogonal LDA projection matrix $W$, the objective function of orthogonal LDA can be reformulated as follows:

$$\begin{array}{l}\underset{W}{\max }Tr\left(\left.{\left(W^T{S}_{w}W\right)}^{-1}{W}^T{S}_{b}W\right)\right.\\ s.t.W^{T}W=I\end{array}$$

(5)

It should be particularly noted that the within-class scatter matrix $S_w$ is assumed to be positive definite. If the input data satisfy $\left|S_w\right|=0$, a situation commonly encountered in high-dimensional or small-sample-size scenarios, a regularization strategy is adopted by replacing $S_w$ with a stabilized matrix $S_w+\delta I$, where $\delta \succ 0$ is a small regularization parameter (In this study, $\delta$ is set to 10⁻⁶). This modification ensures that the resulting within-class scatter matrix becomes positive definite, thereby allowing the Fisher discriminant objective to be formulated as a generalized eigenvalue problem derived from the maximization of a generalized Rayleigh quotient.

First, consider the unconstrained trace optimization problem of Equation (5).

$$\underset{W}{\max }Tr\left(\left.{\left(W^T{S}_{w}W\right)}^{-1}{W}^T{S}_{b}W\right)\right.$$

(6)

Assuming $\widehat{W}$ is the optimal solution to Equation (6), and utilizing the property that similar matrices have equal traces, we derive the following:

$$Tr\left(\left.{\left({\widehat{W}}^T{S}_w\widehat{W}\right)}^{-1}{\widehat{W}}^T{S}_b\widehat{W}\right)\right.=Tr\left(\left.{\left(Q^T{\widehat{W}}^T{S}_w\widehat{W}Q\right)}^{-1}{Q}^T{\widehat{W}}^T{S}_b\widehat{W}Q\right)\right.$$

(7)

where Q is an arbitrary invertible matrix of size $m\times m$.

From Equation (7), it follows that the unconstrained problem defined in Equation (6) possesses the following property. If $\widehat{W}$ is an optimal solution to Equation (6), then $\widehat{W}Q$ is also an optimal solution for any invertible matrix Q. To formally establish this property and facilitate the determination of the optimal solution, we first present Theorem 1 along with its proof.

Theorem 1.

Let $V\in {ℝ}^{d\times m}$ be the matrix consisting of the generalized eigenvectors corresponding to the largest m eigenvalues of the matrix pair ($S_b,S_w$). Q is an arbitrary invertible matrix of dimension $m\times m$. Then, the matrix $VQ$ also constitutes an optimal solution to Equation (6).

Proof. 

By applying an identity transformation to both sides of the equation, we derive the following

$$\begin{array}{l}Tr\left(\left.{\left(W^T{S}_{w}W\right)}^{-1}{W}^T{S}_{b}W\right)\right.\\ =Tr\left(\left.{\left(W^T{S}_{w}W\right)}^{-{\displaystyle \frac{1}{2}}}{W}^T{S}_{b}W{\left(W^T{S}_{w}W\right)}^{-{\displaystyle \frac{1}{2}}}\right)\right.\\ =Tr\left(V^T{S}_{b}V\right)\end{array}$$

(8)

where $V=W{\left(W^T{S}_{w}W\right)}^{-{\displaystyle \frac{1}{2}}}$.

Here, it clearly holds that $V^T{S}_{w}V=I$. Consequently, the unconstrained problem (Equation (6)) can be reformulated as

$$\underset{V^T{S}_{w}V=I}{max}Tr\left(V^T{S}_{b}V\right)$$

(9)

The solution to Equation (9) is derived using the method of Lagrange multipliers. The corresponding Lagrangian function is formulated as follows

$$L\left(V,\lambda \right)=Tr\left(V^T{S}_{b}V\right)-Tr\left(\lambda \left(V^T{S}_{w}V-I\right)\right)$$

(10)

To obtain the optimal solution of Equation (9), the parameters $\lambda$ and V must be determined such that the constraint $V^T{S}_{w}V=I$ is satisfied and the partial derivative of Equation (10) with respect to V equals zero. Assuming $\lambda$ is a symmetric matrix, it can be decomposed as $\lambda =U\Lambda U^T$, where $\Lambda$ is the eigenvalue matrix of $\lambda$ and $U$ is the corresponding eigenvector matrix. Taking the partial derivative of Equation (10) with respect to V and setting it to zero yields:

$$\begin{array}{l}{\displaystyle \frac{\partial L\left(V,\lambda \right)}{\partial V}}=2S_{b}V-2S_{w}V\lambda =0\\ \stackrel{\lambda =U\Lambda U^T}{\Rightarrow }{S}_{b}V=S_{w}VU\Lambda U^T\end{array}$$

(11)

where $U$ can be absorbed through a right-orthogonal transformation of V without altering the optimization problem or its solution space. Therefore, Equation (11) can be rewritten as

$$\begin{array}{l}{S}_{b}V=S_{w}V\Lambda \\ \Rightarrow {\left(S_w\right)}^{-1}{S}_{b}V=V\Lambda \end{array}$$

(12)

Since $S_w$ is symmetric and positive definite, the generalized eigenvectors $V$ in Equation (12) can be chosen and normalized to be $S_w$-orthonormal so that $V^T{S}_{w}V=I$. Thus, when the Lagrange coefficient $\lambda =\Lambda$ and $V$ consists of the generalized eigenvectors of the matrix pair ($S_b,S_w$), both the constraint $V^T{S}_{w}V=I$ and $\partial L\left(V,\lambda \right)/\partial V=0$ remain valid. Consequently, the solution to Equation (9) reduces to solving the generalized eigenvalue decomposition in Equation (12), where $V$ comprises the generalized eigenvectors corresponding to the largest $m$ eigenvalues of the matrix pair ($S_b,S_w$). Therefore, $V$ is the optimal solution to the unconstrained problem defined in Equation (6).

Thus, based on the properties of the unconstrained problem and the relation $V=W{\left(W^T{S}_{w}W\right)}^{-{\displaystyle \frac{1}{2}}}$, it follows that for any invertible $m\times m$ matrix $Q$, $VQ$ remains an optimal solution to the unconstrained problem (Equation (6)). □

Now consider solving the constrained problem (Equation (5)) subject to $W^{T}W=I$.

From Theorem 1, the optimal solution to the unconstrained problem is $VQ\in {ℝ}^{d\times m}$. Applying QR decomposition to $VQ$ yields

$$QR\left(VQ\right)=\widehat{W}R$$

(13)

where $\widehat{W}\in {ℝ}^{d\times m}$ is the orthogonal matrix from the QR decomposition (satisfying $W^{T}W=I$), $R\in {ℝ}^{m\times m}$ is the upper triangular matrix.

By Theorem 1 and the properties of the unconstrained problem, $\widehat{W}$ is the optimal solution to the constrained problem defined in Equation (5).

The pseudo code of the proposed algorithm is as follows (Algorithm 1, proposed method).

Algorithm 1. Proposed Method (QR-FOLDA)

Input: dataset $X\in {ℝ}^{N\times d}$, label of dataset Y, penalty factors δ(=10−6), dimension after mapping (m).

Begin

Calculate ${\mu }_l$ and $\mu$;

Calculate $S_w$ and $S_b$ according to $S_w={\displaystyle \sum _{l=1}^c{\displaystyle \sum _{i=1}^{n_l}(x_i-{\mu }_l)}}{(x_i-{\mu }_l)}^T$ and $S_b={\displaystyle \sum _{l=1}^c{n}_l({\mu }_l-\mu )}{({\mu }_l-\mu )}^T$;    If $\left|S_w\right|=0$    $S_w=S_w+\delta I$;    End if

V is solved by the generalized eigenvalue problem of classical LDA;    Generate a random invertible matrix Q;    Apply a QR decomposition to VQ, yielding $WR=QR(VQ)$.

End

Output: the orthogonal projection matrix $W\in {ℝ}^{d\times m}$.

3.2. Computational Complexity Analysis

Although numerous methods have been proposed for solving orthogonal LDA, notable differences in computational efficiency persist among them in practical applications. Accordingly, this section presents a theoretical analysis and comparison of the computational efficiency of representative approaches.

MMC: The method directly computes the eigenvectors corresponding to the largest m eigenvalues of the matrix $\overline{X}M{\overline{X}}^T-\overline{X}{\overline{X}}^T$. This requires first calculating the matrices $\overline{X}M{\overline{X}}^T$ and $\overline{X}{\overline{X}}^T$ with time complexities of $O\left(n^{2}d+n{d}^2\right)$ and $O\left(n{d}^2\right)$ respectively, followed by computing the top m eigenvectors of $\overline{X}M{\overline{X}}^T-\overline{X}{\overline{X}}^T$ with a complexity of $O\left(m{d}^2\right)$. Consequently, the main computational complexity of MMC is $O\left(n^{2}d+\left(2n+m\right)d^2\right)$.

RFLD: RFLD involves two main computational steps. First, solving for the initial projection vector $w_1$ requires addressing the generalized eigenvalue problem $\overline{X}M{\overline{X}}^{T}w=\lambda {\overline{XX}}^T$, with a computational complexity of $O\left(n^{2}d+n{d}^2+d^3\right)$. Second, computing subsequent vectors $w_k$ (k ≥ 2) involves solving the generalized eigenvalue problem $B^{{\left(k\right)}^T}{C}^{\left(k\right)}w=\lambda B^{{\left(k\right)}^T}{B}^{\left(k\right)}w$, where matrices $B^{\left(k\right)}$ and $C^{\left(k\right)}$ both have dimensions $\left(d+k-1\right)\times d$. The computation of the generalized eigenvalue problem requires first calculating matrices $B^{\left(k\right)}$ and $C^{\left(k\right)}$, corresponding to ${\overline{X}}^{\left(k\right)}M{\overline{X}}^{\left(k\right)T}$ and ${\overline{X}}^{\left(k\right)}{\overline{X}}^{\left(k\right)T}$ respectively, with a time complexity of $O\left(d{n}^2+n{d}^2\right)$. Subsequent computation of $B^{{\left(k\right)}^T}{C}^{\left(k\right)}$ and $B^{{\left(k\right)}^T}{B}^{\left(k\right)}$ incurs $O\left(\left(d+k-1\right)d^2\right)$ operations, followed by solving for $w_k$ (analogous to $w_1$) with $O\left(d^2+d^3\right)$ complexity. Consequently, the total computational complexity for each $w_k$ becomes $O\left(n^{2}d+\left(n+d+k\right)d^2+d^3\right)$. The combined time complexity for obtaining projection vectors $w_2,w_3,\cdot \cdot \cdot ,w_m$ amounts to $O\left(({\displaystyle \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}{m}^2+2mn)d^2+m{d}^3\right)$. Thus, the main computational complexity of RFLD is $O\left(m{n}^{2}d+\left({\displaystyle \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}{m}^2+2mn\right)d^2+m{d}^3\right)$.

FOLDA: Similar to RFLD, the solution process of FOLDA also consists of two main steps. The first step solves for the initial projection vector $w_1$ by optimizing the regression problem $w_1=\arg \underset{w}{\min }{\left({\overline{X}}^{T}w-a\right)}^2$, which has a computational complexity of $O\left(ns\right)$, where s represents the average number of non-zero features per sample in $\overline{X}$. The second step computes subsequent vectors $w_k$ through three operations: QR decomposition of $W^{\left(k-1\right)}$ with $O\left({\left(k-1\right)}^{2}d\right)$ complexity, solving the least squares regression problem $z_k=argmin{\left(G^{\left(k-1\right)}z-a_k\right)}^2$ with $O\left(n{d}^2\right)$, and finally calculating $W_k=Q^{\left(k-1\right)}C{z}_k$ with $o\left(d^2\right)$. The combined complexity for each $w_k$ is approximately $O\left({\left(k-1\right)}^{2}d+\left(n+1\right)d^2+ne\right)$. Similar to RFLD, all preceding $k-1$ projection vectors must be computed to obtain $z_k$. The total complexity for deriving the m-dimensional subspace is $O\left({\displaystyle \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}{m}^{2}d+\left(m-1\right)n{d}^2+\left(m-1\right)ne\right)$. Consequently, the main computational complexity of FOLDA is $O\left({\displaystyle \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}{m}^{2}d+mn{d}^2\right)$.

NFOLDA: From the related works section, the NFOLDA solution primarily involves four steps. The combined computational complexity of steps 1 and 2 is $O(4n^{2}d+4n^3+4n{\gamma }^2+\raisebox{1ex}{$4$}\!\left/ \!\raisebox{-1ex}{$3$}\right.{\gamma }^3)$, where $\gamma =rank[R_2 R_3]＜min(n,d)$. The step 3 has a complexity of $O(2n{\gamma }^2+\raisebox{1ex}{$8$}\!\left/ \!\raisebox{-1ex}{$3$}\right.{\gamma }^3)$ and the computational complexity of step 4 is $O(2n{d}^2+2nd\gamma )$. Therefore, the overall computational complexity of NFOLDA can be approximated as $O(4n^3+4{\gamma }^3+4n^{2}m+2n{m}^2+6n{\gamma }^2+2nm\gamma )$. It is important to note that the stated complexity omits certain lower-order operations. For a more precise and detailed analysis, please refer to reference [27].

QR-FOLDA: The computational complexity analysis of the QR-FOLDA algorithm involves three key components. First, the computational complexity for calculating both the within-class scatter matrix $S_w$ and between-class scatter matrix $S_b$ is $O\left(n{d}^2\right)$. Therefore, the total time complexity for computing $S_b$ and $S_w$ is $O\left(2n{d}^2\right)$. Second, solving the generalized eigenvectors of the matrix pair ($S_b,S_w$) has a complexity of $O\left(d^3\right)$. Finally, generating a random invertible matrix Q incurs a time complexity of $O\left(m^2\right)$, while both computing VQ and performing QR decomposition on VQ have a complexity of $O\left(d{m}^2\right)$. Thus, the main time complexity of QR-FOLDA is $O\left(2m^{2}d+2n{d}^2+d^3\right)$.

Based on the above analysis, it can be concluded that QR-FOLDA achieves a computational complexity comparable to that of MMC ($O\left(n^{2}d+\left(2n+m\right)d^2\right)$), while overcoming its inherent limitation of reducing the feature dimension to at most c − 1. Compared with RFLD ($O\left(m{n}^{2}d+\left({\displaystyle \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}{m}^2+2mn\right)d^2+m{d}^3\right)$), FOLDA ($O\left({\displaystyle \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}{m}^{2}d+mn{d}^2\right)$), and NFOLDA ($O(4n^3+4{\gamma }^3+4n^{2}m+2n{m}^2+6n{\gamma }^2+2nm\gamma )$), the proposed approach demonstrates significantly lower time complexity, highlighting its improved efficiency in solving OLDA and greater potential for handling large-scale datasets. Another noteworthy observation is that, although QR-FOLDA introduces an additional computational cost of $O\left(2m^{2}d\right)$ compared with conventional LDA ($O\left(2n{d}^2+d^3\right)$), this overhead is generally negligible in high-dimensional regimes where $m\ll d$. More importantly, the projection matrix derived by QR-FOLDA is orthogonal, which enhances the discriminative power of the extracted features. As a result, the proposed method is more effective in capturing inter-class differences while preserving intra-class compactness than classic LDA.

4. Experimental Results

4.1. Experimental Environment

All experiments are conducted on a desktop computer with Microsoft Windows 10 operating system (64-bit) with a 13th Gen Intel^® Core™ i5-13500H (Intel Corporation, Santa Clara, CA, USA) of 2.60 GHz, and a RAM of 32 GB. The experiments are conducted with PyCharm compiler, and the main dependencies include Python 3. 12.3, Pytorch 2.8.0, torchvision 0.23.0 and Scikit-learn 1.7.1.

4.2. Experimental Data

To evaluate the effectiveness of QR-FOLDA, comparative experiments are conducted on nine publicly available datasets of varying sizes across different domains. These datasets (Dry Bean [28], MEU-Mobile KSD [29], USPS [30], MNIST [30], Person Classification Gait Data [31], DARWIN [32], IOSLET [33], Toxicity [34], and Gene expression cancer RNA-Seq [35]) are all freely accessible from UCI Machine Learning Repository (website: https://archive.ics.uci.edu/, accessed on 8 May 2025).

Dry Bean [28]: This dataset contains 13,611 dry bean images from seven different bean varieties. Twelve size-related features and four shape-related features are extracted from each image to facilitate bean classification.

MEU-Mobile KSD [29]: Collected using a Nexus 7 device (Asus, Taipei, Taiwan), this dataset consists of 2856 keystroke sensor records from 56 participants (51 records per participant). A total of 71 metric features is extracted from each record for user identification or touch behavior analysis.

USPS [30]: Provided by the U.S. Postal Service, this dataset contains 9298 grayscale images of handwritten digits (0–9), each sized at 16 × 16 pixels. It is widely used in handwritten digit recognition tasks, especially for zip code identification on envelopes.

MNIST [30]: A classic handwritten digit recognition dataset created by the U.S. National Institute of Standards and Technology, comprising 70,000 images (28 × 28 pixels) across 10 digit classes. For consistency with USPS, the images are downsampled to match its resolution.

Person Classification Gait Data [31]: This dataset includes 48 gait sequences from 16 volunteers (7 females and 9 males, with three sequences per person). A total of 321 features—including basic and transient gait parameters—are extracted from each sequence for person identification.

DARWIN [32]: The dataset consists of handwriting data from 174 individuals. A total of 451 nonlinear features associated with Alzheimer’s disease are extracted to differentiate between patients and healthy controls.

ISOLET [33]: Comprising 7797 valid audio recordings collected from 150 subjects (each pronouncing 26 English letters twice), the dataset provides 617 acoustic features per sample for letter classification.

Toxicity [34]: This dataset contains 1203 molecular descriptors for 171 chemical compounds, including 56 toxic and 115 non-toxic substances. It is used for biochemical toxicity prediction.

Gene expression cancer RNA-Seq [35]: As part of the RNA-Seq PANCAN dataset, this dataset includes gene expression profiles from 801 randomly selected patients across five cancer types: breast cancer, clear cell kidney carcinoma, colon adenocarcinoma, lung adenocarcinoma, and prostate adenocarcinoma.

A summary of the datasets is provided in

Table 1. Basic information of the dataset used in this study.

Dataset	Sample Size	Number of Features	Classes	Application Domain	Type of Data
Dry Bean	13,611	16	7	Image Recognition	Legume Images
MEU-Mobile KSD	2856	71	56	Identity Recognition	Pressure Sensor Data
USPS	9298	256	10	Handwriting Recognition	Handwriting Recognition
MNIST	70,000	256	10	Handwriting Recognition	Handwriting Recognition
Person Classification Gait Data	48	321	16	Behavior Analysis/Identity Recognition	Gait Images
DARWIN	174	451	2	Medicine and Healthcare	Medicine and Healthcare
ISOLET	7797	617	26	Speech Recognition	Audio Recordings
Toxicity	171	1203	2	Toxic Substance Identification	Text
Gene expression cancer RNA-Seq	801	20,531	5	Tumor Gene Expression	Tumor Gene Expression

.

4.3. Experimental Results and Analysis

To validate the effectiveness of the proposed algorithm, three sets of experiments are respectively conducted to evaluate its performance in terms of classification performance, computational efficiency (time consumption analysis) and sensitivity to parameter variations.

4.3.1. Classification Performance

To evaluate the classification performance of the proposed algorithm, the hold-out method is employed, in which each dataset is randomly shuffled and split into training and testing sets at a ratio of 7:3. Furthermore, several representative algorithms for solving OLDA projection vectors are selected as comparative approaches. These include RFLD, MMC, and FOLDA. In addition, LDA is adopted as a baseline to evaluate the effectiveness of orthogonal projection vectors in enhancing discriminative capability.

Additionally, parameter settings also play a crucial role in the experimental procedure. Similar to the proposed method, all comparative algorithms contain only a single parameter—the reduced dimensionality m. To ensure fair evaluation, the parameter optimization is performed across different search ranges, determined by the original dimensionality of each dataset. The detailed settings are summarized in

Table 2. Parameter settings of each algorithm on different datasets.

Dataset	LDA	MMC	RFLD	FOLDA	NFOLDA	QR-FOLDA
Dry Bean	(1, 1, 6)	(1, 1, 6)	(1, 1, 15)	(1, 1, 15)	(1, 1, 15)	(1, 1, 15)
MEU-Mobile KSD	(1, 1, 55)	(1, 1, 55)	(1, 1, 70)	(1, 1, 70)	(1, 1, 70)	(1, 1, 70)
USPS	(1, 1, 9)	(1, 1, 9)	(5, 5, 255)	(5, 5, 255)	(5, 5, 255)	(5, 5, 255)
MNIST	(1, 1, 9)	(1, 1, 9)	(5, 5, 255)	(5, 5, 255)	(5, 5, 255)	(5, 5, 255)
Person Classification Gait Data	(1, 1, 15)	(1, 1, 15)	(5, 5, 320)	(5, 5, 320)	(5, 5, 320)	(5, 5, 320)
DARWIN	(1, 1, 1)	(1, 1, 1)	(5, 5, 450)	(5, 5, 450)	(5, 5, 450)	(5, 5, 450)
ISOLET	(5, 5, 25)	(5, 5, 25)	(5, 5, 616)	(5, 5, 616)	(5, 5, 616)	(5, 5, 616)
Toxicity	(1, 1, 1)	(1, 1, 1)	(5, 10, 1195)	(5, 10, 1195)	(5, 10, 1195)	(5, 10, 1195)
Gene expression cancer RNA-Seq	(1, 1, 4)	(1, 1, 4)	(5, 300, 20,405)	(5, 300, 20,405)	(5, 300, 20,405)	(5, 300, 20,405)

. It is worth noting that in Table 2, the triplet (a, b, f) represents the starting dimension a, the step size b, and the ending dimension f in the parameter search process.

It should be noted that to mitigate the influence of randomness on performance evaluation (measured by classification accuracy), each experiment is independently repeated ten times. Moreover, to assess whether the performance differences between the proposed method and baseline algorithms are statistically significant, significance analysis is conducted based on the ten experimental results (with a significance level of $\alpha =0.05$; results significantly different from the proposed method are underlined). The statistical results are summarized in

Table 3. Statistical results of classification performance across different algorithms (Mean ± Standard Deviation).

Dataset	LDA	MMC	RFLD	FOLDA	NFOLDA	QR-FOLDA
Dry Bean	91.11 ± 0.31	91.52 ± 0.98	92.48 ± 0.38	92.26 ± 0.33	92.46 ± 0.32	92.50 ± 0.41
MEU-Mobile KSD	72.13 ± 1.77	74.80 ± 1.71	63.79 ± 2.60	62.91 ± 2.64	77.25 ± 2.00	76.26 ± 1.31
USPS	79.25 ± 1.82	71.50 ± 2.81	91.89 ± 0.70	84.91 ± 1.26	88.78 ± 1.09	91.67 ± 1.01
MNIST	67.73 ± 2.50	48.20 ± 3.95	84.72 ± 1.71	83.10 ± 1.05	85.07 ± 1.91	88.65 ± 1.01
Person Classification Gait Data	64.00 ± 13.77	47.33 ± 13.50	59.33 ± 16.16	69.33 ± 22.04	70.67 ± 16.69	78.67 ± 16.87
DARWIN	49.25 ± 7.20	53.58 ± 7.66	86.04 ± 2.98	82.64 ± 4.05	79.06 ± 6.13	86.98 ± 2.25
ISOLET	96.01 ± 0.56	93.35 ± 0.53	93.62 ± 0.64	95.32 ± 0.50	93.12 ± 0.71	95.62 ± 0.58
Toxicity	61.35 ± 5.16	67.69 ± 8.96	59.62 ± 3.85	70.57 ± 5.37	64.81 ± 7.91	63.65 ± 5.08
Gene expression cancer RNA-Seq	99.58 ± 0.51	99.17 ± 0.72	99.06 ± 0.66	99.17 ± 0.74	98.38 ± 0.61	99.83 ± 0.37

Note: Values in bold indicate the best. Values underlined indicate a significant difference between the method and the proposed algorithm.

.

As shown in Table 3, several key observations can be made regarding the performance of the proposed method. (i) QR-FOLDA achieves the highest classification accuracy on the majority of datasets (5 out of 9), and in most of these cases (32 out of 45), the improvement is statistically significant compared to the competing methods. This highlights the superior classification capability of the proposed approach under various data conditions. (ii) Although the proposed method does not yield the top accuracy on four datasets, it consistently ranks among the top performers—specifically, second on the USPS, ISOLET and MEU-Mobile KSD datasets, and third on the Toxicity dataset. Notably, on USPS and ISOLET, there is no statistically significant difference between the proposed method and the best-performing alternatives (RFLD for USPS and LDA for ISOLET, respectively), indicating that the proposed method achieves competitive performance even in these scenarios. (iii) QR-FOLDA also exhibits the lowest standard deviation in classification accuracy on nearly half of the datasets (4 out of 9), demonstrating its strong robustness and stability across diverse data distributions. Collectively, these findings affirm that the proposed method serves as a highly effective and stable dimensionality reduction technique compared to its competitors. Moreover, an additional interesting finding is that orthogonal LDA-based methods (RFLD, MMC, FOLDA, QR-FOLDA) outperform classical LDA in more than half of the cases (29 out of 45), suggesting that orthogonal transformations are generally more capable of capturing discriminative structures within the data compared to their non-orthogonal counterparts.

To provide an intuitive illustration of the classification stability of the proposed method, boxplots depicting the classification accuracies of different algorithms across various datasets are presented in Figure 1. As shown, the proposed method yields relatively compact accuracy distributions across multiple runs on each dataset, suggesting its strong robustness and consistency in classification performance.

4.3.2. Time Consumption Analysis

To assess the computational efficiency of the proposed algorithm, this section presents a comparative analysis of the time consumption across different methods. It is worth noting that both classical LDA and MMC are constrained to reduce the data dimensionality to at most c − 1, where c denotes the number of classes. Consequently, experiments are conducted on datasets with a relatively large number of classes—specifically, ISOLET and MEU-Mobile KSD. To ensure a fair comparison, the target dimensionality for all methods is uniformly set to c − 1, resulting in reduced dimensions of 25 for ISOLET and 55 for MEU-Mobile KSD. In addition, all reported runtime measurements include only the computation of the orthogonal projection (mapping) matrix. Operations unrelated to the algorithm itself, such as data preprocessing and classifier training, are excluded. The corresponding time consumption results for each algorithm are summarized in

Table 4. Time consumption statistics of different algorithms (in seconds).

Datasets	LDA	MMC	RFLD	FOLDA	NFOLDA	QR-FOLDA
ISOLET	0.206	0.326	19.111	10.478	3.591	0.345
MEU-Mobile KSD	0.006	0.007	0.784	0.534	0.112	0.008

.

As shown in Table 4, the time consumption of the algorithms follows the trend: LDA < MMC ≈ QR-FOLDA < NFOLDA < FOLDA < RFLD. Notably, QR-FOLDA requires substantially less computational time than RFLD, FOLDA and NFOLDA, while incurring only a slight increase in runtime compared to LDA and MMC. The additional computational cost compared to LDA primarily stems from the subsequent QR decomposition step. Compared with MMC, which involves only a single eigenvalue decomposition, the proposed method performs two matrix factorization steps, leading to slightly increased time consumption. Nevertheless, QR-FOLDA successfully addresses the dimensionality constraint inherent in MMC, thereby enabling significantly enhanced classification performance in many cases.

4.3.3. Parameter Analysis

Beyond efficiency and accuracy, the parameter sensitivity of the algorithm remains a noteworthy issue that warrants further discussion. Therefore, we investigate the impact of the dimensionality parameter (dim) on the classification accuracy (acc) of each algorithm. The results are depicted in Figure 2. It is worth noting that, since LDA and MMC are inherently constrained to a maximum of c − 1 dimensions (where c denotes the number of classes), their performance beyond this limit is represented by their accuracy at c − 1. Furthermore, the dimensionality step size is set to 1 to enable a more fine-grained analysis of convergence behavior of each algorithm.

As shown in Figure 2a,c,e–g,i, the proposed method, similar to other compared algorithms, exhibits a pattern wherein classification accuracy initially increases and then reaches a plateau. Notably, this convergence trend typically occurs when the reduced dimensionality reaches approximately 20% of the original data dimension, suggesting that the proposed method achieves rapid convergence on these datasets. In contrast, Figure 2b,d illustrate that the proposed method continues to exhibit performance gains as dimensionality increases, whereas the baseline methods have already converged or stabilized at earlier stages. This observation indicates that the proposed algorithm is more effective at capturing highly discriminative features, particularly in higher-dimensional spaces, compared to its counterparts.

Another noteworthy observation from Figure 2 is that the proposed QR-FOLDA method consistently achieves optimal classification performance at relatively higher dimensionalities. To further evaluate the practical efficiency of each algorithm,

Table 5. Computational Time of Algorithms at Optimal Dimensionality (Dimension, Time: seconds).

Dataset	LDA	MMC	RFLD	FOLDA	NFOLDA	QR-FOLDA
Dry Bean	5, 0.004	6, 0.005	14, 0.099	15, 0.052	6, 0.011	15, 0.006
MEU-Mobile KSD	48, 0.006	54, 0.007	63, 0.172	69, 0.682	52, 0.013	70, 0.009
USPS	9, 0.023	9, 0.029	151, 2.377	143, 0.604	10, 0.134	229, 0.062
MNIST	8, 0.027	9, 0.033	155, 2.912	191, 0.647	10, 0.129	252, 0.055
Person Classification Gait Data	13, 0.026	12, 0.023	17, 1.679	32, 0.115	15, 0.109	207, 0.049
DARWIN	1, 0.037	1, 0.049	47, 2.014	80, 0.671	5, 0.161	251, 0.084
ISOLET	13, 0.204	2, 0.165	14, 5.017	7, 8.551	5, 2.277	16, 0.219
Toxicity	1, 0.008	1, 0.010	136, 2.228	297, 0.992	35, 0.348	168, 0.012
gene expression cancer RNA-Seq	4, 8.889	4, 8.943	793, 178.177	775, 19.989	305, 12.76	806, 9.578

summarizes the time consumption of all methods at their respective optimal dimensions, i.e., when each yields peak classification accuracy.

It can be seen from Table 5 that, although the proposed method requires the highest optimal dimensionality among the compared algorithms, its computational cost still follows the pattern: LDA < MMC ≈ QR-FOLDA < NFOLDA < FOLDA < RFLD, owing to its efficient computational framework. This result suggests that the increased dimensionality has a limited impact on the overall runtime of the proposed algorithm.

5. Conclusions and Further Study

This study introduces a novel orthogonal linear discriminant analysis algorithm, termed QR-FOLDA, to overcome the computational inefficiencies inherent in existing OLDA methods. The approach is grounded in a theoretically established property of classical LDA that any optimal LDA solution remains valid under full-rank linear transformations. By leveraging this property, QR-FOLDA applies QR decomposition to transform the optimal LDA projection into an orthogonal solution without iterative optimization. Extensive experiments on nine publicly available datasets across diverse domains and data scales show that QR-FOLDA consistently delivers substantial reductions in computational cost while achieving classification accuracy on par with, or superior to, existing OLDA algorithms. These results show QR-FOLDA as not only a theoretically grounded contribution but also a practically robust and scalable solution for dimensionality reduction.

Despite these promising results, classification performance on high-dimensional dense data remains an area for improvement. In particular, on the ISOLET dataset (617 features), the method’s advantage over FOLDA is marginal (95.62% vs. 95.32%), while on the Toxicity dataset (1203 features) it falls significantly short of the best result (63.65% vs. 70.57%). This limitation is largely attributed to an inherent drawback of QR decomposition, where numerical precision issues can lead to rapid error accumulation in high-dimensional, non-sparse matrices. Future research could address this numerical stability issue to improve accuracy on such challenging datasets. Meanwhile, the proposed method can further incorporate fast matrix multiplication techniques to improve computational efficiency. For example, replacing the standard QR decomposition with an economic QR factorization [27,36,37] could further reduce the computational cost.