Distributed Sparse Manifold-Constrained Optimization Algorithm in Linear Discriminant Analysis

Abstract

In the field of video image processing, high definition is one of the main directions for future development. Faced with the curse of dimensionality caused by the increasingly large amount of ultra-high-definition video data, effective dimensionality reduction techniques have become increasingly important. Linear discriminant analysis (LDA) is a supervised learning dimensionality reduction technique that has been widely used in data preprocessing for dimensionality reduction and video image processing tasks. However, traditional LDA methods are not suitable for the dimensionality reduction and processing of small high-dimensional samples. In order to improve the accuracy and robustness of linear discriminant analysis, this paper proposes a new distributed sparse manifold constraint (DSC) optimization LDA method, called DSCLDA, which introduces $L_{2,0}$-norm regularization for local sparse feature representation and manifold regularization for global feature constraints. By iterating the hard threshold operator and transforming the original problem into an approximate non-convex sparse optimization problem, the manifold proximal gradient (ManPG) method is used as a distributed iterative solution. Each step of the algorithm has an explicit solution. Simulation experiments have verified the correctness and effectiveness of this method. Compared with several advanced sparse linear discriminant analysis methods, this method effectively improves the average classification accuracy by at least $0.90\%$.

1. Introduction

An ultra-high-definition video image processing system relies on the ability to detect information from multiple targets and over long distances [1,2]. However, the current chip computing power cannot support complex computational imaging processing methods, resulting in the inability to meet the real-time requirements of the system, and the processing performance has not yet reached the demand for high-dimensional information detection and perception. Therefore, it is necessary to quickly and adaptively process high-dimensional video images. Nowadays, complex image data are inherently redundant and non-Gaussian, leading to the unstable performance of traditional methods such as principal component analysis (PCA), linear discriminant analysis (LDA) [3,4], Fisher discriminant analysis (FDA) [5], orthogonal linear discriminant analysis (OLDA) [6], and uncorrelated linear discriminant analysis (ULDA) [7], which has affected the actual video processing performance. Therefore, it is urgent to explore how to utilize high-dimensional spatial data, establish new sparse discrimination models, and design effective optimization schemes to improve existing detection and classification strategies.

From the perspective of data analysis, the key to processing and analyzing high-dimensional data lies in dimensionality reduction and feature extraction, with a focus on sparsity [8,9]. As an emerging optimization branch, sparse constraints have attracted much attention due to their ability to break through traditional Shannon sampling and achieve efficient transmission. Nowadays, sparsity constraints have been widely used in pattern recognition image processing, and its applicability in many other fields has been recognized [10,11]. Sparse constraints refer to the majority of elements being zero. For high-dimensional data, it is necessary to consider sparsity, such as through sparse linear discriminant analysis (SLDA) [12], sparse uncorrelated linear discriminant analysis (SULDA) [13], robust sparse linear discriminant analysis (RSLDA) [14], intra-class and inter-class kernel constraints (IIKCs) [15], hypergraph Laplacian-based semi-supervised discriminant analysis (HSDAFS) [16], adaptive and fuzzy locality discriminant analysis (AFLDA) [17], etc. Compared with traditional LDA methods, sparse discriminant analysis greatly improves the identification ability of the system. However, sparse discriminant analysis methods usually replace the $L_1$-norm with the $L_0$-norm to obtain convex optimization problems. Obviously, the $L_0$-norm can select the most representative feature variables and optimize faster than $L_1$-norm-constrained optimization. Examples include a sparse signal recovery framework based on segmented threshold $L_0$ gradient approximation [18], image non-negative matrix factorization with alternating smooth $L_0$-norm constraints [19], and sparse feature selection based on fast embedding spectral analysis [20]. The above methods also prove that the $L_0$-norm does indeed have better feature vector selection and faster optimization algorithm capabilities.

From the perspective of feature extraction, sparse analysis-based methods have significant data analysis capabilities but can not reveal potential causal relationships between variables during the analysis process [21,22,23]. To address this problem, manifold learning can be introduced to learn local features of potential information in high-dimensional space. To characterize such data, a practicable solution is to map the linear inseparable features in high-dimensional space to a low-dimensional nonlinear feature space, such as through robust sparse manifold discriminative analysis (RSMDA) [24], which captures both global and local geometric information through manifold learning. Zhang et al. [25] proposed a coupled discriminative manifold alignment (CDMA) method, which focuses on aligning the manifold structures of a low resolution (LR) and high resolution (HR) in a common feature subspace. In order to use manifold learning methods, many optimization schemes have been proposed, such as the projection algorithm [26], precise penalty algorithm [27], augmented Lagrangian algorithm [28], iterative hard threshold [29], Newton hard threshold pursuit [30], etc. In addition, in the field of video image processing, the problem of high-dimensional pixel videos essentially requires a minimization optimization with Stiefel manifolds, including the partial least squares [31], principal component analysis [32], and canonical correlation analysis [33]. Manifold-constrained optimization can even be seen frequently in reinforcement learning [34] and federated learning [35].

The optimization methods mentioned above mostly focus on a single constraint. Currently, there is limited research on problems that consider both manifold constraints and sparse constraints. On the one hand, this is because both constraints are non-convex, non-smooth, and even NP-hard, making joint algorithm design difficult. On the other hand, joint constraints require two constraints to share the same variable, making theoretical analysis more difficult. To solve these difficulties, this paper proposes a new distributed sparse manifold-constrained optimization algorithm and explores effective numerical solutions. The proposed joint constraints are introduced to LDA, and the novel method is called distributed sparse manifold-constrained linear discriminant analysis (DSCLDA). The proposed method first divides the process monitoring data into multiple data nodes and performs distributed parallel operations simultaneously. Afterward, a $L_{2,0}$-norm sparse constraint is constructed to regulate local features and preserve the local structure of variables. In addition, by using manifold constraints on global variables, the proposed method can capture the causal correlation and reduce the data structure loss during the projection. By using the manifold proximal gradient (ManPG) to combine local and global variables, sparse constraints and manifold constraints are incorporated into the calculation process during optimization, and explicit solutions for each variable are obtained. The contributions of the proposed method can be presented as follows:

• This paper proposes a novel distributed sparse manifold-constrained linear discriminant analysis (DSCLDA) method, which introduces sparse and manifold constraints to maintain the local and global structure.
• We designed an effective solution scheme that combines local and global variables using the manifold proximal gradient (ManPG) to obtain explicit solutions for each subproblem.
• We conducted a series of experiments on several public datasets to verify the effectiveness of the proposed method and discuss the convergence and feature distribution.

The rest of this paper is organized as follows. Section 2 introduces the notations and related works. Section 3 details the problem of the proposed method and the corresponding optimization algorithm. Section 4 evaluates and discusses the performance of the proposed method. Section 5 concludes this paper.

2. Notations and Preliminaries

2.1. Notations

For convenience, we will define some symbols required for this section. For the matrix $X\in {\mathbb{R}}^{n\times p}$, $X_i$ is represented as the ith row, and $X_{ij}$ is represented as an element of the ith row and jth column. $O_{n\times p}$ is written as an all-zero matrix in $n\times p$, $I_{n\times p}$ represents the identity matrix with the dimensions $n\times p$, and $I_p$ represents the identity matrix with the dimensions $p\times p$. $X^{\top }$ represents the transpose of X, and $\mathrm{vec}\left(X\right)$ represents the vectorization of X. For set $\mathrm{T}$, $\overline{\mathrm{T}}$ is used to represent the complement of T. In addition, for matrices $X,Y\in {\mathbb{R}}^{n\times p}$, the inner product is defined as $⟨X,Y⟩=\mathrm{tr}\left(X^{\top }Y\right)={\sum }_{i=1}^n{\sum }_{i=1}^p{X}_{ij}{Y}_{ij}$, where $\mathrm{tr}(·)$ represents the trace of the matrix.

2.2. Preliminaries

LDA, as a supervised learning method, can use prior experience of categories in the dimensionality reduction process, while unsupervised learning cannot use prior experience of categories. Compared with other methods, a feature of LDA is to learn discriminative projections by maximizing the inter-class distance while minimizing the intra-class distance, thereby achieving a more effective dimensionality reduction ability. Define the inter-class distance matrix $S_b$ and intra-class distance matrix $S_w$ for the training samples, and these two matrices can be defined as

$$S_b={\displaystyle \frac{1}{n}}\sum _{i=1}^c{n}_i({\overline{x}}_i-\overline{x}){({\overline{x}}_i-\overline{x})}^{\top },$$

(1)

$$S_w={\displaystyle \frac{1}{n}}\sum _{i=1}^c\sum _{j=1}^{n_i}({\overline{x}}_{ij}-{\overline{x}}_i){({\overline{x}}_{ij}-{\overline{x}}_i)}^{\top }.$$

(2)

LDA attempts to find a suitable projection direction that minimizes intra-class dispersion and maximizes inter-class dispersion after projection. This search process can be expressed as follows:

$$X=\underset{X^{\top }X=I}{argmax}{\displaystyle \frac{\mathrm{Tr}\left(X^{\top }{S}_{b}X\right)}{\mathrm{Tr}\left(X^{\top }{S}_{w}X\right)}}.$$

(3)

To avoid the distortion of $S_w$, problem (3) can also be extended in the following form:

$$\begin{array}{cc}\hfill \underset{X}{\min }& \mathrm{Tr}\left(X^{\top }(S_w-\mu S_b)X\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& X^{\top }X=I.\hfill \end{array}$$

(4)

However, LDA still has some shortcomings. For example, LDA can only reduce the dimensionality of data with a category of k to $k-1$ at most. Therefore, LDA cannot be used when reducing the dimensionality below $k-1$. In addition, if the original sample size is too small, the dimensionality reduction results of LDA are prone to overfitting. Therefore, a common modification is to add sparse constraints to LDA, commonly known as SLDA. In common SLDA methods, the $L_1$-norm is applied in LDA to induce sparsity, which can remove redundant features from the data and improve the performance of video image processing. The formulation of SLDA, which introduces the $L_{2,1}$-norm, is expressed as follows:

$$\begin{array}{cc}\hfill & \underset{X}{\min }\mathrm{Tr}\left(X^{\top }(S_w-\mu S_b)X\right)+\lambda {\parallel X\parallel }_{2,1}\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.X^{\top }X=I.\hfill \end{array}$$

(5)

To effectively eliminate noise and outliers in SLDA and improve robustness in discriminant analysis, reference [14] proposed RSLDA, which is expressed in the form of

$$\begin{array}{cc}\hfill & \underset{P,X,E}{\min }\mathrm{Tr}\left(X^{\top }(S_w-\mu S_b)X\right)+{\lambda }_1{\parallel X\parallel }_{2,1}+{\lambda }_2{\parallel E\parallel }_1\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.R=P{X}^{\top }R+E,P^{\top }P=I,\hfill \end{array}$$

(6)

where ${\parallel ·\parallel }_1$ is the $L_1$-norm. By selecting different parameters of ${\lambda }_1$ and ${\lambda }_2$, RSLDA can select important features and effectively eliminate noise and outliers, thereby achieving excellent performance in the field of image classification.

Another method to improve the performance of SLDA is to incorporate manifold constraints into the optimization problem, such as in the RSMDA method from reference [24], which is represented as

$$\begin{array}{cc}\hfill & \underset{P,X,E}{\min }\mathrm{Tr}\left(X^{\top }(S_w-\mu S_b)X\right)+\mathrm{Tr}\left(Q^{\top }{X}^{\top }(S_w-\mu S_b)QX\right)+{\lambda }_1{\parallel X\parallel }_{2,1}+{\lambda }_2{\parallel E\parallel }_1\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.Q=P{X}^{\top }Q+E,P^{\top }P=I.\hfill \end{array}$$

(7)

Inspired by the above methods, this paper proposes an LDA variant that utilizes joint sparsity and manifold constraints. The specific optimization problem will be described in detail in Section 3.

3. Methodology

3.1. Optimization Problem

In this paper, for the random matrix X, the proposed distributed sparse manifold constraints can be expressed as the following problem:

$$\begin{array}{cc}\hfill & \underset{X}{\min }\sum _{i=1}^l{f}_i\left(X\right)+\lambda g\left(X\right),\hfill \\ \hfill & {\mathrm{s}.\mathrm{t}.\parallel X\parallel }_{2,0}\le s,X^{T}X=I_p,\hfill \end{array}$$

(8)

where l represents the total number of distributed representations of X. Distributed sparse manifold constraints can fully utilize the spatial information of the current extended variables, further improving the interpretability of variables from the process monitoring data. Therefore, combined with regular LDA, distributed sparse manifold linear discriminant analysis (DSCLDA) is proposed, which can fully utilize the local and global information of process monitoring observations and take into account both causal and structural relationships between variables.

In this model, $f_i\in {\mathbb{R}}^{n\times p}\to \mathbb{R}(i=1,2,\dots ,l)$ is the given Lipschitz locally continuous function, and $g\in {\mathbb{R}}^{n\times p}\to$ is the given global function. ${\parallel X\parallel }_{2,0}\le s$ is introduced as the sparse constraint, and $X^{T}X=I_p$ is used as the manifold constraint. Substituting problem (4) into the distributed sparse constraint yields

$$\begin{array}{cc}\hfill & \underset{X}{\min }\sum _{i=1}^l{f}_i\left(X\right)+\lambda g\left(X\right),\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.X=A+{E,\parallel X\parallel }_{2,0}\le s,X^{T}X=I_p.\hfill \end{array}$$

(9)

3.2. Optimization Algorithm

To obtain an effective algorithm, the distributed variable $X_i$ and the global variable Y are introduced to transform problem (8) into

$$\begin{array}{cc}\hfill & \underset{X_i}{\min }\sum _{i=1}^l{f}_i\left(X_i\right)+\lambda g\left(Y\right),\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.X_i=A_i+E_i,\parallel X_i{\parallel }_{2,0}\le s,Y^{T}Y=I_p,\hfill \end{array}$$

(10)

where $X_i$ represents the variables of the ith distribution. In problem (10), the sparse constraint only includes the local variable $X_i$, and the manifold constraint only includes the global variable Y. Therefore, further consideration can be given to the optimization problem of the following penalty function:

$$\begin{array}{cc}\hfill & \underset{X_i,Y}{\min }\sum _{i=1}^l{f}_i\left(X_i\right)+\lambda g\left(Y\right)+\sum _{i=1}^l{\mu }_i\parallel X_i-Y{\parallel }_F^2,\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.X_i=A_i+E_i,\parallel X_i{\parallel }_{2,0}\le s,Y^{T}Y=I_p,\hfill \end{array}$$

(11)

in which ${\mu }_i$ is the penalty parameter corresponding to each branch.

3.2.1. Updating $X_i$

Problem (30) is an NP-hard problem, and there is no explicit solution. Inspired by the Newton hard threshold tracking method, the proposed optimization algorithm extends it to matrices. Assuming the objective function is $h_i\left(X_i\right)$, then the gradient of this function is represented as

$$\begin{array}{c}\hfill \nabla h_i\left(X_i\right)=\nabla f_i\left(X_i\right)+2{\mu }_i(X_i-Y).\end{array}$$

(12)

The Hessian expression of problem (12) can be written as

$$\begin{array}{c}\hfill {\nabla }^2{h}_i\left(X_i\right)=\nabla f_i\left(X_i\right)+2{\mu }_i{I}_{np}.\end{array}$$

(13)

If $X_i\in {\mathcal{P}}_{ʃ}(X_i-{\alpha }_i\nabla h_i\left(X_i\right))$ is satisfied (where ${\alpha }_i>0$ is the step size parameter), then $X_i$ can be considered as the stable point for problem (29). Let ${\mathbb{T}}_S(X_i,{\alpha }_i)$ represent the set of indicators for the first s rows of $X_i-{\alpha }_i\nabla h_i\left(X_i\right)$ under $L_2$-norm constraints; then, for any $T_i\in {\mathbb{T}}_s(X_i,{\alpha }_i)$, this satisfies a nonlinear relationship, which is written as

$$\begin{array}{c}\hfill H_i(X_i,T_i)=\left(\begin{array}{c}{(\nabla h_i\left(X_i\right))}_{T_i}\\ {\left(X_i\right)}_{{\overline{T}}_i}\end{array}\right)=0,\end{array}$$

(14)

in which ${(\nabla h_i\left(X_i\right))}_{T_i}$ represents the submatrix in $\nabla h_i\left(X_i\right)$, and $T_i$ is the corresponding indicator set. ${\left(X_i\right)}_{{\overline{T}}_i}\in {\mathbb{R}}^{n-a}\times p$ indicates the submatrix in $X_i$ with ${\overline{T}}_i$ as the indicator set. The gradient of $H_i(X_i,T)$ in $X_i$ can be expressed as

$$\begin{array}{c}\hfill \nabla H_i(X_i,T)=\left[\begin{array}{cc}{\left({\nabla }^2{h}_i\left(X_i\right)\right)}_{T_i{T}_i}& {\left({\nabla }^2{h}_i\left(X_i\right)\right)}_{T_i{\overline{T}}_i}\\ O_{(n-s)p\times sp}& I_{(n-s)p}\end{array}\right],\end{array}$$

(15)

where ${\left({\nabla }^2{h}_i\left(X_i\right)\right)}_{T_i{T}_i}\in {\mathbb{R}}^{sp\times sp}$ represents the Hessian submatrix with the indicator set $T_i{T}_i$. Define

$$\begin{array}{c}\hfill X_i\left(\alpha \right)=\left(\begin{array}{c}{\left(X_i\right)}_{T_i}+\alpha {\left(D\right)}_{T_i}\\ {\left(O\right)}_{T_i}\end{array}\right),\end{array}$$

(16)

where D represents the descending direction. The minimum $X_i$ can be obtained using a sparse proximal gradient (SpaPG). The descending direction D is obtained from

$$\begin{array}{c}\hfill \nabla H_i(X_i,T)\mathrm{vec}\left(D\right)=-\mathrm{vec}\left(H_i(X_i,T)\right),\end{array}$$

(17)

Then, the $k+1$th $X_i$, $X_i^{k+1}$, should be represented as

$$\begin{array}{c}\hfill X_i^{k+1}=X_i^k\left({\alpha }_i^k\right),\end{array}$$

(18)

in which ${\alpha }_i^k={\rho }^{\tau }$, while $\tau$ is the smallest positive integer that satisfies the following equation, written as

$$\begin{array}{c}\hfill h_i\left(X_i\left({\rho }^{\tau }\right)\right)\le h_i\left(X_i\right)+\sigma {\rho }^{\tau }⟨\nabla h_i\left(X_i\right),D^k⟩\end{array}$$

(19)

3.2.2. Updating Y

Set $\mathcal{M}=\left\{T\right|Y^{T}Y=I_p\}$; then, the tangent space of manifold $\mathcal{M}$ at Y is expressed as ${\mathcal{T}}_Y\mathcal{M}=\left\{Z\right|Z^{T}Y+Y^{T}Z=0\}$. Assuming the objective function is $\varphi \left(Y\right)$, it has the following approximation function:

$$\begin{array}{c}\hfill \varphi \left(Y^k\right)+⟨\nabla \varphi \left(Y^k\right),Y-Y^k⟩+{\displaystyle \frac{1}{2t}}\parallel Y-Y^k{\parallel }_F^2,\end{array}$$

(20)

where ${\displaystyle \frac{1}{L}}\ge t>0$ is a parameter. To obtain the descending direction D, define

$$\begin{array}{cc}\hfill \underset{D\in {\mathbb{R}}^{R\times P}}{\min }& ⟨\nabla \varphi \left(Y^k\right),D⟩+{\displaystyle \frac{1}{2t}}{\parallel D\parallel }_F^2\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& D\in {\mathcal{T}}_{Y^k}\mathcal{M},\hfill \end{array}$$

(21)

Based on the definition of ${\mathcal{T}}_{Y^k}\mathcal{M}$, set $D^T{Y}^k+Y^{kT}D=0$, and Equation (21) can be represented as

$$\begin{array}{cc}\hfill \underset{D\in {\mathbb{R}}^{R\times P}}{\min }& ⟨\nabla \varphi \left(Y^k\right),D⟩+{\displaystyle \frac{1}{2t}}{\parallel D\parallel }_F^2\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& D^T{Y}^k+Y^{kT}D=0,\hfill \end{array}$$

(22)

Based on Equation (22), the Lagrange function can be obtained, which is written as

$$\begin{array}{cc}\hfill \mathcal{L}(D,\mathrm{\Lambda })& =⟨\nabla \varphi \left(Y^k\right),D⟩+{\displaystyle \frac{1}{2t}}{\parallel D\parallel }_F^2\hfill \\ \hfill & -⟨\mathrm{\Lambda },D^T{Y}^k+Y^{kT}D⟩,\hfill \end{array}$$

(23)

in which $\mathrm{\Lambda }\in {\mathbb{R}}^{n\times p}$ is the Lagrange multiplier. Then, the corresponding Karush–Kuhn–Tucker (KKT) system for the Lagrangian function above is represented as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{c}0\in {\partial }_D\mathcal{L}(D,\mathrm{\Lambda }),\hfill \\ 0=D^T{Y}^k+Y^{kT}D\hfill \end{array}\right.\end{array}\end{array}$$

(24)

By synthesizing Equation (24), the optimization problem for $\{D,\mathrm{\Lambda },Y\}$ can be obtained, which is written as

$$\begin{array}{c}\hfill D{\left(\mathrm{\Lambda }\right)}^T{Y}^k+Y^{kT}D\left(\mathrm{\Lambda }\right)=0.\end{array}$$

(25)

Equation (25) can be solved using the manifold proximal gradient (ManPG) algorithm, and the $k+1$th Y can be represented as

$$\begin{array}{c}\hfill Y^{k+1}={\mathcal{R}}_{Y^k}\left({\gamma }^k{D}^k\right),\end{array}$$

(26)

in which the mapping ${\mathcal{R}}_Y:{\mathcal{T}}_Y\mathcal{M}\to \mathcal{M}$ represents Retraction. ${\mathcal{R}}_Y$ maps the vectors in the tangent space to the manifold, allowing the problem to maintain orthogonality during the optimization. In Equation (26), ${\gamma }^k=\gamma {\eta }^q$, and q is the smallest positive integer that satisfies the following equation, expressed as

$$\begin{array}{c}\hfill \varphi \left(Y^{k+1}\right)\le \varphi \left(Y^k\right)-{\displaystyle \frac{\gamma {\eta }^q}{2t}}\parallel D^k{\parallel }_F^2.\end{array}$$

(27)

3.3. Convergence Analysis

According to the updates of $X_i$ and Y, the optimization algorithm of Equation (11) can be expressed as Algorithm 1. In addition, according to the literature [36], if $(X_i^{*},Y^{*})$ satisfies

$$\begin{array}{c}\hfill \left\{\begin{array}{c}0\in {\mathcal{P}}_{\mathcal{S}}(\nabla f_i\left(X_i^{*}\right)+{\mu }_i(X_i^{*}-Y^{*})),\hfill \\ 0\in {\mathcal{P}}_{\mathcal{M}}(\lambda \nabla g\left(Y^{*}\right)-{\mu }_i(X^{*}-Y^{*})),\hfill \end{array}\right.\end{array}$$

(28)

then $(X_i^{*},Y^{*})$ can be considered as the stable point of Equation (11). The experimental verification of the convergence analysis can be found in Section 4.5.

3.4. Complexity Analysis

To verify how distributed sparse constraints can enhance the performance of existing methods, this section compares the complexity and computational cost of the proposed method with that of the baseline LDA method. For Equation (9), given the original data with a dimensionality of d and n samples, the computational complexity of the objective function is $O\left(n{d}^2\right)$. The sparse constraint ${\parallel X\parallel }_{2,0}\le s$, which checks the number of non-zero elements, has a complexity of $O\left(nd\right)$. The manifold constraint $X^{T}X=I_p$ implies that X is an orthogonal matrix, which, enforcing orthogonality through methods such as QR decomposition, has a complexity of $O\left(n{d}^2\right)$. Therefore, the overall complexity of the proposed method is $O\left(n{d}^2\right)$. In contrast, the computational complexity of traditional LDA-based methods is primarily determined by calculating the within-class scatter matrix $S_w$, the between-class scatter matrix $S_b$, and solving the generalized eigenvalue problem. The complexity of computing $S_w$ is $O\left(n{d}^2\right)$, while that of computing $S_b$ is $O\left(d^2\right)$, due to the calculation based on class means and the global mean. The complexity of solving the eigenvalues and eigenvectors of $S_w^{-1}{S}_b$ is $O\left(d^3\right)$, which is the most time-consuming part of LDA. The overall complexity of LDA is $O(n{d}^2+d^3)$. The proposed distributed sparse constraint method demonstrates superior computational efficiency over traditional LDA methods by reducing the overall complexity from $O(n{d}^2+d^3)$ to $O\left(n{d}^2\right)$ through the enforcement of sparsity and orthogonality constraints, thereby eliminating the most time-consuming generalized eigenvalue problem in LDA.

Algorithm 1 Optimization algorithm for (11)

Input: Data X, parameters s,l,$\lambda$,${\mu }_i>0$. Initialize: Data $Y^0$, parameter $k=0$. Output: Data Y. While not converged do 1: According to Algorithm 2, update $X_i^{k+1}$ by $$\begin{gathered} \begin{array}{c}\underset{X_i}{\min }{f}_i\left(X_i\right)+{\mu }_i\parallel X_i-Y{\parallel }_F^2, \\ \mathrm{s}.\mathrm{t}. X_i=A_i+E_i,\parallel X_i{\parallel }_{2,0}\le s.\hfill \end{array} \end{gathered}$$ (29) 2: According to Algorithm 3, update $Y^{k+1}$ by $$\begin{gathered} \begin{array}{c}\underset{Y}{\min } \lambda g\left(Y\right)+\sum _{i=1}^l{\mu }_i\parallel X_i-Y{\parallel }_F^2, \\ \mathrm{s}.\mathrm{t}. Y^{T}Y=I_p.\hfill \end{array} \end{gathered}$$ (30) 3: If the process meets the shutdown criteria $\parallel X_i{\parallel }_{2,0}\le s$, stop; Otherwise, let $k=k+1$ and return to Step 1. End while

Algorithm 2 Optimization algorithm for (12)

Input: Data X, parameters $\mu ,\alpha >0$, $\rho \in (0,1)$, $\sigma \in (0,1/2)$. Initialize: $X_i^0$, ${\mathrm{T}}_i^0\in {\mathbb{T}}_{\mathcal{S}}(X_i^0,\alpha )$, when $k=0$. Output:$X_i^k$ While not converged do 1: Obtain the nonlinear relationship $H_i(X_i^k,{\mathrm{T}}^k)$ and the gradient $\nabla H_i(X_i^k,{\mathrm{T}}_i^k)$, according to (14), (15); 2: Obtain the descent direction $D^k$, according to (17); 3: Update the local variable $X_i^{k+1}$, according to (9); 4: If the process meets the shutdown criteria, stop; Otherwise, let $k=k+1$, update ${\mathrm{T}}_i^k\in {\mathbb{T}}_{\mathcal{S}}(X_i^k,{\alpha }^k)$, and return to Step 1. End while

Algorithm 3 Optimization algorithm for (20)

Input: Parameters $\gamma ,t>0,\eta \in (0,1)$. Initialize: $Y^0,{\mathrm{\Lambda }}^0$, $k=0$. Output: $X^k,Y^k$. While not converged do 1: Obtain the descent direction D, according to (25); 2: Update the global variable $Y^{k+1}$, according to (26), 3: If the process meets the shutdown criteria, stop; Otherwise, let $k=k+1$ and return to Step 1. End while

4. Simulation Studies

In the experiments, DSCLDA was compared with traditional LDA and six LDA variants, including AFLDA [17], ERSLDA [37], RSLDA+IIKC [15], RSMDA [24], RSLDA [14], SULDA [13], and SLDA [12]. The optimization problem and constraint of each method are shown in Table 1. The datasets used in the experiments in this paper are shown in Table 2, and examples of each dataset are shown in Figure 1. In this experiment, a self-built vehicle dataset, called the CAR_image dataset, was introduced.

Table 1. Information on all comparison methods used in this experiment. The bold method is the proposed method.

Method	Optimization Problem	Constraint
LDA	${\min }_X\mathrm{tr}\left(X^{\top }(S_w-\mu S_b)X\right)$	$\mathrm{s}.\mathrm{t}.X^{\top }X=I$
SLDA	${\min }_X\mathrm{Tr}\left(X\top (S_w-\mu S_b)X\right)+\lambda {\parallel X\parallel }_{2,1}$	$\mathrm{s}.\mathrm{t}.X\top X=I$
SULDA	$\min G$	$\mathrm{s}.\mathrm{t}.U_1^{T}G={\sum }_t^{-1}{P}_{1}Z,Z^{T}Z=I$
RSLDA	${\min }_{P,X,E}\mathrm{Tr}\left(X^{\top }(S_w-\mu S_b)X\right)$	$\mathrm{s}.\mathrm{t}.R=P{X}^{\top }R+E,P^{\top }P=I$
	$+{\lambda }_1{\parallel X\parallel }_{2,1}+{\lambda }_2{\parallel E\parallel }_1$
RSMDA	${\min }_{P,X,E}\mathrm{Tr}\left(X^{\top }(S_w-\mu S_b)X\right)$	$\mathrm{s}.\mathrm{t}.R=P{X}^{\top }R+E,P^{\top }P=I$
	$+\mathrm{Tr}\left(X^{\top }{R}^{\top }(L_w-L_b)RX\right)$
	$+{\lambda }_1{\parallel X\parallel }_{2,1}+{\lambda }_2{\parallel E\parallel }_1$
RSLDA+IIKC	${\min }_{P,X,E}\mathrm{Tr}\left(X^{\top }(S_w-\mu S_b)X\right)$	$\mathrm{s}.\mathrm{t}.R=P{X}^{\top }R+E,P^{\top }P=I$
	$+{\lambda }_1{\parallel X\parallel }_{2,1}+{\lambda }_2{\parallel E\parallel }_1$
	$+\alpha \mathrm{Tr}\left(X^{\top }(S_w^{\prime }-\mu S_b^{\prime })X\right)$
ERSLDA	${\min }_{P,X,E,N}\mathrm{Tr}\left(X^{\top }(S_w-\mu S_b)X\right)$	$\mathrm{s}.\mathrm{t}.R=P{X}^{\top }R+E+N,P^{\top }P=I$
	$+{\lambda }_1{\parallel X\parallel }_{2,p}^p+{\lambda }_2{\parallel E\parallel }_p^p+\eta {\parallel N\parallel }_F^2$
DSCLDA	${\min }_X{\sum }_{i=1}^d\mathrm{tr}\left(X^{\top }(S_i^w-\tau S_i^b)X\right)$	${\mathrm{s}.\mathrm{t}.\parallel \mathrm{X}\parallel }_{2,0}\le \mathrm{s},{\mathrm{X}}^{\top }\mathrm{X}={\mathrm{I}}_{\mathrm{p}}$

Table 2. Information related to the dataset used in this experiment.

Dataset	Image Types	Images	Color Type	Original Resolution
Mnist [38]	10	60,000	Gray	$28\times 28$
Hand Gesture Recognition [39]	10	20,000	Gray	$240\times 640$
Coil20 [40]	20	1440	Gray	$128\times 128$
NEU surface defects [41]	6	1200	Gray	$32\times 32$
Car_image	10	200	RGB	$800\times 600$ to $5000\times 3000$
Caltech-101 [42]	101	9146	RGB and gray	About $300\times 200$

Figure 1. Some image examples from the datasets used in the experiment. (a) Mnist, (b) Hand Gesture Recognition, (c) COIL20, (d) NEU surface defects, (e) Car_image, (f) Caltech-101.

4.1. Experiment Settings

Due to the fact that the datasets were divided into D parts as data nodes for distributed computing during the experiment, D was added before all method names to indicate distributed performance, such as DERSLDA and DRSLDA. In the simulation verification, each method was executed 10 times, with different random samples selected from the same dataset for each run; then, the average classification accuracy was calculated. To improve computational efficiency, all datasets were preconverted into grayscale images. In addition, to improve computational efficiency and achieve better classification accuracy, this experiment used PCA to perform dimensionality reduction on all image datasets, retaining 95% of the original data information. Furthermore, due to the large and inconsistent image sizes in the Car_image dataset, the unified resolution of the images in this dataset was $64\times 128$.

In selecting experimental parameters, the selection of the parameters $\lambda$ and $\mu$ was carried out through a ten-fold cross-validation method based on the content and size of each dataset. The range of the parameters $\lambda$ and $\mu$ is denoted as ${10}^{-5},{10}^{-4},{10}^{-3},{10}^{-2},{10}^{-1},1,{10}^1,$${10}^2,{10}^3,{10}^4$, and ${10}^5$. Prior to numerical validation, a strategy of fixing the value of $\lambda$ while varying $\mu$ was employed to ascertain the corresponding accuracy for each configuration, serving as a basis for evaluation. The experimental results on the COIL20 image dataset are shown in Figure 2. Based on the experimental results, it can be determined that the selection of the parameters $\lambda$ and $\mu$ should be within the range of ${10}^{-5}$, ${10}^{-4}$, ${10}^{-3}$, and ${10}^{-2}$ to achieve better image processing performance. Specifically, for the COIL20 image dataset, the most suitable parameter combination was identified as ${10}^{-3}$ and ${10}^{-5}$, and a similar method for parameter selection was applied to other datasets under investigation. In addition, the shutdown criterion in this experiment was set so that 100 iterations would be reached or the overall objective function value would be less than ${10}^{-3}$.

Figure 2. Parameter cross-validation on the COIL20 image dataset. The parameters $\lambda$ and $\mu$ are derived from Equation (11) In this figure, green indicates high value and blue indicates low value.

4.2. Experiment Based on Sample Size

This experiment used the k nearest neighbors (KNN) classifier to analyze the classification accuracy of the dimensionality reduction results of various methods. The knn classifier is a supervised machine learning algorithm that assigns a new data point to the class most common among the k nearest neighbors in the feature space, based on a distance metric such as the Euclidean distance. In this experiment, four different sample sizes were randomly selected for each dataset as the training set, and the remaining samples were used as the testing set. The classification experiment results under different sample sizes are shown in Table 3, where the highest-performing results are highlighted in bold. The simple image datasets used in the experiment, including the Mnist dataset, Hand Gesture Recognition dataset, and COIL20 dataset, have simple content, a monotonous background, and obvious features. Therefore, each method could achieve better classification performance on the above three datasets. The image features of the NEU surface defect dataset, Car_image dataset, and Caltech-101 dataset are relatively complex or have a high proportion of the background, resulting in relatively low classification accuracy for each model on these datasets. However, the experimental results show that the DSCLDA model still had improvements in the classification performance compared to other methods on these datasets.

Table 3. The classification accuracy obtained on six datasets. The bold value represents the highest value of the column.

Methods	Mnist				Hand Gesture Recognition				COIL20
	10	50	100	200	4	5	6	7	3	6	9	12
DLDA	75.20	85.59	84.02	83.78	75.95	80.38	87.81	89.35	65.29	74.97	81.58	80.40
DSLDA	80.42	85.64	84.85	84.46	81.50	83.69	90.69	92.23	70.84	78.28	84.46	84.02
DSULDA	87.78	87.34	86.37	93.38	84.76	88.62	90.11	91.65	74.10	83.21	83.88	86.37
DRSLDA	84.03	85.85	88.40	96.58	87.40	89.73	88.98	90.52	76.74	84.32	82.75	86.75
DRSMDA	83.62	86.56	90.77	97.51	87.62	85.49	90.85	92.39	76.96	80.08	84.62	87.30
DRSLDA+IIKC	73.27	83.85	85.92	96.94	90.30	90.86	92.89	94.43	79.64	85.45	86.66	86.94
DERSLDA	86.77	90.10	92.06	97.62	88.12	90.26	89.81	91.35	77.46	84.85	83.58	88.73
DAFLDA	85.91	89.11	90.84	96.34	87.93	88.28	88.76	90.67	76.24	82.17	83.36	85.23
DSCLDA	86.92	90.20	92.94	97.82	90.37	91.39	93.48	95.02	79.71	85.98	87.25	90.95
Methods	NEU Surface Defects				Car_IMAGE				Caltech-101
	25	50	75	100	10	15	20	25	10	15	20	25
DLDA	41.27	38.87	43.26	48.08	20.64	25.00	34.50	44.77	51.54	58.16	62.82	65.21
DSLDA	43.03	44.93	48.15	50.53	37.03	39.93	42.15	42.08	55.56	67.60	70.02	74.32
DSULDA	42.18	48.73	52.45	54.82	37.18	42.73	47.45	48.82	67.89	77.09	80.69	86.11
DRSLDA	42.73	46.20	50.89	56.50	36.73	40.20	44.89	50.50	69.22	83.60	83.70	87.04
DRSMDA	52.18	53.33	57.85	60.83	47.18	47.33	51.85	54.83	71.86	83.33	84.51	86.25
DRSLDA+IIKC	52.30	57.80	61.70	64.92	46.30	51.80	55.70	59.92	74.45	87.52	90.32	91.02
DERSLDA	47.52	54.53	55.85	62.92	42.52	49.53	50.85	56.92	73.20	85.10	85.20	88.25
DAFLDA	45.64	50.91	52.68	56.12	40.58	46.37	48.25	53.76	70.45	83.68	84.71	85.99
DSCLDA	54.55	57.80	62.22	65.58	50.32	53.57	57.99	61.35	74.79	88.28	90.98	91.47

Compared with other methods, DSCLDA improved by at least $0.51\%$ on the Mnist dataset; improved by at least $0.44\%$ on the Hand Gesture Recognition dataset; improved by at least $0.85\%$ on the COIL20 image dataset; improved by at least $0.86\%$ on the NEU surface defect dataset; improved by at least $2.16\%$ on the Car_image dataset; and improved by at least $0.55\%$ on the Caltech-101 image dataset. The classification performance of DSCLDA was further improved on two difficult datasets, namely the NEU surface defect dataset and the Car_image dataset. The results can be explained by the fact that the DSCLDA model, which simultaneously extracts features from both global and local structures, can obtain more representative feature data when processing complex images or images with unclear features, thereby achieving better classification performance. The experimental results also demonstrate that DSCLDA divides process monitoring data into multiple data nodes and performs distributed parallel operations, which not only improves computational efficiency but also provides better adaptation to the processing needs of large-scale data.

Compared to other methods, the average classification performance of the proposed DSCLDA method improved by at least $0.90\%$, which proves that the proposed method achieves satisfactory classification performance by introducing joint sparse and manifold constraints. In addition, compared with DRSLDA, DRSMDA, DRSLDA+IIKC, and DERSLDA, the proposed DSCLDA still had a significant improvement, indicating that the proposed method can demonstrate advantages when compared with some of the latest SLDA variants.

4.3. Experiment Based on the Number of Dimensions

In this experiment, $(50,100)$, $(4,6)$, $(4,6)$, $(50,100)$, $(10,20)$, and $(10,20)$ samples were selected as training sets for each type on the six public image datasets, and the remaining samples were used as testing sets with dimensions ranging from 5 to 200. The classification experiment results are shown in Figure 3, Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8. The experimental results indicate that the proposed DSCLDA method achieved relatively better classification performance on the six publicly available datasets mentioned above. From the experimental results, it can be seen that the classification performance of DLDA and DSLDA is very sensitive to the choice of dimensionality. As the dimensionality increases, the classification performance of these two methods may even even decrease. However, the proposed DSCLDA method can still maintain classification accuracy in the presence of dimensional changes. The experimental results demonstrate the flexibility of the proposed method in dimension selection. For the NEU surface defect and Caltech-101 image datasets, the classification performance of DSCLDA did not show significant improvement compared to that of other methods because the features of these two datasets are relatively complex and not clear enough, resulting in similar classification results for the above methods. However, on several other publicly available datasets, the proposed DSCLDA method still performed relatively better in terms of its classification accuracy. DSCLDA normalizes local features by constructing $L_{2,0}$-norm sparse constraints, preserves the local structure of variables, and utilizes manifold constraints to capture causal correlations between global variables, reducing data structure loss during the projection process.

Figure 3. Classification accuracy on the Mnist dataset. (a) Number of samples: 50; (b) number of samples: 100.

Figure 4. Classification accuracy on the Hand Gesture Recognition dataset. (a) Number of samples: 4; (b) number of samples: 6.

Figure 5. Classification accuracy on COIL20 image dataset. (a) Number of samples: 4; (b) number of samples: 6.

Figure 6. Classification accuracy on the NEU surface defect dataset. (a) Number of samples: 50; (b) number of samples: 100.

Figure 7. Classification accuracy on the Car_image dataset. (a) Number of samples: 10; (b) number of samples: 20.

Figure 8. Classification accuracy on the Caltech-101 image dataset. (a) Number of samples: 10; (b) number of samples: 20.

4.4. Experiments with Deep Learning Methods

Deep learning methods, such as Transformer-based feature extraction models, provide new perspectives and powerful tools for feature extraction and dimensionality reduction, which can provide valuable benchmarks. These deep learning methods typically have better feature-learning capabilities and stronger robustness and can achieve excellent performance on large-scale datasets. Therefore, this paper also compares DSCLDA with deep learning-based dimensionality reduction techniques, such as R3D-CNN [43], I3D [44], and Transformer [45], to demonstrate its broader applicability in different scenarios. Through experiments on the Hand Gesture Recognition (HGR) and CIFAR-100 [46] datasets, we validated the advantages of DSCLDA in feature extraction and dimensionality reduction, as well as its competitiveness with deep learning methods. Table 4 demonstrates that the gesture recognition dataset may have certain limitations, such as its number of samples, diversity, and representativeness. On smaller datasets, simpler or more traditional models, such as DSCLDA, may perform better because of their lower complexity. On the other hand, models supported by deep learning methods may be more suitable for handling large and complex datasets, capable of capturing more subtle patterns and relationships.

Table 4. The accuracy of DSCLDA, R3D-CNN, I3D, and Transformer on the HGR and CIFAR-100 datasets. The bold value represents the highest value of the column.

HGR		CIFAR-100
Method	Acc. (%)	Method	Acc. (%)
DSCLDA	90.37	DSCLDA	63.45
R3D-CNN	83.80	R3D-CNN	90.62
I3D	85.70	I3D	94.82
Transformer	87.60	Transformer	95.03

4.5. Convergence Analysis

This section describes how we conducted experimental verification of the convergence analysis proposed in Section 3.3. In the proposed DSCLDA method, the most computationally expensive step is the calculation of the projection matrix X, while the most computationally intensive task is the process of solving the inverse matrix, which significantly affects the computational efficiency of DSCLDA. In this experiment, the computational efficiency of DSCLDA was reflected in the speed of function value reduction and the convergence speed of the classification accuracy. In order to visually demonstrate the convergence of the proposed DSCLDA method, Figure 9 shows the curves of the objective function value and classification accuracy of the functions. As the number of iterations increased, the objective function value of the proposed DSCLDA method rapidly decreased and reached its minimum value, and the classification accuracy also reached its maximum value and converged within 30 iterations. The experimental results validate the fast convergence of DSCLDA.

Figure 9. The relationship between the objective function value, classification accuracy, and the number of iterations. (a) Mnist, (b) Hand Gesture Recognition, (c) COIL20, (d) NEU surface defects, (e) Car_image, (f) Caltech-101.

4.6. t-SNE Comparison

In addition, to further validate the principle and effectiveness of the proposed method, the t-SNE method was utilized to visualize the data distribution before and after projection. The experiment used the top five types of data from the Mnist dataset and randomly selected 100 samples for each type as the training set and the remaining samples as the testing set. The corresponding classification accuracies of each method were 85.85% (DRSLDA), 90.10% (DERSLDA), and 90.20% (DSCLDA), respectively. The experimental results are shown in Figure 10. It can be seen that when not projected, the inter-class and intra-class distributions of the Mnist dataset were not significant. When using the DRSLDA method for projection, it can reduce the distance between types and increase the distance between different types, but DRSLDA cannot fully classify all data, and this classification method is not satisfactory in terms of distribution. With the introduction of sparse constraints, the inter-class distance between different types of data becomes larger, while the intra-class spacing becomes smaller. In the t-SNE distribution of the proposed DERSLDA method, the intra-class spacing is relatively small, but the inter-class distance is not large enough, so there is still a possibility of data confusion during the classification process. In the t-SNE distribution of the proposed DSCLDA method, the distance between different types is the largest, and the distance between types is the smallest, such as between type 1 and type 2, which is more conducive to determining the data category during the classification process. The experimental results show that the proposed method has relatively better classification performance.

Figure 10. The data distribution displayed using the t-SNE method. The images correspond to (a) the local data of the original Mnist dataset; (b) the distribution of the corresponding data after DRSLDA projection; (c) the distribution of the corresponding data after DERSLDA projection; (d) and the corresponding data distribution projected through DSCLDA.

5. Conclusions

In this paper, we constructed a novel distributed sparse manifold constraint and a novel LDA variant, called DSCLDA. The proposed method trains discriminative projections by introducing manifold constraints and $L_{2,0}$-norm sparse constraints, which can obtain the most discriminative features for process monitoring. In addition, in this paper, we designed and developed a novel manifold proximal gradient algorithm to handle the proposed optimization model, while distributed parallel computing could significantly improve computational efficiency. The advantages of DSCs and DSCLDA have been demonstrated through numerical experiments on several public datasets. Compared with other existing LDA methods, the proposed DSCLDA method improves the image classification accuracy by at least $0.90\%$ and also has significant advantages in convergence and feature distribution.

However, the proposed method currently exhibits limitations in terms of its image processing efficiency and feature classification accuracy, necessitating integration with deep learning techniques for improvement. In the future, we will attempt to combine the proposed method with deep learning methods to improve the efficiency of image processing and the accuracy of feature classification. Furthermore, deployment on hardware platforms may be constrained by computational complexity and insufficient flexibility, highlighting the need for further optimization to enhance the processing efficiency and applicability. In addition, this method will also be considered for deployment on hardware to improve the efficiency of the method’s processing and the flexibility of the method’s usage.