Ellipsoid-Structured Localized Generalized Eigenvalue Proximal Support Vector Machines

Abstract

The Generalized Eigenvalue Proximal Support Vector Machine (GEPSVM) introduces a novel large-margin classifier that improves upon standard SVMs by constructing a pair of non-parallel hyperplanes derived from a generalized eigenvalue problem. However, the GEPSVM suffers from severe misclassification in the overlapped hyperplane region, known as the underdetermined hyperplane problem (UHP). A localized GEPSVM (LGEPSVM) alleviates this issue by building convex hulls on the hyperplanes for classification, but it still faces notable drawbacks: (1) an inability to integrate both local and global information, (2) a lack of consideration of the data’s statistical characteristics, and (3) high computational and storage costs. To address these limitations, we propose the Ellipsoid-structured Localized GEPSVM (EL-GEPSVM), which extends the GEPSVM by constructing ellipsoid-structured convex hulls under the Mahalanobis metric. This design incorporates statistical data characteristics and enables a classification scheme that simultaneously considers local and global information. Extensive theoretical analyses and experiments demonstrate that the proposed EL-GEPSVM achieves improved effectiveness and efficiency compared with existing methods.

1. Introduction

Based on statistical learning and structural risk minimization principles, the Support Vector Machine (SVM) is widely used in various classification problems. By maximizing the margin between two-class samples and meanwhile minimizing experience error, SVMs are capable of achieving better generalization and performance. Due to the high computation burden and low interpretability in complex data, the standard SVM is not suitable for learning in multimedia applications. Inherited from the standard SVMs, the Proximal Support Vector Machine (PSVM) uses two parallel hyperplanes to fit two-class samples instead of a single decision plane for classification [1]. The PSVM provided the brand new idea of constructing large-margin classifiers, which was a breakthrough in the SVM research area. Furthermore, the GEPSVM [2] introduced a generalized eigenvalue problem as its solution, which made possible flexible and efficient implementation in applications. Various derivatives were proposed recently to improve the performance and generalization in view of the GEPSVM and PSVM, for example, GEPSVM_L1 [3], TWSVM [4], and PCC [5].

However, a self-contained drawback in the GEPSVM is inevitable, where it may misclassify two overlapped hyperplanes (as shown in Figure 1b). We can call this problem an undetermined hyperplane problem (UHP). Obviously, such misclassifications occur extensively in the intersection region of the two planes in Figure 1b. The LGEPSVM utilizes a convex hull to make classifications to deal with the drawback of the GEPSVM. It assigns samples to classes according to the nearest convex hull, which represents the local region of the decision hyperplane. Several algorithms are proposed that take this localization method and combine it with an SVM and GEPSVM, for instance, the LCTSVM [6,7,8]. However, the LGEPSVM cannot obtain results through batch computation, which takes much more time than the GEPSVM. Since the process of LGEPSVM convex hull construction is iterative and its time complexity is $O\left(mn\right)$ (n, m are the size and dimension of the sample), it is time-consuming when classifying large-scale datasets. In this case, constructing a convex hull differently to make improvements with higher efficiency is necessary. This will enhance the discriminability in the overlapped hyperplanes and meanwhile keep the superiority brought by the GEPSVM with the idea of the large-margin classifier.

Figure 1. Illustration of SVM [9] (a), GEPSVM [2] (b), LGEPSVM [10] (c), and the proposed E-LGEPSVM (proposed) (d). $X_1$ and $X_2$ represent samples from two categories. The green plane p is the hyperplane of the SVM. $p_1$ (blue) and $p_2$ (yellow) are hyperplanes supported by $X_1$ and $X_2$, respectively. The red circle is a confusion area where samples are likely to be misclassified.

Localization is an effective methodology to relieve the problems from the original GEPSVM [10]. Compared with the primitive GEPSVM, the localized GEPSVM produces two major advantages for machine training and classification: (1) it relieves the misclassification problem (see Figure 1c) in UHP region, and (2) it is able to be kernelized for classification in nonlinear space. Specifically, the localization is to construct a convex hull based on the hyperplane so that the classification can be performed with a convex set rather than with the whole hyperplane. This makes the decision boundary of a localized GEPSVM sensitive to the samples located in the UHP region of both planes. In other words, this makes the GEPSVM successful when dealing with ambiguous samples which may lead to failure in classification. Although the localization has success in the misclassification problem, it still lacks consideration of the statistical characteristics of data. Also, the construction of data needs to iterate each sample in the training data. Furthermore, samples involved in the convex set require extra storage space for the classification procedure. Both points above make the algorithm less efficient and more expensive when the scale of the dataset increases.

In this paper, we address the abovementioned problems of GEPSVMs: (1) the misclassification in the UHP region, (2) the mis-consideration of both local and global information, and (3) the lack of geometrical interpretation of the classification mechanism with brand new methodologies with theoretical fundamentals. To address these issues, we propose the Ellipsoid-structured Localized Generalized Eigenvalue Proximal Support Vector Machine (EL-GEPSVM), which constructs an ellipsoidal convex hull using the Mahalanobis metric for large-margin classification. The ellipsoidal convex hull alleviates the misclassification issue in the UHP region by reducing the area of the UHP region (see Figure 1d). Since the Mahalanobis metric is a typical non-Euclidean metric used frequently in different fields for model optimization, it can take global information into account while remaining scale-invariant [11]. Several research areas and applications support the importance and effectiveness of the Mahalanobis metric [8,12]. Compared with the Euclidean metric, the Mahalanobis metric adaptively incorporates the variance and correlation among features, allowing the classifier to model data distributions as anisotropic ellipsoids rather than isotropic spheres. This enables the EL-GEPSVM to better capture intra-class covariance and local geometry, thereby improving separability and robustness in complex, high-dimensional spaces. To provide a proper geometrical interpretation of the classification mechanism for the EL-GEPSVM, we proved that an ellipsoidal hyperplane can act both as a decision plane and as a convex hull with the Mahalanobis metric [13]. In this case, the proposed EL-GEPSVM can take advantage of the localization philosophy (realized by the ellipsoidal convex hull) to address the misclassification of the UHP in feature space. Also, since the constructed ellipsoidal hyperplane is taken as the decision plane, both local and global information are simultaneously considered. To deal with the classification problem in nonlinear space, we extend the standard EL-GEPSVM to a kernelized version. We theoretically proved that the EL-GEPSVM is able to be kernelized through the proof of the existence of a dual representation of the Mahalanobis norm. In addition, the EL-GEPSVM has higher efficiency than imposing localization directly on a GEPSVM (e.g., localized GEPSVM [10]). A series of EL-GEPSVM algorithms is developed and theoretically justified. The main contributions of this paper could be summarized as the following three aspects:

• Ellipsoid-structured Localized Generalized Eigenvalue Proximal Support Vector Machine (EL-GEPSVM). We propose the EL-GEPSVM to provide a solution to the misclassification issue of the UHP region of LGEPSVMs. A series of algorithms is designed for classification.
• Kernelized EL-GEPSVM. We extend the EL-GEPSVM to a nonlinear version with the proof of existence of dual representation of the Mahalanobis norm.
• Geometric interpretation of EL-GEPSVM. We provide the geometric interpretation of the effectiveness of the EL-GEPSVM by theoretically showing that the minimum ellipsoidal convex hull could be achieved with the Mahalanobis metric.

This paper is organized into the following sections: In Section 2, we briefly explain the background knowledge of GEPSVMs. In Section 3, we introduce the newly proposed Ellipsoid-structured Localized Generalized Eigenvalue Proximal Support Vector Machine (EL-GEPSVM). In Section 4, several numerical experiments are designed to verify the excellence of the Mahalanobis classifier mentioned in previous sections. In Section 5, the conclusion and future work are described based on experimental results and comparisons. We summarized the list of notations in Table 1.

Table 1. List of notations.

Notations	Description
X	The training data.
Y	The label set.
$x_t$	The test sample.
$Y_t$	The predicted label of the test sample.
$L_i$	The hyperplane of the $i^{th}$ class.
$w_i$	The weight vector of the $i^{th}$ class.
${\gamma }_i$	The basis of the $i^{th}$ class.
$A,B$	Sample matrices of two classes.
$\delta$	Regularization factor in the GEPSVM.
$\Sigma$	The covariance matrix.
H	The convex hull.
C	The convex set.
$\psi \left(·\right)$	The kernel mapping.
$r_i$	The radius of the ellipsoidal convex hull of the $i^{th}$ class.
${\mu }_i$	The mean of the ellipsoidal convex hull of the $i^{th}$ class.
$H_i$	The convex hull of the $i^{th}$ class.
n	The number of training samples.
m	The number of dimensions of training samples.
${∥·∥}_M$	The Mahalanobis metric.

2. Related Works

2.1. Proximal Support Vector Machines via Generalized Eigenvalues (GEPSVMs)

Given two m-dimensional samples ${\left\{x_i,y_i\right\}}_{i=1}^n,x_i\in {\mathcal{R}}^m,y_i\in \left\{-1,1\right\}$ in a binary classification problem, the task of the GEPSVM is to search for two hyperplanes:

$$L_i:w_i^{T}x+{\gamma }_i=0,i=1,2,$$

(1)

where $\left(w_i,{\gamma }_i\right)$ represent the norm vector and threshold of sample matrices A, B, respectively. Taking hyperplane $L_1$ as an example, to minimize the distance to A and maximize the distance to B, the objective function of optimization can be described as follows:

$$\underset{\left(w_1,{\gamma }_1\right)}{min}{\displaystyle \frac{{∥A{w}_1+e{\gamma }_1∥}_2^2+\delta {∥\left[\begin{array}{c}{w}_1\\ {\gamma }_1\end{array}\right]∥}_2^2}{{∥B{w}_1+e{\gamma }_1∥}_2^2}},$$

(2)

where A and B are the samples from classes 1 and 2, $(w_1,{\gamma }_1)$ defines the hyperplane of class 1, and e represents the column vector filled with element 1. Equation (2) formulates the learning of the first hyperplane in the GEPSVM framework as a Rayleigh quotient minimization problem. The numerator measures the within-class fitting error for class 1, regularized by the term $\left[\begin{array}{c}{w}_1\\ {\gamma }_1\end{array}\right]$ to avoid overfitting. The denominator expresses the distance from the opposite class 2, thereby enforcing large-margin separation. Minimizing this ratio yields a hyperplane that is close to samples in class 1 while distant from those in class 2. This optimization can be rewritten as a generalized eigenvalue problem:

$$Gz=\lambda Hz,$$

(3)

where $G={\left[A+e\right]}^T\left[A+e\right]+\delta I$, $H={\left[B+e\right]}^T\left[B+e\right]$, $z={\left[w^T,\gamma \right]}^T$, and $\delta \left(>0\right)$ is the regularization factor. Likewise, hyperplane $L_2$ can be obtained as above. It is easy to see that the smallest eigenvalue of Equation (3) depends on the corresponding eigenvector. This derivation follows directly from the Rayleigh quotient principle (see Ref. [2]).

Based on the GEPSVM, the localized GEPSVM [10] focuses on the local region of each hyperplane. In other words, the convex hull of each fitted hyperplane is computed in an LGEPSVM. To avoid conventional quadratic programming problem solving as well as to improve classification speed, a convex hull construction algorithm is proposed in [10]. We denote points projected to the hyperplane from a dataset as $\left\{{\varphi }_1,{\varphi }_2,\dots ,{\varphi }_n\right\}$, and · denotes the projection sample as shown in Figure 1. $x_1$ and $x_2$ are the two points farthest from each other, and $\overrightarrow{l}$ is the norm vector of $\left(x_1,x_2\right)$. Taking $x_2$ as a fixed point, we can find ${\varphi }_1$, which represents a projection point in the smallest angle $\overrightarrow{l}$ and $\left({\varphi }_1,x_2\right)$ made through the formulation ${max}_{{\varphi }_i\in \varphi }arccos\left({\displaystyle \frac{⟨\overrightarrow{l},\overrightarrow{{\varphi }_i{x}_2}⟩}{{∥\overrightarrow{l}∥}_2}},{∥\overrightarrow{{\varphi }_i{x}_2}∥}_2\right)$, in which $⟨·,·⟩$ represents the inner product of the vectors. ${\varphi }_i$ will act as the next fixed point in place of $x_2$. Then the process above is repeated until $x_2$ is selected as the fixed point again. All points selected are vertices of a convex set. In terms of [10], a localized GEPSVM is able to be transformed into an unconstrained linear equation problem with a time complexity of $O\left(n^3\right)$. Since convex hull computing is an iterative procedure, the initial condition is critical to the algorithm.

Recent works on GEPSVMs (e.g., GEPSVM_L1 [3], $L_{2,p}$-LIGEPSVM [14], and LGSVM [15]) extend the GEPSVM along functional data, robustness, or training schemes, but they still operate in Euclidean geometry and/or rely on local heuristics for overlap. The EL-GEPSVM introduces a Mahalanobis-based ellipsoidal convex hull around each class and proves a kernelized M-norm dual, giving a principled way to encode intra-class covariance + local shape while remaining Rayleigh quotient solvable—this directly targets the UHP/overlap in a way prior GEPSVM variants did not formalize.

2.2. Metric Learning

Metric learning [16] is a branch of machine learning that aims to learn an appropriate similarity function from data so that samples from the same class are close together while samples from different classes are far apart. Several studies have explored the integration of Mahalanobis metric learning [17,18,19] with classical classifiers to adaptively shape the feature space. We provide a detailed definition of the Mahalanobis metric in Definition 1 below. For example, Ref. [20] proposes learning a global Mahalanobis distance to enhance the performance of k-nearest neighbor classification. Additionally, DMLCN [21] learns a distance metric to simultaneously preserve the class center structure and the nearest neighbor relationship between data samples. Despite these advances, none of the related works integrates statistical metric learning directly into the geometric localization of the GEPSVM [2]. In contrast, the proposed method embeds the Mahalanobis metric for closed-form ellipsoid hull construction, fusing statistical structure, localization, and margin-based classification in a theoretically grounded but computationally efficient framework.

Recent works use metric learning to optimize ellipsoids (e.g., Ref. [17], CHSMOTE [22]), but they are typically decoupled from the GEPSVM decision rule. Our proposed EL-GEPSVM integrates a Mahalanobis structure inside the GEPSVM objective and derives an M-norm dual that is kernelizable, so the ellipsoid statistics actively shape the proximal hyperplanes, they do not just serve as a pre-processing module.

2.3. Efficiency-Oriented Large-Margin Classifiers

Recent margin-based classifiers focus on improving both computational efficiency and generalization [9]. TWSVM [23] improves training speed by solving two smaller-sized quadratic programming problems. To further scale TWSVMs to large datasets and enhance robustness, Tanveer et al. introduced Large-scale Pinball TWSVM (LPTWSVM) [24], which replaces hinge loss with pinball loss for dealing with noisy and high-dimensional data. More recently, capped L1-norm proximal SVMs [3,25] improve both noise resilience and efficiency by replacing squared L2 norms with capped L1 norms. While these classifiers struggle with efficiency and resilience, they still lack the integration of statistical distribution awareness and geometric localization, which motivates us to propose our EL-GEPSVM.

3. Ellipsoid-Structured Localized GEPSVMs

3.1. Motivation

As we discussed before, a hyper-ellipsoid is a trajectory of fixed radius in the Mahalanobis metric. In a two-dimensional space, it degenerates into an ellipse. Since most samples tend to be distributed on a hyper-ellipsoid after mapping [26], this is one of the reasons why the Mahalanobis-based classifier can achieve better performance. Furthermore, the Mahalanobis distance is adopted in this work because it naturally accounts for both the variance and the correlation among features, thereby providing a statistically adaptive notion of distance. In contrast, the Euclidean metric treats all directions as equally scaled and mutually independent, which can distort class boundaries when the feature space exhibits anisotropy or covariance coupling. We first present the definition of the Mahalanobis metric (termed as the M-norm) in Definition 1:

Definition 1 

(M-norm). Let $x,y\in {\mathbf{R}}^n$ be two distinct m-dimensional vectors from data samples X, and let Σ denote the covariance matrix. Assuming that Σ is a symmetric and positive-definite matrix, the Mahalanobis distance between x and y is defined as

$${∥x-y∥}_M=\sqrt{{\left(x-y\right)}^T{\Sigma }^{-1}\left(x-y\right).}$$

(4)

By Definition 1, the Mahalanobis distance can be viewed as a linear re-scaling of the Euclidean distance under the covariance structure $\Sigma$. Based on this equivalence, we can formally establish the distance-preserving property of the Mahalanobis metric in Theorem 1:

Theorem 1 

(Distance-preserving property). In the same normed linear space, the Euclidean metric and Mahalanobis metric are distance-preserving.

$${∥x-y∥}_M^2={∥T\left(x\right)-T\left(y\right)∥}_2,$$

(5)

where Σ is the covariance matrix, $T\left(x\right)={\Sigma }^{-1/2}x$.

Proof. 

As $\Sigma$ is a symmetric matrix [27], the inverse matrix of ${\Sigma }^{-{\displaystyle \frac{1}{2}}}$ is symmetrical.

Therefore,

$${\left(x-y\right)}^T{\Sigma }^{-{\displaystyle \frac{1}{2}}}={\left({\Sigma }^{-{\displaystyle \frac{1}{2}}}\left(x-y\right)\right)}^T.$$

(6)

Considering Definition 1 (Equation (15)) and Equation (6), let $T\left(x\right)={\Sigma }^{-1/2}x$ and we can derive that

$$\begin{array}{cc}\hfill {∥x-y∥}_M^2& ={\left({\Sigma }^{-{\displaystyle \frac{1}{2}}}\left(x-y\right)\right)}^T\left({\Sigma }^{-{\displaystyle \frac{1}{2}}}\left(x-y\right)\right)\hfill \\ \hfill & ={(T\left(x\right)-T\left(y\right))}^T(T\left(x\right)-T\left(y\right))\hfill \\ \hfill & ={∥T\left(x\right)-T\left(y\right)∥}_2^2.\hfill \end{array}$$

(7)

In this case, the Mahalanobis distance between any two points equals their Euclidean distance after the linear transformation $T(·)$. So the transformation $T(·)$ is distance-preserving.    □

Since the properties of an ellipsoid are dependent on its radius and centroid, constructing the convex hull construction of our proposed Ellipsoid-structured Localized Generalized Eigenvalue Proximal Support Vector Machine (EL-GEPSVM) requires only the mean of the projection points, which further enhances the efficiency. We will present the explanation, theory, and algorithm of the proposed method in detail in Section 3.2 and Section 3.3.

The convex hull is especially utilized in the localized GEPSVM (LGEPSVM), as mentioned in Section 1. The convex hull helps to alleviate the misclassification in the UHP region of the GEPSVM [10]. However, it still suffers from high computational and storage requirements, especially when dealing with larger-scale datasets. Based on the idea of a convex hull, our proposed Ellipsoid-structured Localized Generalized Eigenvalue Proximal Support Vector Machine (EL-GEPSVM) improves the localized GEPSVM and further makes it more efficient with less storage requirement. We will introduce it in the next subsection in detail.

3.2. Standard EL-GEPSVMs

We first present Definition 2, which formally defines the proposed Ellipsoid-structured Localized Generalized Eigenvalue Proximal Support Vector Machines (EL-GEPSVM). Then, we present how to realize the proposed EL-GEPSVM with the idea of a convex hull and the Mahalanobis metric in Theorems 2 and 3. We can define the proposed EL-GEPSVM as follows.

Definition 2 

(EL-GEPSVM). Let $X_k^i=\left\{X_1^i,X_2^i,\cdots ,X_n^i\right\}\in {\mathbf{R}}^{m\times n_1}$ be the samples in the $i^{th}$ class and $X_j$ be the samples from the $j^{th}$ class. The Ellipsoid-structured Localized Generalized Eigenvalue Proximal Support Vector Machine (termed EL-GEPSVM) learns the ellipsoidal convex hull $P_i$ and $P_j$ constructed from the GEPSVM hyperplanes to classify $X_i$ from $X_j$.

In our proposed EL-GEPSVM, we established the concept of an ‘ellipsoidal convex hull’ to refer to a convex region bounded by an ellipsoidal surface constructed under the Mahalanobis metric. Although Ref. [28] formulates geometric ellipsoids with a parametric boundary in TWSVM, our proposed method generalizes the notion of a convex hull to Mahalanobis space. Since the classification rule is made based on the idea of the convex hull, the concept of the ‘ellipsoidal convex hull’ is critical. Compared with circular (or spherical) structures that assume isotropic data distributions, the ellipsoidal form naturally accommodates anisotropy and feature correlations within the sample covariance matrix $\Sigma$. In real-world scenarios, class distributions rarely exhibit uniform variance across all dimensions. In this case, an ellipsoidal convex hull offers a more flexible and statistically consistent representation. This allows the proposed EL-GEPSVM to adaptively model the local covariance geometry of each class, improving boundary fitting and class separability. In the standard EL-GEPSVM, we take the maximum distance between the projection point on the hyperplane and the mean point as the radius of the ellipsoid for the ellipsoidal convex hull’s construction. First, we provide Theorem 2 of samples and a convex set:

Theorem 2 

(Samples and convex set). Both of the two points in sample set X with the longest distance are included in the convex set $C=\left\{{\sum }_{i=1}^n{\alpha }_i{x}_i\right|{\alpha }_i\ge 0,{\sum }_{i=1}^n{\alpha }_i=1\}$ as the smallest convex hull that contains all samples in X.

Proof. 

Define the projection samples ${\varphi }_1$, ${\varphi }_2$ as the farthest points from each other: $∥\overrightarrow{{\varphi }_1{\varphi }_2}∥>∥\overrightarrow{{\varphi }_1{\varphi }_i}∥,i=1,2,\cdots ,n$. Define H as the convex hull set. Two assumptions are listed below:

$${\varphi }_i\in H,i=1or2,$$

(8)

$${\varphi }_i\in H,i=1and2.$$

(9)

Suppose (1) (Equation (8)) is established, then connect ${\varphi }_1$, ${\varphi }_2$, and extend the line in direction of $\overrightarrow{{\varphi }_2}\overrightarrow{{\varphi }_1}$. Thus, we note the following:

I

Ray $L_{12}$ does not intersect with convex polygon ${\sum }_{i=1}^n{\lambda }_i{x}_i$;

II

Ray $L_{12}$ intersects with convex polygon ${\sum }_{i=1}^n{\lambda }_i{x}_i$.

Obviously, the above situation (I) breaks the condition that H is a convex hull set, so it will not be established.

In situation (II), by defining $X_0$ as the intersection of Ray $L_{1}2$ and $X_0^{\prime }$, $X_0^{″}$ as the points from H which are closest to $X_0$, it is easy to get that $X_0^{\prime }$ and $X_0^{″}$ are located in the same line. Then, considering triangle ${\varphi }_2{x}_0^{\prime }{x}_0^{″}$, we can find that either $∥\overrightarrow{{\varphi }_2{x}_0^{\prime }}∥>∥\overrightarrow{{\varphi }_2{x}_0}∥$ or $∥\overrightarrow{{\varphi }_2{x}_0^{″}}∥>∥\overrightarrow{{\varphi }_2{x}_0}∥$ is established. Because $X_0$ is located in the extended line of the segment, we can obtain inequality $∥\overrightarrow{{\varphi }_2{x}_0}∥>∥\overrightarrow{{\varphi }_2{x}_1}∥$. As a consequence, a point $x_j=\left\{\begin{array}{cc}{x}_0^{\prime },& ∥{\varphi }_2{x}_0^{\prime }∥>∥{\varphi }_2{x}_0∥\\ x_0^{″},& ∥{\varphi }_2{x}_0^{″}∥>∥{\varphi }_2{x}_0∥\end{array}\right.$ exists and it belongs to H, which contradicts condition with inequality $∥{\varphi }_2{x}_j∥>∥{\varphi }_2{\varphi }_1∥,j=1,2,\cdots ,n$. Therefore, the above situation (II) is rejected.

As both assumptions above are not established, the only assumption left is accepted, which is ${\varphi }_i\in H,I=1and2$.    □

Based on Theorem 2, we have Theorem 3 to construct the ellipsoidal convex hull with Definitions 1 and 2.

Theorem 3 

(Minimum-radius ellipsoidal convex hull). Given $X\in {\mathcal{R}}^{m\times n}$, let ${\mu }_p$ be the projection of the geometric center μ on to hyperplane $L_1:w_1^{T}x+\gamma =0$. The ellipsoid centered at ${\mu }_p$ is the smallest (minimum-radius) ellipsoid, in the Mahalanobis metric, that encloses all samples in X.

$$r=\underset{x_i\in X}{max}{∥{\mu }_p-x_i∥}_M.$$

(10)

Proof. 

We define

$$r=\underset{x_i\in X}{max}{∥{\mu }_p-x_i∥}_M,r^{\prime }=\underset{{\varphi }_i\in \Phi }{max}{∥{\mu }_p-{\varphi }_i∥}_M.$$

(11)

In other words,

$$\begin{array}{cc}\hfill & \forall x_i\in X,r\ge {∥{\mu }_p-x_i∥}_M\hfill \\ \hfill & \forall {\varphi }_i\in \Phi ,r^{\prime }\ge {∥{\mu }_p-{\varphi }_i∥}_M\hfill \end{array}.$$

(12)

According to Theorem 1, the Mahalanobis and Euclidean metrics are distance-preserving under the transformation $T\left(x\right)={\Sigma }^{-1/2}x$, so the radius computation in the Mahalanobis space can be equivalently performed in the Euclidean space:

$$r=\underset{x_i\in X}{max}{∥{\mu }_p-T\left(x_i\right)∥}_2,r^{\prime }=\underset{{\varphi }_i\in \Phi }{max}{∥{\mu }_p-T\left({\varphi }_i\right)∥}_2.$$

(13)

According to Theorem 1, it is easy to get that $x_m\left(\in X\right)$ is the farthest point away from ${\mu }_p$ and its length is $r_d=\sqrt{{r_d^{\prime }}^2+v^2}$. Then, We obtain the equation $v={∥{\mu }_p-X_m∥}_2$. Combining Theorem 1 with Equation (12), then,

$$\forall {\varphi }_i\in \varphi ,r\ge {∥u_p-{\varphi }_i∥}_M.$$

(14)

According to Equation (14), an ellipsoidal convex hull with mean point and radius contains points within the point set. In this case, this indicates that it is the minimum ellipsoidal convex hull.    □

Next, we provide Lemma 1 for the ellipsoid-structured convex hull construction.

Lemma 1 

(Distance in Mahalanobis metric). In Mahalanobis metric space, distance between a sample v and the hyperplane $w^{T}x+\gamma =0$ is given by

$$d={\displaystyle \frac{\left|w^{T}v+\gamma \right|}{\sqrt{w^T\Sigma w}}}.$$

(15)

and projection of a point to hyperplane $w^{T}x+\gamma =0$. $v_p$ can be expressed as

$$v_p=v-{\displaystyle \frac{\left(w^{T}v+\gamma \right)\Sigma w}{w^T\Sigma w}}.$$

(16)

Then we can have the concrete construction algorithm as follows (Algorithm 1).

Algorithm 1 Ellipsoid-structured convex hull construction.

Input Dataset $X=\left\{x_1,x_2,\cdots ,x_n\right\}$, $w^{T}x+\gamma =0$; Output Radius of ellipsoid r, center of projection points ${\mu }_p$; Initialization: $r\leftarrow 0$; $e=\left[1,1,\cdots ,1\right]$;     1.  Compute $\mu \leftarrow Xe/n$, $\Sigma \leftarrow {\displaystyle \frac{1}{n}}{\Sigma }_{i=1}^n\left(x_i-\mu \right){\left(x_i-\mu \right)}^T$;     2.  According to Equation (16), compute ${\mu }_p\leftarrow \mu -{\displaystyle \frac{{\left(w^T\mu +b\right)}^T\Sigma w}{w^T\Sigma w}}$; for $i\leftarrow 1$ to $\tilde{n}$ do     3.    $d\leftarrow \sqrt{{\left(x_i-u_p\right)}^T{\Sigma }^{-1}\left(x_i-u_p\right)}$;     if $d>r$ then     4.       $r\leftarrow d$;     end if end for Output Radius of ellipsoid r, center of projection points ${\mu }_p$;

Depending on the different relationships between a point and the convex hull, the distance of a sample to a convex hull is not absolutely the same in each situation. According to Algorithm 2, there are three relationships that exist between a point and a convex hull, which are overlap ($flag<0$), tangent ($flag=0$), and separation ($flag>0$). In this case, we divided these three situations into two groups, where overlap and tangent are categorized as the same group, while separation is in a second group. For the situation in group I, we calculate $d_0$ as the distance between the test sample and the hyperplane to measure the closeness between the test sample and hyperplane. For the situation in group II, we calculate $d_0$ as the distance between the test sample and the convex hull. Next, we present the distance calculation in Algorithm 2.

Algorithm 2 Calculation of the distance to the ellipsoid-structured convex hull.

Input Dataset $X=\left\{x_1,x_2,\cdots ,x_n\right\}$, $w^{T}x+\gamma =0$; Initialization: Covariance matrix $\Sigma$, mean point $\mu$, radius of ellipsoidal convex hull r, $flag\leftarrow 0$;     1.  Compute the projection point $x_p$ according to Equation (15);     2.  Compute $d\leftarrow \sqrt{{\left(x_p-u\right)}^T{\Sigma }^{-1}\left(x_p-u\right)}$; /*Flag == −1 outside the hull; Flag == 0 on the hull; Flag == 1 inside the hull*/; if  $d>r$  then     3.     $flag\leftarrow -1$ else if  $d<r$  then     4.     $flag\leftarrow 1$ end if if $flag==0$ OR $flag==1$ then     5.     $d_0\leftarrow {\displaystyle \frac{\left|w^{T}x+b\right|}{\sqrt{w^T\Sigma w}}}$; else then     6.     ${\mu }_p=\mu -{\displaystyle \frac{\left(w^T\mu +b\right)\Sigma w}{w^T\Sigma w}}$;     7.     $d_0\leftarrow \sqrt{{\left({\mu }_p-x_p\right)}^T{\Sigma }^{-1}\left({\mu }_p-x_p\right)}-r$; end if Output Distance $d_0$;

3.3. Nonlinear EL-GEPSVMs

To deal with data in the nonlinear space, we can extend our proposed EL-GEPSVM to the nonlinear version. First, we present Theorem 5 to prove the existence of dual representation of the M-norm.

Theorem 4 

(Kernelized EL-GELSVM). Given a d-dimensional dataset $X=\left\{x_1,x_2,\dots ,x_n\right\}$, the covariance matrix is $\Sigma \in {\mathbb{R}}^{d\times d}$, and its inverse matrix is ${\Sigma }^{-1}={\Sigma }_{i=1}^d{\displaystyle \frac{1}{{\lambda }_i}}{u}_i{u}_i^T$, where ${\lambda }_i$ is an eigenvalue of Σ, and $u_i$ represents the symmetric matrix. Let $\psi (·)$ denote the kernel mapping, the dual representation of kernelized $u_i$: $u_i^{\psi }={\Sigma }_{i=1}^n\beta \left(\psi \left(x_i\right)-\overline{\psi }\right)$, where $\overline{\psi }={\displaystyle \frac{1}{n}}{\Sigma }_{i=1}^n\psi \left(x_i\right)$.

Proof. 

Assuming the covariance matrix $\Sigma$ to be positive-definite, we perform singular value decomposition (SVD) [29] and get

$$\Sigma =\sum _{i=1}^d\lambda u_i{u}_i^T,{\Sigma }^{-1}=\sum _{i=1}^d{\displaystyle \frac{1}{{\lambda }_i}}{u}_i{u}_i^T.$$

(17)

According Equation (17), we have

$$\begin{array}{c}\hfill \Sigma U=U\Lambda ,\Sigma u_i={\lambda }_i{u}_i.\end{array}$$

(18)

The covariance matrix with kernel mapping $\psi \left(x\right)$ can be described as follows:

$${\Sigma }^{\psi }={\displaystyle \frac{1}{n}}\sum _{i=1}^n\left(\psi \left(x_i\right)-\overline{\psi }\right){\left(\psi \left(x_i\right)-\overline{\psi }\right)}^T,$$

(19)

where $\overline{\psi }={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n\psi \left(x_i\right)$.

Next, we define $\alpha ={\left(\psi \left(x_i\right)-\overline{\psi }\right)}^T$, and we can get

$$u_i^{\psi }={\displaystyle \frac{1}{{\lambda }_i}}\alpha {\displaystyle \frac{1}{n}}\sum _{i=1}^n\left(\psi \left(x_i\right)-\overline{\psi }\right).$$

(20)

Next, we define ${\beta }_i={\lambda }_i\alpha {\displaystyle \frac{1}{n}}$, and we can get

$$u_i^{\psi }=\sum _{i=1}^n{\beta }_i\left(\psi \left(x_i\right)-\overline{\psi }\right).$$

(21)

Note that the covariance matrix in feature space can be expressed via the kernel trick as

$$\Sigma ={\displaystyle \frac{1}{n}}\Phi K_c\Phi ,$$

(22)

where $K_c$ is the centered kernel matrix and $\Phi =[\psi \left(x_1\right),\cdots ,\psi \left(x_n\right)]$. In this case, ${\Sigma }^{-1}$ and the Mahalanobis distance ${∥z∥}_M^2=\psi {\left(z\right)}^T{\Sigma }^{-1}\psi \left(z\right)$ can be computed solely through kernel evaluations.    □

To generalize the proposed EL-GEPSVM from linear to nonlinear space, it is essential to prove that the Mahalanobis norm used in the ellipsoidal metric admits a kernel-compatible form. Theorem 5 establishes the dual representation of the M-norm, which provides the mathematical basis for kernelizing the EL-GEPSVM.

Theorem 5 

(Dual representation of M-norm). Given d-dimensional dataset $D=\left\{x_1,x_2,\dots ,x_n\right\}$, there exists a dual representation of the squared M-norm:

$${∥z∥}_M^2={\eta }^T{\left[{\beta }^{T}k\left(X,z\right)\right]}^T\left[{\beta }^{T}k(z,X)\right],$$

(23)

where $k(z,X)=\left[k\left(z,x_1\right),k\left(z,x_2\right),\dots ,k\left(z,x_n\right)\right]\in {\mathcal{R}}^n$, $\eta =\left({\displaystyle \frac{1}{{\lambda }_1}},{\displaystyle \frac{1}{{\lambda }_2}},\dots ,{\displaystyle \frac{1}{{\lambda }_d}}\right)\in {\mathcal{R}}^d$, and $\beta =[{\beta }_1,{\beta }_2,\cdots ,{\beta }_n]\in {\mathcal{R}}^{n\times d}$.

Proof. 

According to Definition 1, we square the M-norm and get

$${∥z∥}_M^2=z^T{\Sigma }^{-1}z.$$

(24)

In terms of Lemma 4, we can get ${\Sigma }^{-1}={\sum }_{i=1}^d{\displaystyle \frac{1}{{\lambda }_j}}{u}_i{u}_i^T$, and $u_i={\sum }_{i=1}^n{\beta }_i\left(\psi \left(x_i\right)-\overline{\psi }\right)$. Thus, considering Equations (18), (21), and (24), we deduce and display this in (20) while introducing kernel function $k\left(x,y\right)=\psi {\left(x\right)}^T\psi \left(y\right)$.

$$\begin{array}{cc}\hfill {∥z∥}_M^2& =\psi {\left(z\right)}^T\sum _{i=1}^d{\displaystyle \frac{1}{{\lambda }_i}}\sum _{j=1}^d{\beta }_j\psi \left(x_j\right)\sum _{k=1}^d{\beta }_k\psi {\left(x_j\right)}^T·\psi \left(z\right)\hfill \\ \hfill & =\sum _{i=1}^d{\displaystyle \frac{1}{{\lambda }_i}}\sum _{j=1}^d{\beta }_j\psi {\left(z\right)}^T\psi \left(x_j\right)\sum _{k=1}^d{\beta }_k^T\psi {\left(x_k\right)}^T\psi \left(z\right)\hfill \\ \hfill & =\sum _{i=1}^d{\displaystyle \frac{1}{{\lambda }_i}}\sum _{j=1}^d{\beta }_{j}k\left(z,X_j\right)\sum _{k=1}^d{\beta }_k^{T}k\left(z,X_k\right)\hfill \\ \hfill & =\sum _{i=1}^d{\displaystyle \frac{1}{{\lambda }_i}}{\left[{\beta }_{j}k\left(z,X\right)\right]}^T\left[{\beta }^{T}k\left(z,X\right)\right]\hfill \\ \hfill & =\sum _{i=1}^d{\displaystyle \frac{1}{{\lambda }_i}}{\left[{\beta }^{T}k\left(z,X\right)\right]}^T\left[{\beta }^{T}k\left(z,X\right)\right]\hfill \\ \hfill & ={\displaystyle \frac{1}{\lambda }}{\left[{\beta }^{T}k\left(z,X\right)\right]}^T\left[{\beta }^{T}k\left(z,X\right)\right]\hfill \\ \hfill & =\eta {\left[{\beta }^{T}k\left(z,X\right)\right]}^T\left[{\beta }^{T}k\left(z,X\right)\right]\hfill \end{array},$$

(25)

where (25), $k(z,X)=\left[k\left(z,x_1\right),k\left(z,x_2\right),\dots ,k\left(z,x_n\right)\right]$, $\eta =\left({\displaystyle \frac{1}{{\lambda }_1}},{\displaystyle \frac{1}{{\lambda }_2}},\dots ,{\displaystyle \frac{1}{{\lambda }_d}}\right)$.    □

Theorem 5 shows that the proposed ELGEPSVM can be extended to a nonlinear version since the M-norm has dual representation for kernelization of the EL-GEPSVM. Therefore, considering the above Lemma 4 and Theorem 5, we can have the definition of a nonlinear EL-GEPSVM, which can be described as follows.

Definition 3 

(Nonlinear EL-GEPSVM). Let $X_i=\left\{X_1^i,X_2^i,\cdots ,X_n^i\right\}\in {\mathbf{R}}^{m\times n}$ be the samples in the $i^{th}$ class, and $X^j$ be the samples from the $j^{th}$ class. Considering the standard EL-GEPSVM (as defined in Definition 2), the nonlinear EL-GEPSVM learns the hyperplanes ${\widehat{H}}_1$ and ${\widehat{H}}_2$ in the nonlinear space with the kernelized EL-GEPSVM (Theorem 4) and M-norm (Theorem 5).

We summarize the proposed EL-GEPSVM method in Algorithm 3.

Algorithm 3 EL-GEPSVM algorithm.

Input Dataset $X=\left\{x_1,x_2,\cdots ,x_n\right\}$, Label set $Y=\left\{Y_1,Y_2,\cdots ,Y_n\right\}$, Test sample $x_t$; Initialization: Covariance matrix $\Sigma$, mean point $\mu$, radius of ellipsoidal convex hull r, $flag\leftarrow 0$;     1.  Generate the hyperplanes $H_1:\left(w_1,{\gamma }_1\right)$ of class 1 and $H_2:\left(w_2,{\gamma }_2\right)$ of class 2 with Equations (2) and (3);     2.  Construct the ellipsoidal convex hull with Algorithm 1: $\left(r_1,{\mu }_{p,1}\right)$, $\left(r_2,{\mu }_{p,2}\right)$.     3.  Calculate the distance $\left(d_1,d_2\right)$ from $x_0$ to $H_1:\left(w_1,{\gamma }_1\right)$ and $H_2:\left(w_2,{\gamma }_2\right)$; if  $d_1<d_2$  then     4.     $Y_0\leftarrow 1$ else     5.     $Y_0\leftarrow 2$ end if Output Predicted Label $Y_0$;

3.4. Computational and Space Complexity Analysis

We analyze the computational and space complexity of Algorithms 1–3. In Algorithm 1, the calculation of the mean, covariance, and projection requires $\mathcal{O}\left(nm\right),\mathcal{O}\left(n{m}^2\right),\mathcal{O}\left(m^2\right)$ operations, respectively, where n is the number of samples and m the feature dimension. Considering the iterative operations in steps 3 and 4, the overall computational cost is $\mathcal{O}(m^3+n{m}^2)$. In Algorithm 2, for each data sample, the Mahalanobis distance computation requires $\mathcal{O}\left(m^2\right)$ time, resulting in an overall complexity of $\mathcal{O}\left(m^2\right)$. In Algorithm 3, the total training cost is $\mathcal{O}(n{m}^2+m^3)$, while the inference phase requires $\mathcal{O}\left(m^2\right)$ per sample. For the kernelized version, the formation of the G ram matrix $K(X,X)$ and its inversion introduce an additional $\mathcal{O}\left(n^3\right)$ cost. For the space complexity, the primary storage requirements come from the covariance matrix $\Sigma \in {\mathcal{R}}^{m\times m}$, the hyperplane parameters $(w,\gamma )$, and the mean vectors $(\mu ,{\mu }_p)$. Hence, the total space complexity is $\mathcal{O}\left(m^2\right)$ for the linear case. For the kernelized implementation, storing the Gram matrix requires $\mathcal{O}\left(n^2\right)$ space.

4. Experiments and Analysis

4.1. Experimental Data and Settings

We evaluated the proposed EL-GEPSVM method by designing the following experiments: (1) visualization of EL-GEPSVM, (2) experiments on multi-class classification with EL-GEPSVM, (3) parameter analysis, and (4) computational time analysis. In the first experiment, we visualized the construction of a convex hull in the EL-GEPSVM to show the feasibility and basic idea of the proposed algorithm. We demonstrated the performance of the EL-GEPSVM (the standard and kernelized versions) in the second experiments. Then, we performed parameter analysis to show the stability of the proposed algorithms with changing parameters as mentioned in Algorithms 1 and 3. Lastly, we tested the efficiency of the proposed algorithms in the fourth experiment. All experiments in this section follow the following settings:

Data: We utilized 11 datasets from the UCI repository [30] to evaluate the performance of linear and nonlinear EL-GEPSVMs on multi-class classification.

Comparisons: We made comparisons with 11 state-of-the-art methods including SVM-based methods and popular GEPSVM variants: SVM [9], SRC [31], SVM-2K [32], GEPSVM [2], LGEPSVM [10], TWSVM [27], GNPSVM [33], and L1GEPSVM [25].

Experimental setup: Our experiments and comparisons are based on data from the UCI repository and synthetic datasets. All experiments are conducted on an Intel Core(TM) i5-4210M 2.60GHz on MATLAB R2016a.

4.2. Visualization of EL-GEPSVM

We visualized the EL-GEPSVM in Figure 2 on four datasets: Iris2D, Uniform3D, Iris3D, and Pimadata3D. For the Iris2D dataset, we selected classes 1 and 2 from the full iris dataset, then we projected the raw data samples to a 2D linear space with PCA algorithm. Similarly, for Iris3D, we selected classes 2 and 3 from the full iris dataset, then utilized PCA to reduce the dimensions of the raw data to three. For Uniform 3D, we randomly generated 100 data samples for two different classes. The samples of each class follow the uniform distribution. For Pimadata3D, we reduced the dimensions of raw data samples from the Pima dataset to three. According to Figure 2a–c, we can clearly see that the ellipsoidal convex hull supported by two hyperplanes can cover the sample of each category. Samples of two categories are projected to the hyperplane (in green and cyan colors). After the construction of two hyperplanes with the EL-GEPSVM, two ellipsoidal convex hulls are solved as well (two ellipses), which describe the location relationship between the convex hull and the projection points directly. The ellipsoidal convex hull is able to separate data from two categories for better classification. This shows that the ellipsoid-structured LGEPSVM can feasibly act as a classifier. From Figure 2d, the EL-GEPSVM makes a clear separation with convex hulls to distinguish samples from two different classes in 3D space. Each pair of projection points and the ellipse lie on the same plane, proving the correctness of Theorem 3 and Algorithm 1.

Figure 2. Visualization of the proposed Ellipsoid-structured Localized Generalized Eigenvalue Proximal Support Vector Machine (EL-GEPSVM). The ellipsoids illustrate the localized convex hulls constructed in the Mahalanobis space, showing adaptive shapes that fit intra-class distributions. The subplots correspond to (a) Iris2D, (b) Uniform3D, (c) Iris3D, and (d) PimaData3D datasets.

4.3. Experiments on EL-GEPSVM

This experiment was conducted on the UCI dataset and a part of the synthetic dataset. In the synthetic dataset, iris12 is obtained from the iris dataset after PCA (Principal Component Analysis) processing, and the Cross2plane dataset is two crossing lines with uniform distribution noise added; these were used to validate the feasibility of the GEPSVM classifier in the XOR problem. We used 10-fold cross-validation to test the accuracy and demonstrate results through accuracy and standard deviation. Furthermore, we utilized a paired T-test as the test of statistical significance between the proposed EL-GEPSVM and LGEPSVM methods to make sure that the experimental results are reliable, and we quantify it by using the p-value. We performed the significance test with the LGEPSVM since the proposed method originates from it and can be taken as a variant of this method. In addition, we use the star notation (*) to indicate a significant difference. Results of linear kernel are recorded in Table 2. As to nonlinear circumstances, we utilized a Gaussian kernel whose results are demonstrated in Table 3.

Table 2. Accuracies (%) and comparisons of the linear EL-GEPSVM and the state-of-the-art methods.

	Accuracy ± Standard Deviation
	SVM	GEPSVM	LGE- PSVM	TW- SVM	GN- PSVM	L1- GEPSVM	EL- GEPSVM	p-Value
cross2plane	84.64	99.50	99.50	59.00	58.80	48.53	99.50	0.088
	± 1.39	± 1.12	± 1.12	± 3.53	± 3.20	± 1.14	± 0.25
monk2	70.00	56.28	69.17	50.00	65.00	54.00	80.00	0.074
	± 3.35	± 4.93	± 6.41	± 1.50	± 3.50	± 2.60	± 16.25
monk3	66.67	80.38	80.73	51.16	58.13	91.57	81.00	0.9649
	± 0.33	± 2.05	± 4.41	± 2.13	± 2.98	± 1.12	± 18.23
Muskclean	66.67	38.79	51.09	65.11	51.16	62.72	81.34	0.009 *
	± 3.93	± 1.52	± 1.20	± 2.93	± 1.72	± 2.15	± 20.33
Waveform	49.02	60.62	59.66	91.21	88.46	79.95	91.56	6.07 $\times {10}^{-17}$ *
	± 2.52	± 0.56	± 0.52	± 3.28	± 1.15	± 2.65	± 1.29
Pimadata	48.14	70.71	68.37	71.42	52.63	87.66	71.54	0.183
	± 2.87	± 3.44	± 0	± 2.57	± 1.65	± 2.67	± 6.14
iris12	100	100	100	100	100	78.00	98.00	0.168
	± 0	± 0	± 0	± 0	± 0	± 3.90	± 4.22
Water	58.33	68.22	66.49	100	54.54	43.84	78.00	0.168
	± 4.33	± 1.12	± 2.93	± 0	± 3.47	± 2.23	± 3.94
Sonar	60.00	62.22	48.54	70.00	60.00	59.47	70.00	0.037 *
	± 2.50	± 1.19	± 2.54	± 2.50	± 5.00	± 1.15	± 5.68
Clevedata	20.00	68.00	44.87	44.82	48.27	67.09	60.00	0.043 *
	± 4.65	± 2.50	± 2.83	± 2.79	± 3.83	± 2.36	± 4.92

The highest value in each row is marked in bold font; The ’*’ indicates it is statistically significant.

Table 3. Accuracies (%) and comparisons of the kernelized EL-GEPSVM and the state-of-the-art methods.

Datasets	Accuracy ± Standard Deviation
	TWSVM	GEPSVM	LGEPSVM	EL-GEPSVM (G)	EL-GEPSVM (P)
cross2plane	100 ± 0.00	99.50 ± 1.12	99.50 ± 1.12	99.50 ± 0.42(0.063)	79.00 ± 2.92(0.082)
BUPA	67.64 ± 1.35	63.41 ± 3.29	64.39 ± 2.19	67.85 ± 12.59(0.033 *)	67.14 ± 8.17(0.046 *)
Water	63.63 ± 0.92	68.22 ± 8.15	66.49 ± 10.20	59.64 ± 11.30(0.174)	85.00 ± 10.93(0.025)
Checkdata	51.00 ± 2.37	55.61 ± 5.33	55.88 ± 4.76	76.77 ± 14.52(0.012 *)	62.79 ± 2.76(0.009 *)
Pima	64.47 ± 1.19	45.08 ± 4.23	52.96 ± 2.97	79.62 ± 4.39(0.017 *)	80.77 ± 7.92(0.006 *)
WPBC	73.68 ± 2.27	64.32 ± 5.83	74.21 ± 3.90	52.50 ± 5.27(0.724)	91.25 ± 1.87(0.032 *)

The highest value in each row is marked in bold font; The ‘*’ indicates it is statistically significant. The second best performance of each column is marked with underline.

In Table 2, we bold the highest accuracy of each row (dataset). Results of the linear EL-GEPSVM are listed in Table 2. Based on the results shown in Table 2, we can observe that the EL-GEPSVM not only achieves similar accuracy to the LGEPSVM but performs even better on some of the datasets, such as monk2 and waveform. Accordingly, it is fair to say that the MGEPSVM is equipped with a more powerful classification ability compared with the LGEPSVM in the linear kernel. Noticeably, we found our proposed method achieved lower performance than the highest accuracy in the comparison (L1-GEPSVM) on the monk3 dataset. Since our proposed method directly originates from the LGEPSVM, the achieved accuracy is still higher than that of the LGEPSVM, which demonstrates the superiority of introducing the ellipsoidal structure and Mahalanobis metric into the GEPSVM framework. This result suggests that although Mahalanobis-based convex modeling may not always outperform all variants in specific datasets, it provides a more stable and theoretically grounded decision representation that effectively balances local geometric information and intra-class covariance. Consequently, the performance on Monk3 highlights a trade-off between model generalization and dataset-specific adaptiveness, which will be further optimized in our future work. In addition, the paired T-test results demonstrate that these improvements are statistically significant rather than due to random variation. This consistency across multiple datasets suggests that the proposed Mahalanobis-based convex hull modeling enhances the robustness of large-margin learning. In Table 3, we denote the Gaussian kernel and polynomial kernel as EL-GEPSVM (G) and EL-GEPSVM (P). We can find that the EL-GEPSVM performs better compared with the GEPSVM and LGEPSVM, especially in Checkdata. What is noticeable is that although the EL-GEPSVM in the Water dataset has lower accuracy in the Gaussian kernel, its accuracy in the linear kernel is much higher than the other two classifiers, which reflects the fact that classification accuracy is changeable upon kernel selection. The kernelized EL-GEPSVM benefits from mapping the ellipsoidal boundary into a higher-dimensional reproducing kernel Hilbert space, where the Mahalanobis metric translates into a weighted inner product. This joint kernel–Mahalanobis interaction increases nonlinear discriminability while retaining the convexity of the original optimization. The empirical gains over the kernel GEPSVM highlight the robustness of our dual representation of the M-norm (Theorem 5) and its contribution to improved convergence stability.

We show the performance tendency of the GEPSVM, the LGEPSVM, and the proposed EGEPSVM in Figure 3. As we find, when the number of dimensions of data is higher, the gap between the EL-GEPSVM and the other two methods becomes much larger. This indicates that the proposed method has a stronger ability to deal with data in high-dimensional space. Since the data distribution is more complicated in high-dimensional space, the difficulty will be higher in generating a separable hyperplane. This explains how the GEPSVM fails in high-dimensional space. Furthermore, the statistical information becomes more critical to the performance with a higher number of dimensions, which means the ELGPSVM will have superiority over the LGEPSVM in the high-dimensional space. As we can see in Table 2, our proposed EL-GEPSVM achieved the best performance in most cases (BUPA, Water, Checkdata, Pima). Although in cross2plane dataset, the proposed method did not obtain the highest accuracy, it is still the second highest (99.50%). The gap between the highest one (100%, TWSVM) and the proposed method is 0.5%.

Figure 3. Comparative performance of GEPSVM, LGEPSVM, and the proposed EL-GEPSVM under varying feature dimensions. Bar colors represent the linear versions of each method (blue—GEPSVM, yellow—LGEPSVM, red—EL-GEPSVM), while the overlaid lines of corresponding colors indicate their kernelized counterparts. The x-axis denotes the number of dimensions, and the y-axis shows the classification accuracy (%).

Compared with the standard GEPSVM [2] and its variants, the proposed EL-GEPSVM achieves higher accuracy because the Mahalanobis-based ellipsoidal representation better captures class covariance and intrinsic geometry. Furthermore, we found the use of adaptive covariance weighting stabilizes the optimization, avoiding the numerical divergence observed in large-variance datasets. Theoretically speaking, this corresponds to an implicit regularization that aligns with the minimum-radius ellipsoidal convex hull, improving both margin stability and generalization.

To show how the proposed method deals with high-dimensional data, we show the performance of the GEPSVM, LGEPSVM, and proposed EL-GEPSVM on an increasing number of dimensions. Observing Figure 3, we find that the proposed EL-GEPSVM consistently achieves superior performance across most dimensional settings compared with the GEPSVM and LGEPSVM. In lower dimensions (e.g., two to eight), our proposed EL-GEPSVM yields notably higher accuracies, while in higher-dimensional cases, it maintains a smoother degradation curve, indicating stronger robustness to dimensional expansion. Overall, the EL-GEPSVM achieves the best trade-off between accuracy and scalability among the compared methods.

4.4. Parameter Analysis

We performed parameter analysis of the proposed method on four datasets. We analyzed both regularization factor $\theta$ and the standard deviation $\sigma$ in the E-LGEPSVM. We set $\theta \in \left[0,5\times {10}^{-4}\right]$ and $\sigma \in \left[1\times {10}^{-5},1\times {10}^1\right]$, then showed the performance of the method. The results are shown in Figure 4 and Figure 5.

Figure 4. Parameter analysis of proposed EL-GEPSVM on $\theta$ and $\sigma$ in (a) Pima, (b) Checkdata, (c) Sonar, and (d) Celevedata.

Figure 5. Parameter analysis of proposed E-LGEPSVM on scaling factor $\alpha$, bias c, and degree $\rho$ in (a) BUPA [$\alpha$ vs. c], (b) BUPA [$\alpha$ vs. d], (c) WPBC [$\alpha$ vs. c], and (d) WPBC [$\alpha$ vs. d].

From Figure 4, we can observe the following: (1) The best performance of the proposed method is located in $\theta \in \left[0, 1\times {10}^{-5}\right]$ and $\sigma \in \left[0, 1\times {10}^{-5}\right]$. (2) The setting of $\theta$ is critical to the performance of classification. For example, in the Pima dataset (see Figure 4a), the proposed method shows better performance when setting a lower $\theta$ value. In contrast, the changing $\sigma$ is stable when $\theta ={10}^{-6}$. (3) The accuracy varies when $\sigma \in \left[0,1\times {10}^{-5}\right]$. In Checkdata (see Figure 4b), the highest accuracy is achieved when $\sigma =1$. In Sonar data, the highest accuracy is on $\sigma =1\times {10}^{-3}$. We also tuned the parameters of the polynomial kernel: scaling factor $\alpha$, bias c, and degree $\rho$. The parameter analysis results on the BUPA and WPBC datasets are shown in Figure 5. The parameter analysis on the BUPA and WPBC datasets demonstrates that the performance of the EL-GEPSVM is sensitive to the choice of kernel parameters. However, stable high accuracies can be obtained within certain ranges. For BUPA, the best results are achieved when $c=1$ or $d=3$, combined with moderate $\alpha$ values [0.5, 1]. For WPBC, the method consistently performs well when $c=1$ or $\rho =4$ with an accuracy peak above 90% at $\alpha =0.5$. These results confirm that appropriate parameter selection can significantly enhance the performance.

Overall, the parameter analysis demonstrates that the proposed EL-GEPSVM maintains stable and high performance across a broad range of $\theta$ and $\sigma$ values. For example, in WPBC, the polynomial kernel EL-GEPSVM achieves 91.25% ± 1.87%, outperforming the LGEPSVM (74.21%, Table 3) by nearly 17%, confirming robustness even in small-sample, high-dimension settings. In BUPA, stable regions appear when $c=1$ or $\rho =3$ with $\alpha \in [0.5, 1]$, maintaining accuracies between 66 and 68%, while WPBC keeps accuracies above 90% for $c=1$ or $d=4$ (see Figure 5). These consistent high-accuracy zones reveal that the proposed metric regularization term effectively bounds the covariance eigen-structure $\Sigma$, ensuring smooth convergence and resilience against parameter perturbations. Hence, the method achieves both stability and interpretability under varying $\theta$ and $\sigma$, validating the theoretical claim that Mahalanobis-based regularization provides a well-conditioned optimization landscape.

4.5. Computational Time Analysis

We evaluated the efficiency of the EL-GEPSVM and made comparisons. As mentioned above, we proposed the EL-GEPSVM to improve the efficiency of the LGEPSVM. Since the LGEPSVM spends a large portion of its computational time on convex hull construction in the training stage, we mainly made comparisons on the convex hull construction among the EL-GEPSVM, GEPSVM, and LGEPSVM. The evaluation results are shown in Table 4 with the muskclean dataset. We also compared the efficiency with the state-of-the-art TWSVM method. Besides the computational time (Time) and accuracy (Accuracy), we also calculated the $\mathcal{E}$-score to evaluate the accuracy–time ratio of the comparison methods. This metric directly shows how much accuracy is achieved per second:

$$\mathcal{E}-score={\displaystyle \frac{Accuracy}{Time}}.$$

(26)

Table 4. Computational time comparison on GEPSVM, LGEPSVM, and EL-GEPSVM on muskclean data.

Algorithm	Time (s)	Accuracy (%)	$\mathcal{E}$-Score
TWSVM	17.6071	91.65 ± 2.95	5.21
GEPSVM	0.3197	38.79 ± 1.52	121.22
LGEPSVM	1.1677	51.09 ± 1.20	43.67
EL-GEPSVM	0.0624	81.34 ± 20.33	1303.53

The best performance of each column is marked in bold font.

In Table 4, the LGEPSVM takes 3.6 times the computation time that the GEPSVM takes in the classification period, while the EL-GEPSVM takes only 0.624 s, which is much less than the LGEPSVM. Obviously, the EL-GEPSVM increases the classification speed to a large extent (approximately 187% increase in speed) by reducing complexity in the convex hull construction. Due to the similarity in classification rules, both the LGEPSVM and EL-GEPSVM should compute distance for each point rather than compute in batches like the GEPSVM. In this case, they are slower than the GEPSVM. However, the EL-GEPSVM requires less storage room, which can be considered another advantage over the LGEPSVM and GEPSVM. Using the $\mathcal{E}$-score as an efficiency metric, the EL-GEPSVM achieves the highest score (1303.53), far surpassing the GEPSVM (121.22), LGEPSVM (43.67), and TWSVM (5.21), which highlights its outstanding efficiency advantage. We can see that although the TWSVM achieved the best performance on the muskclean data, it had the lowest $\mathcal{E}$-score in the comparison, indicating the low efficiency of the model. In contrast, the EL-GEPSVM obtained moderately acceptable performance with the least computational time and highest $\mathcal{E}$-score, showing its stronger ability to maintain a balanced trade-off between the performance and computational time.

4.6. Application to Real-World Data

We performed validation using our proposed EL-GEPSVM method in a real-world application. The dataset is collected for medical biometrics [34], and it contains multimodal data such as face, tongue, sublingual vein, pulse, odor, etc. The idea of medical biometrics is to perform computerized disease diagnosis by observing human biometrics characteristics. In this experiment, we adopted face biometrics to detect diabetes and heart disease. Follow the idea in [35], four blocks from specific regions from the candidate’s face are extracted in the experiment. Examples of facial blocks are shown in the Figure 6. We show the validation results in Table 5. As shown in the results, we can observe that the proposed EL-GEPSVM achieved satisfactory performance when detecting diabetes and heart disease. Noticeably, we can see the precision of block A when detecting diabetes is 100%, indicating that the EL-GEPSVM could precisely capture the characteristics of diabetes and made a highly separable decision boundary. We also recorded the computational time of disease detection. The proposed method could use 0.0033 s and 0.0019 s to perform the detection. This shows the EL-GEPSVM’s efficiency and practical applicability in a real-world application.

Figure 6. Samples of four blocks of diabetes, health control, and heart disease. A: block A, B: block B, C: block C, D: block D.

Table 5. Validation results of proposed method for medical biometrics.

Diabetes
Metric	A	B	C	D
Accuracy (%)	91.67	89.58	90.42	90.83
Precision (%)	100.00	83.33	91.67	84.62
Recall (%)	83.33	83.33	91.67	91.67
F1 (%)	90.91	83.33	91.67	88.00
Time (s)	0.0033	0.0043	0.0037	0.0033
Heart Disease
Metric	A	B	C	D
Accuracy (%)	73.00	84.29	90.00	82.86
Precision (%)	100.00	87.50	100.00	87.50
Recall (%)	80.00	100.00	85.71	100.00
F1 (%)	88.89	93.33	92.31	93.33
Time (s)	0.0019	0.0024	0.0027	0.0032

5. Conclusions

In this paper, we proposed the Ellipsoid-structured Localized Generalized Eigenvalue Proximal Support Vector Machine (EL-GEPSVM) for dealing with the underdetermined hyperplane problem (UPH). We utilized Mahalanobis distance for ellipsoidal convex hull construction to fully consider the local information and global information with statistical awareness. With the convex hull, the EL-GEPSVM unifies local and global information in pattern classification. In addition, we proved that the EL-GEPSVM is able to be kernelized for dealing with nonlinear data. The main novelty of this work lies in introducing the Mahalanobis-based ellipsoidal constraint into the GEPSVM framework, which enables adaptive class modeling through intra-class covariance and achieves a theoretically grounded distance-preserving property between Euclidean and Mahalanobis spaces. Furthermore, we established a kernelized dual representation for the M-norm, proving that the proposed ellipsoidal GEPSVM can be extended to nonlinear feature spaces while maintaining computational efficiency. Theoretical analysis and experiments showed that the EL-GEPSVM achieves higher accuracy and efficiency, especially in high-dimensional spaces. Experiments on several benchmarks show consistent gains over the GEPSVM/LGEPSVM and competitive baselines, together with a lower runtime in the linear case, indicating that modeling intra-class covariance and local geometry improves separability and stability of the learned hyperplanes. Since the establishment of the ellipsoidal convex hull in our proposed method does not organically integrate with GEPSVM optimization, in the future, we plan to explore the implementation of second-order cone programming to minimize the Mahalanobis distance on an ellipsoid under GEPSVM and LGEPSVM optimization frameworks. Furthermore, we will extend the EL-GEPSVM to larger and multimodal datasets using scalable kernel approximations and task-specific features.