Kernel-Based Optimal Subspaces (KOS): A Method for Data Classification

Abstract

Support Vector Machine (SVM) is a popular kernel-based method for data classification that has demonstrated high efficiency across a wide range of practical applications. However, SVM suffers from several limitations, including the potential failure of the optimization process, especially in high-dimensional spaces; the inherently high computational cost; the lack of a systematic approach to multi-class classification; difficulties in handling imbalanced classes; and the prohibitive cost of real-time or dynamic classification. This paper proposes an alternative method, referred to as Kernel-based Optimal Subspaces (KOS), which belongs to the family of kernel subspace methods. Mathematically similar to Kernel PCA (KPCA), KOS achieves performance comparable to SVM while addressing the aforementioned weaknesses. The method is based on computing the minimum distance to optimal feature subspaces of the mapped data. Because no optimization process is required, KOS is robust, fast, and easy to implement. The optimal subspaces are constructed independently, enabling high parallelizability and making the approach well-suited for dynamic classification and real-time applications. Furthermore, the issue of imbalanced classes is naturally handled by subdividing large classes into smaller sub-classes, thereby creating appropriately sized sub-subspaces within the feature space.

1. Introduction

Machine Learning (ML), a branch of Artificial Intelligence (AI), is the science of designing algorithms that can automatically improve by learning from data (see, for instance, [1]). This multidisciplinary field has a wide range of applications, including handwriting recognition, automated disease detection, robotics, and others.

Among the core problems in AI is data classification, which relies heavily on ML techniques. Kernel methods are one of the most popular subsets of data classifiers, with Support Vector Machine (SVM) being among the most widely used. The SVM algorithm is grounded in statistical learning theory, originating from Vapnik’s work on the Structural Risk Minimization principle [2,3,4].

The fundamental idea of kernel methods is to map the original data, referred to as attributes, into a higher-dimensional space, known as the feature space, where the data can be more easily separated. While SVM was originally designed for linear separation, this mapping enables it to handle data that are non-linearly separable in the original space by achieving linear separation in the feature space. Note that the mapping function is not explicitly defined, which is a tedious task; instead, a suitable kernel function is used, as long as all operations can be expressed using dot products. The existence of such kernels is guaranteed under certain conditions by Mercer’s theorem [4,5].

SVM is widely recognized for its high performance (see [6,7,8,9,10,11,12,13] for recent improvements). However, despite significant advancements, it still has several limitations:

• As an optimization problem, SVM may fail, particularly in high dimensions, even though the cost function is convex (quadratic).
• Extending SVM to multi-class classification is not straightforward and becomes computationally expensive in high dimensions or when the number of classes is large. For instance, a common approach combining one-against-one and one-against-all strategies requires computing a total of $n+{\displaystyle \frac{n(n-1)}{2}}$ separating hyperplanes (i.e., solving n optimization problems), where n is the number of classes.
• Imbalanced classes pose another challenge. A popular approach to addressing this involves adding synthetic attributes to underrepresented classes. While effective, the process is not straightforward and can alter the structure of the original class data.
• In dynamic classification, where classes may be added or removed, all separating hyperplanes must be recalculated, which can be prohibitively expensive for real-time applications.

To address these issues, a preliminary version of an alternative method, along with basic initial results, was introduced in a recent conference paper [14]. In this extended journal version, we provide the complete formulation of the method for centered data in the feature space, which offers more efficiency, along with a thorough theoretical analysis and corresponding proofs. Furthermore, we propose an effective strategy to tackle the inherent imbalanced class problem. Extensive experiments are conducted on large, multi-class datasets for a thorough evaluation of the method’s performance. In addition, a detailed analysis of time efficiency is included. Finally, when using the radial basis function (RBF) kernel, we describe a robust learning procedure for selecting the parameter $\sigma$.

In this work, the proposed method is referred to as Kernel-based Optimal Subspaces (KOS). This is a kernel subspace method [15,16,17] and is similar to Kernel PCA (KPCA) [18,19]. The approach relies, as in subspace methods, on computing a minimum distance between an unseen feature vector and a set of optimal subspaces, in a sense that will be defined, in the feature space. The unseen vector is then assigned to the appropriate attribute class. Proper orthogonal decomposition (POD) is used to build the feature subspaces, and their optimality is shown.

Fundamentally, the method requires the computation of the eigenpairs of the Mercer kernel matrix for each training class independently. No optimization is involved, making the method straightforward to implement, significantly faster than SVM (with linear complexity with respect to the number of classes), and robust.

KOS is highly parallelizable with respect to the number of classes and well-suited for dynamic and real-time classification, as only the eigenpairs for new classes need to be computed. If a class is removed, no further computation is required.

The issue of imbalanced classes is handled naturally by splitting large attribute classes into balanced sub-classes and then labeling all corresponding feature subspaces as belonging to a single class. An unseen feature vector is classified into a given class if it is close to any of the subspaces generated by that class’s sub-classes.

The performance of the proposed method is demonstrated through both 2D and high-dimensional, multi-class test cases. The results are compared against those of Support Vector Machine (SVM), using the popular LIBSVM software package [20]. The findings show that KOS performs comparably to SVM while offering several significant advantages, as discussed earlier, making it a strong alternative to SVM. Although the primary objective is to propose an alternative to SVM, the method is also compared with Kernel Neural Networks (KNNs) and Kernel PCA (KPCA).

The experiments use the radial basis function (RBF) kernel, which, as is well known in the context of SVM, involves tuning two key parameters: the standard deviation $\sigma$ (in the Gaussian form) or $\gamma$ (in the RBF form) and the penalty parameter C. The most widely used method for selecting these parameters is cross-validation-based grid search, as described in [21,22,23,24,25], including both automated and manual search approaches.

In contrast, the KOS method does not require the penalty parameter C. Only one parameter, $\sigma$ or $\gamma$, needs to be learned. Through a class-wise re-scaling process, this parameter is learned in a robust and effective manner, contributing to the overall performance of the method.

In Section 2, a short overview of proper orthogonal decomposition (POD) is provided first, then details of the proposed KOS method are developed. Tests to validate the proposed method by comparison to SVM results are performed and reported in Section 3. Conclusions are drawn in Section 4.

2. KOS: Kernel-Based Optimal Subspaces Method

Before detailing the KOS method, we first present a brief overview of the proper orthogonal decomposition technique, which is a crucial component of the approach.

2.1. Proper Orthogonal Decomposition (POD): Overview

Proper orthogonal decomposition (POD) [26], also known as Karhunen–Loeve expansion (KLE) [26], popular in the data analysis field, is a technique that aims to represent a big number of data by a reduced number of basis elements built from the data. More precisely, if the data are stored in an $m\times N$ matrix A, with N being a large number, it is possible to represent the N columns of A by p orthonormal vectors, with p being very small compared with N. The mathematical formulation can be summarized as follows: Let $A=\left[Y_1,Y_2,\dots ,Y_N\right]$; the problem is to find $B=\left[V^1,V^2,\dots ,V^p\right]$, an orthonormal set of p vectors that better represent the N columns of A in the following sense.

$V^1$ satisfies the optimization problem

$$\begin{array}{c}\hfill V^1=Arg[Ma{x}_{V\in I{R}^m}\left(\sum _{i=1}^N{\left(〈V,Y_i〉\right)}^2\right)\\ \hfill {\mathit{subject}\mathit{to}\parallel V\parallel }^2=1].\end{array}$$

(1)

$$\begin{array}{c}\hfill V^2=Arg[Ma{x}_{V\in I{R}^m}\left(\sum _{i=1}^N{\left|〈V,Y_i〉\right|}^2\right)\\ \hfill {\mathit{subject}\mathit{to}\parallel V\parallel }^2=1\mathit{and}〈V,V^1〉=0].\end{array}$$

(2)

$$⋮$$

$$\begin{array}{c}\hfill V^k=Arg[Ma{x}_{v\in I{R}^m}\left(\sum _{i=1}^N{\left|〈V,Y_i〉\right|}^2\right)\\ \hfill {\mathit{subject}\mathit{to}\parallel V\parallel }^2=1\\ \hfill \mathit{and}〈V,V^1〉=0,〈V,V^2〉=0,\dots ,〈V,V^{k-1}〉=0].\end{array}$$

(3)

The vectors $V^k,k=1,p$, are the eigenvectors of $A{A}^T$. A simplified proof is provided in Appendix A for self-content. For more details, see [26].

POD Basis System

The POD basis system is built as follows:

1.

If $n\le m$, let M be the correlation matrix $M_{i,j}={\left(A^{T}A\right)}_{i,j}$ where $(,)$ is any inner product. Let V be the matrix of eigenvectors of M; then the basis functions ${\mathrm{\Psi }}_i$, called modes, are given by:

$${\mathrm{\Psi }}_i=\sum _{j=1}^N{v}_j^i{Y}_j$$

(4)

where $v_j^i$ are the components of the ith eigenvector of M.

2.

If $n>m$, the POD basis is defined by the eigenvectors of the matrix ${\overline{M}}_{i,j}={\left(A{A}^T\right)}_{i,j}$.

This paper will use the first definition since the original data will be mapped into an infinite-dimension vector space.

Now calculate the inner product $({\mathrm{\Psi }}_i,{\mathrm{\Psi }}_j)$:

$$\begin{array}{c}\hfill ({\mathrm{\Psi }}_i,{\mathrm{\Psi }}_j)=\\ \hfill (\sum _{l=1}^N{v}_l^i{Y}_l,\sum _{k=1}^N{v}_k^j,Y_k)=\sum _{l,k}{v}_l^i{v}_k^j(Y_l,Y_k)=\sum _{l,k}{v}_l^i{v}_k^j{M}_{l,k}=\\ \hfill M{V}^i.V^j={\lambda }_i{V}^i.V^j\end{array}$$

(5)

This shows that the functions ${\mathrm{\Psi }}_i,1\le i\le N$, are orthogonal if and only the eigenvectors $V^i,1\le i\le N$, are, which is generally the case. For $i=j$ we get the norm $\parallel {\mathrm{\Psi }}_i\parallel =\sqrt{{\lambda }_i}$, which shows that the norm of each mode represents part of the energy of the system. In conclusion, modes with low energy (small eigenvalues) can be neglected, reducing the POD basis to only modes with significant energy. Practically, the ratio of the modeled to the total energy contained in the system given by

$$\epsilon \left(l\right)={\displaystyle \frac{{\sum }_{i=1}^l{\lambda }_i}{{\sum }_{i=1}^N{\lambda }_i}}$$

(6)

is calculated to decide if l modes are enough to describe the system.

2.2. KOS: Global Description and Some Theoretical Aspects

Similar to SVM and other kernel methods, KOS first maps the original data into a high-dimensional feature space using a kernel function. For each class, an optimal subspace that best fits its feature vectors is constructed. An unseen sample is classified, as is common in subspace-based methods, by projecting its feature vector onto each class subspace and assigning the label corresponding to the minimum distance. Unlike SVM, KOS replaces separating hyperplanes, often a source of practical limitations, with optimal class subspaces that directly model the feature data, as illustrated in Figure 1. A summary of the method is provided in Algorithm 1.

Note that in some cases, different classes may be mapped to the same feature subspace, leading to a misclassification. Therefore, it is desirable for the subspaces to be as distinct as possible to reduce this risk. In theory, as proved later in this paper, it is always possible to construct subspaces that are even completely disjoint. However, in practice, this is cannot be guaranteed or controlled. For optimal performance, the kernel should be chosen such that the resulting feature space is of infinite dimension, which increases the likelihood of separability between class subspaces. It is well established, for example, that the RBF kernel meets this requirement.

Algorithm 1: Main KOS Steps

Let $C_{k,k=1,P}$ denote P attribute sets representing P different classes used during the learning stage. Let $B_{k=1,P}$ denote the corresponding mapped sets in the feature space. The KOS classification procedure is as follows: 1. Check whether the classes $C_k$ are imbalanced. If so, split the larger classes into smaller sub-classes according to procedure described in Section 2.6. 2. Select a Mercer kernel K, then compute the kernel matrices $K_k$ for each feature class $C_k$ (of feature sub-classes if applicable). 3. Compute the POD correlation matrices $M_k$ for each feature class (or sub-class) $C_k$ using either non-centered formulation (9) or centered formulation (14). 4. Compute the eigenvalues and eigenvectors of each matrix $M_{k,k=1,P}$. 5. For each unseen vector $\widehat{X}$, compute its coordinates ${\alpha }_i^k,i=1,size\left(C_k\right)$ in each POD feature subspace using Formula (20) for the non-centered case or Formula (26) for the centered case. 6. Compute the distance of $\widehat{X}$ to each POD feature subspace using Formula (21) for the non-centered case or Formula (27) for the centered case. 7. Decision: Assign $\widehat{X}$ to the class corresponding to the minimum computed distance.

Theorem 1.

There exist Mercer kernels such that the attribute classes can be mapped into disjoint feature subspaces.

Proof. 

For simplicity, the proof is given for the two-class case; the result can then be extended recursively to the multi-class case. Let $\chi$ denote the attribute (original) space and let $\mathit{H}$ be the feature space obtained through a mapping $\mathrm{\Phi }$ associated with a Mercer kernel $K(x,y)$. Suppose that $K(x,y)$ is such that $\mathit{H}$ is infinite-dimensional.

$$\mathrm{\Phi }:\chi ⟶\mathit{H}$$

(7)

Let $C_1$ and $C_2$ be two classes, and let $\mathit{E}$ and $\mathit{F}$ be two finite-dimensional subspaces of the feature space $\mathit{H}$ such that $\mathrm{\Phi }\left(C_1\right)\subset E$ and $\mathrm{\Phi }\left(C_2\right)\subset F$. Let ${\mathit{E}}^T$ denote the orthogonal complement of $\mathit{E}$ in $\mathit{H}$. Since $\mathit{H}$ is infinite-dimensional and $\mathit{E}$ is finite-dimensional, $\mathit{E}$ is also infinite-dimensional.

Let $\mathit{G}\subset {\mathit{E}}^T$ be any subspace with the same dimension as $\mathit{F}$. Then there exist a one-to-one mapping $\varphi :\mathit{F}\to \mathit{G}$.

Now, define a new mapping $\psi :\chi ⟶\mathit{H}$ as follows:

$$\psi \left(x\right)=\left\{\begin{array}{c}\varphi \left(\mathrm{\Phi }\left(x\right)\right)ifx\in C_2\hfill \\ \mathrm{\Phi }\left(x\right)ifnot\hfill \end{array}\right.$$

(8)

We have $\mathrm{\Phi }\left(C_1\right)\subset E$ and $\mathrm{\Phi }\left(C_2\right)\subset G$ with E and G orthogonal then disjoint. The kernel defined by $\overline{K}(x,y)=\psi \left(x\right)\ast \psi \left(y\right)$ is a Mercer kernel by construction and satisfies the theorem, which ends the proof. □

2.3. POD Feature Subspaces

The POD technique is proposed to build the optimal subspaces in the feature vector space. Note that similar techniques, such as PCA (Principal Component Analysis) and KPCA (Kernel Principal Component Analysis), are commonly used in kernel theory for data classification, typically as a preprocessing step for feature extraction and enhancement [27]. The optimality of the POD feature subspaces will be discussed later.

Assume that a suitable kernel $K(x,y)$ is given, and let $\varphi$ be the associated mapping function. It is important to note that choosing a kernel such that the feature space $\mathit{H}$ is of infinite dimension is highly recommended, as it helps map data into subspaces with minimal or no overlap.

Let $C_{l,l=1,P}$ denote the P attribute classes. Denote by $X_i^l$ the ith element of class $C_l$ and by $Y_i^l=\varphi \left(X_i^l\right)$ its corresponding mapped feature vector. To construct the POD feature subspaces, we only need to build the POD basis, also known as the modes, as described in Section 2.1.

In general, it is well established that centering the data is recommended when applying the POD method. Therefore, we provide formulations of the POD basis for both non-centered and centered data.

To lighten the notations, the superscript l, which refers to a specific class, is omitted and will be reintroduced as needed. In the following, we use $K_{i,j}$ to denote the Mercer’ kernel K matrix, namely, $K_{i,j}=Y_i{Y}_j=\varphi \left(X_i\right)\varphi \left(X_j\right)$, and $K(X,Y)$ when X or Y does not belong to the training dataset.

1.

Non-centered data formulation: In this case, the data in the feature space are used without centering. Let $Y_i=\varphi \left(X_i\right),i=1,N$ be the mapped data points of a selected class of size N. First, we construct the correlation matrix:

$$M(i,j)=Y_i{Y}_j=\varphi \left(X_i\right)\varphi \left(X_j\right)=K(i,j),$$

(9)

where $K(i,j)$ denotes the kernel function evaluated at $Y_i$ and $Y_j$. As seen, the POD correlation matrix is equivalent to the kernel matrix derived from the mapping.

The normalized POD modes are then given by

$${\psi }_i={\displaystyle \frac{1}{{\sqrt{\sigma }}_i}}\sum _{k=1}^N{V}_k^i{Y}_k,$$

(10)

where $V^i$ is the ith eigenvector of the kernel matrix M and ${\sigma }_i$ the associated eigenvalue.

2.

Centered data formulation: In this case, the feature vectors $Y_i$ are centered. Let $\overline{Y}={\displaystyle \frac{1}{N}}{\sum }_{k=1}^N{Y}_k$ be the mean vector and $Z_i=Y_i-\overline{Y}$ be the centered data. The correlation matrix is then given by:

$$M(i,j)=Z_i{Z}_j=(Y_i-\overline{Y})(Y_j-\overline{Y})=Y_i{Y}_j-Y_i\overline{Y}-Y_j\overline{Y}+\overline{Y}\overline{Y}.$$

(11)

Each of the inner products can be expressed in terms of the kernel function:

$$Y_i\overline{Y}=Y_i{\displaystyle \frac{1}{N}}\sum _{k=1}^N{Y}_k={\displaystyle \frac{1}{N}}\sum _{k=1}^N{Y}_i{Y}_k={\displaystyle \frac{1}{N}}\sum _{k=1}^{N}K(i,k),$$

(12)

$$\overline{Y}\overline{Y}={\displaystyle \frac{1}{N}}\sum _{k=1}^N{Y}_k{\displaystyle \frac{1}{N}}\sum _{l=1}^N{Y}_l={\displaystyle \frac{1}{N^2}}\sum _{k=1}^N\sum _{l=1}^{N}K(l,k).$$

(13)

By substituting back into the expression for $M(i,j)$, the centered correlation matrix becomes:

$$M(i,j)=K(i,j)-{\displaystyle \frac{1}{N}}\sum _{k=1}^N(K(i,k)+K(j,k))+{\displaystyle \frac{1}{N^2}}\sum _{k=1}^N\sum _{l=1}^{N}K(l,k).$$

(14)

As in the non-centered case, the correlation matrix can be entirely expressed using the kernel matrix.

The normalized POD modes for centered data are then given by

$${\psi }_i={\displaystyle \frac{1}{{\sqrt{\sigma }}_i}}\sum _{k=1}^N{V}_k^i(Y_k-\overline{Y})$$

(15)

where $V^i$ is the ith eigenvector of the correlation matrix M defined by (14) and ${\sigma }_i$ the associated eigenvalue.

Note that for calculating the minimum distance in classification (as will be shown later), it is not necessary to explicitly compute the feature vectors $Y_k$ or the POD modes.

2.4. Optimality of the POD Feature Subspaces

The optimality of the feature subspaces is considered from both geometric and algebraic perspectives.

From a geometric viewpoint, the subspace basis should approximate the original feature vectors as closely as possible, thereby preserving the intrinsic structure of the data in the feature space.

From an algebraic viewpoint, the feature subspaces should be the smallest, in terms of dimension, subspaces that contain the feature vectors, thereby enhancing discrimination.

In essence, the POD method provides a set of orthonormal basis functions that span the most “informative” subspace, allowing for an efficient and discriminative representation of each class in the feature space.

• Geometrical Optimality of POD Feature Spaces:
  As shown in [26], if $\mathrm{\Psi }$ is the POD basis matrix constructed to represent the columns of a matrix $A=\left[Y_1,Y_2,\dots ,Y_N\right]$ and $\overline{\mathrm{\Psi }}$ is any other basis of the same size constructed for the same purpose, then the following inequality holds:

  $${\parallel A-\mathrm{\Psi }\parallel }_F\le {\parallel A-\overline{\mathrm{\Psi }}\parallel }_F,$$

  (16)

  where ${\parallel .\parallel }_F$ is the Frobenius norm, defined by

  $${\parallel A\parallel }_F=\sqrt{\sum _{i=1}^m\sum _{j=1}^N\left|A_{i,j}\right|}=\sqrt{\mathit{Trace}\left(A^{T}A\right)}$$

  (17)
  This result demonstrates the geometric optimality of the POD basis: among all possible orthonormal bases of the same dimension, the POD basis provides the best approximation (in the least-squares sense) of the original data matrix A. Therefore, the POD basis vectors are the closest possible representatives of the original feature vectors, ensuring maximum fidelity in capturing the data structure.
• Algebraic Optimality of POD Feature Spaces:

Lemma 1.

The POD subspace is algebraically optimal; it is the smallest subspace containing the mapped data.

Proof. 

The proof is straightforward. To show that the POD subspace is the smallest subspace containing the mapped (feature) data, it is sufficient to show that any subspace containing the feature data must also contain the POD subspace. □

Let F be a subspace that contains the given feature data. To prove that F also contains the POD subspace, it suffices to show that F contains the POD modes (i.e., the basis of the POD subspace). Since the POD modes are linear combinations of the feature data and F is a subspace (i.e., closed under linear combinations), the modes necessarily lie in F. Hence, the POD subspace is contained in any subspace that contains the mapped data, proving its algebraic optimality.

2.5. Decision Criterion

The decision criterion for an unseen sample $\widehat{X}$ is, as in subspace-based methods, based on the minimum distance between its feature vector $\widehat{Y}=\varphi \left(\widehat{X}\right)$ and the POD feature subspaces. This requires computing the POD coordinates of $\widehat{Y}$ within each subspace. The procedures for both non-centered and centered data are detailed below:

1.

Non-centered data formulation: The coordinates ${\alpha }_i,i=1,N$, are given by the projection of the element on the POD modes

$${\alpha }_i=<{\psi }_i,\widehat{Y}>$$

(18)

with

$${\psi }_i={\displaystyle \frac{1}{{\sqrt{\sigma }}_i}}\sum _{k=1}^N{V}_k^i{Y}_k,$$

(19)

Then

$${\alpha }_i={\displaystyle \frac{1}{{\sqrt{\sigma }}_i}}<\sum _{k=1}^N{V}_k^i{Y}_k,\widehat{Y}>={\displaystyle \frac{1}{{\sqrt{\sigma }}_i}}\sum _{k=1}^N{V}_k^i\left(Y_k\widehat{Y}\right)={\displaystyle \frac{1}{{\sqrt{\sigma }}_i}}\sum _{k=1}^N{V}_k^{i}K(X_k,\widehat{X})$$

(20)

Using the Pythagoras rule, the distance of $\overline{Y}$ to the POD subspace F is given by

$$dis(\widehat{Y},F)=\sqrt{\parallel \widehat{Y}{\parallel }^2-\sum _{i=1}^N{\alpha }_i^2}.$$

(21)

In the above,

$$\parallel \widehat{Y}{\parallel }^2=\widehat{Y}\widehat{Y}=\varphi \left(\widehat{X}\right)\varphi \left(\widehat{X}\right)=K(\widehat{X},\widehat{X}).$$

(22)

2.

Centered data formulation: For the centered case, $\widehat{Y}$ and $Y_i$ are replaced by $\widehat{Y}-\overline{Y}$ and $Y_i-\overline{Y}$ respectively. That is,

$${\alpha }_i=<{\psi }_i,(\widehat{Y}-\overline{Y})>,$$

(23)

with

$${\psi }_i={\displaystyle \frac{1}{{\sqrt{\sigma }}_i}}\sum _{k=1}^N{V}_k^i(Y_k-\overline{Y}),$$

(24)

Then

$$\begin{array}{c}\hfill {\alpha }_i={\displaystyle \frac{1}{{\sqrt{\sigma }}_i}}<\sum _{k=1}^N{V}_k^i(Y_k-\overline{Y}),(\widehat{Y}-\overline{Y})>=\\ \hfill {\displaystyle \frac{1}{{\sqrt{\sigma }}_i}}[\sum _{k=1}^N{V}_k^i\left(Y_k\widehat{Y}\right)-\sum _{k=1}^N{V}_k^i\left(Y_k\overline{Y}\right)\\ \hfill -\sum _{k=1}^N{V}_k^i\overline{Y}\widehat{Y}+\sum _{k=1}^N{V}_k^i\overline{Y}\overline{Y}].\end{array}$$

(25)

By substituting $\overline{Y}={\displaystyle \frac{1}{N}}{\sum }_{j=1}^N{Y}_j$ we obtain

$$\begin{array}{c}\hfill {\alpha }_i={\displaystyle \frac{1}{{\sqrt{\sigma }}_i}}[\sum _{k=1}^N{V}_k^{i}K(X_k,\widehat{X})-{\displaystyle \frac{1}{N}}\sum _{k=1}^N\sum _{j=1}^N{V}_k^i{K}_{k,j}\\ \hfill -\left({\displaystyle \frac{1}{N}}\sum _{j=1}^{N}K(X_j,\widehat{X})\right)\sum _{k=1}^N{V}_k^i+\left({\displaystyle \frac{1}{N^2}}\sum _{k=1}^N\sum _{j=1}^N{K}_{k,j}\right)\sum _{k=1}^N{V}_k^i].\end{array}$$

(26)

Then

$$dis(\widehat{Y},F)=\sqrt{\parallel \widehat{Y}-\overline{Y}{\parallel }^2-\sum _{i=1}^N{\alpha }_i^2}.$$

(27)

In the above,

$$\parallel \widehat{Y}-\overline{Y}{\parallel }^2={\widehat{Y}}^2-2\widehat{Y}\overline{Y}+{\overline{Y}}^2=K(\widehat{X},\widehat{X})-{\displaystyle \frac{2}{N}}\sum _{j=1}^{N}K(\widehat{X},X_j)+{\displaystyle \frac{1}{N^2}}\sum _{k=1}^N\sum _{j=1}^N{K}_{k,j}.$$

(28)

The unseen attribute vector is assigned to the class that corresponds to the minimum distance calculated above.

2.6. Imbalanced Classes

Class imbalance is a common issue in many classifiers, including SVM. Popular approaches include adjusting class weights (as in LIBSVM) and the SMOTE algorithm [28], which generates synthetic samples for smaller classes. While both methods show acceptable effectiveness, they lack a systematic way to determine weight distributions or the placement of synthetic vectors, potentially altering the data structure.

The proposed KOS method addresses this naturally by subdividing large attribute classes into smaller sub-classes of similar sizes. Each sub-class retains the original class label, effectively representing large subspaces as a collection of smaller, more balanced subspaces.

Specifically, a reference size, often the size of the smallest class, is defined. Any class at least twice this size is split into subsets roughly equal to the reference size. For classification, if the minimum distance is obtained from a POD subspace of a subdivided class, the unseen sample is assigned to that class. This strategy is illustrated in Figure 2 and validated in the test section.

2.7. RBF Parameter Learning

As previously mentioned, when using the RBF kernel, only the parameter $\sigma$ (equivalently $\gamma$) needs to be learned, as in the case of hard-margin SVM. There is no need for a penalty parameter.

To enable a fast and efficient search algorithm, we propose scaling $\sigma$ according to the class attribute diameter. This approach is motivated by multi-scale theory in image processing (see, for instance, [29,30]), where the Gaussian kernel is used to extract features at different scales corresponding to different values of $\sigma$. The latter determines the width of the Gaussian, allowing each object to be observed at an appropriate scale. Small values of $\sigma$ correspond to fine image details, whereas larger values capture more global structures.

Applying this idea to each class and selecting a value of $\sigma$ proportional to the class diameter allow the entire class structure to be captured. Only minor adjustments are then required to avoid missing important details or overfitting. Moreover, vectors that are far apart may produce many entries of the RBF co-variance matrix close to 0, which can lead to an ill-conditioned matrix. Choosing $\sigma$ proportional to the class diameter helps avoid this issue.

We decompose $\sigma$ as follows:

$$\sigma =\omega D,$$

(29)

where D is the diameter of the set, computed as the maximum distance between any two elements in the set. The scaling factor $\omega$ is varied from $0.1$ to 1 in increments of $0.1$. This strategy significantly reduces the complexity of the parameter search.

For comparison, in standard SVM, approximately $250\times 250$ grid searches are typically required to determine the optimal parameter pair $(\sigma ,C)$, where C is the penalty parameter.

2.8. Some Remarks

PCA and POD share the same mathematical foundation but differ in data interpretation. Since the original data are mapped into an infinite-dimensional space, where the dimensionality exceeds the number of samples, POD modes are more appropriately computed using (10) or (15) and the eigenvectors of $Y^{T}Y$ [26], rather than the eigenvectors of $Y{Y}^T$ as in KPCA. This approach simplifies the projection process.

Additionally, POD treats the data as snapshots, interpreting the entire class as a dynamical system in which each element represents a temporal deformation of a standard reference. From this perspective, POD aims to optimize the system’s energy content. This distinction is meaningful, as will be shown in the test section along with the proposed strategy for handling imbalanced classes. Notably, the POD-snapshots approach is widely used in fluid dynamics for reduced-order modeling.

2.9. Summary of the Algorithm

The KOS method is summarized below, and its main computational steps are described to facilitate practical implementation.

3. Results: Validation and Discussion

The KOS method is verified and validated using the RBF kernel. Initial tests are conducted on 2D cases to visually assess its ability to handle non-linear classification. Subsequently, KOS is tested on selected real-world datasets from the LIBSVM [20] and OpenML [31] repositories, and the results are compared against those obtained using SVM implemented in the widely used LIBSVM software package. While the main objective is to propose an alternative to SVM, the performance and computational cost of the method are also compared with Kernel Neural Networks (KNNs) and Kernel PCA (KPCA). Accuracy and macro-precision are used as performance metrics.

It is important to note that no specific preprocessing is applied to the data for KOS, apart from re-scaling the input features to the interval $[-1,1]$.

3.1. 2D Test Cases

To evaluate KOS’s capability to separate non-linearly separable data, three increasingly challenging scenarios are tested: (i) connected sets, (ii) non-connected sets, and (iii) a spiral configuration. Figure 3, Figure 4 and Figure 5 demonstrate that KOS successfully separates the data in all cases. A fixed kernel parameter value of $\sigma =1.2$ is used for all three tests.

3.2. Higher-Dimension Test Cases

Two-class and multi-class test cases are selected from the LIBSVM [20] and OpenML [31] databases. For all experiments, accuracy and macro-precision are computed to evaluate performance.

3.2.1. KOS vs. SVM

KOS and SVM (using LIBSVM software) are tested, and their performances compared. Note that for KOS, there is no penalty parameter C; only the standard deviation $\sigma$ is learned. This is done by splitting the training set into two parts: 2/3 for building the KOS model (eigenpairs of the correlation matrices) and 1/3 for validation. The value of $\sigma$ is varied from $\sigma =0.1$, to $\sigma =1$ in increments of $0.1$. Once the optimal $\sigma$ is found, the entire training set is used to build the KOS model. The tests are briefly described in

Table 1. Test information (two classes).

Test Name	No. of Classes	Description/Reference	Training Set Size/ Class Sizes	Test Set Size/ Feature Vector Size
Leukemia	2	Molecular classification of cancer [32]	$38/$ size$\left(C_1\right)$ = 27 size$\left(C_2\right)=11$	$34/$ 7129
svmguide1	2	Astroparticle application [33]	$3089/$ size$\left(C_1\right)=200$ size$\left(C_2\right)=200$	$4000/$ 4
splice	2	Splice junctions in a DNA sequence [34]	$1000/$ size$\left(C_1\right)=483$ size$\left(C_2\right)=517$	$2175/$ 60
austrian	2	Credit Approval dataset [35]	$349/$ size$\left(C_1\right)=191$ size$\left(C_2\right)=158$	$341/$ 14
madelon	2	Analysis of the NIPS 2003 [36]	$200/$ size$\left(C_1\right)=1000$ size$\left(C_2\right)=1000$	$600/$ 14

and

Table 2. Test information (multiple classes).

Test Name	No. of Classes	Description/ Reference	Training Set Size/ Class Sizes	Test Set Size/ Feature Vector Size
DNA	3	DNA [37]	$2000/$ size$\left(C_1\right)=464$ size$\left(C_2\right)=485$ size$\left(C_3\right)=1051$	$1186/$ 180
Satimage	6	Satellite images [37]	4435 size$\left(C_1\right)=1072$ size$\left(C_2\right)=479$ size$\left(C_3\right)=961$ size$\left(C_4\right)=415$ size$\left(C_5\right)=470$ size$\left(C_6\right)=1038$	$2000/$ 36
USPS	10	Handwritten text recognition dataset [38]	$7291/$ size$\left(C_1\right)=1194$ size$\left(C_2\right)=1005$ size$\left(C_3\right)=731$ size$\left(C_4\right)=658$ size$\left(C_5\right)=652$ size$\left(C_6\right)=556$ size$\left(C_7\right)=664$ size$\left(C_8\right)=645$ size$\left(C_9\right)=542$ size$\left(C_{10}\right)=644$	$2007/$ 256
letter	26	Letter recognition dataset [38]	15,000/ size$\left(C_1\right)=789$ size$\left(C_2\right)=766$ size$\left(C_3\right)=736$ size$\left(C_4\right)=805$ size$\left(C_5\right)=768$ size$\left(C_6\right)=775$ size$\left(C_7\right)=773$ size$\left(C_8\right)=734$ size$\left(C_9\right)=755$ size$\left(C_{10}\right)=747$ size$\left(C_{11}\right)=739$ size$\left(C_{12}\right)=761$ size$\left(C_{13}\right)=792$ size$\left(C_{14}\right)=783$ size$\left(C_{15}\right)=753$ size$\left(C_{16}\right)=803$ size$\left(C_{17}\right)=783$ size$\left(C_{18}\right)=758$ size$\left(C_{19}\right)=748$ size$\left(C_{20}\right)=796$ size$\left(C_{21}\right)=813$ size$\left(C_{22}\right)=764$ size$\left(C_{23}\right)=752$ size$\left(C_{24}\right)=787$ size$\left(C_{25}\right)=786$ size$\left(C_{26}\right)=734$	$500/$ 16
shuttle	7	Space shuttle sensors [38]	43,500/ size$\left(C_1\right)=$ 34,108 size$\left(C_2\right)=37$ size$\left(C_3\right)=132$ size$\left(C_4\right)=6748$ size$\left(C_5\right)=2458$ size$\left(C_6\right)=6$ size$\left(C_7\right)=11$	14,500/ 9

.

Table 3. KOS-SVM performance comparison on LIBSVM database.

Name/Classes	Test Size	SVM Accuracy|Precision	KOS Accuracy|Precision
Leukemia/2	Train.#38 Test.#34	$82.35\%|86.42\%$	$\mathbf{82}.\mathbf{35}\%|\mathbf{82}.\mathbf{58}\%$
svmguide1/2	Train.#3089 Test.#4000	$96.87\%|96.87\%$	$\mathbf{96}.\mathbf{30}\%|\mathbf{96}.\mathbf{31}\%$
splice/2	Train.#1000 Test.#2175	$90.43\%|90.50\%$	$\mathbf{89}.\mathbf{42}\%|\mathbf{89}.\mathbf{41}\%$
austrian/2	Train.#349 Test.#341	$85.04\%|85.01\%$	$\mathbf{85}.\mathbf{63}\%|\mathbf{85}.\mathbf{36}\%$
madelon/2	Train.#2000 Test.#600	$61.16\%|59.51\%$	$\mathbf{61}.\mathbf{50}\%|\mathbf{61}.\mathbf{55}\%$
DNA/3	Train.#2000 Test.#1186	$94.43\%|94.14\%$	$\mathbf{92}.\mathbf{32}\%|\mathbf{92}.\mathbf{83}\%$
Satimage/6	Train.#4435 Test.#200	$91.85\%|91.68\%$	$\mathbf{91}.\mathbf{35}\%|\mathbf{90}.\mathbf{40}\%$
USPS/10	Train.#7291 Test.#2007	$95.26\%|95.32\%$	$\mathbf{95}.\mathbf{76}\%|\mathbf{95}.\mathbf{76}\%$
letter/26	Train.#15,000 Test.#5000	$97.9\%|97.90\%$	$\mathbf{97}.\mathbf{42}\%|\mathbf{97}.\mathbf{44}\%$
shuttle/7	Train.#43,500 Test.#14,500	$99.9\%|99.93\%$	$\mathbf{99}.\mathbf{91}\%|\mathbf{99}.\mathbf{91}\%$

and

Table 4. KOS-SVM performance comparison on OpenML database.

Name/Classes	Test Size	SVM Accuracy|Precision	KOS Accuracy|Precision
BreastCancer/2	Train.#398 Test.#171	$97.07\%|97.07\%$	$\mathbf{97}.\mathbf{07}\%|\mathbf{97}.\mathbf{01}\%$
Zernike/10	Train.#1400 Test.#600	$82.00\%|82.03\%$	$\mathbf{81}.\mathbf{00}\%|\mathbf{80}.\mathbf{82}\%$
Diabetes/2	Train.#537 Test.#231	$74.026\%|74.40\%$	$\mathbf{74}.\mathbf{46}\%|\mathbf{72}.\mathbf{20}\%$
mfeat-morphological/10	Train.#1400 Test.#600	$73.66\%|69.36\%$	$\mathbf{73}.\mathbf{833}\%|\mathbf{75}.\mathbf{07}\%$

compare the performance of KOS and SVM. Overall, the tables show that KOS performs comparably to SVM while offering significant advantages, outlined in the next subsection.

The effectiveness of the proposed imbalanced strategy is demonstrated, for instance, in the Leukemia and Shuffle tests, which contain the most imbalanced classes. Specifically, the performance rates without splitting the classes were $67.64\%$ and $80.6\%$, respectively. After applying the splitting strategy, these scores improved to $82.35\%$ and $99.9\%$, respectively.

3.2.2. Comparison with KNN and KPCA

The Kernel Neural Network (KNN) used in this study is a Multilayer Perceptron (MLP) with two hidden layers of 64 and 32 neurons, respectively. Note that in this model, the kernel is used only for feature extraction and enhancement. Both the KNN and KPCA are implemented using the TensorFlow library [39].

Table 5. KOS-KNN-KPCA performance comparison on LIBSVM and OpenML databases.

Name/Classes	KNN Accuracy|Precision	KPCA Accuracy|Precision	KOS Accuracy|Precision
Leukemia/2	$67.6\%|86.42\%$	$85.29\%|84.51\%$	$\mathbf{82}.\mathbf{3}\%|\mathbf{82}.\mathbf{58}\%$
svmguide1/2	$96.5\%|96.5\%$	$88.98\%|88.94\%$	$\mathbf{96}.\mathbf{30}\%|\mathbf{96}.\mathbf{31}\%$
splice/2	$86.3\%|86.3\%$	$83.49\%|83.50\%$	$\mathbf{89}.\mathbf{42}\%|\mathbf{89}.\mathbf{41}\%$
austrian/2	$84.5\%|84.2\%$	$85.04\%|84.98\%$	$\mathbf{85}.\mathbf{63}\%|\mathbf{85}.\mathbf{36}\%$
madelon/2	$70.5\%|71.2\%$	$58.33\%|58.33\%$	$\mathbf{61}.\mathbf{50}\%|\mathbf{61}.\mathbf{55}\%$
DNA/3	$90.1\%|89.5\%$	$89.04\%|89.12\%$	$\mathbf{92}.\mathbf{32}\%|\mathbf{92}.\mathbf{83}\%$
Satimage/6	$90.0\%|89.2\%$	$83.49\%|83.50\%$	$\mathbf{91}.\mathbf{35}\%|\mathbf{90}.\mathbf{40}\%$
USPS/10	$92.9\%|95.32\%$	$66.62\%|60.49\%$	$\mathbf{95}.\mathbf{76}\%|\mathbf{95}.\mathbf{76}\%$
letter/26	$90.50\%|92.3\%$	$61.56\%|61.12\%$	$\mathbf{97}.\mathbf{42}\%|\mathbf{97}.\mathbf{44}\%$
shuttle/7	$93.8\%|93.3\%$	$84.78\%|85.50\%$	$\mathbf{99}.\mathbf{91}\%|\mathbf{99}.\mathbf{91}\%$
BreastCancer/2	$97.1\%|96.3\%$	$93.57\%|93.53\%$	$\mathbf{97}.\mathbf{076}\%|\mathbf{97}.\mathbf{01}\%$
Zernike/10	$80.8\%|80.2\%$	$72.50\%|72.25\%$	$\mathbf{81}.\mathbf{00}\%|\mathbf{80}.\mathbf{82}\%$
Diabetes/2	$73.6\%|71.0\%$	$69.26\%|69.90\%$	$\mathbf{74}.\mathbf{46}\%|\mathbf{72}.\mathbf{20}\%$
mfeat-morphological/10	$69.0\%|68.7\%$	$70.00\%|69.79\%$	$\mathbf{73}.\mathbf{83}\%|\mathbf{75}.\mathbf{07}\%$

reports the obtained results, which indicate that KOS (and similarly SVM) generally outperforms the KNN, particularly on imbalanced datasets such as Leukemia. A similar conclusion holds for KPCA, even though KPCA is conceptually closer to KOS. This difference, as discussed in Section 2.8, can be attributed to the snapshot POD formulation. On one hand, this formulation makes the computation of distances to subspaces in the feature space straightforward; on the other hand, it treats each class as a dynamical system, where each vector is interpreted as a temporal deformation of a representative element, thereby reinforcing intra-class cohesion. Moreover, KPCA computes distances to subspaces through reconstruction via the inverse transform, which introduces an optimization step that negatively impacts both accuracy and computation time. The proposed imbalance classes-handling strategy further contributes to improving the performance of KOS on classes that typically exhibit such behavior.

3.2.3. Feature Space Orthogonality

We proved in Section 2.2 that from a theoretical perspective, it is always possible for feature subspaces to be distinct and even orthogonal. However, in practice, this is not guaranteed. Considering kernels whose feature subspaces are infinite-dimensional (such as the RBF kernel) increases the likelihood that orthogonality occurs, thereby improving the efficiency of the method. In this section, to quantify the separation between subspaces in the conducted experiments, the principal angles between subspaces and the chordal metric are computed.

Let E and F be two subspaces of dimensions p and m, respectively. Let ${\left\{{\phi }_i^E\right\}}_{i=1}^p$ and ${\left\{{\phi }_j^F\right\}}_{j=1}^m$ be orthonormal bases of E and F. The orthogonality between the two subspaces is quantified using the principal angles ${\mathrm{\Theta }}_k$, for $k=1,\dots ,min(p,m)$, defined as follows.

First, compute the $p\times m$ correlation matrix between the two subspaces:

$$K^{E,F}(i,j)=\langle {\phi }_i^E,{\phi }_j^F\rangle ,i=1,\dots ,p,j=1,\dots ,m.$$

(30)

Next, compute the singular value decomposition (SVD) of $K^{E,F}$:

$$K^{E,F}=U\Sigma V^{\top }.$$

(31)

The principal angles are obtained from the singular values as

$$cos\left({\mathrm{\Theta }}_k\right)={\sigma }_k,k=1,\dots ,min(p,m),$$

(32)

where ${\sigma }_k$ denotes the k-th singular value of $K^{E,F}$.

The distribution of the angles $\left\{{\mathrm{\Theta }}_k\right\}$ provides a measure of the separation between the two subspaces. If all angles are equal to ${\displaystyle \frac{\pi }{2}}$, the subspaces are perfectly orthogonal. Conversely, if all angles are equal to 0, the two subspaces coincide.

One of the most popular metrics used to estimate the orthogonality between two subspaces based on principal angles is the chordal metric, defined as

$$d_c^2=\sum _{k=1}^{min(p.m)}sin{\left({\mathrm{\Theta }}_k\right)}^2,$$

(33)

It is well established that $d_c^2\le r=min(p,m)$. Therefore, values of $d_c^2$ close to zero indicate weak orthogonality, whereas values of $d_c^2$ close to r indicate strong orthogonality. In this study $d_c^2$ values are normalized by the minimum dimension.

Table 6. Subspaces principal angles (PAs) and chordal distances (CDs).

Name/Classes	PA Average	PA Standard Deviation	CD
Leukemia/2	$0.9431607{\displaystyle \frac{\pi }{2}}$	$0.1098443$	$0.9676548\times min(p,m)$
svmguide1/2	$0.8779066{\displaystyle \frac{\pi }{2}}$	$0.3009425$	$0.8771604\times min(p,m)$
splice/2	$0.7883427{\displaystyle \frac{\pi }{2}}$	$0.1908748$	$0.8458960\times min(p,m)$
austrian/2	$0.6836720{\displaystyle \frac{\pi }{2}}$	$0.2971302$	$0.8458960\times min(p,m)$
madelon/2	$0.9539925{\displaystyle \frac{\pi }{2}}$	$0.05631952$	$0.9878780\times min(p,m)$
DNA/3	$0.8459874{\displaystyle \frac{\pi }{2}}$	$0.1555859$	$0.8984641\times min(p,m)$
Satimage/6	$0.9486068{\displaystyle \frac{\pi }{2}}$	$0.1346845$	$0.9541381\times min(p,m)$
USPS/10	$0.9438224{\displaystyle \frac{\pi }{2}}$	$0.09998060$	$0.9714052\times min(p,m)$
letter/26	$0.9431607{\displaystyle \frac{\pi }{2}}$	$0.1098443$	$0.9676548\times min(p,m)$
shuttle/7	$0.8868192{\displaystyle \frac{\pi }{2}}$	$0.09998060$	$0.9714052\times min(p,m)$
BreastCancer/2	$0.4914156{\displaystyle \frac{\pi }{2}}$	$0.2680962$	$0.4980344\times min(p,m)$
Zernike/10	$0.6886542{\displaystyle \frac{\pi }{2}}$	$0.2393689$	$0.7285724\times min(p,m)$
Diabetes/2	$0.2689528{\displaystyle \frac{\pi }{2}}$	$0.2955982$	$0.2431952\times min(p,m)$
mfeat-morphological/10	$0.9094226{\displaystyle \frac{\pi }{2}}$	$0.2617106$	$0.9099942\times min(p,m)$

shows the average principal angles ${\mathrm{\Theta }}_k$ between subspaces; the corresponding standard deviations, which indicate how far the remaining angles deviate from the average; and the chordal distance metric. For multi-class problems, the average of the normalized chordal distance is reported.

The table shows that overall, there is a clear correlation between subspace orthogonality (separability) and the performance of the method. However, there are two exceptions. In the Madelon dataset, the subspaces exhibit high orthogonality, while the classification performance is poor. Conversely, the Breast Cancer dataset shows low orthogonality but high performance. These cases can be explained by the fact that orthogonality alone is not a decisive criterion for classification performance; the representativeness of the training data with respect to the real-world distribution is crucial.

3.2.4. CPU Time Discussion

The proposed method is implemented on the MATLAB-R2018b platform. Note that LIBSVM is written in C++; therefore, any direct comparison of execution time is not meaningful, as C++ implementations are reported in many studies to be up to 100 times faster than MATLAB (see, for instance, [40]. For fairness, the MATLAB execution time was divided by 50. The KNN and KPCA methods are mainly implemented in Python, and for a fair comparison, their processing time was also divided by 50. All experiments were conducted on an HP Victus 15L Gaming Desktop (TG02-0xxx), Victus, shipped from Riyadh KSA running Ubuntu 24.04.3 LTS, equipped with an AMD Ryzen 7 5700G processor (16 threads) and 32 GB of RAM. Only one CPU core was used. The results reported in

Table 7. KOS-SVM processing time comparison.

Name/Classes	Test Size	SVM Processing Time	KOS Processing Time
Leukemia/2	Train.#38 Test.#34	0m8.646s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{005}\mathbf{s}$
svmguide1/2	Train.#3089 Test.#4000	0m33.327s	$\mathbf{0}\mathbf{m}\mathbf{1}.\mathbf{680854}\mathbf{s}$
splice/2	Train.#1000 Test.#2175	0m29.190s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{262854}\mathbf{s}$
austrian/2	Train.#349 Test.#341	0m1.721s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{025804}\mathbf{s}$
madelon/2	Train.#2000 Test.#600	16m50.721s	$\mathbf{0}\mathbf{m}\mathbf{1}.\mathbf{571}\mathbf{s}$
DNA/3	Train.#2000 Test.#1186	4m24.300s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{614}\mathbf{s}$
Satimage/6	Train.#4435 Test.#200	3m53.042s	$\mathbf{0}\mathbf{m}\mathbf{1}.\mathbf{468212}\mathbf{s}$
USPS/10	Train.#7291 Test.#2007	81m18.931s	$\mathbf{0}\mathbf{m}\mathbf{6}.\mathbf{347}\mathbf{s}$
letter/26	Train.#15,000 Test.#5000	44m11.996s	$\mathbf{0}\mathbf{m}\mathbf{21}.\mathbf{330}\mathbf{s}$
shuttle/7	Train.#43,500 Test.#14,500	62m28.783s	$\mathbf{0}\mathbf{m}\mathbf{55}.\mathbf{077}\mathbf{s}$
BreastCancer/2	Train.#398 Test.#171	0m1.696s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{037422}\mathbf{s}$
Zernike/10	Train.#1400 Test.#600	0m48.875s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{126698}\mathbf{s}$
Diabetes/2	Train.#537 Test.#231	0m6.959s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{062452}\mathbf{s}$
mfeat-morphological/10	Train.#1400 Test.#600	0m23.421s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{1069}\mathbf{s}$

and

Table 8. KOS-KNN-KPCA processing time comparison.

Name/Classes	KNN Processing Time	KPCA Processing Time	KOS Processing Time
Leukemia/2	0m0.0630s	0m0.28780s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{005}\mathbf{s}$
svmguide1/2	0m1.2287s	0m0.8382s	$\mathbf{0}\mathbf{m}\mathbf{1}.\mathbf{680854}\mathbf{s}$
splice/2	0m0.5270s	0m0.66s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{262854}\mathbf{s}$
austrian/2	0m0.2475s	0m0.1202s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{025804}\mathbf{s}$
madelon/2	0m0.8191s	0m0.541s	$\mathbf{0}\mathbf{m}\mathbf{1}.\mathbf{571}\mathbf{s}$
DNA/3	0m1.0193s	0m0.6386s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{614}\mathbf{s}$
Satimage/6	1m35.10s	0m0.5332s	$\mathbf{0}\mathbf{m}\mathbf{1}.\mathbf{468212}\mathbf{s}$
USPS/10	0m24.9070s	0m1.3262s	$\mathbf{0}\mathbf{m}\mathbf{6}.\mathbf{347}\mathbf{s}$
letter/26	2m47.025s	0m4.1978s	$\mathbf{0}\mathbf{m}\mathbf{21}.\mathbf{330}\mathbf{s}$
shuttle/7	0m3.2707s	0m56.142s	$\mathbf{0}\mathbf{m}\mathbf{55}.\mathbf{077}\mathbf{s}$
BreastCancer/2	0m0.3280s	0m0.2946s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{037422}\mathbf{s}$
Zernike/10	0m1.0684s	0m0.5318s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{126698}\mathbf{s}$
Diabetes/2	0m0.4419s	0m0.3174s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{062452}\mathbf{s}$
mfeat-morphological/10	0m0.743s	0.3104s	$\mathbf{0}\mathbf{m}\mathbf{0}.\mathbf{1069}\mathbf{s}$

show that KOS is significantly faster than SVM, faster than the KNN, and comparable to KPCA. Notably, KOS remains faster than SVM even without re-scaling the CPU time.

3.2.5. Complexity Analysis

Let d denote the attribute space dimension, c the class size (assuming all classes have approximately the same size), and n the number of classes. The SVM method requires solving $n+{\displaystyle \frac{n(n-1)}{2}}=O\left(n^2\right)$ optimization problems in order to determine the separating hyperplanes. If Newton’s method is employed for the optimization process, the computational complexity of each iteration is $O\left(d^3\right)$. This results in an overall complexity of $O\left(k{n}^2{d}^3\right),$ where k denotes the average number of iterations required for each optimization problem.

For KOS, computing the eigenpairs of the kernel (Mercer) matrix for each class has a complexity of $O\left(c^3\right)$. Since this computation is performed independently for each class, the total complexity is $O\left(n{c}^3\right)$.

To achieve a comparable processing time, we require

$$c=d\sqrt[3]{kn}.$$

A typical accuracy of $\epsilon ={10}^{-6}$ is generally required, and since $k\sim$1/$\epsilon$, we obtain $k\sim$${10}^6$. This leads to

$$c=d{10}^3\sqrt[3]{n}.$$

This indicates that for KOS and SVM to have comparable processing times, the class size must be several orders of magnitude larger than the attribute space dimension. Such a situation is highly unlikely in practice, particularly in high-dimensional cases.

Finally, if both the feature space dimension d and the class size c are assumed constant and only the number of classes n increases, the computational complexities of SVM and KOS reduce to

$$O\left(n^2\right)\mathrm{and}O\left(n\right),$$

respectively. This confirms that KOS scales linearly with respect to the number of classes, as stated in the introduction.

3.3. KOS: Main Advantages Compared to SVM

As shown by the tests, KOS performs similarly to SVM, with many significant advantages, which can be listed as:

1.

The KOS algorithm is easier to implement: it only requires the computation of the correlation matrix eigenpairs (Formulas (9) or (14)), followed by a direct application of Formulas ((20), (21) or (26), (27)).

2.

Robustness: The KOS algorithm is robust and faster; it requires no optimization process which can slow down the algorithm in high dimensions and can fail even if the cost function is quadratic because of rounding-off errors, as in the case of SVM.

3.

Complexity: SVM complexity grows quadratically with the number of classes (requiring $n+{\displaystyle \frac{n(n-1)}{2}}$ hyperplanes), while KOS scales linearly, needing only n sets of eigenpairs.

4.

Time efficiency: As a result of the two previous proprieties, KOS is significantly faster than SVM, up to more than 60 times faster in certain tests, as shown in Table 7.

5.

Parallelization: KOS is highly parallelizable with respect to the number of classes since all subspaces are independent.

6.

Dynamic classification: In the case of class creation or cancellation, SVM requires recalculating all feature space hyperplanes. By contrast, KOS only requires computing eigenpairs of the Mercer kernel matrix for the new class, with no update needed for canceled classes.

7.

Imbalanced classes: In KOS, this issue is naturally handled by subdividing large classes into smaller, balanced subclasses; see Section 2.6. This avoids the need for artificial attributes, which may distort the data structure, or the use of balancing weights, which often lack a clear and systematic procedure.

4. Conclusions

In this paper, we proposed an alternative to the SVM method, referred to as KOS (Kernel-based Optimal Subspaces), which belongs to the family of kernel subspace methods. The approach involves constructing optimal subspaces in the feature space and classifying an unseen attribute vector based on its minimum distance to these subspaces. Some theoretical foundations are presented, and proofs are provided. The POD subspaces are shown to be optimal in the sense that they are the smallest subspaces containing the feature classes, with basis systems that serve as the best representatives of the data. All necessary formulations for practical implementation are derived.

The issue of imbalanced classes is addressed using a simple and intuitive mechanism, eliminating the need for additional preprocessing or resampling techniques. Experimental results on both 2D and high-dimensional multi-class datasets from real-world scenarios demonstrate that KOS achieves performance comparable to optimized SVM. The method is also compared with KNNs and KPCA. Beyond accuracy, KOS offers several practical advantages over SVM: it is easier to implement, significantly faster, and more robust; systematically handles imbalanced classes; is suitable for dynamic classification tasks; and is highly parallelizable with respect to the number of classes. These strengths make KOS a compelling alternative to SVM across a wide range of applications.