A Sparse ${\mathcal{L}}_{\infty }$-Norm Regularized Least Squares Support Vector Regression

Abstract

Although Least Squares Support Vector Regression (LSSVR) reduces the hyperparameter space to two, it sacrifices sparsity, causing all training samples to become support vectors and increasing storage costs. In contrast, standard Support Vector Regression (SVR) preserves sparsity but requires tuning three highly coupled hyperparameters, leading to higher computational burden. To address these limitations, this paper proposes a sparse ${\mathcal{L}}_{\infty }$-norm regularized least squares SVR framework that incorporates the infinity norm of approximation errors into both the objective function and inequality constraints. The resulting optimization problem minimizes model complexity while controlling the maximum prediction deviation through a single slack variable, thereby transforming the conventional three-hyperparameter SVR tuning task into a two-parameter problem involving only the regularization coefficient and kernel width. This formulation restores sparsity by enabling a compact support vector set, while preserving the stability and convexity advantages of LSSVR. Experiments on both static and dynamic datasets demonstrate that the proposed method consistently achieves higher predictive accuracy and improved robustness compared with standard SVR and LSSVR. These results indicate that the proposed ${\mathcal{L}}_{\infty }$-norm regularized framework offers a mathematically principled and computationally efficient alternative for sparse, robust, and scalable regression modeling.

1. Introduction

Guaranteeing the sparsity, modeling accuracy, and efficient optimization of a reduced number of hyperparameters in constructing robust regression models is one of the critical unresolved issues in the realm of machine learning. Practitioners in fields ranging from theoretical research to engineering applications are hesitant to implement machine learning solutions, as the process of constructing predictive regression models with a better generalization performance is not transparent. Support Vector Regression (SVR) [1,2] is a well-known type of non-parametric supervised learning model that integrates the control of model structural risk into machine learning. The risk minimization theory allows for a balance between empirical risk and structural risk, and therefore, SVR surmounts a significant shortcoming of other, nonlinear modeling methods, namely over-fitting [3]. Shortly after its introduction, SVR has been extended in numerous applications, ranging from fault detection [4,5,6], pattern recognition [7,8] and predictive control [9] to brain–computer interfaces [10].

SVR is viewed as one of the most effective and commonly applied techniques in predictive modeling, with improvements in its predictive performance and generalization capacity significantly relying on the adjustment of its hyperparameters. Although it reveals the generalization capability in regression by introducing the $ϵ$-insensitive margin, providing new insights into controlling the structural complexity of models, the optimization process implementing the selection of three hyperparameters, which includes regularization factor $\gamma$, kernel parameter $\sigma$, and $ϵ$, makes the regression modeling method more complex [11,12,13,14,15,16,17]. As mentioned in literature [18], the training of SVR is time-consuming on large-scale datasets, especially when it comes to hyperparameter optimization. It is currently reported that finding the optimal hyperparameters in machine learning models efficiently without sacrificing prediction accuracy poses major hurdles [19]. Prevailing techniques like Grid-Search and Random-Search with cross-validation for optimizing the hyperparameters require considerable time in large-scale search spaces, and they do not provide a guarantee of excellent predictive results. Parameters can be categorized into two groups: learnable parameters and hyperparameters. Learnable parameters are obtained from training data using optimization techniques such as quadratic programming (QP), linear programming (LP), or gradient descent. Hyperparameters are not obtained from the data; instead, they must be chosen before starting the learning process, the optimization of which involves identifying the best combination of hyperparameters that either reduces prediction error or enhances prediction accuracy. As a result, hyperparameter optimization plays a vital role in developing powerful and efficient machine learning models.

Various methods exist for hyperparameter optimization in machine learning models, including grid search, probabilistic model-based optimization, random search [20,21], gradient descent algorithms, and nature-inspired optimization methods. Grid search methods incorporating performance metric [22,23] or a weighted error function [19] aim to find the optimal combination of hyperparameters through iterative updates. From the standpoint of Bayesian methods, probabilistic model-based optimization [24,25] employs a Bayesian model to assess the performance of various hyperparameter configurations and chooses the most advantageous options for evaluation. Gradient descent-based approaches are a widely used class of conventional optimization techniques for adjusting real-valued hyperparameters by computing their gradients. Nature-inspired optimization is another method that employs evolution-based techniques, among them genetic algorithms [26] or particle swarm optimization [27], to explore the range of potential hyperparameters. Additionally, the literature [28] proposed an effective warm-start technique as a practical tool for parameter selection in linear SVR, and the approach has been available for public use. The aforementioned optimization methods aim to find the optimal selection to guarantee a better generalization ability by exhaustively exploring all possible combinations in the hyperparameter space. For instance, there are three hyperparameters in SVR, each with 100 options within its respective space, resulting in a total of ${10}^6$ combinations. However, if the number of hyperparameters is reduced to two, the total size of possible combinations will decrease from one million to ten thousand. Thus, the hyperparameter optimization process for SVR will be greatly simplified and improved in terms of both time and spatial scale.

Significant efforts have been dedicated to exploring solutions for the redundancy during the hyperparameter optimization of SVR. In this regard, LSSVR, as a variant of SVR and first introduced in [29,30], has been successfully explored and emerged as a prominent technique in machine learning, particularly for classification and regression problems. In the LSSVR framework—rather than addressing a quadratic programming problem with inequality constraints as in the conventional SVR—equality constraints and an $L2$-loss function are employed. This results in an optimization problem whose dual solution is found by solving a system of linear equations. More importantly, compared to SVR, LSSVR requires the optimization of only two hyperparameters, namely the kernel width $\sigma$ and the regularization factor $\gamma$, significantly improving the optimization process. As a result, LSSVR has seen widespread application and research in recent years, with over 15,000 citations. For instance, it has been successfully applied in short-term traffic flow prediction [31], optimal control [32], interval-censored data [33] and model identification [34]. Unfortunately, since the emergence of LSSVR, researchers have found that the method lacks model sparsity, meaning that all or most of the samples involved in model training contribute to the regression model, which can easily lead to poor generalization performance. Addressing model sparsity has become a current research hotspot. An early sparse approximation method [35,36] was proposed by Johan Suykens, which can perform pruning based on the physical interpretation of the magnitude of the contribution of ordered support values. Follow-up works have sought to address these challenges by introducing an adaptive pruning algorithm [37,38], Nyström approximation method along with prototype vectors (PVs) [39], gradient descent technique [40] and non-iterative sparse scheme with global representative point selection [41].

However, these methods remain fundamentally limited in achieving true sparsity. Most approaches, such as pruning or global representative point selection, are inherently a posteriori approximations or filtering processes that artificially induce sparsity by discarding samples from the dense solution, inevitably leading to information loss and performance trade-offs. Moreover, variants employing re-weighted or $ϵ$-insensitive loss functions, while introducing sparsity-inducing mechanisms, still rely on heuristic thresholds or complex regularization without fundamentally altering the underlying structure of the least squares problem, resulting in suboptimal, incomplete, or heavily parameter-dependent sparse solutions. The so-called model sparsity, as described in SVR, refers to the fact that when the value of the Lagrange multiplier is relatively small (e.g., its absolute value is less than ${10}^{-8}$), the corresponding training samples are not support vectors (SVs) and can be ignored. From the perspective of practical applications or current software tools, the core ideas of traditional SVR and LSSVR still dominate the mainstream, including the widely used MATLAB 2025a software. To the best of our knowledge, the following issues remain to be investigated: (1) The optimization of the three hyperparameter combinations in SVR is complex. (2) The introduction of slack variables within SVR results in additional inequality constraints, making the parameter estimation more challenging. (3) The expected sparsity of LSSVR has essentially not been addressed, and despite some proposed improvements, they still come at the cost of model accuracy.

It is worth noting that the use of the ${\ell }_{\infty }$-norm, min–max formulations, and Chebyshev-type approximation has been extensively studied in the optimization and robust regression literature. Therefore, the contribution of this work does not aim to introduce a fundamentally new norm-based regression theory. Instead, this study focuses on embedding an ${\ell }_{\infty }$-norm constraint into the SVR framework, with particular attention to kernel-based modeling, sparsity, and computational efficiency.

Unlike classical Chebyshev regression or generic ${\ell }_{\infty }$-norm minimization approaches, the proposed LI-SVR is derived within the primal–dual structure of SVR and LSSVR. By reformulating the approximation error using a single global variable $\lambda$, the resulting optimization problem remains convex and requires only two hyperparameters, namely the regularization parameter and kernel width. Moreover, the number of inequality constraints is reduced to $2n$, which is half of that required by standard $ϵ$-SVR formulations.

More importantly, the proposed formulation naturally restores model sparsity through the Karush–Kuhn–Tucker (KKT) conditions. Only samples whose prediction errors attain the maximum bound actively constrain the solution and become support vectors, while the remaining samples are strictly inactive. This sparsity mechanism differs fundamentally from post hoc pruning strategies or explicit sparsity-inducing regularizers, and leads to a compact kernel regression model without sacrificing convexity or introducing additional tuning parameters.

From this perspective, the proposed LI-SVR should be viewed as a structurally simplified and sparsity-enhanced kernel regression framework inspired by ${\ell }_{\infty }$-norm optimization, rather than as a novel norm definition or a replacement of existing robust regression theories.

Inspired by the core ideas of classical SVR and LSSVR, the main contributions of this work are as follows.

• By integrating the concept of infinite norm into our research, we investigate the problem of lacking sparsity of LSSVR and optimizing hyperparameters of $ϵ$ in SVR, exploring a more efficient and generalizable regression prediction method that ensures both model sparsity and modeling accuracy;
• We propose a novel minimum infinite norm Support Vector Regression (LI-SVR) method, distinct from classical SVR and LSSVR, to explore the optimal balance between the complex hyperparameter optimization process and sparsity, overcoming the limitations of traditional regression modeling methods;
• We demonstrate the sparsity of the proposed method and explain why it does not require optimization of the epsilon-insensitive parameter of $ϵ$ in SVR. Furthermore, we find the maximum approximation error, which is adaptively computed using the proposed method to reflect model accuracy, thereby avoiding the complicated optimization of $ϵ$.

The rest of this article is structured as follows. Section 2 introduces the theory underlying the algorithms, while the proposed LI-SVR is detailed in Section 3. Section 4 presents the experimental analysis of the results. Finally, the conclusions would be gathered in Section 5.

2. A Concise Overview of Classical SVR and LSSVR

2.1. SVR

As our method in this paper is inspired by SVR and LSSVR, a concise summary of their principles is provided. It can be asserted that SVR developed by Vladimir Vapnik with colleagues [1] offers notable and distinct advantages in small sample learning and is among the most extensively researched models, founded on the statistical learning principles of VC theory. Given a training set with N input–output samples, ${\mathbf{x}}_k\in {\mathbb{R}}^d$ and $y_k\in \mathbb{R}$,

$$\left({\mathbf{x}}_k,y_k\right),k=1,2,\dots ,N$$

(1)

SVR aims to determine a nonlinear regression function,

$$f_{SVR}\left(\mathbf{x}\right)={\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b$$

(2)

where $f_{SVR}\left(\mathbf{x}\right)$ can accurately simulate all training data. Since linear regression falls under the umbrella of nonlinear modeling techniques, the following discussion will shift to nonlinear SVR, where its solution commences with the incorporation of high-dimensional feature mapping $\varphi (·)$ to formulate the subsequent optimization problem [2],

$$\begin{array}{cc}& \min :J(\mathbf{w},b)=\frac{1}{2}{\mathbf{w}}^T\mathbf{w}+\gamma {\sum }_{k=1}^N({\xi }_k+{\xi }_k^{*})\end{array}$$

(3)

$$\begin{array}{cc}& s.t.\left\{\begin{array}{c}{y}_k-{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)-b⩽ϵ+{\xi }_k,\hfill \\ -y_k+{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b⩽ϵ+{\xi }_k^{*},\hfill \\ {\xi }_k,{\xi }_k^{*}\ge 0,k=1,2,\dots ,N,\hfill \end{array}\right.\end{array}$$

(4)

where the constant $\gamma$ regulates the balance between smoothness of $f_{SVR}$ and the extent to which deviations exceeding $ϵ$ are permissible, and the slack variables ${\xi }_k,{\xi }_k^{*}$ are employed as a means to address potentially unattainable constraints within the optimization framework, ensuring feasibility is maintained.

The essence lies in crafting a Lagrange function L, fusing the objective function (3) with its associated constraints (4) through the introduction of a dual variable set. Importantly, this function possesses a saddle point, relative to both primal and dual variables, precisely at the optimal solution, as elaborated below,

$$\begin{array}{cc}\hfill L=& {\displaystyle \frac{1}{2}}{\mathbf{w}}^T\mathbf{w}+\gamma {\sum }_{k=1}^N({\xi }_k+{\xi }_k^{*})-{\sum }_{k=1}^N({\eta }_k{\xi }_k+{\eta }_k^{*}{\xi }_k^{*})\\ & +{\sum }_{k=1}^n{\alpha }_k(y_k-{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)-b-ϵ-{\xi }_k)\\ & +{\sum }_{k=1}^n{\alpha }_k^{*}(-y_k+{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b-ϵ-{\xi }_k^{*})\end{array}$$

(5)

where ${\alpha }_k,{\alpha }_k^{*},{\eta }_k,{\eta }_k^{*}$ are Lagrange multipliers, all of which are greater than or equal to zero. By eliminating the primal variables $w,b,{\xi }_k,{\xi }_k^{*}$ based on the saddle point conditions, we can compute $\mathbf{w}$ as,

$${\displaystyle \frac{\partial L}{\partial \mathbf{w}}}=0\Rightarrow \mathbf{w}=\sum _{k=1}^n({\alpha }_k-{\alpha }_k^{*})\varphi \left({\mathbf{x}}_k\right)$$

(6)

and the corresponding dual optimization problem is obtained,

$$\begin{array}{cc}\hfill \min :& J({\alpha }_k,{\alpha }_k^{*})=-{\sum }_{k=1}^N\left({\alpha }_k-{\alpha }_k^{*}\right)y\left(k\right)+ϵ{\sum }_{k=1}^N\left({\alpha }_k+{\alpha }_k^{*}\right)\\ & +\frac{1}{2}{\sum }_{j=1}^N{\sum }_{k=1}^N\left({\alpha }_j-{\alpha }_j^{*}\right)\left({\alpha }_k-{\alpha }_k^{*}\right){\varphi }^T\left({\mathbf{x}}_j\right)\varphi \left({\mathbf{x}}_k\right))\\ & s.t.{\sum }_{k=1}^N\left({\alpha }_k-{\alpha }_k^{*}\right)=0,0\le {\alpha }_k,{\alpha }_k^{*}\le \gamma \end{array}$$

(7)

Based on the Karush–Kuhn–Tucker (KKT) conditions [2], it can be inferred that ${\alpha }_k{\alpha }_k^{*}=0$. Additionally, it has been proven that the Lagrange multiplier is non-zero only when the condition $f\left(x_k\right)-y_k\ge ϵ$ is satisfied, meaning that the k-th sample lies precisely on or outside the $ϵ$-tube and is referred to as a support vector (SV). To reiterate, the KKT conditions are met solely when the Lagrange multipliers corresponding to all samples that lie within the $ϵ$-tube (i.e., $f\left(x_k\right)-y_k\le ϵ$) are exactly zero, and this is why SVR exhibits sparsity.

2.2. LSSVR

LSSVR is based on a standard SVR but substitutes the $ϵ$-insensitive loss function with a least squares. It transforms the inequality constraints of SVR into an equality one, thereby converting the quadratic programming problem into a linear equation system. Similarly, LSSVR also attempts to find a better regression function $f\left(\mathbf{x}\right)$ to approximate all training data. By introducing a regularization factor $\gamma$ to fine-tune the degree of optimality between model complexity and error $e_k$, it yields the following optimization [29],

$$\begin{array}{c}\hfill \min :J(\mathbf{w},b,\mathbf{e})={\displaystyle \frac{1}{2}}{\mathbf{w}}^T\mathbf{w}+{\displaystyle \frac{\gamma }{2}}\sum _{k=1}^n{e}_k^2\end{array}$$

(8)

$$\begin{array}{c}\hfill s.t{y}_k={\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b+e_k.\end{array}$$

(9)

To solve this optimization, the Lagrangian function is constructed using Lagrange multipliers ${\alpha }_k$,

$$\begin{array}{c}\hfill L(\mathbf{w},b,\mathbf{e},\mathit{\alpha })={\displaystyle \frac{1}{2}}{\mathbf{w}}^T\mathbf{w}+{\displaystyle \frac{\gamma }{2}}\sum _{k=1}^n{e}_k^2\\ \hfill +\sum _{k=1}^n{\alpha }_k(y_k-{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)-b-e_k)\end{array}$$

(10)

Taking partial derivatives of the Lagrangian function with respect to $\mathbf{w},b,\mathbf{e},\mathit{\alpha }$ respectively, and setting them to zero, it gets,

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial L}{\partial \mathbf{w}}}=0\Rightarrow \mathbf{w}=\sum _{k=1}^n{\alpha }_i\varphi \left({\mathbf{x}}_i\right)\end{array}$$

(11)

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial L}{\partial b}}=0\Rightarrow \sum _{k=1}^n{\alpha }_k=0\end{array}$$

(12)

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial L}{\partial e_k}}=0\Rightarrow {\alpha }_k=\gamma e_k\end{array}$$

(13)

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial L}{\partial {\alpha }_k}}=0\Rightarrow y_k={\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b+e_k\end{array}$$

(14)

Substituting the expression for $\mathbf{w}$ into the constraint (9) and introducing a kernel function $K({\mathbf{x}}_i,{\mathbf{x}}_j)=\varphi {\left({\mathbf{x}}_i\right)}^T\varphi \left({\mathbf{x}}_j\right)$, Formula (14) can be rewritten as,

$$\begin{array}{c}\hfill y_k=\sum _{j=1}^n{\alpha }_{j}K({\mathbf{x}}_j,{\mathbf{x}}_k)+b+e_k,k=1,2,\dots ,n\end{array}$$

(15)

Combining Formulas (13)–(15), a system of linear equations represented in vectors is formed,

$$\begin{array}{c}\hfill \left[\begin{array}{cc}0& {\mathbf{1}}_n^T\\ {\mathbf{1}}_n& \mathbf{K}+{\gamma }^{-1}{\mathbf{I}}_n\end{array}\right]\left[\begin{array}{c}b\hfill \\ \mathit{\alpha }\hfill \end{array}\right]=\left[\begin{array}{c}0\hfill \\ \mathbf{y}\hfill \end{array}\right],\end{array}$$

(16)

where, the elements of kernel matrix $\mathbf{K}$ are computed by kernel function $K({\mathbf{x}}_j,{\mathbf{x}}_k),j,k=1,2,\dots ,n$, $\mathbf{y}={(y_1,y_2,\dots ,y_n)}^T,\mathit{\alpha }={({\alpha }_1,{\alpha }_2,\dots ,{\alpha }_n)}^T,{\mathbf{1}}_n={(1,1,\dots ,1)}^T$. After solving Equation (16), the predicted output of $f\left(\mathbf{x}\right)$ for the LSSVR can be obtained,

$$\begin{array}{c}\hfill f\left(\mathbf{x}\right)={\mathbf{w}}^T\varphi \left(\mathbf{x}\right)+b=\sum _{k=1}^n{\alpha }_{k}K(\mathbf{x},{\mathbf{x}}_k)+b.\end{array}$$

(17)

The process of constructing the LSSVR has the following advantages: (1) converting quadratic programming problems into linear equations significantly improves computational efficiency; (2) introducing the regularization parameter $\gamma$ to control model complexity effectively can prevent overfitting and enhance model generalization capability; (3) importantly, the number of selected hyperparameters is only two, while SVR has three with an additional insensitivity margin $ϵ$. Unfortunately, the use of ${\mathcal{L}}_2$-norm of error in LSSVR deprives the model sparsity existing in SVR. The fundamental reason lies in its optimization formulation. The ${\mathcal{L}}_2$-norm loss, coupled with equality constraints, forces all training samples to satisfy the Karush–Kuhn–Tucker conditions strictly, resulting in non-zero Lagrange multipliers for every data point. This mechanism lacks the threshold-induced sparsity inherent in the $ϵ$-insensitive loss of standard SVR, where only samples outside the $ϵ$-tube contribute support vectors, thereby eliminating the model’s capacity for sparse representation.

2.3. Comparison with Related SVR Variants and Robust Regression Methods

To further clarify the position of the proposed LI-SVR, this section provides a conceptual and structural comparison with SVR and LSSVR variants, as shown in Figure 1. From Equations (8) and (9), it can be observed that in the parameter optimization of LSSVR, the training errors of all training samples are considered, which gives rise to a loss of the model’s sparsity. Standard $ϵ$-SVR achieves sparsity by introducing an $ϵ$-insensitive tube, where samples lying inside the tube do not contribute to the regression model. However, the performance and sparsity of $ϵ$-SVR critically depend on the preselection of $ϵ$, which is typically determined heuristically or through an extensive grid search [3]. Adaptive $ϵ$ strategies have been proposed to alleviate this issue, but they often introduce additional algorithmic complexity, data-dependent heuristics, or iterative update rules. In contrast, the proposed LI-SVR eliminates the explicit $ϵ$ parameter and instead introduces a single global envelope variable $\lambda$, which is jointly optimized with the regression function. This formulation avoids manual tuning of $ϵ$ while preserving sparsity through the KKT conditions. ${\ell }_1$-SVR and least absolute deviation regression promote robustness by penalizing absolute errors, while quantile regression focuses on estimating conditional quantiles rather than central tendencies. Although these methods are effective for handling non-Gaussian noise or asymmetric error distributions, their objectives differ fundamentally from the proposed LI-SVR. Specifically, LI-SVR aims to control the maximum approximation error via an ${\ell }_{\infty }$-type envelope, rather than minimizing cumulative absolute deviations or targeting specific quantiles. As a result, LI-SVR is particularly suitable for applications requiring bounded worst-case prediction errors. Classical robust regression methods, such as Huber regression and Chebyshev approximation, have long employed ${\ell }_{\infty }$-norm or hybrid loss functions to enhance robustness against outliers. However, these approaches are typically formulated in linear or non-kernelized settings and do not explicitly exploit the primal–dual structure of support vector machines. Moreover, they do not directly address model sparsity in kernel-based regression. By contrast, the proposed LI-SVR integrates an ${\ell }_{\infty }$-type constraint into the SVR/LSSVR framework, yielding a sparse kernel regression model with a compact set of support vectors.

Motivated by both $ϵ$-SVR and LSSVR, we introduce the infinite norm on approximation errors to address the aforementioned issues. In the proposed LI-SVR, sparsity emerges intrinsically from the optimization structure and KKT conditions, without requiring additional sparsity-inducing regularizers or post hoc pruning. This leads to a simpler and more interpretable sparse regression model.

3. SVR and LSSVR-Inspired Least ${\mathcal{L}}_{\infty }$-Norm Support Vector Regression

Based on the comparative analysis in Section 2, it is evident that SVR achieves sparsity at the cost of increased hyperparameter complexity, whereas LSSVR simplifies optimization but loses sparsity due to its ${\mathcal{L}}_2$-norm error formulation. Section 3 builds directly upon this observation by reformulating the LSSVR objective using an ${\mathcal{L}}_{\infty }$-norm error criterion, with the aim of restoring intrinsic sparsity while preserving convexity and limiting the number of tunable hyperparameters to two.

Actually, model sparsity is guaranteed by whether the absolute values of the Lagrange multipliers solved are sufficiently small. Given that LSSVR utilizes the ${\mathcal{L}}_2$-norm of the error to assess modeling accuracy, it inherently lacks model sparsity. In contrast, SVR obtains sparsity by introducing $ϵ$, but it adds the complexity of optimizing an additional hyperparameter. To deal with these issues, we propose the least infinity (${\mathcal{L}}_{\infty }$)-norm Support Vector Regression (LI-SVR) method, as shown in Figure 2.

3.1. The ${\mathcal{L}}_{\infty }$-Norm on Approximation Error

The ${\mathcal{L}}_{\infty }$-norm (also known as the maximum norm) is defined as the largest element of the absolute value in a vector. In contrast, the ${\mathcal{L}}_2$-norm is a square root of the sum of squares in all elements, which calculates the length or distance of the vector in Euclidean space.

Assuming the obtained input–output data and the regression model to be solved are represented as in (1) and (2), respectively, where the measurement of $y_k$ satisfies the following nonlinear system,

$$y_k=g\left({\mathbf{x}}_k\right),k=1,2,\dots ,n.$$

(18)

According to statistical theory, there exists an arbitrary approximation of the measurement model g by the nonlinear regression model f described in Equation (2). As the approximation precision decreases, fewer SVs are needed in SVR; conversely, more SVs are required. Hence, for any given real continuous function g and $\eta >0$, a regression model f can be found,

$$\underset{{\mathbf{x}}_k\in \mathbf{S}}{\sup }|f\left({\mathbf{x}}_k\right)-g\left({\mathbf{x}}_k\right)|<\eta ,\forall k.$$

(19)

In traditional LSSVR, the following deviation $e_k,k=1,2,\dots ,n$ is defined,

$$\begin{array}{c}{e}_1=f\left({\mathbf{x}}_1\right)-y_1={\mathbf{w}}^T\varphi \left({\mathbf{x}}_1\right)+b-y_1\\ e_2=f\left({\mathbf{x}}_2\right)-y_2={\mathbf{w}}^T\varphi \left({\mathbf{x}}_2\right)+b-y_2\\ ⋮\\ e_n=f\left({\mathbf{x}}_n\right)-y_n={\mathbf{w}}^T\varphi \left({\mathbf{x}}_n\right)+b-y_n.\end{array}$$

(20)

and its optimization problem employs the ${\mathcal{L}}_2$-norm of deviations, implying that all deviations in Formula (20) contribute to the solution of Lagrange multipliers, which is the primary reason for the loss of sparsity.

Now, we introduce ${\mathcal{L}}_{\infty }$-norm on the approximation error to replace the ${\mathcal{L}}_2$-norm and solve for the regression model f satisfying condition (19). Initially, we search for the maximum absolute deviation $\lambda$ in Formula (20), defined as the ${\mathcal{L}}_{\infty }$-norm of the approximation error. Next, the minimization of $\lambda$ can be transformed into a $\min -\max$ optimization problem to get intrinsic sparsity,

$$\begin{array}{cc}\hfill \mathit{\alpha }& =\min \underset{{\mathbf{x}}_k\in \mathbf{Z}}{\max }|y_k-f\left({\mathbf{x}}_k\right)|,\forall k,\hfill \\ \hfill & =\arg \underset{\mathbf{w},b}{\min }\underset{{\mathbf{x}}_k\in \mathbf{Z}}{\max }|y_k-{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)-b|.\hfill \end{array}$$

(21)

where $\mathbf{Z}$ denotes the finite training set. For SVR, the interpretation of sparsity implies that contributions of input samples within the pipeline are disregarded by the regression model. Actually, the essentially sparse nature proposed in this paper borrows from the idea of SVR. That is to say, when the deviation $e_k$ satisfies the condition of $-\lambda <e_k<\lambda$, the contribution of the corresponding sample ${\mathbf{x}}_k$ will be negligible in (21).

Lemma 1. 

The $\min -\max$ optimization problem (21) can be addressed through linear programming,

$$\begin{array}{c}\min :\lambda ,\\ s.t.\left\{\begin{array}{c}{y}_k-{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)-b⩽\lambda ,\hfill \\ -y_k+{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b⩽\lambda \hfill \\ \lambda ⩾0,k=1,2,\dots ,n,\hfill \end{array}\right.\end{array}$$

(22)

where $\mathbf{w}$ and b are the parameters to be determined, and λ represents the maximum approximation error.

Proof. 

Define $\lambda$ as follows,

$$\lambda =\max \left|y_k-{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)-b\right|$$

(23)

and the following inequality can be directly derived,

$$\left|y_k-{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)-b\right|⩽\lambda$$

(24)

By applying the absolute value operation, the system of inequalities (24) can be rewritten in the form of (22). This completes the proof of Lemma 1. □

The sparsity-promoting property of the ${\ell }_{\infty }$-norm formulation can be intuitively understood from its min–max structure. As shown in (22), the optimization seeks to minimize a single scalar variable $\lambda$, which represents the maximum approximation error shared by all training samples. Unlike ${\ell }_2$-based formulations that accumulate errors over all samples, the objective here is governed solely by the worst-case deviation. As a result, only those samples whose residuals attain the maximum bound $\lambda$ actively constrain the solution, while samples with smaller errors remain strictly inactive. Consequently, the regression model is effectively determined by a limited subset of critical samples, which naturally leads to a sparse representation. Next, this lemma will be used to establish a sparse regression model in our approach.

3.2. Establishment of Sparse Kernel Regression Model

In fact, early studies [42,43] on the classification of machine learning methods already discussed how the sparsity of models could be enhanced by leveraging the concept of the ${\mathcal{L}}_{\infty }$-norm. A good regression model f should be a comprehensive embodiment of both model sparsity and modeling accuracy, as exemplified by SVR and LSSVR. However, SVR’s optimization problem has the following two drawbacks compared to LSSVR: (1) with N training samples, the introduction of slack variables of ${\xi }_k^{+},{\xi }_k^{-}$ increases the number of inequality constraints by $2N$; (2) it requires the selection of three hyperparameters, including the kernel parameter $\sigma$, regularization factor $\gamma$, and insensitivity margin $ϵ$, while LSSVR only has two of $\sigma ,\gamma$. In light of this [42,43], while maintaining the same number of hyperparameters as LSSVR, we propose the least ${\mathcal{L}}_{\infty }$-norm SVR(LI-SVR). Hence, drawing inspiration from LSSVR, we substitute the ${\mathcal{L}}_2$-norm optimization with the $\min -\max$ problem (22) and yield a new optimization,

$$\begin{array}{cc}& \min :J(\mathbf{w},b,\mathbf{e})={\displaystyle \frac{1}{2}}{\mathbf{w}}^T\mathbf{w}+{\displaystyle \frac{\gamma }{2}}{\lambda }^2\end{array}$$

(25)

$$\begin{array}{cc}& s.t.\left\{\begin{array}{c}{y}_k-{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)-b⩽\lambda ,\hfill \\ -y_k+{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b⩽\lambda \hfill \\ \lambda >0,k=1,2,\dots ,n,\hfill \end{array}\right.\end{array}$$

(26)

where the first term ${\displaystyle \frac{1}{2}}{\mathbf{w}}^T\mathbf{w}$ in (25) controls the model structure, which is minimized to favor smoother model characteristics; the second term ${\lambda }^2$ primarily focuses on improving model accuracy; $\gamma >0$ is a regularization factor, used to balance the weights between model structure and modeling accuracy. What should be emphasized is that in inequality constraint (26), we only consider $\lambda >0$. This is because $\lambda$ represents the maximum one among all absolute deviations. If $\lambda =0$, then $\left|{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b-y_k\right|=0$, indicating that regression model $f\left(\mathbf{x}\right)$ can completely fit the actual outputs y, which has a high potential to trigger severe overfitting and result in extreme degradation of the model’s generalization performance. It should be emphasized that $\lambda$ is an optimization variable rather than a hyperparameter. Unlike the $ϵ$ parameter in SVR, which must be manually specified prior to training, $\lambda$ is automatically determined during optimization as the maximum approximation error, as shown in Equation (29). Therefore, the proposed method involves only two hyperparameters: the regularization coefficient $\gamma$ and the kernel width $\sigma$.

To solve for the weight vector $\mathbf{w}$ and the bias term b, we leverage the Lagrange function with Lagrange multipliers ${\alpha }_k$ and ${\beta }_k$ to convert the constraints into an unconstrained optimization problem,

$$\begin{array}{cc}\hfill \mathcal{L}(\mathbf{w},b,\lambda ,{\alpha }_k,{\beta }_k)=& \frac{1}{2}{\mathbf{w}}^T\mathbf{w}+\frac{\gamma }{2}{\lambda }^2+\sum _{k=1}^n{\alpha }_k(y_k-{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)-b-\lambda )\hfill \\ \hfill & +\sum _{k=1}^n{\beta }_k({\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b-y_k-\lambda ).\hfill \end{array}$$

(27)

To achieve the optimal solution, partial derivatives of the Lagrange function with regard to $\mathbf{w}, b, \lambda , {\alpha }_k, {\beta }_k$ are computed using the Karush–Kuhn–Tucker condition and equated to zero, respectively,

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial \mathcal{L}}{\partial \mathbf{w}}}& =\mathbf{w}-\sum _{k=1}^n{\alpha }_k\varphi \left({\mathbf{x}}_k\right)+\sum _{k=1}^n{\beta }_k\varphi \left({\mathbf{x}}_k\right)=0\hfill \\ \hfill & \Rightarrow \mathbf{w}=\sum _{k=1}^n({\alpha }_k-{\beta }_k)\varphi \left({\mathbf{x}}_k\right),\hfill \end{array}$$

here, the weight vector $\mathbf{w}$ can be viewed as a linear expansion of coefficients ${\alpha }_k-{\beta }_k$ in relation to the nonlinear feature mapping $\varphi \left(\mathbf{x}\right)$.

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial \mathcal{L}}{\partial \lambda }}& =\lambda \gamma -\sum _{k=1}^n{\alpha }_k-\sum _{k=1}^n{\beta }_k=0\hfill \\ \hfill & \Rightarrow \lambda ={\displaystyle \frac{1}{\gamma }}\sum _{k=1}^n({\alpha }_k+{\beta }_k),\hfill \end{array}$$

(28)

From Equation (28), it can be observed that $\lambda$ is automatically calculated via ${\alpha }_k$ and ${\beta }_k$, eliminating the need for prior determination.

$${\displaystyle \frac{\partial \mathcal{L}}{\partial b}}=\sum _{k=1}^n(-{\alpha }_k)+\sum _{k=1}^n{\beta }_k=0\Rightarrow \sum _{k=1}^n{\alpha }_k=\sum _{k=1}^n{\beta }_k$$

(29)

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial \mathcal{L}}{\partial {\alpha }_k}}& =y_k-({\mathbf{w}}^T\varphi \left(\mathbf{x}\right)+b)-\lambda =0\hfill \\ \hfill & \Rightarrow y_k={\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b+\lambda \hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial \mathcal{L}}{\partial {\beta }_k}}& ={\mathbf{w}}^T\varphi \left(\mathbf{x}\right)+b-y_k-\lambda =0\hfill \\ \hfill & \Rightarrow y_k={\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b-\lambda \hfill \\ \hfill & {\beta }_k\ge 0,{\alpha }_k\ge 0,\lambda >0,k=1,2,\dots ,n\hfill \end{array}$$

(31)

where Equations (30) and (31) will be employed for analyzing the sparsity properties and determining whether the k-th sample ${\mathbf{x}}_k$ qualifies as an SV.

Now, substituting Formula (28) into (27), we obtain,

$$\begin{array}{cc}\hfill \mathcal{L}({\alpha }_k,{\beta }_k)=& -{\displaystyle \frac{1}{2\gamma }}{\left(\sum _{k=1}^n({\alpha }_k+{\beta }_k)\right)}^2+\sum _{k=1}^n\left({\alpha }_k-{\beta }_k\right)y_k\hfill \\ \hfill & -{\displaystyle \frac{1}{2}}\sum _{j=1}^n\sum _{k=1}^n({\alpha }_j-{\beta }_j)\left({\alpha }_k-{\beta }_k\right){\varphi }^T\left({\mathbf{x}}_j\right)\varphi \left({\mathbf{x}}_k\right)\hfill \end{array}$$

(32)

where the constraints of ${\beta }_k\ge 0$ and ${\alpha }_k\ge 0$ in (32) must be met. Maximizing Formula (27) is equivalent to the maximization of (32). However, the maximization of (32) can be reformulated as its dual problem,

$$\begin{array}{cc}\hfill \min :& J(\mathit{\alpha },\mathit{\beta })={\displaystyle \frac{1}{2\gamma }}{\left(\sum _{k=1}^n({\alpha }_k+{\beta }_k)\right)}^2+\sum _{k=1}^n\left(-{\alpha }_k+{\beta }_k\right)y_k\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}\sum _{j=1}^n\sum _{k=1}^n({\alpha }_j-{\beta }_j)\left({\alpha }_k-{\beta }_k\right){\varphi }^T\left({\mathbf{x}}_j\right)\varphi \left({\mathbf{x}}_k\right)\hfill \\ \hfill s.t& \sum _{k=1}^N({\alpha }_k-{\beta }_k)=0,\hfill \\ \hfill & {\alpha }_k\ge 0,{\beta }_k\ge 0,k=1,2,\dots ,n.\hfill \end{array}$$

(33)

Theorem 1. 

Optimization problem (33) can be solved through quadratic programming, and the solution of ${\alpha }_k$ and ${\beta }_k$ is globally optimal.

Proof. 

We only need to transform optimization (33) into vector form, as seen in Equation (34), and the theorem is proven. □

$$\begin{array}{c}\min :J(\mathit{\alpha },\mathit{\beta })=\frac{1}{2}{\left(\begin{array}{c}\mathit{\alpha }\hfill \\ \mathit{\beta }\hfill \end{array}\right)}^T\left(\begin{array}{cc}\mathbf{K}+\frac{1}{\gamma }\mathbf{I}& -\mathbf{K}+\frac{1}{\gamma }\mathbf{I}\\ -\mathbf{K}+\frac{1}{\gamma }\mathbf{I}& \mathbf{K}+\frac{1}{\gamma }\mathbf{I}\end{array}\right)\left(\begin{array}{c}\mathit{\alpha }\hfill \\ \mathit{\beta }\hfill \end{array}\right)\\ +{\left(\begin{array}{c}-\mathit{y}\\ \mathit{y}\end{array}\right)}^T\left(\begin{array}{c}\mathit{\alpha }\hfill \\ \mathit{\beta }\hfill \end{array}\right)\\ s.t.\left({\mathbf{\Theta }}^T-{\mathbf{\Theta }}^T\right)\left(\begin{array}{c}\mathit{\alpha }\hfill \\ \mathit{\beta }\hfill \end{array}\right)=0,\\ \mathit{\beta }\ge 0,\mathit{\alpha }\ge 0.\end{array}$$

(34)

Here,

$$\begin{array}{cc}\hfill & \mathit{\alpha }={\left({\alpha }_1,{\alpha }_2,\dots ,{\alpha }_n\right)}^T,\mathit{\beta }={\left({\beta }_1,{\beta }_2,\dots ,{\beta }_n\right)}^T,\hfill \\ \hfill & \mathbf{y}={(y_1,y_2,\dots ,y_n)}^T,\mathbf{Z}={(0,0,\dots ,0)}^T\hfill \\ \hfill & K(j,k)=K({\mathbf{x}}_j,{\mathbf{x}}_k)={\varphi }^T\left({\mathbf{x}}_j\right)\varphi \left({\mathbf{x}}_k\right),\hfill \\ \hfill & \mathbf{K}=\left(\begin{array}{cccc}K(1,1)& K(1,2)& \dots & K(1,n)\\ ⋮& ⋮& & ⋮\\ K(n,1)& K(n,2)& \dots & K(n,n)\end{array}\right),\hfill \end{array}$$

$\mathbf{I}$ is an n-dimensional matrix filled with elements that are all equal to 1, $\mathbf{\Theta }$ represents a column vector with all elements equal to 1, the entries of kernel matrix $\mathbf{K}$ are calculated using kernel function $K({\mathbf{x}}_i,{\mathbf{x}}_j)={\varphi }^T\left({\mathbf{x}}_j\right)\varphi \left({\mathbf{x}}_k\right)$, which includes commonly used ones such as polynomial, radial basis, and sigmoid kernels, etc. It has been demonstrated that the radial basis kernel, also known as the Gaussian kernel, exhibits superior performance over other kernel functions in handling nonlinear regression problems. Therefore, the Gaussian kernel function with kernel width $\sigma$ is adopted in this study,

$$\begin{array}{c}\hfill K({\mathbf{x}}_i,{\mathbf{x}}_j)=\exp ({\displaystyle \frac{-\parallel {\mathbf{x}}_i-{\mathbf{x}}_j{\parallel }^2}{2{\sigma }^2}}).\end{array}$$

(35)

As the optimization (32) is a typical quadratic programming problem, the solution obtained is globally optimal, thus avoiding local optima. After solving the optimization problem of (32), we found that the magnitudes of most Lagrange multipliers of ${\alpha }_k$ and ${\beta }_k$ are sufficiently small, even negligible. The maximum approximation error $\lambda$ can be computed based on (28). Subsequently, we derive a bias term b from Equation (30),

$$\begin{array}{cc}\hfill b& =y_k-{\sum }_{j=1}^n({\alpha }_j-{\beta }_j)K({\mathbf{x}}_k,{\mathbf{x}}_j)-\lambda \\ & ={\displaystyle \frac{\left(y_k-{\sum }_{j=1,{\mathbf{x}}_j\in SV}^{n_{sv1}}{\alpha }_{j}K({\mathbf{x}}_k,{\mathbf{x}}_j)-\lambda \right)}{n_{sv1}}}\end{array}$$

(36)

or (31),

$$\begin{array}{cc}\hfill b& =y_k-{\sum }_{j=1}^n({\alpha }_j-{\beta }_j)K({\mathbf{x}}_k,{\mathbf{x}}_j)+\lambda \\ & ={\displaystyle \frac{\left(y_k-{\sum }_{j=1,{\mathbf{x}}_j\in SV}^{n_{sv2}}-{\beta }_{j}K({\mathbf{x}}_k,{\mathbf{x}}_j)+\lambda \right)}{n_{sv2}}}\end{array}$$

(37)

Here, $n_{sv1}$ and $n_{sv2}$ represent the number of significant differences from zero for components ${\alpha }_k$ and ${\beta }_k$ in $\mathit{\alpha }$ and $\mathit{\beta }$, respectively. Assuming that the total number of SVs (support vectors) is $n_{sv}$, and these SVs are represented as $S{V}_j,j=1,2,\dots ,n_{sv}$, we can derive an essentially sparse regression model, namely LI-SVR, comparable to SVR and LSSVR,

$$\begin{array}{c}\hfill f\left(\mathbf{x}\right)=\sum _{j=1}^{n_{sv}}({\alpha }_j-{\beta }_j)K(\mathbf{x},{SV}_j)+b.\end{array}$$

(38)

It is also revealed in Section 3.3 that $n_{sv}$ is substantially smaller than n. In practice, the quadratic programming problem in Theorem 1 is solved using standard interior-point solvers. Numerical stability is ensured by kernel matrix regularization and normalization of input features. All experiments employ identical solver tolerances to ensure fair comparisons.

3.3. Support Vectors Analysis for Sparse LI-SVR

The number of SVs plays a crucial role in determining the model complexity of SVR. Traditionally, LSSVR lacks sparsity, meaning that all samples contribute to the model construction. This not only increases the structural complexity of the model, leading to potential overfitting issues, but also deteriorates its generalization performance. Therefore, the method proposed in this paper can essentially produce a sparse regression model after solving the optimization problem (33). The quality of sparsity depends on the number of SVs. In general, more SVs indicate a poorer sparsity, while fewer SVs imply a better one. Below, we will analyze how the SVs are generated in the proposed sparse LI-SVR.

Firstly, we define a deviation ${\tilde{e}}_k$ between the actual output $y_k$ and predicted values ${\tilde{y}}_k$ as follows,

$$\begin{array}{cc}\hfill {\tilde{e}}_k& =y_k-{\tilde{y}}_k\hfill \\ \hfill & =y_k-\sum _{j=1}^n({\alpha }_j-{\beta }_j)K({\mathbf{x}}_k,{\mathbf{x}}_j)-b.\hfill \end{array}$$

(39)

where $k=1,2,\dots ,n$. Upon completing the solution to the convex quadratic programming problem (33), the maximum approximation error $\lambda$ can be computed from Equation (28). According to conditions (23) and (30), we further obtain,

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\lambda =\max \left\{{\tilde{e}}_k\right\},\hfill \\ k=1,2,\dots ,n,\lambda >0.\hfill \end{array}\right.\end{array}$$

(40)

It is worth noting that the difference between ${\tilde{e}}_k$ and $\lambda$ is that Equation (39) includes both ${\tilde{e}}_k<\lambda$ and ${\tilde{e}}_k=\lambda$, whereas Equation (40) contains only those subject to condition of ${\tilde{e}}_k=\lambda$. Considering that we derive Equation (40) by computing the partial derivative of the Lagrangian function with respect to ${\alpha }_j$, the sparse solution of ${\alpha }_j$ is dominated by Equation (40). Logically, the k-th sample ${\mathbf{x}}_k$ is selected as an SV only when $\lambda ={\tilde{e}}_k$ strictly, and the contribution of the corresponding ${\alpha }_k$ for sparse LI-SVR should not be overlooked. That is to say, only a small fraction of all deviations of (39) satisfies the condition of $\lambda ={\tilde{e}}_k$. Similarly, we can conduct an analysis on the sparsity of the Lagrange multiplier ${\beta }_k$. From, Equations (23) and (30), we have,

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\lambda =\max \{-{\tilde{e}}_k,k=1,2,\dots ,n\}\hfill \\ k=1,2,\dots ,n,\lambda >0.\hfill \end{array}\right.\end{array}$$

(41)

It should be noted that from Equation (30), $\lambda$ denotes the maximum value of all negative deviations at this point. Equation (41) allows us to ascertain an index, k, of all SVs in which its negative deviations $-{\tilde{e}}_k$ equal exactly $\lambda$. It is evident that the numbers $n_{sv1}$ and $n_{sv2}$ of the SVs are markedly smaller than the total sample size n. Finally, the total number of SVs used in constructing the sparse LI-SVR is computed as $n_{sv}$ (namely, $n_{sv1}+n_{sv2})$, where the size of coefficient $|{\alpha }_k-{\beta }_k|$ corresponding to these SVs is not equal to zero. The essence of the obtained sparse LI-SVR can be reflected through Theorem 2.

Theorem 2. 

Among the Lagrange multipliers $\mathit{\alpha }=({\alpha }_1,{\alpha }_2,\dots ,{\alpha }_n)$ and $\mathit{\beta }=({\beta }_1,{\beta }_2,\dots ,{\beta }_n)$, at most, one of $({\alpha }_k,{\beta }_k)$ is non-zero, and one of the following three conditions is always satisfied for $k=1,2,\dots ,n$ and $\lambda >0$:

$$\begin{array}{c}\hfill \left(a\right):{\alpha }_k>0,{\beta }_k=0\\ \left(b\right):{\alpha }_k=0,{\beta }_k>0\\ \left(c\right):{\alpha }_k=0,{\beta }_k=0\end{array}$$

Proof. 

According to the original constraints (26), it is apparent that any one in the Equations (42)–(44) remains valid,

$$\begin{array}{c}\hfill y_k-{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)-b=\lambda ,\end{array}$$

(42)

$$\begin{array}{c}\hfill -\lambda <y_k-{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)-b<\lambda \end{array}$$

(43)

$$\begin{array}{c}\hfill -y_k+{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b=\lambda .\end{array}$$

(44)

From the Karush–Kuhn–Tucker (KKT) conditions, we have,

$$\begin{array}{c}\hfill {\alpha }_k\left(y_k-{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)-b-\lambda \right)=0\end{array}$$

(45)

$$\begin{array}{c}\hfill {\beta }_k\left(-y_k+{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b-\lambda \right)=0.\end{array}$$

(46)

where ${\alpha }_k\ge 0$, ${\beta }_k\ge 0$ and $\lambda >0$.

Equation (45) is evidently true when ${\alpha }_k=0$, whereas Equation (42) must hold when ${\alpha }_k>0$. Substituting it into Equation (46) results in ${\beta }_k·(-2\lambda )=0$, implying ${\beta }_k=0$. Condition $\left(a\right)$ subjected to both ${\alpha }_k>0$ and ${\beta }_k=0$ in Theorem 2 is proven. In addition, Equation (46) also holds true when ${\beta }_k=0$. If ${\beta }_k>0$, then from Equations (44) and (45), it follows that ${\alpha }_k·(-2\lambda )=0$, which leads to ${\alpha }_k=0$. This further confirms the condition $\left(b\right)$ of Theorem 2. Similarly, when Equation (43) holds, both ${\alpha }_k=0$ and ${\beta }_k=0$ must be met from Equations (45) and (46). We derive the third condition $\left(c\right)$ of Theorem 2 again.

However, it is impossible for both ${\alpha }_k>0$ and ${\beta }_k>0$ to occur simultaneously. If ${\alpha }_k>0$ and ${\beta }_k>0$ are both met, then we have,

$$\begin{array}{c}\hfill \lambda =y_k-{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)-b\end{array}$$

(47)

$$\begin{array}{c}\hfill \lambda =-y_k+{\mathbf{w}}^T\varphi \left({\mathbf{x}}_k\right)+b,\end{array}$$

(48)

and it is inferred that $\lambda =0$ is contradictory to the known fact that $\lambda >0$. This concludes the proof of Theorem 2. □

Although $\lambda$ is influenced by the largest residual, the regularization term jointly constrains the model complexity and prevents overreaction to individual samples. This trade-off limits the sensitivity of $\lambda$ to moderate outliers in practice, as confirmed by the experimental results.

In conclusion, we can clearly see that Theorem 2 further elaborates on how to obtain sparse solutions using the proposed method. Additionally, we also observe that the number of samples satisfying condition $\left(c\right)$ in Theorem 2 is much higher than the one corresponding to the other two conditions $\left(a\right)$ and $\left(b\right)$. Therefore, the proposed sparse LI-SVR method has a fundamentally evolutionary development over the traditional LSSVR presented by Johan Suykens and Joos Vandewalle in enhancing model sparsity and generalization ability.

Further, the proposed LI-SVR formulation explicitly minimizes a regularized upper bound of the prediction error, which inherently limits the influence of noise on the regression function. As further demonstrated in the experimental section, the model exhibits stable performance across different noise realizations and noise distributions, indicating robustness against stochastic disturbances.

3.4. Relationship Between the Global Variable $\lambda$ and the $ϵ$-Insensitive Loss

In classical $ϵ$-SVR, sparsity is induced by the $ϵ$-insensitive loss, where prediction errors within a predefined tolerance band do not contribute to the objective function. In contrast, the proposed LI-SVR replaces the fixed $ϵ$-tube with a single global envelope variable $\lambda$, which is jointly optimized with the regression function. This subsection provides a formal discussion on the relationship between $\lambda$ and $ϵ$.

For a fixed regression function $f\left(\mathbf{x}\right)={\mathbf{w}}^T\varphi \left(\mathbf{x}\right)+b$, the $ϵ$-SVR constraint enforces

$$|y_k-f\left({\mathbf{x}}_k\right)|\le ϵ+{\xi }_k,$$

(49)

where $ϵ$ is user-specified and ${\xi }_k$ are slack variables. In LI-SVR, the corresponding constraint can be written as

$$|y_k-f\left({\mathbf{x}}_k\right)|\le \lambda ,$$

(50)

where $\lambda$ is an optimization variable shared by all samples.

It can be observed that, for any feasible solution of $ϵ$-SVR, choosing $\lambda ={\max }_k|y_k-f\left({\mathbf{x}}_k\right)|$ yields a feasible solution for LI-SVR. Conversely, for a given LI-SVR solution $(f,\lambda )$, setting $ϵ=\lambda$ leads to an $ϵ$-SVR feasible set that contains the LI-SVR solution as a special case. Therefore, LI-SVR can be interpreted as optimizing an upper bound of the $ϵ$-insensitive loss, where the tolerance parameter is no longer fixed a priori but is determined adaptively from the data.

More formally, the LI-SVR objective can be viewed as minimizing a regularized upper bound of the maximum absolute deviation, while $ϵ$-SVR minimizes the sum of deviations exceeding a fixed threshold. When the optimal solution of $ϵ$-SVR satisfies that all active errors lie exactly on the boundary of the $ϵ$-tube, the two formulations coincide with $\lambda =ϵ$. In general cases, LI-SVR yields a conservative envelope that guarantees bounded worst-case errors, whereas $ϵ$-SVR allows a trade-off between sparsity and tolerance through a predefined margin.

This analysis clarifies that $\lambda$ should not be regarded as a direct replacement of $ϵ$, but rather as an adaptive envelope parameter that implicitly controls the insensitivity region through optimization.

In addition, we emphasize that interpreting $\lambda$ as an “adaptive $ϵ$” is not intended to imply a strict theoretical equivalence. Instead, this interpretation arises naturally from the optimality conditions of the proposed formulation.

From the Karush–Kuhn–Tucker (KKT) conditions of the LI-SVR optimization problem, it follows that only samples satisfying,

$$|y_k-f\left({\mathbf{x}}_k\right)|=\lambda$$

(51)

are associated with nonzero dual variables and thus act as support vectors. All remaining samples strictly satisfy

$$|y_k-f\left({\mathbf{x}}_k\right)|<\lambda$$

(52)

and are inactive at the optimum. This property closely parallels the role of the $ϵ$-insensitive region in classical $ϵ$-SVR, where samples inside the $ϵ$-tube do not contribute to the solution.

However, a key distinction is that $\lambda$ is not predefined but is jointly optimized together with the regression function. As a result, the effective insensitivity region is determined adaptively by the data distribution and regularization trade-off, rather than by user specification. From an optimization viewpoint, $\lambda$ represents the smallest envelope that simultaneously satisfies all constraints while minimizing a regularized worst-case error.

4. Experimental Analysis

In comparison with LSSVR and SVR methods, the regression prediction using our approach has the following three significant characteristics: (1) for LSSVR, the proposed method not only achieves inherent sparsity but also outperforms SVR; (2) while SVR requires the optimization of three hyperparameters, the proposed method only optimizes two hyperparameters; (3) by introducing the infinite norm of error, the proposed method adaptively computes the maximum error, which is somewhat similar to the insensitive zone in SVR. Based on this, the following three metrics are introduced to demonstrate the effectiveness of the proposed method: the percentage of support vectors ($SVs\%$), root mean square error ($RMSE$). The first metric is used to evaluate the complexity of the model structure, while the latter one is used to assess the accuracy of the model. Clearly, these metrics are conflicting, and the evaluation of regression prediction should strike a balance between them. $SVs\%$, $RMSE$ and $MAE$ (maximum absolute error) are defined as, respectively,

$$\begin{array}{c}\hfill SVs\%={\displaystyle \frac{N_{SV}}{N}}\times 100\%\end{array}$$

(53)

$$\begin{array}{c}\hfill RMSE=\sqrt{{\displaystyle \frac{1}{N}}\sum _{j=1}^N{\left(y_k-\widehat{y_k}\right)}^2},\end{array}$$

(54)

here, $N_{SV}$ denotes the total number of SVs, which is calculated by applying the conditions of $\mid {\alpha }_k-{\beta }_k\mid >1E-8$, N represents the total number of training samples, $y_k$ and $\widehat{y_k}$ are the actual and predicted output, respectively. MAE is reported to reflect the worst-case performance, which is particularly relevant for the proposed ${\mathcal{L}}_{\infty }$-norm formulation. To demonstrate the rationality and superiority of the proposed method, we compare its performance with that of classical SVR and LSSVR on several benchmark datasets, including static nonlinear functions and linear/nonlinear dynamic systems. All experimental simulations were conducted in MATLAB 2022b, with the hyperparameters of SVR and LSSVR optimized using the built-in MATLAB packages and the LS-SVMlab Toolbox, respectively. Hyperparameter selection is not the primary focus of this study. In the experiments, the regularization parameter $\gamma$ and kernel width $\sigma$ are selected using grid search with cross-validation to ensure fair and stable performance across different datasets.

4.1. Computational Complexity and Scalability

The proposed LI-SVR is formulated as a convex quadratic programming problem with $2n$ inequality constraints, which is structurally simpler than the standard $ϵ$-SVR formulation involving $4n$ constraints. In terms of computational complexity, standard SVR requires solving a quadratic programming (QP) problem with $2N$ inequality constraints, resulting in a worst-case training complexity between $\mathcal{O}\left(N^2\right)$ and $\mathcal{O}\left(N^3\right)$. LSSVR reduces this burden by solving a linear system of size N, leading to a complexity of $\mathcal{O}\left(N^3\right)$. The proposed LI-SVR also involves a convex QP problem, but without slack variables and with only $2N$ non-negativity constraints. In practice, the effective number of active constraints is significantly smaller due to sparsity, yielding training times comparable to or lower than standard SVR. In addition, LI-SVR requires only two hyperparameters, namely the kernel width and the regularization parameter, whereas classical $ϵ$-SVR introduces an additional $ϵ$-insensitive parameter. This reduction in hyperparameter dimensionality significantly alleviates the computational burden associated with hyperparameter optimization, especially when grid search or cross-validation is employed.

In terms of convergence behavior, LI-SVR can be efficiently solved using standard quadratic programming solvers and exhibits stable convergence due to its convex objective and reduced constraint set. Since the kernel matrix structure is identical to that of LSSVR and SVR, the memory footprint scales quadratically with the number of training samples, which is common to kernel-based methods.

Regarding scalability, LI-SVR inherits the same limitations and advantages as kernel SVR methods. While the current work focuses on moderate-scale datasets, the proposed formulation is compatible with existing large-scale extensions, such as decomposition methods or approximate kernel techniques, which can be incorporated in future work.

4.2. Static Nonlinear Functions

Initially, we evaluate the performance of our method by applying it to a simple regression task of the sinc function [44], which is defined as:

$$y=sinc\left(x\right)={\displaystyle \frac{sinx}{x}}+\nu ,x\in U[-4\pi ,4\pi ]$$

(55)

where U denotes a random variable with a uniform distribution in the interval $[-4\pi ,4\pi ]$, and v is Gaussian noise with a mean of 0 and a variance of ${0.2}^2$. To mitigate potential bias in the regression prediction experiments, the mean value derived from ten independent runs for each dataset is used as the evaluation metric. Then, 500 training samples are generated from Equation (55) to establish the regression prediction model using the LI-SVR method. Likewise, 10 independent test datasets are generated from Equation (55), with each test dataset consisting of 500 samples. The results obtained for a classical SVR, LSSVR and LI-SVR (our approach) are listed in

Table 1. The proposed LI-SVR is compared with the classical SVR and LSSVR in terms of training/testing RMSE and $SVs\%$, where the hyperparameter set $(\sigma ,\gamma )$ in our approach is selected as (0.8, 100).

Model	RMSE (Training)	RMSE (Testing)	$\mathit{S}\mathit{V}\mathit{s}\%$
LSSVR	0.1119	0.1137	100.00%
SVR	0.1127	0.1155	89.00%
LI-SVR	$0.1133$	$0.1143$	$2.60\%$

. For training RMSE, our method performs slightly worse than both SVR and LSSVR, while the testing RMSE is between the two. However, the sparsity of our method significantly outperforms them. The $SVs\%$ in our method is 2.60%, indicating that only 13 samples (as shown in Figure 3) out of 500 training samples were used to construct the regression prediction model, compared to 445 and 500 samples required by SVR and LSSVR, respectively. Figure 4 gives the indices of these SVs obtained by our method, along with the corresponding magnitudes. Clearly, while maintaining modeling accuracy, the proposed LI-SVR exhibits superior sparsity compared to the classical SVR and LSSVR in terms of model sparsity.

4.3. Linear Dynamic System

Consider a linear dynamic system represented by the following transfer function,

$$G\left(s\right)={\displaystyle \frac{C\left(s\right)}{R\left(s\right)}}={\displaystyle \frac{20}{0.05s^2+3s+20}}$$

(56)

with a sampling time $T_s=0.01s$, a total of $N=500$ samples, and input/output delays both set to 2. The input–output pairs of $\left\{\mathbf{X}\right(k),Y(k\left)\right\}$ for constructing LI-SVR are given as follows,

$$\begin{array}{c}\hfill \mathbf{X}\left(k\right)=\left\{y\right(k-1),y(k-2),u(k-1),u(k-2\left)\right\}\\ \hfill Y\left(k\right)=y\left(k\right),k=3,4,\dots ,500\end{array}$$

(57)

here $u\left(k\right)$ is a random signal defined as $u\left(k\right)=2.0\times rand$ in our MATLAB simulation. Figure 5 shows the training output of LI-SVR with hyperparameter set $(\sigma ,\gamma )=(0.8,1000)$, where the blue solid circles represent the SVs. The corresponding magnitude of the LI-SVR model coefficients for these SVs is given in Figure 6, accounting for 10.08% of the total sample size, compared to 66.47% for SVR and 100% for LSSVR. The detailed training/testing RMSEs are presented in

Table 2. The proposed LI-SVR is compared with the classical SVR and LSSVR in terms of training/testing RMSE and $SVs\%$, where the hyperparameter set $(\sigma ,\gamma )$ in our approach is selected as (0.8, 1000).

Model	RMSE (Training)	RMSE (Testing)	$\mathit{S}\mathit{V}\mathit{s}\%$
LSSVR	6.27 × ${10}^{-4}$	8.02 × ${10}^{-4}$	100.00%
SVR	0.0652	0.0833	66.47%
LI-SVR	$0.0055$	$0.0106$	$10.84\%$

. Although LSSVR achieves the best modeling accuracy, its model structure is significantly more complex, reaching a magnitude of $1\times {10}^7$ as indicated by the y-axis in Figure 7. Further, test data were generated based on Equation (57), and LI-SVR achieved favorable predicted outputs, as shown in Figure 8.

4.4. Nonlinear Dynamic System

We consider a nonlinear dynamic system [44],

$$\begin{array}{c}\hfill y(k+1)=0.5y\left(k\right)+0.5{\displaystyle \frac{y^2(k-1)}{1+y^2(k-1)}}-0.5y(k-1)u\left(k\right)+u\left(k\right)\end{array}$$

(58)

with a sampling time $T_s=0.01s$, a total of $N=200$ samples. The input–output pairs of $\left\{\mathbf{X}\right(k),Y(k\left)\right\}$ for constructing LI-SVR are the same as (57). When hyperparameter set $(\sigma ,\gamma )$ are chosen as $(0.2,1000)$, the predicted output and SV are shown in Figure 9, where the indices associated with the SVs, along with the corresponding model coefficients, are illustrated in Figure 10. The predictive outputs derived from the test data, as governed by Equation (58), are presented in Figure 11. In

Table 3. The proposed LI-SVR is compared with the classical SVR and LSSVR in terms of training/testing RMSE and $SVs\%$, where the hyperparameter set $(\sigma ,\gamma )$ in our approach is selected as (0.2, 1000).

Model	RMSE (Training)	RMSE (Testing)	$\mathit{S}\mathit{V}\mathit{s}\%$
LSSVR	3.88 × ${10}^{-4}$	3.56 × ${10}^{-3}$	100.00%
SVR	0.0170	0.0261	23.74%
LI-SVR	$0.0056$	$0.0067$	$7.07\%$

, the modeling precision of the LI-SVR is observed to be intermediate between SVR and LSSVR, yet it demonstrates the most favorable sparsity. In particular, the proposed method constructs a regression model utilizing only 13 SVs out of 200 data points.

5. Conclusions

Although LSSVR greatly improves training efficiency by converting inequality into equality constraints, it also forfeits the crucial model sparsity of SVR. To deal with the fundamental flaw of LSSVR, we propose and thoroughly analyze another least infinity-norm (${\mathcal{L}}_{\infty }$-norm) version for LSSVR and SVR. Essentially, the proposed method exhibits excellent sparsity compared to traditional LSSVR, by adopting the ${\mathcal{L}}_{\infty }$-norm and introducing new constraints to replace the ${\mathcal{L}}_2$-norm and its constraints used in the original LSSVR optimization problem. Without the additional insensitivity margin $ϵ$ and slack variables ${\xi }_k$ seen in SVR, the number of hyperparameters in our approach is the same as LSSVR’s. Furthermore, another strength of using the idea of least ${\mathcal{L}}_{\infty }$-norm is that it not only generates truly sparse properties in the regression model, but also automatically obtains the maximum approximation error $\lambda$ reflecting the model accuracy.

Despite its advantages, the proposed method may face scalability challenges on extremely large datasets due to kernel matrix construction. In addition, its performance depends on kernel selection, and the ${\mathcal{L}}_{\infty }$-norm criterion may be overly conservative in noise-dominated scenarios. Future work will focus on extending the proposed framework using low-rank kernel approximations or online learning strategies to improve scalability in high-dimensional or large-scale settings.