A Globally Optimal Alternative to MLP

Abstract

In deep learning, achieving the global minimum poses a significant challenge, even for relatively simple architectures such as Multi-Layer Perceptrons (MLPs). To address this challenge, we visualized model states at both local and global optima, thereby identifying the factors that impede the transition of models from local to global minima when employing conventional model training methodologies. Based on these insights, we propose the Lagrange Regressor (LReg), a framework that is mathematically equivalent to MLPs. Rather than updates via optimization techniques, LReg employs a Mesh-Refinement–Coarsening (discrete) process to ensure the convergence of the model’s loss function to the global minimum. LReg achieves faster convergence and overcomes the inherent limitations of neural networks in fitting multi-frequency functions. Experiments conducted on large-scale benchmarks including ImageNet-1K (image classification), GLUE (natural language understanding), and WikiText (language modeling) show that LReg consistently enhances the performance of pre-trained models, significantly lowers test loss, and scales effectively to big data scenarios. These results underscore LReg’s potential as a scalable, optimization-free alternative for deep learning in large and complex datasets, aligning closely with the goals of innovative big data analytics.

1. Introduction

Current research suggests that in sufficiently large neural networks, most local minima in associated loss functions have relatively small values. Consequently, the requirement of locating a global minimum becomes less crucial, as it is more significant to identify a point within the parameter space with a low loss, though not necessarily minimal [1,2,3,4]. This perspective has been substantiated through various studies on large language models. These studies demonstrate that independent training of an identical model often results in final cost function values that display minimal variation. This observation suggests that the differences between most local minima, including the global minimum, are potentially negligible. Additionally, scaling laws [5,6] establish a relationship between the number of model parameters and the (minimum) value of cost function.

Nonetheless, these regular patterns have yet to be thoroughly investigated, while they have been predominantly inferred based on experimental results. For example, in the case of scaling laws, the constants in the empirical formula can be approximated solely through experimental data. Furthermore, practitioners face significant challenges in determining whether a model’s cost function has attained the global minimum during training, as there is a lack of research providing methodologies to calculate the lower bound of a given cost function or to visualize a model’s state upon reaching the global minimum.

In this study, we begin by establishing the equivalence between MLPs and Lagrange linear interpolation functions. Subsequently, we explore the performance of a Multi-Layer Perceptron (MLP) in regression tasks, with a visualization of the loss function’s states at both local minima (Figure 1a) and global minima (Figure 1d). Our analysis demonstrates that conventional training methods, which depend on continuous parameter updates, are fundamentally inadequate for guiding a loss function from a local minimum to a global minimum. To address this issue, we propose a Mesh-Refinement–Coarsening (MRC) loop (Figure 1b,c) for updating model parameters, inspired by Finite Element Method (FEM) theory [7]. This algorithm reformulates the MLP as an equivalent Lagrange linear interpolation function, and we refer to the alternative model of MLP as Lagrange Regressor (LReg).

LReg serves as an alternative to MLPs, with the ability to ensure a rapid convergence of a loss function towards its global minimum. It is applicable to a range of recognition domains, including regression, image classification, image super-resolution, text classification, and language modeling. Furthermore, our experimental results on large datasets, such as ImageNet, demonstrate that LReg, when explained as a Parameter-Efficient Fine-Tuning (PEFT) method, promotes further convergence of the validation loss function in large pre-trained models towards global minima and enhances the model performance. In contrast to other PEFT methods, LReg has a versatile structure that is not limited to transformer-based architectures, thereby increasing its applicability. Our contributions are as follows:

• Investigating and visualizing the challenge encountered by the loss function of an MLP in overcoming to escape local minima to attain the global minimum.
• Deriving the explicit scaling law for MLPs through error bounds and conducting experimental validation on regression tasks.
• Overcoming the limitations of neural networks in fitting multi-frequency data, thereby making LReg more versatile for complex applications.
• Applying LReg as a PEFT method for large models (not confined to transformer architectures) to investigate the potential of guiding their loss functions toward global minima, consequently further reducing test loss and improving the performance of pre-trained models.

In Section 2, we describe our method, including the mesh generation and the architecture of the model encoder. In Section 3, we provide experiments to illustrate the scaling law of our model and compare its performance with other regressors and neural networks across diverse datasets and application scopes. In Section 4, we discuss the limitations and future work. In Section 5, we conclude our work, highlighting its theoretical and practical value. Additionally, we provide Appendix A for mathematical proofs and visualizations, Appendix B for further ablation studies, and Appendix C for expanded applications.

2. Methods

2.1. Global Minimum and Linear Interpolation

It is well known in finite element approximation theory that the linear interpolation operator $I_h$ reproduces all polynomials exactly. Consequently, the interpolation error depends solely on the higher-order derivatives of the target function. Let ${\mathcal{T}}_h$ denote a shape-regular triangulation of the domain $\Omega$ with mesh size parameter $h={max}_{K\in {\mathcal{T}}_h}{h}_K$. For any sufficiently smooth function $u\in H^2(\Omega )$, the following error estimates hold (see, e.g., [1,2]):

$$\parallel u-I_h{u\parallel }_{L^2(\Omega )}\le C{h}^2{\left|u\right|}_{H^2(\Omega )},|u-I_h{u|}_{H^1(\Omega )}\le Ch{\left|u\right|}_{H^2(\Omega )},$$

(1)

where C is a constant independent of h.

These inequalities imply that as the mesh is refined ($h\to 0$), the interpolation error necessarily converges to zero. Intuitively, since linear elements already capture the affine component of the function exactly, the remaining curvature term is suppressed quadratically (in $L^2$) or linearly (in $H^1$) with respect to the mesh size.

Therefore, linear interpolation on sufficiently fine triangular meshes is guaranteed to approximate any smooth function arbitrarily well. This property ensures that when doing the function fitting, mesh refinement reduces the loss function and makes it possible to accurately locate the global minimum of the loss function.

2.2. Preliminary: Function Fitting via MLP

Mathematically, the composition of any piecewise linear functions remains piecewise linear. Furthermore, any piecewise linear function $f\left(x\right)$ can be represented as a linear combination of Lagrange basis functions:

$$f\left(x\right)=\sum _i{c}_i{\psi }_i\left(x\right),$$

(2)

where $c_i$ is a coefficient of the basis function ${\psi }_i\left(x\right)$.

It is noteworthy that both linear layers and ReLU activations are piecewise linear functions. Consequently, an MLP composed of these elements inherently constitutes a piecewise linear mapping. For such an MLP, it is always possible to determine a set of coefficients $\{c_1,c_2,\dots \}$ and basis functions $\{{\psi }_1,{\psi }_2,\dots \}$ such that the aforementioned equation can represent the MLP.

The geometry of piecewise linear functions is straightforward: in one-dimensional space, these functions appear as segmented lines, while in two-dimensional space, they form triangulated surfaces. Figure 1a illustrates the performance of an MLP on a function-fitting task. In this example, the MLP fits the function by the dashed segmented line, while the ground truth is represented by a solid curve. The area of the shaded region between them represents the Mean Absolute Error (MAE). Notably, any minor alteration to any node on the segmented line will result in an increase in the MAE, because the MAE is at a local minimum in this configuration.

2.3. Optimization: Discrete vs. Continuous Processes

Upon examining Figure 1 closely, the transitioning from the piecewise linear function depicted in Figure 1a to that in Figure 1d requires two key modifications: (1) relocation of nodes ${\mathit{p}}_1$ and ${\mathit{p}}_2$, and (2) the shift of nodes ${\mathit{p}}_7$ and ${\mathit{p}}_8$ into the interval between ${\mathit{p}}_4$ and ${\mathit{p}}_5$. This is a discrete updating process as it requires reordering $({\mathit{p}}_4,{\mathit{p}}_5,{\mathit{p}}_6,{\mathit{p}}_7,{\mathit{p}}_8,{\mathit{p}}_9)$ to $({\mathit{p}}_4,{\mathit{p}}_7,{\mathit{p}}_8,{\mathit{p}}_5,{\mathit{p}}_6,{\mathit{p}}_9)$, followed by adjusting the positions of ${\mathit{p}}_7$ and ${\mathit{p}}_8$. The former adjustment might be achievable through gradient-based methods, whereas the latter is almost impossible. This difficulty arises while updating parameters continuously with first-order adaptive learning rate methods. With the learning rate inevitably diminishing to very small values, the parameter updates become negligible, reducing the likelihood of the model undergoing the significant changes required for the transition from Figure 1a–d.

2.4. Lagrange Basis Function

Our MRC algorithm is designed for discrete updates of model parameters. However, it is not directly applicable to the training of MLPs. In order to employ it effectively, we first need to reformulate the MLP as a linear combination of basis functions, as shown in Equation (2). The Lagrange basis functions are defined as tent-shaped linear functions that take a value of 1 at their specified nodes and 0 at all other nodes. In Appendix A.2, we derive the formula for the Lagrange basis function in detail. For the one-dimensional case, when fitting a continuous function $F:\mathbb{R}\to \mathbb{R}$ using an MLP, the Lagrange basis function can be expressed as:

$${\psi }_i\left(\mathit{x}\right)=min\left({\displaystyle \frac{\mathrm{ReLU}(\mathit{x}-{\mathit{p}}_{i-1})}{{\mathit{p}}_i-{\mathit{p}}_{i-1}}},{\displaystyle \frac{\mathrm{ReLU}({\mathit{p}}_{i+1}-\mathit{x})}{{\mathit{p}}_{i+1}-{\mathit{p}}_i}}\right).$$

(3)

For example, ${\psi }_1$ and ${\psi }_9$ are shaded blue in Figure 1a for illustrative purposes. In FEM, a mesh refers to a network of interconnected points, called nodes (e.g., ten points $\{{\mathit{p}}_0,{\mathit{p}}_1,\dots \}$ in Figure 1a), which divide a complex geometry into smaller, simpler simplices (e.g., nine intervals $\{({\mathit{p}}_0,{\mathit{p}}_1),({\mathit{p}}_1,{\mathit{p}}_2),\dots \}$ in Figure 1a).

When fitting a continuous function $F:{\mathbb{R}}^d\to \mathbb{R}$ using an MLP, the formulation for the Lagrange basis becomes substantially more intricate. This necessitates an understanding of the terminologies associated with Delaunay Triangulation. In general scenarios, the mathematical expressions of the Lagrange basis are described in Appendix A.2, and the visualizations are provided in Appendix A.5.

2.5. Mesh-Refinement–Coarsening Loop

Inspired by the FEM, we designed the MRC loop to update model parameters. The process begins with identifying underfitting regions, where the fitted value significantly deviates from the target. As shown in Figure 1b, the model exhibits MAE in the elements $({\mathit{p}}_1,{\mathit{p}}_2)$, $({\mathit{p}}_2,{\mathit{p}}_3)$, and $({\mathit{p}}_4,{\mathit{p}}_5)$; next, we perform mesh refinement by inserting fine nodes ${\mathit{p}}_{1.5}$, ${\mathit{p}}_{2.5}$, and ${\mathit{p}}_{4.5}$ at the center of these regions (for the two-dimensional case $F:{\mathbb{R}}^2\to \mathbb{R}$, we provide a visual example of fine node insertion in Appendix A.5) and denote others as coarse nodes. The corresponding coefficients $c_{1.5}$, $c_{2.5}$, and $c_{4.5}$ are computed via interpolation to construct the scaled-up model; subsequently, we further update these new coefficients to continuously reduce MAE using model training or the least squares method (see Appendix A.5 for details); the scaled-up model will have a reduced MAE value but with a bigger model size. For example, there are three more basis functions involved. In order to keep model size unchanged, we perform coarse node removal to scale the model down after maintaining the MAE value as much as possible. In Figure 1c, three nodes are chosen to be removed, and the adjacent simplices are merged to form a scaled Lagrange basis function (see Figure 1c); we then reindex the nodes, denoting them as coarse nodes to start the next iteration of the loop. The following transformations illustrate the MRC loop in Figure 1, detailing the key steps in mesh refinement, coefficient interpolation, mesh coarsening, and reindexing.

[Mesh Refinement] Insert three nodes:

$$({\mathit{p}}_0,{\mathit{p}}_1,{\mathit{p}}_2,{\mathit{p}}_3,{\mathit{p}}_4,{\mathit{p}}_5,{\mathit{p}}_6,{\mathit{p}}_7,{\mathit{p}}_8,{\mathit{p}}_9)\to ({\mathit{p}}_0,{\mathit{p}}_1,{\mathit{p}}_{1.5},{\mathit{p}}_2,{\mathit{p}}_{2.5},{\mathit{p}}_3,{\mathit{p}}_4,{\mathit{p}}_{4.5},{\mathit{p}}_5,{\mathit{p}}_6,{\mathit{p}}_7,{\mathit{p}}_8,{\mathit{p}}_9).$$

[Mesh Refinement] Add three basis functions:

$$({\psi }_0,{\psi }_1,{\psi }_2,{\psi }_3,{\psi }_4,{\psi }_5,{\psi }_6,{\psi }_7,{\psi }_8,{\psi }_9)\to ({\psi }_0,{\psi }_1,{\psi }_{1.5},{\psi }_2,{\psi }_{2.5},{\psi }_3,{\psi }_4,{\psi }_{4.5},{\psi }_5,{\psi }_6,{\psi }_7,{\psi }_8,{\psi }_9).$$

[Interpolation] Compute three coefficients:

$$(c_0,c_1,c_2,c_3,c_4,c_5,c_6,c_7,c_8,c_9)\to (c_0,c_1,c_{1.5},c_2,c_{2.5},c_3,c_4,c_{4.5},c_5,c_6,c_7,c_8,c_9).$$

[Mesh Coarsening] Remove three coarse nodes:

$$({\mathit{p}}_0,{\mathit{p}}_1,{\mathit{p}}_{1.5},{\mathit{p}}_2,{\mathit{p}}_{2.5},{\mathit{p}}_3,{\mathit{p}}_4,{\mathit{p}}_{4.5},{\mathit{p}}_5,{\mathit{p}}_6,{\mathit{p}}_7,{\mathit{p}}_8,{\mathit{p}}_9)\to ({\mathit{p}}_0,{\mathit{p}}_{1.5},{\mathit{p}}_2,{\mathit{p}}_{2.5},{\mathit{p}}_3,{\mathit{p}}_4,{\mathit{p}}_{4.5},{\mathit{p}}_5,{\mathit{p}}_6,{\mathit{p}}_9).$$

[Mesh Coarsening] Merge simplices that are adjacent to the removed nodes:

$$({\mathit{p}}_0,{\mathit{p}}_1),({\mathit{p}}_1,{\mathit{p}}_{1.5})\to ({\mathit{p}}_0,{\mathit{p}}_{1.5})({\mathit{p}}_6,{\mathit{p}}_7),({\mathit{p}}_7,{\mathit{p}}_8),({\mathit{p}}_7,{\mathit{p}}_9)\to ({\mathit{p}}_8,{\mathit{p}}_9).$$

[Mesh Coarsening] Keep the number of basis functions:

$$({\psi }_0,{\psi }_1,{\psi }_{1.5},{\psi }_2,{\psi }_{2.5},{\psi }_3,{\psi }_4,{\psi }_{4.5},{\psi }_5,{\psi }_6,{\psi }_7,{\psi }_8,{\psi }_9)\to ({\psi }_0,{\psi }_{1.5},{\psi }_2,{\psi }_{2.5},{\psi }_3,{\psi }_4,{\psi }_{4.5},{\psi }_5,{\psi }_6,{\psi }_9).$$

[Node Reindexing]:

$$({\mathit{p}}_0,{\mathit{p}}_{1.5},{\mathit{p}}_2,{\mathit{p}}_{2.5},{\mathit{p}}_3,{\mathit{p}}_4,{\mathit{p}}_{4.5},{\mathit{p}}_5,{\mathit{p}}_6,{\mathit{p}}_9)\to ({\mathit{p}}_0,{\mathit{p}}_1,{\mathit{p}}_2,{\mathit{p}}_3,{\mathit{p}}_4,{\mathit{p}}_5,{\mathit{p}}_6,{\mathit{p}}_7,{\mathit{p}}_8,{\mathit{p}}_9).$$

These transformations ensure the MRC loop efficiently adapts the model structure, reducing underfitted regions while maintaining computational efficiency.

Figure 1 shows that as this loop progresses, the underfitting regions gradually shrink, and the MAE consistently decreases. This process significantly deviates from first-order adaptive learning rate methods. Our methodology utilizes discrete operations to update model parameters—check Algorithms 1–3 for details.

Algorithm 1 Overall 1D Pipeline on the domain [0, 1]

Require: Training pairs ${(x_i,y_i)}_{i=1}^N$, initial DoF D, training budget T, top-k selector k Ensure: Final bins $p[0:D]$ and coefficients $c[0:D-1]$ appended as the model 1: === (a) Initialize 1D remainder === 2: Initialize uniform bin edges $p[0:D]$ on $[0, 1]$ and zero coefficients $c[0:D-1]$ 3: Full model: $f\left(x\right)={\sum }_{j=0}^{D-1}{\psi }_j\left(x\right)c_j$ 4: === (b) Alternate train and mesh-update === 5: repeat 6:     TrainPhase1D$(f,p,c,k)$    (updates c and accumulates interval counts) 7:     MeshUpdate1D$(p,c,D,k)$    (refine → coarsen → reindex; resets counts) 8: until validation no longer improves or budget T used 9: return $(p,c)$

Algorithm 2 TrainPhase1D (local fit + interval counting)

Require: Model f, bins p, coeffs c, selector k; batch size B Ensure: Updated c and accumulated counts $u[0:D-1]$ 1: Zero optimizer; set counts $u\leftarrow 0$ 2: for minibatches $\mathcal{B}=\left\{\right(x,y\left)\right\}$ do 3:     Forward: $\widehat{y}=f\left(x\right)$ 4:     Per-sample loss: ${\ell }_i=\ell ({\widehat{y}}_i,y_i)+\lambda {\parallel c_i\left[1:-1\right]-\frac{1}{2}\left(c_i[2:]+c_i[:-2]\right)\parallel }_1$; total $\mathcal{L}={\displaystyle \frac{1}{B}}{\sum }_i{\ell }_i$ 5:     Backprop and optimizer step (updates c) 6:     Select hard samples $I_k\leftarrow \mathtt{TopK}\left({\left\{{\ell }_i\right\}}_{i=1}^B\right)$ 7:     For each $i\in I_k$, find j with $z_i\in [p_j,p_{j+1})$ and set $u_j\leftarrow u_j+1$ 8: end for 9: return $c,u$

Algorithm 3 MeshUpdate1D (Refine → Coarsen → Reindex)

Require: Counts $u[0:D-1]$, bins $p[0:D]$, coeffs $c[0:D-1]$, target DoF D, selector k Ensure: Updated $(p,c)$; counts reset   1: Refine ($(p,c)\leftarrow (p_{\mathrm{fine}},c_{\mathrm{fine}})$).   2: $J\leftarrow \mathtt{TopKIndices}(u,k)$   3: for each $j\in J$ do   4:     Insert midpoint $\frac{p_j+p_{j+1}}{2}$ into p   5: end for   6: Interpolate coefficients to fine mesh: $c_{\mathrm{fine}}\leftarrow \mathtt{linear}\text{\_}\mathtt{interp}(p_{\mathrm{fine}},p,c)$   7: Coarsen & Reindex ($(p,c)\leftarrow (p_{\mathrm{fine}},c_{\mathrm{fine}})$).   8: Estimate the Probability Density Function (PDF) on $p_{\mathrm{fine}}$ from u   9: Compute the Cumulative Distribution Function (CDF) on $p_{\mathrm{fine}}$ 10: Choose $D-1$ interior edges of $b_{\mathrm{fine}}$ for Equal frequency binning; form p 11: Interpolate coefficients to coarse grid: $c\leftarrow \mathtt{linear}\text{\_}\mathtt{interp}(p,p_{\mathrm{fine}},c_{\mathrm{fine}})$ 12: Reset counts. 13: $u\leftarrow 0$ 14: return $(p,c)$

2.6. Scaling Law and Error-Bound Formula

Our approach demonstrates significant interpretability, supported by the universal approximation theorem and our quantitative experimental results. In the FEM theory, the number of parameters N is proportional to the number of simplices $n_t$ in the triangular mesh (refer to Appendix A.2). Since ${(n_t/d!)}^{1/d}=O\left(h^{-1}\right)$, the error bound formula for fitting function $F:{\mathbb{R}}^d\to \mathbb{R}$ is derived as:

$$L\left(N\right)={|f\left(\mathit{x}\right)-F\left(\mathit{x}\right)|}_{\mathit{x}\in \Omega }=O(\underset{\xi \in \Omega }{max}\parallel \nabla F\left(\xi \right)\parallel ·n_t^{-1/d})=O\left(N^{-1/d}\right),$$

(4)

where $h={max}_i{max}_j{\mathbb{1}}_{\begin{array}{c}\mathrm{dist}({\mathit{p}}^{\left(i\right)},{\mathit{p}}^{\left(j\right)})=1\end{array}}·\parallel {\mathit{p}}^{\left(i\right)}-{\mathit{p}}^{\left(j\right)}\parallel$ represents the maximum length of mesh edges (see Appendix A), and $\Omega$ is the domain over which the approximation occurs.

Our experiment shown in Section 3.3 demonstrates that we perfectly predicted the explicit scaling law.

2.7. LReg Applied to Large Models

Based on the interpolation formula Equation (2), we propose LReg (see Supplementary Materials), which is equivalent to MLP. LReg adheres to the universal approximation theorem and exhibits strong interpretability for low-dimensional recognition tasks (see Section 3.1). However, as the data dimension d increases, the output dimension of LReg grows factorially ($O(d!)$) as described in Section 2.6. This exponential growth makes it computationally expensive for high-dimensional data, such as large images, thereby imposing significant challenges in environments with constrained computational resources.

To address this issue, LReg can be incorporated as a module within PEFT methodologies. As shown in Figure 2, consider the target $F\left(\mathit{x}\right)$ and a pre-trained model $f\left(\mathit{x}\right)$. When $\tilde{f}\left(\mathit{x}\right)$ is well trained, the residual $F\left(\mathit{x}\right)-\tilde{f}\left(\mathit{x}\right)$ remains nearly flat around the value of 0 throughout the input space. This sparsity in the residual’s support set suggests that dimensionality reduction on features could serve as an effective strategy for applying LReg to high-dimensional inputs.

In Figure 1c, we use a SigmoidNorm layer to reduce the input dimensionality:

$$\mathit{x}\leftarrow \mathrm{Sigmoid}\left(\mathrm{PCA}\right(\mathit{x}\left)\right).$$

(5)

The built-in Principal Component Analysis (PCA) module can be obtained through standard unsupervised methods or by training. LReg is then applied to learn the residual:

$$F\left(\mathit{x}\right)-\tilde{f}\left(\mathit{x}\right)\approx \sum _i{c}_i{\psi }_i\left(\mathit{x}\right).$$

(6)

Subsequently, the composed $\mathit{x}$ is converted to ${\left\{{\psi }_i\left(\mathit{x}\right)\right\}}_i$ by the encoder of LReg. Finally, the linear combination ${\sum }_i{c}_i{\psi }_i\left(\mathit{x}\right)$ is completed via the Linear layer.

In this scenario, LReg enables the loss function of large pre-trained models to move from local minima. Experiments demonstrate that LReg connection further reduces cross-entropy loss and significantly improves the performance of pre-trained models within a single training epoch (see Section 3.3). This approach effectively fine-tunes the model while preserving computational efficiency.

3. Results

We begin by training LReg from scratch to validate the explicit scaling law in Section 2.6 and identify the limitations of conventional models in fitting multi-frequency data (Section 3.2). The exploration of model fine-tuning for achieving reduced loss function values and enhanced performance is discussed in Section 3.3. For extending the PEFT method to non-transformer-based models via LReg connection, along with comparison results, refer to Section 3.4. Implementation details are provided in Appendix A, while additional applications are discussed in Appendix C.

3.1. Scaling Law and Addressing Regression Challenges

Figure 3 illustrates the performance of the LReg method with varying numbers of model parameters. In these experiments, LReg is trained to approach convergence on fitting the mapping function $F:{\mathbb{R}}^d\to \mathbb{R}$. As shown, the $L\left(N\right)$, denoted as test Mean Absolute Error (MAE) loss, is observed to be proportional to the power of the number of model parameters, $N^{-1/d}$, and the inverse power of the number of simplices, $n_t^{-1/d}$, on the triangulation mesh. This observation is in perfect agreement with the derived error-bound formula in Section 2.6.

Notably, the data learned by the model in Figure 3, represented as $\{(x,y)|{\displaystyle \frac{1}{X}}\sim U(0.02,1.0),$$Y\sim \mathcal{N}(sin{\displaystyle \frac{1}{x}},0.01)\}$, exhibits multi-frequency characteristics that present significant challenges for conventional neural networks to learn effectively. It is worth noting that although distribution $\{(x,y)|{\displaystyle \frac{1}{X}}\sim U(0.02,1.0),Y\sim \mathcal{N}(sin{\displaystyle \frac{1}{x}},0.01)\}$ and distribution $\{(x,y)|X$$\sim U(1,50),Y\sim \mathcal{N}(sinx,0.01)\}$ are equivalent in mathematics, the numerical results of function fitting are completely different. In our open-source GitHub repository, we provide such a demo: for the neural network without LReg connection, the model fits the latter well, but cannot fit the former anyway.

Traditional regressors often struggle with overfitting, while neural networks address this but perform poorly when fitting multi-frequency functions, a phenomenon known as the frequency principle [8]. As shown in Figure 4, Support Vector Regression [9] fits dataset $\mathbb{B}=\{(x,y)|{\displaystyle \frac{1}{X}}\sim u(0.02,0.5),y=sin{\displaystyle \frac{1}{x}}\}$ but overfits on dataset $\mathbb{A}$, whereas neural networks perform well on $\mathbb{A}=\{(x,y)|X\sim u(0,{\displaystyle \frac{\pi }{4}}),Y\sim N(sin8x,|cos8x|)\}$ but underfit on $\mathbb{B}$. Neither MLPs, Convolutional Neural Networks (CNNs), nor Transformers can effectively fit sharp transitions in dataset $\mathbb{B}$, often treating these regions as noise to avoid overfitting. Importantly, no existing model can fit the challenging dataset $\mathbb{C}=\{(x,y)|{\displaystyle \frac{1}{X}}\sim u(0.02,0.5),Y\sim N(sin{\displaystyle \frac{1}{x}},0.5x^2)\}$.

LReg addresses this gap by combining the strengths of traditional regressors and neural networks. It leverages a multiscale mesh to fit high-frequency regions using fine simplices adaptively. Figure 4 (right) demonstrates the LReg’s unique ability to handle dataset $\mathbb{C}$, a task no other model can achieve. In the next section, we further illustrate its gradual fitting process during training. Comparison experiments in

Table 1. A comprehensive comparison between the LReg and traditional regressors. The left half of each paired column displays the training $R^2$ score, while the right half showcases the corresponding test $R^2$ score.

Method	${\mathbb{A}}^1$		${\mathbb{B}}^1$		${\mathbb{A}}^2$		${\mathbb{B}}^2$
OLS Linear	0.037	0.042	0.963	0.951	0.085	0.092	0.984	0.984
Theil-Sen	−44.7	−54.4	0.958	0.946	−0.41	−3.81	0.982	0.982
RANSAC	−1.21	−1.43	0.963	0.951	−27.0	−27.1	0.983	0.983
Huber	0.036	0.041	0.962	0.949	0.085	0.092	0.984	0.984
Ridge	0.031	0.038	0.963	0.951	0.055	0.061	0.984	0.984
RidgeCV	0.037	0.042	0.963	0.951	0.085	0.092	0.984	0.984
SGD	0.009	0.01	0.962	0.95	0.005	0.004	0.983	0.983
KRR	0.0036	0.04	0.97	0.962	0.056	0.051	0.993	0.992
SVR	0.11	0.101	0.97	0.962	0.29	0.308	0.992	0.992
Voting	0.852	0.852	0.942	0.917	0.869	0.868	0.951	0.946
LReg	1.0	1.0	0.971	0.963	0.999	0.999	0.992	0.992

show that the LReg consistently achieves high $R^2$ scores across all test sets, demonstrating its robustness and effectiveness, even in high-noise and multi-frequency scenarios.

3.2. Comparison with Regressors: Training from Scratch

As mentioned earlier, neural networks often struggle to fit multi-frequency datasets effectively. Therefore, our primary focus is comparing the LReg with traditional regressors. To evaluate the effectiveness and generalization of the LReg, we have devised four diverse datasets, each generated from distinct probability distributions:

• ${\mathbb{A}}^1$: Generated from the distribution $\{(x,y)|X\sim U(-\pi ,\pi ),Y\sim \mathcal{N}(sinx,{\displaystyle \frac{1}{5}}{cos}^{2}x)\}$, with 1000 training examples and 200 test examples. The LReg was trained with a learning rate of 0.1.
• ${\mathbb{B}}^1$: Generated from the distribution $\{(x,y)|{\displaystyle \frac{1}{X}}\sim U(0.02,1.0),Y\sim \mathcal{N}(sin{\displaystyle \frac{1}{x}},0.01)\}$, with 1000 training examples and 200 test examples. We trained the LReg with a learning rate of 0.9.
• ${\mathbb{A}}^2$: Generated from the distribution $\{(\mathit{x},y)|{\mathit{X}}_i\sim U(-\pi ,\pi ),Y_i\sim \mathcal{N}(sin{\mathit{x}}_i,{\displaystyle \frac{1}{10}}{cos}^2{\mathit{x}}_i),$
  $Y={\displaystyle \frac{1}{2}}(Y_1+Y_2)\}$, with 7500 training examples and 1500 test examples. The LReg was trained with a learning rate of 0.1.
• ${\mathbb{B}}^2$: Generated from the distribution $\{(\mathit{x},y)|{\displaystyle \frac{1}{{\mathit{x}}_i}}\sim U(0.05,0.5),Y_i\sim \mathcal{N}(sin{\displaystyle \frac{1}{{\mathit{x}}_i}},0.01),$
  $Y={\displaystyle \frac{1}{2}}(Y_1+Y_2)\}$, with 50,000 training examples and 10,000 test examples. Training utilized a learning rate of 0.9.

Table 1 displays the coefficient of determination ($R^2$) scores for the LReg and traditional regressors [9,10,11,12,13,14,15,16,17,18] across fitting the four datasets. The LReg consistently achieves high $R^2$ scores across all test sets, demonstrating the effectiveness of the InterpolationNet on both high-noise and multi-frequency datasets. Furthermore, the minimal gap between training and test set evaluations underscores the robustness of the LReg, indicating its capability of generalization.

We also explore training LReg from scratch on non-regression tasks. Here, we delve into the practical application of LReg for text feature extraction using the AG News dataset [19] for classification tasks. Our experimental findings reveal that LReg can directly extract features from raw text. However, recognizing that tokens are unordered categorical variables, to enhance performance, we add a preprocessing layer that converts each token to a four-dimensional vector, the proportion of the token appearing in four categories.

Our experiment showcases the LReg achieving 90.01% test accuracy after the first epoch, with 90.4% test accuracy reached within just five epochs. It possesses a unique characteristic in text classification tasks: the number of its parameters remains independent of the token count. Compared to word2vec-based networks, our model performs equally well in classification but boasts only 256 model parameters, a significant reduction from the word2vec-based network with more than 6.13 million parameters.

3.3. Improving Pre-Trained Model Performance and Continuously Reducing Loss

As described in Section 2.7, LReg can serve as a connection to enable the loss function of large pre-trained models to escape local minima (see Figure 5). In this module, the SigmoidNorm layer and linear head contain trainable parameters (see Figure 2c). In this section, we illustrate the effectiveness of LReg connection through ablation studies. To ensure a fair comparison with baseline models, we adopt a stricter experimental setup: the original training recipe of the pre-trained model remains unchanged, the entire pre-trained model is frozen, and no additional training data is used.

This restriction is crucial to ablation studies because the training recipes for pre-trained models often have room for optimization [20]. Changes to batch size, optimizer, weight decay rate, or the order of applying data augmentations could potentially improve the performance of the pre-trained model [21,22]. To avoid such effects, we selected pre-trained models from TorchVision with publicly available training recipes as our baselines. The weights of these pre-trained models closely reproduce the results from the original papers on the ImageNet-1K dataset [23], with recipes available at [24].

Our method offers significant practical value. Unlike model scaling and adaptation approaches, it requires no modifications to the raw training recipe, converges within a single epoch, and is applicable to any model, not limited to the transformer-based architecture. For instance, on 4× A6000 GPUs, it adds fewer than one million parameters to a ResNet-50 model and achieves a 0.2% accuracy gain in just 40 min of training. Due to space constraints, we present some of the comparison results in

Table 2. Comparison of LReg with baseline models across different architectures. LReg improves top-1 accuracy while introducing only a negligible increase in trainable parameters. It also achieves either comparable or superior training efficiency by converging within five training epochs, thereby significantly reducing training time compared to full model training. Unlike baseline models that require days of training, LReg enhances performance within an hour. These results demonstrate the effectiveness of LReg as a general, parameter-efficient fine-tuning method for large-scale models, without being limited to transformer-based architectures. Results that surpass competing methods are bold. * indicates numbers published in prior works.

Model	Method	# Trainable Parameters	Acc@1	Acc@5	Speed (img/s)	Training Time (Total)
MobileNet-V2	Baseline *	3.5 M	71.878	90.286	701.51	16 h 37 m
	LReg	328,720	71.934	90.268	685.17	32 m 12 s
ResNet-50	Baseline *	25.6 M	76.130	92.862	440.18	2 d 1 h 15 m
	LReg	540,688	76.274	92.932	411.50	40 m 42 s
ResNeXt-50_32x4d	Baseline *	25.0 M	77.618	93.698	334.72	3 d 1 h 32 m
	LReg	135,172	77.650	93.672	315.33	51 m 9 s
ViT-B16	Baseline *	86.6 M	81.072	95.318	331.02	3 d 3 h 26 m
	LReg	67,076	81.082	95.316	316.31	56 m 54 s

.

In Appendix B, we highlight the strong stability of LReg, showing ±0.07% fluctuation in validation accuracy across independent model training (see Appendix B,

Table A1. Classification accuracy on the ImageNet-1K validation set and one epoch training, with all methods adhering to the raw training recipes. We report the changes in the number of model parameters and their ratio of changes and total. For each model, we conducted multiple experiments and show the accuracy as “${\mathrm{mean}}_{\pm \mathrm{std}}(\mathrm{mean}-\mathrm{baseline})$”.

Model	d	n	# Params	Acc@1 (%)	Acc@5 (%)
MobileNet-V2	32	8	0.298 M (+8.51%)	${71.920}_{\pm 0.016}$  $(+0.045)$	${90.302}_{\pm 0.021}$  $(+0.016)$
ResNet50	32	8	0.324 M (+1.27%)	${76.334}_{\pm 0.054}$  $(+0.204)$	${92.976}_{\pm 0.024}$  $(+0.114)$
ResNeXt50	32	8	0.324 M (+1.30%)	${77.796}_{\pm 0.063}$  $(+0.178)$	${93.653}_{\pm 0.038}$  $(+0.178)$
ViT-B-16	8	32	1.032 M (+1.19%)	${81.092}_{\pm 0.012}$  $(+0.020)$	${95.316}_{\pm 0.004}$  $(-0.002)$

). Trainable parameters in our method originate from the SigmoidNorm and linear layer (see Figure 2c). The total number of trainable parameters can be calculated as $N=(C_{\mathrm{in}}+1)\times d+d\times n\times C$, where N is the number of model parameters, $C_{\mathrm{in}}$ is the SigmoidNorm input dimension, d the PCA output dimension, n is the mesh node count (degrees of freedom), and C the total output dimension.

Increasing n and d improves model performance (see Appendix B,

Table A2. ResNet-50 Performance with different n and d. The table explores the impact of varying n (PCA output dimension) and d (degrees of freedom) on the performance of the LagEncoder-based ResNet-50 model. Metrics include the number of additional parameters, top-1 (Acc@1) and top-5 (Acc@5) validation accuracy, and training speed (images per second). The baseline ResNet-50 has 25.6 M parameters, Acc@1: 76.130%, and Acc@5: 92.862%. Results that surpass competing methods are bold.

Coeff (n, d)	(4, 4)	(4, 8)	(4, 16)	(4, 32)	(8, 4)	(8, 8)	(8, 16)	(8, 32)
# Parameters	36,868	69,636	135,172	266,244	73,736	139,272	270,344	532,488
Acc@1	76.226	76.226	76.242	76.222	76.238	76.256	76.268	76.276
Acc@5	92.952	92.960	92.948	92.972	92.964	92.954	92.966	92.952
Speed (img/s)	405.31	411.18	390.26	418.84	382.22	394.63	409.52	404.43
Coeff (n, d)	(16, 4)	(16, 8)	(16, 16)	(16, 32)	(32, 4)	(32, 8)	(32, 16)	(32, 32)
# Parameters	147,472	278,544	540,688	1,064,976	294,944	557,088	1,081,376	2,129,952
Acc@1	76.272	76.244	76.248	76.284	76.298	76.272	76.248	76.294
Acc@5	92.940	92.946	92.968	92.958	92.956	92.958	92.968	92.932
Speed (img/s)	393.19	394.86	379.72	380.30	393.34	414.39	379.72	368.94

). We also study empirical scaling laws for model performance on cross-entropy loss to demonstrate how our method furthers the loss value of pre-trained models. Ref. [5] proposed an empirical formula where the loss scales as a power-law with model size:

$$L(N)={\left({\displaystyle \frac{c}{N}}\right)}^{\alpha }$$

(7)

where $L(N)$ is the cross-entropy loss on the validation dataset, and c and $\alpha$ are constants related to the model type. Figure 5 shows that our method is more effective than simply scaling the model size. Our method increases the $\alpha$ for MobileNet [25] from 0.27 to 0.47, for ResNet [26] from 0.25 to 2.1, for ResNeXt [27] from 0.01 to 3.1, and for ViT [28] from −0.06 to 1.2.

3.4. Transfer Learning via LReg Connection

Through experiments, we identified three key limitations of existing PEFT methods: (1) Dependence on Transfer Learning: Methods like LoRA [29] struggle to outperform pre-trained models on the same or similar datasets, when no domain adaptation is needed. This means they cannot continuously reduce the loss function on the raw training set. Additional training data is often required for these methods to be effective. Therefore, their strength lies mainly in transfer learning scenarios (see Appendix B). (2) Sensitive Training Requirements: PEFT methods require specific training recipes, such as small learning rates. Without these, performance often deteriorates from the first epoch. (3) Task and Architecture Limits: PEFT methods, like LoRA, LoHA [30], and IA3 [31], are restricted to transformer-based architectures.

We compare fine-tuning (FT), BitFit [32], LoRA, and LReg connection on the GLUE dataset [33]; subsequently, we conduct additional experiments on WikiText benchmarks [34] using GPT2 [35] for causal language modeling and Roberta [36] for masked language modeling with a default random seed (42). Pre-trained models of the HuggingFace Transformers library [37] are used. A reduced learning rate (≤10⁻⁵) is applied for LoRA to prevent performance degradation in these experiments. As shown in

Table 3. GPT2 and RoBERTa with Different Adaptation Methods on the GLUE dataset and WikiText Benchmark. We report the overall (both matched and mismatched) accuracy for MNLI and the accuracy for other tasks. For each model, we conducted multiple experiments and show the accuracy as “${\mathrm{mean}}_{\pm \mathrm{std}}$”. For a fair comparison, we adjusted the number of parameters of our method to be similar to those of LoRA. * indicates numbers published in prior works. We also report the number of trainable parameters, perplexity (PPL, with lower values being preferable), and training throughput (measured in sequences per second) for language modeling tasks. Results that surpass competing methods are bold.

	MNLI		SST-2		MRPC		QNLI		QQP		RTE
Method	# Params	Accuracy	# Params	Accuracy	# Params	Accuracy	# Params	Accuracy	# Params	Accuracy	# Params	Accuracy
FT *	125 M	87.6	125 M	94.8	125 M	90.2	125 M	92.8	125 M	91.9	125 M	78.7
BitFit *	0.1 M	84.7	0.1 M	93.7	0.1 M	92.7	0.1 M	91.8	0.1 M	84.0	0.1 M	81.5
LoRA *	0.295 M	87.5±0.3	0.295 M	${95.1}_{\pm 0.2}$	0.295 M	${89.7}_{\pm 0.7}$	0.295 M	${93.3}_{\pm 0.3}$	0.295 M	${90.8}_{\pm 0.1}$	0.295 M	${86.6}_{\pm 0.7}$
LReg	0.259 M	${87.9}_{\pm 0.5}$	0.259 M	${95.3}_{\pm 0.6}$	0.259 M	${90.5}_{\pm 0.4}$	0.259 M	${93.5}_{\pm 0.2}$	0.259 M	${89.7}_{\pm 0.4}$	0.259 M	${83.5}_{\pm 0.3}$
	GPT2			GPT2-Medium			GPT2-Large			Roberta-Base
Method	# Params	PPL	Seq/s	# Params	PPL	Seq/s	# Params	PPL	Seq/s	# Params	PPL	Seq/s
FT	124.4 M	21.4234	6.98	354.8 M	15.8900	3.50	774.0 M	13.8468	1.58	124.7 M	3.6415	10.05
LoRA	0.295 M	21.4188	12.90	0.393 M	15.8873	6.29	0.737 M	13.8486	1.67	0.295 M	3.6425	9.85
LReg	0.203 M	21.3401	9.58	0.404 M	15.8674	6.66	0.607 M	13.8443	2.52	0.303 M	3.6414	12.37

, our method overcomes the mentioned challenges, performing well even without additional training data. Therefore, it is also versatile for transfer learning.

4. Discussion

It is important to acknowledge the limitations associated with LReg. As described in Section 2.7, the substantial computational cost of extracting features from high-dimensional data restricts its direct application to large-scale datasets. Presently, we address this by incorporating LReg as an adaptation method, but we aim to extract features directly from large-scale data in the future. Additionally, the experiments in Section 3.3 indicate that, although our adaptation method significantly reduces cross-entropy on the ImageNet-1K validation set, the extent of accuracy improvement is relatively modest, warranting further investigation. We did not compare LReg with models such as Support Vector Machine (SVM) known for their robust interpretability, due to fundamentally different underlying principles, and the usual missing experimental results from interpretable models on large datasets. Specifically, we now outline that since our method is plug-and-play and can be integrated into any existing models, and given that large generative models are a prominent trend in AI research, we plan to apply our method to diffusion models and large language models in future studies.

5. Conclusions

In this paper, we reformulate MLPs and introduce a mathematically equivalent framework named LReg. Furthermore, we propose a discrete method, referred to as the MRC loop, which serves as an alternative to traditional continuous optimization techniques, thereby enabling the loss function to reach global minima. For general pre-trained models, we demonstrate that our approach can significantly reduce the loss function value.

From a theoretical perspective, we construct a visual example to elucidate the intrinsic challenge encountered by traditional optimization methods in reducing the loss function from local minima to global minima, and highlight the benefits of discrete parameter updates in addressing this challenge. Furthermore, we analyze the error-bound formula and, for the first time, derive an explicit scaling law, while we empirically validate through regression tasks. From an application perspective, LReg effectively mitigates the core limitation of traditional neural networks in learning multi-frequency data. Additionally, LReg serves as a plug-and-play module, and our analysis of the scaling law shows that it substantially reduces the loss function value of pre-trained models while providing moderate improvement in performance. In contrast to scaling up a model and training it from scratch, our approach is more efficient. Furthermore, LReg broadens the application scope of PEFT methods in transfer learning, without requiring the pre-trained model to be transformer-based.

This work opens new directions for research in model training and establishes a foundation for further advancements in discrete optimization.