An Adaptive Multipath Linear Interpolation Method for Sample Optimization

Abstract

When using machine learning methods to make predictions, the problem of small sample sizes or highly noisy observation samples is common. Current mainstream sample expansion methods cannot handle the data noise problem well. We propose a multipath sample expansion method (AMLI) based on the idea of linear interpolation, which mainly solves the problem of insufficient prediction sample size or large error between the observed sample and the actual distribution. The rationale of the AMLI method is to divide the original feature space into several subspaces with equal samples, randomly extract a sample from each subspace as a class, and then perform linear interpolation on the samples in the same class (i.e., K-path linear interpolation). After the AMLI processing, valid samples are greatly expanded, the sample structure is adjusted, and the average noise of the samples is reduced so that the prediction effect of the machine learning model is improved. The hyperparameters of this method have an intuitive explanation and usually require little calibration. We compared the proposed method with a variety of machine learning prediction methods and demonstrated that the AMLI method can significantly improve the prediction result. We also propose an AMLI plus method based on the linear interpolation between classes by combining the idea of AMLI with the clustering method and present theoretical proofs of the effectiveness of the AMLI and AMLI plus methods.

1. Introduction

When using machine learning models to make predictions, a series of problems, such as insufficient sample sizes, missing parts of data, or large observation errors, are common. Especially for small sample datasets, how to improve model performance by using effective sample optimization techniques is crucial.

Improving model performance by increasing the number of numerical data can be traced back to the interpolation method at the earliest. Boor and Carl [1] proposed the cubic spline interpolation method, and Mitas and Mitasova [2] proposed the spatial interpolation method. Lu and Wong [3] proposed an adaptive inverse distance spatial interpolation algorithm, which uses the inverse proportional relationship between the distance between neighbors and the interpolation weight. Efron [4] proposed using the bootstrap resampling method based on jackknife; Chawla et al. [5] proposed the synthetic minority oversampling technique (SMOTE); Pan and Yang [6] proposed a transfer learning method to simultaneously model different types of label samples to increase sample size for model training. Fernandez et al. [7] proposed the SMOTE smoothing method, which constructs a new sample by randomly selecting a sample and randomly selecting multiple samples from its K-nearest neighbor.

With the advent of the era of big data, there are also some studies on sample optimization in machine learning and deep learning. Zhu (2005) [8] proposed active learning and semi-supervised learning using the existing samples in the original sample space and using certain algorithms to label the unlabeled samples with high quality to achieve the effect of sample optimization. Eisenberger et al. [9] proposed an unsupervised shape interpolation method based on a neural network.

Recent technology research found that Few-Shot Learning has become a promising future development direction. Wang et al. [10] pointed out that the most important point in Few-Shot Learning was the use of prior knowledge. Prior knowledge of Few-Shot Learning comes from three sources: data, models, and algorithms. Current machine learning methods are in stark contrast to human perception, and their sample learning efficiency is very low. Few-Shot Learning is an exciting area of machine learning; it can solve the problem of low sample learning efficiency. Based on matching neural networks (matching learning), Vinyals et al. [11] proposed an LSTM to calculate the supported FCE, and they also optimized the sample by adding another LSTM to modify the embedding of the query samples. Snell et al. [12] proposed a striking inductive bias in the form of a class of prototypes in prototypical networks, and its achieved Few-Shot Learning performance can exceed matched networks without FCE complexity. Kokol et al. (2022) [13] proposed the synthetic data learning method and demonstrated that small samples can be better than large samples of low quality in the context of statistical machine learning. Zhou et al. (2022) [14] proposed a new improved multiscale edge-labeling graph neural network (MEGNN) to address the small sample size problem by acquiring as much feature information as possible.

When the sample size is expanded, the distribution of the added samples often deviates greatly from that of the actual samples (unknown). Moreover, the data observed daily often contain noise, and the expansion of samples with noise often aggravates the influence of the noise on the prediction result. The adaptive multipath linear interpolation AMLI method proposed in this paper can effectively solve these problems and ensure that most of the added samples are valid ones (i.e., their distribution generally deviates only slightly from the actual distribution). The AMLI method is mainly based on the linear interpolation method to expand the samples of the original data. The idea is to divide the original feature space into several subspaces with an equal number of samples, extract one sample from each subspace as a class, and then perform linear interpolation for the samples in the same class, which is K-path linear interpolation. This method requires two hyperparameters (K and $\eta$) in advance. The visual interpretation of parameter K is the number of samples existing in each feature subspace, while $\eta$ is the number of samples interpolated per unit distance in the linear interpolation of the samples. In the simulation and empirical research, we found that the selection of parameter K is critical, which varies with different samples. By selecting appropriate hyperparameters, many valid samples can be expanded, and the proportion of samples in which the observed value deviates greatly from the actual value is reduced so that the composition structure of samples with error is adjusted and sample optimization is achieved; consequently, the impact of observation noise on the prediction result is also greatly reduced.

2. Research Hypothesis and Methodology Statement

This section describes the steps of the AMLI method and some assumptions that should be satisfied when using the AMLI method.

For a given training dataset:

$$T=\{({\mathit{x}}_1,y_1),({\mathit{x}}_2,y_2),\dots ,({\mathit{x}}_N,y_N)\},$$

where ${\mathit{x}}_i=(x_i^{(1)},\dots ,x_i^{(n)})\in \mathcal{X}\subseteq R^n$ is the feature vector of the example, and n is the feature dimension. $y_i\in R$ is the corresponding output; $i=1,2,\cdots ,N$, where N represents the sample size. Assuming

$$y_i=f({\mathit{x}}_i-{ϵ}_i)+\widetilde{ϵ},i=1,2,\dots ,N,$$

where $f(·)$ is a continuous function; ${ϵ}_i$ is the set of independent and identically distributed observation noise; and $\widetilde{ϵ}$ is the model error. The specific steps of the AMLI method are as follows.

First, the hyperparameter K is determined, and the feature space $\mathcal{X}$ is divided into $N/K$ feature subspaces, each subspace containing K observation samples. Second, a sample is randomly selected from each subspace to form a set $S_d,$$d=1,2,\dots ,K,$ and then, we have:

$$T=\{S_1,S_2,\dots ,S_K\},$$

where each $S_d$ contains $N/K$ samples.

∃${\mathit{x}}_0\in$$\mathcal{X}$, for $\forall i,$ we have ${\mathit{x}}_0^{(i)}=\underset{\mathit{x}\in \mathcal{X}}{inf}\left\{x^{(i)}\right\}$ and call ${\mathit{x}}_0$ the feature space minimum point. Let $L(·)$ be the distance measurement function of the feature space $\mathcal{X}$ ($S_d=\{{\mathit{x}}^{(d,1)},\dots ,{\mathit{x}}^{(d,N/K)}\}$, ${\mathit{x}}^{(d,1)}=\underset{\mathit{x}\in s_d}{argmin}L(\mathit{x},{\mathit{x}}_0)$); we call the point in each sample class nearest to ${\mathit{x}}_0$ the minimum point of the sample class ($d=1,\dots ,K.$); and ${\mathit{x}}^{(d,h+1)}=\underset{\{\mathit{x}:\mathit{x}\in S_d,\mathit{x}\ne {\mathit{x}}^{(d,1)},\dots ,{\mathit{x}}^{(d,h)}\}}{argmin}L(\mathit{x},{\mathit{x}}^{(d,h)})$, in which $S_d$ is the observation point having the second nearest distance to ${\mathit{x}}^{(d,h)}$ (after ${\mathit{x}}^{(d,1)},\dots ,{\mathit{x}}^{(d,h)}$) ($h=1,2,\dots ,N/K-1$).

After determining the minimum point ${\mathit{x}}^{(d,1)}$ of the sample class, all the samples in the set are searched to find the ${\mathit{x}}^{(d,1)}$ point in the feature space nearest to ${\mathit{x}}^{(d,2)}$, and ${\mathit{x}}^{(d,3)},\dots ,{\mathit{x}}^{(d,N/K)}.$ We define the unit distance filling parameter $\eta$ and use linear interpolation in the feature space to interpolate ${\sum }_{i=1}^{N/K-1}\lfloor \eta ·L({\mathit{x}}^{(d,i)},{\mathit{x}}^{(d,i+1)})\rfloor$ (for the sake of convenience, the number of interpolated samples in this paper is rounded down and will not be specifically mentioned below) virtual samples, and the interpolated samples are equally spaced. ${\mathit{x}}_d^{(h,h+1,i)}$ represents the ith dummy sample interpolated between ${\mathit{x}}^{(d,h)}$ and ${\mathit{x}}^{(d,h+1)}$($i=1,\dots ,\eta ·L({\mathit{x}}^{(d,h)},{\mathit{x}}^{(d,h+1)})$) in $S_d$, and for ${\mathit{x}}_d^{(h,h+1,i)}$ and its corresponding output $y_d^{(h,h+1,i)}$, it satisfies:

$${\mathit{x}}_d^{(h,h+1,i)}={\mathit{x}}^{(d,h)}+i·\frac{{\mathit{x}}^{(d,h+1)}-{\mathit{x}}^{(d,h)}}{\eta ·L({\mathit{x}}^{(d,h)},{\mathit{x}}^{(d,h+1)})+1},$$

$$y_d^{(h,h+1,i)}=y^{(d,h)}+i·\frac{y^{(d,h+1)}-y^{(d,h)}}{\eta ·L({\mathit{x}}^{(d,h)},{\mathit{x}}^{(d,h+1)})+1}.$$

The above linear interpolation is performed on all sample classes, the number of interpolations is ${\sum }_{d=1}^K{\sum }_{i=1}^{N/K-1}\eta ·L({\mathit{x}}^{(d,i)},{\mathit{x}}^{(d,i+1)})$, and all dummy samples are added to the dataset.

The AMLI method divides the original feature space into $N/K$ subspaces with equal samples and then randomly selects a sample in each subspace as a sample class. When performing sample classification operations in practical applications, we can calculate the distances of all observation samples in the dataset to the minimum point of the feature space and classify each sample into its respective sample class in ascending order. The specific implementation process of the AMLI method is detailed in Algorithm 1.

Algorithm 1:AMLI

3. Simulation Experiments

3.1. Monte Carlo Simuations

In this section, we will demonstrate the optimization effect of the samples added by using the AMLI method on the overall samples by using six groups of Monte Carlo simulations. For the sake of simplicity and to achieve a better visualization effect, the feature dimension n = 1 is defined, and the real function relationship selected is $y=x^3$. Since there is a certain error between the observed value of the sample and the actual value, to simulate this effect, we add noise to the samples after data generation.

The distribution of data and the settings of noise and sample size are as follows:

Simulation 1: $N=200,$ $x\sim U(-2.5,2.5),$ $ϵ\sim N(0,1);$

Simulation 2: $N=500,$ $x\sim U(-2.5,2.5),$ $ϵ\sim N(0,1);$

Simulation 3: $N=800,$ $x\sim U(-2.5,2.5),$ $ϵ\sim N(0,1);$

Simulation 4: $N=200,$ $x\sim N(0,2.5),$ $ϵ\sim N(0,1);$

Simulation 5: $N=200,$ $x\sim t(5),$ $ϵ\sim N(0,1);$

Simulation 6: $N=200,$ $x\sim U(-2.5,2.5),$ $ϵ\sim U(-1.732,1.732).$

Simulation 1 is used as the control group, and simulations 2 and 3 are used as experimental groups with different sample sizes; simulations 4 and 5 are experimental groups with different feature distributions; simulation 6 is an experimental group with different types of noise (noise distribution parameters are selected to unify the noise variance and to ensure that the expectation is 0). Some verification indicators (e.g., the proportion of samples with an error greater than 0.5, 1, 1.5, 2, or 2.5; and the mean square error ($MSE$) between the sample observation values and the actual value) are selected to test the optimization effect of the AMLI method on the original sample after processing.

$$p_{(\alpha )}=\frac{{\sum }_{i=1}^{N+{\sum }_{d=1}^K{\sum }_{i=1}^{N/K-1}\eta ·L({\mathit{x}}^{(d,i)},{\mathit{x}}^{(d,i+1)})}I(|{\mathit{x}}_i-{\mathit{x}}_i^{*}|>\alpha )}{N+{\sum }_{d=1}^K{\sum }_{i=1}^{N/K-1}\eta ·L({\mathit{x}}^{(d,i)},{\mathit{x}}^{(d,i+1)})},$$

$$MSE(\mathit{x},{\mathit{x}}^{*})=\frac{{\sum }_{i=1}^N{({\mathit{x}}_i-{\mathit{x}}_i^{*})}^2}{N},$$

where $\mathit{x}$ is the observed value of the sample; ${\mathit{x}}^{*}$ is the actual value of the sample; $N+{\sum }_{d=1}^K{\sum }_{i=1}^{N/K-1}\eta ·L({\mathit{x}}^{(d,i)},{\mathit{x}}^{(d,i+1)})$ is the number of total samples after AMLI optimization; and $\alpha$ = 0.5, 1, 1.5, 2, 2.5. Due to the randomness of each experiment, we selected different hyperparameters to perform multiple calibrations in each simulation and calculated the average value of each of the verification indicators for every 100 experiments.

To reflect the optimization effect of the AMLI method more visually, below, we describe the AMLI processing process of simulation 1 in detail. First, 200 samples are generated evenly in the interval of (−2.5, 2.5) (Figure 1(1)). Second, noise that obeys the standard normal distribution is added to the feature variables (Figure 1(2)). Third, to achieve a better visualization effect, original samples are divided into only four categories (just to achieve a better visualization effect, and the verification index is not necessarily optimal) (i.e., the hyperparameter K = 4) with labels of different colors (Figure 1(3)). Fourth, given the small interval of the definition domain, high values of the unit distance filling parameter are selected, and sample filling is performed after setting the hyperparameter $\eta$ = 100. After the filling, the sample size reaches 3035 (Figure 1(4)).

The comparison of the visualization effects in Figure 1(2) and Figure 1(4) indicates that after processing with the AMLI method, the samples can adaptively fit the functional relationship between x and y.

By using the above method, the parameters of the six simulations are calibrated, and the verification indicators are calculated, as shown in

Table 1. Monte Carlo simulation results.

Simulation	MSE		${\mathit{p}}_{(0.5)}$		${\mathit{p}}_{(1)}$		${\mathit{p}}_{(1.5)}$		${\mathit{p}}_{(2)}$		${\mathit{p}}_{(2.5)}$
	Before	After	Before	After	Before	After	Before	After	Before	After	Before	After
1	0.957	0.762	0.604	0.570	0.316	0.251	0.130	0.083	0.036	0.022	0.010	0.006
2	0.980	0.774	0.614	0.567	0.313	0.258	0.132	0.089	0.043	0.023	0.011	0.005
3	0.977	0.789	0.619	0.571	0.308	0.254	0.131	0.090	0.043	0.025	0.011	0.005
4	0.990	0.701	0.609	0.549	0.317	0.232	0.133	0.073	0.044	0.017	0.011	0.003
5	0.982	0.756	0.611	0.552	0.306	0.245	0.127	0.085	0.038	0.024	0.010	0.005
6	0.987	0.742	0.706	0.632	0.411	0.297	0.126	0.057	0.000	0.000	0.000	0.000

.

Remark 1.

The optimal effect of each simulation is determined by the lowest $MSE$ after AMLI processing by traversing all the hyperparameter values according to the grid search method.

Clearly, after the AMLI processing, the $MSE$ of the samples and the proportion of various error samples have been optimized, most of the added dummy samples are valid ones, the increase in the sample size does not significantly weaken the AMLI’s optimization effect, the samples that obey the normal distribution have better optimization performance, and even the uniformly distributed noise shows good robustness. In many cases, the AMLI method performs well in data optimization.

3.2. Analysis of Hyperparameter Taking Values

In this section, we discuss the patterns of setting the parameters of K and $\eta$ in the AMLI method. First, we examine the setting of parameter K. For the above simulation, we set the parameter $\eta$ = 100; perform traversal iterations consecutively by setting $K=1,2\dots ,200$; and obtain the average value of the index for one hundred iterations at each setting. The change trend of the MSE index before and after the AMLI processing is shown in Figure 2.

At fixed parameter $\eta$ values, the optimal value of K increases with the increase in the sample size; the value range of K varies under different distributions of variables; and the change in the noise distribution has little impact on the optimal value of K.

Next, we examine the pattern of the setting of parameter $\eta$. In the case of simulation 1, under the condition of $K=21$ to take the optimal value, traversal iterations are performed consecutively by setting the parameter $\eta =1,2,\dots ,200$; the results are shown in Figure 3. Clearly, the fluctuation of the sample $MSE$ after optimization by using the AMLI method decreases with the increase in the value of filling parameter $\eta$.

3.3. Comparison with Other Interpolation Methods

The above experiments show that the AMLI method can greatly expand valid samples and reduce the uniform error of samples as a whole so that the proportion of samples with large errors is low. The AMLI method achieves sample optimization based mainly on the idea of interpolation. The main interpolation methods include linear interpolation, quadratic spline interpolation and cubic spline interpolation. In this section, we will compare the AMLI method with other interpolation methods to demonstrate its superiority.

Similarly, simulation 1 is used as the control group, and simulations 2–6 are used as the experimental group, using various interpolation methods with a fixed interpolation number of 3500–4000. Only the MSE between the processed samples and the actual values is selected as the evaluation index, and for each simulation, one hundred experiments are carried out, and their average value is taken. The results are shown in

Table 2. $MSE$ values of samples under different interpolation methods.

Simulation	MSE
	AMLI	Linear Interpolation	Quadratic Spline	Cubic Spline
1	0.762	1.385	5.542	9.452
2	0.774	1.618	6.434	8.976
3	0.789	1.812	6.441	8.338
4	0.701	1.578	18.156	42.997
5	0.756	1.203	10.322	29.542
6	0.742	0.959	4.802	6.619

. In terms of the $MSE$, the $AMLI$ method clearly outperforms the other methods, indicating that the $AMLI$ method has achieved a very significant optimization effect.

4. Application of AMLI Method in Machine Learning

In this section, we will describe the performance of the AMLI method in both simulated and actual data prediction when the method is combined with various machine learning models. Using the side-by-side method, we divide the training set and the test set in the ratio of 7:3; the training set is optimized by using the AMLI method. For the machine learning method, we select the K-Nearest Neighbor ($KNN$) method, Feedforward Neural Network ($FNN$), Gradient Boosting Decision Tree ($GBDT$) and Random Forest ($RF$), with the $MSE$ as the loss function. The hyperparameters of each model and the parameters of K and $\eta$ of the AMLI method are calibrated multiple times to obtain the optimal values. The Euclidean distance is adopted as the distance function:

$$MSE=\frac{1}{N}\sum _{i=1}^N{(y_i-{\widehat{y}}_i)}^2,$$

$$L(x_i,x_j)=\sqrt{\sum _{l=1}^n{(x_i^l-x_j^l)}^2}.$$

4.1. Simulated Data Prediction

We assume that for simulated samples with a size of 1000, the simulated data feature dimension $n=3$, different feature dimensions obey different distributions, ${\mathit{x}}^1\sim N(0,3),$${\mathit{x}}^2\sim U(-3,3),{\mathit{x}}^3\sim t(5),$ and the weight vector of ${\mathit{w}}_{(3,2)}^1,{\mathit{w}}_{(2,1)}^2$ is randomly generated. Let $\mathit{Y}=\mathit{X}·{\mathit{w}}_{(3,2)}^1·{\mathit{w}}_{(2,1)}^2+\widetilde{ϵ}$. After the data are generated, noise that obeys the Gaussian distribution is added to $\mathit{X}$, which is then further divided into the test set and the training set. The training set is processed using the $AMLI$ method (K = 40, $\eta$ = 5), and the total number of samples after the processing is 9810.

As shown in

Table 3. Simulation data prediction results.

MSE	KNN	FNN	GBDT	RF
Before AMLI processing	1.70	0.942	1.210	1.507
After AMLI processing	1.07	0.713	1.008	1.320

, the $MSE$ of the trained model using the data processed by the $AMLI$ method is smaller in the prediction, indicating that the prediction result is more accurate.

4.2. Actual Data Prediction

4.2.1. Demand Forecast for Shared Bike Rental

In this section, we present a case study by predicting the demand for shared bicycle rentals in a certain city. The dataset includes multiple variables, such as season, holiday, temp, and registered (

Table 4. Description of variable indicators.

Variable Name	Variable Definition
Season	1 = Spring
	2 = Summer
	3 = Autumn
	4 = Winter
Holiday	1 = Holiday
	0 = Non-holiday
Workdays	1 = Working day
	0 = Weekend
Weather	1 = Sunny, cloudy
	2 = Foggy, overcast
	3 = Light snow, drizzle
	4 = Heavy rain, heavy snow, heavy fog
Temp	Temperature in Celsius
Atemp	Apparent temperature
Humidity	Humidity
Windspeed	Wind speed
Casual	Number of nonregistered users
Registered	Number of registered users
Count	Total number of car rentals

). We use the AMLI method to optimize samples of actual data. On the one hand, we combine the AMLI method with machine learning methods to examine its optimization performance in actual predictions; on the other hand, given that the dataset contains multiple categorical data, we can explore whether the AMLI method can achieve good optimization in prediction when the assumptions of the AMLI method are violated.

The dataset contains 7620 observation samples. We select 1000, 3000, and 7620 samples consecutively to examine the prediction optimization of the AMLI method in combination with machine learning methods when the sample size is insufficient, fair, or sufficient.

As shown in

Table 5. Prediction results of shared bicycle rental demand (MSE).

N	Hyper-Parameter	N (After Processing)	AMLI Processing	KNN	FNN	GBDT	RF
1000	$K=25$	191,057	Before	208.230	0.9677	88.152	231.210
	$\eta =5$		After	46.340	0.0791	68.394	158.940
3000	$K=40$	290,856	Before	42.757	0.3342	22.794	43.679
	$\eta =3$		After	26.020	0.0084	14.301	25.451
7620	$K=65$	428,277	Before	17.464	0.1168	9.879	23.521
	$\eta =1$		After	6.798	0.0051	6.981	2.472

, at various sample size levels, the AMLI method has achieved a certain optimization of prediction effect.

4.2.2. Concentration Forecast for $P{M}_{2.5}$

In this section, we present a case study by predicting the concentration for $P{M}_{2.5}$ in a certain city to rule out the randomness of the dataset. The dataset includes multiple concentrations of air pollutants such as $P{M}_{10}$, CO, $S{O}_2$, $N{O}_2$ and $O_3$. The dataset contains 3193 observation samples. We select 500, 1500, and 3193 samples consecutively to examine the prediction optimization of the $AMLI$ method. As shown in

Table 6. Prediction results of shared bicycle rental demand ($MSE({10}^{-3}))$.

N	Hyperparameter	N (After Processing)	AMLI Processing	$\mathit{KNN}$	$\mathit{FNN}$	$\mathit{GBDT}$	$\mathit{RF}$
500	$K=17$	1089	Before	3.16	4.45	3.14	3.39
	$\eta =10$		After	2.89	2.55	2.36	3.01
1500	$K=23$	7003	Before	2.69	2.54	2.33	2.62
	$\eta =30$		After	2.18	2.07	1.86	2.33
3193	$K=50$	24,559	Before	1.77	1.63	1.48	1.54
	$\eta =60$		After	1.41	1.13	1.07	1.16

, the $AMLI$ method still has a good optimization effect.

5. Proof

In this section, we will present the proof for the samples that satisfy the AMLI hypothesis that after processing with the AMLI method, the average observation error of the samples is reduced, and the proportion of samples with different errors is adjusted.

The rationale of AMLI is to divide the original feature space into $N/K$ subspaces with K samples, randomly extract a sample from each subspace as a class to divide the original dataset into K classes, and then perform linear interpolation between two adjacent samples. In essence, AMLI can be viewed as a method of linear interpolation between two subspaces with close distances through K paths.

For the dataset $T=\{(x_1,y_1),(x_2,y_2),\dots ,(x_N,y_N)\}$, we select two subspaces that are assumed to have equal samples: ${\mathcal{X}}^{(1)},{\mathcal{X}}^{(2)}\subseteq \mathcal{X}\subseteq R,$ and ${\mathcal{X}}^{(1)}\cap {\mathcal{X}}^{(2)}=\varnothing$. In these two subspaces, we assume the following common relation is present:

$$\begin{array}{c}\hfill y=f(x^{*})+\widetilde{ϵ}=f(x-ϵ)+\widetilde{ϵ},\end{array}$$

(1)

where $x^{*}$ is the actual value of x after removing the observation noise; and where $x=x^{*}+ϵ$, where $ϵ$ is the noise term, $ϵ\sim N(0,{\sigma }^2),$ and ${\widetilde{ϵ}}_i$ is the model error.

According to our assumption, $f(·)$ is a continuous function; if ${\mathcal{X}}^{(1)},{\mathcal{X}}^{(2)}\to 0$, and ${\mathcal{X}}^{(1)}\cap {\mathcal{X}}^{(2)}=\varnothing ,dis({\mathcal{X}}^{(1)},{\mathcal{X}}^{(2)})\to 0$, $f(·)$ can be approximated as a linear function of $g(·)$, and then (1) can be transformed into

$$\begin{array}{c}\hfill y=g(x^{*})+{ϵ}_1+\widetilde{ϵ}=g(x-ϵ)+{ϵ}_1+\widetilde{ϵ},\end{array}$$

(2)

where ${ϵ}_1$ is the linear fitting error term and ${ϵ}_1\to 0.$

Since the samples we select in ${\mathcal{X}}^{(1)},{\mathcal{X}}^{(2)}$ are equal in size, we assume there are K observation samples; i.e., ${\mathit{x}}^{(j)}=(x_1^{(j)},\dots ,x_K^{(j)})=(x_1^{*(j)}+{ϵ}_1^{(j)},\dots ,x_K^{*(j)}+{ϵ}_K^{(j)})\in {\mathcal{X}}^{(j)},j=1,2.$ The expectation of the absolute value of the uniform noise in the space at this time is ${\mathcal{X}}^{(1)},{\mathcal{X}}^{(2)}$:

$$\begin{array}{ccc}\hfill E(\frac{{\sum }_{j=1}^2{\sum }_{i=1}^K\left|{ϵ}_i^{(j)}\right|}{2K})& =& {\int }_{-\infty }^{+\infty }\left|t\right|\frac{1}{\sqrt{2\pi }·\sigma }{e}^{-\frac{t^2}{2{\sigma }^2}}dt\hfill \\ & =& \sqrt{\frac{2}{\pi }}·\sigma {\int }_0^{+\infty }\frac{t}{\sigma }{t}^{-\frac{t^2}{2{\sigma }^2}}d\frac{t}{\sigma }\hfill \\ & =& \sqrt{\frac{2}{\pi }}·\sigma {\int }_0^{+\infty }{e}^{-\frac{t^2}{2{\sigma }^2}}d\frac{{(\frac{t}{\sigma })}^2}{2}\hfill \\ & =& \sqrt{\frac{2}{\pi }}·\sigma ·\mathsf{\Gamma }(1)=\sqrt{\frac{2}{\pi }}·\sigma .\hfill \end{array}$$

The expectation of the proportion of observed samples with a noise greater than 0.5 is

$$\begin{array}{ccc}\hfill E(\frac{{\sum }_{j=1}^2{\sum }_{i=1}^{K}I(|x_i^{(j)}-x_i^{*(j)}|)>0.5}{2K})& =& E(\frac{{\sum }_{j=1}^2{\sum }_{i=1}^{K}I(|{ϵ}_i^{(j)}|)>0.5}{2K})\hfill \\ & =& \frac{{\sum }_{j=1}^2{\sum }_{i=1}^K(P({ϵ}_i^{(j)}>0.5)+P({ϵ}_i^{(j)}<-0.5))}{2K}\hfill \\ & =& \frac{{\displaystyle {\sum }_{j=1}^2{\sum }_{i=1}^K(P(\frac{{ϵ}_i^{(j)}}{\sigma }>\frac{0.5}{\sigma })+P(\frac{{ϵ}_i^{(j)}}{\sigma }<\frac{-0.5}{\sigma }))}}{2K}\hfill \\ & =& {\displaystyle \frac{{\displaystyle {\sum }_{j=1}^2{\sum }_{i=1}^K\mathsf{\Phi }(\frac{-0.5}{\sigma })}}{K}=2\mathsf{\Phi }(\frac{-0.5}{\sigma }).}\hfill \end{array}$$

We randomly select two samples from the subspace to perform linear interpolation on them and iterate the process K times. The AMLI method determines the number of interpolation samples according to the distance between two samples. For the sake of simplicity and convenience, we assume that the number of samples in each interpolation is m, and the rest of the situation is similar and provable. The sample for the ith interpolation is

$$\begin{array}{ccc}\hfill {\mathit{x}}_i^{{}^{\prime }}& =& (x_{i,1}^{{}^{\prime }},x_{i,2}^{{}^{\prime }},\dots ,x_{i,m}^{{}^{\prime }})\hfill \\ & =& (x_i^{(1)}+\frac{x_i^{(2)}-x_i^{(1)}}{m+1},x_i^{(1)}+2·\frac{x_i^{(2)}-x_i^{(1)}}{m+1},\dots ,x_i^{(1)}+m·\frac{x_i^{(2)}-x_i^{(1)}}{m+1})\hfill \\ & =& (x_i^{*(1)}+{ϵ}_i^{(1)}+\frac{x_i^{*(2)}-x_i^{*(1)}+{ϵ}_i^{(2)}-{ϵ}_i^{(1)}}{m+1},\cdots ,x_i^{*(1)}+{ϵ}_i^{(1)}+m·\frac{x_i^{*(2)}-x_i^{*(1)}+{ϵ}_i^{(2)}-{ϵ}_i^{(1)}}{m+1}),\hfill \end{array}$$

where $i=1,\dots ,K$ and the corresponding output is $y_{i,d}^{{}^{\prime }}=y_i^{(1)}+d·\frac{y_i^{(2)}-y_i^{(1)}}{m+1}$. Based on (2), the noise of ${\mathit{x}}_{i,d}^{{}^{\prime }}$ is ${\displaystyle {ϵ}_{i,d}^{{}^{\prime }}={ϵ}_i^{(1)}+d·\frac{{ϵ}_i^{(2)}-{ϵ}_i^{(1)}}{m+1}}$, $d=1,\dots ,m.$

After K interpolations, the uniform noise expectation is

$$\begin{array}{cc}\hfill & E(\frac{{\sum }_{j=1}^2{\sum }_{i=1}^K|{ϵ}_i^{(j)}|+{\sum }_{i=1}^K{\sum }_{d=1}^m|{ϵ}_i^{(1)}+d·\frac{{ϵ}_i^{(2)}-{ϵ}_i^{(1)}}{m+1}|}{2K+K·m})\hfill \\ \hfill & =\frac{{\sum }_{j=1}^2{\sum }_{i=1}^{k}E(|{ϵ}_i^{(j)}|)+{\sum }_{i=1}^K{\sum }_{d=1}^{m}E(|{ϵ}_i^{(1)}+d·\frac{{ϵ}_i^{(2)}-{ϵ}_i^{(1)}}{m+1}|)}{2K+K·m}\hfill \\ \hfill & =\frac{{\sum }_{j=1}^2{\sum }_{i=1}^{K}E(|{ϵ}_i^{(j)}|)+\frac{1}{m+1}{\sum }_{i=1}^K{\sum }_{d=1}^{m}E(|(m-d+1){ϵ}_i^{(1)}+d{ϵ}_i^{(2)}|)}{2K+K·m}\hfill \\ \hfill & =\frac{2+\frac{1}{m+1}{\sum }_{d=1}^m\sqrt{{(m-d+1)}^2+d^2}}{2+m}·\sigma \sqrt{\frac{2}{\pi }}\hfill \\ \hfill & <\frac{2+\frac{1}{m+1}{\sum }_{d=1}^m\sqrt{{(m-d+1)}^2+d^2+2d(m-d+1)}}{2+m}·\sigma \sqrt{\frac{2}{\pi }}\hfill \\ \hfill & =\frac{2+\frac{1}{m+1}{\sum }_{d=1}^m\sqrt{{(m+1)}^2}}{2+m}·\sigma \sqrt{\frac{2}{\pi }}=\sigma \sqrt{\frac{2}{\pi }}.\hfill \end{array}$$

The above shows that the uniform noise of the samples after the optimization through the AMLI method is reduced. The expectation for the proportion of samples with a noise greater than 0.5 is

$$\begin{array}{cc}\hfill & E(\frac{{\sum }_{j=1}^2{\sum }_{i=1}^{K}I(|{ϵ}_i^{(j)}|>0.5)+{\sum }_{i=1}^K{\sum }_{d=1}^{m}I(|{ϵ}_i^{(1)}+d·\frac{{ϵ}_i^{(2)}-{ϵ}_i^{(1)}}{m+1}|>0.5)}{2K+K·m})\hfill \\ \hfill & =\frac{{\displaystyle 4K·\mathsf{\Phi }(\frac{-0.5}{\sigma })+\sum _{i=1}^K\sum _{d=1}^{m}p(|{ϵ}_i^{(1)}+d·\frac{{ϵ}_i^{(2)}-{ϵ}_i^{(1)}}{m+1}|>0.5)}}{2K+K·m}\hfill \\ \hfill & =\frac{{\displaystyle 4K·\mathsf{\Phi }(\frac{-0.5}{\sigma })+\sum _{i=1}^K\sum _{d=1}^{m}p(\frac{|(m-d+1){ϵ}_i^{(1)}+d·{ϵ}_i^{(2)}|}{\sqrt{{(m-d+1)}^2+d^2}·\sigma }>\frac{0.5(m+1)}{\sqrt{{(m-d+1)}^2+d^2}·\sigma })}}{2K+K·m}\hfill \\ \hfill & =\frac{{\displaystyle 4\mathsf{\Phi }(\frac{-0.5}{\sigma })+2\sum _{d=1}^m\mathsf{\Phi }(-\frac{0.5(m+1)}{\sqrt{{(m-d+1)}^2+d^2}·\sigma })}}{2+m}\hfill \\ \hfill & {\displaystyle <\frac{{\displaystyle 4\mathsf{\Phi }(\frac{-0.5}{\sigma })+2\sum _{d=1}^m\mathsf{\Phi }(-\frac{0.5(m+1)}{\sqrt{{(m-d+1)}^2+d^2+2d(m-d+1)}·\sigma })}}{2+m}=2\mathsf{\Phi }(\frac{-0.5}{\sigma }).}\hfill \end{array}$$

Thus, after being processed by the AMLI method, the proportion of samples with an error greater than 0.5 in the data decreases.

6. Extension

6.1. AMLI Plus

The above simulation experiments indicate that the selection of parameter K in the AMLI algorithm is very important. The selection of the optimal value of parameter K involves many parameter adjustment calculations while under the influence of randomness, thereby making it difficult to guarantee that the selected K value results in optimal performance at all times. If the AMLI method fails to achieve a good result, we can consider another interpolation method that combines the clustering method to perform linear interpolation between classes; we name this method AMLI plus. Below, we describe specific steps of the AMLI plus method.

First, all observed samples are clustered according to their distribution. Assuming that the number of clusters is K, and accordingly, the virtual space is divided into K subspaces, each containing all observed samples of the same category:

$$T=\{({\mathit{x}}_1,y_1),({\mathit{x}}_2,y_2),\dots ,({\mathit{x}}_k,y_k)\}=\{G_1,G_2,\dots ,G_K\},$$

where $G_1$ represents the class whose center is closest to the minimum point of the feature space ${\mathit{x}}_0$; the center of the class is ${\displaystyle {\overline{\mathit{x}}}^{(d)}=\frac{1}{n^{(d)}}\sum _{i=1}^{n^{(d)}}{\mathit{x}}_i^{(d)}}$, $d=1,\dots ,K,$$n^{(d)}$$G_d,d=1,\dots ,K;$$n^{(d)}$ is the number of samples in $G_d$; and $G_{d+1}$ satisfies ${\overline{\mathit{x}}}^{(d+1)}=\underset{\{\overline{\mathit{x}}:\overline{\mathit{x}}\ne {\overline{\mathit{x}}}^{(1)},\dots ,{\overline{\mathit{x}}}^{(d)}\}}{argmin}(L({\overline{\mathit{x}}}^{(d)},\overline{\mathit{x}}))$. The choice of clustering method can be diverse; additionally, the K-$means$, the X-$means$ or the $DBSCAN$ method, which can eliminate noise points according to Ester et al. [15], can be selected.

Second, interpolation is performed between classes, in which the unit distance filling parameter $\eta$ is defined; additionally, $n^{(d)}·n^{(d+1)}$ linear interpolations are performed between ${\mathit{x}}_i^{(d)}\in G_d$ and all samples in $G_{d+1}$. The number of interpolation samples is ${\sum }_{j=1}^{n^{(d+1)}}{\sum }_{i=1}^{n^{(d)}}\eta ·L({\mathit{x}}_i^{(d)},{\mathit{x}}_j^{(d+1)}),$ and the interpolated samples are also equally spaced.

The specific implementation process of the AMLI method is detailed in Algorithm 2.   

Algorithm 2: AMLI plus

6.2. The Proof of AMLI Plus

In this section, we will present evidence of the effectiveness of the AMLI plus method. The proof idea is essentially the same as that of the AMLI method, and we will focus on the differences between the two.

Unlike the AMLI method, the AMLI plus method divides the virtual space into K subspaces according to the number of clusters, and each subspace contains all observation samples of the same category; thus, the sample size of each subspace may vary. It is assumed that two adjacent subspaces, namely, ${\mathcal{X}}^{(1)},{\mathcal{X}}^{(2)}$, contain $n^{(1)},$$n^{(2)}$ samples, respectively; additionally, it is assumed that $n^{(1)}·n^{(2)}$ linear interpolations are performed. For the sake of simplicity, assuming that the number of samples interpolated each time is m, then the uniform noise after the interpolation is

$$\begin{array}{cc}\hfill & E(\frac{{\sum }_{i=1}^{n^{(1)}}|{ϵ}_i^{(1)}|+{\sum }_{i=1}^{n^{(2)}}\left|{ϵ}_i^{(2)}\right|+{\sum }_{i=1}^{n^{(1)}}{\sum }_{j=1}^{n^{(2)}}{\sum }_{d=1}^m\left|{ϵ}_i^{(1)}+d·\frac{{ϵ}_j^{(2)}-{ϵ}_i^{(1)}}{m+1}\right|}{n^{(1)}+n^{(2)}+n^{(1)}{n}^{(2)}m})\hfill \\ \hfill & =\frac{{\sum }_{i=1}^{n^{(1)}}E(|{ϵ}_i^{(1)}|)+{\sum }_{i=1}^{n^{(2)}}E(|{ϵ}_i^{(2)}|)+{\sum }_{i=1}^{n^{(1)}}{\sum }_{j=1}^{n^{(2)}}{\sum }_{d=1}^{m}E(\left|{ϵ}_i^{(1)}+d·\frac{{ϵ}_j^{(2)}-{ϵ}_i^{(1)}}{m+1}\right|)}{n^{(1)}+n^{(2)}+n^{(1)}{n}^{(2)}m}\hfill \\ \hfill & =\frac{n^{(1)}\sigma \sqrt{\frac{2}{n}}+n^{(2)}\sigma \sqrt{\frac{2}{n}}+\frac{1}{m+1}{\sum }_{i=1}^{n^{(1)}}{\sum }_{j=1}^{n^{(2)}}{\sum }_{d=1}^{m}E(\left|(m-d+1){ϵ}_i^{(1)}+d{ϵ}_i^{(2)}\right|)}{n^{(1)}+n^{(2)}+n^{(1)}{n}^{(2)}m}\hfill \\ \hfill & =\frac{n^{(1)}+n^{(2)}+\frac{n^{(1)}{n}^{(2)}}{m+1}{\sum }_{d=1}^m\sqrt{{(m-d+1)}^2+d^2}}{n^{(1)}+n^{(2)}+n^{(1)}{n}^{(2)}m}·\sigma \sqrt{\frac{2}{\pi }}\hfill \\ \hfill & <\frac{n^{(1)}+n^{(2)}+\frac{n^{(1)}{n}^{(2)}}{m+1}{\sum }_{d=1}^m\sqrt{{(m-d+1)}^2+d^2+2d(m-d+1)}}{n^{(1)}+n^{(2)}+n^{(1)}{n}^{(2)}m}·\sigma \sqrt{\frac{2}{\pi }}\hfill \\ \hfill & =\frac{n^{(1)}+n^{(2)}+\frac{n^{(1)}{n}^{(2)}}{m+1}{\sum }_{d=1}^{m}m+1}{n^{(1)}+n^{(2)}+n^{(1)}{n}^{(2)}m}·\sigma \sqrt{\frac{2}{\pi }}=\sigma \sqrt{\frac{2}{\pi }}.\hfill \end{array}$$

Thus, the uniform noise after optimization by the AMLI plus method is reduced. The proportion of samples with a noise error greater than 0.5 is expected to be

$$\begin{array}{cc}\hfill & E(\frac{{\sum }_{i=1}^{n^{(1)}}I(|{ϵ}_i^{(1)}|>0.5)+{\sum }_{i=1}^{n^{(2)}}I(|{ϵ}_i^{(2)}|>0.5)+{\sum }_{i=1}^{n^{(1)}}{\sum }_{j=1}^{n^{(2)}}{\sum }_{d=1}^{m}I(\left|{ϵ}_i^{(1)}+d·\frac{{ϵ}_j^{(2)}-{ϵ}_i^{(1)}}{m+1}\right|>0.5)}{n^{(1)}+n^{(2)}+n^{(1)}{n}^{(2)}m})\hfill \\ \hfill & =\frac{{\sum }_{i=1}^{n^{(1)}}P(|{ϵ}_i^{(1)}|>0.5)+{\sum }_{i=1}^{n^{(2)}}P(|{ϵ}_i^{(2)}|>0.5)+{\sum }_{i=1}^{n^{(1)}}{\sum }_{j=1}^{n^{(2)}}{\sum }_{d=1}^{m}P(\left|{ϵ}_i^{(1)}+d·\frac{{ϵ}_j^{(2)}-{ϵ}_i^{(1)}}{m+1}\right|>0.5)}{n^{(1)}+n^{(2)}+n^{(1)}{n}^{(2)}m}\hfill \\ \hfill & =\frac{{\displaystyle 2(n^{(1)}+n^{(2)})\mathsf{\Phi }\left({\displaystyle \frac{-0.5}{\sigma }}\right)+{\sum }_{i=1}^{n^{(1)}}{\sum }_{j=1}^{n^{(2)}}{\sum }_{d=1}^{m}P(\frac{|(m-d+1){ϵ}_i^{(1)}+d·{ϵ}_j^{(2)}|}{\sqrt{{(m-d+1)}^2+d^2}·\sigma }>\frac{0.5(m+1)}{\sqrt{{(m-d+1)}^2+d^2}·\sigma })}}{n^{(1)}+n^{(2)}+n^{(1)}{n}^{(2)}m}\hfill \\ \hfill & =\frac{2(n^{(1)}+n^{(2)})\mathsf{\Phi }\left({\displaystyle \frac{-0.5}{\sigma }}\right)+2n^{(1)}{n}^{(2)}{\sum }_{d=1}^m\mathsf{\Phi }\left({\displaystyle -\frac{0.5(m+1)}{\sqrt{{(m-d+1)}^2+d^2·\sigma }}}\right)}{n^{(1)}+n^{(2)}+n^{(1)}{n}^{(2)}m}\hfill \\ \hfill & <\frac{2(n^{(1)}+n^{(2)})\mathsf{\Phi }\left({\displaystyle \frac{-0.5}{\sigma }}\right)+2n^{(1)}{n}^{(2)}{\sum }_{d=1}^m\mathsf{\Phi }\left({\displaystyle -\frac{0.5(m+1)}{\sqrt{{(m-d+1)}^2+d^2+2d(m-d+1)}·\sigma }}\right)}{n^{(1)}+n^{(2)}+n^{(1)}{n}^{(2)}m}=2\mathsf{\Phi }\left(\frac{-0.5}{\sigma }\right).\hfill \end{array}$$

Therefore, after the samples are processed by the AMLI plus method, the proportion of samples with an error greater than 0.5 in the data decreases.

Remark 2.

The proof rationale of the AMLI plus method is to divide the clustered samples of different categories into different subspaces and to assume that neighboring subspaces have a common linear relationship; therefore, when the number of clusters and the clustering method are selected, the above requirement should be satisfied as much as possible.

7. Conclusions

In this study, we propose a multipath sample interpolation method based on the idea of linear interpolation, which can solve the problem of insufficient sample sizes or large errors between observed samples and actual distribution when making predictions. The AMLI method, simple to implement and flexible, can greatly expand valid samples, reduce the influence of sample noise, and thus significantly improve the prediction effect. Finally, we propose the AMLI plus method, another class-to-class-based linear interpolation method, which can also achieve good optimization results. In general, we find that the AMLI method is robust, effective, and very suitable for addressing a series of problems in machine learning, e.g., insufficient sample sizes and large amounts of observation noise.