Parameter Estimation of Noise-Disturbed Multivariate Systems Using Support Vector Regression Integrated with Random Search and Bayesian Optimization

Abstract

To achieve accurate control of Multi-Input and Multi-Output (MIMO) physical plants, it is crucial to obtain correct model expressions. In practice, the prevalence of both outliers and colored noise can cause serious interference with the industrial process, thus reducing the accuracy of the identification algorithm. The algorithm of support vector regression (SVR) is proposed to address the problem of parameter estimation for MIMO systems under interference from outliers and colored noise. In order to further improve the speed of parameter estimation, random search and Bayesian optimization algorithms were introduced, and the support vector regression combining stochastic search and Bayesian optimization (RSBO-SVR) algorithm was proposed. It was verified by simulation and tank experiments. The results showed that the method has strong anti-interference ability and can achieve high-precision parameter identification. The maximum relative error of the RSBO-SVR algorithm did not exceed 4% in both the simulation and experiment. It had a maximum reduction of 99.38% in runtime compared to SVR.

1. Introduction

It is known that multivariable systems are referred to as Multi-Input and Multi-Output (MIMO) systems. In the field of identification, methods for the identification of univariate systems are becoming mature, such as the least squares (LS) method [1,2], the stochastic gradient method [3,4], the maximum-likelihood method [5,6], and so on. With the development of control theory and the needs of engineering practice, most industrial plants are expressed by multivariable systems with complex structures, the existence of nonlinearities, incomplete information, and uncertain disturbances [7,8,9,10]. Therefore, the study of multivariate systems is more valuable than univariate systems [11,12].

There are two main models for describing MIMO systems: one is the state space equation (SSE) model, and the other is the transfer function matrix (TFM) model [13,14]. Over the years, scholars have proposed different parameter identification methods for these two models.

When the SSE model has a measurable state, the controlled model can be converted into a regression model, and the conventional identification algorithm can be used to identify the regression model; e.g., the iterative identification algorithm derived in the literature [15] for the SSE model with unknown parameters and a measurable state can effectively identify the parameters of the system. Although the ultimate goal of the literature [16] is to identify the TFM model, it first identifies the SSE model and then explores the relationship between the SSE model and the TFM model. The paper first utilized the subspace method to estimate the state space matrix (A, B, C, D). Then, the covariance expression of the input correlation matrix (B, D) was derived based on the first-order perturbation method, and the covariance of the transfer function was calculated by combining the covariance of (B, D) with the covariance of (A, C). Subspace identification methods have developed particularly rapidly in recent decades, and when the state of the SSE model is not measurable, the method identifies the linear time-invariant state-space model directly from the input and output data [17]. A novel system identification method based on invariant subspace theory has been proposed in the literature [18], aiming at solving the problem of identifying continuous-time linear time-invariant systems by combining time-domain and frequency-domain methods.

When the TFM is utilized to describe the system model for parameter identification, the MIMO system is always decoupled by a decoupling method before identification [19,20,21,22]. Liu derived a coupled stochastic gradient algorithm for multivariate systems, which decomposes the system model into multiple single-output subsystems for parameter identification. Although this method has high accuracy, it does not match the realistic industrial production model [19]. Ref. [23] presented a dynamic model of a plate-fin heat exchanger, from the heat exchanger mechanism model, using the Laplace transform to derive the calculation formula of the time constant, and put forward the identification method based on the heat exchanger efficiency of the heat transfer thermal resistance calculation relationship. Ref. [24] avoided the calculation of matrix inversion by decomposing the multivariate class CAR system into two subsystems and deriving a joint identification algorithm of stochastic gradient and least squares to estimate the system parameters.

Scholars who have previously proposed traditional identification methods consider only empirical risks, and these methods are prone to overfitting problems. In 1995, Vapnik et al. proposed a support vector machine (SVM) based on the Vapnik–Chervonenkis (VC) theory of statistical learning theory and structural risk minimization theory [25]. SVM has a strict theoretical and mathematical foundation, does not easily fall into the problems of local optimum and dimension difficulty, has a weak dependence on the number of samples, and has a strong generalization ability [26].

SVM was first solely used for classification problems [27,28,29,30]; however, as research into the topic has progressed, the SVM algorithm has been refined and enhanced, and its use has been extended to regression problems.

Huang and colleagues used SVM and the least square support vector machine (LS-SVM) to identify the difference equation models of linear and nonlinear systems [31]. The findings indicate that the SVM and LS-SVM techniques outperform neural networks in terms of generalization capacity and system identification accuracy. The LS-SVM algorithm is better suited for dynamic system identification since it is quicker and more resilient to noise than SVM. Castro-Garcia R. et al. presented a technique that uses the LS-SVM’s regularization capability to identify MIMO Hammerstein systems. The identification process only necessitates a few basic presumptions and performs well in the presence of Gaussian white noise [32].

Systems with colored noise have more challenges in parameter identification than systems with white noise disturbances [33,34,35]. Systems with colored noise, however, are more in accordance with real-world production processes [36,37].

Zhao divided the system disturbed by colored noise into a system part and a noise part and estimated the parameters of the two parts separately [38]. However, specific noise levels cannot be observed in real industrial activities and the method does not apply to on-site production activities. Xu and others used the filtering idea to convert the colored noise system into two models containing white noise [39]. This method directly changes the system structure and also generates new noise during the conversion process. Zhang et al. considered the problem of parameter identification in the presence of additive noise and used the a priori knowledge of the instantaneous mixing matrix in the independent component analysis (ICA) model to accurately estimate the additive noise at the output of the system [40].

Real-time parameter estimation is of great significance in modern engineering and control systems, which not only improves the performance and robustness of the system, but also supports fault detection, optimizes the production process, reduces the maintenance cost, and enhances the system safety. In order to realize real-time parameter estimation, the time for each parameter estimation process should not be too long. To achieve the reduction of estimation time, parameter optimization can be used. Parameter optimization has a very wide range of applications in many fields. These include hyperparameter optimization [41], process optimization [42,43], and so on.

The parameter estimation problem for multivariate systems under the interference of outliers and colored noise is considered in this paper. The main contributions are summarized as follows.

(1) To solve the problem of estimating the parameters of the TFM, the parameters of the equivalent difference equation model were first estimated, and then the TFM parameters were derived from the equation;

(2) In this paper, a method was proposed to deal with the parameter identification problem effectively using support vector regression (SVR). The method can well solve the problem of parameter estimation under the interference of outliers and colored noise;

(3) To address the issue of long parameter search time in the abovementioned method, a new SVR method combining stochastic search and Bayesian optimization was proposed;

(4) A two-input single-output simulation system was used to verify the effectiveness and accuracy of the algorithm. In addition, a water tank system was constructed for system identification to verify the effectiveness of the algorithm in real systems.

The remainder of this article is organized as follows. Section 2 introduces the related knowledge of SVR, analyzes the anti-interference property of SVR, and introduces the parameter estimation algorithm based on SVR in detail. Section 3 introduces the SVR algorithm combining random search and Bayesian optimization. Section 4 verifies the effectiveness of the algorithm through a dual-input, single-output simulation system, and verifies the usability of the algorithm in real systems through data from actual water tank systems. Section 5 contains the concluding remarks.

2. Parameter Estimation for MIMO Systems Under Noise Interference

2.1. Support Vector Regression Algorithm

There are m sets of input and output data samples $\left\{(X_i,Y_i),i=1,2,\cdots ,m\right\}$, where $X_i\in ℝ$ is the input data set matrix of the ith data sample, which is expressed as $\left[\begin{array}{cccc}{x}_i^{11}& x_i^{12}& \cdots & x_i^{1n}\\ x_i^{21}& x_i^{22}& \cdots & x_i^{2n}\\ ⋮& ⋮& \cdots & ⋮\\ x_i^{r1}& x_i^{r2}& ⋮& x_i^{rn}\end{array}\right]$. $Y_i\in R^r$ is the corresponding output data set matrix.

The established linear regression function is shown in Equation (1).

$$f(X)=〈w,X_i〉+b,w\in \aleph ,b\in ℝ$$

(1)

where $〈\cdot ,\cdot 〉$ denotes the dot product in $\aleph$.

We construct an SVR regression problem for each output $Y_i$.

The SVR algorithm is obtained by introducing an insensitive loss function on the basis of SVM classification. The basic idea is to find an optimal plane that minimizes the error of all training samples away from that optimal plane. Currently, the most widely used SVR model is $\epsilon$-SVR, which introduces a $\epsilon$ linear insensitive loss function (as shown in Equation (2)).

$$L(f(X),Y_i,\epsilon )=\left\{\begin{array}{cc}0,& \left|Y_i-f(X)\right|\le \epsilon \\ \left|f(X)-Y_i\right|-\epsilon ,& \left|Y_i-f(X)\right|>\epsilon \end{array}\right.$$

(2)

The goal of $\epsilon$-SVR is to find a function $f(x)$ that deviates from the actual obtained target $Y_i$ by at most $\epsilon$ for all training data while being as flat as possible [44]. To satisfy the flatness of Equation (1), it can be achieved by minimizing the norm, i.e., ${‖w‖}^2$ [45]. This is shown in Equation (3). The principle of SVR is shown in Figure 1.

$$\left\{\begin{array}{c}\min {\displaystyle \frac{1}{2}}{‖w‖}^2\\ s.t.\left|y_i-〈w,X_i〉+b\right|\le \epsilon ,\forall i\end{array}\right.$$

(3)

The biggest difference between SVR and traditional regression algorithms is that SVR gives some tolerance to fitting.

In practice, setting $\epsilon$ too small cannot guarantee that all sample points are within the $\epsilon$-pipeline, and setting $\epsilon$ too large will cause the regression hyperplane to be affected by outliers. Relaxation variables ${\xi }_i$ and ${\xi }_i^{*}$ are introduced to describe the degree of deviation of the sample points from the $\epsilon$-pipeline so that both empirical and structural risks are taken into account.

The objective function is updated as Equation (4).

$$\left\{\begin{array}{c}\underset{w,b,\xi ,{\xi }^{*}}{\min }{\displaystyle \frac{1}{2}}{‖w‖}^2+C{\displaystyle \sum _{i=1}^m({\xi }_i+{\xi }_i^{*})}\\ s.t.〈w,X_i〉+b-Y_i\le \epsilon +{\xi }_i,i=1,2,\cdots ,m\\ Y_i-〈w,X_i〉-b\le \epsilon +{\xi }_i^{*},i=1,2,\cdots ,m\\ {\xi }_i,{\xi }_i^{*}\ge 0,i=1,2,\cdots ,m\end{array}\right.$$

(4)

where C is the penalty factor, a human-set parameter.

Based on the constraints, Lagrange multipliers $\alpha , {\alpha }^{*}, \beta , {\beta }^{*}$ are introduced, and constructing the Lagrange function (as in Equation (5)) transforms the optimization problem into a dyadic problem.

$$L={\displaystyle \frac{1}{2}}{‖w‖}^2+C{\displaystyle \sum _{i=1}^m({\xi }_i+{\xi }_i^{*})-{\displaystyle \sum _{i=1}^m{\alpha }_i[\epsilon +}}{\xi }_i-Y_i+f(X_i)]-{\displaystyle \sum _{i=1}^m{\alpha }_i^{*}[\epsilon +{\xi }_i^{*}+Y_i-f(X_i)]-{\displaystyle \sum _{i=1}^m({\beta }_i{\xi }_i+{\beta }_i^{*}{\xi }_i^{*})}}$$

(5)

The above-shown equation is obtained by taking the derivations of $\omega , b, {\xi }_i, {\xi }_i^{*}$, respectively, and making the result zero:

$$\left\{\begin{array}{c}\omega ={\displaystyle \sum _{i=1}^m({\alpha }_i-{\alpha }_i^{*})X_i}\\ {\displaystyle \sum _{i=1}^m({\alpha }_i-{\alpha }_i^{*})=0}\\ C-{\alpha }_i-{\xi }_i=0\\ C-{\alpha }_i^{*}-{\xi }_i^{*}=0\end{array}\right.$$

(6)

The dyadic problem of the original question is

$$\left\{\begin{array}{c}\underset{\alpha ,{\alpha }^{*}}{\max }[-{\displaystyle \frac{1}{2}}{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^m({\alpha }_i-{\alpha }_i^{*})({\alpha }_j-{\alpha }_j^{*})〈X_i,X_j〉}}-{\displaystyle \sum _{i=1}^m({\alpha }_i+{\alpha }_i^{*})\epsilon +{\displaystyle \sum _{i=1}^m({\alpha }_i-{\alpha }_i^{*})Y_i}}]\\ s.t.{\displaystyle \sum _{i=1}^m({\alpha }_i-{\alpha }_i^{*})=0}\\ 0\le {\alpha }_i\le C\\ 0\le {\alpha }_i^{*}\le C\end{array}\right.$$

(7)

A kernel function is a very important concept in SVR algorithms, which can map the input feature vectors to a high-dimensional space for computation. Commonly used kernel functions include linear kernel function, polynomial kernel function, radial basis kernel function (RBF), and sigmoid kernel function. Different kernel functions correspond to different problems and are applicable to different problems, and choosing the appropriate kernel function can improve the performance of SVR. The specific calculation formula is shown in

Table 1. The kernel functions and their calculation formulas.

Kernel Function	Calculation Formula
Linear kernel function	$K(x_i,x_j)=x_i^T{x}_j$
Polynomial kernel function	$K(x_i,x_j)={(\gamma x_i^T{x}_j+b)}^d$
RBF	$K(x_i,x_j)=\exp \left(-\gamma {‖x_i-x_j‖}^2\right)$
Sigmoid kernel function	$K(x_i,x_j)=\tanh (\gamma x_i^T{x}_j+b)$

.

Support vector machines that introduce kernel functions and neural networks have a similar structure. Since we are studying a linear MIMO system, we use a linear kernel function.

Solving Equation (5) yields the optimal solution $\alpha , {\alpha }^{*}$, and the regression function is shown in Equation (8).

$$f(X)={\displaystyle \sum _{i=1}^{\rho }({\alpha }_i-{\alpha }_i^{*})}〈X_i,X〉+b$$

(8)

where $\rho$ represents the number of support vectors.

Currently, there are many toolkits about SVM, such as LIBLINEAR, mySVM, LIBSVM, and so on. Among them, LIBSVM (Version 3.35) is an open-source software package for SVM developed by Prof. Chi-Jen Lin’s team at National Taiwan University, which provides an interface with MATLAB and is easy to use. In this paper, this package was used for data training and prediction.

2.2. Anti-Interference Analysis of SVR

Observations or values that deviate significantly from other data points in the data collection are called outliers. Numerous factors such as measurement errors, data entry problems, natural variability, and atypical events can lead to outliers.

Colored noise is a noise signal with an uneven frequency–energy distribution. In contrast to Gaussian white noise, colored noise does not have a uniform energy distribution over the frequency spectrum but has a frequency dependence or bias.

White noise and outliers are independent at different points in time, whereas colored noise is time-dependent. In addition, colored noise signals are often mixed with useful signals and are difficult to distinguish.

Due to the principle and structural properties of SVR, high interference resistance of parameter identification using SVR is guaranteed.

(1) Support Vector Regression employs the principle of structural risk minimization, which aims to minimize the empirical error and the confidence range, i.e., to minimize both the training error and the model complexity. In this way, SVR is able to find a balance between reducing model complexity and improving fitting accuracy, thus enhancing the generalization ability of the model.

Specifically, the goal of SVR is to optimize the model by minimizing structural risk, which involves not only minimizing training error but also ensuring that the model complexity is manageable. The L2 regularization term plays a key role here. The L2 regularization ensures that the model parameters do not become too large by restricting the L2 norms of the model parameter w. This helps to control the model complexity and to improve the generalization ability of the model. This helps control model complexity and prevents the model from overfitting the noise in the training data. L2 regularization reduces the model’s sensitivity to the input features by compressing the parameter values, thus improving the model’s generalization ability. Noise is usually random and irregular, and if the model is too complex, it may capture these noise signals, leading to overfitting. L2 regularization, on the other hand, makes the model smoother and avoids overreacting to chance noise patterns in the data;

(2) The generalization performance of SVR depends heavily on the appropriate choice of parameter C, which controls the weight of the empirical risk in the structural risk. To find the optimal value of C, a grid search method can be used. The specific size of the grid search will be elaborated in Section 2.3;

(3) SVR improves the robustness and immunity of the model by maximizing the distance between the support vector and the regression hyperplane. Maximizing this spacing means that the model pays more attention to trends in the overall data distribution, rather than being overly sensitive to small changes (i.e., noise) in individual sample points. This strategy allows SVR to better handle training data with noise and avoid overfitting the model by fitting noise;

(4) Using an $\epsilon -$ insensitive loss function allows a width interval to exist between the regression predictions and the true values, and the loss function does not take into account the prediction error within that interval. This means that the model is insensitive to small noises, as small noises do not lead to additional penalties.

2.3. Parameter Estimation of SVR-Based for MIMO Systems

A linear multivariate system is shown in Figure 2. The state space equation description of a system is not unique and when the state of the state space equation model is not measurable, not only the parameters of the model but also the state of the model has to be estimated, which makes the identification more complicated, and it is more advantageous to use the transfer function matrix model to describe the identification system model.

The TFM model of the MIMO system is described as shown in Equation (9).

$$Y(s)=\left[\begin{array}{ccc}{\displaystyle \frac{K_{11}}{T_{11}s+1}}& \cdots & {\displaystyle \frac{K_{1r}}{T_{1r}s+1}}\\ ⋮& \ddots & ⋮\\ {\displaystyle \frac{K_{n1}}{T_{n1}s+1}}& \cdots & {\displaystyle \frac{K_{nr}}{T_{nr}s+1}}\end{array}\right]U(s)$$

(9)

The parameter set K described in Equation (10) and the parameter set T described in Equation (11) are the sets of parameters that the system needs to recognize.

$$K=\left[\begin{array}{cccccc}{K}_{11}& K_{12}& \cdots & K_{ij}& \cdots & K_{nr}\end{array}\right]$$

(10)

$$T=\left[\begin{array}{cccccc}{T}_{11}& T_{12}& \cdots & T_{ij}& \cdots & T_{nr}\end{array}\right]$$

(11)

It is first necessary to discretize the discrete model (9) of the multivariate system into an impulse TFM model, as shown in Equation (12). In this paper, the discretization was carried out using the bilinear transformation method ($s={\displaystyle \frac{2}{T_s}}{\displaystyle \frac{1-z^{-1}}{1+z^{-1}}}$).

$$Y(z)=\left(\begin{array}{ccc}{G}_{11}(z^{-1})& \dots & G_{1r}(z^{-1})\\ ⋮& \ddots & ⋮\\ G_{n1}(z^{-1})& \cdots & G_{nr}(z^{-1})\end{array}\right)U(z)$$

(12)

Regarding the above-shown equation, the specific description of $G_{ij}(z^{-1})$ is shown in Equation (13).

$$G_{ij}(z^{-1})={\displaystyle \frac{n_{ij,0}+n_{ij,1}{z}^{-1}+\cdots +n_{ij,q_{b_{ij}}}{z}^{-q_{b_{ij}}}}{1+m_{ij,1}{z}^{-1}+\cdots +m_{ij,p_{a_{ij}}}{z}^{-p_{a_{ij}}}}}$$

(13)

where $q_{b_{ij}}$ is the input signal order and $p_{a_{ij}}$ is the output signal order.

Extracting the least common multiple of each row of the impulse TFM leads to Equation (14) and then converting the impulse TFM model to a difference equation model (Equation (15)).

$$1+{\mu }_{i,1}{z}^{-1}+\cdots +{\mu }_{i,n_{a_i}}{z}^{-n_{a_i}}=(1+\cdots +m_{i1,p_{a_{i1},i1}}{z}^{-p_{a_{i1},i1}})\cdots (1+\cdots +m_{in,p_{a_{in},in}}{z}^{-p_{a_{in},in}})$$

(14)

$$\left[\begin{array}{c}{y}_1(k)\\ ⋮\\ y_n(k)\end{array}\right]=-\left[\begin{array}{c}{Y}_1(k)\\ ⋮\\ Y_n(k)\end{array}\right]+\left[\begin{array}{c}{U}_1(k)\\ ⋮\\ U_n(k)\end{array}\right]$$

(15)

where $Y_i(k)={\mu }_{i,1}{y}_i(k-1)+\cdots +{\mu }_{i,n_{a_i}}{y}_i(k-n_{a_i})$, $U_i={\lambda }_{i1,0}{u}_1(k)+\cdots +{\lambda }_{i1,q_{b_{i1}}}$ $u_1(k-q_{b_{i1}})+\cdots +{\lambda }_{ir,0}{u}_r(k)+\cdots +{\lambda }_{ir,q_{b_{ir}}}{u}_r(k-q_{b_{ir}})$.

The difference equation model parameters (Equation (16)) are first estimated and then the parameters K and T are inverted according to the discretization method used.

$${\theta }_1=\left[\begin{array}{ccc}{\mu }_{1,1}& \cdots & {\mu }_{1,n_{a1}}\\ {\mu }_{2,1}& \cdots & {\mu }_{2,n_{a2}}\\ ⋮& ⋮& ⋮\\ {\mu }_{n,1}& \cdots & {\mu }_{n,n_{an}}\end{array}\right.\left.\begin{array}{ccccccc}{\lambda }_{11,0}& \cdots & {\lambda }_{11,q_{b_{11}}}& \cdots & {\lambda }_{1r,0}& \cdots & {\lambda }_{1r,q_{b_{1r}}}\\ {\lambda }_{21,0}& \cdots & {\lambda }_{21,q_{b_{21}}}& \cdots & {\lambda }_{2r,0}& \cdots & {\lambda }_{2r,q_{b_{2r}}}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ {\lambda }_{n1,0}& \cdots & {\lambda }_{n1,q_{b_{n1}}}& \cdots & {\lambda }_{nr,0}& \cdots & {\lambda }_{nr,q_{b_{nr}}}\end{array}\right]$$

(16)

Specifically, after determining the differential equation model, K and T are solved inversely according to the formulation of the bilinear transformation method. This can be achieved by using the d2c function in MATLAB.

For the MIMO system, we split the system into multiple Multi-Input and Single-Output (MISO) systems based on each output $y_i$ and study each MISO system.

Take the example of a two-input, two-output system (Equation (17)). It is discretized into a difference equation, which is described as Equation (18).

$$Y(s)=\left[\begin{array}{cc}{\displaystyle \frac{K_1}{T_{1}s+1}}& {\displaystyle \frac{K_2}{T_{2}s+1}}\\ {\displaystyle \frac{K_3}{T_{3}s+1}}& {\displaystyle \frac{K_4}{T_{4}s+1}}\end{array}\right]U(s)$$

(17)

$$\left[\begin{array}{c}{y}_1(k)\\ y_2(k)\end{array}\right]=-\left[\begin{array}{cc}{\mu }_{11}{y}_1(k-1)& {\mu }_{12}{y}_1(k-2)\\ {\mu }_{21}{y}_2(k-1)& {\mu }_{22}{y}_2(k-2)\end{array}\right]+\left[\begin{array}{cccccc}{\lambda }_{11}{u}_1(k)& {\lambda }_{12}{u}_1(k)& {\lambda }_{13}{u}_1(k)& {\lambda }_{21}{u}_2(k)& {\lambda }_{22}{u}_2(k)& {\lambda }_{23}{u}_2(k)\\ {\lambda }_{31}{u}_1(k)& {\lambda }_{32}{u}_1(k)& {\lambda }_{33}{u}_1(k)& {\lambda }_{41}{u}_2(k)& {\lambda }_{42}{u}_2(k)& {\lambda }_{43}{u}_2(k)\end{array}\right]$$

(18)

The inverse solution of $K_1, K_2, K_3, K_4, T_1, T_2, T_3, T_4$ is performed according to the formulation of the bilinear transformation method, the expression of which is shown in Equation (19).

$$\left\{\begin{array}{c}{K}_1={\displaystyle \frac{2{\lambda }_{11}{\lambda }_{21}}{{\lambda }_{22}}}\\ K_2={\displaystyle \frac{2{\lambda }_{11}{\lambda }_{21}}{{\lambda }_{12}}}\\ K_3={\displaystyle \frac{2{\lambda }_{31}{\lambda }_{41}}{{\lambda }_{42}}}\\ K_4={\displaystyle \frac{2{\lambda }_{31}{\lambda }_{41}}{{\lambda }_{32}}}\end{array}\right.\hspace{1em}\hspace{1em}\left\{\begin{array}{c}{T}_1={\displaystyle \frac{2T_s{\lambda }_{21}-T_s{\lambda }_{22}}{2{\lambda }_{22}}}\\ T_2={\displaystyle \frac{2T_s{\lambda }_{11}-T_s{\lambda }_{12}}{2{\lambda }_{12}}}\\ T_3={\displaystyle \frac{2T_s{\lambda }_{41}-T_s{\lambda }_{42}}{2{\lambda }_{42}}}\\ T_4={\displaystyle \frac{2T_s{\lambda }_{31}-T_s{\lambda }_{32}}{2{\lambda }_{32}}}\end{array}\right.$$

(19)

where $T_s$ is the system sampling time.

Parameter estimation for MIMO systems can be viewed as finding the relationship between the input vector X and multiple outputs $y_j$. For each output $y_j$, an SVR model is built to describe its relationship with the input vector X.

$$y_j=f_j(X)+{\mu }_j$$

(20)

For each output $y_j$, we define an independent SVR optimization problem with the following objective function.

$$\left\{\begin{array}{c}\underset{w,b,\xi ,{\xi }^{*}}{\min }{\displaystyle \frac{1}{2}}{‖w_j‖}^2+C{\displaystyle \sum _{i=1}^m({\xi }_i^j+{\xi }_i^{*j})}\\ s.t.〈w_j,X_i〉+{\mu }_j-y_j\le \epsilon +{\xi }_i^j\\ y_j-〈w_j,X_i〉-{\mu }_j\le \epsilon +{\xi }_i^{*j}\\ {\xi }_i^j,{\xi }_i^{*j}\ge 0\end{array}\right.$$

(21)

where $w_j$ is the weight vector of the corresponding SVR regression problem for each output signal $y_j$.

By training the dataset $\left\{\left(X_i,Y_i\right)\right\}$, the output of each SVR model is trained individually.

After collecting the input and output signals, the dataset is constructed according to the data requirements of LIBSVM, including the label dataset and data dataset; for the regression problem, the label dataset is the independent variable dataset, and the data dataset is the dependent variable dataset. Parameter identification of multivariate systems based on SVR requires the construction of datasets as shown in Equations (22) and (23).

$$label=\left[\begin{array}{cc}-y_i(k-1)& -y_i(k-2)\end{array}\right.\begin{array}{ccc}{u}_1(k)& u_1(k-1)& u_1(k-2)\end{array}\left.\begin{array}{ccc}{u}_2(k)& u_2(k-1)& u_2(k-2)\end{array}\right]$$

(22)

$$data=[y_i(k)]$$

(23)

The dataset was divided into training and test sets according to a certain ratio. The linear kernel function was selected based on the characteristics of the linear multivariate system. We then optimized the parameter C using the grid search method to achieve the best prediction result.

When the kernel function is a linear kernel function, the training system parameters, support vector weights, and support vectors can be used to find the discriminant parameters, as shown in Equation (24).

$$w=\left((model.sv\_coef)’•full\left(model.SVs\right)\right)$$

(24)

The specific process of SVR-based parameter identification is shown in Figure 3.

2.4. Unbiased and Error Convergence

For traditional estimation algorithms, such as the LS method, the most important properties of the method are unbiasedness and convergence of the error variance to zero.

SVR-based parameter estimation methods do not directly optimize for or guarantee unbiasedness and are more concerned with finding a model that strikes a balance between model complexity and prediction error. This bias facilitates the reduction of overfitting and improves the generalization ability of the model. Therefore, this paper considers the problem of parameter estimation under noise interference, and better results can be achieved by selecting SVR-based parameter estimation algorithms compared to traditional parameter estimation methods.

Whether the error variance of SVR-based parameter estimation methods can converge to zero depends on a number of factors, including the choice of model, the complexity of the feature space, the nature of the dataset, and the setting of the regularization parameters. Since the goal of SVR is to minimize the model complexity and prediction error, it is not possible to achieve complete convergence of the error variance to 0.

Although the complexity of the feature equations is performed to improve the accuracy of the parameter estimation, the model may be overfitted in this case. For the data studied in this paper, i.e., data containing noise, overfitting can lead to a parameter estimation problem that is heavily influenced by noise.

3. SVR Algorithm Using Stochastic Search and Bayesian Optimization

Stochastic search is a global search method that does not rely on gradient information, and it searches for optimal solutions by randomly sampling in the solution space. Unlike deterministic search algorithms, stochastic search does not rely on a priori knowledge and is insensitive to the choice of initial points, and thus exhibits unique robustness in dealing with discontinuous and nonconvex optimization problems. The core of the stochastic search algorithm is to utilize randomness to traverse the solution space, and it does not use any directed search strategy, which means that the decision at each step of the algorithm is independent of the previous step.

Its main process is as follows:

(1) Define the parameter space. Define the parameter space to be optimized according to the specific problem;

(2) Initial sampling. Randomly sample a set of initial parameter values from the parameter space according to a certain sampling method, such as uniform distribution, Gaussian distribution, and so on;

(3) Evaluation of objective function. Design the objective function according to the specific problem, and then calculate the corresponding objective function value for each set of parameter values;

(4) Update the best parameters. Record the best combination of parameters found and their corresponding objective function values;

(5) Iterative search. Repeat steps 2 to 4 until a predetermined number of iterations is reached or the stopping condition is met.

Compared with grid search, random search does not need to search for all parameter combinations, which greatly reduces the number of searches and time. Especially when the parameter space is very large, stochastic search can find the near-optimal parameter combinations with less computational resources and time cost.

Bayesian optimization is a global optimization method based on a probabilistic model, which searches for the optimal hyperparameters by constructing a probabilistic model of the objective function.

The core of Bayesian optimization lies in the use of a probabilistic model to model a black-box function and the use of this model to guide the process of finding the optimal solution. Probabilistic models use prior knowledge and observations to update the posterior distribution as a means of predicting the value of a function at points that have not yet been observed. The first step in constructing the model is to determine the prior distribution, which is usually based on assumptions about the prior knowledge of the problem.

The probabilistic model is then updated based on Bayes’ Law as new observations are obtained. Constructing a probabilistic model requires the selection of an appropriate family of probability distributions, which is commonly used as a Gaussian process (GP).

Its specific implementation process is as follows:

(1) Initial sampling. Randomly select some sampling points and calculate the corresponding objective function values;

(2) Construct an agent model of the objective function based on the initial sampling using the Gaussian process;

(3) Optimize the proxy model. Use the proxy model to predict the objective function values and uncertainties of the new parameter points. Calculate the acquisition function based on the above predictions to balance the trade-off between exploration and exploitation. In this case, exploration is the selection of points with higher uncertainty to discover new optimal points. Exploitation is conducted to select points with better expected objective function values to improve the optimal solution by utilizing existing information;

(4) Updating the agent model. The next parameter point is selected under the guidance of the acquisition function and its objective function value is calculated. Add the new data point to the existing data set to update the agent model;

(5) Iterative repetition. Repeat steps 3 and 4 to gradually reduce the parameter space and find the optimal parameters. The iterative process ends when the preset number of iterations or convergence conditions are reached.

Bayesian optimization predicts which parameter combinations are likely to lead to the best performance by constructing a probabilistic model of the objective function and then selects the parameters to be evaluated next based on this predictive model. This approach is more efficient compared to random and lattice searches and is able to find near-optimal parameters with fewer evaluations, especially if the objective function evaluation is costly. In addition, Bayesian optimization is an iterative optimization process where each step uses the information from the previous steps to improve the next search. This means that as the search proceeds, the optimization process becomes progressively more efficient and precise and is able to better focus on the possible regions of optimal parameters.

When the search space is large and the problem complexity is high, stochastic search covers the solution space well but lacks relevance and efficiency. On the other hand, Bayesian optimization methods provide good relevance and high efficiency, but require high prior knowledge of the problem and perform erratically in high-dimensional spaces. Combining the two aims to exploit the global search capability of stochastic search and the local search efficiency of Bayesian optimization.

The specific steps of the algorithm for combining the two are as follows:

(1) Define the parameter space. Define the range of values and distribution of each hyperparameter;

(2) Initial exploration using random search. Determine the number of initial searches according to the specific problem and computational resources, and collect the initial data points after a certain number of random searches;

(3) Construct a Bayesian optimization agent model. Construct the agent model of the objective function using the initial data points in step 2;

(4) Optimize the agent model. Use the agent model to predict the objective function values and uncertainties for the new parameter points. Calculate the acquisition function based on these predictions to balance the tradeoff between exploration and exploitation. In this case, exploration is the selection of points with higher uncertainty to discover new optimal points. Exploitation is conducted to select points with better expected objective function values to improve the optimal solution by utilizing existing information;

(5) Updating the agent model. The next parameter point is selected under the guidance of the acquisition function and its objective function value is calculated. Add the new data point to the existing data set to update the agent model;

(6) Iterative repetition. Repeat steps 4 and 5 to gradually reduce the parameter space and find the optimal parameters. The iterative process ends when the preset number of iterations or convergence conditions are reached.

4. Simulation and Experiment

4.1. Simulation

In order to demonstrate the effectiveness of the proposed method, a two-input single-output system was built for simulation. Its specific formula is shown in Equation (25).

$$Y(s)=\left[\begin{array}{cc}{\displaystyle \frac{1}{s+1}}& {\displaystyle \frac{3}{5s+1}}\end{array}\right]U(s)$$

(25)

Bilinear discretization of the above transfer function matrix using MATLAB gives the impulse TFM as Equation (26).

$$\left[\begin{array}{cc}{\displaystyle \frac{0.04762(1+z^{-1})}{1-0.9048z^{-1}}}& {\displaystyle \frac{0.0297(1+z^{-1})}{1-0.9802z^{-1}}}\end{array}\right]$$

(26)

The least common multiple is extracted for each row of the impulse TFM and transformed into the form of a difference equation, as described below:

$$\begin{array}{cc}\hfill y(k)& =1.885y(k-1)-0.8869y(k-2)+0.04762u_1(k)+0.00094u_1(k-1)-0.046676u_1(k-2)\hfill \\ & +0.0297u_2(k)+0.028u_2(k-1)-0.05516u_2(k-2)\hfill \end{array}$$

(27)

The parameters to be estimated for this system are shown in Equation (28).

$$\left\{\begin{array}{c}{a}_1=1.885\\ a_2=-0.8869\end{array}\right.\hspace{1em}\left\{\begin{array}{c}{b}_{11}=0.04762\\ b_{12}=0.00094\\ b_{13}=-0.046676\\ b_{21}=0.0297\\ b_{22}=0.028\\ b_{23}=-0.05516\end{array}\right.$$

(28)

The input signals 1 and 2 are both random signals, as shown in Figure 4.

The PC hardware configuration is a Windows 11 operating system, 16 g RAM, and the CPU is an Intel i7-1260p. The MATLAB version is R2018b.

The estimation of model parameters (see Equation (27)) was performed using the recursive least square (RLS), SVR, and RSBO-SVR methods in the presence of outliers or colored noise. The data were divided into a training set and a test set, where 2/3 is the training set and 1/3 is the test set.

4.1.1. Presence of Outliers

We assumed that there were two outliers y (100) = 100 and y (120) = −50 in the dataset. The recursive process of RLS identification is shown in Figure 5. The training and prediction results of SVR are shown in Figure 6. The training and prediction results of RSBO-SVR are shown in Figure 7. The specific recognition results and errors are shown in

Table 2. The identification results and errors of SVR and RSBO-SVR (with outliers).

Parameters	Setpoints	SVR	RSBO-SVR	SVR Error	RSBO-SVR Error
$a_1$	1.885	1.885467	1.885583	0.02%	0.03%
$a_2$	−0.8869	−0.887002	−0.88585	0.01%	−0.12%
$b_{11}$	0.04762	0.047306	0.047903	−0.66%	0.59%
$b_{12}$	0.00094	0.000937	0.000935	−0.32%	−0.53%
$b_{13}$	−0.046676	−0.047586	−0.046977	1.95%	0.64%
$b_{21}$	0.0297	0.030307	0.030254	2.04%	1.87%
$b_{22}$	0.0028	0.002851	0.002831	1.82%	1.11%
$b_{23}$	−0.05516	−0.054962	−0.0554611	−0.36%	0.55%

.

Using the above-estimated data, the values of the relevant parameters of the TFM are calculated according to Equation (19), and their specific values and errors from the set values are shown in

Table 3. Estimation results for the parameters of the transfer function matrix (with outliers).

Parameters	Setpoints	SVR	RSBO-SVR	SVR Error	RSBO-SVR Error
$K_1$	1	1.005754	1.023848	0.58%	2.38%
$K_2$	3	3.060198	3.100016	2.01%	3.33%
$T_1$	1	1.013031	1.018668	1.30%	1.87%
$T_2$	5	4.998666	5.073316	−0.03%	1.47%

.

From Figure 6, it can be seen that the outliers had a great influence on the parameter identification of RLS, which will make RLS completely invalid and unable to identify the correct parameters. On the contrary, SVR had a strong anti-interference ability to the outliers, and its parameter identification results still had a high correct rate, as shown in Table 2; the maximum error was 2.04%, and the maximum error of the transfer function matrix parameters obtained after the operation was 2.01%. The RSBO-SVR algorithm also had very good identification results; the maximum error of the parameters of the difference equation was not more than 2%, and the maximum error of the transfer function matrix parameters obtained after the operation was 2.01%. The maximum error of the transfer function matrix parameters was only 3.33%.

4.1.2. Colored Noise States

Assuming that the model of Equation (27) contains colored noise, the expression is shown in Equation (29).

$$e(k)=1.5e(k-1)-0.7e(k-2)-0.1e(k-3)+0.5v(k)+0.2v(k-1)$$

(29)

where $v(k)$ is a white noise signal with mean 0 and variance 1.

The training and prediction results of SVR and RSBO-SVR are shown in Figure 8 and Figure 9. The specific identification results and errors are shown in

Table 4. The identification results and errors of RLS, SVR, and RSBO-SVR (with colored noise).

Parameters	Setpoints	RLS	SVR	RSBO-SVR	RLS Error	SVR Error	RSBO-SVR Error
$a_1$	1.885	1.7199	1.885467	1.885395	−8.76%	0.02%	0.02%
$a_2$	−0.8869	−0.8937	−0.887002	−0.88687	0.77%	0.01%	0.00%
$b_{11}$	0.04762	0.1609	0.047306	0.047441	237.88%	−0.66%	−0.38%
$b_{12}$	0.00094	0.000534	0.000937	0.000942	−43.19%	−0.32%	0.21%
$b_{13}$	−0.046676	−0.0043	−0.047586	−0.048103	−90.79%	1.95%	3.06%
$b_{21}$	0.0297	0.0383	0.030307	0.028976	28.96%	2.04%	−2.44%
$b_{22}$	0.0028	0.0087	0.002851	0.002849	210.71%	1.82%	1.75%
$b_{23}$	−0.05516	−0.0092	−0.054962	−0.055081	−83.32%	−0.36%	−0.14%

.

From the above-shown table, it can be seen that in most of the parameter identification results, SVR and RSBO-SVR performed closer to the set values than RLS, and the errors were generally lower than RLS. The maximum errors were 2.04% for SVR, 3.06% for RSBO-SVR, and 237.88% for RLS, which was more than 100 times higher than that of SVR, and more than 100 times higher than that of RSBO-SVR. The maximum error of RLS was 237.88%, which is more than 100 times greater than SVR and more than 70 times greater than RSBO-SVR. This shows that SVR and RSBO-SVR identify the parameters better than RLS in the presence of colored noise. The values of the relevant parameters of the transfer function matrix were calculated using the above-shown data, and the exact values and the errors with respect to the set values are shown in

Table 5. Estimation results for the parameters of the transfer function matrix (with colored noise).

Parameters	Setpoints	RLS	SVR	RSBO-SVR	RLS Error	SVR Error	RSBO-SVR Error
$K_1$	1	1.417	1.0059	0.965006	41.7%	0.59%	−3.50%
$K_2$	3	23.08	3.0599	2.918578	669.3%	1.997%	−2.71%
$T_1$	1	0.39	1.013	0.967059	−61%	1.3%	−3.29%
$T_2$	5	30.08	4.9987	4.9862	501.6%	−0.026%	−0.28%

.

In the presence of colored noise in the system, although RLS could identify the relevant parameter values, the error was very large; as can be seen from the table above, the maximum error was 669.3% and the minimum error was more than 40%, and the model built by using this identification parameter could not accurately describe the system. In contrast, the maximum error of SVR was only 1.997%, and the maximum error of RSBO-SVR was only −3.5%, which could describe the relevant system better.

Both SVR and RSBO-SVR have good estimation accuracy, and we compared their estimation times, as shown in

Table 6. Estimated times for SVR and RSBO-SVR.

State	SVR	RSBO-SVR
With outliers	201 s	20 s
With colored noise	251 s	53 s

below.

In the presence of outliers, the estimation time of RSBO-SVR decreased by 90.05% compared to SVR; in the presence of colored noise interference, the estimation time of RSBO-SVR decreased by 78.88% compared to SVR.

4.2. Experimental Verification

In order to verify the practical application of the algorithm, a water tank model was created. The diagram of the water tank model is shown in Figure 10.

Three systems need to be identified in the above-shown system. To explain in detail, the following is an example of Tank 1.

To identify Tank 1, we used a pressure sensor to obtain level information and a pump and ball valve to control the level. The current value of the pressure sensor was obtained, and the current values of the input pump and ball valve were used as identification data.

The input data and output data are shown in Figure 11. The complete data are shown in Appendix A. Parameter estimation was carried out using SVR and RSBO-SVR algorithms and the training and prediction process is shown in Figure 12 and Figure 13.

The final recognized model of SVR is shown in Equation (30). The recognized model of RSBO-SVR is shown in Equation (31).

$$y(k)=0.8107y(k-1)+0.1717y(k-2)-0.0074u_1(k-2)-0.0055u_1(k-3)-0.0222u_2(k-2)+0.0015u_2(k-3)+0.4492$$

(30)

$$y(k)=0.7998y(k-1)+0.2056y(k-2)-0.0077u_1(k-2)-0.0053u_1(k-3)-0.0219u_2(k-2)+0.0015u_2(k-3)+0.4001$$

(31)

To verify the accuracy of the estimated model, the same input signals were given to the estimated model and the actual system. The output signals and errors of the two systems are shown in Figure 14 and Figure 15.

The error between the estimated output signal and the actual output signal of the SVR algorithm was effectively controlled within ±1.5%. The maximum error between the estimated output signal and the actual output signal of the RSBO-SVR algorithm did not exceed 3%. This indicates that the accuracy of the estimation process of SVR and RSBO-SVR was very high. The estimation time of SVR was 4085 s and the estimation time of RSBO-SVR was 25 s. The estimation accuracy of RSBO-SVR decreased in comparison with SVR, but its estimation speed improved dramatically, being 99.38% faster when compared with SVR.

5. Conclusions

In this study, we investigated the performance of SVR and RSBO-SVR algorithms for parameter identification in multivariate systems, including challenging conditions such as common outliers and colored noise disturbances. The results showed that both the RSBO-SVR and SVR algorithms utilize the structural risk minimization principle and the L2 regularization term to effectively reduce the complexity of the model while maintaining robustness to noise disturbances. Specifically, the use of the maximum margin strategy and insensitive loss function enabled the model to effectively resist noise and outliers.

For parameter identification of multivariate systems, the RLS algorithm was completely ineffective when there were outliers in the collected input and output data. The SVR algorithm and RSBO-SVR algorithm, however, still had high identification accuracy under the same conditions. In addition, in order to solve the problem of the long optimization time of SVR parameters, the RSBO-SVR algorithm combining random search and Bayesian optimization is proposed. When the system contains colored noise, RLS could identify the parameters, but the error was very large, and the identified system could not correctly reflect the system characteristics. In contrast, color noise had less of an effect on the SVR and RSBO-SVR algorithms. Compared with SVR, RSBO-SVR had a faster estimation speed.

In addition, the tank experiments prove that these algorithms can be applied to the identification of real systems; the maximum output error between the estimated model and the actual model was only 1.5% for the SVR algorithm, and the maximum output error between the estimated model and the actual model was not more than 3% for the RSBO-SVR algorithm. Although the accuracy is lower than SVR, it still meets the requirements of industrial production. Compared with SVR, the estimation time of RSBO-SVR was 99.38% shorter.

6. Limitation of Study

There are three main limitations in this study: first, the parameter estimation model was constructed based on a linear multivariate system; although fast and accurate parameter identification can be achieved by the proposed method, it may generate modeling bias for an actual engineering system with strong nonlinear coupling characteristics. Second, although the SVR-based parameter estimation algorithm balances the model complexity and prediction error via L2 regularization, its loss function design does not impose explicit constraints on the unbiasedness of parameter estimation. Third, the current framework does not introduce a compensation mechanism for non-minimum phase zeros and time lag links, which may widen the phase margin prediction error for hydraulic servo systems with significant inverted response characteristics or process control objects with transmission delays of more than 40% of the sampling period, potentially affecting their applicability in specific industrial scenarios. These limitations point the way to improvements in subsequent research, including the introduction of a nonlinear model predictive control framework [46,47], the incorporation of Bayesian regularization constraints [48], and the embedding of Padé time-lag approximation modules [49].