Neural Computing Enhanced Parameter Estimation for MultiInput and MultiOutput Total NonLinear Dynamic Models
Abstract
:1. Introduction
1.1. Literature Survey
1.2. Motivation and Contributions
2. Total NonLinear Model
 (i)
 The input layer consists of regression terms ${p}_{k}^{n}(t)(k=1,\dots ,N)$ and ${p}_{k}^{d}(t)(k=1,\dots ,D)$; here, a neuron in the hidden layer is not connected to all the neurons in the input layer, that is, the network is a noncompletely connected feedforward neural network.
 (ii)
 The action function of the neurons in the hidden layer is linear, and the output of the hidden layer neurons is ${a}_{i}(t)$ or ${b}_{i}(t)$.
 (iii)
 The action function of the output layer neurons is linear, and the output of the ith output layer neuron is ${b}_{i}(t){e}_{i}(t)$.
 (iv)
 The connection weights between the input layer neurons and the hidden layer neurons are the parameters ${\mathsf{\theta}}_{k}^{n}$ and ${\mathsf{\theta}}_{k}^{d}$ of the model.
 (v)
 The connection weight between the hidden layer neurons and the ith output layer neurons are $1$ and the observed output ${\mathrm{y}}_{i}(t)$.
 (i)
 By setting parameter $i=1$, Zhu’s [18] model can be a special case of the model in Formula (1).
 (ii)
 The model is nonlinear in parameters and regression terms, which was caused by denominator polynomials.
 (iii)
 When the denominator ${b}_{i}(t)$ of the model is close to 0, the output deviation would be large. In this paper, considering this point, division operation was avoided in the action function of the neuron when the neural network model was being built.
 (iv)
 The structure of the neural network corresponding to the total nonlinear model is a noncompletely connected feedforward neural network, or a partially connected feedforward neural network. Therefore, the convergence of the network becomes a big problem, which is the difficulty of this paper.
 (v)
 The model has a wide range of application prospects. In many nonlinear system modeling and control applications, the total nonlinear model has been gradually adopted. Some nonlinear models, such as the exponential model ${\mathrm{e}}^{x}$, which describes the change of dynamic rate constant with temperature, cannot be directly used. The exponential model can be firstly transformed into a nonlinear model (${\mathrm{e}}^{x}=\frac{1\frac{x}{2}+\frac{{x}^{2}}{12}}{1+\frac{x}{2}+\frac{{x}^{2}}{12}}$), and then, system identification can be implemented [19,21,22].
3. Gradient Descent Calculation of Parameter Estimation
Algorithm 1. Gradient Descent Algorithm 
1: Initialization: The weights of the neural network (parameters of a total nonlinear model) are set as random little numbers with uniform distribution; the average value is zero, and the variance is small. Set the maximum number of iterations T, the minimum error ε, and the maximum number of samples $P$. 2: Generate training sample set {$X$,$\text{}Y$} of the neural network according to Formula (1), where $X=\left\{{X}_{1},{X}_{2},\dots ,{X}_{I}\right\}$, $Y$ = {${Y}_{1},{Y}_{2},\dots ,{Y}_{I}$}, ${X}_{i}\ni ${${p}_{1}^{\mathrm{n}}(t)$,${p}_{2}^{\mathrm{n}}(t)$,…,${p}_{N}^{\mathrm{n}}(t),{p}_{1}^{d}$,${p}_{2}^{d},\dots ,{p}_{D}^{d}$}, ${\mathrm{Y}}_{i}$={${y}_{i}(t)$}. 3: Input a training sample p to the neural network. 4: Calculate the output value ${a}_{i}(\mathrm{k})$,${y}_{i}(t){e}_{i}(t)$ and ${f}_{i}(t)$ of the neurons in the hidden layer and the output layer according to Formulas (2), (3), and (4), respectively. 5: Adjust the weight of the neural network according to Formulas (10) and (13). 6: Calculate the error $E(t)$ according to Formula (4) and calculate the total error according to Formula (14).
$$E={{\displaystyle \sum}}^{\text{}}E(t)$$
7: $p=p+1$ 8: If p > P, then t = t + 1; otherwise, run step 3. 9: If $E<$ ε or $t>T$, stop training; otherwise, run step 3. 
4. Model Structure Detection
Algorithm 2. KnockOut Algorithm 
1: Using the network structure shown in Figure 1, all the items contained in the whole items set are taken as the input of the network. 2: The algorithm in Section 3 is used to train the network, and network error ${E}_{1}$ is obtained. 3: A new network structure is obtained by randomly removing a network input. The algorithm in Section 3 is used to train the new network, and network error ${E}_{2}$ is obtained. If ${E}_{2}\le {E}_{1}$, then ${E}_{1}={E}_{2}$. Otherwise, this operation should be invalid (the input is reserved). 4: Another input is selected, and step 3 is executed again until all the input items are executed once. 5: The N connection weights between the input layer and the hidden layer are sorted in descending order. The first n weights are selected to make the significance reach 95%. Meanwhile, Formulas (15) and (16) are met, and the network input items corresponding to the first n weights are retained.
$$\frac{{{\displaystyle \sum}}_{\mathrm{i}=1}^{n}\left{w}_{i}\right}{{{\displaystyle \sum}}_{i=1}^{N}\left{w}_{i}\right}\ge 0.95$$
$$\frac{{{\displaystyle \sum}}_{i=1}^{n1}\left{w}_{i}\right}{{{\displaystyle \sum}}_{i=1}^{N}\left{w}_{i}\right}<0.95$$

5. Convergence Analysis of the Algorithm
6. Simulation Results and Discussions
7. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
 Billings, S.A.; Chen, S. Identification of nonlinear rational systems using a predictionerror estimation algorithm. Int. J. Syst. Sci. 1989, 20, 467–494. [Google Scholar] [CrossRef]
 Billings, S.A.; Zhu, Q.M. Rational model identification using an extended leastsquares algorithm. Int. J. Control 1991, 54, 529–546. [Google Scholar] [CrossRef]
 Sontag, E.D. Polynomial Response Maps. Lecture Notes in Control & Information Sciences; Springer: Berlin/Heidelberg, Germany, 1979; Volume 13. [Google Scholar]
 Narendra, K.S.; Parthasarathy, K. Identification and control of dynamical systems using neural networks. IEEE Trans. Neural Netw. 2002, 1, 4–27. [Google Scholar] [CrossRef] [PubMed] [Green Version]
 Zhu, Q.M.; Ma, Z.; Warwick, K. Neural network enhanced generalised minimum variance selftuning controller for nonlinear discretetime systems. IEE Proc. Control Theory Appl. 1999, 146, 319–326. [Google Scholar] [CrossRef]
 Billings, S.A.; Zhu, Q.M. A structure detection algorithm for nonlinear dynamic rational models. Int. J. Control 1994, 59, 1439–1463. [Google Scholar] [CrossRef]
 Zhu, Q.M.; Billings, S.A. Recursive parameter estimation for nonlinear rational models. J. Syst. Eng. 1991, 1, 63–67. [Google Scholar]
 Zhu, Q.M.; Billings, S.A. Parameter estimation for stochastic nonlinear rational models. Int. J. Control 1993, 57, 309–333. [Google Scholar] [CrossRef]
 Aguirre, L.A.; Barbosa, B.H.G.; Braga, A.P. Prediction and simulation errors in parameter estimation for nonlinear systems. Mech. Syst. Signal Process. 2010, 24, 2855–2867. [Google Scholar] [CrossRef]
 Huo, M.; Duan, H.; Luo, D.; Wang, Y. Parameter Estimation for a VTOL UAV Using Mutant Pigeon Inspired Optimization Algorithm with Dynamic OBL Strategy. In Proceedings of the 2019 IEEE 15th International Conference on Control and Automation (ICCA), Edinburgh, UK, 16–19 July 2019; pp. 669–674. [Google Scholar]
 Zhu, Q.M.; Yu, D.; Zhao, D. An Enhanced Linear Kalman Filter (EnLKF) algorithm for parameter estimation of nonlinear rational models. Int. J. Syst. Sci. 2016, 48, 451–461. [Google Scholar] [CrossRef] [Green Version]
 Türksen, Ö.; Babacan, E.K. Parameter Estimation of Nonlinear Response Surface Models by Using Genetic Algorithm and Unscented Kalman Filter. In Chaos, Complexity and Leadership 2014; Erçetin, S., Ed.; Springer Proceedings in Complexity; Springer: Cham, Switzerland, 2016. [Google Scholar]
 Billings, S.A.; Mao, K.Z. Structure detection for nonlinear rational models using genetic algorithms. Int. J. Syst. Sci. 1998, 29, 223–231. [Google Scholar] [CrossRef]
 Plakias, S.; Boutalis, Y.S. Lyapunov Theory Based Fusion Neural Networks for the Identification of Dynamic Nonlinear Systems. Int. J. Neural Syst. 2019, 29, 1950015. [Google Scholar] [CrossRef] [PubMed]
 Kumar, R.; Srivastava, S.; Gupta, J.R.P.; Mohindru, A. Diagonal recurrent neural network based identification of nonlinear dynamical systems with lyapunov stability based adaptive learning rates. Neurocomputing 2018, 287, 102–117. [Google Scholar] [CrossRef]
 Chen, S.; Liu, Y. Robust Distributed Parameter Estimation of Nonlinear Systems with Missing Data over Networks. IEEE Trans. Aerosp. Electron. Syst. 2019. [Google Scholar] [CrossRef]
 Zhu, Q.M. An implicit least squares algorithm for nonlinear rational model parameter estimation. Appl. Math. Model. 2005, 29, 673–689. [Google Scholar] [CrossRef]
 Zhu, Q.M. A back propagation algorithm to estimate the parameters of nonlinear dynamic rational models. Appl. Math. Model. 2003, 27, 169–187. [Google Scholar] [CrossRef]
 Zhu, Q.M.; Wang, Y.; Zhao, D.; Li, S.; Billings, S.A. Review of rational (total) nonlinear dynamic system modelling, identification, and control. Int. J. Syst. Sci. 2015, 46, 2122–2133. [Google Scholar] [CrossRef]
 Leung, H.; Haykin, S. Rational function neural network. Neural Comput. 1993, 5, 928–938. [Google Scholar] [CrossRef]
 Jain, R.; Narasimhan, S.; Bhatt, N.P. A priori parameter identifiability in models with nonrational functions. Automatica 2019, 109, 108513. [Google Scholar] [CrossRef]
 Kambhampati, C.; Mason, J.D.; Warwick, K. A stable onestepahead predictive control of nonlinear systems. Automatica 2000, 36, 485–495. [Google Scholar] [CrossRef]
 Kumar, R.; Srivastava, S.; Gupta, J.R.P. Lyapunov stabilitybased control and identification of nonlinear dynamical systems using adaptive dynamic programming. Soft Comput. 2017, 21, 4465–4480. [Google Scholar] [CrossRef]
 Ge, H.W.; Du, W.L.; Qian, F.; Liang, Y.C. Identification and control of nonlinear systems by a timedelay recurrent neural network. Neurocomputing 2009, 72, 2857–2864. [Google Scholar] [CrossRef]
 Verdière, N.; Zhu, S.; DenisVidal, L. A distribution input–output polynomial approach for estimating parameters in nonlinear models. Application to a chikungunya model. J. Comput. Appl. Math. 2018, 331, 104–118. [Google Scholar] [CrossRef]
 Li, F.; Jia, L. Parameter estimation of hammersteinwiener nonlinear system with noise using special test signals. Neurocomputing 2019, 344, 37–48. [Google Scholar] [CrossRef]
 Chen, C.Y.; Gui, W.H.; Guan, Z.H.; Wang, R.L.; Zhou, S.W. Adaptive neural control for a class of stochastic nonlinear systems with unknown parameters, unknown nonlinear functions and stochastic disturbances. Neurocomputing 2017, 226, 101–108. [Google Scholar] [CrossRef] [Green Version]
${\mathit{u}}_{1}(\mathit{t})$  ${\mathit{u}}_{2}(\mathit{t})$  ${\mathsf{\theta}}_{1}$  ${\mathsf{\theta}}_{2}$  ${\mathsf{\theta}}_{3}\text{}$  ${\mathsf{\theta}}_{4}\text{}$  ${\mathsf{\theta}}_{5}\text{}$  ${\mathsf{\theta}}_{6}\text{}$  ${\mathsf{\theta}}_{7}\text{}$  ${\mathsf{\theta}}_{8}\text{}$  ${\mathsf{\theta}}_{9}\text{}$  ${\mathsf{\theta}}_{10}\text{}$  MSE 

sine  sine  0.5002  0.8025  1.0003  1.0034  1.0000  0.2006  0.5010  1.0004  1.0018  0.9991  2.351E06 
sine  square  0.5000  0.8000  1.0000  1.0000  1.0000  0.1996  0.4982  1.0182  0.9677  1.0473  0.0003 
square  square  0.4973  0.8760  1.0110  1.0031  1.0153  0.2013  0.5072  1.0354  0.9744  1.0840  0.0015 
${\mathit{u}}_{1}(\mathit{t})$  ${\mathit{u}}_{2}(\mathit{t})$  ${\mathsf{\theta}}_{1}$  ${\mathsf{\theta}}_{2}$  ${\mathsf{\theta}}_{3}\text{}$  ${\mathsf{\theta}}_{4}\text{}$  ${\mathsf{\theta}}_{5}\text{}$  ${\mathsf{\theta}}_{6}\text{}$  ${\mathsf{\theta}}_{7}\text{}$  ${\mathsf{\theta}}_{8}\text{}$  ${\mathsf{\theta}}_{9}\text{}$  ${\mathsf{\theta}}_{10}\text{}$  MSE 

sine  sine  0.5003  0.8041  1.0005  1.0054  1.0001  0.2008  0.5014  1.0005  1.0016  0.9987  5.342E06 
sine  square  0.5000  0.8001  1.0000  1.0001  1.0000  0.2045  0.5019  1.073  1.1364  1.0898  0.0032 
square  square  0.4953  0.8765  1.0085  1.0327  1.0095  0.2969  0.7030  0.9971  1.0007  0.9953  0.0058 
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Liu, L.; Ma, D.; Azar, A.T.; Zhu, Q. Neural Computing Enhanced Parameter Estimation for MultiInput and MultiOutput Total NonLinear Dynamic Models. Entropy 2020, 22, 510. https://doi.org/10.3390/e22050510
Liu L, Ma D, Azar AT, Zhu Q. Neural Computing Enhanced Parameter Estimation for MultiInput and MultiOutput Total NonLinear Dynamic Models. Entropy. 2020; 22(5):510. https://doi.org/10.3390/e22050510
Chicago/Turabian StyleLiu, Longlong, Di Ma, Ahmad Taher Azar, and Quanmin Zhu. 2020. "Neural Computing Enhanced Parameter Estimation for MultiInput and MultiOutput Total NonLinear Dynamic Models" Entropy 22, no. 5: 510. https://doi.org/10.3390/e22050510