The Euler-Type Universal Numerical Integrator (E-TUNI) with Backward Integration

Abstract

The Euler-Type Universal Numerical Integrator (E-TUNI) is a discrete numerical structure that couples a first-order Euler-type numerical integrator with some feed-forward neural network architecture. Thus, E-TUNI can be used to model non-linear dynamic systems when the real-world plant’s analytical model is unknown. From the discrete solution provided by E-TUNI, the integration process can be either forward or backward. Thus, in this article, we intend to use E-TUNI in a backward integration framework to model autonomous non-linear dynamic systems. Three case studies, including the dynamics of the non-linear inverted pendulum, were developed to verify the computational and numerical validation of the proposed model.

1. Introduction

The Euler-Type Universal Numerical Integrator (E-TUNI) is a particular case of a Universal Numerical Integrator (UNI). In turn, the UNI is the coupling of a conventional numerical integrator (e.g., Euler, Runge–Kutta, Adams-Bashforth, Predictive-Corrector, among others) with some kind of universal approximator of functions (e.g., MLP neural networks, SVM, RBF, wavelets, fuzzy inference systems, among others). Furthermore, UNIs generally work exclusively with neural networks with feed-forward architecture through supervised learning with input/output patterns.

In [1], the first work involving the mathematical modelling of an artificial neuron is presented. In [2,3], it is mathematically demonstrated, independently, that feed-forward neural networks with Multi-Layer Perceptron (MLP) architecture are universal approximators of functions. For the mathematical demonstration of the universality of feed-forward networks, a crucial starting point is the Stone-Weierstrass theorem [4]. An interesting work on the universality of neural networks involving the ReLU neuron can be found in [5]. In summary, many neural architectures were developed in the last half of the twentieth century, involving shallow neural networks [6]. On the other hand, in this century, many architectures of deep neural networks demonstrated the great power of neural networks [7].

So, artificial neural networks solve an extensive range of computational problems. However, in this article, we limit the application of artificial neural networks with feed-forward architecture in the resolution of mathematical problems involving modelling autonomous non-linear dynamic systems governed by a system of ordinary differential equations. With the neural networks, it is possible to solve the inverse problem of modelling dynamical systems, i.e., given the solution of the dynamical system, then obtain the instantaneous derivative functions or the mean derivative functions of the system.

The starting point for the proper design of a Universal Numerical Integrator (UNI) is the conventional numerical integrators [8,9,10,11,12,13,14,15]. In [16], a qualitative and very detailed description of the design of a UNI is presented. Also, in [16], an appropriate classification is given for the various types of Universal Numerical Integrators (UNIs) found in the literature.

According to these authors, the UNIs can be divided into three large groups, giving rise to three different methodologies, namely: (i) the NARMAX methodology (Nonlinear Auto Regressive Moving Average with Exogenous input); (ii) the instantaneous derivatives methodology (e.g., Runge–Kutta neural network, Adams-Bashforth neural network, Predictive-Corrector neural network, among others), and (iii) mean derivatives methodologies or E-TUNI. For a detailed understanding of the similarities and differences between the existing UNIS types, see again [16].

In practice, the leading scientific works involving the proper design of a UNI are: (i) the NARMAX methodology in [17,18,19,20,21]; (ii) the Runge–Kutta neural network in [22,23,24,25]; (iii) the Adams-Bashforth neural network in [26]; and (iv) the E-TUNI in [27,28]. For this article, it should be noted that E-TUNI works exclusively with mean derivative functions instead of instantaneous derivative functions. Furthermore, the NARMAX and mean derivative methodologies are of fixed steps in the simulation phase, while the instantaneous derivative methodologies are of varied integration steps in the simulation phase [16]. However, all of them are fixed steps in the training phase.

In this paragraph, we briefly describe existing work on E-TUNI. In [27], it is a technological application of E-TUNI using neural networks with MLP architecture. In [28], it is a technological application of E-TUNI using fuzzy logic and genetic algorithms. Still in [28], the fuzzy membership functions are adjusted automatically, using genetic algorithms through supervised learning using input/output patterns.

However, none of the previous works used E-TUNI with backward integration. Therefore, this article’s originality lies in using E-TUNI in a backward integration process coupled with a shallow neural network with MLP architecture. As far as we know, this has not yet been done.

This article is divided as follows: Section 2 briefly describes the symbology and notation adopted in this paper. Section 3 describes the context in which this work can be applied to real-world problems. Section 4 briefly describes the basic structure of a Universal Numerical Integrator (UNI). Section 5 describes the detailed mathematical development of E-TUNI, designed with backward integration. Section 6 analyzes the numerical accuracy of E-TUNI using Landau symbols. Section 7 presents the numerical and computational results validating the proposed model. Section 8 presents the main conclusions of this work.

2. Symbols and Notations Adopted

For a more detailed understanding of the theoretical development presented in this paper, we define all the symbols and variables used in this work. The proposed method is called Euler-Type Universal Numerical Integrator (E-TUNI) and uses backward integration. This method is discrete. Thus, we propose a discrete method to approximate a continuous system of autonomous ordinary differential equations. Therefore, below, we present all the symbologies and variables used here. They are divided into two groups: (i) continuous variables and (ii) discrete variables.

(i) 

Continuous Variables

• $\dot{y}=f\left(y\right)$—System of continuous differential equations.
• $y={[y_1{y}_2\dots y_n]}^T$—State Variables.
• $f\left(y\right)={[f_1\left(y\right)f_2\left(y\right)\dots f_n\left(y\right)]}^T$—Instantaneous derivative functions.
• d—Plant noise.
• $d^{\prime }$—Measuring instrument noise.
• $y_j^i\left(t\right)=g_j^i\left(t\right)$—Particular continuous and differentiable curve of a family of solution curves of the dynamical system $\dot{y}=f\left(y\right)$.
• ${\dot{y}}_j^i\left(t\right)={\dot{g}}_j^i\left(t\right)$—First derivative of $y_j^i\left(t\right)$.

(ii) 

Discrete Variables

• ${}^{k}y^i=y^i(t_0+k·\Delta t)$—Vector of state variables at time $t_k$.
• ${}^{k}y_j^i$—Scalar state variable for $j=1,2,\dots ,n$ at time $t_k$. It is a generic discretization point of the state variables generated by the integers i, j, and k.
• n—Total number of state variables.
• ${}^{k}u=u(t_0+k·\Delta t)$—Vector of control variables at time $t_k$.
• ${}^{k}u_j$—Scalar control variable for $j=1,2,\dots ,m$ at time $t_k$.
• m—Total number of control variables.
• ${}^{k+1}y^i=y^i[t_0+(k+1)·\Delta t]$—Exact vector of state variables at time $t_{k+1}$.
• ${}^{k+1}y_j^i$—Exact scalar state variable for $j=1,2,\dots ,n$ at time $t_{k+1}$.
• ${}^{k+1}\widehat{y}^i={\widehat{y}}^i[t_0+(k+1)·\Delta t]$—Estimated Vector of state variables by UNI or E-TUNI at time $t_{k+1}$.
• ${}^{k+1}\widehat{y}_j^i$—Scalar state variable estimated by UNI or E-TUNI for $j=1,2,\dots ,n$ at time $t_{k+1}$.
• ${}^{k+1}\tilde{y}^i$—Estimated Vector of state variables when using only the integrator and without using the neural network at time $t_{k+1}$.
• $ta{n}_{\Delta t}{}^k\alpha ^i=ta{n}_{\Delta t}^{+}{}^k\alpha ^i={[ta{n}_{\Delta t}^{+}{}^k\alpha _1^{i}ta{n}_{\Delta t}^{+}{}^k\alpha _2^i\dots ta{n}_{\Delta t}^{+}{}^k\alpha _n^i]}^T$—Exact vector of positive mean derivative functions at time $t_k$.
• $ta{n}_{\Delta t}{}^k\alpha _j^i=ta{n}_{\Delta t}^{+}{}^k\alpha _j^i={\displaystyle \frac{{}^{k+1}y_j^i-{}^{k}y_j^i}{\Delta t}}$—Scalar positive mean derivative functions for $j=1,2,\dots ,n$ at time $t_k$.
• $ta{n}_{\Delta t}{}^k\widehat{\alpha }^i=ta{n}_{\Delta t}^{+}{}^k\widehat{\alpha }^i={[ta{n}_{\Delta t}^{+}{}^k\widehat{\alpha }_1^{i}ta{n}_{\Delta t}^{+}{}^k\widehat{\alpha }_2^i\dots ta{n}_{\Delta t}^{+}{}^k\widehat{\alpha }_n^i]}^T$—Estimated vector of positive mean derivative functions by the E-TUNI at time $t_k$.
• $ta{n}_{\Delta t}^{-}{}^k\alpha ^i={[ta{n}_{\Delta t}^{-}{}^k\alpha _1^{i}ta{n}_{\Delta t}^{-}{}^k\alpha _2^i\dots ta{n}_{\Delta t}^{-}{}^k\alpha _n^i]}^T$—Exact vector of negative mean derivative functions at time $t_k$.
• $ta{n}_{\Delta t}^{-}{}^k\alpha _j^i={\displaystyle \frac{{}^{k}y_j^i-{}^{k-1}y_j^i}{\Delta t}}$—Scalar negative mean derivative functions for $j=1,2,\dots ,n$ at time $t_k$.
• $ta{n}_{\Delta t}^{-}{}^k\widehat{\alpha }^i={[ta{n}_{\Delta t}^{-}{}^k\widehat{\alpha }_1^{i}ta{n}_{\Delta t}^{-}{}^k\widehat{\alpha }_2^i\dots ta{n}_{\Delta t}^{-}{}^k\widehat{\alpha }_n^i]}^T$—Estimated vector of negative mean derivative functions by the E-TUNI at time $t_k$.
• $tan{}^k\theta ^i=ta{n}^{+}{}^k\theta ^i={[ta{n}^k{\theta }_1^{i}ta{n}^k{\theta }_2^i\dots ta{n}^k{\theta }_n^i]}^T$—Vector of positive instantaneous derivatives at time $t_k$.
• $ta{n}^k{\theta }_j^i=ta{n}^{+}{}^k\theta _j^i=\underset{\Delta t\to 0}{lim}{\displaystyle \frac{{}^{k+1}y_j^i-{}^{k}y_j^i}{\Delta t}}$—Scalar positive instantaneous derivative for $j=1,2,\dots ,n$ at instant $t_k$.
• $ta{n}^{-}{}^k\theta ^i={[ta{n}^{-}{}^k\theta _1^{i}ta{n}^{-}{}^k\theta _2^i\dots ta{n}^{-}{}^k\theta _n^i]}^T$—Vector of negative instantaneous derivatives at instant $t_k$.
• $ta{n}^{-}{}^k\theta _j^i=\underset{\Delta t\to 0}{lim}{\displaystyle \frac{{}^{k}y_j^i-{}^{k-1}y_j^i}{\Delta t}}$—Scalar negative instantaneous derivative for $j=1,2,\dots ,n$ at time $t_k$.
• $t_k$—Time instant $t_k=t_0+k·\Delta t$.
• $t_{k+1}$—Time instant $t_{k+1}=t_0+(k+1)·\Delta t$.
• $\Delta t$—Integration step.
• i—Over-index that enumerates a particular curve from the family of curves of the dynamical system to be modelled ($i=1,2,\dots ,q$).
• j—Under-index that enumerates the state and control variables.
• k—Over-index that enumerates the discrete time instants ($k=1,2,\dots ,r$).
• r—Total number of horizons of the time variable.
• q—Total number of curves from the family of curves curves of the dynamic system to be modelled.
• $t_k^{*}$—Instant of time within the interval $[t_k,t_{k+1}]$ as a result of the Differential Mean Value Theorem (see Theorem 2).
• $t_k^x$—Instant of time within the interval $[t_k,t_{k+1}]$ as a result of the Integral Mean Value Theorem (see Theorem 3).

To understand this symbology, the first step is to understand the difference between ${}^{k+1}y^i$, ${}^{k+1}\tilde{y}^i$, and ${}^{k+1}\widehat{y}^i$. The vector variable ${}^{k+1}y^i$ is the exact value of the state variables at time $t_{k+1}=t_0+(k+1)·\Delta t$. The vector variable ${}^{k+1}\tilde{y}^i$ is the estimated value of the state variables using only the numerical integrator. Finally, the vector variable ${}^{k+1}\widehat{y}^i$ is the estimated value of the state variables using the numerical integrator coupled with an artificial neural network. Figure 1, Figure 2, Figure 3 and Figure 4 follow this convention for a better understanding of the model proposed in this work. Additionally, it is important to understand the meaning of the auxiliary variables i, j, and k. This is described in the next paragraph.

Here, an important issue is to elucidate the use of the over-indexes k and i and the sub-index j in the variables ${}^{k}y_j^i$ and $ta{n}_{\Delta t}{}^k\alpha _j^i$. Figure 1 will help in this explanation. When the auxiliary variables i, j, and k are used simultaneously, they uniquely identify the secant ($ta{n}_{\Delta t}{}^k\alpha _j^i$) at the point ${}^{k}y_j^i$. Notice that the over-index k indicates the time instant $t_k=t_0+k·\Delta t$ of the secant, the sub-index j indicates the state variable in question, where $j=1,2,\dots ,n$, and the over-index i$(i=1,2,\dots ,q)$ indicates the particular curve, where the respective secant is located, of the family of possible curves of the system of differential equations considered. The value of q can be as large as one wants. The larger the value of q the more different curves will be trained by the neural network, and the better its generalization will be.

This unique mapping of the secant $ta{n}_{\Delta t}{}^k\alpha _j^i$ will be of fundamental importance for the demonstration of Theorem 4 that will be presented and demonstrated in Section 5. This theorem establishes that if the Euler integrator uses the secant instead of the instantaneous derivatives, it will be a discrete and exact solution for a system of ordinary and autonomous differential equations. Furthermore, the mapping mentioned above is unique because of the uniqueness theorem for systems of ordinary differential equations. Finally, an artificial neural network will be used to learn the secants of the considered dynamic system through supervised learning using input/output training patterns.

3. Contextualization

This section clarifies some critical issues regarding using the E-TUNI in mathematics, physics, and engineering problems. These issues will help provide a better theoretical and practical understanding of the method proposed in this article. To this end, in the following, we formulate five essential questions regarding the proper design of E-TUNI and give succinct and objective answers to all of them.

What is the difference between E-TUNI and the Conventional Euler method? The traditional Euler method works with instantaneous derivative functions, and the E-TUNI works with mean derivative functions. Although this difference may seem insignificant, in reality, it is not. We also can refer to Figure 2 to answer this question. In this figure, ${}^{k+1}\tilde{y}_j^i$, ${}^{k+1}\widehat{y}_j^i$, and ${}^{k+1}y_j^i$ are, respectively, the solution obtained by conventional Euler, the solution obtained by E-TUNI, and the exact solution of the dynamical system at time $t_{k+1}$. As can be seen, the mean derivative gives a more accurate solution than the instantaneous derivative. Another advantage of E-TUNI is that it can be designed using real-world training patterns through supervised learning. To this end, E-TUNI uses an artificial neural network to learn the mean derivative functions with high precision. Notice also a difference between the values ${}^{k+1}\widehat{y}_j^i$ and ${}^{k+1}y^i$. This difference is caused by the training of the neural network since the error generated by this same neural network is non-zero.

What are the advantages of E-TUNI with backward integration compared to other conventional numerical integration methods? Since E-TUNI uses a neural network, its accuracy is equivalent to any other higher-order numerical integrator. This property is because artificial neural networks are universal approximators of functions [2,3,4,5]; that is, the training error of the neural network can be as small as desired. The other advantage of E-TUNI is that it has a mathematical structure only of the first-order, which means it is much simpler than higher-order integrators. The disadvantage of E-TUNI, concerning higher-order integrators, is that the former has a fixed step, and the latter has a variable step. This difference means that if it is desired to vary the integration step of E-TUNI, a new neural training will be necessary. Furthermore, as previously mentioned, E-TUNI can work with real-world data and not only with theoretical models.

How does the algorithm guarantee accuracy and stability when finding the endpoints of the integration interval during the backward integration process? In this paper, we only empirically verify that this is possible through the practical examples presented in the numerical results section at the end of this paper. However, a convergence proof of the numerical stability of the proposed model is left for future work. However, here, we prove that the general expression of E-TUNI (which works with mean derivative functions) is an exact solution for a system of Ordinary Differential Equations (ODEs) and as demonstrated by Theorem 4.

Is this valuable method for dynamical systems of nonlinear differential equations of any dimension n? As exemplified by Equations (1)–(3), E-TUNI is capable of solving a non-linear dynamic modelling problem of any order. However, it seems reasonable to assume that the larger the dimension n of the proposed problem, the greater the computational effort required to train the neural network used. This article exemplifies its technological application with physics problems of up to four state variables.

Does the E-TUNI model have application value for new problems in a specific discipline such as Physics or Engineering? Yes, this method can be applied to any physical problem governed by an autonomous non-linear ordinary differential equation with n state variables and m control variables. However, its use in partial differential equations requires further studies. A good application of the proposed method would be, for example, forecasting the stock market, forecasting river flows in hydro-graphic basins, control problems applied to aerospace engineering, among others. The E-TUNI models for forward and backward integration can also be used in optimal control to solve problems involving Pontryagin’s principle. Additionally, the E-TUNI can be an alternative to the traditional NARMAX model and Recurrent Neural Networks (RNNs). Finally, Figure 3 graphically schematizes the coupling of an E-TUNI model to any real-world plant, which can be learned through supervised learning with input/output training patterns.

4. Universal Numerical Integrator (UNI)

Figure 3 illustrates a general scheme for adequately designing a Universal Numerical Integrator (UNI). Notice that in Figure 3, when one couples a conventional numerical integrator (e.g., Euler, Runge–Kutta, Adams-Bashforth, predictive-corrector, among others) to some kind of universal approximators of functions (e.g., MLP neural networks, RBF, SVM, fuzzy inference systems, among others), the UNI design can be achieved high numerical accuracy through supervised learning using input/output training patterns. This allows for solving real-world problems involving the modeling of non-linear dynamic systems.

Still, regarding Figure 3, note that ${}^{k+1}\widehat{y}_j^i$ is the prediction made at instant $t_{k+1}$ by the UNI used and ${}^{k+1}y_j^i$ is the real value generated by the plant at the same instant. Note that these two values are compared in the “Training Error” block, and the weights present in the UNI are updated through supervised learning with input/output training patterns. Since artificial neural networks are considered universal approximators of functions, the neural training error can be as small as desired. Notice that ${}^{k+1}\tilde{y}_j^i$ is the output generated if only the conventional numerical integrator is used. However, in the latter case, training it with real-world training patterns is impossible.

As previously mentioned, a Universal Numerical Integrator (UNI) can be of three types, thus giving rise to three distinct methodologies, namely: (i) NARMAX model, (ii) instantaneous derivatives methodology (e.g., Runge–Kutta neural network, Adams-Bashforth neural network, predicted-corrector neural network, among others), and (iii) mean derivatives methodology or E-TUNI. The three types of universal numerical integrators are equally precise, as the Artificial Neural Network (ANN) is a universal approximator of functions [2,3,4,5], every UNI can have its approximation error as close to zero, as much as one likes.

To fully understand the difference between the instantaneous and mean derivative methodologies, see Figure 2 again. This figure presents the geometric difference between the instantaneous and mean derivative functions. It is well-known that the mean derivative ($ta{n}_{\Delta t}{}^k\alpha ^i$) is the tangent of the angle of the secant line connecting two distinct points of a mathematical function. On the other hand, the instantaneous derivative ($ta{n}^k{\theta }^i$) is the tangent of the angle of the tangent line to any given point of a mathematical function.

The mean derivatives methodology occurs when a universal approximator of functions is coupled to the Euler-type first-order integrator. In this case, the first-order integrator would have a substantial error if instantaneous derivative functions were used. However, when the mean derivative functions replace the instantaneous derivative functions, the integrator error is reduced to zero [16,27]. In this way, it is possible to train an artificial neural network to learn the mean derivative functions through two possible approaches [16]: (i) the direct approach and (ii) the indirect or empirical approach.

Figure 4 schematically represents the direct approach. Figure 5 schematically represents the indirect or empirical approach. In the direct approach, the artificial neural network is trained separately from the first-order integrator. The artificial neural network is trained and coupled to the first-order integrator in the indirect or empirical approach. Both approaches are equivalent to each other. However, the back-propagation algorithm must be recalculated using the indirect or empirical approach. In the direct approach, this is unnecessary [16].

However, in this article, only the direct approach will be used to train the neural network with the mean derivative functions. In this sense, giving a more detailed explanation of Figure 4 is useful. The “Dynamic System” block is the state equations of the dynamic system that we want to reproduce through E-TUNI. A higher-order numerical integrator solves the theoretical dynamical system (e.g., Runge–Kutta 4-5, among others). After this, the values ${}^{k}y^i$ and ${}^{k+1}y^i$ are used to calculate the mean derivative of this theoretical system, i.e., $ta{n}_{\Delta t}{}^k\alpha ^i$. In parallel, the “NN” block contains the neural network. The neural network inputs will be ${}^{k}y^i$ (state variable) and ${}^{k}u$ (control variable), calculated at time $t_k$. The output of the neural network will be the mean derivative $ta{n}_{\Delta t}{}^k\widehat{\alpha }^i$. Notice that the same inputs ${}^{k}y^i$ and ${}^{k}u$ are simultaneously stimulated in the original theoretical dynamical system and the neural network. Thus, after the dynamical system is adequately trained, the output variable of the neural network ($ta{n}_{\Delta t}{}^k\widehat{\alpha }^i$) should be the same as the output variable of the original dynamical system ($ta{n}_{\Delta t}{}^k\alpha ^i$), at least for a small residual error. Furthermore, if we want to learn a real-world plant, we must only replace the theoretical model with a computational data acquisition system.

The same explanation applies to Figure 5. However, since the integrator is trained with the neural network in this case, the reference value in training will not be the mean derivative functions but an instant future ahead of state variables, i.e., ${}^{k+1}y^i$.

Additionally, in the remainder of this article, although the dynamical system can be used with both state variables and control variables, we will consider dynamical systems with only state variables. It is also essential to note that in the E-TUNI, the integration step ($\Delta t$) can be any (it does not need to be infinitesimal). It should be small only in the case where control variables are used.

This difference between the direct and indirect approach also applies to the instantaneous derivatives methodology. However, in this case, the direct approach only applies to theoretical models, as instantaneous derivative functions cannot be determined directly in real-world problems [16]. Furthermore, this difference between the direct and indirect approach does not apply to the NARMAX model. However, the NARMAX model is also a particular case of UNI. This is because, although a conventional integrator is not coupled to the NARMAX model, the universal approximator of functions behaves as if it were a numerical integrator [16].

5. The Euler-Type Universal Numerical Integrator (E-TUNI)

Firstly, in this section, we briefly introduce the Euler-Type Universal Numerical Integrator (E-TUNI) and present the general expression of this unique type of Universal Numerical Integrator (UNI). Next, we present the mathematical foundations of E-TUNI with forward and backward integrations. It is also important to highlight that this integrator is inherently of first-order.

As a result, Equation (1) mathematically represents a system of n autonomous non-linear ordinary differential equations with dependent variables $y_1,y_2,\dots ,y_n$. Based on this, it is possible to establish an initial value problem for a first-order system, also described by (1):

$$\left\{\begin{array}{cc}{\dot{y}}_1=f_1(y_1,y_2,\dots y_n),& y_1\left(t_0\right)={\eta }_1\\ {\dot{y}}_2=f_2(y_1,y_2,\dots y_n),& y_2\left(t_0\right)={\eta }_2\\ ⋮& ⋮\\ {\dot{y}}_n=f_n(y_1,y_2,\dots y_n),& y_n\left(t_0\right)={\eta }_n\end{array}\right.$$

(1)

The system represented by Equation (1) is a non-linear and autonomous (time-invariant) system of ordinary differential equations. It is important to notice that the method presented here will be limited to this type of equation. Thus, neither non-autonomous systems nor Partial Differential Equations (PDEs) will be considered here.

Additionally, the solution of the differential Equation (1) can be obtained through a first-order approximation. Such a solution was obtained by the mathematician Leonard Euler in 1768. This solution, keeping the convention described by Figure 2, is given by Equation (2):

$$\left\{\begin{array}{cc}{}^{k+1}y_1^i\cong tan{}^k\theta _1^i·\Delta t+{}^{k}y_1^i,& {}^{0}y_1^i={\eta }_1\\ {}^{k+1}y_2^i\cong tan{}^k\theta _2^i·\Delta t+{}^{k}y_2^i,& {}^{0}y_2^i={\eta }_2\\ ⋮& ⋮\\ {}^{k+1}y_n^i\cong tan{}^k\theta _n^i·\Delta t+{}^{k}y_n^i,& {}^{0}y_n^i={\eta }_n\end{array}\right.$$

(2)

However, it is essential to notice that the accuracy of Equation (2) is awful. Because of this, if the mean derivative functions replace the instantaneous derivative functions, then the Euler-type first-order integrator becomes precise (see demonstration of Theorem 4). This new solution is given by Equation (3):

$$\left\{\begin{array}{cc}{}^{k+1}y_1^i=ta{n}_{\Delta t}{}^k\alpha _1^i·\Delta t+{}^{k}y_1^i,& {}^{0}y_1^i={\eta }_1\\ {}^{k+1}y_2^i=ta{n}_{\Delta t}{}^k\alpha _2^i·\Delta t+{}^{k}y_2^i,& {}^{0}y_2^i={\eta }_2\\ ⋮& ⋮\\ {}^{k+1}y_n^i=ta{n}_{\Delta t}{}^k\alpha _n^i·\Delta t+{}^{k}y_n^i,& {}^{0}y_n^i={\eta }_n\end{array}\right.$$

(3)

for

$$ta{n}_{\Delta t}{}^k\alpha _j^i={\displaystyle \frac{{}^{k+1}y_j^i-{}^{k}y_j^i}{\Delta t}}$$

(4)

Note that Equation (3) is straightforward to obtain, as the mean derivative functions sound easily computable, for real-world problems, via Equation (4), for $j=1,2,\dots ,n$. Here, the variable n is the total number of state variables of the dynamical system given by (1).

Before we proceed with the rest of this article, we will give an intuitive justification for the passage from Equation (2) to Equation (3), with the help of Figure 2. If an artificial neural network is trained to learn the instantaneous derivative function ($ta{n}^k{\theta }^i$) of any non-linear dynamical system and this neural network is subsequently coupled to the first-order Euler integrator, then the output of this model will be ${}^{k+1}\tilde{y}_j^i$ (see Figure 2 again). Therefore, this value is very different from the real value ${}^{k+1}y_j^i$. However, if the neural training on the Euler integrator, mentioned above, is continued through the indirect approach, its training error will be reduced to almost zero, since the neural network is a universal approximator of functions. Therefore, the new output will be ${}^{k+1}\widehat{y}_j^i$. Notice that graphically, the instantaneous derivative function has been forced to converge to the mean derivative function to reduce the integrator error to almost zero. Additionally, by the uniqueness theorem (Theorem 1 which will be presented later) there is no other way for this problem to converge.

Finally, it is crucial to notice that a neural network can compute the mean derivative functions, as schematized by Figure 6. In this way, the E-TUNI works with a first-order numerical integrator coupled to a neural network with any feed-forward architecture (e.g., MLP neural network, RBF, SVM, wavelets, fuzzy inference systems, Convolutional Neural Network (CNN), among others).

5.1. Fundamental Theorem

In this section, the general expression of E-TUNI is demonstrated. For this purpose, let the system of non-linear differential Equations (5) be:

$$\dot{y}=f\left(y\right)$$

(5)

where, $y=$ ${[y_1{y}_2\dots y_n]}^T$ and $f\left(y\right)=$ ${[f_1\left(y\right)f_2\left(y\right)\dots f_n\left(y\right)]}^T$. The uniqueness theorem is a fundamental theorem about Equation (5).

Theorem 1

(T1). Assume that each of the functions $f_1(y_1,y_2,\dots ,y_n),\dots ,f_n(y_1,y_2,\dots ,y_n)$ has continuous partial derivatives with respect to $y_1,y_2,\dots ,y_n$. So the known initial value problem $\dot{y}=f\left(y\right)$ for $y\left(t_0\right)$ has one and only one solution $y=y\left(t\right)$ in $\mathbb{R}$, starting from each $y\left(t_0\right)$. If two solutions $y=\varphi \left(t\right)$ have a common point, they must be identical.

For the formal demonstration of the general expression of E-TUNI, which will numerically and discretely solve Equation (5), two hypotheses will be assumed: (i) a set of points that is an exact solution of the dynamical system (5) is known and which we will call ${}^{k}y_j^i$ and (ii) the existence of two sets of continuous and differentiable functions that pass through these points is assumed, namely, $y_j^i\left(t\right)$ and ${\dot{y}}_j^i\left(t\right)$. Figure 7 mathematically illustrates these two hypotheses when $i=1,2,3,4$, $j=1,2$, and $k=0,1,2,3,4$. The meanings of the auxiliary variables i, j, and k are the same as those described in Figure 1. Furthermore, only the points ${}^{k}y_j^i$ must be known. The functions $y_j^i\left(t\right)$ and ${\dot{y}}_j^i\left(t\right)$ just need to exist; that is, it is not necessary to know them.

What do the matrices mean in Figure 7? The first line of these point matrices represents a set of exact points belonging to the first curve of the family of solution curves of the considered system, and so on. For example, all points ${}^{0}y_1^1$, ${}^{1}y_1^1$, ${}^{2}y_1^1$, ${}^{3}y_1^1$, and ${}^{4}y_1^1$ pass through the continuous and differentiable function $y_1^1\left(t\right)$, and so on. Since $i=4$, the family of curves, in this case, is composed of four curves. Based on the continuous hypothesis, i can be as large as one wants. Since the neural network, responsible for modelling the system (5), will be designed with the supervised learning approach through input/output training patterns, knowing in advance the discrete solution of the system (5), is not absurd. Thus, the system (5) can be replaced by the following set of solutions (6):

$$\left\{\begin{array}{c}{\dot{y}}_j^i\left(t\right)={\dot{g}}_j^i\left(t\right)\\ y_j^i\left(t\right)=g_j^i\left(t\right)\end{array}\right.$$

(6)

where $i=1,2,\dots ,q$; $j=1,2,\dots ,n$; $k=0,1,2,\dots ,r$. It is essential to note that changing the values of q, n, r, and $\Delta t$ can obtain any finite density of discrete point solutions of (5) over a finite domain. Thus, the great advantage of Equation (6) concerning Equation (5) is that the former is an explicit function only of the variable t. Thus, the differential and integral mean value theorems can be applied in (6) to arrive at the general expression of E-TUNI, which uses the functions of mean derivatives. The differential and integral mean value theorems cannot be applied in (5) because, in this case, the system in question is an explicit function of y and not of t.

Theorem 2

(The Differential Mean Value Theorem). If a function $g_j^i\left(t\right)$ for $j=1,2,\dots ,n$ is a continuous function and defined over the closed interval $[t_k,t_{k+1}]$ is differentiable over the open interval $(t_k,t_{k+1})$, then there exists at least one number $t_k^{*}$ with $t_k<t_k^{*}<t_k+\Delta t=t_{k+1}$ such that, ${\dot{g}}_j^i\left(t_k^{*}\right)={\displaystyle \frac{{}^{k+1}g_j^i-{}^{k}g_j^i}{\Delta t}}$.

Theorem 3

(The Integral Mean Value Theorem). If a function ${\dot{g}}_j^i\left(t\right)$ for $j=1,2,\dots ,n$ is a continuous function and defined over the closed interval $[t_k,t_{k+1}]$, then there exists at least one internal point $t_k^x$ in $[t_k,t_{k+1}]$ such that, ${\dot{g}}_j^i\left(t_k^x\right)·\Delta t={\int }_{t_k}^{t_{k+1}}{\dot{g}}_j^i\left(t\right)dt$.

The notation used to represent ${}^{k}y_j^i$ is sufficient to map it uniquely to its respective region of interest, where its associated characteristic secant $ta{n}_{\Delta t}{}^k\alpha _j^i$ is confined. Thus, it can be stated that for the state ${}^{k}y_j^i=y_j^i(t_0+k·\Delta t)$, we have a single characteristic secant associated with it and given by $ta{n}_{\Delta t}{}^k\alpha _j^i$. The existence of a unique secant associated with each point ${}^{k}y_j^i$ is an immediate consequence of the uniqueness theorem (Theorem 1).

Consider also $y_j^i=y_j^i\left(t\right)$ for $j=1,2,$ $\dots ,n$ a particular trajectory of a family of solutions of the system of differential equations $\dot{y}=f\left(y\right)$ passing through $y_j^i\left(t_0\right)$ at time $t_0$, i.e., initializing within a finite domain of interest $[y_j^{min}\left(t_0\right),$ $y_j^{max}\left(t_0\right){]}^n$.

In [29,30], by definition, the secant curve between two points ${}^{k}y_j^i$ and ${}^{k+1}y_j^i,$ belonging to the curve $y_j^i\left(t\right)$ for $j=1,2,\dots ,n$ is the straight line segment that joins these two points. Thus, the tangents of the secants between the points ${}^{k}y_1^i$ and ${}^{k+1}y_1^i$, ${}^{k}y_2^i$ and ${}^{k+1}y_2^i$, …, ${}^{k}y_n^i$ and ${}^{k+1}y_n^i$ are defined as:

$$\begin{gathered} ta{n}_{\Delta t}{\alpha }^i(t+k·\Delta t)=ta{n}_{\Delta t}{}^k\alpha ^i= \\ {\left[\begin{array}{cccc}ta{n}_{\Delta t}{}^k\alpha _1^i& ta{n}_{\Delta t}{}^k\alpha _2^i& \dots & ta{n}_{\Delta t}{}^k\alpha _n^i\end{array}\right]}^T \end{gathered}$$

(7)

where $ta{n}_{\Delta t}{}^k\alpha _j^i={\displaystyle \frac{{}^{k+1}y_j^i-{}^{k}y_j^i}{\Delta t}}$ for $j=1,2,·,n$.

Property 1.

Applying Theorem 2 on the curve ${\dot{g}}_j^i\left(t\right)$ is equivalent to applying Theorem 1 on the curve $g_j^i\left(t\right)$ both on the same interval closed $[t_k,t_{k+1}]$, i.e., ${\dot{g}}_j^i\left(t_k^x\right)={\dot{g}}_j^i\left(t_k^{*}\right)$.

Proof. 

If one applies Theorem 2 to the continuous curve ${\dot{g}}_j^i\left(t\right)$, this results in a ${\dot{g}}_j^i\left(t_k^x\right)$ for $t_k<t_k^x<t_{k+1}$ such that ${\dot{g}}_j^i\left(t_k^x\right)·\Delta t=$ ${\int }_{t_k}^{t_{k+1}}{\dot{g}}_j^i\left(t\right)dt=$ ${}^{k+1}g_j^i-{}^{k}g_j^i$ as a result of the fundamental theorem of calculus. This way, ${\dot{g}}_j^i\left(t_k^x\right)=$ ${\displaystyle \frac{{}^{k+1}g_j^i-{}^{k}g_j^i}{\Delta t}}\stackrel{\mathrm{def}}{=}$ $ta{n}_{\Delta t}{}^k\alpha _j^i$. On the other hand, the application of Theorem 1 on the continuous and differentiable curve $g_j^i\left(t\right)$ implies the existence of a ${\dot{g}}_j^i\left(t_k^{*}\right)$ for $t_k<t_k^x<t_{k+1}$, such that ${\dot{g}}_j^i\left(t_k^{*}\right)=$ ${\displaystyle \frac{{}^{k+1}g_j^i-{}^{k}g_j^i}{\Delta t}}\stackrel{\mathrm{def}}{=}$ $ta{n}_{\Delta t}{}^k\alpha _j^i$. This way, ${\dot{g}}_j^i\left(t_k^x\right)=$ ${\dot{g}}_j^i\left(t_k^{*}\right)$. However, this demonstration does not prove that $t_k^x=t_k^{*}$. □

Theorem 4.

The general discrete and exact solution ${}^{k+1}y_j^i$ for $j=1,2,\dots ,n$ of the autonomous system of nonlinear differential equations of the type ${\dot{y}}^i=f\left(y^i\right)$ can be established through the first-order Euler relation of the type ${}^{k+1}y_j^i=$ $ta{n}_{\Delta t}{}^k\alpha _j^i·\Delta t$ $+{}^{k}y_j^i$ for a given ${}^{k}y_j^i$ and $\Delta t$ fixed, since the general solution of this dynamical system, given by, $y_j^i\left(t\right)=g_j^i\left(t\right)$ and ${\dot{y}}_j^i\left(t\right)={\dot{g}}_j^i\left(t\right)$ are previously known for $j=1,2,\dots ,n$; $i=1,2,\dots ,\infty$ and tϵ$[t_o,t_f]$.

Proof. 

Let the first-order non-linear and autonomous dynamical system be given by ${\dot{y}}_j^i\left(t\right)=$ $f_j(y_1^i,y_2^i,\dots ,y_n^i)$ $=f_j\left(y^i\right)$ for $j=1,2,\dots ,n$ and $i=1,2,\dots ,\infty$. If the solution of the dynamical system is known and equal to $y_j^i\left(t\right)=$ $g_j^i\left(t\right)$, then the function $f_j\left(y^i\right)$ can be replaced by ${\dot{g}}_j^i\left(t\right)=$ $h_j^i\left(t\right)$, i.e., ${\dot{y}}_j^i\left(t\right)=h_j^i\left(t\right)$. In this way, we can write that ${\int }_{{}^{k}y_j^i}^{{}^{k+1}y_j^i}d{y}_j^i\left(t\right)=$ ${\int }_{t_k}^{t_{k+1}}{h}_j^i\left(t\right)dt$. Note that the indices i, j, and k uniquely map the secant we are interested in and in accordance with Figure 1. This last integral can still be simplified as ${}^{k+1}y_j^i-{}^{k}y_j^i=$ ${\int }_{t_k}^{t_{k+1}}{h}_j^i\left(t\right)dt$. Thus, applying the mean integral value theorem, in this last expression, we obtain ${}^{k+1}y_j^i-{}^{k}y_i^j=$ $h_j^i\left(t_k^x\right)·\Delta t=$ ${\dot{g}}_j^i\left(t_k^x\right)·\Delta t$ where $t_k^x$ $ϵ$ $[t_k,t_{k+1}]$ and for $h_j^i\left(t\right)=$ ${\dot{g}}_j^i\left(t\right)$. On the other hand, applying the differential mean value theorem on the function $y_j^i\left(t\right)=$ $g_j^i\left(t\right)$ for $j=1,2,\dots ,n$ in the interval $[t_k,t_{k+1}]$ we obtain that ${\dot{g}}_j^i\left(t_k^{*}\right)=$ ${\displaystyle \frac{{}^{k+1}g_j^i-{}^{k}g_j^i}{\Delta t}}$ $\stackrel{\mathrm{def}}{=}ta{n}_{\Delta t}{}^k\alpha _j^i$. Thus, by Property 1 we have that ${\dot{g}}_j^i\left(t_k^{*}\right)=$ ${\dot{g}}_j^i\left(t_k^x\right)=$ $ta{n}_{\Delta t}{}^k\alpha _j^i$ for $j=1,2,\dots ,n$. Therefore, ${}^{k+1}y_j^i=$ $ta{n}_{\Delta t}{}^k\alpha _i^j·\Delta t$ $+{}^{k}y_j^i$ for $j=1,2,\dots ,n$ or, in vector form, it turns out that ${}^{k+1}y^i=$ $ta{n}_{\Delta t}{}^k\alpha ^i·\Delta t$ $+{}^{k}y^i$. □

Note that the functions ${\dot{y}}_j^i\left(t\right)$ and $y_j^i\left(t\right)$, used in the previous demonstration, need to be continuous and differentiable because of Theorems 2 and 3, but they do not need to be known. Their existence is enough. What needs to be known are the points ${}^{k}y_j^i$. Finally, through a mathematical demonstration, very similar to Theorem 4, one can demonstrate the general expression of E-TUNI with backward integration, i.e., ${}^{k-1}y_j^i=-ta{n}_{\Delta t}^{-}{}^k\alpha _j^i.\Delta t{+}^k{y}_j^i$ for $ta{n}_{\Delta t}^{-}{}^k\alpha _j^i={\displaystyle \frac{{}^{k}y_j^i-{}^{k-1}y_j^i}{\Delta t}}$ and $j=1,2,\dots ,n$. For more details on E-TUNI and its relationship with other types of UNIs, see [16,31].

5.2. General E-TUNI Algorithm with Forward and Backward Integrations

This section describes first-order Euler integrators with forward and backward integration. They are divided into two classes: (i) the conventional Euler integrator, which works with instantaneous derivative functions (low numerical precision and variable step), and (ii) the E-TUNI type integrator, which works with mean derivative functions (high numerical precision and fixed step).

Therefore, it is important to mathematically relate the mean derivatives functions with the instantaneous derivatives functions (see again Figure 2). Thus, it is easily seen that if $\dot{y}=f\left(y\right)$ then,

$$\begin{array}{c}\hfill {}^k\dot{y}_i^i=f\left({}^{k}y^i\right)=\underset{\Delta t\to 0}{lim}ta{n}_{\Delta t}{}^{k}y_j^i=\underset{\Delta t\to 0}{lim}{\displaystyle \frac{{}^{k+1}y_j^i-{}^{k}y_j^i}{\Delta t}}\\ \hfill forj=1,2,\dots ,n\end{array}$$

(8)

or

$$\begin{array}{c}\hfill \underset{\Delta t\to 0}{lim}ta{n}_{\Delta t}{}^{k}y_j^i=ta{n}^k{\mathrm{\Theta }}_j^i\\ \hfill forj=1,2,\dots ,n\end{array}$$

(9)

Figure 2 graphically illustrates the convergence of mean derivatives functions to instantaneous derivatives functions when $\Delta t$ tends to zero. Based on this, it is possible to enunciate two basic types of first-order numerical integrators or Euler integrators. An Euler integrator that uses instantaneous derivatives functions, given by $ta{n}^k{\mathrm{\Theta }}_j^i=f_j\left({}^{k}y^i\right)$ for $j=1,2,\dots ,n$ and an Euler integrator that uses mean derivatives functions, given by $ta{n}_{\Delta t}^k{y}_j^i={\displaystyle \frac{{}^{k+1}y_j^i-{}^{k}y_j^i}{\Delta t}}=g_j[{}^{k}y^i{,}^{k+1}{y}^i,\Delta t]$.

A significant difference between the mean derivatives functions and the instantaneous derivatives functions is that the instantaneous derivatives functions $f_j\left({}^{k}y^i\right)$ for $j=1,2,\dots ,n$ is only dependent on ${}^{k}y^i$, but the mean derivatives functions $g_j[{}^{k}y^i{,}^{k+1}{y}^i,\Delta t]$ for $j=1,2,\dots ,n$ is dependent on ${}^{k}y^i$, ${}^{k+1}y^i$ and $\Delta t$ at the point ${}^{k}y^i$. As will be seen later, this difference is significant.

The first important point to be considered here is the possibility of carrying out both forward and backward integration of the Euler Integrator with instantaneous derivatives. So, the instantaneous derivative functions at the points $y_j^i\left(t\right)$ for $j=1,2,\dots ,n$ can be defined in two different ways:

$${}^k\dot{y}_j^i=f_j^{+}\left({}^{k}y^i\right)=\underset{\Delta t\to 0}{lim}{\displaystyle \frac{{}^{k+1}y_j^i-{}^{k}y_j^i}{\Delta t}}=ta{n}^{+}{}^k\theta _j^i$$

(10)

or

$${}^k\dot{y}_j^i=f_j^{-}\left({}^{k}y^i\right)=\underset{\Delta t\to 0}{lim}{\displaystyle \frac{{}^{k}y_j^i-{}^{k-1}y_j^i}{\Delta t}}=ta{n}^{-}{}^k\theta _j^i$$

(11)

However, if the instantaneous derivatives functions $f_j(.)$ exist at the point ${}^{k}y^i$, then necessarily $f_j^{+}\left({}^{k}y^i\right)=f_j^{-}\left({}^{k}y^i\right)=f_j\left({}^{k}y^i\right)$ or $ta{n}^{+}{}^k\theta _j^i=ta{n}^{-}{}^k\theta _j^i=ta{n}^k{\theta }_j^i$ (see Figure 8). In this way, the expressions of forward and backward propagating Euler integrators, necessarily designed with the instantaneous derivative functions, are given, respectively, by (i.e., with low numerical precision):

$$\begin{array}{c}\hfill {}^{k+1}y_j^i\approx f_j\left({}^{k}y^i\right)·\Delta t{+}^k{y}_j^i=\\ \hfill {}^{k+1}y_j^i\approx tan{}^k\theta _j^i·\Delta t{+}^k{y}_j^i=\end{array}$$

(12)

$$\begin{array}{c}\hfill {}^{k-1}y_j^i\approx -f_j\left({}^{k}y^i\right)·\Delta t{+}^k{y}_j^i=\\ \hfill {}^{k-1}y_j^i\approx -tan{}^k\theta _j^i·\Delta t{+}^k{y}_j^i=\end{array}$$

(13)

Note that Equation (13) can also be obtained directly from Equation (12) by making the substitution of the variable $\Delta t=-\Delta t$.

Additionally, like $f_j^{+}\left({}^{k}y^i\right)=f_j^{-}\left({}^{k}y^i\right)=$ $f_j\left({}^{k}y^i\right)$, the Euler integrator with instantaneous derivatives functions has a symmetric solution between intervals $[t_{k-1},t_k]$ and $[t_k,t_{k+1}]$, thus implying low numerical precision, for the outputs ${}^{k-1}y_j^i$ and ${}^{k+1}y_j^i$ for $j=1,2,\dots ,n$. To say that a solution is symmetric at the point k means to say that ${|}^k{y}_j^i-{}^{k-1}y_j^i|=$ ${|}^{k+1}{y}_j^i-{}^{k}y_j^i|$ to $j=1,2,\dots ,n$.

This Euler integrator symmetry property only occurs when instantaneous derivative functions are used. However, in Euler’s integrator, designed with the mean derivatives functions, this symmetry of the solution around the point $t_k$ is not verified (see Figure 8 again). This is important because in Euler’s integrator, designed with mean derivatives functions, the backward solution does not necessarily decrease when the forward solution grows.

In [32], several difference operators can be applied to a function ${}^{k}y_j^I$ for $j=1,2,\dots ,n$. However, the two operators that interest us are the forward difference and the backward difference, given below, respectively:

$$\begin{array}{c}\hfill {\Delta }^k{y}_j^i={}^{k+1}y_j^i-{}^{k}y_j^{i}forj=1,2,\dots ,n\end{array}$$

(14)

$$\begin{array}{c}\hfill {\nabla }^k{y}_j^i={}^{k}y_j^i-{}^{k-1}y_j^{i}forj=1,2,\dots ,n\end{array}$$

(15)

Thus, using the finite difference operators, there are two different ways to represent the secant at the point ${}^{k}y_j^i$ and with integration step $\Delta t$:

$$\begin{array}{c}\hfill ta{n}_{\Delta t}^{+}{}^k\alpha _j^i={\displaystyle \frac{{\Delta }^k{y}_j^i}{\Delta t}}={\displaystyle \frac{{}^{k+1}y_j^i-{}^{k}y_j^i}{\Delta t}}\\ \hfill forj=1,2,\dots ,n\end{array}$$

(16)

$$\begin{array}{c}\hfill ta{n}_{\Delta t}^{-}{}^k\alpha _j^i={\displaystyle \frac{{\nabla }^k{y}_j^i}{\Delta t}}={\displaystyle \frac{{}^{k}y_j^i-{}^{k-1}y_j^i}{\Delta t}}\\ \hfill forj=1,2,\dots ,n\end{array}$$

(17)

Comparing Equation (16) with the result of Theorem T4, one can obtain the following recursive and exact equation to represent the solution of autonomous non-linear ordinary differential equations system for a forward integration process (i.e., with high numerical accuracy):

$$\begin{array}{c}\hfill {}^{k+1}y_j^i=ta{n}_{\Delta t}^{+}{}^k\alpha _j^i·\Delta t{+}^k{y}_j^i\\ \hfill forj=1,2,\dots ,n\end{array}$$

(18)

or

$$\begin{array}{c}\hfill {}^{k+1}y_j^i={\displaystyle \frac{{\Delta }^k{y}_j^i}{\Delta t}}·\Delta t{+}^k{y}_j^i\\ \hfill forj=1,2,\dots ,n\end{array}$$

(19)

Equations (18) and (19) represent an Euler-type integrator, which uses mean derivatives functions and forward integration. However, to obtain the Euler integrator with backward recursion, it is enough to analyze the asymmetry of the mean derivative functions at the point ${}^{k}y_j^i$ and as is illustrated in Figure 8. In this way, we can obtain the following equation to perform the backward integration of E-TUNI (i.e., with high numerical precision):

$$\begin{array}{c}\hfill {}^{k-1}y_j^i=-ta{n}_{\Delta t}^{-}{}^k\alpha _j^i·\Delta t{+}^k{y}_j^i\\ \hfill forj=1,2,\dots ,n\end{array}$$

(20)

or

$$\begin{array}{c}\hfill {}^{k-1}y_j^i=-{\displaystyle \frac{{\nabla }^k{y}_j^i}{\Delta t}}·\Delta t{+}^k{y}_j^i\\ \hfill forj=1,2,\dots ,n\end{array}$$

(21)

Thus, the iterative equations of the first-order Euler integrator, using mean derivatives, which perform forward and backward recursions, are, respectively, Equations (18) and (20). The negative sign that appears in Equation (20) means, not necessarily, that when ${}^{k+1}y_j^i$ increases, then ${}^{k-1}y_j^i$ decreases, because they are two neural networks independent of each other. Therefore, it is observed that, in general, $|ta{n}_{\Delta t}^{+}{}^k\alpha _j^i|\ne |ta{n}_{\Delta t}^{-}{}^k\alpha _j^i|$, and therefore Euler integrators with mean derivatives functions do not have a symmetric solution, i.e., in general, $|{}^{k+1}y_j^i-{}^{k}y_j^i{|\ne |}^k{y}_j^i-{}^{k-1}y_j^i|$ for $j=1,2,\dots ,n$. As can be easily seen in Figure 8, the negative mean derivative has a different slope than the positive mean derivative for the same point in $t_k$.

It is interesting to note that the negative sign that appears in Equations (13), (20) and (21) is because of the inverted direction of the flow of time (from present to past) in backward integration.

Additionally, the following algorithm is proposed to determine the mean derivatives through the direct methodology involving both forward and backward integrations in the Universal Numerical Integrator (UNI) of the E-TUNI type. This approach requires five steps.

1. Given a finite domain of interest $[{}^{k}y_j^{min},$ ${}^{k}y_j^{max}{]}^n$ computed at time $t_k$ for $j=1,2,\dots ,n$ state variables, generate, according to a uniform probability distribution, q initial conditions such that,

$$p^i={\left[\begin{array}{ccccc}{}^{k}y_1^i& {}^{k}y_2^i& {}^{k}y_3^i& \dots & {}^{k}y_n^i\end{array}\right]}^T$$

and

$$P={\left[\begin{array}{ccccc}{p}^1:& p^2:& p^3:& \dots :& p^q\end{array}\right]}_{nxq}$$

which are the E-TUNI’s input training patterns at time $t_k$ for both the forward and backward integrators. The number q of training patterns should be large enough to ensure adequate supervised training. To this end, the training patterns should be appropriately divided, with one portion for training and another for testing.

2. Using a high-order integrator, propagate all initial conditions $p^i$ to obtain the corresponding states at time $t_{k+1}$ and at time $t_{k-1}$. Thus, assemble the output patterns ${}^{+}T^i$ and ${}^{-}T^i$ to train the positive and negative mean derivatives.

$${}^{+}T^i={\left[\begin{array}{ccccc}ta{n}_{\Delta t}^{+}{}^k\alpha _1^i& ta{n}_{\Delta t}^{+}{}^k\alpha _2^i& ta{n}_{\Delta t}^{+}{}^k\alpha _3^i& \dots & ta{n}_{\Delta t}^{+}{}^k\alpha _n^i\end{array}\right]}^T$$

$${}^{+}T={\left[\begin{array}{ccccc}{}^{+}T^1:& {}^{+}T^2:& {}^{+}T^3:& \dots :& {}^{+}T^q\end{array}\right]}_{nxq}$$

and

$${}^{-}T^i={\left[\begin{array}{ccccc}ta{n}_{\Delta t}^{-}{}^k\alpha _1^i& ta{n}_{\Delta t}^{-}{}^k\alpha _2^i& ta{n}_{\Delta t}^{-}{}^k\alpha _3^i& \dots & ta{n}_{\Delta t}^{-}{}^k\alpha _n^i\end{array}\right]}^T$$

$${}^{-}T={\left[\begin{array}{ccccc}{}^{-}T^1:& {}^{-}T^2:& {}^{-}T^3:& \dots :& {}^{-}T^q\end{array}\right]}_{nxq}$$

Note: Alternatively, in this step, a data acquisition system could capture real-world dynamic systems’ behavior.

3. Once input vectors P and output vectors ${}^{+}T$ and ${}^{-}T$ are required by the universal approximator of functions, train two neural networks through supervised learning. One training pair $\left[P{,}^{+}T\right]$ to learn the positive mean derivative function and one training pair $\left[P{,}^{-}T\right]$ to learn the negative mean derivative function. These two neural networks will look similar to the one in Figure 6.

4. When the two supervised trainings are consolidated, it is possible to simulate the dynamics of the system through the following discrete recursive expressions:

$${}^{k+1}\widehat{y}^i=ta{n}_{\Delta t}^{+}{}^k\widehat{\alpha }^i·\Delta t{+}^k{y}^i$$

and

$${}^{k-1}\widehat{y}^i=-ta{n}_{\Delta t}^{-}{}^k\widehat{\alpha }^i·\Delta t{+}^k{y}^i$$

5. Using the E-TUNI expression to propagate the positive mean derivative, integrate the dynamical system from the initial instant $t_o$ to the final instant $t_f$. Then, integrate the E-TUNI expression from $t_f$ to $t_o$ using the negative mean derivative. By the uniqueness theorem, the initial starting point of the first integration must coincide with the final arrival point of the second integration.

6. Analysis of the Numerical Accuracy of E-TUNI Using Landau Symbols

In this section, an analysis of the accuracy of the E-TUNI solution is performed. For this purpose, the numerical accuracy of the conventional Euler method, which works with instantaneous derivative functions, is initially analyzed. Next, an analysis of the accuracy of the E-TUNI solution, i.e., the Euler method that works with mean derivative functions, is performed. It is essential to note that the mean derivative functions will be obtained through an artificial neural network.

6.1. Numerical Accuracy of the Conventional Euler Method

The explicit Euler method is a numerical integration technique for solving ordinary differential equations of the form:

$${\displaystyle \frac{dy}{dt}}=f(t,y),y\left(t_0\right)=y_0.$$

(22)

The formula for the method is:

$$y_{n+1}=y_n+hf(t_n,y_n),$$

(23)

where h is the integration step; $y_n$ is the approximation of the exact solution $y\left(t_n\right)$; $f(t_n,y_n)$ is the instantaneous derivative of the solution at the point $(t_n,y_n)$. Therefore, it is a well-known result that the difference between the exact solution and the Euler approximation is given by:

$$y\left(t_{n+1}\right)-y_{n+1}={\displaystyle \frac{h^2}{2}}{y}^{″}\left(t_n\right)+O\left(h^3\right).$$

(24)

Therefore, the accuracy of the method is evaluated by two errors: (i) The local truncation error $E_{\mathrm{local}}=O\left(h^2\right)$ and (ii) the global truncation error $E_{\mathrm{global}}=O\left(h\right)$. Understanding the global error is quite simple; since the Euler method performs $N={\displaystyle \frac{T}{h}}$ steps to integrate up to a time T, the global error is obtained by adding the local errors over this interval. Finally, the accuracy of the Euler method is first order since the global error is $O\left(h\right)$. This result shows that the Euler method is accurate only for small steps, and higher-order methods such as Runge–Kutta are preferable to obtain greater accuracy.

6.2. The E-TUNI Numerical Accuracy

In the modified Euler method, i.e., in E-TUNI, we employ the mean derivative functions ${\overline{f}}_n$, which can be obtained by an Artificial Neural Network (ANN):

$$y_{n+1}=y_n+h{\overline{f}}_n,$$

(25)

where the mean derivative is defined as:

$${\overline{f}}_n={\displaystyle \frac{1}{h}}{\int }_{t_n}^{t_{n+1}}f(t,y\left(t\right))dt.$$

(26)

If the mean derivative is estimated by an Artificial Neural Network (ANN), we will have:

$${\overline{f}}_n^{\mathrm{ANN}}\approx {\overline{f}}_n+{\epsilon }_{\mathrm{ANN}},$$

(27)

where ${\epsilon }_{\mathrm{ANN}}$ represents the error of the neural network. Expanding the exact solution $y\left(t\right)$ around $t_n$, we have that:

$$y\left(t_{n+1}\right)=y(t_n+h)=y\left(t_n\right)+h{y}^{\prime }\left(t_n\right)+{\displaystyle \frac{h^2}{2}}{y}^{″}\left(t_n\right)+O\left(h^3\right).$$

(28)

The mean derivative can be approximated by expanding $y^{\prime }\left(t\right)$:

$$y^{\prime }\left(t\right)=y^{\prime }\left(t_n\right)+(t-t_n)y^{″}\left(t_n\right)+O\left(h^2\right).$$

(29)

Thus, substituting Equation (29) into the integral (26) and integrating over the interval $[t_n,t_{n+1}]$, we have:

$${\overline{f}}_n=y^{\prime }\left(t_n\right)+{\displaystyle \frac{h}{2}}{y}^{″}\left(t_n\right)+O\left(h^2\right).$$

(30)

Thus, substituting the mean derivative Equation (30) into the modified Euler method, we finally have:

$$y_{n+1}=y_n+h{y}^{\prime }\left(t_n\right)+{\displaystyle \frac{h^2}{2}}{y}^{″}\left(t_n\right)+O\left(h^3\right).$$

(31)

Equation (31) shows that the local error of the method has increased to $O\left(h^3\right)$, improving accuracy. On the other hand, if the mean derivative is approximated by an Artificial Neural Network (ANN):

$${\overline{f}}_n^{\mathrm{ANN}}={\overline{f}}_n+{\epsilon }_{\mathrm{ANN}}.$$

(32)

So, the modified method becomes:

$$y_{n+1}=y_n+h({\overline{f}}_n+{\epsilon }_{\mathrm{ANN}}).$$

(33)

So, the total error will be:

$$y\left(t_{n+1}\right)-y_{n+1}=O\left(h^3\right)+O\left(h{\epsilon }_{\mathrm{ANN}}\right).$$

(34)

Thus, if the artificial neural network is accurate enough, we can make ${\epsilon }_{\mathrm{ANN}}\to 0$, making the integration error close to $O\left(h^3\right)$. Thus, the local error being equal to $O\left(h^3\right)+O\left(h{\epsilon }_{\mathrm{ANN}}\right)$ will make the global error, summed over $N=O(1/h)$ steps, equal to $E_{\mathrm{global}}=O\left(h^2\right)+O\left({\epsilon }_{\mathrm{ANN}}\right)$. Therefore, the modified Euler method with mean derivatives improves the global accuracy to $O\left(h^2\right)$. If an artificial neural network is used to estimate the mean derivative, its error ${\epsilon }_{\mathrm{ANN}}$ can be reduced to a minimum, making the method highly accurate. Thus, this method can be seen as a transition to higher-order Runge–Kutta methods.

7. Results and Analysis

In this section, we perform a numerical and computational analysis of the Universal Numerical Integrator (UNI), which exclusively uses mean derivative functions. As previously described, this is the Euler-Type Universal Numerical Integrator (E-TUNI). Therefore, we present here three case studies: (i) a one-dimensional simple case, (ii) the non-linear simple pendulum (two state variables), and (iii) the non-linear inverted pendulum (four state variables).

Thus, for all experiments that it is presented below, the use of MLP neural networks was standardized with the traditional Levenberg–Marquardt training algorithm [33] from 1994. Concerning the architecture of the neural network used, all experiments were trained using only one inner layer (with thirty neurons with tangent hyperbolic activation function) in an artificial neural network of the Multi-Layer Perceptron (MLP) type. In all experiments, $80\%$ of the total patterns were left for training the MLP neural network, $10\%$ of the patterns for testing, and $10\%$ for validation. Concerning the architecture of E-TUNI, the direct approach was exclusively used. However, it must be said that the indirect or empirical approach of E-TUNI was not tested in this article. Thus, the general algorithm for training and testing the methodology presented in this article can be briefly described as follows:

Step 1. Using a high-order 4-5 Runge–Kutta integrator, over the considered autonomous non-linear ordinary differential equation system, randomly generate p training patterns input/output for E-TUNI concerning the initial instant $y^i\left(t_0\right)$. Perform both forward and backward integration.

Step 2. Determine both the values of the positive mean derivatives $ta{n}_{\Delta t}^{+}{}^k\alpha _j^i$, as well as the values of the negative mean derivatives $ta{n}_{\Delta t}^{-}{}^k\alpha _j^i$ in relation to initial instants $y^i\left(t_0\right)$ for $i=1,2,\dots ,p$.

Step 3. Train two distinct neural networks. One is to output positive mean derivative functions, and the other is to output negative mean derivative functions. Do this following the graphic diagram in Figure 6.

Step 4. Simulate, for several different $\Delta t$ integration steps, the performance of E-TUNI both with forward and backward integration. Observe that if starting from an initial instant $y^i\left(t_0\right)$, the solution is propagated in n distinct instants forward and, after that, one returns to the same n instants with backward integration because of the uniqueness theorem, one returns perfectly to the original initial instant $y^i\left(t_0\right)$. This trick will be used to test the efficiency of E-TUNI when it is propagated backward.

Step 5. Do the same, using the Euler-type first-order integrator, but now using the original instantaneous derivative functions of the dynamical system in question. Then, compare E-TUNI with conventional Euler. Thus, to verify the superiority of E-TUNI concerning the conventional Euler, the accuracy of the discrete numerical solutions obtained must be verified.

Therefore, next, we briefly describe the systems of non-linear differential equations of the three dynamical systems considered here to test the proposed algorithm to perform the backward integration of E-TUNI. Table 1 also summarizes the training data of the eight experiments conducted here to test the proposed methodology.

Example 1.

Simulate the one-dimensional non-linear dynamics problem, given by Equation (35). The variable y was trained on the range $[-1,13]$. Note that this dynamic has infinite stability points since $siny=0$ has infinitely many solutions. Note that the points of stability recovering the initial condition through backward integration is very difficult for any integrator.

$$\dot{y}=sin\left(y\right)$$

(35)

Example 2.

Simulate the non-linear simple pendulum problem given by the system of differential Equations (36). Note that this example does not involve control variables. The acceleration values due to gravity constants and the pendulum’s length are, respectively, given by $g=9.81m/s^2$ and $l=0.30m$. The variable θ was trained on the range $[-1.2,1.2]rad$ and the variable $\dot{\theta }$ was trained on the range $[-5.2,+5.2]{\displaystyle \frac{rad}{s}}$.

$$\begin{array}{c}\dot{\theta }=w\\ \dot{w}=-{\displaystyle \frac{g}{l}}sin\theta \end{array}$$

(36)

Example 3.

Do the same for the non-linear inverted pendulum problem described by the system of differential Equations (37). The constants adopted for this problem were $M=1.0\left[kg\right]$, $m=0.4\left[kg\right]$, $l=0.5\left[m\right]$, $I=0.05[kg·m^2]$, $b=0.1\left[{\displaystyle \frac{N}{m/s}}\right]$, $g=9.81\left[fracm{s}^2\right]$, e $F=1\left[N\right]$. This problem has four state variables ($x_1=x$, $x_2=\dot{x}$, $x_3=\theta$, and $x_4=\dot{\theta }$) and a constant control ($u_1=F=1\left[N\right]$). The state variables were trained on the intervals $x_1$ between $[-1,+35]\left[m\right]$, $x_2$ between $[-1,+7][m/s]$, $x_3$ between $[-1,+1]\left[rad\right]$, and $x_4$ between $[-1.2,+1.2][rad/s]$.

$$\begin{array}{c}\dot{x_1}=x_2\\ \dot{x_2}={\displaystyle \frac{[F-b{x}_2+ml{x}_4^{2}sin\left(x_3\right)](I+m{l}^2)+{\left(ml\right)}^{2}gsin\left(x_3\right)cos\left(x_3\right)}{(I+m·l^2)(M+m)-{\left[mlcos\left(x_3\right)\right]}^2}}\\ \dot{x_3}=x_4\\ \dot{x_4}={\displaystyle \frac{[\left(F-b{x}_2+ml{x}_4^{2}sin\left(x_3\right))cos\left(x_3\right)+(M+m)gsin\left(x_3\right)\right]ml}{{\left(mlcos\left(x_3\right)\right]}^2-(I+m{l}^2)(M+m)}}\end{array}$$

(37)

Figure 9 graphically illustrates the numerical simulation obtained for Example 1. This figure is not yet the result obtained by E-TUNI, but by Runge–Kutta 4-5, available in Matlab, to numerically solve Equation (35). Therefore, the result obtained by E-TUNI will be compared later, numerically and graphically, with the graphic result of this figure. Also, in Figure 9, the following convention was used: (i) the blue dots in “x” represent the forward propagation, in high precision, of Runge–Kutta 4-5 over the dynamics of Equation (35), (ii) the blue dots in “o” represent the high precision backward propagation of Runge–Kutta 4-5 over the same dynamics, and (iii) the solid red lines represent the slope of the mean derivatives functions between two consecutive points.

Note that the integration step, as graphically presented, was $\Delta t=0.1$. However, it is essential to notice that the Runge–Kutta 4-5 works internally with a varied integration step much smaller than this one to achieve the high numerical accuracy shown in Figure 9. So, both forward and backward integration were performed from $t_0=0$ to $t_f=5$. In this case, as can be seen, the result obtained is perfect, as the points obtained by backward integration overlapped perfectly over the points with forward integration.

Figure 10 presents the first graphic result of E-TUNI, for the dynamics given by Equation (35). The blue dots represent the results obtained by E-TUNI and follow the same convention as in the previous figure. The green dots represent the results obtained by the conventional Euler Integrator, using the instantaneous derivative functions, also given by Equation (35). The two black dash-dot horizontal lines delimit the E-TUNI training range. Note that within the training range, the E-TUNI result was perfect, with both forward and backward integration. The line in Table 1 with the $Number=1$ indicates the training data obtained for this case. In this way, 400 training patterns were generated, and $80\%$ of them, 320 patterns, were effectively left for training. Note that in Table 1 the symbols $\left(MS{E}_F\right)$ and $\left(MS{E}_B\right)$ represent, respectively, the mean square error of forward integration and the mean square error of backward integration.

Note that outside the training region, E-TUNI has become inaccurate. However, to solve this problem, the training domain of the state variable should be increased. Note also that the conventional Euler presented a slight deviation in the backward integration concerning the forward integration in the given domain. Thus, in conventional Euler, this deviation becomes more critical and accentuated, which increases the integration step. However, in E-TUNI, the result does not deteriorate with the increase of the integration step.

Figure 11 presents the same result as the previous figure, but now using an integration step $\Delta t=0.5$. The E-TUNI result was perfect, but the conventional Euler result was even worse. This confirms what was said in the previous paragraph. See Table 1 again in line $Number=2$ to find out about the main parameters used in this training. Figure 12 presents the same result as the previous figure, but now using an integration step $\Delta t=0.01$. In this case, the result achieved by E-TUNI is still perfect, and the result by conventional Euler is as well. The result, obtained by conventional Euler, was perfect in this case because a tiny integration step was used and equal to $\Delta t=0.01$. Figure 13 presents a simulation using the integration step $\Delta t=2>1$. Again, E-TUNI presented a perfect result, but the conventional Euler degenerated its solution.

Figure 14 and Figure 15 deserve a little more attention. In Figure 14, as can be seen, the solution obtained by E-TUNI, through backward integration, was degenerate (see the zoom). In this case, the blue circle should lie perfectly on the solid red line. In Figure 15, the backward integration solution obtained by Runge–Kutta 4-5 was also degenerate. Both solutions turned out bad because the integration process was conducted for too long, over a point of stability. The zoom made in Figure 14 will be used to better explain this.

What is explained for Figure 14 is also valid for Figure 15. In Figure 14, the integration process was conducted from $t_0=0$ to $t_f=20$. The biggest problem occurred when returning from $t_f=20$ to $t_0=0$. For example, all families of curves confined between $y\left(0\right)=0$ and $y\left(0\right)=2·\pi$ at the initial instant $t=0$ were confined in a much smaller interval than this one at the instant $t_f=20$. Therefore, to work properly, backward integration would have to be trained with a much smaller mean squared error than the one achieved in Table 1 (line $Number=5$). However, this is impossible to achieve using just the algorithm in [33]. Note that this task was also impossible for Runge–Kutta 4-5, as schematized in Figure 15. Therefore, the problem presented here is critical, regardless of the integrator used.

Figure 16 presents the first graphic result of E-TUNI for the dynamics given by Equation (36), which represents the dynamics of the non-linear simple pendulum. In this case, the superiority of E-TUNI over the conventional Euler is evident. An integration step $\Delta t=0.02$ was used in this case. Figure 17 concerns the same previous example, but using an integration step $\Delta t=5$. The conventional Euler was not presented in this figure because its result was bad, which would impair, therefore, the scale of the graph. Still referring to Figure 17, the main reason the backward integration turned out to be a bit bad at the end of the integration process is that the average training error became slightly oversized (see Table 1 in cell $Number=7$). To solve this problem, it is enough to reduce the average training error of the artificial neural network used, but that is not always an easy task. Figure 18 demonstrates that, even for Runge–Kutta 4-5, the backward integration process of the simple pendulum can also be imperfect (see blue circles misaligned with the blue “x”).

Note that the solution from Equation (36), graphically presented in Figure 17, was much worse than the other solutions given in Table 1 (compare lines 6 and 7). This fact can be seen by checking the training mean square errors in Table 1 (line $Number7$), which were too high. This was most likely caused by two fundamental reasons. The first reason is that the E-TUNI local error is amplified by the square of the Integration step. However, the analysis of the global error would require further studies. The other reason, which is purely speculative, is the following: it is believed that when the E-TUNI integration step is very large if the solution of the real dynamical system becomes a non-convex function (concerning the line of mean derivatives in this same interval $\Delta t$), then the training becomes more difficult. However, this last argument still needs to be verified empirically and theoretically.

Figure 19 presents the first graphic result of E-TUNI, for the dynamics given by Equation (37), which represents the dynamics of the non-linear inverted pendulum. As can be seen, in this figure, Runge–Kutta 4-5 perfectly simulated this dynamic, both for forward and backward integrations. Figure 20 is the same case simulation but performed by E-TUNI. Theoretically, Figure 20 is the same as Figure 19. They are just on different scales. Notice that in Figure 20 the E-TUNI simulation was perfect. However, the conventional Euler simulation (green dots) became confusing due to the numerical imprecision achieved by it. Furthermore, this caused an exaggerated change in the scale of the figure.

Table 1. Training data summary.

N.	Simul.	$\left({\mathbf{MSE}}_{\mathit{F}}\right)$	${(\Delta \mathit{t})}^2·{\mathbf{MSE}}_{\mathit{F}}$	$\left({\mathbf{MSE}}_{\mathit{B}}\right)$	${(\Delta \mathit{t})}^2·{\mathbf{MSE}}_{\mathit{B}}$	Patterns
1	Figure 10	$1.951\times {10}^{-10}$	$1.951\times {10}^{-12}$	$1.035\times {10}^{-11}$	$1.035\times {10}^{-13}$	320
2	Figure 11	$2.157\times {10}^{-13}$	$5.393\times {10}^{-14}$	$1.322\times {10}^{-14}$	$3.305\times {10}^{-15}$	320
3	Figure 12	$3.097\times {10}^{-12}$	$3.097\times {10}^{-16}$	$4.768\times {10}^{-14}$	$4.768\times {10}^{-18}$	320
4	Figure 13	$8.636\times {10}^{-12}$	$3.454\times {10}^{-11}$	$2.169\times {10}^{-12}$	$8.676\times {10}^{-12}$	320
5	Figure 14	$8.636\times {10}^{-12}$	$3.454\times {10}^{-11}$	$2.169\times {10}^{-12}$	$8.676\times {10}^{-12}$	320
6	Figure 16	$7.921\times {10}^{-11}$	$3.168\times {10}^{-14}$	$6.770\times {10}^{-11}$	$2.708\times {10}^{-14}$	580
7	Figure 17	$1.845\times {10}^{-6}$	$4.613\times {10}^{-5}$	$1.892\times {10}^{-6}$	$4.730\times {10}^{-5}$	580
8	Figure 20	$1.774\times {10}^{-10}$	$1.774\times {10}^{-12}$	$6.645\times {10}^{-11}$	$6.645\times {10}^{-13}$	640

Table 1. Training data summary.

N.	Simul.	$\left({\mathbf{MSE}}_{\mathit{F}}\right)$	${(\Delta \mathit{t})}^2·{\mathbf{MSE}}_{\mathit{F}}$	$\left({\mathbf{MSE}}_{\mathit{B}}\right)$	${(\Delta \mathit{t})}^2·{\mathbf{MSE}}_{\mathit{B}}$	Patterns
1	Figure 10	$1.951\times {10}^{-10}$	$1.951\times {10}^{-12}$	$1.035\times {10}^{-11}$	$1.035\times {10}^{-13}$	320
2	Figure 11	$2.157\times {10}^{-13}$	$5.393\times {10}^{-14}$	$1.322\times {10}^{-14}$	$3.305\times {10}^{-15}$	320
3	Figure 12	$3.097\times {10}^{-12}$	$3.097\times {10}^{-16}$	$4.768\times {10}^{-14}$	$4.768\times {10}^{-18}$	320
4	Figure 13	$8.636\times {10}^{-12}$	$3.454\times {10}^{-11}$	$2.169\times {10}^{-12}$	$8.676\times {10}^{-12}$	320
5	Figure 14	$8.636\times {10}^{-12}$	$3.454\times {10}^{-11}$	$2.169\times {10}^{-12}$	$8.676\times {10}^{-12}$	320
6	Figure 16	$7.921\times {10}^{-11}$	$3.168\times {10}^{-14}$	$6.770\times {10}^{-11}$	$2.708\times {10}^{-14}$	580
7	Figure 17	$1.845\times {10}^{-6}$	$4.613\times {10}^{-5}$	$1.892\times {10}^{-6}$	$4.730\times {10}^{-5}$	580
8	Figure 20	$1.774\times {10}^{-10}$	$1.774\times {10}^{-12}$	$6.645\times {10}^{-11}$	$6.645\times {10}^{-13}$	640

8. Conclusions

As far as we know, all previous work on E-TUNI was developed using only forward integration. Therefore, we present here, for the first time, a work that uses E-TUNI in a backward integration process. Three case studies were carried out. Non-linear and autonomous ordinary differential equations govern all three dynamical systems considered in this study.

The Euler-type Integrator designed with instantaneous derivatives is not very accurate. However, the E-TUNI has numerical precision equivalent to that of the highest-order integrators. It is interesting to note that the elegance of the theory of ordinary differential equations can be improved when combined with the theory of universal approximator of functions. This propriety is possible because artificial neural networks allow working with real-world input/output training patterns. This fact enables refining the final solution obtained by the first-order Euler integrator and estimator. Because of this, the E-TUNI can also be used to predict real-world dynamics (e.g., river flow and stock market forecasts, among others).

Additionally, we can call the forward mean derivatives functions positive mean derivatives functions. Equivalently, we can call the backward mean derivatives functions negative mean derivatives functions. Thus, we also conclude that the mean derivative functions depend on the integration step and are asymmetric. This propriety means training two distinct neural networks to simultaneously perform forward and backward integration. Unfortunately, the positive and negative instantaneous derivative functions are symmetric to each other.

Additionally, it would be interesting to carry out a future study of E-TUNI using deep neural networks instead of shallow neural networks, aiming to reach two essential points: (i) to be able to increase the number of training patterns (and, consequently, the domain of state variables) and (ii) try to reduce the training mean squared error, as this quantity is critical, in the case of modeling non-linear dynamic systems designed through artificial neural networks. Interestingly, E-TUNI is an excellent option to replace the NARMAX model and even the Runge–Kutta Neural Network (RKNN).

Finally, as another future contribution, one could consider changing the notation used in this article to something more straightforward, as explained in [34,35]. It is essential to notice that the same notation used in articles [16,26,27,28,31] was used here. In this sense, the most significant criticism would be not using the over-index i in the algebraic development of E-TUNI. However, in [31], the over-index i was used to help demonstrate the general expression of E-TUNI. Furthermore, E-TUNI could also be tested for dynamic systems using control variables.