On Stability of Perturbed Nonlinear Switched Systems with Adaptive Reinforcement Learning

Abstract

In this paper, a tracking control approach is developed based on an adaptive reinforcement learning algorithm with a bounded cost function for perturbed nonlinear switched systems, which represent a useful framework for modelling these converters, such as DC–DC converter, multi-level converter, etc. An optimal control method is derived for nominal systems to solve the tracking control problem, which results in solving a Hamilton–Jacobi–Bellman (HJB) equation. It is shown that the optimal controller obtained by solving the HJB equation can stabilize the perturbed nonlinear switched systems. To develop a solution to the translated HJB equation, the proposed neural networks consider the training technique obtaining the minimization of square of Bellman residual error in critic term due to the description of Hamilton function. Theoretical analysis shows that all the closed-loop system signals are uniformly ultimately bounded (UUB) and the proposed controller converges to optimal control law. The simulation results of two situations demonstrate the effectiveness of the proposed controller.

1. Introduction

Many power electronic converters play a remarkable role in industrial applications, such as electrical drives, renewable energy systems, etc. [1,2,3,4,5]. The modelling technique in power electronic converters is usually implemented by considering the average small-signal analysis. This method enables us to design the voltage, current controller, as well as implement the space vector modulation technique. However, the average small-signal model method is disadavantageous for complicated applications, with the requirement of tracking control systems. Switched systems have been known as a special class of hybrid systems describing an active application area in the field of power electronics, such as DC–DC converter, power source, etc. [6,7,8,9,10,11,12,13,14,15]. The approach of employing switched systems representation in control design has been the efficient method to model many practical systems based on multi-subsystem. Basic issues in the control of switched systems are to find the control input under arbitrary switching signal and/or to find the appropriate switching signal obtaining the tracking, stability of closed-loop systems. Additionally, the identification of the active mode based on estimating the delay to analyze the switched systems was implemented in [6]. The authors in [7] proposed a novel Lyapunov function combining linear matrix inequalities (LMIs) for switched linear systems in the presence of exogenous disturbance. The control design problem with stochastic stability based on a complete probability space under the general random switching signal was investigated in [8]. The stabilizing switching signal was designed using average dwell time with proposed existence conditions [9]. Moreover, several conditions to characterize the mode-dependent dwell-times have been given in [10]. The switching event triggered and back stepping technique-based tracking controller achieved the bound of all signals and tracking error in the closed-loop system [11]. However, the optimal control with effectiveness in dealing with constraint has not been mentioned in switched systems yet [6,7,8,9,10,11]. For the optimal control design, it is necessary to find the result of a Ricatti equation for linear systems or a Hamilton–Jacobi–Bellman (HJB) equation for nonlinear systems [16]. However, in the general case, it is hard to solve the HJB (Hamilton–Jacobi–Bellman) equation for finding the optimal controller. In [16,17,18,19], thanks to the approximation description of Neural Networks (NNs), the weight of the actor and critic were updated simultaneously under the optimization problem for continuous/discrete time systems. The convergence effectiveness of weights in actor/critic policy is guaranteed by the persistence excitation (PE) condition. It is worth emphasizing that to deal with the uncertainties of nonlinear continuous systems, the identifier was inserted into the control structure [19] and the off-policy technique under the separation between actor/critic policy and control input in the first stage is presented [20]. Moreover, the constraint in input/output was mentioned in the adaptive/approximate dynamic programming (ADP) technique [17,18,21] by using the appropriate cost function and the dynamic programming principle. The framework of ADP was utilized in control scheme for surface vessel systems [22], a spring-Mass-Damper [23], two mass-spring systems [20], Wheeled mobile robotic systems [24]. Based on approximating the modified critic term in each control loop, the stability of whole cascade systems was guaranteed in surface vessel systems due to the consideration of the derivative of Lyapunov candidate function along the closed system [22,25]. However, all the ARL-based controllers presented in [20,22,23,24] are implemented under the consideration of nonlinear systems, which are not mentioned in the switching signals. The model-free problem, as well as the connection between continuous time systems and appropriate discrete time systems, ensure the development of off-policy algorithm [26,27]. To deal with the complete dynamic uncertainties, the modified cost function and critic enable us to obtain the approximately optimal control law [28,29,30,31]. Additionally, the general policy iteration (GPI) in linear discrete time systems [32] achieves the simultaneous computation and appropriate data collection.

To the best knowledge of the authors, adaptive reinforcement learning (ARL) for uncertain switched systems is still an open problem. Therefore, this paper attempts to implement an ARL algorithm for uncertain continuous-time nonlinear switched systems under arbitrary switching signals based on an optimization problem in NNs training. Inspired by the above works, this paper studies the ARL-based optimal control problem for a class of perturbed switched nonlinear continuous systems. First, the optimal control algorithm is established for switched systems. Then, the neural network is employed to approximate the Critic part in policy. Based on optimization principle, the training law is proposed to develop the optimal control strategy. The main contributions of this paper are as follows:

(1)

In comparison with the previous papers [12,13,14,15,18,22,29,30,33], an optimal control obtained from the nominal system is proposed for perturbed switched nonlinear continuous systems based on dynamic programming principle.

(2)

The Neural Network training law with optimization principle is developed to achieve the ARL-based optimal control strategy.

(3)

The strict proof concerning UUB stability of the closed system and the convergence of the controller to optimal control input are given based on Lyapunov stability theory and reinforcement learning scheme.

The rest of this article is organized as follows. The preliminaries and problem statements are presented in Section 2. The main results are given in Section 3. Two simulation cases are presented in Section 4 to illustrate the effectiveness of the proposed solution. Finally, the conclusions are exhibited in Section 5.

2. Problem Statement and Preliminaries

In [15], it can be seen that switched systems can represent a useful framework for modelling general power electronics converters. Therefore, in this section, we consider the perturbed continuous-time nonlinear switched system described by:

$$\dot{x}=f_{\sigma }\left(x\right)+g_{\sigma }\left(x\right)\left(u+\Delta \left(x,t\right)\right)$$

(1)

where $x\left(t\right)\in {\mathsf{\Omega }}_x\subset {\mathbb{R}}^n$ and $u\left(t\right)\in {\mathsf{\Omega }}_u\subset {\mathbb{R}}^m$ are the vector of state variables and the control inputs vector, respectively. The function $\sigma :\left[\left.0,+\infty \right)\to \right.\mathsf{\Omega }=\left\{1,2,...,n\right\}$ is an unknown switching signal and n is the number of subsystems. $f_i\left(x\right)$ $\left(\forall i\in \mathsf{\Omega }\right)$ are unknown smooth vector functions satisfying $f_i\left(0\right)=0$ and $g_i\left(x\right)$$\left(\forall i\in \mathsf{\Omega }\right)$ are known smooth vector functions such that $G_{min}\le ∥g_i\left(x\right)∥\le G_{max}$.

Assumption 1.

There exists a known function $\rho \left(x\right)$ satisfying that: $∥\Delta \left(x,t\right)∥\le \rho \left(x\right)$.

Regarding the perturbed continuous-time nonlinear switched system (1), we introduce the cost function formulated as:

$$J\left(x\left(t\right),u\left(t\right)\right)=\underset{t}{\overset{\infty }{\int }}r\left(x\left(\tau \right),u\left(\tau \right)\right)d\tau$$

(2)

where $r\left(x\left(\tau \right),u\left(\tau \right)\right)=x^{T}Qx+u^{T}Ru$ and $Q=Q^T\ge 0;$ $R=R^T>0$

Control Objective: This article aims at designing the optimal guaranteed cost control scheme $u^{\ast }=arg\underset{u\in {\mathsf{\Omega }}_u}{min}K\left(x,u\right)$ despite arbitrary switching law, in which the feedback control law u and a finite upper bound function $K\left(x,u\right)$ are satisfied, not only is the closed-loop system (1) robustly stable but the cost function (2) also satisfies the condition: $J\left(x,u\right)\le K\left(x,u\right)\le M$.

Definition 1.

The function $K(x,u)$ can be known as the guaranteed cost function. Therefore, the control law $u^{\ast }$ with $u^{\ast }=arg\underset{u\in {\mathsf{\Omega }}_u}{min}K\left(x,u\right)$ is known as the optimal guaranteed cost control law.

Remark 1.

It is worth emphasizing that the main objective of this work is to find the optimal control for the equivalent nominal system with the performance index being modified. Compared with the control objective in [33], this work investigates the tracking control for switched systems, for which it is hard to develop the optimal control algorithm. Additionally, it is obviously different from the existing work of robust ADP in [31], since the proposed optimal control design is implemented based on the upper bound function $K(x,u)$.

3. Adaptive Reinforcement Learning-Based Control Design

In this section, we investigate the ARL-based optimal control for perturbed nonlinear switched systems. Due to the difficulties in implementing directly the optimal control scheme for perturbed nonlinear systems, the strategy is proposed by three steps as follows. First, based on Dynamic Programming, we obtain the optimal control design for the corresponding nominal switched systems obtained by eliminating the uncertainties. Then, the ARL algorithm is developed by Neural Network technique for this nominal system. Finally, we carry out a stability analysis of the closed-loop system involving the perturbed switched system and the proposed ARL controller.

According to perturbed system (1), the nominal system is obtained by eliminating the term of uncertainties as:

$$\dot{x}=f_{\sigma }\left(x\right)+g_{\sigma }\left(x\right)u$$

(3)

Because the purpose is to consider the tracking problem of optimal control law and the influence of uncertainties, the cost function is modified to satisfy that $J\left(x\left(t\right),u\left(t\right)\right)\le J_1\left(x\left(t\right),u\left(t\right)\right)$. Therefore, the cost function associated with (3) can be represented as:

$$J_1\left(x\left(t\right),u\left(t\right)\right)=\underset{t}{\overset{\infty }{\int }}\left[r\left(x\left(\tau \right),u\left(\tau \right)\right)+\lambda {\left(\rho \left(x\right)\right)}^2\right]d\tau$$

(4)

It should be noted that $J_1\left(x\left(t\right),u\left(t\right)\right)$ with $\lambda \ge ∥R∥$ is one of the guaranteed cost function associated with system (1). According to (2) and (4), it can be seen that:

$$J\left(x\left(t\right),u\left(t\right)\right)\le J_1\left(x\left(t\right),u\left(t\right)\right)$$

(5)

Based on the Dynamic programming principle, establishing the Bellman function as: $V^{\ast }\left(t\right)=\underset{u\in {\mathsf{\Omega }}_u}{min}{J}_1\left(x\left(t\right),u\left(t\right)\right)$, we have the cost function formulated as:

$$V^{\ast }\left(t\right)=\underset{u\in {\mathsf{\Omega }}_u}{min}\underset{t}{\overset{\infty }{\int }}\left[r\left(x\left(\tau \right),u\left(\tau \right)\right)+\lambda {\left(\rho \left(x\right)\right)}^2\right]d\tau$$

(6)

$$V^{\ast }\left(t\right)=\underset{u\in {\mathsf{\Omega }}_u}{min}\underset{t}{\overset{t+\Delta t}{\int }}\left[r\left(x\left(\tau \right),u\left(\tau \right)\right)+\lambda {\left(\rho \left(x\right)\right)}^2\right]d\tau +\underset{u\in {\mathsf{\Omega }}_u}{min}\underset{t+\Delta t}{\overset{\infty }{\int }}\left[r\left(x\left(\tau \right),u\left(\tau \right)\right)+\lambda {\left(\rho \left(x\right)\right)}^2\right]d\tau$$

(7)

$$V^{\ast }\left(x\right)=\underset{u\in {\mathsf{\Omega }}_u}{min}\underset{t}{\overset{t+\Delta t}{\int }}\left[r\left(x\left(\tau \right),u\left(\tau \right)\right)+\lambda {\left(\rho \left(x\right)\right)}^2\right]d\tau +V^{\ast }\left(t+\Delta t\right)$$

(8)

$$\underset{u\in {\mathsf{\Omega }}_u}{min}\left\{\frac{1}{\Delta t}\underset{t}{\overset{t+\Delta t}{\int }}\left[r\left(x\left(\tau \right),u\left(\tau \right)\right)+\lambda {\left(\rho \left(x\right)\right)}^2\right]d\tau +\frac{V^{\ast }\left(t+\Delta t\right)-V^{\ast }\left(t\right)}{\Delta t}\right\}=0$$

(9)

As the convergence of $\Delta t\to 0^{+}$, we can derive that:

$$\underset{u\in {\mathsf{\Omega }}_u}{min}\left[r\left(x\left(t\right),u\left(t\right)\right)+\lambda {\rho }^2\left(x\right)+{\left(\nabla V^{\ast }\right)}^T\left(f_{\sigma }\left(x\right)+g_{\sigma }\left(x\right)u\right)\right]=0$$

(10)

where $\nabla V^{\ast }=\frac{\partial V^{\ast }}{\partial x}$.

Consider the Hamilton function obtaining from the nominal system and performance index (4):

$$H\left(x,u,V^{\ast }\right)=r\left(x\left(t\right),u\left(t\right)\right)+\lambda {\rho }^2\left(x\right)+{\left(\nabla V^{\ast }\right)}^T\left(f_{\sigma }\left(x\right)+g_{\sigma }\left(x\right)u\right)$$

(11)

The control input is computed by minimizing this function with $u\in {\mathsf{\Omega }}_u$ as

$$H\left(x,u^{\ast },V^{\ast }\right)=\underset{u\in {\mathsf{\Omega }}_u}{min}H\left(x,u,V^{\ast }\right)=0$$

(12)

Therefore, we find that (13) yields

$${\left.\frac{\partial H\left(x,u,V^{\ast }\right)}{\partial u}\right|}_{u=u^{\ast }}=0\Rightarrow u^{\ast }=-\frac{1}{2}{R}^{-1}{\left(g_{\sigma }\left(x\right)\right)}^T\nabla V^{\ast }$$

(13)

We implement to develop this control algorithm (13) for nonlinear switched system (1) and achieve the following result:

Theorem 1.

Take system (1) with the feedback control law $u^{\ast }\left(x\right)=-\frac{1}{2}{R}^{-1}{\left(g_{\sigma }\left(x\right)\right)}^T\nabla V^{\ast }$ into consideration. Then, the cost function $V^{\ast }\left(t\right)=\underset{t}{\overset{\infty }{\int }}\left[r\left(x^{\ast }\left(\tau \right),u^{\ast }\left(\tau \right)\right)+\lambda {\left(\rho \left(x^{\ast }\right)\right)}^2\right]d\tau$ is the Lyapunov function candidate with $\lambda \ge ∥R∥$ can guarantee that the system (1) is stable.

Proof. 

Consider the derivative of $V^{\ast }\left(t\right)$ along the solution of Equation (1), we achieve the result formulated as:

$${\dot{V}}^{\ast }\left(t\right)={\left(\nabla V^{\ast }\right)}^T\left(f_i\left(x^{\ast }\right)+g_i\left(x^{\ast }\right)\left(u^{\ast }+\Delta \left(x^{\ast },t\right)\right)\right)$$

(14)

It follows from $u^{\ast }\left(x\right)=-\frac{1}{2}{R}^{-1}{\left(g_i\left(x\right)\right)}^T\nabla V^{\ast }$ and according to (14), (11) and (12), we have:

$$\begin{array}{cc}8{\dot{V}}^{\ast }\left(t\right)\hfill & =-r\left(x^{\ast }\left(t\right),u^{\ast }\left(t\right)\right)-\lambda {\rho }^2\left(x\right)+{\left(g_i\left(x^{\ast }\right)\right)}^T\nabla V^{\ast }\Delta \left(x^{\ast },t\right)\hfill \\ \hfill & =-x^{\ast T}Q{x}^{\ast }-u^{\ast }TR{u}^{\ast }-\lambda {\rho }^2\left(x^{\ast }\right)-2u^{\ast T}R\Delta \left(x^{\ast },t\right)\hfill \end{array}$$

(15)

$$=-x^{\ast T}Q{x}^{\ast }-\lambda {\rho }^2\left(x^{\ast }\right)-\left(u^{\ast T}+\Delta {\left(x^{\ast },t\right)}^T\right)R\left(u^{\ast }+\Delta \left(x,t\right)\right)+\Delta {\left(x^{\ast },t\right)}^{T}R\Delta \left(x^{\ast },t\right)$$

(16)

$$=-x^{\ast T}Q{x}^{\ast }-\left(\lambda {\rho }^2\left(x^{\ast }\right)-\Delta {\left(x^{\ast },t\right)}^{T}R\Delta \left(x^{\ast },t\right)\right)-{\left(u^{\ast }+\Delta \left(x^{\ast },t\right)\right)}^{T}R\left(u^{\ast }+\Delta \left(x,t\right)\right)$$

(17)

According to Assumption 1 and $\lambda \ge ∥R∥$, it can be seen that:

$${\dot{V}}^{\ast }\left(t\right)\le -x^{\ast T}Q{x}^{\ast }-\left(\lambda {\rho }^2\left(x^{\ast }\right)-∥R∥{∥\Delta \left(x^{\ast },t\right)∥}^2\right)\le -x^{\ast T}Q{x}^{\ast }-\left(\lambda -∥R∥\right){\rho }^2\left(x^{\ast }\right)$$

(18)

$$\dot{V}\left(t\right)\le -x^{\ast T}Q{x}^{\ast }$$

(19)

Therefore, the system (1) is stable under the optimal control for equivalent nominal system. □

It is noteworthy that Theorem 1 has a new extension in switched systems compared with nonlinear systems in the existing result [33].

However, it is impossible to find the Bellman function $V^{\ast }\left(x\right)$ based on solving analytically the HJB nonlinear Equation (12). Hence, to solve it, we construct a critic network under the framework of adaptive critic learning. Using the approximation property of neural networks described in [28], the critic associated with system (3) can be described as:

$$V^{\ast }\left(x\right)=w^T\sigma \left(x\right)+\epsilon \left(x\right)$$

(20)

where $\sigma \left(x\right):{\mathbb{R}}^n\to {\mathbb{R}}^N;\sigma \left(0\right)=0$ is the vector activation function including N linearly independent elements, N is the number of neurons in the hidden layer of Radial Basis Function (RBF) network [28] and it is also the number of dimensions of $\sigma \left(x\right):{\mathbb{R}}^n\to {\mathbb{R}}^N$, and $\epsilon \left(x\right)$ is the function reconstruction error playing the role in finding the training law the in next steps, $w\in R^N$ is the idea weight vector and is generally unavailable. As $N\to \infty$ converges to infinity, we obtain that $\epsilon \left(x\right)\to 0$ and $\nabla \epsilon \left(x\right)\to 0$. The following assumption is considered for each fixed N.

Assumption 2.

The estimation of terms in NNs can be described as: $∥\epsilon \left(x\right)∥\le {\epsilon }_{max};∥\nabla \epsilon \left(x\right)∥\le \nabla {\epsilon }_{max};\nabla {\sigma }_{min}\le ∥\nabla \sigma \left(x\right)∥\le \nabla {\sigma }_{max};∥w∥\le w_{max}$

According two Equations (12) and (13), it holds that:

$$H\left(x,u^{\ast },V^{\ast }\right)=r\left(x\left(t\right),u^{\ast }\left(t\right)\right)+\lambda {\rho }^2\left(x\right)+{\left(\nabla V^{\ast }\right)}^T\left(f_i\left(x\right)+g_i\left(x\right)u^{\ast }\right)=0$$

(21)

$$=x^{T}Qx+\lambda {\rho }^2\left(x\right)+{\left(\nabla V^{\ast }\right)}^T{f}_i\left(x\right)-\frac{1}{4}{\left(\nabla V^{\ast }\right)}^T{g}_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\left(\nabla V^{\ast }\right)=0$$

(22)

It follows from (20) to:

$$\nabla V^{\ast }={\left(\nabla \sigma \left(x\right)\right)}^{T}w+\nabla \epsilon \left(x\right)$$

(23)

The NN-based HJB equation can be represented as follows:

$$e_{NN}=x^{T}Qx+\lambda {\rho }^2\left(x\right)+{\left(\nabla \sigma {\left(x\right)}^{T}w\right)}^T{f}_i\left(x\right)-\frac{1}{4}{\left(\nabla \sigma {\left(x\right)}^{T}w\right)}^T{g}_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\left(\nabla \sigma {\left(x\right)}^{T}w\right)$$

(24)

$$e_{NN}=x^{T}Qx+\lambda {\rho }^2\left(x\right)+w^T\nabla \sigma \left(x\right)f_i\left(x\right)-\frac{1}{4}{w}^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^{T}w$$

(25)

Thus, the residual error is formulated by the function approximation error:

$$e_{NN}=-\nabla \epsilon {\left(x\right)}^T\left(f_i\left(x\right)+g_i\left(x\right)u^{\ast }\right)+\frac{1}{4}\nabla \epsilon {\left(x\right)}^T{g}_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \epsilon \left(x\right)$$

(26)

It should be noted that, as N goes to infinity, $e_{NN}$ converges uniformly to zero. Hence, the residual error $e_{NN}$ is bounded for each fixed N. Under the framework of ADP-based approximate optimal control design, a critic neural network is established by an estimated weight vector $\hat{w}$ given as:

$$\hat{V}={\hat{w}}^T\sigma \left(x\right)=\sigma {\left(x\right)}^T\hat{w};\hat{u}=-\frac{1}{2}{R}^{-1}{\left(g_i\left(x\right)\right)}^T\nabla \hat{V}$$

(27)

The approximate error of the critic network can be formulated as:

$$e_{HJB}=r\left(x\left(t\right),\hat{u}\left(t\right)\right)+\lambda {\rho }^2\left(x\right)+{\left(\nabla \sigma {\left(x\right)}^T\hat{w}\right)}^T\left(f_i\left(x\right)+g_i\left(x\right)\hat{u}\right)$$

(28)

$$e_{HJB}=x^{T}Qx+\lambda {\rho }^2\left(x\right)+{\hat{w}}^T\nabla \sigma \left(x\right)f_i\left(x\right)-\frac{1}{4}{\hat{w}}^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^T\hat{w}$$

(29)

The weight vector is determined by a steepest descent algorithm:

$$\frac{d}{dt}\hat{w}=-\alpha \frac{\partial E}{\partial \hat{w}}$$

(30)

with $E=\frac{1}{2}{e}_{HJB}^T{e}_{HJB}$

Remark 2.

The weight vector $\hat{w}$ is designed as in (30) to minimize the quadratic function $E=\frac{1}{2}{e}_{HJB}^T{e}_{HJB}$. The fact is that the following inequality:

$$\dot{E}=\frac{\partial E}{\partial t}=\frac{\partial E}{\partial \hat{w}}.\frac{\partial \hat{w}}{\partial t}=\frac{\partial E}{\partial \hat{w}}.\left(\frac{d}{dt}\hat{w}\right)=-\alpha {\left(\frac{\partial E}{\partial \hat{w}}\right)}^2\le 0$$

(31)

Theorem 2.

The proposed optimal control (27) with the weight vector of the critic network (30) can force the system (1) to be uniform ultimate bounded (UUB) stability.

Proof. 

Define: $\tilde{w}=w-\hat{w}\Rightarrow \frac{d}{dt}\tilde{w}=-\frac{d}{dt}\hat{w}$. Consider the Lyapunov function:

$$V\left(t\right)=V_1\left(t\right)+V_2\left(t\right),\mathrm{where}:V_1\left(t\right)=\frac{1}{2\alpha }\tilde{w}{\left(t\right)}^T\tilde{w}\left(t\right),V_2\left(t\right)=V^{\ast }$$

For deriving the term $V_1\left(t\right)$, we obtain that:

$$\frac{d}{dt}{V}_1\left(t\right)=\frac{1}{\alpha }\tilde{w}{\left(t\right)}^T\left(\frac{d}{dt}\tilde{w}\left(t\right)\right)=-\frac{1}{\alpha }\tilde{w}{\left(t\right)}^T\left(\frac{d}{dt}\hat{w}\left(t\right)\right)=\tilde{w}{\left(t\right)}^T\frac{\partial E}{\partial \hat{w}}$$

$${\dot{V}}_1={\tilde{w}}^T{e}_{HJB}\nabla \sigma \left(x\right)\left(f_i\left(x\right)+g_i\left(x\right)\hat{u}\right)$$

According to (27) and (13), it follows that:

$$\hat{u}-u^{\ast }=-\frac{1}{2}{R}^{-1}{\left(g_i\left(x\right)\right)}^T\left(\nabla \hat{V}-\nabla V^{\ast }\right)$$

(32)

$$=-\frac{1}{2}{R}^{-1}{\left(g_i\left(x\right)\right)}^T\left(\nabla \sigma {\left(x\right)}^T\hat{w}-{\left(\nabla \sigma \left(x\right)\right)}^{T}w-\nabla \epsilon \left(x\right)\right)$$

(33)

$$=\frac{1}{2}{R}^{-1}{\left(g_i\left(x\right)\right)}^T\left({\left(\nabla \sigma \left(x\right)\right)}^T\tilde{w}+\nabla \epsilon \left(x\right)\right)$$

(34)

Moreover, we have:

$$\nabla \sigma \left(x\right)\left(f_i\left(x\right)+g_i\left(x\right)\hat{u}\right)=\nabla \sigma \left(x\right)\left(f_i\left(x\right)+g_i\left(x\right)u^{\ast }\right)+\nabla \sigma \left(x\right)g_i\left(x\right)\left(\hat{u}-u^{\ast }\right)$$

(35)

$$\nabla \sigma \left(x\right)\left(f_i\left(x\right)+g_i\left(x\right)\hat{u}\right)=\nabla \sigma \left(x\right)\left(f_i\left(x\right)+g_i\left(x\right)u^{\ast }+\frac{1}{2}{g}_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\left(\nabla \sigma {\left(x\right)}^T\tilde{w}+\nabla \epsilon \left(x\right)\right)\right)$$

(36)

From (26) and (29), it follows:

$$\begin{array}{c}{e}_{HJB}-e_{NN}=\left({\left(\nabla \sigma {\left(x\right)}^T\hat{w}\right)}^T{f}_i\left(x\right)-{\left(\nabla \sigma {\left(x\right)}^{T}w\right)}^T{f}_i\left(x\right)\right)\hfill \\ +\frac{1}{4}\left(w^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^{T}w-{\hat{w}}^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^T\hat{w}\right)\hfill \\ +\frac{1}{2}{\left(\nabla \sigma {\left(x\right)}^T\hat{w}\right)}^T{g}_i\left(x\right)R^{-1}{\left(g_i\left(x\right)\right)}^T\left({\left(\nabla \sigma \left(x\right)\right)}^T\tilde{w}+\nabla \epsilon \left(x\right)\right)\hfill \end{array}$$

(37)

$$\begin{array}{c}{e}_{HJB}-e_{NN}=-{\left(\nabla \sigma {\left(x\right)}^T\tilde{w}\right)}^T{f}_i\left(x\right)\hfill \\ +\frac{1}{4}\left(\left(w^T-{\hat{w}}^T\right)\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^{T}w+{\hat{w}}^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^T\left(w-\hat{w}\right)\right)\hfill \end{array}$$

(38)

$$\begin{array}{c}{e}_{HJB}-e_{NN}=-{\tilde{w}}^T\nabla \sigma \left(x\right)f_i\left(x\right)\hfill \\ +\frac{1}{4}\left({\tilde{w}}^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^{T}w+{\hat{w}}^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^T\tilde{w}\right)\hfill \end{array}$$

(39)

$$\begin{array}{c}{e}_{HJB}-e_{NN}=-{\tilde{w}}^T\nabla \sigma \left(x\right)f_i\left(x\right)\hfill \\ +\frac{1}{4}\left({\tilde{w}}^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^{T}w+{\left(w-\tilde{w}\right)}^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^T\tilde{w}\right)\hfill \end{array}$$

(40)

$$\begin{array}{c}{e}_{HJB}-e_{NN}=-{\tilde{w}}^T\nabla \sigma \left(x\right)f_i\left(x\right)\hfill \\ +\frac{1}{2}{\tilde{w}}^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^{T}w-\frac{1}{4}{\tilde{w}}^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^T\tilde{w}\hfill \end{array}$$

(41)

Because $u^{\ast }=-\frac{1}{2}{R}^{-1}{\left(g_i\left(x\right)\right)}^T\left({\left(\nabla \sigma \left(x\right)\right)}^{T}w+\nabla \epsilon \left(x\right)\right)$, we have:

$$\begin{array}{c}{e}_{HJB}-e_{NN}=-{\tilde{w}}^T\nabla \sigma \left(x\right)f_i\left(x\right)+{\tilde{w}}^T\nabla \sigma \left(x\right)g_i\left(x\right)\left(-u^{\ast }-\frac{1}{2}{R}^{-1}{g}_i{\left(x\right)}^T\nabla \epsilon \left(x\right)\right)\hfill \\ -\frac{1}{4}{\tilde{w}}^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^T\tilde{w}\hfill \end{array}$$

(42)

$$\begin{array}{c}{e}_{HJB}=e_{NN}-{\tilde{w}}^T\nabla \sigma \left(x\right)\left(f_i\left(x\right)+g_i{u}^{\ast }\right)\hfill \\ -\frac{1}{2}{\tilde{w}}^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \epsilon \left(x\right)-\frac{1}{4}{\tilde{w}}^T\nabla \sigma \left(x\right)g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\nabla \sigma {\left(x\right)}^T\tilde{w}\hfill \end{array}$$

(43)

Assumption 3.

$∥f_i\left(x\right)+g_i\left(x\right)u^{\ast }∥\le {\mu }_{max}$

Define: ${\mu }_i=f_i\left(x\right)+g_i\left(x\right)u^{\ast };G_i=g_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T;\nabla \sigma =\nabla \sigma \left(x\right);\nabla \epsilon =\nabla \epsilon \left(x\right)$

From (35) we obtain:

$${\dot{V}}_1\left(t\right)=-{\tilde{w}}^T\left(-e_{NN}+{\tilde{w}}^T\nabla \sigma {\mu }_i+\frac{1}{2}{\tilde{w}}^T\nabla \sigma G_i\nabla \epsilon +\frac{1}{4}{\tilde{w}}^T\nabla \sigma G_i\nabla {\sigma }^T\tilde{w}\right)\nabla \sigma \left(x\right)\left({\mu }_i+\frac{1}{2}{G}_i\left(\nabla {\sigma }^T\tilde{w}+\nabla \epsilon \right)\right)$$

(44)

$$\begin{array}{c}{\dot{V}}_1\left(t\right)=-\frac{1}{8}{\tilde{w}}^T\left({\tilde{w}}^T\nabla \sigma G_i\nabla {\sigma }^T\tilde{w}\right)\nabla \sigma \left(G_i\nabla {\sigma }^T\tilde{w}\right)\hfill \\ -\left(\frac{1}{2}{\tilde{w}}^T\left({\tilde{w}}^T\nabla \sigma {\mu }_i\right)\nabla \sigma \left(G_i\nabla {\sigma }^T\tilde{w}\right)+\frac{1}{4}{\tilde{w}}^T\left({\tilde{w}}^T\nabla \sigma G_i\nabla \epsilon \right)\nabla \sigma \left(G_i\nabla {\sigma }^T\tilde{w}\right)+\frac{1}{4}{\tilde{w}}^T\left({\tilde{w}}^T\nabla \sigma G_i\nabla {\sigma }^T\tilde{w}\right)\nabla \sigma {\mu }_i\right)\hfill \\ -{\tilde{w}}^T\left({\tilde{w}}^T\nabla \sigma {\mu }_i+\frac{1}{2}{\tilde{w}}^T\nabla \sigma G_i\nabla \epsilon \right)\nabla \sigma {\mu }_i+\frac{1}{2}{\tilde{w}}^T{e}_{NN}\nabla \sigma \left(G_i\nabla {\sigma }^T\tilde{w}\right)+{\tilde{w}}^T{e}_{NN}\nabla \sigma {\mu }_i\hfill \end{array}$$

(45)

$$\begin{array}{c}{\dot{V}}_1\left(t\right)=-\frac{1}{8}{\left({\tilde{w}}^T\nabla \sigma G_i\nabla {\sigma }^T\tilde{w}\right)}^2\hfill \\ -\left[\frac{1}{2}\left({\tilde{w}}^T\nabla \sigma {\mu }_i\right)+\frac{1}{4}\left({\tilde{w}}^T\nabla \sigma G_i\nabla \epsilon \right)+\frac{1}{4}\left({\tilde{w}}^T\nabla \sigma {\mu }_i\right)+\frac{1}{2}{e}_{NN}\right]\left({\tilde{w}}^T\nabla \sigma G_i\nabla {\sigma }^T\tilde{w}\right)\hfill \\ -{\left({\tilde{w}}^T\nabla \sigma {\mu }_i\right)}^2+\left[e_{NN}-\frac{1}{2}\left({\tilde{w}}^T\nabla \sigma G_i\nabla \epsilon \right)\right]\left({\tilde{w}}^T\nabla \sigma {\mu }_i\right)\hfill \end{array}$$

(46)

$$\begin{array}{c}{\dot{V}}_1\left(t\right)=-\frac{1}{8}{\left({\tilde{w}}^T\nabla \sigma G_i\nabla {\sigma }^T\tilde{w}\right)}^2\hfill \\ -\left[\frac{3}{4}\left({\tilde{w}}^T\nabla \sigma {\mu }_i\right)+\frac{1}{4}\left({\tilde{w}}^T\nabla \sigma G_i\nabla \epsilon \right)+\frac{1}{2}{e}_{NN}\right]\left({\tilde{w}}^T\nabla \sigma G_i\nabla {\sigma }^T\tilde{w}\right)-{\left({\tilde{w}}^T\nabla \sigma {\mu }_i\right)}^2\hfill \\ +\left[e_{NN}-\frac{1}{2}\left({\tilde{w}}^T\nabla \sigma G_i\nabla \epsilon \right)\right]\left({\tilde{w}}^T\nabla \sigma {\mu }_i\right)\hfill \end{array}$$

(47)

Define: $\begin{array}{cc}& A={\tilde{w}}^T\nabla \sigma G_i\nabla {\sigma }^T\tilde{w};B=\frac{3}{4}\left({\tilde{w}}^T\nabla \sigma {\mu }_i\right)+\frac{1}{4}\left({\tilde{w}}^T\nabla \sigma G_i\nabla \epsilon \right)+\frac{1}{2}{e}_{NN};\hfill \\ & C={\tilde{w}}^T\nabla \sigma {\mu }_i;D=e_{NN}-\frac{1}{2}\left({\tilde{w}}^T\nabla \sigma G_i\nabla \epsilon \right)\hfill \end{array}$

It is obvious that:

$$\begin{array}{cc}\hfill & {\dot{V}}_1\left(t\right)=-\frac{1}{8}{A}^2-BA-C^2-DC=-\frac{1}{8}{\left(A+4B\right)}^2+2B^2-{\left(C+\frac{D}{2}\right)}^2+\frac{D^2}{4}\hfill \\ \hfill & -\frac{1}{8}\left[{\left(A+4B\right)}^2-\left(16B^2+2D^2\right)\right]\hfill \end{array}$$

(48)

We have:

$$A+4B\ge {∥\tilde{w}∥}^2{\left(G_{min}\right)}^2{\lambda }_{min}\left(R^{-1}\right){\left(\nabla {\sigma }_{min}\right)}^2-∥\tilde{w}∥\left(3\nabla {\sigma }_{max}{\mu }_{max}+\nabla {\sigma }_{max}{\left(G_{max}\right)}^2{\lambda }_{max}\left(R^{-1}\right)\nabla {\epsilon }_{max}\right)-2e_{max}$$

(49)

$$\begin{array}{c}16B^2+2D^2\le {\left(\begin{array}{c}∥\tilde{w}∥\left(3\nabla {\sigma }_{max}{\mu }_{max}+\nabla {\sigma }_{max}{\left(G_{max}\right)}^2{\lambda }_{max}\left(R^{-1}\right)\nabla {\epsilon }_{max}\right)\hfill \\ +2e_{max}\hfill \end{array}\right)}^2\hfill \\ +2{\left(e_{max}+\frac{1}{2}∥\tilde{w}∥\nabla {\sigma }_{max}\nabla {\sigma }_{max}{\left(G_{max}\right)}^2{\lambda }_{max}\left(R^{-1}\right)\nabla {\epsilon }_{max}\right)}^2\hfill \end{array}$$

(50)

According to (49) and (50), the inequality can be obtained

$${\left(A+4B\right)}^2-\left(16B^2+2D^2\right)\ge {\pi }_1$$

It can be obtained that ${\pi }_1>0$ with a big enough variable $∥\tilde{w}∥$ and the highest order coefficient ${\left({\left(G_{min}\right)}^2{\lambda }_{min}\left(R^{-1}\right){\left(\nabla {\sigma }_{min}\right)}^2\right)}^2>0$. Therefore, we can determine the positive number ${\vartheta }_1$ such that: $\forall ∥\tilde{w}∥>{\vartheta }_1$, we imply that ${\left(A+4B\right)}^2-\left(16B^2+2D^2\right)\ge {\pi }_1$, and from (48) we obtain: ${\dot{V}}_1\left(t\right)\le -{\pi }_1$.

Regarding the term of $V_2\left(t\right)$, from (20) we compute the derivative of them:

$$\begin{array}{l}{\dot{V}}_2={\left(\nabla V^{\ast }\right)}^T\left(f_i+g_i\left(\hat{u}+\Delta \right)\right)\hfill \\ =-\left(x^{T}Qx+\lambda {\rho }^2\left(x\right)\right)+\frac{1}{4}{\left(\nabla V^{\ast }\right)}^T{g}_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\left(\nabla V^{\ast }\right)+{\left(\nabla V^{\ast }\right)}^T{g}_i\left(-\frac{1}{2}{R}^{-1}{\left(g_i\left(x\right)\right)}^T\nabla \sigma {\left(x\right)}^T\hat{w}+\Delta \right)\hfill \\ =-\left(x^{T}Qx+\lambda {\rho }^2\left(x\right)\right)-\frac{1}{4}{\left(\nabla V^{\ast }\right)}^T{g}_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\left(\nabla V^{\ast }\right)\hfill \\ +\frac{1}{2}{\left(\nabla V^{\ast }\right)}^T{g}_i{R}^{-1}{\left(g_i\left(x\right)\right)}^T\left(\nabla V^{\ast }-\nabla \sigma {\left(x\right)}^T\hat{w}\right)+{\left(\nabla V^{\ast }\right)}^T{g}_i\Delta \hfill \\ =-\left(x^{T}Qx+\lambda {\rho }^2\left(x\right)\right)-\frac{1}{4}{\left(\nabla V^{\ast }\right)}^T{g}_i\left(x\right)R^{-1}{g}_i{\left(x\right)}^T\left(\nabla V^{\ast }\right)\hfill \\ +\frac{1}{2}{\left(\nabla V^{\ast }\right)}^T{g}_i{R}^{-1}{\left(g_i\left(x\right)\right)}^T\left(\nabla \sigma {\left(x\right)}^T\tilde{w}+\nabla \epsilon \left(x\right)\right)+{\left(\nabla V^{\ast }\right)}^T{g}_i\Delta \hfill \\ =-\left(x^{T}Qx+\lambda {\rho }^2\left(x\right)\right)-\frac{1}{4}{\left(\nabla V^{\ast }\right)}^T{g}_i{R}^{-1}{g_i}^T\left(\nabla V^{\ast }\right)+\frac{1}{2}{\left(\nabla V^{\ast }\right)}^T{g}_i{R}^{-1}{g_i}^T\left(\nabla \sigma {\left(x\right)}^T\tilde{w}+\nabla \epsilon \left(x\right)\right)\hfill \\ +{\left(\nabla V^{\ast }\right)}^T{g}_i\Delta \hfill \end{array}$$

(51)

It can be considered that $\rho \left(x\right)=\varpi ∥x∥$. According to (51), we obtain:

$${\dot{V}}_2\le -\left({\lambda }_{min}\left(Q\right)+\lambda \varpi \right){∥x∥}^2+{\theta }^2$$

(52)

with

$${\theta }^2=-\frac{1}{4}{\left(\nabla V^{\ast }\right)}^T{g}_i{R}^{-1}{g}_i^T\left(\nabla V^{\ast }\right)+\frac{1}{2}{\left(\nabla V^{\ast }\right)}^T{g}_i{R}^{-1}{g}_i^T\left(\nabla \sigma {\left(x\right)}^T\tilde{w}+\nabla \epsilon \left(x\right)\right)+{\left(\nabla V^{\ast }\right)}^T{g}_i\Delta$$

(53)

According to the assumption 1 and assumption 2, it follows that:

$$\begin{array}{c}{\theta }^2\le \frac{1}{4}{\left(w_{max}\nabla {\sigma }_{max}+\nabla {\epsilon }_{max}\right)}^2{g}_{max}^2{\lambda }_{max}\left(R^{-1}\right)+\frac{1}{2}{\left(\vartheta \nabla {\sigma }_{max}+\nabla {\epsilon }_{max}\right)}^2{g}_{max}^2{\lambda }_{max}\left(R^{-1}\right)\hfill \\ +\left(w_{max}\nabla {\sigma }_{max}+\nabla {\epsilon }_{max}\right)g_{max}\varpi ∥x∥\hfill \end{array}$$

(54)

It can be seen that $\left({\lambda }_{min}\left(Q\right)+\lambda \varpi \right){∥x∥}^2-{\theta }^2\ge {\pi }_2$ with ${\pi }_2>0$ also leads to other inequalities, including the polynomial quadratic form with the variable $∥x∥$ and the highest order coefficient $\left({\lambda }_{min}\left(Q\right)+\lambda \varpi \right)>0$. Hence, we can find the positive number ${\vartheta }_2$ such that: $\forall ∥x∥>{\vartheta }_2$ we get $\left({\lambda }_{min}\left(Q\right)+\lambda \varpi \right){∥x∥}^2-{\theta }^2\ge {\pi }_2$, and from (52) we obtain

$${\dot{V}}_2\left(t\right)\le -{\pi }_2.$$

 □

Remark 3.

The numbers ${\vartheta }_1;{\vartheta }_2$ can be changed by establishing the neural network of the optimal cost function. Additionally, for any switching index, the variable $∥x∥$ and $∥\tilde{w}∥$ converge to the specified domains. The approximate optimal control law $\hat{u}$ is considered in the equation as (27), which converges to the neighbourhood of the optimal control as $u^{\ast }$. Different from the controller in [33], the switched system with unknown switching signal is mentioned in this work by proposed adaptive optimal control.

Proof. 

It should be noted that according to (34), we imply that:

$$∥\hat{u}-u^{\ast }∥=\frac{1}{2}∥R^{-1}{\left(g_i\left(x\right)\right)}^T\left({\left(\nabla \sigma \left(x\right)\right)}^T\tilde{w}+\nabla \epsilon \left(x\right)\right)∥\le \frac{1}{2}{\lambda }_{max}\left(R^{-1}\right).G_{max}.\left(\nabla {\sigma }_{max}.{\upsilon }_1+\nabla {\epsilon }_{max}\right)={\vartheta }_3$$

(55)

 □

4. Simulation Results

In this section, we verify the effectiveness and performance of the proposed controller. The ARL algorithm is developed for solving the optimal and tracking problem by RBF network-based ADP, where a single critic Neural Network is used to approximate the Bellman function. The ARL control law and the weight of the Critic part are established as in (27) and (30) with the appropriate learning rate $\alpha$ and the coefficient $\lambda$. Moreover, to carry out the ARL algorithm (27) and (30), we need to employ the term of model $g_i\left(x\right)$, the term of cost function $Q,R$ and the appropriate function $\sigma \left(x\right)$ in Neural Network. To verify the proposed method of our algorithm, two different situations are implemented to do simulations with all the parameters and functions being listed in each case as follows:

4.1. The Second-Order Switched Nonlinear Systems

In this simulation experiment, we consider the case of switched systems including $N=2$ subsystems to be described as Equations (56) and (57):

$$\left\{\begin{array}{c}{\dot{x}}_1=-x_1^3-2x_2+\left(u+{\Delta }_1\left(x,t\right)\right)\hfill \\ {\dot{x}}_2=x_1+0.5cos\left(x_1^2\right)sin\left(x_2^3\right)-\left(u+{\Delta }_1\left(x,t\right)\right)\hfill \end{array}\right.$$

(56)

$$\left\{\begin{array}{c}{\dot{x}}_1=-x_1^{5}sin\left(x_2\right)+\left(u+{\Delta }_2\left(x,t\right)\right)\hfill \\ {\dot{x}}_2=\frac{1}{2}{x}_1-cos\left(x_1\right)cos\left(x_2^3\right)-\left(u+{\Delta }_2\left(x,t\right)\right)\hfill \end{array}\right.$$

(57)

The initial value of state vector is selected as

$$x\left(0\right)={\left[\begin{array}{cc}5& -5\end{array}\right]}^T$$

and choosing the parameters and matrices in cost function as follows: $R=\left[\begin{array}{cc}2& 0\\ 0& 2\end{array}\right];Q=\left[\begin{array}{cc}1& 0\\ 0& 3\end{array}\right];\alpha =0.1;\lambda =5$. Additionally, the function $\sigma \left(x\right)$ in RBF Neural Network can be chosen as $\sigma \left(x\right)={\left[x_1^2,x_1{x}_2,x_2^2\right]}^T$.

Under the arbitrary switching law (Figure 1), the results are shown in Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6 by using proposed ARL law, in which the responses of state variables are described in Figure 2 and Figure 3 under the control input (Figure 5) and the weights are trained as in Figure 6. It is obvious that the closed system is stable due to the convergence of state variables to zero (Figure 2 and Figure 3). On the other hand, the convergence is also shown in the weights of Critic part (Figure 6).

Remark 4.

The second-order switched nonlinear system was also mentioned in [14] by using nonlinear control law after the equivalent transform operation (Figure 1, Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6). However, it is noteworthy that the response of control input of proposed method has a better transient and steady performance compared with the nonlinear controller in [14] because of the effectiveness of handling the constraint in optimal control (Figure 4 and Figure 5).

4.2. The Third-Order Switched Nonlinear Systems

Here, we continue to investigate the case of switched systems including $N=2$ subsystems as follows:

$$\left\{\begin{array}{c}\dot{\mathbf{x}}=\left[\begin{array}{c}-x_1+x_2\hfill \\ -0.2x_2-sin{x}_{1}cos{x}_3\hfill \\ x_1{x}_2-sin{x}_3\hfill \end{array}\right]+\left[\begin{array}{c}1\hfill \\ -1\hfill \\ -2\hfill \end{array}\right]u+{\Delta }_{\mathbf{1}}\left(\mathbf{x},t\right)\hfill \\ \dot{\mathbf{x}}=\left[\begin{array}{c}-x_1+x_2^2{x}_3\hfill \\ -x_2-sin\left(x_{1}cos{x}_3\right)\hfill \\ sin{x}_2-x_3\hfill \end{array}\right]+\left[\begin{array}{c}1\hfill \\ -1\hfill \\ -2\hfill \end{array}\right]u+{\Delta }_{\mathbf{2}}\left(\mathbf{x},t\right)\hfill \end{array}\right.$$

(58)

where $x_0={\left[5;-3;2\right]}^T;{\Delta }_i\left(\mathbf{x},t\right)=\frac{1}{10}\left[\begin{array}{c}{d}_1{x}_1\hfill \\ d_2{x}_2\hfill \\ d_3{x}_3\hfill \end{array}\right];∥{\Delta }_i∥\le \frac{1}{10}∥x∥$ with $d_1,d_2,d_3$ are the random functions. The parameters of proposed control design can be chosen as $R=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right];\alpha =0.01;Q=\left[\begin{array}{ccc}1& 0& 0\\ 0& 2& 0\\ 0& 0& 3\end{array}\right];\lambda =5$. Moreover, the function $\sigma \left(x\right)$ in RBF Neural Network can be chosen as $\sigma \left(x\right)={\left[x_1^2,x_2^2,x_3^2,x_1{x}_2,x_2{x}_3,x_1{x}_3\right]}^T$. In this case, under the arbitrary switching law in Figure 7, the results are obtained in Figure 8, Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13 with the responses of state variables are shown in Figure 8, Figure 9 and Figure 10 based on the control input (Figure 11 and Figure 12), and the training weights are shown in Figure 13. It can be seen that the closed system is stable (Figure 8, Figure 9 and Figure 10) and the convergence is also shown in the weights of Critic part (Figure 13).

Remark 5.

It is worth noting that the third-order switched nonlinear system was also considered in [15] based on the adaptive backstepping output feedback nonlinear control scheme combined with the transform operation obtaining the response of state variables, control input in Figure 8, Figure 9, Figure 10 and Figure 11 and adaptation law (Figure 14). However, thanks to the property of optimal control, the control input response of the proposed controller has a better transient compared with the adaptive backstepping nonlinear controller in [15] (Figure 11 and Figure 12).

5. Conclusions

This paper investigated the optimal control design for perturbed switched nonlinear systems based on the adaptive dynamic programming technique. The optimal control is first designed under the consideration of nominal systems. Then, the ADP technique is developed by using the Neural Networks. Due to the description of nonlinear systems, as well as unknown switch index, the neural network is considered to approximate the critic part of the iterative algorithm. Moreover, the UUB stability problem of the closed-loop system and the convergence of weight training are guaranteed under this solution. Finally, two simulation examples are given to verify the effectiveness of the presented ARL algorithm.