Cooperative Differential Game-Based Modular Unmanned System Approximate Optimal Control: An Adaptive Critic Design Approach

Abstract

An approximate optimal control issue for modular unmanned systems (MUSs) is presented via a cooperative differential game for solving the trajectory tracking problem. Initially, the modular unmanned system’s dynamic model is built with the joint torque feedback technique. The moment of inertia of the motor rotor has positive symmetry. Each MUS module is deemed as a participant in the cooperative differential game. Then, the MUS trajectory tracking problem is transformed into an approximate optimal control problem by means of adaptive critic design (ACD). The approximate optimal control is obtained by the critic network, approaching the joint performance index function of the system. The stability of the closed-loop system is proved through Lyapunov theory. The feasibility of the proposed control algorithm is verified by an experimental platform.

1. Introduction

Unmanned systems [1,2] refer to collections of intelligent equipment that realize unmanned or semi-autonomous operation through advanced sensing, artificial intelligence, autonomous navigation, and other technologies, including unmanned aerial vehicles [3], unmanned vehicles [4], unmanned ships [5], robots [6], and cross-domain collaborative unmanned cluster systems. Their core goals are to improve task efficiency, reduce human risk, and push the limits of human physiology in complex, dangerous, or repetitive scenarios through robot replacement or human–robot collaboration. However, traditional unmanned systems usually have fixed configurations, thus limiting their further application. In order to solve the above challenges, modular unmanned systems (MUSs) [7] have been proposed and applied to many aspects, such as autonomous assembly of large-aperture telescopes in space via the handling, assembly, and connection of trusses and lenses. In addition, modular reconfigurable robots use the same ideas as MUSs.

An ideal MUS should not only ensure its stability and accuracy under control, but also consider its optimality. Adaptive critic design (ACD) [8,9], as a branch of optimal control, is a cutting-edge method at the intersection of machine learning, control theory, and operations research which aims to solve the real-time optimization and decision problems of complex dynamic systems. Through the integration of reinforcement learning, neural networks, and dynamic programming, it overcomes the dependence of traditional dynamic programming on precise mathematical models, and realizes online learning and adaptive adjustment of strategies in unknown or time-varying environments, known as the intelligent optimization brain of dynamic systems. Adaptive critic design has many applications, such as in continuous-time systems [10,11], discrete-time systems [12,13], data-driven systems [14,15], event-triggered systems [16,17], fault diagnosis systems [18,19], and model uncertain system [20,21]. However, most of the systems mentioned above have a single controller, and with the gradual expansion of the system scale, multi-controller or multi-player systems should be further considered to optimize overall system performance.

Differential game theory [22,23] is a frontier field combining game theory and dynamic system theory, focusing on the strategic interaction between multiple decision-makers under continuous time frames. Different from traditional static games, differential games model the strategy selection of the participants as a time-dependent state variable control process, and describe the dynamic evolution of the system state through differential equations. Each participant adjusts their control strategy according to real-time information. In the process of pursuing the optimization of individual objective functions, it is necessary for one player to predict the influence of the opponent’s behavior on the system trajectory and consider the chain reaction of its own strategy on the global dynamics. There are many types of games, such as zero-sum games [24,25], non-zero-sum games [26,27], etc. Cooperative games [28,29], as special non-zero-sum games, have only one performance index, and each player hopes to optimize the overall performance index function of the system through cooperation. Therefore, the control problem of MUSs is suitable to be solved by the concept of cooperative differential games.

This paper proposes an approximate optimal control method based on cooperative differential games for MUSs and addresses the trajectory tracking problem. First, a dynamic model is established. Then, each module of the MUS is treated as a participant in a cooperative differential game, transforming the trajectory tracking problem into an approximate optimal control problem based on a cooperative differential game via ACD. A critic network is employed to approximate the system’s joint performance index function, and control law is derived through a policy iteration algorithm. The stability is proven using Lyapunov theory. The effectiveness of the proposed algorithm is validated through an experimental platform. This paper’s contributions are as follows:

• It is the first paper to use cooperative differential games for MUSs via ACD to guarantee accuracy and optimality. The developed control method is verified on the actual platform.
• The experimental results are verified via tracking error and control torque under the developed cooperative game method using ACD.

Notation

x is the global state of the MUS; E is the position tracking error of the MUS; U is the global optimal control; W is the global coupled strategy; $Q,R,P$ are a given positive definite matrix; $U_{lti}(\dot{E},U,W)$ is the utility function; and $V_{al}(\dot{E},U,W)$ is the cost function.

2. Dynamic Model of MUS

According to the dynamic modeling method of MUS based on the JTF [26], we obtain

$$I_{moi}{\gamma }_{rai}{\ddot{\theta }}_i+f_{lui}({\theta }_i,{\dot{\theta }}_i)+I_{dci}(\theta ,\dot{\theta },\ddot{\theta })+{\displaystyle \frac{{\tau }_{coi}}{{\gamma }_{rai}}}={\tau }_i,$$

(1)

where $I_{moi}$ is the moment of inertia of the motor rotor in relation to the rotating shaft; ${\gamma }_{rai}$ is the reduction ratio of the motor; ${\theta }_i,{\dot{\theta }}_i$, and ${\ddot{\theta }}_i$ are the position vector, velocity vector, and acceleration vector of the joint of the ith subsystem of the MUS; $f_{lui}({\theta }_i,{\dot{\theta }}_i)$ is the concentrated joint friction torque; $I_{dci}(\theta ,\dot{\theta },\ddot{\theta })$ is the coupling term between joint subsystems; ${\tau }_{coi}$ is the coupling joint torque measurement; and ${\tau }_i$ indicates control torque.

The joint module of the MUS is shown in Figure 1.

Property 1.

The moment of inertia of the motor rotor $I_{moi}$ has positive symmetry. It will be useful in the stability analysis.

(1)

Coupling joint torque measurement ${\tau }_{coi}$

Coupling joint torque ${\tau }_{coi}$ is measured and mainly consists of

$${\tau }_{coi}={\tau }_{cofi}+{\tau }_{coci},$$

(2)

where ${\tau }_{cofi}$ is the joint torque measured in free space, and ${\tau }_{coci}$ represents the value of the external torque generated by the continuous or instantaneous contact environment. Additionally, the joint torque values in the constrained space ${\tau }_{coi}$ and free space ${\tau }_{cofi}$ are easily found; therefore, the external torque ${\tau }_{coci}$ can be obtained by Formula (2).

(2)

Concentrated joint friction torque $f_{lui}({\theta }_i,{\dot{\theta }}_i)$

Concentrated joint friction torque $f_{lui}({\theta }_i,{\dot{\theta }}_i)$ reflects the friction torque on the DC motor and reducer of the joint module. $f_{lui}({\theta }_i,{\dot{\theta }}_i)$ is composed of nonlinear functions related to joint position and velocity:

$$f_{lui}({\theta }_i,{\dot{\theta }}_i)=f_{bvi}{\dot{\theta }}_i+\left(f_{coi}+f_{sti}{e}^{(-f_{\tau si}{\dot{\theta }}_i^2)}\right)sgn\left({\dot{\theta }}_i\right)+f_{pdi}({\theta }_i,{\dot{\theta }}_i),$$

(3)

where $f_{bvi},f_{coi},f_{sti}$, and $f_{\tau si}$ represent the viscous, Coulomb, static, and Stribeck friction of subsystem module i, respectively; $f_{pdi}({\theta }_i,{\dot{\theta }}_i)$ represents position-dependent friction term; and $sgn\left({\dot{\theta }}_i\right)$ is a symbolic function.

According to linearization criteria, $f_{pdi}$ is approximated by the following formula:

$${\widehat{f}}_{lui}({\theta }_i,{\dot{\theta }}_i)={\widehat{f}}_{bvi}{\dot{\theta }}_i+\left({\widehat{f}}_{coi}+{\widehat{f}}_{sti}{e}^{(-{\widehat{f}}_{\tau si}{\dot{\theta }}_i^2)}\right)sgn\left({\dot{\theta }}_i\right)+f_{pdi}({\theta }_i,{\dot{\theta }}_i)+K\left({\dot{\theta }}_i\right){\tilde{F}}_{pui},$$

(4)

where ${\widehat{f}}_{bvi},{\widehat{f}}_{coi},{\widehat{f}}_{sti}$, and ${\widehat{f}}_{\tau si}$ are the approximate approximation values of $f_{bvi},f_{coi},f_{sti}$, and $f_{\tau si}$; ${\tilde{F}}_{pui}={\left[\begin{array}{cccc}{f}_{bvi}-{\widehat{f}}_{bvi}& f_{coi}-{\widehat{f}}_{coi}& f_{sti}-{\widehat{f}}_{sti}& f_{\tau si}-{\widehat{f}}_{\tau si}\end{array}\right]}^T$ is the parameter uncertainty of friction torque; and $K\left({\dot{\theta }}_i\right)=\left[{\dot{\theta }}_i,sgn\left({\dot{\theta }}_i\right),e^{(-{\widehat{f}}_{\tau si}{\dot{\theta }}_i^2)}sgn\left({\dot{\theta }}_i\right),-{\widehat{f}}_{sti}{\dot{\theta }}_i^2{e}^{(-{\widehat{f}}_{\tau si}{\dot{\theta }}_i^2)}sgn\left({\dot{\theta }}_i\right)\right].$

Remark 1.

Friction torque parameters $f_{bvi},f_{coi},f_{sti},f_{\tau si}$ and their estimates ${\widehat{f}}_{bvi},{\widehat{f}}_{coi},{\widehat{f}}_{sti},{\widehat{f}}_{\tau si}$ are uniformly bounded, and the upper bound of ${\tilde{F}}_{pui}$ can be defined as $∥{\tilde{F}}_{pui}∥\le {\rho }_{puik}(k=1,2,3,4)$, where ${\rho }_{puik}$ is a known upper bound constant. Therefore, the friction modeling error $K\left({\dot{\theta }}_i\right){\tilde{F}}_{pui}$ has the following relationship $∥K\left({\dot{\theta }}_i\right){\tilde{F}}_{pui}∥\le ∥K\left({\dot{\theta }}_i\right)∥{\rho }_{puik}$. In addition, the upper bound of the position-dependent friction term $f_{pdi}$ is $∥f_{pdi}({\theta }_i,{\dot{\theta }}_i)∥\le {\rho }_{fpdi}$, where ${\rho }_{fpdi}$ is a known upper bound constant.

(3)

Coupling term $I_{dci}(\theta ,\dot{\theta },\ddot{\theta })$

The coupling term $I_{dci}(\theta ,\dot{\theta },\ddot{\theta })$, related to the coupling dynamics of the robot’s global vector, is

$$\begin{array}{c}{I}_{dci}(\theta ,\dot{\theta },\ddot{\theta })=I_{moi}\sum _{k=1}^{i-1}{a}_{moi}^T{a}_{lnk}{\ddot{\theta }}_k+I_{moi}\sum _{k=2}^{i-1}\sum _{m=1}^{k-1}{a}_{moi}^T(a_{lnm}\times a_{lnk}){\dot{\theta }}_m{\dot{\theta }}_k\hfill \\ =I_{moi}\sum _{k=1}^{i-1}{\Lambda }_k^i{\ddot{\theta }}_k+I_{moi}\sum _{k=2}^{i-1}\sum _{m=1}^{k-1}{\Omega }_{mk}^i{\dot{\theta }}_m{\dot{\theta }}_k\hfill \\ =\sum _{k=1}^{i-1}\left[\begin{array}{cc}{I}_{moi}{\widehat{\Lambda }}_k^i& I_{moi}\end{array}\right]{\left[\begin{array}{cc}{\ddot{\theta }}_k& {\tilde{\Lambda }}_k^i{\ddot{\theta }}_k\end{array}\right]}^T+\sum _{k=2}^{i-1}\sum _{m=1}^{k-1}\left[\begin{array}{cc}{I}_{moi}{\widehat{\Omega }}_{mk}^i& I_{moi}\end{array}\right]{\left[\begin{array}{cc}{\dot{\theta }}_m{\dot{\theta }}_k& {\tilde{\Omega }}_{mk}^i{\dot{\theta }}_m{\dot{\theta }}_k\end{array}\right]}^T,\hfill \end{array}$$

(5)

where $a_{moi}$, $a_{lnm}$, and $a_{lnk}$ are unit vectors around the rotation axes of the ith motor and the m and k joints, respectively; ${\Lambda }_k^i=a_{moi}^T{a}_{lnk}$; and ${\Omega }_{mk}^i=a_{moi}^T(a_{lnm}\times a_{lnk})$. In addition, ${\widehat{\Lambda }}_k^i={\Lambda }_k^i-{\tilde{\Lambda }}_k^i$ and ${\widehat{\Omega }}_{mk}^i={\Omega }_{mk}^i-{\tilde{\Omega }}_{mk}^i$, where ${\widehat{\Lambda }}_k^i$ and ${\widehat{\Omega }}_{mk}^i$ are estimated values of ${\Lambda }_k^i$ and ${\Omega }_{mk}^i$, and ${\tilde{\Lambda }}_k^i$ and ${\tilde{\Omega }}_{mk}^i$ are calibration errors.

Remark 2.

For coupling term $I_{dci}(\theta ,\dot{\theta },\ddot{\theta })$, the dot product between $a_{moi}$, $a_{lnm}$, and $a_{lnk}$ is bounded, and satisfies $∥{\Lambda }_k^i∥=∥a_{moi}^T{a}_{lnk}∥<1$ and $∥{\Omega }_{mk}^i∥=∥a_{moi}^T(a_{lnm}\times a_{lnk})∥<1$. When the k and m$(1<k,m<i-1)$ joints near the base joint are stable, the coupling term $I_{dci}(\theta ,\dot{\theta },\ddot{\theta })$ can be given the following upper bounds: $∥I_{dci}(\theta ,\dot{\theta },\ddot{\theta })∥$$<{\rho }_{dci}$, where ${\rho }_{dci}$ is bounded by a positive number. Therefore, while stabilizing the current joint, the MUS can be stabilized step-by-step, thus making the system globally stable.

Define $x_i={[x_{i1},x_{i2}]}^T={[{\theta }_i,{\theta }_i]}^T$, $u_i={\tau }_i=R^{1\times 1}$. Then, (1) can be converted into a state space equation:

$$\left\{\begin{array}{c}{\dot{x}}_{i1}=x_{i2}\hfill \\ {\dot{x}}_{i2}=f_i(x_{i1},x_{i2})+g_i\left(x_{i1}\right)u_i+{\psi }_i\left(x\right)\hfill \\ y_i=x_{i1}\hfill \end{array}\right.,$$

(6)

where $f_i(x_{i1},x_{i2})$, $g_i\left(x_{i1}\right)$, and ${\psi }_i\left(x\right)$ are the measurable dynamics, control input matrix, and uncertainty of MUS, respectively, and

$$\begin{array}{cc}& g_i\left(x_{i1}\right)={\displaystyle \frac{1}{I_{moi}{\gamma }_{rai}}}\hfill \\ & f_i(x_{i1},x_{i2})=-g_i\left(x_{i1}\right)\left({\widehat{f}}_{bvi}{\dot{\theta }}_i+\left({\widehat{f}}_{coi}+{\widehat{f}}_{sti}{e}^{(-{\widehat{f}}_{\tau si}{\dot{\theta }}_i^2)}\right)sgn\left({\dot{\theta }}_i\right)+f_{pdi}({\theta }_i,{\dot{\theta }}_i)+K\left({\dot{\theta }}_i\right){\tilde{F}}_{pui}+{\displaystyle \frac{{\tau }_{coi}}{{\gamma }_{rai}}}\right)\hfill \\ & {\psi }_i\left(x\right)=-g_i\left(x_{i1}\right)I_{dci}(\theta ,\dot{\theta },\ddot{\theta }).\hfill \end{array}$$

3. Approximate Optimal-Control-Based Cooperative Differential Game via ACD

3.1. Problem Description

In this section, an approximate optimal control method based on a cooperative differential game created via ACD is proposed. In order to ensure the convenience of controller design, the state space equation of the following system is considered:

$$\left\{\begin{array}{c}{\dot{x}}_1=x_2\hfill \\ {\dot{x}}_2=f\left(x\right)+\sum _{m=1}^n{G}_m{u}_m+W\left(x\right)\hfill \\ y=x_1\hfill \end{array}\right.,$$

(7)

where $x={\left[x_1^T,x_2^T\right]}^T\in R^{2n}$ is the global state of the MUS, and vectors $x_1$ and $x_2$, respectively, are expressed by the following formula:

$$x_m={\left[x_{1m},\dots ,x_{im}\dots ,x_{nm}\right]}^T\in R^n,m=1,2.$$

(8)

In addition, the following equation is established:

$$\begin{array}{cc}\hfill & f\left(x\right)={\left[f_1\left(x\right),\dots ,f_i\left(x\right)\dots ,f_n\left(x\right)\right]}^T,\hfill \\ \hfill & G_m={\left[0,\dots ,0,g_m,0,\dots ,0\right]}^T,\hfill \\ \hfill & W\left(x\right)={\left[{\psi }_1\left(x\right),\dots ,{\psi }_i\left(x\right)\dots ,{\psi }_n\left(x\right)\right]}^T,\hfill \end{array}$$

(9)

where $g_m={\left(I_{mom}{\gamma }_{ram}\right)}^{-1}$ and $m=1,2,\dots ,n$. $G_m$ is a positive symmetry matrix.

Define the following cost function:

$$V_{al}(\dot{E},U,W)={\int }_t^{\infty }\left({\dot{E}}^{T}Q\dot{E}+U^{T}RU+W^{T}PW\right)d\tau ={\int }_t^{\infty }{U}_{lt}(\dot{E},U,W)d\tau ,$$

(10)

where E is the error of the MUS, and the velocity error is defined as $\dot{E}=x_2-{\dot{x}}_{1d}$; $x_{1d}$ is the desired trajectory; $Q,R=diag[R_1,R_2,\dots ,R_n],P$ are a given positive definite matrix; and $U_{lti}(\dot{E},U,W)$ is the utility function.

$$H_{am}(\dot{E},U,W)=U_{lti}(\dot{E},U,W)+\left(\nabla V_{al}^T\right)\left(f\left(x\right)+GU+W-{\ddot{x}}_{1d}\right),$$

(11)

where $\nabla V_{al}\left(\dot{E}\right)={\displaystyle \frac{\partial V_{al}\left(\dot{E}\right)}{\partial \dot{E}}}$.

Define the optimal cost function:

$$V_{al}^{*}(\dot{E},U,W)=\underset{U,W}{min}{\int }_0^{\infty }\left(U_{lt}(\dot{E},U,W)\right)d\tau .$$

(12)

According to optimality conditions ${\displaystyle \frac{\partial H_{am}}{\partial U}}=0$ and ${\displaystyle \frac{\partial H_{am}}{\partial W}}=0$, the approximate optimal control strategy is obtained a follows:

$$U^{*}=-{\displaystyle \frac{1}{2}}{R}^{-1}{G}^T\nabla V_{al}^{*}.$$

(13)

$$W^{*}=-{\displaystyle \frac{1}{2}}{P}^{-1}\nabla V_{al}^{*}.$$

(14)

Considering the cooperative differential game represented by Formula (10), each player needs to minimize the cost function of the corresponding coupling and define the control input matrix $G=[G_1,G_2,\dots ,G_n]$; then, the cooperative game problem is transformed to solve the approximate optimal control related to the cost function (10).

Then, Equations (10), (13) and (14) are brought into the Hamiltonian function (11), and the coupled HJB equation is as follows:

$$0=\left(\nabla V_{al}^T\right)\left(f\left(x\right)-{\ddot{x}}_{1d}\right)+{\dot{E}}^{T}Q\dot{E}-{\displaystyle \frac{1}{4}}{(\nabla V_{al}^{*})}^{T}G{P}^{-1}{G}^T(\nabla V_{al}^{*})-{\displaystyle \frac{1}{4}}{(\nabla V_{al}^{*})}^T{P}^{-1}(\nabla V_{al}^{*}).$$

(15)

The cost function can be deduced by Formula (15) $\nabla V_{al}^{*}$, and the corresponding equilibrium solution of the Pareto equation is obtained. However, since Formula (15) is a nonlinear PDE, it is difficult to directly obtain the Pareto equilibrium solution of the system by analytical methods. Therefore, in the next section, the critic NN is used in the MUS so as to obtain the approximate optimal control strategy.

3.2. An Approximate Solution of Decentralized Approximate Optimal Control in a Cooperative Differential Game Based on a Critic Network

Dynamic compensation is important in MUS control, and a controller based on dynamic model compensation, local expectation information, and approximate optimal control law is figured as follows:

$$u_i^{*}=u_{i1}+u_{i2}+u_{i3}^{*},$$

(16)

where $u_{i1}$ is used to process dynamic models $f_i(x_{i1},x_{i2})$, $u_{i2}$ deals with the coupling terms of the MUS using locally desired control information, and $u_{i3}^{*}$ is the optimal compensation for uncertainty.

According to Formula (6), the design control law $u_{i1}$ to compensate a subsystem model that has been accurately modeled and measurable is as follows:

$$u_{i1}=-\left(-{\widehat{f}}_{bvi}{x}_{i2}-\left({\widehat{f}}_{coi}+{\widehat{f}}_{sti}{e}^{(-{\widehat{f}}_{\tau si}{x}_{i2}^2)}\right)sgn\left({\dot{\theta }}_i\right)-g_i^{-1}{\ddot{x}}_{i1d}-{\displaystyle \frac{{\tau }_{coi}}{{\gamma }_{rai}}}\right).$$

(17)

The critic neural network is

$$V_{al}\left(\dot{E}\right)=W_{cr}^T{\delta }_{cr}\left(\dot{E}\right)+{\epsilon }_{cr},$$

(18)

where $W_{cr}$ is the ideal weight vector of the critic neural network; ${\delta }_{cr}\left(\dot{E}\right)$ represents the activation function; and ${\epsilon }_{cr}$ is the finite approximation error.

Taking the partial derivative of (18) yields

$$\nabla V_{al}\left(\dot{E}\right)={\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{W}_{cr}+\nabla {\epsilon }_{cr},$$

(19)

where $\nabla {\delta }_{cr}\left(\dot{E}\right)={\displaystyle \frac{\partial \left({\delta }_{cr}\left(\dot{E}\right)\right)}{\partial \dot{E}}}$ and $\nabla {\epsilon }_{cr}$ are the partial derivatives of the activation function and approximation error.

By substituting Formula (19) into (13) and (14), we get

$$U^{*}=-{\displaystyle \frac{1}{2}}{R}^{-1}{G}^T\left({\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{W}_{cr}+\nabla {\epsilon }_{cr}\right),$$

(20)

$$W^{*}=-{\displaystyle \frac{1}{2}}{P}^{-1}\left({\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{W}_{cr}+\nabla {\epsilon }_{cr}\right).$$

(21)

According to Formulas (20) and (21), each element of the expansion matrix can be obtained as

$$u_i^{*}=-{\displaystyle \frac{1}{2}}{R}_i^{-1}{G}_i^T\left({\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{W}_{cr}+\nabla {\epsilon }_{cr}\right).$$

(22)

$${{\psi }_i}^{*}=-{\displaystyle \frac{1}{2}}{P}_i^{-1}\left({\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{W}_{cr}+\nabla {\epsilon }_{cr}\right).$$

(23)

Substituting Formulas (19), (20), and (21) into (11), we get

$$H_{am}(\dot{E},U,W)={\dot{E}}^{T}Q\dot{E}+U^{T}RU+W^{T}PW+\left(\nabla V_{al}^T\right)\left(f\left(x\right)+GU+W-{\ddot{x}}_{1d}\right)-e_{Jh}=0,$$

(24)

where $e_{Jh}$ is the approximation residual of the neural network.

In order to implement a decentralized control mechanism, the assumption of nominal limitation between coupling terms needs to be eliminated, so that the state of the coupled subsystem can be represented by the expected state of the other subsystems.

$$\begin{array}{cc}\hfill & f_i\left(x\right)=f_i(x_i,x_{md})+\Delta f_i(x,x_{md}),\hfill \\ \hfill & u_{m2}=G_m^{-1}({\dot{x}}_{m2d}-f_m\left(x_d\right)),m\ne i.\hfill \end{array}$$

(25)

Here, $x_{md}$ represents the desired state of the coupled subsystem $m=1,\dots ,i-1,i+1,\dots ,n$, and $\Delta f_i(x,x_{md})$ is substitution error. Since the coupling term satisfies the local Lipschitz condition, there are

$$∥\Delta f_i(x,x_{md})∥\le \sum _{m=1,m\ne i}^n{d}_{im}{E}_m,$$

(26)

where $E_m=∥x_m-x_{md}∥$ and $d_{im}\ge 0$.

The ideal critic neural network weight vector $W_{cr}$ is unknown, but it can be approximated as

$${\widehat{V}}_{al}\left(\dot{E}\right)={\widehat{W}}_{cr}^T{\delta }_{cr}\left(\dot{E}\right),$$

(27)

where ${\widehat{W}}_{cr}$ and ${\widehat{V}}_{al}\left(\dot{E}\right)$ are estimated values of $W_{cr}$ and $V_{al}\left(\dot{E}\right)$.

Taking the partial derivative of (27) yields

$$\nabla {\widehat{V}}_{al}\left(\dot{E}\right)={\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{\widehat{W}}_{cr}.$$

(28)

Combined with Formulas (22), (23), and (28), the approximate decentralized optimal control laws are as follows:

$${\widehat{u}}_{i3}^{*}=-{\displaystyle \frac{1}{2}}{R}_i^{-1}{G}_i^T\left({\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{\widehat{W}}_{cr}\right).$$

(29)

$${\widehat{\psi }}_i^{*}=-{\displaystyle \frac{1}{2}}{P}_i^{-1}\left({\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{\widehat{W}}_{cr}\right).$$

(30)

Define $F\left(x\right)=\left[f_1(x_1,x_{md}),f_2(x_2,x_{md}),\dots ,f_n(x_n,x_{md})\right]$. According to Formula (24), the approximate Hamiltonian function can be obtained:

$${\widehat{H}}_{am}(\dot{E},\widehat{U},\widehat{W})={\dot{E}}^{T}Q\dot{E}+{\widehat{U}}^{T}R\widehat{U}+{\widehat{W}}^{T}P\widehat{W}+\left(\nabla V_{al}^T\right)\left(F\left(x\right)+G\widehat{U}+\widehat{W}-{\ddot{x}}_{1d}\right)=e_J.$$

(31)

The approximation error of the Hamiltonian function is

$$e_J={\widehat{H}}_{am}-H_{am},$$

(32)

where $e_J={\widehat{H}}_{am}$ is obtained by Formulas (24) and (31).

The weight approximation error is defined as

$${\tilde{W}}_{cr}=W_{cr}-{\widehat{W}}_{cr}.$$

(33)

According to (24), (31), and (32), we can deduce

$$e_J=e_{Jh}-{\tilde{W}}_{cr}^T\nabla {\delta }_{cr}\left(\dot{E}\right)\ddot{E}.$$

(34)

Define the following objective function:

$$E_J={\displaystyle \frac{1}{2}}{e}_J^T{e}_J.$$

(35)

The weight update law for critic NNs is obtained as follows:

$${\dot{\widehat{W}}}_{cr}=-{\alpha }_{le}{e}_J\nabla {\delta }_{cr}\left(\dot{E}\right)\ddot{E},$$

(36)

where ${\alpha }_{le}$ is the critic neural network learning law. Define ${\upsilon }_{au}=\nabla {\delta }_{cr}\left(\dot{E}\right)\ddot{E}$, and assume that normal numbers for ${\upsilon }_{Lb}$ satisfy $∥{\nu }_{au}∥\le {\nu }_{Lb}$. The dynamic equation for weight approximation error can be deduced as follows:

$${\dot{\tilde{W}}}_{cr}=-{\dot{\widehat{W}}}_{cr}={\alpha }_{le}{e}_J{\upsilon }_{au}={\alpha }_{le}(e_{Jh}-{\tilde{W}}_{cr}^T{\upsilon }_{au}){\upsilon }_{au}.$$

(37)

Finally, according to Formulas (17), (25), and (29), the decentralized approximate optimal control law based on the cooperative differential game via ACD is obtained as follows:

$$\begin{array}{cc}\hfill & {\widehat{u}}_i^{*}=u_{i1}+u_{i2}+{\widehat{u}}_{i3}^{*}=-\left(-{\widehat{f}}_{bvi}{x}_{i2}-\left({\widehat{f}}_{coi}+{\widehat{f}}_{sti}{e}^{(-{\widehat{f}}_{\tau si}{x}_{i2}^2)}\right)sgn\left({\dot{\theta }}_i\right)-g_i^{-1}{\ddot{x}}_{i1d}-{\displaystyle \frac{{\tau }_{coi}}{{\gamma }_{rai}}}\right)\hfill \\ \hfill & +G_m^{-1}({\dot{x}}_{m2d}-f_m\left(x_d\right))-{\displaystyle \frac{1}{2}}{R}_i^{-1}{G}_i^T\left({\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{\widehat{W}}_{cr}\right).\hfill \end{array}$$

(38)

3.3. Fulfillment of Policy Iteration

In order to solve the difficulty of the HJB equation, a policy iteration method is proposed.

Step 1: Let $k=0$ and begin with admissible control $u_i^{\left(0\right)}$, then choose a small positive number ${\epsilon }_i$.

Step 2: Let $k=0$; based on admissible control $u_i^{\left(k\right)}$, solve $V_{al}^k\left(\dot{E}\right)$ through ${\dot{E}}^{T}Q\dot{E}+{\left(U^{\left(k\right)}\right)}^{T}R{U}^{\left(k\right)}+{\left(W^{\left(k\right)}\right)}^{T}PW+{\left(\nabla V_{al}^{(k+1)}\right)}^T\left(f\left(x\right)+G{U}^{\left(K\right)}+W^{\left(K\right)}-{\ddot{x}}_{1d}\right)-e_{Jh}=0,$ with $V_{al}^{(k+1)}\left(0\right)=0$.

Step 3: Update the control policy $u_i^{\left(k\right)}$ with $u_i^{(k+1)}=-{\displaystyle \frac{1}{2}}{R}_i^{-1}{G}_i^T\left(x_i\right)\nabla V_{al}^{(k+1)}$.

Step 4: If $∥V_{al}^{(k+1)}-V_{al}^{\left(k\right)}∥\le {\epsilon }_i$, stop; else, $k=k+1$, return to Step 2.

Theorem 1.

Given the initial control policy $U^{\left(0\right)}$ and the base policy iteration, the improved cost function and control policy converge to the optimal ones as $k\to \infty$, i.e., $V_{al}^{\left(k\right)}\to V_{al}^{(*)},U^{\left(k\right)}\to U^{(*)}.$

One has $U^{\left(k\right)}$ for any $k\ge 0$ with initial policy $U^{\left(0\right)}$. Then, one has integer $k_{0i}$ for any ${\omega }_i$. For $k\ge k_{0i}$, one has:

$$sup\left|V_{al}^{\left(k\right)}-V_{al}^{(*)}\right|<{\omega }_i,sup\left|U^{\left(k\right)}-U^{(*)}\right|<{\omega }_i.$$

The algorithm will converge to the improved cost function and optimal control.

Theorem 2.

Considering the dynamic model (1) of the MUS subsystem, its state space equation is shown in (7). Under the proposed decentralized approximate optimal control law (38) based on a cooperative differential game, the tracking error of the MUS is guaranteed to be UUB.

Choose $V_{al}\left(\dot{E}\right)=V_{cl}\left(t\right)$ as a Lyapunov candidate function, the derivative of which can be obtained as follows:

$${\dot{V}}_{cl}\left(t\right)=\left(\nabla V_{al}^T\right)\left(F\left(x\right)+GU+W-{\ddot{x}}_{1d}\right),$$

(39)

where $V_{si}={J_i}^{*}\left({\dot{e}}_{si}\right)$ and $V_{sji}={J_i}^{*}\left({\dot{e}}_{sji}\right)$.

Considering the HJB Equation (15), it can be obtained as follows:

$$\left(\nabla V_{al}^T\right)\left(F\left(x\right)-{\ddot{x}}_{1d}\right)=-{\dot{E}}^{T}Q\dot{E}+{\displaystyle \frac{1}{4}}{(\nabla V_{al}^{*})}^{T}G{R}^{-1}{G}^T(\nabla V_{al}^{*})+{\displaystyle \frac{1}{4}}{(\nabla V_{al}^{*})}^T{P}^{-1}(\nabla V_{al}^{*}).$$

(40)

By substituting Formula (40) into (39), we get

$${\dot{V}}_{cl}\left(t\right)=-{\dot{E}}^{T}Q\dot{E}+{\displaystyle \frac{1}{4}}{(\nabla V_{al}^{*})}^{T}G{R}^{-1}{G}^T(\nabla V_{al}^{*})+{\displaystyle \frac{1}{4}}{(\nabla V_{al}^{*})}^T{P}^{-1}(\nabla V_{al}^{*})+\left(\nabla V_{al}^T\right)\left(GU+W\right).$$

(41)

Combining this with Formula (41), it can be deduced that

$$\begin{array}{cc}\hfill & {\dot{V}}_{cl}\left(t\right)=-{\dot{E}}^{T}Q\dot{E}-\left(\nabla V_{al}^T\right)\left(G(U^{*}-\widehat{U})+(W^{*}-\widehat{W})\right)\hfill \\ \hfill & +{\displaystyle \frac{1}{4}}{(\nabla V_{al}^{*})}^{T}G{R}^{-1}{G}^T(\nabla V_{al}^{*})+{\displaystyle \frac{1}{4}}{(\nabla V_{al}^{*})}^T{P}^{-1}(\nabla V_{al}^{*}).\hfill \end{array}$$

(42)

Substituting Formula (39) into (42) yields

$$\begin{array}{cc}\hfill & {\dot{V}}_{cl}\left(t\right)=-{\dot{E}}^{T}Q\dot{E}+{\displaystyle \frac{1}{4}}{(\nabla V_{al}^{*})}^T\left(G{R}^{-1}{G}^T(\nabla V_{al}^{*})+P^{-1}(\nabla V_{al}^{*})\right)+{\displaystyle \frac{1}{2}}{\left({\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{W}_{cr}+\nabla {\epsilon }_{cr}\right)}^T\hfill \\ \hfill & \left(G{R}^{-1}\left(G^T{\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{W}_{cr}+G^T\nabla {\epsilon }_{cr}\right)+P^{-1}\left({\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{W}_{cr}+\nabla {\epsilon }_{cr}\right)\right)\hfill \\ \hfill & =-{\dot{E}}^{T}Q\dot{E}+{\Pi }_J,\hfill \end{array}$$

(43)

where ${\Pi }_J$ is with the upper bound:

$$\begin{array}{c}{\Pi }_J\le ∥\begin{array}{c}{\displaystyle \frac{1}{4}}{(\nabla V_{al}^{*})}^T\left(G{R}^{-1}{G}^T(\nabla V_{al}^{*})+P^{-1}(\nabla V_{al}^{*})\right)+{\displaystyle \frac{1}{2}}{\left({\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{W}_{cr}+\nabla {\epsilon }_{cr}\right)}^T\hfill \\ \left(G{R}^{-1}\left(G^T{\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{W}_{cr}+G^T\nabla {\epsilon }_{cr}\right)+P^{-1}\left({\left(\nabla {\delta }_{cr}\left(\dot{E}\right)\right)}^T{W}_{cr}+\nabla {\epsilon }_{cr}\right)\right)\hfill \end{array}∥\hfill \\ \le {\pi }_J,\hfill \end{array}$$

(44)

where ${\pi }_J$ is a computable positive number.

According to Formula (44), ${\dot{V}}_{cl}\left(t\right)$ has the following upper bounds:

$$\begin{array}{cc}\hfill & {\dot{V}}_{cl}\left(t\right)\le -{\dot{E}}^{T}Q\dot{E}+{\pi }_J\hfill \\ \hfill & \le -{\lambda }_{min}\left(Q\right){∥\dot{E}∥}^2+{\pi }_J,\hfill \end{array}$$

(45)

where ${\lambda }_{min}$ is the minimum of the eigenvalue.

If $\dot{E}$ is outside the compact set

$${\Omega }_{cl}=\left\{\dot{E}:\dot{E}\le \sqrt{{\displaystyle \frac{{\pi }_J}{{\lambda }_{min}\left(Q\right)}}}\right\},$$

(46)

it follows that Formula (39) is negative definite; that is, if Formula (46) is satisfied for any $\dot{E}\ne 0$, then ${\dot{V}}_{cl}\left(t\right)<0$. Therefore, we can conclude that the trajectory tracking error is ultimately uniformly bounded under the decentralized approximate optimal control law (30) based on a cooperative differential game via ACD. Proof complete.

4. Experiment

4.1. Experimental Setup

The proposed approximate optimal control law based on the cooperative differential game was verified on a 2-DOF MUS experimental platform, the details of which are shown in Figure 2. The measurable joint control torque is measured by the torque sensor, and the joint position information can be obtained by a combination of the absolute encoder and the incremental encoder. Because the control system is built in the Simulink environment, the sampling interval of the system can be set by the software. We conducted several experiments with different related algorithms. To reduce the length of the article, two representative different control methods are used in this section, namely non-zero-sum game-based optimal control [7] and the proposed cooperative differential game-based adaptive critic design method. The critic neural network chose a 2-3-1 structure with a single hidden layer structure. The activation function was ${\delta }_{cr}\left(\dot{E}\right)=e^{{\displaystyle \frac{-{(\dot{E}-\varsigma )}^T(\dot{E}-\varsigma )}{l}}}$, with $\varsigma =[-0.5,0.5]$ and $l=[1.28,1.28]$. The learning rate of weight update was 0.82. $Q=I$, $R=0.97I$, and $P=0.94I$, where I is the identity matrix.

4.2. Experimental Results

(1)

Position tracking performance

Figure 3, Figure 4, and Figure 5, respectively, show the position tracking curves under the non-zero-sum game-based optimal control method [7,30] cooperative differential game-based approximate optimal control strategy. The desired values are those which the MUS is expected to achieve. The actual values are those of the MUS under the different control algorithms’ curves. If the difference between the desired values and actual values is small, we say that the performance is good. It can be seen from the figures that both the traditional control method and the proposed control strategy can ensure good position tracking performance. Figure 4 and Figure 5 show the position tracking error curves of the traditional and proposed control methods. Both methods guarantee that the position tracking error is within 0.002 rad. It can be seen from the graph that the actual position immediately deviates from the expected position when the curve deflects, but due to the good robustness of the proposed control system, the actual trajectory coincides with the expected trajectory again in a short time. The proposed method’s tracking error is reduced by nearly 30% in both joint one and joint two. The proper learning law shows good performance. Based on different experiments, we chose a learning law with a better performance of 0.82.

(2)

Control torque

Figure 6 and Figure 7 show the control torque curve under the existing and proposed control methods. Control torque has a serious tremor effect, which will affect the position tracking performance and durability of the DC motor. Each joint of the MUS has a certain cost function under the existing non-zero-sum game method. However, this method cannot optimize the whole performance of the MUS. The developed cooperative differential game method is the only one with a whole cost function, and it can minimize the total performance. Therefore, in Figure 7, the control torque is reasonably optimized due to the adoption of the cooperative game strategy based on ACD. The root mean squares of the tracking error and control torque are shown in

Table 1. Root mean squares.

	Joint 1	Joint 2
Position error of proposed method	1.88 × ${10}^{-3}$ rad	1.23 × ${10}^{-3}$ rad
Position error of existing method	2.56 × ${10}^{-3}$ rad	1.79 × ${10}^{-3}$ rad
Control torque of proposed method	0.36 Nm	0.19 Nm
Control torque of existing method	0.42 Nm	0.22 Nm

. The values of the control torque when completing the same task should be approximately the same. However, the cooperative differential game-based ACD method can minimize the value of the control torque to the greatest extent possible. According to Table 1, the control torque in the proposed method is reduced by nearly 15%.

(3)

Critic NN weight

Figure 8 shows the weight curves of the neural network with the proposed control method. Due to the training of the neural network and the critic neural network’s policy iteration, the weights of the neural network are aggregated. However, because of the nature of RBFNNs, the aggregation of the curves is not to a specific value but within a small range.

5. Conclusions

This paper proposes an MUS approximate optimal control method based on cooperative differential game theory and solves the trajectory tracking problem. Firstly, a dynamic model of an MUS is established. Then, each module of the MUS is treated as a participant in the cooperative differential game, and the trajectory tracking problem of the MUS is transformed into an approximate optimal control problem through ACD based on the cooperative differential game. Using the critic network to approximate the joint performance index function of the system, an approximate optimal control law is obtained through a policy iteration algorithm. According to Lyapunov theory, the stability has been proven. Finally, the effectiveness was verified through an experimental platform.