Event-Trigger Reinforcement Learning-Based Coordinate Control of Modular Unmanned System via Nonzero-Sum Game

Abstract

Decreasing the position error and control torque is important for the coordinate control of a modular unmanned system with less communication burden between the sensor and the actuator. Therefore, this paper proposes event-trigger reinforcement learning (ETRL)-based coordinate control of a modular unmanned system (MUS) via the nonzero-sum game (NZSG) strategy. The dynamic model of the MUS is established via joint torque feedback (JTF) technology. Based on the NZSG strategy, the existing coordinate control problem is transformed into an RL issue. With the help of the ET mechanism, the periodic communication mechanism of the system is avoided. The ET-critic neural network (NN) is used to approximate the performance index function, thus obtaining the ETRL coordinate control policy. The stability of the closed-loop system is verified via Lyapunov’s theorem. Experiment results demonstrate the validity of the proposed method. The experimental results show that the proposed method reduces the position error by 30% and control torque by 10% compared with the existing control methods.

1. Introduction

With the rapid development of the space industry and the continuous increase in the demand for space exploration, the complexity of the environment and the precision of the control requirements faced by space operations are also constantly improving [1,2,3]. The problems of high risk and low efficiency caused by traditional astronaut operations relying on them leaving the capsule are becoming increasingly prominent. In recent years, thanks to the rapid improvement and development of unmanned systems research and development technology, the use of high-precision and -performance unmanned systems to solve the assembly and maintenance of space operations in orbit is gradually becoming a scientific value of the goal-oriented basic research topic in the field of space exploration. So far, unmanned systems operating in conventional ground environments have achieved good reliability and accuracy. However, under the high standard requirements of coordinate missions in complex space environments, traditional unmanned systems are difficult to meet the transportation requirements of launch vehicles and spacecraft due to their large volume, heavy mass, and difficulty in disassembly and assembly. However, for the space unmanned systems that are in service to overcome the above difficulties, there are still some limitations in the configuration of the mechanism, and it is difficult to change its assembly configuration and working mode according to different task requirements. The modular unmanned system (MUS) [4,5] is a kind of autonomous unmanned system with standard modules and interfaces that can reassemble and configure itself according to different task requirements. Through the reconfiguration of modules, the unmanned system can show a variety of assembly configurations to complete different tasks, thus showing advantages that traditional unmanned systems do not possess.

As an important branch of game theory, differential game [6,7] focuses on the dynamic decision-making process of continuous time systems described by differential equations. It is an ideal tool to deal with multi-participant decision making and control problems and to solve optimal strategies, and it is widely used in economics [8], management [9], computer science [10], and other fields [11,12]. Reinforcement learning [13,14] originated as an imitation of the human brain learning mechanism, reflecting the mapping of learning environment state to action, so that the system can obtain the maximum cumulative reward from the environment and then optimize the system performance through the optimal strategy selection. In recent years, it has been widely used in complex nonlinear differential games because it can effectively solve the problem of “dimensionality disaster” in traditional dynamic programming [15,16]. As a kind of game, the nonzero-sum game (NZSG) [17,18] needs to solve the corresponding coupled Hamilton–Jacobi (HJ) equation for each player in order to obtain its Nash equilibrium solution.

As an important part of modern control theory, the core problem of optimal control is to select control strategies to make some performance indexes of a given controlled system optimal. For a large number of nonlinear systems in practical engineering, to obtain the optimal control strategy, it is necessary to solve the HJ(-Bellman) (HJB) equation, which is a class of nonlinear partial differential equation, and it is difficult to obtain the optimal solution by analytical methods. The reinforcement learning method is a powerful tool to solve the optimal control problem of nonlinear systems. In reinforcement learning systems, a neural network (NN) [19,20] is designed to approximate the performance index function and estimate the solution of the HJ(B) equation. Due to its strong advantages in solving nonlinear optimal control, reinforcement learning has attracted extensive attention from scholars both domestic and abroad in recent years, and has made rich achievements in solving problems such as discrete time optimal control [21,22,23], continuous time optimal control [24,25,26], and data-driven optimal control [27,28,29] of complex nonlinear systems. However, these results are based on periodic sampling or event triggering, resulting in a waste of resources and high computational costs.

Motivated by the above, this paper develops event-trigger (ET) reinforcement learning (ETRL)-based coordinate control of MUS via the NZSG strategy. The main contributions of this paper are mainly the following two aspects:

1. To the best of the authors’ knowledge, it is the first time to introduce the NZSG via reinforcement learning applied to an MUS. By considering the control torque of n modules in the MUS as decision-makers, the optimal control problem for the MUS system is morphed into an NZSG issue with n players.

2. The stability of the developed method is guaranteed and the experiment on MUS is conducted. Through the experimental results, we can conclude that the proposed method produces less tracking errors and power consumption.

2. Background and Related Work

2.1. Reinforcement Learning

Optimal control is widely utilized and holds great importance in many areas. However, with the increase in the system’s dimension, the issue of the dimensionality curse has appeared. Reinforcement learning, as an effective solution to the dimensionality curse in optimal control, has emerged as a crucial approach for addressing approximate optimal control issues. Vamvoudakis et al. [30] published a book about RL-based control for cognitive autonomy. Wang et al. [12] developed a review in the field of RL for advanced control applications. Liu and Xue et al. [31] concluded the RL with applications in control. The above three papers are all surveys about the RL or adaptive dynamic programming. The proposed method in this paper deals with the coordinate control of an modular unmanned system that is a specific application environment using RL. Dong et al. [32] proposed safe RL for the sake of trajectory tracking for a modular robot system. Event-trigger is not mentioned, as it causes computation and communication burdens. An et al. [17] designed a cooperative game-based RL method for human–robot interaction. Liu et al. [33] used the RL control method to deal with the vehicle path tracking issue. The particle swarm optimization method is utilized in the vehicle path tracking issue [34]. However, the above methods only consider a single controller to guarantee optimality. The modern industry needs more than one player/controller to finish the task using other player’s information. Hence, the developed nonzero-sum game has great importance.

2.2. Nonzero-Sum Game

Differential game theory focuses on the dynamic decision-making process in multi-player interactive systems and with advantages in dealing with uncertain interaction and disturbance. Differential games include the zero-sum game, nonzero-sum game, cooperative game, etc. Each module in the MUS system functions as a participant in NZSG, each with its own policy, collectively operating within the group using a general quadratic performance index function as the basis for the game. Wu et al. [35] proposed NZSG for an unmanned aerial vehicle with uncertain as well as asymmetric information. Coordinate control is not considered. Zheng et al. [36] developed a Q-learning-based NZSG for spacecraft system under a pursuit–evasion condition. The above method is based on a time-triggered mechanism. Besides optimum, the communication burden between the sensor and actuator needs to be considered. Therefore, an event-trigger has been developed to decrease the quantity of sampling. An et al. [37] used a dynamic event-trigger to complete a robot’s tracking task via NZSG. Dong et al. [38] proposed event-trigger value iteration RL for a coordinated task under the framework of NZSG. The above methods are only applied on robots; thus, they are unsuitable for modular unmanned systems.

3. Dynamic Model

For an MUS employing the JTF technique, the dynamic model of the ith subsystem is presented below:

$$I_{im}{\gamma }_i{\ddot{q}}_i+{\displaystyle \frac{{\tau }_{is}}{{\gamma }_i}}+f_{ir}(q_i,{\dot{q}}_i)+I_i(q,\dot{q},\ddot{q})={\tau }_i+J_i^{T}f,$$

(1)

where f is the contact force between the MUS and object; $f_{ir}(q_i,{\dot{q}}_i)$ means lumped joint friction; ${\gamma }_i$ indicates the gear ratio; $q_i$ reflects joint position; ${\tau }_{is}$ represents coupled joint torque; $I_i(q,\dot{q},\ddot{q})$ is the IDC effect among MUS subsystems; ${\tau }_i$ indicates control torque; and subscript i is ith joint module subsystem. The property analyses are described below:

(1)

The lumped joint friction

The joint friction term $f_{ir}(q_i,{\dot{q}}_i)$ is formulated as

$$\begin{array}{cc}\hfill f_{ir}(q_i,{\dot{q}}_i)& ={\widehat{f}}_{ib}{\dot{q}}_i+({\widehat{f}}_{is}{e}^{(-{\widehat{f}}_{ir}{\dot{q}}_i^2)}+{\widehat{f}}_{ic})sgn\left({\dot{q}}_i\right)+f_{ip}(q_i,{\dot{q}}_i)+Y_i\left({\dot{q}}_i\right){\tilde{F}}_{ir},\hfill \end{array}$$

(2)

in which

$$Y_i\left({\dot{q}}_i\right)={\left[f_{ib}-{\widehat{f}}_{ib},f_{ic}-{\widehat{f}}_{ic},f_{is}-{\widehat{f}}_{is},f_{i\tau }-{\widehat{f}}_{i\tau }\right]}^T,$$

(3)

where $f_{ip}(q_i,{\dot{q}}_i)$ is the position dependency friction term; $f_{ib},f_{i\tau }$ are viscous and Stribect friction effects; and $f_{is},f_{ic}$ are static and Coulomb friction parameters. Furthermore, ${\widehat{f}}_{ib},{\widehat{f}}_{ic},$ ${\widehat{f}}_{is},$ and ${\widehat{f}}_{i\tau }$ are the estimated values.

Remark 1.

The variables $f_{ib},f_{ic},f_{is},f_{i\tau }$ are bounded, and their corresponding estimates also possess boundedness. Consequently, this ensures that the variable ${\tilde{F}}_{ir}$ is bounded, as indicated by $\left|{\tilde{F}}_{ir}\right|\le b_{iFrm}$, where $b_{iFrm}$ represents a known positive constant for each m in (1,2,3,4). Consequently, $Y_i\left({\dot{q}}_i\right){\tilde{F}}_{ir}$ can be derived, which is designated as $\left|Y_i\left({\dot{q}}_i\right){\tilde{F}}_{ir}\right|\le Y_i\left({\dot{q}}_i\right)b_{iFrm}$. Additionally, $\left|f_{ip}(q_i,{\dot{q}}_i)\right|\le b_{iFp}$, in which $b_{iFp}$ is a known positive constant.

(2)

The interconnected dynamic coupling

The IDC is expressible as a nonlinear function of the coupled vectors of the entire modular subsystem in this way:

$$\begin{array}{cc}\hfill I_i& =I_{im}\sum _{j=1}^{i-1}{v}_{mi}^T{v}_{lj}{\ddot{q}}_j+I_{im}\sum _{j=2}^{i-1}\sum _{k=1}^{j-1}{v}_{mi}^T(v_{lk}\times v_{lj}){\dot{q}}_k{\dot{q}}_j\hfill \\ \hfill & =I_{im}\sum _{j=1}^{i-1}{D}_j^i{\ddot{q}}_j+I_{im}\sum _{j=2}^{i-1}\sum _{k=1}^{j-1}{\Theta }_{kj}^i{\dot{q}}_k{\dot{q}}_j\hfill \\ \hfill & =\sum _{j=1}^{i-1}\left[I_{im}{\widehat{D}}_j^i,I_{im}\right]{\left[{\ddot{q}}_j,{\tilde{D}}_j^i{\ddot{q}}_j\right]}^T+\sum _{j=2}^{i-1}\sum _{k=1}^{j-1}\left[I_{im}{\widehat{\Theta }}_{kj}^i,I_{im}\right]{\left[{\ddot{q}}_j,{\tilde{\Theta }}_{kj}^i{\dot{q}}_k{\dot{q}}_j\right]}^T,\hfill \end{array}$$

(4)

in which $v_{mi},v_{lj},v_{lk}$ denote the unit vectors along with the ith, jth, and kth joint rotation axes, respectively. Consequently, define $D_j^i=v_{mi}^T{v}_{lj}$ and ${\Theta }_{kj}^i=v_{mi}^T(v_{lk}\times v_{lj})$. We also have the relation that ${\widehat{D}}_j^i=D_j^i-{\tilde{D}}_j^i$ and ${\widehat{\Theta }}_{kj}^i={\Theta }_{kj}^i-{\tilde{\Theta }}_{kj}^i$, in which ${\widehat{D}}_j^i,{\widehat{\Theta }}_{kj}^i$ represent the estimated values of $D_j^i,{\Theta }_{kj}^i$, and ${\tilde{D}}_j^i,{\tilde{\Theta }}_{kj}^i$ are alignment errors.

Remark 2.

Based on (4), which characterizes $v_{mi},v_{lk},v_{lj}$, it is inferred that the magnitudes of the associated vector products are bounded, where $\left|D_j^i\right|=\left|v_{mi}^T{v}_{lj}\right|<1$ and $\left|{\Theta }_{kj}^i\right|=\left|v_{mi}^T(v_{lk}\times v_{lj})\right|<1$. Additionally, our findings indicate that $I_i$ is bounded and the up-bound is given as $\left|I_i\right|\le b_{iI}$ with a positive constant.

Define state vector $x_i={[x_{i1},x_{i2}]}^T={[q_i,{\dot{q}}_i]}^T$ and the control input $u_i={\tau }_i$. The state space of the ith subsystem is

$$\left\{\begin{array}{c}{\dot{x}}_{i1}=x_{i2}\hfill \\ {\dot{x}}_{i2}=f_i\left(x\right)+g_i{u}_i\hfill \end{array}\right.,$$

(5)

where

$$\begin{array}{c}{g}_i={\left(I_{im}{\gamma }_i\right)}^{-1}\hfill \\ f_i=g_i\left(\begin{array}{c}-({\widehat{f}}_{is}{e}^{(-{\widehat{f}}_{ir}{\dot{x}}_{i1}^2)}+{\widehat{f}}_{ic})sgn\left(x_{i2}\right)-f_{ip}(x_{i1},x_{i2})-\hfill \\ {\widehat{f}}_{ib}{x}_{i2}-Y_i\left(x_{i2}\right){\tilde{F}}_{ir}-{\displaystyle \frac{{\tau }_{is}}{{\gamma }_i}}-I_i(x,\dot{x},\ddot{x})-J_i^{T}f\hfill \end{array}\right).\hfill \end{array}$$

(6)

Control objectives aim to ensure optimal tracking error performance for the MUS in coordinate control. Within the subsequent section, we introduce an event-trigger reinforcement learning-based coordinate control via nonzero-sum game framework.

4. Event-Trigger Reinforcement Learning-Based Coordinate Control via Nonzero-Sum Game

4.1. Problem Transformation

Based on the dynamic model (1) and state space (5), the control object of this paper is completing optimal trajectory tracking. Therefore, to facilitate designing the controller, the augmenting subsystem is deduced:

$$\left\{\begin{array}{c}{\dot{x}}_1=x_2\hfill \\ {\dot{x}}_2=f\left(x\right)+\sum _{m=1}^n{G}_m{u}_m\hfill \end{array}\right.,$$

(7)

where $x={[x_1^T,x_2^T]}^T\in R^{2n}$ is global state of the MUS, in which the vectors $x_1,x_2$ are given by $x_1=[x_{11},\cdots ,x_{i1},\cdots ,x_{n1}{]}^T\in R^n$ and $x_2={[x_{12},\cdots ,x_{i2},\cdots ,x_{n2}]}^T\in R^n$. Moreover, $f\left(x\right)=[f_1\left(x\right),\cdots ,f_i\left(x\right),\cdots ,f_n\left(x\right){]}^T$, $G_m={[0,\cdots ,0,g_m,0,\cdots ,0]}^T$, where $g_m={\left(I_{mm}{\gamma }_m\right)}^{-1},m=1,\cdots ,n$.

Define the cost function:

$$\begin{array}{cc}\hfill J_i({\dot{e}}_s,u_1,\cdots ,u_n)& ={\int }_t^{\infty }\left({{\dot{e}}^T}_s{Q}_i{\dot{e}}_s+\sum _{m=1}^n{u}_m^T{R}_{im}{u}_m\right)d\tau \hfill \\ \hfill & ={\int }_t^{\infty }{U}_i({\dot{e}}_s,u_1,\cdots ,u_n)d\tau ,\hfill \end{array}$$

(8)

where position error is $e={\left[e_1,e_2,\cdots ,e_n\right]}^T=x_1-x_d$ and velocity error vector $\dot{e}={\left[{\dot{e}}_1,{\dot{e}}_2,\cdots ,{\dot{e}}_n\right]}^T=x_2-{\dot{x}}_d$; $e_s=\dot{e}+\beta e$ means fusion error; $x_d,{\dot{x}}_d,{\ddot{x}}_d$ represent the determined reference vectors; $Q_i,R_{im}$ denote determined positive definite matrices; and $U_i({\dot{e}}_s,u_1,\cdots ,u_n)$ indicates the utility function. Employing the infinitesimal version of (8), the Hamiltonian function can be derived:

$$H_i({\dot{e}}_s,u_1,\cdots ,u_n,\nabla J_i)=U_i({\dot{e}}_s,u_1,\cdots ,u_n)+{(\nabla J_i)}^T\left(f\left(x\right)+\sum _{m=1}^n{G}_m{u}_m-b_d\right),$$

(9)

where $\nabla J_i\left({\dot{e}}_s\right)={\displaystyle \frac{\partial J_i\left({\dot{e}}_s\right)}{\partial {\dot{e}}_s}}$ is the partial derivative of $J_i\left({\dot{e}}_s\right)$, $b_d={\ddot{x}}_d+\beta e$. Additionally, the optimal value function can be described as

$$J_i^{*}({\dot{e}}_s,u_1,\cdots ,u_n)=\underset{u_i}{min}{\int }_t^{\infty }{U}_i({\dot{e}}_s,u_1,\cdots ,u_n)d\tau .$$

(10)

Based on the stationary condition ${\displaystyle \frac{\partial H_i}{\partial u_i}}=0$, the local optimal control policy $u_i^{*}$ is defined as

$$u_i^{*}=-{\displaystyle \frac{1}{2}}{R}_{ii}^{-1}{G}_i^T\nabla J_i^{*}.$$

(11)

By substituting (8) and (11) into the Hamiltonian function (9), the coupled Hamilton–Jacobi (HJ) equation can be derived:

$$\begin{array}{cc}\hfill 0& ={(\nabla J_i^{*})}^T\left(f\left(x\right)-{\displaystyle \frac{1}{2}}\sum _{m=1}^n{G}_m{R}_{mm}^{-1}{G}_m^T\nabla J_m^{*}+\varpi \left(x\right)-b_d\right)\hfill \\ \hfill & +{\displaystyle \frac{1}{4}}\sum _{m=1}^n{(\nabla J_i^{*})}^T{G}_m{R}_{mm}^{-1}{R}_{im}{R}_{mm}^{-1}{G}_m^T(\nabla J_m^{*})+{{\dot{e}}^T}_s{Q}_i{\dot{e}}_s\hfill \end{array}$$

(12)

It is hard to obtain an analytical solution because of the nonlinearity system. Therefore, an event-trigger reinforcement learning-based coordinate control is introduced.

4.2. Event-Trigger Reinforcement Learning-Based Coordinate Control

The optimal control policy (11) is addressed from periodic sampling as well as the coupled HJ equation (12). Fixed sampling control not only escalates computational demands but also excessively taps into communication resources, jeopardizing the timeliness of control in environments with constrained bandwidth. Therefore, an event-triggered strategy is introduced to optimize efficiency.

Set a series of monotonously increasing ${\left\{t_j\right\}}_{j=0}^{+\infty }$, which contains trigger instants $t_j$. Then, define the sampling state

$${\dot{e}}_{sji}\left(x_{ji}\right)={\dot{e}}_{sji}\left(x_i\left(t_j\right)\right),$$

(13)

where ${\dot{e}}_{sji}\left(x_{ji}\right)$ denotes triggering instant state for $t\in \left[t_j,t_{j+1}\right)$. To obtain the trigger condition, the subsequent gap function is introduced:

$$g_{eji}\left(t\right)={\dot{e}}_{si}\left(x_i\right)-{\dot{e}}_{sji}\left(x_i\right).$$

(14)

Upon event triggering, based on (13), the actual state undergoes sampling to become the sampled state, after which $g_{eji}\left(t\right)$ is reset to zero. The optimal control law is updated to ${u_i}^{*}\left({\dot{e}}_{si}\left(t_j\right)\right)$$={u_i}^{*}\left({\dot{e}}_{sji}\right)$ during $\left[t_j,t_{j+1}\right)$, $j\in N$. It should be noted that ${u_i}^{*}\left({\dot{e}}_{sji}\right)$ are discrete values updated irregularly, necessitating conversion to continuous values. Therefore, a zero-order holder is derived to cope with this issue.

According to the dynamic model of MUS (7), one gives the event-trigger value function as follows:

$$J_i({\dot{e}}_{sji},u_1,\cdots ,u_n)={\int }_t^{t+T}\left({{\dot{e}}_{sji}}^T{Q}_i{\dot{e}}_{sji}+\sum _{m=1}^n{u}_m^T\left({\dot{e}}_{sji}\right)R_{im}{u}_m\left({\dot{e}}_{sji}\right)\right)d\tau .$$

(15)

One has the event-triggered HJ equation:

$$\begin{array}{cc}\hfill & H_i({\dot{e}}_{sji},u_1,\cdots ,u_n,\nabla J_i\left({\dot{e}}_{sji}\right))=U_i({\dot{e}}_{sji},u_1,\cdots ,u_n)\hfill \\ \hfill & +{(\nabla J_i\left({\dot{e}}_{sji}\right))}^T\left(f\left(x\right)+\sum _{m=1}^n{G}_m{u}_m\left({\dot{e}}_{sji}\right)-b_d\right),\hfill \end{array}$$

(16)

where $\nabla J_i\left({\dot{e}}_{sji}\right)=\partial J_i\left({\dot{e}}_{sji}\right)/\partial {\dot{e}}_{sji}$ is the partial derivative of $J_i\left({\dot{e}}_{sji}\right)$ with regard to ${\dot{e}}_{sji}$. To eliminate the assumption of norm-boundness regarding interconnections, the desired states of coupled subsystems are used as a replacement for their actual states. Consequently, the interconnection term is depicted:

$$\begin{array}{cc}\hfill & f\left(x\right)=f_i(x_i,x_{md})+\Delta f_i(x,x_{md}),\hfill \\ \hfill & u_m=G_m^{-1}({\dot{x}}_{m2d}-f_m\left(x_d\right)),m\ne i.\hfill \end{array}$$

(17)

where $x_{md}$ is the desired state of coupled subsystems for $m=1,\cdots ,i-1,i+1,\cdots ,n,$ and $\Delta f_i(x,x_{md})$ represents the substitution error. Given the interconnection’s compliance with the global Lipschitz condition, this indicates

$$∥\Delta f_i(x,x_{md})∥\le \sum _{m=1,m\ne i}^n{d}_{im}{E}_m,$$

(18)

where $E_m=∥x_m-x_{md}∥,$ and $d_{im}\ge 0$ denotes an unknown global Lipschitz constant.

The improved event-triggered optimal value function is

$$J_i^{*}\left({\dot{e}}_{sji},u_1,\cdots ,u_n\right)=\underset{{\mu }_i}{min}\left({\int }_t^{t+T}\left({{\dot{e}}_{sji}}^T{Q}_i{\dot{e}}_{sji}+\sum _{m=1}^n{u}_m^T\left({\dot{e}}_{sji}\right)R_{im}{u}_m\left({\dot{e}}_{sji}\right)\right)d\tau \right).$$

(19)

Through the substitution of (19) into (16), it can be inferred that

$$0=\underset{u_i\left({\dot{e}}_{sji}\right)\in {\mathsf{\Psi }}_i\left(\mathsf{\Omega }\right)}{min}{H}_i\left({\dot{e}}_{sji},u_i\left({\dot{e}}_{sji}\right),\nabla {J_i}^{*}\left({\dot{e}}_{sji}\right)\right).$$

(20)

Based on (21), one has the event-triggered optimal control law

$${u_i}^{*}\left({\dot{e}}_{sji}\right)=-{\displaystyle \frac{1}{2}}{R}_{ii}^{-1}{G}_i^T\nabla {J_i}^{*}\left({\dot{e}}_{sji}\right).$$

(21)

For any ${\dot{e}}_{si},{\dot{e}}_{sji}\in \mathsf{\Omega }$, the control law is Lipschitz continuous. Then, one has a constant ${\chi }_i$ satisfying

$$\begin{array}{cc}\hfill ∥{u_i}^{*}\left({\dot{e}}_{si}\right)-{u_i}^{*}\left({\dot{e}}_{sji}\right)∥& =∥{u_i}^{*}\left({\dot{e}}_{si}+g_{eji}\left(t\right)\right)-{u_i}^{*}\left({\dot{e}}_{sji}\right)∥\le {\chi }_i∥g_{eji}\left(t\right)∥.\hfill \end{array}$$

(22)

Given the challenging nature of solving the coupled HJ equation and the curse of dimensionality that arises with increasing dimensions, we employ the reinforcement learning algorithm for deriving an approximate solution for the event-triggered HJ equation in real-time.

The improved value function ${J_i}^{*}\left({\dot{e}}_{sji}\right)$ can be obtained by the radial basis function neural network (RBFNN) as follows:

$${J_i}^{*}\left({\dot{e}}_{sji}\right)={W_{ci}}^T{\delta }_{ci}\left({\dot{e}}_{sji}\right)+{\epsilon }_{ci}\left({\dot{e}}_{sji}\right),$$

(23)

where $W_{ci}\in R^{K_i}$ is the desired critic NN weight vector; $K_i$ is the number of neurons in the hidden-layer; ${\delta }_{ci}\left({\dot{e}}_{sji}\right)=exp\left(-∥{\dot{e}}_{sji}-c_{ij}∥/2b_{ij}^2\right)$ denotes the activation function; and ${\epsilon }_{ci}\left({\dot{e}}_{sji}\right)$ is critic NN approximation error, which is bounded as $∥{\delta }_{ci}\left({\dot{e}}_{sji}\right)∥$ $\le {\delta }_{cimax}$, $∥{\epsilon }_{ci}\left({\dot{e}}_{sji}\right)∥\le {\epsilon }_{cimax}$ with the positive constants ${\delta }_{cimax}$ and ${\epsilon }_{cimax}$.

Therefore, the partial derivative of ${J_i}^{*}\left({\dot{e}}_{sji}\right)$ can be obtained as follows:

$$\nabla {J_i}^{*}\left({\dot{e}}_{sji}\right)=\nabla {\delta }_{ci}^T\left({\dot{e}}_{sji}\right)W_{ci}+\nabla {\epsilon }_{ci}\left({\dot{e}}_{sji}\right),$$

(24)

where $\nabla {\delta }_{ci}\left({\dot{e}}_{sji}\right)$ is Lipschitz continuous.

The relationship $∥\nabla {\delta }_{ci}\left({\dot{e}}_{si}\right)-\nabla {\delta }_{ci}\left({\dot{e}}_{sji}\right)∥\le p_i∥g_{eji}\left(t\right)∥$ can be derived, and $p_i$ is a positive constant.

Substituting (24) into (21) can yield the following:

$$\begin{array}{cc}\hfill {u_i}^{*}\left({\dot{e}}_{sji}\right)& =-{\displaystyle \frac{1}{2}}{R}_{ii}^{-1}{G}_i^T\nabla {J_i}^{*}\left({\dot{e}}_{sji}\right)\hfill \\ \hfill & =-{\displaystyle \frac{1}{2}}{R}_{ii}^{-1}{G}_i^T\left(\nabla {\delta }_{ci}^T\left({\dot{e}}_{sji}\right)W_{ci}+\nabla {\epsilon }_{ci}\left({\dot{e}}_{sji}\right)\right).\hfill \end{array}$$

(25)

Therefore, one obtains the event-triggered HJ equation as follows:

$$\begin{array}{c}{H}_i\left({\dot{e}}_{sji},u_i^{*}\left({\dot{e}}_{sji}\right),\nabla J_i^{*}\left({\dot{e}}_{sji}\right)\right)=U_i({\dot{e}}_{sji},u_1,\cdots ,u_i^{*},\cdots ,u_n)\hfill \\ +{(\nabla J_i\left({\dot{e}}_{sji}\right))}^T\left(\begin{array}{c}{f}_i(x_i,x_{md})-b_d+\sum _{m=1}^n{G}_m{u}_m\left({\dot{e}}_{sji}\right)\hfill \end{array}\right)\triangleq e_{cHi},\hfill \end{array}$$

(26)

where $e_{cHi}=-\nabla {\epsilon }_{ci}^T\left({\dot{e}}_{sji}\right){\ddot{e}}_{sji}$ means residual error, and the positive constant $e_{cHimax}$ is the upper bound of $e_{cHi}$.

Since we cannot obtain the desired critic NN weight vector, we approximate the improved value function

$${\widehat{J}}_i\left({\dot{e}}_{sji}\right)={\widehat{W}}_{ci}^T{\delta }_{ci}\left({\dot{e}}_{sji}\right).$$

(27)

Furthermore, the partial derivative ${\widehat{J}}_i\left({\dot{e}}_{sji}\right)$ is formulated as follows:

$$\nabla {\widehat{J}}_i\left({\dot{e}}_{sji}\right)=\nabla {\delta }_{ci}^T\left({\dot{e}}_{sji}\right){\widehat{W}}_{ci}.$$

(28)

Therefore, merging (28) with (21), the event-trigger-based approximate optimal control law ${\widehat{u}}_i\left({\dot{e}}_{sji}\right)$ is obtained as

$${\widehat{u}}_i\left({\dot{e}}_{sji}\right)=-{\displaystyle \frac{1}{2}}{R}_{ii}^{-1}{G}_i^T\left(\nabla {\delta }_{ci}^T\left({\dot{e}}_{sji}\right){\widehat{W}}_{ci}\right).$$

(29)

According to (25), (28) and (29), we can obtain the event-triggered approximate HJ equation as

$$\begin{array}{c}{\widehat{H}}_i\left({\dot{e}}_{sji},{{\widehat{u}}_i}^{*}\left({\dot{e}}_{sji}\right),\nabla {{\widehat{J}}_i}^{*}\left({\dot{e}}_{sji}\right)\right)=U_i({\dot{e}}_{sji},{\widehat{u}}_1,\cdots ,{\widehat{u}}_i^{*},\cdots ,{\widehat{u}}_n)\hfill \\ +{(\nabla J_i\left({\dot{e}}_{sji}\right))}^T\left(\begin{array}{c}{f}_i(x_i,x_{md})-b_d+\sum _{m=1}^n{G}_m{\widehat{u}}_m\left({\dot{e}}_{sji}\right)\hfill \end{array}\right)\triangleq e_{ci}.\hfill \end{array}$$

(30)

Define the critic approximation error vector as ${\tilde{W}}_{ci}=W_{ci}-{\widehat{W}}_{ci}$; from (30), we can define $\partial e_{ci}/\partial {\widehat{W}}_{ci}=\nabla {\delta }_{ci}\left({\dot{e}}_{sji}\right)$${\ddot{e}}_{sji}={\theta }_i$. To refine the estimation of the desired vector ${\widehat{W}}_{ci}$, we employ the gradient descent algorithm to minimize the objective function $E_{ci}={\displaystyle \frac{1}{2}}{e}_{ci}^2$, with the update rate given by

$$\begin{array}{cc}\hfill {\dot{\widehat{W}}}_{ci}& =-{\alpha }_{ci}{\displaystyle \frac{\partial E_{ci}}{\partial {\widehat{W}}_{ci}}}=-{\alpha }_{ci}\left(U_i({\dot{e}}_{sji},{\widehat{u}}_1,\cdots ,{\widehat{u}}_i^{*},\cdots ,{\widehat{u}}_n)+{\theta }_i{\widehat{W}}_{ci}^T\right){\theta }_i.\hfill \end{array}$$

(31)

Then, the approximation error vector is

$${\dot{\tilde{W}}}_{ci}=-{\alpha }_{ci}\left({\theta }_i{\tilde{W}}_{ci}^T-e_{cHi}\right){\theta }_i.$$

(32)

Theorem 1.

Taking the value function (23) into account, it is estimated by the critic NN with weights $W_{ci}$. The cost function, as given by Equation (27), is approximated using the weights ${\widehat{W}}_{ci}$. Assuming the update law for the critic NN is defined by (31), the weight approximation error is proven to be UUB.

Proof. 

The candidate for the Lyapunov function is selected as

$$V_{1i}\left(t\right)={\displaystyle \frac{1}{2{\alpha }_{ci}}}{\tilde{W}}_{ci}^T{\tilde{W}}_{ci}.$$

(33)

The derivative of $V_{1i}\left(t\right)$ can be obtained as

$$\begin{array}{c}{\dot{V}}_{1i}\left(t\right)={\displaystyle \frac{1}{{\alpha }_{ci}}}{\tilde{W}}_{ci}^T{\dot{\tilde{W}}}_{ci}={\tilde{W}}_{ci}^T\left(e_{cHi}-{\theta }_i{\tilde{W}}_{ci}^T\right){\theta }_i\hfill \\ ={\tilde{W}}_{ci}^T{e}_{cHi}{\theta }_i-{∥{\tilde{W}}_{ci}^T{\theta }_i∥}^2\le {\displaystyle \frac{1}{2}}{e}_{cHi}^2-{\displaystyle \frac{1}{2}}{∥{\tilde{W}}_{ci}^T{\theta }_i∥}^2.\hfill \end{array}$$

(34)

Upon analyzing (34), we observe that if $∥{\tilde{W}}_{ci}∥\ge ∥{\displaystyle \frac{e_{cHi}}{{\theta }_i}}∥$, this leads to ${\dot{V}}_{1i}\left(t\right)<0$, which in turn confirms the UUB of the critic approximation error vector. □

Theorem 2.

Given an MUS with joint subsystem dynamic model (1) and state space (7), the closed-loop MUS with coordinate control is UUB under the presented event-triggered reinforcement learning-based coordinate control law (35) if

$$\begin{array}{cc}\hfill {∥g_{eji}∥}^2& \le {\displaystyle \frac{\left(1-{\alpha }_i^2\right){\sigma }_{min}\left(Q_i\right){∥{\dot{e}}_{si}∥}^2+{\displaystyle \sum _{m=1}^n}{∥r_m∥}^2{∥{\widehat{u}}_m\left({\dot{e}}_{sjm}\right)∥}^2}{2R_i{\chi }_i^2}}\hfill \\ & -{\displaystyle \frac{G_i^2{\left(\nabla {\delta }_{cimax}{W}_{cimax}+\nabla {\epsilon }_{cimax}\right)}^2}{4R_i^2{\chi }_i^2}},\hfill \end{array}$$

(35)

holds, where ${\alpha }_i\in \left(0,1\right)$ is the designed sampling frequency parameter, ${\sigma }_{min}\left(·\right)$ means the minimum eigenvalue of the matrix, and $r_{im}$ is a positive constant that satisfies $R_{im}=r_{im}^T{r}_{im}$, assuming $∥{\tilde{W}}_{ci}∥\le ∥W_{cimax}∥$.

Proof. 

We select the Lyapunov candidate function

$$V_i\left(t\right)=V_{si}+V_{sji},$$

(36)

where $V_{si}={J_i}^{*}\left({\dot{e}}_{si}\right)$ and $V_{sji}={J_i}^{*}\left({\dot{e}}_{sji}\right)$. □

The following proof is divided into two cases.

Case 1: 

The events are not triggered, i.e., $t\in \left[t_j,t_{j+1}\right)$.

Computing the derivative with respect to time of (36), the result is obtained as follows:

$${\dot{V}}_{si}\left(t\right)={(\nabla J_i^{*}\left({\dot{e}}_{si}\right))}^T\left(\begin{array}{c}{f}_i(x_i,x_{md})-b_d+\sum _{m=1}^n{G}_m{u}_m\hfill \end{array}\right),$$

(37)

$${\dot{V}}_{sji}=0.$$

(38)

Based on the optimal control law (11) and time-triggered HJ equation (12), one has

$$\begin{array}{cc}\hfill {(\nabla J_i^{*})}^T\left(f_i(x_i,x_{md})-b_d\right)=& -{\dot{e}}_{si}^T{Q}_i{\dot{e}}_{si}+{\displaystyle \frac{1}{2}}\sum _{m=1}^n{G}_m{R}_{mm}^{-1}{G}_m^T\nabla J_m^{*}\hfill \\ \hfill & -{\displaystyle \frac{1}{4}}\sum _{m=1}^n{(\nabla J_m^{*})}^T{G}_m{R}_{mm}^{-1}{R}_{im}{R}_{mm}^{-1}{G}_m^T(\nabla J_m^{*}),\hfill \end{array}$$

(39)

$$\nabla {J_i}^{*}\left({\dot{e}}_{si}\right)G_i=-2\sum _{m=1}^n{G}_m^T{u}_m^{*}.$$

(40)

Substituting (39) and (40) into (37), we can obtain the following equation:

$$\begin{array}{cc}\hfill {\dot{V}}_{si}\left(t\right)& =-{\dot{e}}_{si}^T{Q}_i{\dot{e}}_{si}-{(\nabla J_i^{*})}^T\left(\sum _{m=1}^n{G}_m\left(u_m^{*}-{\widehat{u}}_m\right)\right)-{\displaystyle \frac{1}{4}}\sum _{m=1}^n\left({(\nabla J_m^{*})}^T{G}_m{R}_{mm}^{-1}{R}_{im}{R}_{mm}^{-1}{G}_m^T(\nabla J_m^{*})\right)\hfill \\ \hfill & =-{\dot{e}}_{si}^T{Q}_i{\dot{e}}_{si}-{\displaystyle \frac{1}{4}}\sum _{m=1}^n\left(\begin{array}{c}{(\nabla J_m^{*}\left({\dot{e}}_{sji}\right))}^T{G}_m{R}_{mm}^{-1}{R}_{im}{R}_{mm}^{-1}{G}_m^T(\nabla J_m^{*}\left({\dot{e}}_{sji}\right))\hfill \end{array}\right)\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{\left(\begin{array}{c}\nabla {\delta }_{ci}^T\left({\dot{e}}_{sji}\right)W_{ci}+\nabla {\epsilon }_{ci}\hfill \end{array}\right)}^T\left(\sum _{m=1}^n{G}_m{R}_{mm}^{-1}\left(\begin{array}{c}{G}_m^T\nabla {\delta }_{cm}^T\left({\dot{e}}_{sji}\right){\tilde{W}}_{cm}+G_m^T\nabla {\epsilon }_{cm}\hfill \end{array}\right)\right)\hfill \\ \hfill & =-{\dot{e}}_{si}^T{Q}_i{\dot{e}}_{si}-{\displaystyle \frac{1}{4}}\sum _{m=1}^n\left(\begin{array}{c}{(\nabla J_m^{*}\left({\dot{e}}_{sji}\right))}^T{G}_m{R}_{mm}^{-1}{R}_{im}{R}_{mm}^{-1}{G}_m^T(\nabla J_m^{*}\left({\dot{e}}_{sji}\right))\hfill \end{array}\right)+{\mathsf{\Pi }}_{iJ},\hfill \end{array}$$

(41)

in which the function term ${\mathsf{\Pi }}_{iJ}$ has the following up-bound:

$$\begin{array}{cc}\hfill {\mathsf{\Pi }}_{iJ}& \le ∥\begin{array}{c}{\displaystyle \frac{1}{2}}{\left(\nabla {\delta }_{ci}^T\left({\dot{e}}_{sji}\right)W_{ci}+\nabla {\epsilon }_{ci}\right)}^T\left(\sum _{m=1}^n{G}_m{R}_{mm}^{-1}{G}_m^T\left(\nabla {\varphi }_{cm}^T\left({\dot{e}}_{sji}\right){\tilde{W}}_{cm}+\nabla {\epsilon }_{cm}\right)\right)\hfill \end{array}∥\le {\pi }_{iJ},\hfill \end{array}$$

(42)

where ${\pi }_{iJ}$ is a computable positive constant.

According to (24) and (28), (41) can be transformed into

$${\dot{V}}_{si}\le -{{\dot{e}}_{si}}^T{Q}_i{\dot{e}}_{si}-\sum _{m=1}^n{{\widehat{u}}_m}^T\left({\dot{e}}_{sjm}\right)R_{im}{\widehat{u}}_m\left({\dot{e}}_{sjm}\right)+2{\chi }^2{R}_{ii}{∥g_{eji}\left(t\right)∥}^2+{\displaystyle \frac{1}{2}}{R}_{ii}^{-1}{G}_i^2{\left(\nabla {\delta }_{ci}^T\left({\dot{e}}_{sji}\right){\tilde{W}}_{ci}+\nabla {\epsilon }_{ci}\left({\dot{e}}_{sji}\right)\right)}^2.$$

(43)

Thus, we have $\dot{V}\left(t\right)$ as

$$\dot{V}\left(t\right)=\sum _{i=1}^n{\dot{V}}_i\left(t\right)=\sum _{i=1}^n\left({\dot{V}}_{si}+{\dot{V}}_{sji}\right).$$

(44)

According to (38) and (43) as well as (44), one has

$$\begin{array}{cc}\hfill \dot{V}\left(t\right)& \le \sum _{i=1}^n\left(\begin{array}{c}-{{\dot{e}}_{si}}^T{Q}_i{\dot{e}}_{si}+2R_{im}{\chi }^2{∥g_{eji}∥}^2-\sum _{m=1}^n{{\widehat{u}}_m}^T\left({\dot{e}}_{sjm}\right)R_{im}{\widehat{u}}_m\left({\dot{e}}_{sjm}\right)\hfill \\ +{\displaystyle \frac{1}{2}}{R}_{im}^{-1}{G}_i^2{\left(\nabla {\delta }_{ci}^T\left({\dot{e}}_{sji}\right){\tilde{W}}_{ci}+\nabla {\epsilon }_{ci}\left({\dot{e}}_{sji}\right)\right)}^2\hfill \end{array}\right)\hfill \\ \hfill & \le \sum _{i=1}^n\left(\begin{array}{c}-{\alpha }_i^2{\sigma }_{min}\left(Q_i\right){∥{\dot{e}}_{si}∥}^2+\left({\alpha }_i^2-1\right){\sigma }_{min}\left(Q_i\right){∥{\dot{e}}_{si}∥}^2-{∥r_{im}∥}^2{∥{\widehat{u}}_i\left({\dot{e}}_{sjm}\right)∥}^2\hfill \\ +2R_{im}{\chi }^2{∥g_{eji}∥}^2+{\displaystyle \frac{1}{2}}{R}_{im}^{-1}{G}_i^2{\left(\nabla {\delta }_{cimax}{W}_{cimax}+\nabla {\epsilon }_{cimax}\right)}^2\hfill \end{array}\right).\hfill \end{array}$$

(45)

Given that (35) is valid, (45) is compliant with the requirement $\dot{V}\left(t\right)\le {\displaystyle \sum _{i=1}^n}({\pi }_{iJ}-{\alpha }_i^2{\sigma }_{min}\left(Q_i\right){∥{\dot{e}}_{si}∥}^2)$. The condition that ensures the negativity of ${\dot{V}}_i\left(t\right)$ is that ${\dot{e}}_{si}$ does not fall within the confines of $\mathsf{\Omega }=\left\{{\dot{e}}_{si}:∥{\dot{e}}_{si}∥\le \sqrt{{\displaystyle \frac{{\pi }_{iJ}}{{\alpha }_i^2{\sigma }_{min}\left(Q_i\right)}}}\right\}$, a requirement critical for affirming the negativity of the proposed Lyapunov function.

Case 2: 

When events are triggered, $\forall t=t_{j+1}$, the difference of (36) is rewritten as

$$\begin{array}{cc}\hfill \Delta V_i\left(t\right)& =V_i\left({\dot{e}}_{sji+1}\right)-V_i\left({\dot{e}}_{si}\left(x_{ji+1}^{-}\right)\right)\hfill \\ \hfill & =J_i^{*}\left({\dot{e}}_{sji+1}\right)-J_i^{*}\left({\dot{e}}_{si}\left(x_{ji+1}^{-}\right)\right)+J_i^{*}\left({\dot{e}}_{sji+1}\right)-J_i^{*}\left({\dot{e}}_{sji}\right).\hfill \end{array}$$

(46)

Based on (35), one has $\dot{V}\left(t\right)\le 0$. Therefore, we have ${J_i}^{*}\left({\dot{e}}_{sji+1}\right)$$\le {J_i}^{*}\left({\dot{e}}_{si}\left(x_{ji+1}^{-}\right)\right)$.

Then, one has (46) as the following form:

$$\Delta V\left(t\right)\le {J_i}^{*}\left({\dot{e}}_{sji+1}\right)-{J_i}^{*}\left({\dot{e}}_{sji}\right)\le -\nu ∥g_{eji}\left(t_j\right)∥,$$

(47)

where $\nu$ denotes a class-k function, $g_{eji}\left(t_j\right)={\dot{e}}_{sji+1}-{\dot{e}}_{sji}$.

Taking into account Cases 1 and 2 collectively, it follows that under the condition specified by (35), the closed-loop MUS’s tracking error is UUB. Thus, the conclusion of the proof is established.

4.3. Exclusion of Zeno Behaviors

The minimum trigger interval $t_{min}=min\left\{t_{j+1}-t_j\right\}$ is likely to be 0—that is, Zeno behavior. Therefore, we give the following theorem to avoid the phenomenon:

Theorem 3.

Considering MUS (1), the triggering condition (35) and the event-triggered approximate optimal control law (29), the minimum trigger interval $t_{min}$ is with a positive lower bound by

$$t_{min}\ge {\displaystyle \frac{1}{S_{iZ}}}ln\left(1+{\pi }_{j,min}\right)>0,$$

(48)

where ${\pi }_{j,min}=min\left({\displaystyle \frac{∥g_{eji}\left({\dot{e}}_{si}\left(x_{ji+1}^{-}\right)\right)∥}{\left({\dot{e}}_{sji}\right)+{\Theta }_i}}\right)>0$, $S_{iZ},{\Theta }_i$ are positive constants.

Proof. 

The time derivative of the event-triggered gap function (14) can be derived as follows:

$${\displaystyle \frac{\mathrm{d}\left(g_{eji}\right)}{\mathrm{d}t}}={\dot{g}}_{eji}={\ddot{e}}_{si}\left(x_i\right)-{\ddot{e}}_{sji}\left(x_{ji}\right)={\ddot{e}}_{si}\left(x_i\right).$$

(49)

The upper bound of ${\ddot{e}}_{si}\left(x_i\right)$ is derived as

$$∥{\ddot{e}}_{si}\left(x_i\right)∥\le S_{iZ}∥x_i∥+S_{iZ}{\Theta }_i.$$

(50)

Combining (14) and (49) with (50), it can be obtained that

$$\begin{array}{cc}\hfill ∥{\dot{g}}_{eji}∥& \le {\int }_{t_j}^t\left(e^{\left(S_{iZ}\left(t-w\right)\right)}{S}_{iZ}\left(∥{\dot{e}}_{sji}∥+{\Theta }_i\right)\right)\mathrm{d}w\le \left(∥{\dot{e}}_{sji}∥+{\Theta }_i\right)\left(e^{\left(S_{iZ}\left(t-t_j\right)\right)}-1\right).\hfill \end{array}$$

(51)

When $t=t_{j+1}$, the event-triggered condition satisfies

$$∥g_{eji}\left({\dot{e}}_{sji}\left(x_{ji+1}^{-}\right)\right)∥=∥g_{eji}\left(x_{ji+1}^{-}\right)∥.$$

(52)

Based on (51) and (52), the jth triggering interval $\Delta t_j$ has the lower bound by

$$\Delta t_j=t_{j+1}-t_j\ge {\displaystyle \frac{1}{S_{iZ}}}ln\left(1+{\displaystyle \frac{∥g_{eji}\left({\dot{e}}_{sji}\left(x_{ji+1}^{-}\right)\right)∥}{\left({\dot{e}}_{sji}\right)+{\Theta }_i}}\right).$$

(53)

This concludes the proof. □

5. Experiment

5.1. Experimental Setup

The validation of the proposed control method’s effectiveness is demonstrated through experiments on a 2 degrees of freedom (DOF) MUS platform. Detailed information about the experimental setup can be found in Figure 1. Joint control torque is measured using a joint torque sensor, while joint position information is acquired from both absolute and incremental encoders. The data acquisition board acts as the intermediary allowing interaction between the software environment (Simulink of Matlab 2016a) and hardware components. It is noted that the proposed ETRL via NZSG, which is in the form of continuous time, needs to be realized discretely when it is implemented in experiments. Fortunately, the control system, which is constructed under the Simulink environment, may complete the discrete realization automatically and adjust the sampling period adaptively. The model parameters are as follows: $I_{mi}$ = 120 g·cm², ${\gamma }_i$ = 100, ${\widehat{f}}_{ib}$ = 12 m·Nm/rad, ${\widehat{f}}_{ic}$ = 30 m·Nm, ${\widehat{f}}_{is}$ = 40 m·Nm, ${\widehat{f}}_{i\tau }$ = 20 s²/rad². We consider the coordinate control, which is illustrated in Figure 1. The purpose of the experiment is satisfying the requirements of position tracking performance and control torque optimization under a coordinate operation with MUS. The critic NN is selected as RBFNN, and the activation function of (23) is ${\delta }_{ci}=e^{{\displaystyle \frac{-{({\dot{e}}_{sji}-{\ell }_i)}^T({\dot{e}}_{sji}-{\ell }_i)}{{\zeta }_i}}}$ with initial value $W_{ci}={[0.3,0.3,0.3,0.3,0.3]}^T.$ ${\ell }_i,{\zeta }_i$ denote the center and width of the activation function. The purpose of the control method is to decrease the position error and control torque as much as possible. The experimental results show that the proposed method reduces the position error by 30% and control torque by 10% compared with the existing control methods.

5.2. Experimental Results

The experimental outcomes are utilized to evaluate the system’s position tracking accuracy, tracking error magnitude, applied control torque, contact forces, event-triggering mechanism’s efficiency, and neural network (NN) weights’ performance individually. Two distinct control methodologies are implemented: the established learning-based tracking approach, as seen in references [37,39], and the novel proposed control strategy. The coordinate control was subjected to the implementation of two distinct control approaches. The upper figure corresponds to joint one, while the lower one illustrates joint two.

(1) Position tracking

Figure 2, Figure 3 and Figure 4 depict the position tracking and tracking error curves in joint space during coordinate control using both the existing learning-based tracking control method and the proposed approximate optimal control method. The graphical analysis indicates that the position tracking error is notably lower and smoother with the newly proposed control method as opposed to the previously established method. The proposed method reduces the position error by 30%. This is attributed to the accurate solution of the coordinate control problem achieved by the proposed method. At the corners of the trajectory, the tracking error tends to increase but is effectively mitigated back to an acceptable range along the smooth path by the proposed approximate optimal control method. Figure 5 shows the 3D tracking curves.

(2) Control torque

Figure 6 displays the control torque curves during coordinate control using both the existing learning-based tracking control method and the proposed approximate optimal control method. The illustrations indicate that the control torque experiences a sharp increase during sudden trajectory changes, potentially impacting the lifespan of the DC motors. The proposed method reduces the control torque by 10% compared with the existing control methods. Furthermore, the control torque curves under the current control method display pronounced chattering, potentially degrading the accuracy of trajectory tracking. However, by employing the developed approximate optimal control method, the output torques are optimized to minimize motor power consumption and instantaneous increases in control torques are maintained within safe boundaries.

(3) Contact force

Figure 7 depicts the contact force curves during coordinate control using the proposed approximate optimal control method. Since the MUS has 2-DOF and the joint axes are assembled in parallel, the contact force curves appear in a two-dimensional space. From the figures, it can be observed that the proposed approximate optimal control method ensures that the contact force remains below 2N, with minimal chattering phenomenon.

(4) Event-triggered mechanism

Figure 8 and Figure 9 depict the trigger threshold and trigger condition curves. Owing to the incorporation of the NN within the reinforcement learning process, both the trigger condition and the trigger threshold exhibit large values. The proposed method’s trigger time is nearly half that of the existing method. However, the trigger condition stays within the threshold limits, confirming the reliability of the newly introduced strategy. Figure 9 demonstrates that the developed controller substantially reduces the communication burden of the MUS.

(5) NN weight

Figure 10 illustrates the behavior of the critic NN via RBFNN under coordinate control facilitated by the proposed approximate optimal control method. The converged weights obtained from the proposed approximate optimal control policies allow the NN to accurately reflect the ongoing coordinate operations in real-time.

Based on the experimental results, the closed-loop MUS systems have better performance than the existing methods in terms of position tracking and control torque under the proposed ETRL via the NZSG approach (cf.

Table 1. Performance comparisons.

	Mean Absolute Value of Position Error	Mean Absolute Value of Control Torque
The existing method (Joint 1)	$1.73\times {10}^{-3}$ rad	0.32 Nm
The proposed method (Joint 1)	$1.03\times {10}^{-3}$ rad	0.29 Nm
The existing method (Joint 2)	$1.62\times {10}^{-3}$ rad	0.30 Nm
The existing method (Joint 2)	$0.98\times {10}^{-3}$ rad	0.26 Nm

). Drawing from the experimental figure findings, when compared to existing methods, the closed-loop MUS demonstrates enhanced performance in position tracking, control torque, contact force, and event-triggered conditions under the proposed approximate optimal control method.

6. Conclusions

An ETRL-based coordinate control of MUS via NZSG is proposed in the paper. JTF is utilized to form the MUS’s dynamic. The coordinate control problem is transformed into an RL issue via the NZSG strategy. Conventional periodic communication is avoided by the ET mechanism. The performance index function is approximated by the critic NN to obtain the optimal control strategy. According to the Lyapunov theorem, the closed-loop system is guaranteed to be stable. The experimental results show that the proposed method reduces the position error by 30% and control torque by 10% compared with the existing control methods. The mentioned control algorithm only concerns the static event-trigger. However, the computation burden and power consumption can be optimized by the dynamic event-trigger or self-event-trigger. This is the future research direction that we will work on.