Research on Motion Control of Hydraulic Manipulator Based on Prescribed Performance and Reinforcement Learning

Abstract

Achieving high-precision motion control for hydraulic manipulators presents a challenging task. Addressing the issue of low motion control accuracy caused by the strong electromechanical-hydraulic coupling characteristics of hydraulic manipulator systems, this paper innovatively introduces an RBF neural network and an Actor–Critic reinforcement learning architecture within a performance-based control framework designed using the inverse method. This approach enables dual compensation for both internal uncertainties and external disturbances within the manipulator, thereby enhancing the system’s control performance. First, within the control architecture, the performance function ensures system transient performance while employing an RBF neural network to estimate and compensate for internal unmodeled errors caused by mechanical coupling and hydraulic parameter uncertainties. Stability proofs are used to derive the network weight update rate. Second, a disturbance compensator is designed based on reinforcement learning. Deployed into the controller through offline training and online adaptation, it compensates for external system disturbances, further improving control accuracy. Finally, comparative and ablation experiments conducted on a hydraulic manipulator testbed demonstrate the effectiveness of the disturbance compensator. Compared to PID control, the proposed approach achieves a 60–65% improvement in control accuracy.

1. Introduction

Manipulators serve as indispensable high-tech equipment in energy and public welfare sectors such as mining operations, emergency rescue, urban and rural infrastructure development, and industrial applications. They represent high-value-added, multifunctional flagship products [1,2]. They increasingly meet market demands for high-precision motion control amid the trend toward intelligentization. Not only do they significantly enhance operational efficiency and work quality, but they also reduce human workload, improve construction safety and automation levels, and lay the foundation for intelligent development [3].

The primary methods currently applied to the motion control of manipulators include Proportional Integral Derivative (PID) control [4], adaptive robust control [5,6], fuzzy logic control [7], improved Linear Quadratic Regulator [8], and sliding mode control [9]. Although PID, as a typical model-free controller, offers advantages such as convenient parameter tuning and simple structure, its inability to effectively address the strong electromechanical-hydraulic coupling characteristics of manipulators results in low control accuracy, making it difficult to meet current market demands [10]. Furthermore, unlike electrically driven manipulators, hydraulically driven manipulators present unpredictable, complex operating conditions and external disturbances. Their inherent joint strong coupling characteristics and uncertainties in hydraulic system parameters, specifically, that manipulators commonly exhibit nonlinear differential equation structures, friction nonlinearities, and uncertainties in parameters such as load, hydraulic oil volumetric elastic modulus, and viscous friction coefficients. Additionally, external disturbances, oil leakage rates, and dynamic friction forces present difficult-to-model uncertain nonlinearities [11,12]. These factors compromise the accuracy of model-based control algorithms. There is an urgent need for a control algorithm that can overcome the challenges inherent in hydraulic manipulators and enhance their motion control precision.

To enhance control accuracy, prior work introduced extended state observers (ESOs) and nonlinear disturbance observers (NDOs) to monitor system uncertainties and disturbances [13]. A classic approach integrates these observers into adaptive robust control strategies, enabling both precise estimation of unknown disturbances and effective handling of parameter uncertainties [14]. Building upon this foundation, Zhang et al. proposed an adaptive robust control method based on a dual extended state observer to observe the uncertainties in a hydraulic manipulator system, achieving good tracking accuracy [15]. Shi et al. effectively resolved the trade-off between tracking accuracy and disturbance rejection capability by employing a disturbance compensation mechanism based on NDO to address external disturbances and unmodeled dynamics [16]. Although the aforementioned methods effectively compensate for parameter and unmodeled nonlinearities, they remain sensitive to sensor noise. The high computational demands for real-time operation introduce significant control feedback delays, hindering practical implementation.

In manipulator motion control, neural networks are widely employed as feedforward compensators to address system nonlinearity and model uncertainty due to their strong approximation capabilities and learning capacity [17,18]. Asko Rouvinen et al.approximate the uncertainties in the system by using different methods of neural network, and obtain good control performance [19]. Tran et al. designed an adaptive control method for multi-degree-of-freedom hydraulic manipulators, employing RBF neural networks to approximate and compensate for uncertainties, achieving high control performance [20]. Su et al. proposed a manipulator control method combining adaptive neural networks with backstepping control to compensate for unknown system dynamics, enhancing control accuracy [21].

The aforementioned control algorithms primarily focus on nonlinear compensation and steady-state tracking performance improvement, without considering the transient performance of the control system. In current research, prescriptive performance control is achieved by setting different performance functions [22,23], keeping the tracking error within an adjustable range and demonstrating significant transient and steady-state performance. Wang Shubo et al. proposed an adaptive funnel control scheme for manipulators with unknown blind zones. Error transformation was applied in controller design [24]. Yang et al. introduced a novel adaptive dynamic surface asymptotic tracking controller based on neural networks, approximating unknown dynamics while ensuring tracking error remains within predetermined boundaries via performance functions, achieving high transient tracking performance [25]. Liang Xianglong et al. proposed an adaptive neural network admittance control method based on integral obstacle Lyapunov functions, achieving excellent position and force tracking control while ensuring system outputs remain within preset limits [26].

In recent years, researchers have explored applying reinforcement learning to hydraulic manipulators. As an intelligent algorithm applicable to both model-based and model-free scenarios, its Actor–Critic architecture has seen the widest industrial adoption [27]. Yao et al. combined the Actor–Critic reinforcement learning architecture with PID control, achieving high motion accuracy through online PID parameter tuning [28]. Additionally, when integrated with RISE controllers, reinforcement learning estimated and compensated for unmodeled errors in the system, similarly yielding high control precision [29]. These studies demonstrate the feasibility of applying reinforcement learning to hydraulic manipulators.

Building on the above research, this paper addresses the current challenge in hydraulic manipulator control algorithms, which struggle to simultaneously accommodate both system transient response and internal/external uncertainties. By employing a performance-based controller as the foundational architecture, the system’s position output error can be constrained within predetermined boundaries, thereby ensuring transient performance. Furthermore, an RBF neural network is introduced to estimate and compensate for unmodeled internal errors. Simultaneously, a disturbance compensator based on the Actor–Critic reinforcement learning architecture is designed to handle external disturbance uncertainties. This dual compensation mechanism significantly enhances the system’s motion control accuracy.

The rest of this paper is as follows. Section 2 constructs the excavator model; Section 3 designs the proposed control algorithm; Section 4 analyzes the algorithm’s stability and convergence; Section 5 experimentally validates the algorithm’s application; Section 6 summarizes the paper.

2. Hydraulic Manipulator Model

The hydraulic manipulator studied in this paper is modified from an excavator, with its structure shown in Figure 1. This paper only considers the compound motion of the boom and arm joints within a single plane. The dynamics of the hydraulic manipulator in the joint space can be described as follows:

The force balance equation of inertial load can be expressed as:

$$M(q)\ddot{q}+C(q,\dot{q})\dot{q}+B\dot{q}+G(q)+\Delta M(q)\ddot{q}+\Delta C(q,\dot{q})\dot{q}+\Delta B\dot{q}+\Delta G(q)=\tau$$

(1)

where $M(q)\in {\mathrm{R}}^{3\times 3}$ represents the inertia matrix; $C(q,\dot{q})\in {\mathrm{R}}^{3\times 3}$ denotes the Coriolis and centrifugal force matrix; $G(q)\in {\mathrm{R}}^3$ signifies the gravitational force matrix; $B\in {\mathrm{R}}^{3\times 3}$ indicates the viscous friction coefficient matrix; $q\in {\mathrm{R}}^3$, $\dot{q}\in {\mathrm{R}}^3$, and $\ddot{q}\in {\mathrm{R}}^3$ respectively denote the joint angular displacement, angular velocity, and angular acceleration vectors; $\Delta M(q)$, $\Delta C(q,\dot{q})$, $\Delta G(q)$, $\Delta B$ represent the unmodeled portions of joint coupling dynamics; $\tau$ denotes the torque vector acting on the joint.

The motion of this hydraulic manipulator is driven by a single-rod hydraulic cylinder, as shown in Figure 2. Let $x_{ha}={[x_{ha1},x_{ha2}]}^T$ denote the displacement vector of the boom and arm hydraulic cylinder, i.e., the displacement vector in the actuator space. Then, the velocity vector ${\dot{x}}_{ha}$ can be expressed in terms of the joint vectors $q$ and $\dot{q}$ as follows:

$${\dot{x}}_{ha}={\displaystyle \frac{\partial x_{ha}}{\partial q}}\dot{q}=diag\left\{{\displaystyle \frac{\partial x_{ha1}}{\partial q_1}},{\displaystyle \frac{\partial x_{ha2}}{\partial q_2}}\right\}\dot{q}$$

(2)

The torque vector acting on the manipulator joint can be expressed as:

$$\tau =J_{ha}(A_1{P}_1-A_2{P}_2)={\displaystyle \frac{\partial x_{ha}}{\partial q}}(A_1{P}_1-A_2{P}_2)$$

(3)

where $A_1$ is the piston area of the rodless chamber of the hydraulic cylinder; $A_2$ is the piston area of the rod chamber of the hydraulic cylinder; $P_1$ is the pressure in the rodless chamber of the hydraulic cylinder; $P_2$ is the pressure in the rod chamber of the hydraulic cylinder;

Neglecting internal leakage, the dynamic pressure equations for the rod chamber and rodless chamber of the hydraulic cylinder can be expressed as [30]:

$${\dot{P}}_1={\beta }_e\left(-A_1{\dot{x}}_{ha}+Q_1\right)/V_1 {\dot{P}}_2={\beta }_e\left(A_2{\dot{x}}_{ha}-Q_2\right)/V_2$$

(4)

where ${\beta }_e$ is the effective volumetric modulus of the oil; $V_1=V_{01}+A_1\cdot x_{ha}$ is the rodless chamber volume of the hydraulic cylinder; $V_{01}$ is the initial volume of the rodless chamber; $V_2=V_{02}-A_2\cdot x_{ha}$ is the rod chamber volume; $V_{02}$ is the initial volume of the rod chamber; $Q_1$ is the supply flow rate to the hydraulic cylinder; $Q_2$ is the return flow rate from the hydraulic cylinder.

The hydraulic cylinder flow equation is:

$$\begin{array}{l}{Q}_1=k_{v}u\left[s\left(u\right)\sqrt{\left(P_s-P_1\right)}+s\left(-u\right)\sqrt{\left(P_1-P_r\right)}\right] \\ Q_2=k_{v}u\left[s\left(u\right)\sqrt{\left(P_2-P_r\right)}+s\left(-u\right)\sqrt{\left(P_s-P_2\right)}\right]\end{array}$$

(5)

where $k_v$ represents flow gain; $u$ denotes the proportional valve control signal; $P_s$ indicates the supply pressure; and $P_r$ signifies the return pressure.

Define state variable $x={[x_1,x_2,x_3]}^T$, where $x_1=q$, $x_2=\dot{q}$, $x_3=A_1{P}_1-A_2{P}_2$. The state equation is:

$$\begin{array}{c}{\dot{x}}_1=x_2\hfill \\ {\dot{x}}_2=M^{-1}(J_{ha}{x}_3-N_1\left(x_2\right)+N_2\left(x_2\right))\hfill \\ {\dot{x}}_3=g_1(x_3)u-g_2(x_2)\hfill \end{array}$$

(6)

where

$$\left\{\begin{array}{l}{N}_1\left(x_2\right)=C{x}_2+B{x}_2+G N_2\left(x_2\right)=-\Delta M{\dot{x}}_2-\Delta C{x}_2-\Delta B{x}_2-\Delta G\\ g_1(x_3)={\beta }_e{k}_v\left(A_1{R}_1/V_1+A_2{R}_2/V_2\right) g_2(x_2)={\beta }_e{J}_{ha}\left(A_1^2/V_1+A_2^2/V_2\right)x_2\\ R_1=s\left(u\right)\sqrt{P_s-P_1}+s\left(-u\right)\sqrt{P_1-P_r} R_2=s\left(u\right)\sqrt{P_2-P_r}+s\left(-u\right)\sqrt{P_s-P_2}\end{array}\right.$$

(7)

3. Controller Design

This section first designs a performance-based controller that employs an RBF neural network to estimate and compensate for unmodeled mechanical and hydraulic errors in the manipulator. Subsequently, building upon this performance controller, a disturbance compensator is designed using the Actor–Critic reinforcement learning architecture to counteract external disturbances, thereby further enhancing control accuracy.

3.1. RBF Neural Network

The RBF neural network consists of an input layer, a hidden layer, and an output layer. Due to its simple structure, fast learning convergence, and ability to approximate continuous nonlinear functions on compact sets with arbitrary precision, it is commonly used in controllers for real-time fitting of unmodeled errors. Adaptive laws designed based on Lyapunov stability theory can adjust network weights online, enabling the controller to compensate for unmodeled errors while ensuring the stability and tracking performance of the closed-loop system [31]. The transformation from the input layer to the hidden layer is nonlinear, while the transformation from the hidden layer to the output layer is linear. The activation function of the neurons in the hidden layer is the radial basis function. Any continuous unknown function can be approximated as:

$$F(X)=W^{*T}\varphi (X)+\epsilon$$

(8)

where $W^{*T}\in {ℝ}^{c\times v}$ is the weight vector, $c$ and $v$ represent the number of outputs and the number of hidden layer neurons, respectively; $X\in {ℝ}^{z\times l}$ is the input to the neural network, $z$ is the number of inputs; $\epsilon$ is the neural network approximation error satisfying $‖\epsilon ‖\le \overline{\epsilon }$, $\overline{\epsilon }$ is a positive constant; $\varphi (X)=[{\varphi }_1(X),{\varphi }_2(X),\cdots ,{\varphi }_l(X)]$ is the activation function, expressed as:

$${\varphi }_i(X)=\exp \left(\begin{array}{r}\hfill -{(X-{\mu }_i)}^T(X_1-{\mu }_i)/{\sigma }_i^2\end{array}\right),i=1,2,l$$

(9)

where ${\mu }_i$ and ${\sigma }_i$ represent the center and width of the Gaussian function for the $i$ th neuron, respectively.

3.2. Performance Function

The time-varying logarithmic Lyapunov function is designed as follows:

$$V={\displaystyle \frac{1}{2}}\ln {\displaystyle \frac{k_a^2\left(t\right)}{k_a^2\left(t\right)-z^{T}z}}$$

(10)

where $k_a\left(t\right)$ represents the error constraint boundary, $k_a\left(t\right)>0$; $z$ denotes tracking errors.

As shown in Equation (10), when the system tracking error variable exceeds the time-varying boundary, the function value of the time-varying log-type Lyapunov function tends toward infinity. Based on this characteristic, by rigorously proving the boundedness of this Lyapunov function under the controller’s action, it is ensured that the closed-loop system tracking error strictly remains within the preset event constraint boundary $\left(-k_a\left(t\right),k_a\left(t\right)\right)$, where $k_{ai}\left(t\right)$ is expressed as:

$$k_a(t)=(k_{a0}-k_{a\infty }){\exp }^{-nt}+k_{a\infty }$$

(11)

where $k_{a0}$, $k_{a\infty }$, $n$ are constants greater than zero and satisfy $0<k_{a\infty }<k_{a0}$, $‖z\left(0\right)‖<k_{a0}$; $z\left(0\right)$ is the initial value of $z$. The image of $k_a(t)$ is shown in Figure 3:

3.3. Prescribed Performance Controller Design

Define the error variable as:

$$z_1=x_1-x_{1d}, z_2=x_2-{\alpha }_1, z_3=x_3-{\alpha }_2$$

(12)

where ${\alpha }_1$, ${\alpha }_2$ are virtual control variables.

Define the first Lyapunov function as:

$$V_1={\displaystyle \frac{1}{2}}\ln {\displaystyle \frac{k_{b1}^2\left(t\right)}{k_{b1}^2\left(t\right)-z_1^T{z}_1}}$$

(13)

The derivation of $V_1$ yields:

$${\dot{V}}_1={\displaystyle \frac{z_1^T}{k_{b1}^2-z_1^T{z}_1}}\left(x_2-{\dot{x}}_{1d}-{\displaystyle \frac{{\dot{k}}_{b1}}{k_{b1}}}{z}_1\right)={\displaystyle \frac{z_1^T}{k_{b1}^2-z_1^T{z}_1}}\left(z_2+{\alpha }_1-{\dot{x}}_{1d}-{\displaystyle \frac{{\dot{k}}_{b1}}{k_{b1}}}{z}_1\right)$$

(14)

For the convenience of representation, in the following, $\ln {\left(·\right)}_i$ is used to denote $\ln (k_{bi}^2\left(t\right)/k_{bi}^2\left(t\right)-z_i^T{z}_i)$, $i=1,2,3$, and the virtual control rate ${\alpha }_1$ is designed based on Equation (14) and the fixed time theory as:

$${\alpha }_1={\dot{x}}_{1d}+{\displaystyle \frac{{\dot{k}}_{b1}}{k_{b1}}}{z}_1-k_{11}^{\ast }({\displaystyle \frac{{\ln }^{{\gamma }_1}{(·)}_1(k_{b1}^2-z_1^T{z}_1)}{z_1^T}})-k_{12}^{\ast }({\displaystyle \frac{{\ln }^{{\gamma }_2}{(·)}_1(k_{b1}^2-z_1^T{z}_1)}{z_1^T}})$$

(15)

where $k_{11}^{\ast }$ and $k_{12}^{\ast }$ are constants greater than zero; $0<{\gamma }_1<1$, $1<{\gamma }_2<\infty$.

Substituting Equation (15) into Equation (14) yields:

$${\dot{V}}_1={\displaystyle \frac{z_1^T{z}_2}{k_{b1}^2-z_1^T{z}_1}}-k_{11}^{\ast }{\ln }^{{\gamma }_1}{(·)}_1-k_{12}^{\ast }{\ln }^{{\gamma }_2}{(·)}_1$$

(16)

Further, the second Lyapunov function is designed:

$$V_2=V_1+{\displaystyle \frac{1}{2}}\ln {\displaystyle \frac{k_{b2}^2\left(t\right)}{k_{b2}^2\left(t\right)-z_2^T{z}_2}}+{\displaystyle \frac{1}{2{\eta }_{w1}}}{\tilde{W}}_1^T{\tilde{W}}_1+{\displaystyle \frac{1}{2{\eta }_{\vartheta 1}}}{\tilde{\vartheta }}_1^T{\tilde{\vartheta }}_1$$

(17)

where ${\tilde{W}}_1=W_1^{\ast }-{\widehat{W}}_1$, ${\tilde{\vartheta }}_1={\vartheta }_1-{\widehat{\vartheta }}_1$, ${\eta }_{w1}$ and ${\eta }_{\vartheta 1}$ are constants greater than zero.

The derivation of Equation (17) is obtained:

$$\begin{array}{cc}\hfill {\dot{V}}_2=& {\dot{V}}_1+{\displaystyle \frac{z_2^T}{k_{b2}^2-z_2^T{z}_2}}\left[{\displaystyle \frac{J_{ha}}{M}}{x}_3-{\displaystyle \frac{N_1(x_2)}{M}}+{\displaystyle \frac{N_2(x_2)}{M}}-\right.\left.{\dot{\alpha }}_1-{\displaystyle \frac{{\dot{k}}_{b2}}{k_{b2}}}{z}_2\right]-{\displaystyle \frac{1}{{\eta }_{w1}}}{\tilde{W}}_1^T{\dot{\widehat{W}}}_1-{\displaystyle \frac{1}{{\eta }_{\vartheta 1}}}{\tilde{\vartheta }}_1^T{\dot{\widehat{\vartheta }}}_1\hfill \\ \hfill =& -k_{11}^{\ast }{\ln }^{{\gamma }_1}{(·)}_1-k_{12}^{\ast }{\ln }^{{\gamma }_2}{(·)}_1+{\displaystyle \frac{z_2^T}{k_{b2}^2-z_2^T{z}_2}}\left[{\displaystyle \frac{J_{ha}}{M}}{z}_3+{\displaystyle \frac{J_{ha}}{M}}{\alpha }_2\right.-\hfill \\ & \left.{\displaystyle \frac{N_2(x_2)}{M}}+f_1-{\displaystyle \frac{{\dot{k}}_{b2}}{k_{b2}}}{z}_2\right]-{\displaystyle \frac{1}{{\eta }_{w1}}}{\tilde{W}}_1^T{\dot{\widehat{W}}}_1-{\displaystyle \frac{1}{{\eta }_{\vartheta 1}}}{\tilde{\vartheta }}_1^T{\dot{\widehat{\vartheta }}}_1\hfill \end{array}$$

(18)

where $f_1={\displaystyle \frac{N_2(x_2)}{M}}+{\displaystyle \frac{d_1}{M}}-{\dot{\alpha }}_1+{\displaystyle \frac{(k_{b2}^2-z_2^T{z}_2)z_1}{k_{b1}^2-z_1^T{z}_1}}$.

The virtual control volume ${\alpha }_2$ is designed according to Equation (18) as:

$${\alpha }_2= {\displaystyle \frac{M}{J_{ha}}}\left[-{\widehat{f}}_1+{\displaystyle \frac{N_1(x_2)}{M}}+{\displaystyle \frac{{\dot{k}}_{b2}}{k_{b2}}}{z}_2\right]-{\displaystyle \frac{M}{J_{ha}}}{\displaystyle \frac{k_{a2}^2-z_2^T{z}_2}{z_2^T}}\left[k_{21}^{\ast }{\ln }^{{\gamma }_1}{(·)}_2+k_{22}^{\ast }{\ln }^{{\gamma }_2}{(·)}_2\right]$$

(19)

where $k_{21}^{\ast }$ and $k_{22}^{\ast }$ are constants greater than zero.

Let ${\widehat{f}}_1={\widehat{W}}_1^T\varphi (X_1)+{\widehat{\vartheta }}_1$ and substituting together with Equation (19) into Equation (18) yields:

$$\begin{array}{cc}{\dot{V}}_2& ={\displaystyle \sum _{i=1}^2\left[-k_{i1}{\ln }^{{\gamma }_1}{(·)}_i-k_{i2}{\ln }^{{\gamma }_2}{(·)}_i\right]}+{\displaystyle \frac{z_2^T}{k_{b2}^2-z_2^T{z}_2}}\left[{\tilde{W}}_1^T\varphi (X_1)+{\tilde{\vartheta }}_1\right]\hfill \\ & -{\displaystyle \frac{1}{{\eta }_{w1}}}{\tilde{W}}_1^T{\dot{\widehat{W}}}_1+{\displaystyle \frac{z_2^T}{k_{b2}^2-z_2^T{z}_2}}{\displaystyle \frac{J_{ha}}{M}}{z}_3-{\displaystyle \frac{1}{{\eta }_{\vartheta 1}}}{\tilde{\vartheta }}_1^T{\dot{\widehat{\vartheta }}}_1\hfill \end{array}$$

(20)

According to Equation (20), the neural network weight update rate can be obtained as:

$${\dot{\widehat{W}}}_1={\eta }_{w1}{\displaystyle \frac{z_2^T}{k_{b2}^2-z_2^T{z}_2}}\varphi (X_1)-{\sigma }_{w1}{\widehat{W}}_1 {\dot{\widehat{\vartheta }}}_1={\eta }_{\vartheta 1}{\displaystyle \frac{z_2^T}{k_{b2}^2-z_2^T{z}_2}}-{\sigma }_{\vartheta 1}{\widehat{\vartheta }}_1$$

(21)

where both ${\sigma }_{w1}$ and ${\sigma }_{\vartheta 1}$ are constants greater than zero; $X_1=[q,\dot{q},q_d,{\dot{q}}_d,{\alpha }_1]$. Substituting Equation (21) has:

$${\dot{V}}_2={\displaystyle \sum _{i=1}^2\left[-k_{i1}{\ln }^{{\gamma }_1}{(·)}_i-k_{i2}{\ln }^{{\gamma }_2}{(·)}_i\right]}+{\displaystyle \frac{{\sigma }_{w1}}{{\eta }_{w1}}}{\tilde{W}}_1^T{\widehat{W}}_1+{\displaystyle \frac{{\sigma }_{\vartheta 1}}{{\eta }_{\vartheta 1}}}{\tilde{\vartheta }}_1^T{\widehat{\vartheta }}_1+{\displaystyle \frac{z_2^T}{k_{b2}^2-z_2^T{z}_2}}{\displaystyle \frac{J_{ha}}{M}}{z}_3$$

(22)

According to Young’s inequality,

$${\tilde{W}}_1^T{\widehat{W}}_1\le -{\displaystyle \frac{1}{2}}{\tilde{W}}_1^T{\tilde{W}}_1+{\displaystyle \frac{1}{2}}{W}_1^{\ast T}{W}_1^{\ast } {\tilde{\vartheta }}_1^T{\widehat{\vartheta }}_1\le -{\displaystyle \frac{1}{2}}{\tilde{\vartheta }}_1^T{\tilde{\vartheta }}_1+{\displaystyle \frac{1}{2}}{\vartheta }_1^{\ast T}{\vartheta }_1^{\ast }$$

(23)

Further, the deflation of Equation (22) yields:

$$\begin{array}{ll}{\dot{V}}_2\le & {\displaystyle \sum _{i=1}^2\left[-k_{i1}{\ln }^{{\gamma }_1}{(·)}_i-k_{i2}{\ln }^{{\gamma }_2}{(·)}_i\right]}-{\displaystyle \frac{{\sigma }_{w1}}{2{\eta }_{w1}}}{\tilde{W}}_1^T{\tilde{W}}_1-{\displaystyle \frac{{\sigma }_{\vartheta 1}}{2{\eta }_{\vartheta 1}}}{\tilde{\vartheta }}_1^T{\tilde{\vartheta }}_1+\hfill \\ & {\displaystyle \frac{{\sigma }_{w1}}{2{\eta }_{w1}}}{W}_1^{\ast T}{W}_1^{\ast }+{\displaystyle \frac{{\sigma }_{\vartheta 1}}{2{\eta }_{\vartheta 1}}}{\vartheta }_1^{\ast T}{\vartheta }_1^{\ast }+{\displaystyle \frac{z_2^T}{k_{b2}^2-z_2^T{z}_2}}{\displaystyle \frac{J_{ha}}{M}}{z}_3\hfill \end{array}$$

(24)

It can be shown that if $z_3$ converges, then $V_2$ eventually reaches consistent boundedness. Define the third Lyapunov function as:

$$V_3=V_2+{\displaystyle \frac{1}{2}}\ln {\displaystyle \frac{k_{b3}^2\left(t\right)}{k_{b3}^2\left(t\right)-z_3^T{z}_3}}+{\displaystyle \frac{1}{2{\eta }_{w2}}}{\tilde{W}}_2^T{\tilde{W}}_2+{\displaystyle \frac{1}{2{\eta }_{\vartheta 2}}}{\tilde{\vartheta }}_2^T{\tilde{\vartheta }}_2$$

(25)

Derivation of Equation (25) yields:

$$\begin{array}{cc}\hfill {\dot{V}}_3=& {\dot{V}}_2+{\displaystyle \frac{z_3^T}{k_{b3}^2-z_3^T{z}_3}}\left[g_1(x_3)u-g_2(x_2)-{\dot{\alpha }}_2-{\displaystyle \frac{{\dot{k}}_{b3}}{k_{b3}}}{z}_3\right]-{\displaystyle \frac{1}{{\eta }_{w2}}}{\tilde{W}}_2^T{\dot{\widehat{W}}}_2-{\displaystyle \frac{1}{{\eta }_{\vartheta 2}}}{\tilde{\vartheta }}_2^T{\dot{\widehat{\vartheta }}}_2\hfill \\ \hfill \le & {\displaystyle \sum _{i=1}^2\left[-k_{i1}{\ln }^{{\gamma }_1}{(·)}_i-k_{i2}{\ln }^{{\gamma }_2}{(·)}_i\right]}-{\displaystyle \frac{{\sigma }_{w1}}{2{\eta }_{w1}}}{\tilde{W}}_1^T{\tilde{W}}_1-{\displaystyle \frac{{\sigma }_{\vartheta 1}}{2{\eta }_{\vartheta 1}}}{\tilde{\vartheta }}_1^T{\tilde{\vartheta }}_1+{\displaystyle \frac{{\sigma }_{w1}}{2{\eta }_{w1}}}{W}_1^{\ast T}{W}_1^{\ast }+\hfill \\ & {\displaystyle \frac{{\sigma }_{\vartheta 1}}{2{\eta }_{\vartheta 1}}}{\vartheta }_1^{\ast T}{\vartheta }_1^{\ast }+{\displaystyle \frac{1}{2}}{‖\epsilon ‖}^2+{\displaystyle \frac{z_3^T}{k_{b3}^2-z_3^T{z}_3}}\left[g_1(x_3)u-g_2(x_2)\right.+\left.f_2-{\displaystyle \frac{{\dot{k}}_{b3}}{k_{b3}}}{z}_3\right]\hfill \\ & -{\displaystyle \frac{1}{{\eta }_{w2}}}{\tilde{W}}_2^T{\dot{\widehat{W}}}_2-{\displaystyle \frac{1}{{\eta }_{\vartheta 2}}}{\tilde{\vartheta }}_2^T{\dot{\widehat{\vartheta }}}_2\hfill \end{array}$$

(26)

where $f_2=-{\dot{\alpha }}_2+{\displaystyle \frac{(k_{b3}^2-z_3^T{z}_3)z_2{J}_{ha}{M}^{-1}}{k_{b2}^2-z_2^T{z}_2}}$.

The control input $u$ is designed according to Equation (26) as:

$$u={\displaystyle \frac{1}{g_1(x_3)}}\left[g_2(x_2)\right.-{\widehat{f}}_2+{\displaystyle \frac{{\dot{k}}_{b3}}{k_{b3}}}{z}_3-\left.{\displaystyle \frac{k_{b3}^2-z_3^T{z}_3}{z_3^T}}(-k_{31}{\ln }^{{\gamma }_1}{(·)}_3-k_{32}{\ln }^{{\gamma }_2}{(·)}_3)\right]$$

(27)

where $k_{31}^{\ast }$ and $k_{32}^{\ast }$ are constants greater than zero;

Let ${\widehat{f}}_2={\widehat{W}}_2^T\varphi (X_2)+{\widehat{\vartheta }}_2$ and substituting together with Equation (27) into Equation (26) has:

$$\begin{array}{ll}{\dot{V}}_3\le & {\displaystyle \sum _{i=1}^3\left[-k_{i1}{\ln }^{{\gamma }_1}{(·)}_i-k_{i2}{\ln }^{{\gamma }_2}{(·)}_i\right]}-{\displaystyle \frac{{\sigma }_{w1}}{2{\eta }_{w1}}}{\tilde{W}}_1^T{\tilde{W}}_1-{\displaystyle \frac{{\sigma }_{\vartheta 1}}{2{\eta }_{\vartheta 1}}}{\tilde{\vartheta }}_1^T{\tilde{\vartheta }}_1+\hfill \\ & {\displaystyle \frac{{\sigma }_{w1}}{2{\eta }_{w1}}}{W}_1^{\ast T}{W}_1^{\ast }+{\displaystyle \frac{{\sigma }_{\vartheta 1}}{2{\eta }_{\vartheta 1}}}{\vartheta }_1^{\ast T}{\vartheta }_1^{\ast }-{\displaystyle \frac{1}{{\eta }_{w2}}}{\tilde{W}}_2^T{\dot{\widehat{W}}}_2-{\displaystyle \frac{1}{{\eta }_{\vartheta 2}}}{\tilde{\vartheta }}_2^T{\dot{\widehat{\vartheta }}}_2+\hfill \\ & {\displaystyle \frac{z_3^T}{k_{b3}^2-z_3^T{z}_3}}({\tilde{W}}_2^T\varphi (X_2)+{\tilde{\vartheta }}_2)\hfill \end{array}$$

(28)

According to Equation (28), the neural network weight update rate can be obtained as:

$${\dot{\widehat{W}}}_2={\eta }_{w2}{\displaystyle \frac{z_3^T}{k_{b3}^2-z_3^T{z}_3}}\varphi (X_2)-{\sigma }_{w2}{\widehat{W}}_2 {\dot{\widehat{\vartheta }}}_2={\eta }_{\vartheta 2}{\displaystyle \frac{z_3^T}{k_{b3}^2-z_3^T{z}_3}}-{\sigma }_{\vartheta 2}{\widehat{\vartheta }}_2$$

(29)

where both ${\sigma }_{w2}$ and ${\sigma }_{\vartheta 2}$ are constants greater than zero; $X_2=[q,\dot{q},q_d,{\dot{q}}_d,{\alpha }_2]$.

3.4. Disturbance Compensator

This subsection designs the disturbance compensator based on the reinforcement learning Actor–Critic architecture to compensate for external disturbances based on Equation (27), i.e., [32]:

$$u={\displaystyle \frac{1}{g_1(x_3)}}[g_2(x_2)-{\widehat{f}}_2+{\displaystyle \frac{{\dot{k}}_{b3}}{k_{b3}}}{z}_3-{\displaystyle \frac{k_{b3}^2-z_3^T{z}_3}{z_3^T}}(-k_{31}{\ln }^{{\gamma }_1}{(·)}_3-k_{32}{\ln }^{{\gamma }_2}{(·)}_3)]+F_D$$

(30)

where $F_D=F_D(x(k))$, is the disturbance compensator designed in this paper.

Define the stage cost function as:

$$\delta (k)=Z_1^T(k)M{Z}_1(k)+u^T(k)Pu(k)$$

(31)

where $M\in {ℝ}^{3\times 3}$, $P\in {ℝ}^{3\times 3}$, are semipositive definite matrices. Parameter $M$ serves to fully account for the impact of error on the system’s control accuracy. Meanwhile, to prevent the output signal from becoming excessively large, parameter $P$ is added to constrain the output signal.

According to reinforcement learning theory, the long-term reward function is defined as:

$$J(x(k),F_D(k))={\displaystyle \sum _{j=k}^{\infty }{\lambda }^{j-k}}\delta (k)$$

(32)

where $0<\lambda <1$, representing the reward discount factor, according to Equation (32), the following Bellman equation holds:

$$J^{\ast }(x(k),F_D(k))=\delta (k)+\lambda J^{\ast }(x(k+1),F_D^{\ast }(k+1))$$

(33)

where $J^{\ast }(x(k),F_D(k))$ and $F_D^{\ast }(k)$ are the optimal value function and optimal strategy, respectively, and $F_D^{\ast }(k)$ is expressed as:

$$F_D^{\ast }(k)=\arg \min J^{\ast }(x(k),F_D(k))$$

(34)

This paper employs the Q-learning algorithm to solve the equations, where the actor–critic neural network utilizes a 2 × 64 fully connected layer with a hyperbolic tangent activation function.

The state action function $Q(k)$ is defined as based on Equation (32):

$$Q(x(k),F_D(k))=\delta (k)+{\displaystyle \sum _{j=1}^{\infty }{\lambda }^j}\delta (k+j)$$

(35)

This paper starts with $Q_0(x(k),F_D(k))=0$ and solves the initial control strategy ${\pi }_0(k)$ using the following equation:

$$F_{D0}(k)=\arg \min Q_0(x(k),F_D(k))$$

(36)

Then, $Q_1$ can be computed in the next iteration using the determined strategy $F_{D0}(k)$:

$$Q_1(x(k),F_D(k))=\delta (k)+\lambda Q_0(x(k+1),F_{D0}(k+1))$$

(37)

By means of mathematical induction, there is the following:

$$F_{Di}(k)=\arg \min Q_i(x(k),F_D(k))$$

(38)

$$Q_i(x(k),F_D(k))=\delta (k)+\lambda Q_i(x(k+1),F_{Di}(k+1))$$

(39)

where $i$ is the iteration index.

By means of Equations (38) and (39), there is the following Bellman equation:

$$Q_{i+1}(x(k),F_D(k))=r(k)+\lambda \min Q_i(x(k+1),F_D(k+1))$$

(40)

In this paper, critic neural network is used to approximate the $Q(k)$. There is:

$${\widehat{Q}}_i(k)={\widehat{w}}_{c2,i}(k)\ast \sigma ({\widehat{w}}_{c1,i}(k)\ast x_c(k))={\widehat{w}}_{c2,i}(k)\ast {\sigma }_{c,i}(k)$$

(41)

where ${\widehat{w}}_{c1,i}$ and ${\widehat{w}}_{c2,i}$ represent neural network weight matrices, and $\ast$ denotes the inner product.

The input of the critic neural network is:

$$x_c(k)={[x_a(k),u(k)]}^T$$

(42)

where $x_a(k)={[x_1(k),x_2(k),x_3(k),e_1(k),x_d(k)]}^T$.

From the Bellman Equation (39), the prediction error of the critic neural network is:

$$e_{c,i+1}(k)=\delta (k)+\lambda {\widehat{Q}}_i(k+1)-{\widehat{Q}}_{i+1}(k)$$

(43)

To design the weight update rate, the following minimization approximation cost equation is designed:

$$E_{c,i+1}(k)=e_{c,i+1}^T(k)e_{c,i+1}(k)/2$$

(44)

In this paper, the gradient descent method is used to update the weights as:

$$\begin{array}{c}\Delta {\widehat{w}}_{c1,i+1}(k)=K_c[-{\displaystyle \frac{\partial E_{c,i+1}(k)}{\partial {\widehat{w}}_{c1,i+1}(k)}}]\hfill \\ {\displaystyle \frac{\partial E_{c,i+1}(k)}{\partial {\widehat{w}}_{c1,i+1}(k)}}={\displaystyle \frac{\partial E_{c,i+1}(k)}{\partial {\widehat{Q}}_i(k)}}{\displaystyle \frac{\partial {\widehat{Q}}_i(k)}{\partial {\sigma }_{c,i}(k)}}{\displaystyle \frac{\partial {\sigma }_{c,i}(k)}{\partial s_{c,i}(k)}}{\displaystyle \frac{\partial s_{c,i}(k)}{\partial {\widehat{w}}_{c1,i}(k)}}=\lambda e_{c,i+1}(k){\widehat{w}}_{c2,i}(k)[{\displaystyle \frac{1}{2}}(1-({\sigma }_{c,i}{(k))}^2)]x_c(k)\hfill \\ \Delta {\widehat{w}}_{c2,i+1}(k)=K_c[-{\displaystyle \frac{\partial E_{c,i+1}(k)}{\partial {\widehat{w}}_{c2,i+1}(k)}}] {\displaystyle \frac{\partial E_{c,i+1}(k)}{\partial {\widehat{w}}_{c2,i+1}(k)}}={\displaystyle \frac{\partial E_{c,i+1}(k)}{\partial {\widehat{Q}}_i(k)}}{\displaystyle \frac{\partial {\widehat{Q}}_i(k)}{\partial {\widehat{w}}_{c2,i}(k)}}=\lambda e_{c,i+1}(k){\sigma }_{c,i}(k)\hfill \end{array}$$

(45)

where $K_c>0$ is the weighted learning step.

In this paper, using actor neural network to output the strategy function $F_D(k)$.

$$F_{Di}(k)={\widehat{w}}_{a2,i}(k)\ast \sigma ({\widehat{w}}_{a1,i}\ast x_a(k))={\widehat{w}}_{a2,i}(k)\ast {\sigma }_{a,i}(k)$$

(46)

where ${\widehat{w}}_{a1,i}$ and ${\widehat{w}}_{a2,i}$ represent neural network weight matrices, and $\ast$ denotes the inner product.

The purpose of the Actor neural network is to minimize the value function ${\widehat{Q}}_i(k)$. Therefore, the following minimization approximation equation is designed:

$$E_{a,i+1}(k)={\widehat{Q}}_{i+1}(k){\widehat{Q}}_{i+1}(k)/2$$

(47)

The weights are updated using the gradient descent method as follows:

$$\begin{array}{cc}\hfill \Delta {\widehat{w}}_{a1,i+1}(k)& =K_a[-{\displaystyle \frac{\partial E_{a,i+1}(k)}{\partial {\widehat{w}}_{a1,i+1}(k)}}]\hfill \\ \hfill {\displaystyle \frac{\partial E_{a,i+1}(k)}{\partial {\widehat{w}}_{a1,i+1}(k)}}& ={\displaystyle \frac{\partial E_{a,i+1}(k)}{\partial {\widehat{Q}}_i(k)}}{[{\displaystyle \frac{\partial {\widehat{Q}}_i(k)}{\partial {\pi }_i(k)}}]}^T{\displaystyle \frac{\partial {\pi }_i(k)}{{\sigma }_{a,i}(k)}}{\displaystyle \frac{{\sigma }_{a,i}(k)}{\partial s_{a,i}(k)}}{\displaystyle \frac{\partial s_{a,i}(k)}{\partial {\widehat{w}}_{a1,i}(k)}}={\widehat{Q}}_i(k){\displaystyle \sum _{l=1}^n{\displaystyle \sum _{r=1}^N[{\widehat{w}}_{c2,i}(k)\frac{1}{2}(1}}\hfill \\ & -({\sigma }_{a,i}{(k))}^2){\widehat{w}}_{c1,i}(k)]{\widehat{w}}_{a2,i}(k){\displaystyle \frac{1}{2}}(1-({\sigma }_{a,i}{(k))}^2)x_a(k)\hfill \\ \hfill \Delta {\widehat{w}}_{a2,i+1}(k)& =K_a[-{\displaystyle \frac{\partial E_{a,i+1}(k)}{\partial {\widehat{w}}_{a2,i+1}(k)}}]\hfill \\ \hfill {\displaystyle \frac{\partial E_{a,i+1}(k)}{\partial {\widehat{w}}_{a2,i+1}(k)}}& ={\displaystyle \frac{\partial E_{a,i+1}(k)}{\partial {\widehat{Q}}_i(k)}}{[{\displaystyle \frac{\partial {\widehat{Q}}_i(k)}{\partial {\pi }_i(k)}}]}^T{\displaystyle \frac{\partial {\pi }_i(k)}{\partial {\widehat{w}}_{a2,i}(k)}}\hfill \\ & =e_a(k){\displaystyle \sum _{l=1}^n{\displaystyle \sum _{r=1}^N[{\widehat{w}}_{c2,i}(k)\frac{1}{2}(1-{({\sigma }_{a,i}(k))}^2){\widehat{w}}_{c1,i}(k)]}}{\sigma }_{a,i}(k)\hfill \end{array}$$

(48)

where $K_a>0$ is the weighted learning step.

4. Stability and Optimality Analysis

This subsection first analyzes the global stability of the prescribed-performance controller based on Lyapunov theory, and second, since the disturbance compensator is an externally attached compensation signal, this subsection analyzes the optimality and suboptimality of its compensation signal.

4.1. Global Stability Analysis of Prescribed Performance Controller

Theorem 1. 

Let a continuous function $V\left(t\right)$ be positive definite and satisfy the following differential equation:

$$\dot{V}(t)\le -{\vartheta }_{a}V+{\vartheta }_b$$

(49)

where ${\vartheta }_a>0$, $0<{\vartheta }_b<1$, at this time, $V\left(t\right)$ satisfies consistent eventual bounded convergence.

Theorem 2. 

[33] Suppose that there exists a Lyapunov function $V(x)$ with the constants $\alpha$, $\beta$, $r_1$, $r_2$, ${\eta }_0$ satisfying $\alpha >0$, $\beta >0$, $0<{\eta }_0<\infty$, $0<r_1<1$ and $r_2>1$, If $\dot{V}\left(x\right)\le -\alpha V{\left(x\right)}^{r_1}-\beta V{\left(x\right)}^{r_2}+{\eta }_0$ holds, then at a fixed time, $V\left(x\right)$ is able to converge to a neighborhood of the equilibrium point, which can be denoted as:

$$\left\{\underset{t\to T}{\lim }x\left|V(x)\le \min \left\{{\left({\displaystyle \frac{{\eta }_0}{\alpha (1-\theta )}}\right)}^{r_1},{\left({\displaystyle \frac{{\eta }_0}{\beta (1-\theta )}}\right)}^{r_2}\right\}\right.\right\}$$

(50)

where $\theta$ is a constant and satisfies $0<\theta <1$. The time required for the system variables to converge to this neighborhood satisfies the following inequality:

$$T\le {\displaystyle \frac{1}{\alpha \theta }}{\displaystyle \frac{1}{1-r_1}}+{\displaystyle \frac{1}{\beta \theta }}{\displaystyle \frac{1}{r_2-1}}$$

(51)

Proof. 

By substituting Equation (29) into Equation (28) and based on the comparison principle in Lemma 1, the following is obtained:

$$\begin{array}{cc}\hfill {\dot{V}}_3& \le {\displaystyle \sum _{i=1}^3\left[-k_{i1}{\ln }^{{\gamma }_1}{(·)}_i-k_{i2}{\ln }^{{\gamma }_2}{(·)}_i\right]}-{\displaystyle \sum _{j=1}^2\left[\frac{{\sigma }_{wj}}{2{\eta }_{wj}}{\tilde{W}}_j^T{\tilde{W}}_j+\frac{{\sigma }_{\vartheta j}}{2{\eta }_{\vartheta j}}{\tilde{\vartheta }}_j^T{\tilde{\vartheta }}_j\right]}\hfill \\ & +{\displaystyle \sum _{j=1}^2\left[\frac{{\sigma }_{wj}}{2{\eta }_{wj}}{W}_j^{\ast T}{W}_j^{\ast }+\frac{{\sigma }_{\vartheta j}}{2{\eta }_{\vartheta j}}{\vartheta }_j^{\ast T}{\vartheta }_j^{\ast }\right]}+{\displaystyle \frac{1}{2}}{‖\epsilon ‖}^2\hfill \\ & \le -\rho V_3+\zeta \hfill \end{array}$$

(52)

where

$$\rho =\min \left\{k_{11},k_{12},k_{21},k_{22},k_{31},k_{32},{\sigma }_{w1},{\sigma }_{w2},{\sigma }_{\vartheta 1},{\sigma }_{\vartheta 2}\right\}$$

(53)

$$\zeta ={\displaystyle \sum _{j=1}^2\left[\frac{{\sigma }_{wj}}{2{\eta }_{wj}}{W}_j^{\ast T}{W}_j^{\ast }+\frac{{\sigma }_{\vartheta j}}{2{\eta }_{\vartheta j}}{\vartheta }_j^{\ast T}{\vartheta }_j^{\ast }\right]}+{\displaystyle \frac{1}{2}}{‖\epsilon ‖}^2$$

(54)

There are integrals on both sides of Equation (52):

$$0\le V_3\le \zeta /\rho +\left[V_3(0)-\zeta /\rho \right]e^{-\rho t}$$

(55)

When $t$ tends to infinity, there is $0\le V_3\le \zeta /\rho$. From the fact that $V_3$ is bounded it follows that the tracking error of the system is always bounded within the time-varying boundary $\left(-k_{bi}(t),k_{bi}(t)\right)$.

Since $V_3$ is bounded, the error signals $z_3$, ${\tilde{W}}_j$ and ${\tilde{\vartheta }}_j$ are all bounded, i.e., there exist positive constants ${\overline{W}}_j$ and ${\overline{\vartheta }}_j$ satisfying ${\tilde{W}}_j\le {\overline{W}}_j$ and ${\tilde{\vartheta }}_j\le {\overline{\vartheta }}_j$. This leads to the following inequality holding:

$$\begin{array}{l}-{\left({\displaystyle \frac{{\sigma }_{w_j}}{2{\eta }_{w_j}}}{\tilde{W}}_j^T{\tilde{W}}_j\right)}^{{\gamma }_1}-{\left({\displaystyle \frac{{\sigma }_{{\vartheta }_j}}{2{\eta }_{{\vartheta }_j}}}{\tilde{\vartheta }}_j^T{\tilde{\vartheta }}_j\right)}^{{\gamma }_1}+{\left({\displaystyle \frac{{\sigma }_{w_j}}{2{\eta }_{w_j}}}{\overline{W}}_j^T{\overline{W}}_j\right)}^{{\gamma }_1}+{\left({\displaystyle \frac{{\sigma }_{{\vartheta }_j}}{2{\eta }_{{\vartheta }_j}}}{\overline{\vartheta }}_j^T{\overline{\vartheta }}_j\right)}^{{\gamma }_1}\ge 0\\ -{\left({\displaystyle \frac{{\sigma }_{w_j}}{2{\eta }_{w_j}}}{\tilde{W}}_j^T{\tilde{W}}_j\right)}^{{\gamma }_2}-{\left({\displaystyle \frac{{\sigma }_{{\vartheta }_j}}{2{\eta }_{{\vartheta }_j}}}{\tilde{\vartheta }}_j^T{\tilde{\vartheta }}_j\right)}^{{\gamma }_2}+{\left({\displaystyle \frac{{\sigma }_{w_j}}{2{\eta }_{w_j}}}{\overline{W}}_j^T{\overline{W}}_j\right)}^{{\gamma }_2}+{\left({\displaystyle \frac{{\sigma }_{{\vartheta }_j}}{2{\eta }_{{\vartheta }_j}}}{\overline{\vartheta }}_j^T{\overline{\vartheta }}_j\right)}^{{\gamma }_2}\ge 0\end{array}$$

(56)

Substituting Equation (56) into Equation (52) yields:

$$\begin{array}{ll}\hfill {\dot{V}}_3\le & {\displaystyle \sum _{i=1}^3\left[-k_{i1}{\ln }^{{\gamma }_1}{(·)}_i-k_{i2}{\ln }^{{\gamma }_2}{(·)}_i\right]}+{\displaystyle \sum _{j=1}^2\left[\frac{{\sigma }_{wj}}{2{\eta }_{wj}}{W}_j^{\ast T}{W}_j^{\ast }+\frac{{\sigma }_{\vartheta j}}{2{\eta }_{\vartheta j}}{\vartheta }_j^{\ast T}{\vartheta }_j^{\ast }\right]}-\hfill \\ & {\displaystyle \sum _{j=1}^2\left[{\left(\frac{{\sigma }_{w_j}}{2{\eta }_{w_j}}{\tilde{W}}_j^T{\tilde{W}}_j\right)}^{{\gamma }_1}+{\left(\frac{{\sigma }_{{\vartheta }_j}}{2{\eta }_{{\vartheta }_j}}{\tilde{\vartheta }}_j^T{\tilde{\vartheta }}_j\right)}^{{\gamma }_1}\right]}-{\displaystyle \sum _{j=1}^2\left[{\left(\frac{{\sigma }_{w_j}}{2{\eta }_{w_j}}{\tilde{W}}_j^T{\tilde{W}}_j\right)}^{{\gamma }_2}+{\left(\frac{{\sigma }_{{\vartheta }_j}}{2{\eta }_{{\vartheta }_j}}{\tilde{\vartheta }}_j^T{\tilde{\vartheta }}_j\right)}^{{\gamma }_2}\right]}+\hfill \\ & {\displaystyle \sum _{j=1}^2\left[{\left(\frac{{\sigma }_{w_j}}{2{\eta }_{w_j}}{\overline{W}}_j^T{\overline{W}}_j\right)}^{{\gamma }_1}+{\left(\frac{{\sigma }_{{\vartheta }_j}}{2{\eta }_{{\vartheta }_j}}{\overline{\vartheta }}_j^T{\overline{\vartheta }}_j\right)}^{{\gamma }_1}\right]}+{\displaystyle \sum _{j=1}^2\left[{\left(\frac{{\sigma }_{w_j}}{2{\eta }_{w_j}}{\overline{W}}_j^T{\overline{W}}_j\right)}^{{\gamma }_2}+{\left(\frac{{\sigma }_{{\vartheta }_j}}{2{\eta }_{{\vartheta }_j}}{\overline{\vartheta }}_j^T{\overline{\vartheta }}_j\right)}^{{\gamma }_2}\right]}+\hfill \\ & {\displaystyle \frac{1}{2}}{‖\epsilon ‖}^2\hfill \end{array}$$

(57)

This can be obtained using Lemma 2:

$${\dot{V}}_3\le -\alpha V_3^{{\gamma }_1}-\beta V_3^{{\gamma }_2}+{\eta }_0$$

(58)

where $\alpha$, $\beta$ and ${\eta }_0$ can be expressed, respectively, as:

$$\alpha =\min \left\{k_{i1},{\sigma }_{w_j}^{{\gamma }_1},{\sigma }_{{\vartheta }_j}^{{\gamma }_1}\right\}$$

(59)

$$\beta =\min \left\{{(2n)}^{1-{\gamma }_2}{k}_{i2},{(2n)}^{1-{\gamma }_2}{\sigma }_{w_j}^{{\gamma }_2},{(2n)}^{1-{\gamma }_2}{\sigma }_{{\vartheta }_j}^{{\gamma }_2}\right\}$$

(60)

$$\begin{array}{ll}\hfill {\eta }_0=& {\displaystyle \sum _{j=1}^2\left[{\left(\frac{{\sigma }_{w_j}}{2{\eta }_{w_j}}{\overline{W}}_j^T{\overline{W}}_j\right)}^{{\gamma }_1}+{\left(\frac{{\sigma }_{{\vartheta }_j}}{2{\eta }_{{\vartheta }_j}}{\overline{\vartheta }}_j^T{\overline{\vartheta }}_j\right)}^{{\gamma }_1}+{\left(\frac{{\sigma }_{w_j}}{2{\eta }_{w_j}}{\overline{W}}_j^T{\overline{W}}_j\right)}^{{\gamma }_2}\right]}+\hfill \\ & {\displaystyle \sum _{j=1}^2\left[{\left(\frac{{\sigma }_{{\vartheta }_j}}{2{\eta }_{{\vartheta }_j}}{\overline{\vartheta }}_j^T{\overline{\vartheta }}_j\right)}^{{\gamma }_2}+\frac{{\sigma }_{wj}}{2{\eta }_{wj}}{W}_j^{\ast T}{W}_j^{\ast }+\frac{{\sigma }_{\vartheta j}}{2{\eta }_{\vartheta j}}{\vartheta }_j^{\ast T}{\vartheta }_j^{\ast }\right]}+{\displaystyle \frac{1}{2}}{‖\epsilon ‖}^2\hfill \end{array}$$

(61)

Using Lemma 2, it can be obtained that the controller proposed in this section can converge in a neighborhood near the zero point in a fixed time, where the neighborhood expression is $\left\{\underset{t\to T}{\lim }x\left|V(x)\le \min \left\{{\left({\eta }_0/k_{i1}(1-\theta )\right)}^{{\gamma }_1},{\left({\eta }_0/k_{i2}(1-\theta )\right)}^{{\gamma }_2}\right\}\right.\right\}$ and the convergence time expression is $T=1/k_{i1}\theta (1-{\gamma }_1)+1/k_{i2}\theta ({\gamma }_2-1)$.□

4.2. Suboptimality Analysis of Disturbance Compensator

Next, the optimality and suboptimality of the compensation signal are analyzed:

Define the value function under any strategy as:

$$V_{i+1}(x(k),F_D(k))=r(k)+\lambda V_i(x(k+1),F_D(k+1))$$

(62)

According to Equation (40), $Q_{i+1}(x(k),F_D(k))$ is the result of minimizing the value function, while $V_{i+1}(x(k),F_D(k))$ is the result of the value function of an arbitrary action, so there is:

$$Q_{i+1}(x(k),F_D(k))\le V_{i+1}(x(k),F_D(k))$$

(63)

When $Q_0=V_0=0$, this holds for both $i=0,1,2\dots$.

According to Equation (62) there is:

$$\begin{array}{cc}\hfill V_{i+1}(x(k),F_D(k))-V_i(x(k),F_D(k))& =\lambda V_i(x(k+1),F_D(k+1))-\lambda V_{i-1}(x(k+1),F_D(k+1))\hfill \\ & ={\lambda }^2{V}_{i-1}(x(k+2),F_D(k+2))-V_{i-2}(x(k+2),F_D(k+2))\hfill \\ & ={\lambda }^i{V}_1(x(k+i),F_D(k+i))-V_0(x(k+i),F_D(k+i))\hfill \\ & ={\lambda }^i{V}_1(x(k+i),F_D(k+i))\hfill \end{array}$$

(64)

Therefore, the value function is expressed iteratively as:

$$\begin{array}{cc}\hfill V_{i+1}(x(k),F_D(k))& ={\lambda }^i{V}_1(x(k+i),F_D(k+i))+V_i(x(k),F_D(k))\hfill \\ & ={\lambda }^i{V}_1(x(k+i),F_D(k+i))+{\lambda }^{i-1}{V}_1(x(k+i-1),F_D(k+i-1))\hfill \\ & +V_{i-1}(x(k),F_D(k))\hfill \\ & ={\lambda }^i{V}_1(x(k+i),F_D(k+i))+{\lambda }^{i-1}{V}_1(x(k+i-1),F_D(k+i-1))\hfill \\ & +\cdots +\lambda V_1(x(k+1),F_D(k+1))+V_1(x(k),F_D(k))\hfill \\ & ={\displaystyle \sum _{j=0}^i{\lambda }^j{V}_1(x(k+i),F_D(k+i))}\hfill \end{array}$$

(65)

According to Equation (62) there is:

$$V_{i+1}(x(k),F_D(k))\le {\displaystyle \sum _{j=0}^{\infty }{\lambda }^{j}r(x(k+i),F_D(k+i))}$$

(66)

Due to the positive characterization of the stage cost function $\delta (x(k+i),F_D(k+i))$, it is assumed that $0\le \delta (x(k+i),F_D(k+i))\le C$, there exists an upper bound $ℂ$.

$${\displaystyle \sum _{j=0}^{\infty }{\lambda }^j\delta (x(k+i),F_D(k+i))}\le {\displaystyle \sum _{j=0}^{\infty }{\lambda }^{j}C}\le {\displaystyle \sum _{j=0}^{\infty }{\lambda }^jℂ}$$

(67)

There is through Equation (63):

$$Q_{i+1}(x(k),F_D(k))\le V_{i+1}(x(k),F_D(k))\le ℂ$$

(68)

Next, the optimality of the value function and control policy will be proven.

Since $Q_0=V_0=0$, it follows that when $i=0$, there is $Q_0(x(k),F_D(k))\le Q_1(x(k),F_D(k))=\delta (k)$, there is:

$$Q_1(x(k),F_D(k))-V_0=\delta (k)\ge 0$$

(69)

Thus, by mathematical induction, there is:

$$V_{i-1}(x(k),F_D(k))\le Q_i(x(k),F_D(k))$$

(70)

Further, there is:

$$\begin{array}{c} Q_{i+1}(x(k),F_D(k))-V_i(x(k),F_D(k))\hfill \\ =\lambda [Q_i(x(k),F_D(k))-V_{i-1}(x(k),F_D(k))]\ge 0\hfill \end{array}$$

(71)

According to inequality (63), there is:

$$Q_i(x(k),F_D(k))\le V_i(x(k),F_D(k))\le Q_{i+1}(x(k),F_D(k))$$

(72)

For any control strategy $F_D(k)$, there is the following inequality:

$$Q_i(x(k),F_D(k))\le \delta (k)+\lambda Q_{i-1}(x(k+1),F_D(k+1))$$

(73)

According to Equation (72),

$$Q_i(x(k),F_D(k))\le \delta (k)+\lambda Q_{\infty }(x(k+1),F_D(k+1))$$

(74)

It applies to $i\to \infty$:

$$Q_{\infty }(x(k),F_D(k))\le \delta (k)+\lambda Q_{\infty }(x(k+1),F_D(k+1))$$

(75)

When there is an arbitrary control strategy $\pi (k+1)$, there is:

$$Q_{\infty }(x(k),F_D(k))\le \delta (k)+\lambda \underset{\pi (k+1)}{\min }{Q}_{\infty }(x(k+1),F_D(k+1))$$

(76)

Also, according to Equations (40) and (72),

$$Q_{\infty }(x(k),F_D(k))\ge \delta (k)+\lambda Q_{\infty }(x(k+1),F_D(k+1))$$

(77)

Obviously, there is:

$$Q_{\infty }(x(k),F_D(k))=\delta (k)+\lambda \underset{\pi (k+1)}{\min }{Q}_{\infty }(x(k+1),F_D(k+1))=\delta (k)+\lambda Q_{\infty }(x(k+1),F_D(k+1))$$

(78)

The optimal control strategy can be solved by the following equation:

$$F_{D\infty }(k)=\arg \min Q_{\infty }(x(k),F_D(k))$$

(79)

Therefore, according to Equations (33) and (34), there is:

$$Q_{\infty }(x(k),F_D(k))=J^{\ast }(x(k),F_D(k)) F_{D\infty }(k)=F_D^{\ast }(k)$$

(80)

However, due to the inherent error in neural networks, the optimal convergence properties of the value function and control policy only hold under ideal conditions. Therefore, considering the approximation errors in Equations (41) and (46),

$${\widehat{F}}_{Di}(k)=\arg \min {\widehat{Q}}_i(x(k),F_D(k))+{\epsilon }_i(x(k))$$

(81)

$${\widehat{Q}}_{i+1}(x(k),{\widehat{F}}_D(k))=\delta (k)+\lambda {\widehat{Q}}_i(x(k+1),{\widehat{F}}_D(k+1))+{\delta }_i(x(k),{\widehat{F}}_D(k))$$

(82)

where ${\epsilon }_i(x(k))$ and ${\delta }_i(x(k),{\widehat{F}}_D(k))$ are the estimation errors and ${\epsilon }_i(0)={\delta }_i(0,0)=0$.

Next, a new convergence criterion will be established at each iteration, taking into account the error, i.e., the value iterates converge to a neighborhood of the optimal value function, and the iterative value function is:

$$Q_{t,i}(x(k),F_D(k))=\delta (k)+\lambda \min {\widehat{Q}}_{i-1}(x(k+1),F_D(k+1))$$

(83)

where ${\widehat{Q}}_0(x(k),F_D(k))={\widehat{Q}}_{t,0}(x(k),F_D(k))=0$ and there exists a constant $1\le \varpi <\infty$ such that the following inequality holds:

$${\widehat{Q}}_i(x(k),F_D(k))\le \varpi Q_{t,i}(x(k),F_D(k))$$

(84)

Defining the constant $0<\nu <\infty$, the following inequality holds:

$$J^{\ast }(x(k+1),F_D(k+1))\le \varpi \delta (k)$$

(85)

According to Equation (84), when $i=0$,

$${\widehat{Q}}_0(x(k),F_D(k))\le \varpi J^{\ast }(x(k),F_D(k))$$

(86)

When $i=1$,

$$\begin{array}{cc}\hfill Q_{t,1}(x(k),F_D(k))& =\delta (k)+\lambda \underset{\pi (k+1)}{\min }{\widehat{Q}}_0(x(k+1),F_D(k+1))\hfill \\ & \le \delta (k)+\lambda \varpi J^{\ast }(x(k+1),F_D(k+1))\hfill \\ & \le (1+\nu (\varpi -1)/(\varpi +1))\delta (k)+\hfill \\ & \lambda (\varpi -(\varpi -1)/(\varpi +1))J^{\ast }(x(k+1),F_D(k+1))\hfill \\ & =(1+\nu (\varpi -1)/(\varpi +1))J^{\ast }(x(k+1),F_D(k+1))\hfill \end{array}$$

(87)

According to Equations (33) and (85),

$$\lambda J^{\ast }(x(k+1),F_D(k+1))\le J^{\ast }(x(k+1),F_D(k+1))\le \nu \delta (k)$$

(88)

Therefore, the following expression holds:

$${\widehat{Q}}_1(x(k),F_D(k))\le \varpi (1+\nu (\varpi -1)/(\varpi +1))J^{\ast }(x(k),F_D(k))$$

(89)

Define the following equation:

$$\beta ={\displaystyle \sum _{j=1}^{i-1}({\nu }^{j-1}{\varpi }^{j-1}(\varpi -1))/{(\nu +1)}^j} \omega =({\nu }^{j-1}{\varpi }^{j-1}(\varpi -1))/{(\nu +1)}^i$$

(90)

There is:

$$\begin{array}{cc}\hfill Q_{t,i}(x(k),\pi (k))& =\delta (k)+\lambda \underset{\pi (k+1)}{\min }{\widehat{Q}}_{i-1}(x(k+1),F_D(k+1))\hfill \\ & \le \delta (k)+\lambda \varpi (1+\nu \beta )J^{\ast }(x(k+1),F_D(k+1))\hfill \\ & \le (1+\varpi \beta +\varpi \omega )\delta (k)+\lambda (\varpi (1+\nu \beta )-\beta -\omega )J^{\ast }(x(k+1),F_D(k+1))\hfill \\ & =(1+{\displaystyle \sum _{j=1}^i{\nu }^j{\varpi }^{j-1}(\varpi -1)/{(\nu +1)}^j{J}^{\ast }(x(k+1),F_D(k+1))}\hfill \end{array}$$

(91)

According to Equation (84), by mathematical induction,

$${\widehat{Q}}_i(x(k),F_D(k))\le (1+{\displaystyle \sum _{j=0}^i{\nu }^j{\varpi }^{j-1}(\varpi -1)/{(\nu +1)}^j)}{J}^{\ast }(x(k+1),F_D(k+1))$$

(92)

According to inequality (91), it is known that ${\nu }^j{\varpi }^{j-1}(\varpi -1)/{(\nu +1)}^j$ is a geometric progression, so inequality (91) can be expressed as:

$$\begin{array}{cc}\hfill Q_{t,i}(x(k),F_D(k))& =(1+(\nu (\varpi -1)/(\nu +1))(1-(\nu \varpi /{(\nu +1))}^i)/(1\hfill \\ & -\nu \varpi /(\nu +1)))J^{\ast }(x(k+1),F_D(k+1))\hfill \end{array}$$

(93)

When $i\to \infty$, inequality (93) becomes:

$$\underset{i\to \infty }{\lim }{Q}_{t,i}(x(k),F_D(k))\le (1+\nu (\varpi -1)/(1-\nu (\varpi -1)))J^{\ast }(x(k),F_D(k))$$

(94)

Combined with inequality (84), there is:

$${\widehat{Q}}_{\infty }(x(k),F_D(k))\le \varpi \underset{i\to \infty }{\lim }{Q}_{t,i}(x(k),F_D(k))$$

(95)

If inequality $1\le \varpi \le (\nu +1)/\nu$ holds, according to Equations (94) and (95), there is:

$$\underset{i\to \infty }{\lim }{\widehat{Q}}_i(x(k),F_D(k))\le \varpi (1+\nu (\varpi -1)/(1-\nu (\varpi -1)))J^{\ast }(x(k),F_D(k))$$

(96)

When network errors exist, ${\widehat{Q}}_i(x(k),F_D(k))$ converges within the scaled range of the optimal value function $J^{\ast }(x(k),F_D(k))$, i.e., converging to a suboptimal solution. Therefore, the suboptimality of the disturbance compensator is proven.

The overall block diagram of the proposed controller is shown in Figure 4.

5. Experimental Verification

The hydraulic manipulator arm experiment platform is shown in Figure 5, the boom and arm are driven by two hydraulic cylinders, each cylinder is controlled by two Rexroth 4WREE-10 proportional servo valves (Bosch Rexroth, Lohr am Main, Germany), the displacement measurement of the hydraulic cylinders is performed by Shanghai Loxin NS-WY06B tie-wire displacement sensors (Loxin, Shanghai, China), with the measurement noise of about ±0.5 mm, and the pressure measurement is performed by Schneider-Ford RPT8304-C-02 pressure sensors (Schneider, Shanghai, China), with the measurement noise of about ±0.07 MPa, and the pump used for the experiment is Rexroth SYDFEE-20/071R electric proportional pump (Bosch Rexroth, Lohr am Main, Germany), with the displacement of 71 mL/rad. The measurement and control system adopts the closed-loop architecture of upper computer-controller-CAN communication-lower computer-sensor, and the real-time data acquisition and control of the system is based on dSPACE (DS4302 and DS2004 Germany) and MATLAB (2015b) with a sampling period of 10 ms. The sampling period is 10 ms.

As the controller-related parameters need the real-time angle of each joint, the control algorithm joins the hydraulic cylinder and the joint angle conversion equation. The length of the boom cylinder with the angle change equation is:

$$l_1=\sqrt{L_{AE}^2+L_{AF}^2-2L_{AE}{L}_{AF}\cos (\angle EAN+\angle BAF+{\theta }_1)}$$

(97)

The length of the arm cylinder with the angle change equation is:

$$\left\{\begin{array}{l}\angle GBH=\arccos ({\displaystyle \frac{L_{BG}^2+L_{BH}^2-{\lambda }_2^2}{2L_{BG}{L}_{BH}}})\\ {\theta }_2=\pi -\angle ABG-\angle GBH-\angle HBC\\ l_2=\sqrt{L_{BG}^2+L_{BH}^2-2L_{BG}{L}_{BH}\cos (\pi -\angle ABG-\angle HBC-{\theta }_2)}\end{array}\right.$$

(98)

Since the neural network does not converge at the initial stage, it is necessary to train the disturbance compensator offline and deploy it online to the controller. This enables fine-tuning of network weights to estimate and compensate for external disturbances. The relevant structural parameters of the manipulator are shown in

Table 1. Some parameters of the hydraulic manipulator.

Parameter	Value	Parameter	Value	Parameter	Value	Parameter	Value	Parameter	Value
$A_{11}$	3.12 × 10−3 m2	$A_{12}$	2.4 × 10−3 m2	$A_{21}$	2.38 × 10−3 m2	$A_{22}$	1.7 × 10−3 m2	$g$	9.8 m/s2
$P_r$	0 Pa	$I_1$	28 kg∙m2	$I_2$	10.2 kg∙m2	$I_3$	1.95 kg∙m2	$m_1$	86.6 kg
$m_2$	64 kg	$m_3$	43 kg	$L_1$	1.806 m	$L_2$	1.151 m	$L_3$	0.535 m
${\beta }_e$	1 × 109	$k_{v1}$	3.3 × 10−8	$k_{v2}$	3.3 × 10−8

. By constructing the manipulator model using Equation (6) and introducing irregular external disturbances as depicted in Figure 6, the network weights are ensured to learn real disturbances of the same magnitude. Given the excitation trajectory shown in Figure 7, the disturbance compensator is trained using the performance-specific controller designed in this paper for tracking. When the tracking error $Z_1\le E_{\mathrm{A}}$, the training is stopped and the trained actor and critic neural network weights are deployed in dSPACE. Taking the boom controller training as an example, the training process is shown in Figure 8, consisting of 30 iterations. Figure 8a represents the first iteration, while Figure 8b shows the last iteration.

In this subsection, the superiority of the proposed control algorithm is verified through comparison and ablation experiments, firstly, the PI controller is compared with the prescribed performance controller based on neural network compensation to verify the effectiveness of the prescribed performance function and the neural network in the controller, and secondly, the prescribed performance controller is compared with the addition of the trained disturbance compensator for ablation experiments to verify the effectiveness of the disturbance compensator, and the relevant parameters are shown below:

C1: This is a typical Proportional Integral (PI) controller with control gain $k_P=\mathrm{diag}\left\{55, 85\right\}$, $k_I=\mathrm{diag}\left\{35, 75\right\}$.

C2: This is a prescribed performance controller based on neural network compensation. The constraint boundaries of the moving arm and the boom are the same, and the constraint boundaries are set as: $k_{a1}=0.8e^{-0.7t}+0.8$, $k_{a2}=1.3e^{-0.8t}+1.6$, $k_{a3}=1.8e^{-0.4t}+2.1$. Select controller parameters through a combination of simulation and experimentation: $k_{11}^{\ast }=k_{21}^{\ast }=k_{31}^{\ast }=\mathrm{diag}\left\{16, 16\right\}$, $k_{12}^{\ast }=k_{22}^{\ast }=k_{32}^{\ast }=\left\{24, 24\right\}$, ${\gamma }_1=1.2$, ${\gamma }_2=3$. the number of nodes in the hidden layer of the RBF neural network is set to 32, the initial value of the neural network weights $W_{1i0}=W_{2i0}=0.2,$ $i=1,2$$,\dots$$,32$, the initial value of the approximation error ${\vartheta }_{10}={\vartheta }_{20}=0$, ${\eta }_{w1}={\eta }_{w2}=1$, ${\eta }_{\vartheta 1}={\eta }_{\vartheta 2}=210$, ${\sigma }_{\vartheta 1}={\sigma }_{\vartheta 2}=0.16$, center value interval is [−1.5, 1.5]. The relevant structural parameters of the manipulator are shown in Table 1.

C3: This is the prescribed performance controller with the addition of the disturbance compensator, and its training parameters are: $h_c=h_a=0.01$, $\lambda =\mathrm{diag}[0.9,0.9,0.9]$. Its base control parameters are the same as those in C2 to ensure fairness in comparison.

The experiment is used to simulate the normal operation of the hydraulic manipulator arm by giving the time-varying sinusoidal signals of the boom and arm, while 20 kg sandbags are added to the tip of the bucket teeth to simulate the load during normal operation. The signals are as follows:

$$q_d={\left[10\sin \left(0.2t+8\right)+25,25\sin \left(0.2t+10.5\right)-59.8\right]}^T$$

(99)

Figure 9 shows the coverage effect of desired trajectories and tracking trajectories under three controllers for the boom and arm. Figure 10 depicts the angular tracking error of the boom and arm. Figure 11 illustrates the compensation signal for the boom and arm generated by the disturbance compensator. Figure 12 presents the unmodeled error estimation compensation value for the boom and arm derived from the neural network. Based on experimental results, it is evident that for continuously varying sinusoidal velocity signals, the tracking performance of C3 and C2 significantly outperforms that of C1. This improvement stems from the fact that C2 incorporates a predefined performance function to constrain system errors while ensuring stability, thereby mitigating the manipulator’s inherent joint strong coupling characteristics and hydraulic system parameter uncertainties to a certain extent. Concurrently, the RBF neural network estimates and compensates for internal system uncertainties, thereby mitigating the manipulator’s inherent joint strong coupling characteristics and hydraulic, which the PI controller cannot resolve. Consequently, C2 achieves significantly higher control accuracy than C1. Furthermore, as shown in the error results of Figure 10, C3 demonstrates markedly superior control accuracy compared to C2. This is because C3 incorporates an additional disturbance compensator on top of C2, enabling compensation for disturbances encountered during the hydraulic manipulator’s operation. In this experiment, a 20 kg sandbag was introduced to simulate external disturbances. However, due to the inherent robustness of the specified performance controller, this robustness effectively suppresses disturbances during the motion of the heavy-duty hydraulic manipulator. Due to its substantial mass and the time-varying characteristics of the inertia matrix, Coriolis torque matrix, and gravity matrix, the improvement in control accuracy achieved by C3 over C2 is less pronounced than the improvement C2 achieves over C1.

To further analyze the performance of each controller, this paper employs widely used evaluation metrics: the maximum absolute value, mean, and standard deviation of tracking error as assessment criteria [34], expressed as:

$$M_e=\underset{i=1,\dots ,N}{\max }\left\{\left|z_1\left(i\right)\right|\right\} \mu ={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N\left|z_1\left(i\right)\right|} \sigma =\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{\left[\left|z_1(i)-\mu \right|\right]}^2}}$$

(100)

where $M_e$ is the maximum absolute value error; $\mu$ is the mean error; and $\sigma$ is the standard deviation.

The results are shown in

Table 2. Control performance index of each controller.

Joint	Controller	${\mathit{M}}_{\mathit{e}}$	$\mathit{\mu }$	$\mathit{\sigma }$
Boom	C1	1.6104	0.6107	0.5701
	C2	0.7913	0.2364	0.2296
	C3	0.7026	0.2136	0.2108
Arm	C1	1.6073	0.6046	0.5658
	C2	0.7924	0.2309	0.2217
	C3	0.7043	0.2136	0.2102

. Based on the quantified data, it is evident that due to the performance controller imposing boundary constraints while the neural network compensates for unmodeled internal system errors, coupled with the control architecture’s inherent robustness, suppressing certain disturbances, C2 achieved a 61.3% and 61.8% reduction in average tracking error compared to C1 for the boom and arm, respectively. The standard deviation decreased by 59.7% and 60.8%, respectively. Furthermore, since C3 incorporates a disturbance compensator on top of C2, its control accuracy improvement surpasses that of C2. Specifically, C3 reduced the mean tracking error for the boom and arm by 65% and 64.7% compared to C1. The standard deviation decreased by 63% and 62.8%, respectively. To more clearly demonstrate the effectiveness of the disturbance compensator, comparing the control accuracy between C2 and C3 shows that C3 reduced the average tracking error compared to C2 by 9.6% and 7.5% for the boom and arm, respectively. The standard deviation decreased by 8.2% and 5.2%, respectively, fully validating the effectiveness of the disturbance compensator.

Through the aforementioned experiments, the high-precision motion control algorithm for hydraulic manipulators proposed in this paper demonstrates that by defining a performance function, errors can be stabilized within a preset range. Concurrently, the RBF neural network compensates for internal system uncertainties, achieving a significant improvement in precision compared to PID control algorithms. Furthermore, a disturbance compensator based on the Actor–Critic reinforcement learning architecture was introduced. Ablation experiments confirmed the effectiveness of this disturbance compensator.

6. Conclusions

This paper addresses the challenge of achieving high-precision motion control for hydraulic manipulators with strong electromechanical-hydraulic coupling. A high-precision motion control algorithm with a dual compensation mechanism and specified performance is designed, significantly enhancing the system’s control accuracy. The following key conclusions are drawn:

(1)

Within a performance-based control architecture based on the inverse method, a neural network is introduced to estimate and compensate for internal system uncertainties: unmodeled mechanical and hydraulic errors. The network update rate is obtained through a global stability proof, significantly enhancing system control accuracy.

(2)

An Actor–Critic reinforcement learning architecture is employed to design a disturbance compensator for external system uncertainties: external disturbances. The network’s online update rate was obtained using local stability, further enhancing the system’s motion control precision.

(3)

Through a hydraulic manipulator experimental platform, the proposed control algorithm demonstrated a 60–65% improvement in control accuracy compared to the PID algorithm. Ablation experiments confirmed that the disturbance compensator designed using the reinforcement learning Actor–Critic architecture further enhances the hydraulic manipulator’s control accuracy by 7–10%, validating its effectiveness.

Furthermore, due to the network’s local updates and proven optimality, the disturbance compensator is not only applicable to the proposed algorithm but can also be extended to adaptive robust control algorithms to further enhance system motion control accuracy.