An Improved H_∞ Tracking Controller for Uncertain Systems Based on DDPG with Improved Exploration Strategy

Abstract

This paper proposes an integrated robust–learning control framework for uncertain systems with external disturbances. An $H_{\infty }$ state-feedback controller is first synthesized to ensure closed-loop stability and a prescribed disturbance attenuation level under norm-bounded uncertainties. Building on this robust baseline, the Deep Deterministic Policy Gradient (DDPG) is used to refine the $H_{\infty }$ feedback gains online to improve tracking and transient performance while reducing the conservatism of fixed robust gains. To improve the exploration process in reinforcement learning, a tracking-error-guided mechanism is developed. It adaptively adjusts the exploration intensity by means of tracking error energy, promoting reasonable exploration under steady-state conditions while suppressing excessive exploration during large transients, thereby improving both learning efficiency and system transient performance. The simulation results verify the effectiveness of the proposed method.

1. Introduction

In modern control systems, uncertainties and disturbances are almost unavoidable, stemming from modeling simplifications, parameter variations, identification errors, unmodeled dynamics, and exogenous perturbations. These factors can markedly degrade closed-loop performance and may even jeopardize stability, thereby imposing stringent requirements on controller synthesis. To cope with such challenges, various methodologies have been proposed, including adaptive control for compensating parametric variations [1] and model predictive control for handling multivariable constraints and optimizing finite-horizon performance [2], as well as disturbance-observer-based, sliding-mode [3], and fractional-order [4] control techniques for disturbance rejection and performance enhancement. Among these approaches, robust control provides a systematic framework for guaranteeing stability and performance in the presence of bounded uncertainties, especially for uncertain discrete-time systems. Nevertheless, robust control methods are often developed under worst-case uncertainty descriptions and may therefore suffer from a certain degree of conservatism, which can limit transient performance in practical applications. In addition, their synthesis may become computationally demanding, particularly when the controller design is formulated through matrix inequalities or other optimization-based conditions [5]. Compared with advanced approaches such as sliding mode control and fractional PID control, robust control presents a different trade-off between performance and robustness: sliding mode control is well known for strong disturbance rejection and fast response, but may suffer from chattering and implementation difficulties, while fractional PID control offers additional tuning flexibility and can achieve favorable dynamic performance, although its robustness guarantees are often less explicit for uncertain systems. Therefore, although robust control remains attractive for systems requiring rigorous stability and disturbance attenuation analysis, its conservatism and performance limitations still motivate further improvement.

Motivated by these observations, extensive research efforts have been devoted to robust control design for uncertain systems [6,7,8,9,10]. Existing studies have reported a variety of effective approaches from different perspectives. In dealing with uncertain parameters and model complexity of aerial manipulators, an adaptive robust control scheme [11] was introduced. In the context of autonomous ground vehicle control, Ref. [12] designed an $H_{\infty }$ output feedback controller to handle parameter uncertainties and ensure stable lane-keeping performance. For dynamic systems with time delays that are influenced by parameter uncertainties and external perturbations, researchers [13] proposed a robust optimal fault-tolerant controller and validated its effectiveness through numerical simulations. In the study of time-varying reference and disturbance signals, Ref. [14] formulated a robust integral control strategy to achieve control objectives in delay-uncertain systems. For wastewater treatment processes characterized by strong nonlinearity and external disturbances, Ref. [15] developed an adaptive robust fuzzy sliding mode control method, which effectively improved dynamic performance and disturbance rejection capability. The work in [16] combines $H_{\infty }$ control and fractional-order sliding mode control to achieve the tracking control of robots. Although robust control explicitly addresses bounded uncertainties and disturbances, its design is typically conservative and strongly dependent on uncertainty descriptions, which may lead to degraded performance in practical applications.

Over the past few years, reinforcement learning has been widely investigated for controlling systems with uncertain dynamics and disturbances [17,18,19,20,21]. A robust control scheme leveraging reinforcement learning is presented in [22] to address uncertain nonlinear systems subject to input constraints. In [23], an optimal finite-time adaptive performance-constrained controller for nonlinear systems via reinforcement learning is constructed. The authors of [24] employ neural networks to approximate unknown nonlinear dynamics while simultaneously establishing actor and critic neural networks to execute reinforcement learning algorithms, ensuring that the output signal follows the reference trajectory with prescribed accuracy. The authors of [25] employ reinforcement learning methods, utilizing neural networks to approximate both the cost function and the controller, thereby achieving optimal robust tracking. However, reinforcement-learning-based control methods generally lack explicit stability and robustness guarantees, which limits their reliability in safety-critical control systems.

To address the above limitations, this paper considers an $H_{\infty }$-based robust–learning scheme for uncertain tracking control with external disturbances. It should be noted that the combination of robust control and reinforcement learning has been investigated in previous studies. Therefore, the contribution of this work does not lie in the generic integration of the two paradigms, but in the specific way they are coordinated for the considered tracking problem. In the proposed method, an LMI-based $H_{\infty }$ state-feedback controller is first designed to guarantee closed-loop stability and a prescribed disturbance attenuation level under norm-bounded uncertainties. On this basis, DDPG is employed to optimize a structured gain increment on top of the robust baseline so as to improve tracking accuracy and transient response while alleviating the conservatism of fixed robust control. Furthermore, a tracking-error-guided exploration strategy is introduced to adaptively regulate the exploration intensity according to the tracking performance evolution with the aim of improving the stability of the training process and reducing oscillatory policy updates.

The main contributions of this work are summarized as follows:

(1) An $H_{\infty }$-based robust–learning tracking design is developed for uncertain systems with disturbances, in which the robust controller provides a stabilizing baseline with explicit $H_{\infty }$ performance guarantees, while DDPG is used for structured gain refinement rather than unconstrained policy generation.

(2) A tracking-error-guided exploration mechanism is proposed to enhance the training stability of DDPG by adjusting the exploration intensity according to the tracking error dynamics, thereby promoting smoother convergence and more reliable policy learning.

The remainder of this paper is organized as follows: Section 2 presents the problem formulation and derives the augmented tracking error system. Section 3 develops the proposed control method, including the LMI-based $H_{\infty }$ controller design, the DDPG-based gain refinement scheme, and the tracking-error-guided exploration strategy. Section 4 provides simulation results and comparative analysis. Finally, Section 5 concludes this work.

2. Preliminaries

Consider the following discrete-time uncertain system:

$$\left\{\begin{array}{c}x(k+1)=(A+\Delta A)x\left(k\right)+B_{1}u\left(k\right)+B_{2}w\left(k\right)\hfill \\ y\left(k\right)=Cx\left(k\right)+Dw\left(k\right)\hfill \end{array}\right.$$

(1)

where $x\in {\mathbb{R}}^n$ represents the system state, $u\in {\mathbb{R}}^m$ represents the control input, $w\in {\mathbb{R}}^p$ represents the disturbance acting on the system, and $y\in {\mathbb{R}}^q$ represents the system output; $A,B_1,B_2,C,D$ represent matrices of compatible dimensions, and $\Delta A$ represents the uncertainty of the system, which satisfies the following constraint:

$$\Delta A=H_1{F}_1{E}_1,\parallel F_1\parallel \le 1$$

(2)

where $H_1,E_1$ are known matrices with compatible dimensions.

This paper aims to address the tracking control problem, and the required reference trajectory is generated by the following system:

$$\left\{\begin{array}{cc}\hfill x_d(k+1)& =A_d{x}_d\left(k\right),\hfill \\ \hfill y_d\left(k\right)& =C_d{x}_d\left(k\right).\hfill \end{array}\right.$$

(3)

where $x_d\in {\mathbb{R}}^d$ represents the state of the reference system (3), and $y_d\in {\mathbb{R}}^q$ denotes the desired trajectory of y, where $A_d$ and $C_d$ are matrices of compatible dimensions.

Define $\chi \left(k\right)={\left[x{\left(k\right)}^{\mathrm{T}}{x}_d{\left(k\right)}^{\mathrm{T}}\right]}^{\mathrm{T}}$ and the tracking error $Z\left(k\right)=y\left(k\right)-y_d\left(k\right)$, where the superscript T denotes the matrix transpose. Using (1) and (3), the following tracking error system is formulated:

$$\left\{\begin{array}{c}\chi (k+1)=(A^{\prime }+\Delta A^{\prime })\chi \left(k\right)+B_1^{\prime }u\left(k\right)+B_2^{\prime }w\left(k\right)\hfill \\ Z\left(k\right)=C^{\prime }\chi \left(k\right)+Dw\left(k\right)\hfill \end{array}\right.$$

(4)

where

$$A^{\prime }=\left[\begin{array}{c}\begin{array}{cc}A& 0\\ 0& A_d\end{array}\end{array}\right],\Delta A^{\prime }=\left[\begin{array}{c}\begin{array}{cc}\Delta A& 0\\ 0& 0\end{array}\end{array}\right]=\overline{H_1}{F}_1\overline{E_1}$$

$${\overline{H}}_1={\left[H_1^{\mathrm{T}},0\right]}^{\mathrm{T}},{\overline{E}}_1=\left[\begin{array}{cc}{E}_1& 0\end{array}\right]$$

$$B_1^{\prime }={\left[B_1^{\mathrm{T}},0\right]}^{\mathrm{T}},B_2^{\prime }={\left[B_2^{\mathrm{T}},0\right]}^{\mathrm{T}},C^{\prime }=\left[C,-C_d\right].$$

For the uncertain system (1), the controller is designed in the form specified by:

$$u\left(k\right)=\overline{K}\chi \left(k\right)$$

(5)

Apply this controller to the tracking error system (6):

$$\left\{\begin{array}{c}\chi (k+1)={\overline{A}}_c\chi \left(k\right)+B_2^{\prime }w\left(k\right)\hfill \\ Z\left(k\right)=C^{\prime }\chi \left(k\right)+Dw\left(k\right)\hfill \end{array}\right.$$

(6)

where ${\overline{A}}_c=A^{\prime }+\Delta A^{\prime }+B_1^{\prime }\overline{K}$.

The controller given in (5) is developed in this paper to achieve stability of the closed-loop tracking error system and to satisfy a prescribed $H_{\infty }$ performance, i.e., the following two conditions are satisfied:

• Asymptotic stability [26]: ${lim}_{k\to \infty }\chi \left(k\right)=0$ holds for $\forall \chi \left(0\right)\in {\mathbb{R}}^{n+d}$ and $w\left(k\right)\equiv 0$.
• $H_{\infty }$ performance [27]: ${\sum }_{k=0}^{\infty }{Z}^{\mathrm{T}}\left(k\right)Z\left(k\right)<{\gamma }^2{\sum }_{k=0}^{\infty }{w}^{\mathrm{T}}\left(k\right)w\left(k\right)$ holds for any disturbance input $w\left(k\right)\in L_2[0,+\infty )$ under zero initial condition.

For the subsequent analysis, the following is introduced.

Petersen’s Lemma [28]: for any real matrices $M=M^{\mathrm{T}}$, $\tilde{H}$, and $\tilde{E}$ of compatible dimensions, and any matrix $\overline{F}$ satisfying ${\overline{F}}^{\mathrm{T}}\overline{F}\le I$, the  inequality

$$M+\tilde{H}\overline{F}\tilde{E}+{\tilde{E}}^{\mathrm{T}}{\overline{F}}^{\mathrm{T}}{\tilde{H}}^{\mathrm{T}}<0$$

is guaranteed if there exists a scalar $\beta >0$ such that

$$M+\beta \tilde{H}{\tilde{H}}^{\mathrm{T}}+{\displaystyle \frac{1}{\beta }}{\tilde{E}}^{\mathrm{T}}\tilde{E}<0.$$

3. Results

In this section, an $H_{\infty }$ tracking controller $u_1\left(k\right)$ will be first designed taking into consideration the DDPG-based controller $u_2\left(k\right)$ to guarantee closed-loop stability, a certain disturbance attenuation level and tracking performance. Building on this robust baseline controller, the auxiliary control input $u_2\left(k\right)$ will be subsequently optimized based on the improved DDPG algorithm. Thus the overall controller can be written as follows:

$$u\left(k\right)=u_1\left(k\right)+u_2\left(k\right)$$

(7)

where

$$u_1\left(k\right)=K\chi \left(k\right),u_2\left(k\right)=\Delta K\chi \left(k\right).$$

The controller gain $\Delta K$ to be optimized is defined as follows:

$$\Delta K=H_2{F}_2{E}_2,\parallel F_2\parallel \le 1,$$

where $H_2$ and $E_2$ are known matrices with compatible dimensions. then $\overline{K}=K+\Delta K,$ $\overline{A_c}=A^{\prime }+B_1^{\prime }K+\Delta A^{\prime }+B_1^{\prime }\Delta K$. Let $A_c=A^{\prime }+B_1^{\prime }K,\Delta A^{\prime }+B_1^{\prime }\Delta K=\overline{H}\overline{F}\overline{E}$, where:

$$\overline{H}=[\begin{array}{cc}{\overline{H}}_1\hfill & B_1^{\prime }{H}_2\hfill \end{array}],\overline{F}=\left[\begin{array}{cc}{F}_1\hfill & 0\hfill \\ 0\hfill & F_2\hfill \end{array}\right],\overline{E}=\left[\begin{array}{c}{E}_1\hfill \\ E_2\hfill \end{array}\right]$$

The resulting control framework is illustrated in Figure 1.

Figure 1. Overall control framework.

3.1. Robust State-Feedback Controller Design

Theorem 1. 

The closed-loop tracking error system (6) is asymptotically stable with $H_{\infty }$ performance $\gamma >0$ if there exists a matrix $P>0$ and a scalar $\beta >0$, such that the following condition holds:

$$\left[\begin{array}{ccccc}-P& C^{\prime \mathrm{T}}D& A_c^{\mathrm{T}}& {\overline{E}}^{\mathrm{T}}& C^{\prime \mathrm{T}}\\ D^{\mathrm{T}}{C}^{\prime }& -{\gamma }^{2}I+D^{\mathrm{T}}D& B_2^{\prime \mathrm{T}}& 0& 0\\ A_c& B_2^{\prime }& -P^{-1}+\beta \overline{H}{\overline{H}}^{\mathrm{T}}& 0& 0\\ \overline{E}& 0& 0& -\beta I& 0\\ C^{\prime }& 0& 0& 0& -I\end{array}\right]<0$$

(8)

Proof. 

Equation (8) can be rewritten by using the Schur complement lemma:

$$\left[\begin{array}{ccc}-P+C^{\prime \mathrm{T}}{C}^{\prime }+{\displaystyle \frac{1}{\beta }}{\overline{E}}^{\mathrm{T}}\overline{E}& C^{\prime \mathrm{T}}D& A_c^{\mathrm{T}}\\ D^{\mathrm{T}}{C}^{\prime }& -{\gamma }^{2}I+D^{\mathrm{T}}D& B_2^{\prime \mathrm{T}}\\ A_c& B_2^{\prime }& -P^{-1}+\beta \overline{H}{\overline{H}}^{\mathrm{T}}\end{array}\right]<0$$

(9)

For convenience of proof, let

$$M=\left[\begin{array}{ccc}-P+C^{\prime \mathrm{T}}{C}^{\prime }& C^{\prime \mathrm{T}}D& A_c^{\mathrm{T}}\\ D^{\mathrm{T}}{C}^{\prime }& -{\gamma }^{2}I+D^{\mathrm{T}}D& B_2^{\prime \mathrm{T}}\\ A_c& B_2^{\prime }& -P^{-1}\end{array}\right]$$

and $\tilde{H}=\left[\begin{array}{ccc}0& 0& {\overline{H}}^{\mathrm{T}}\end{array}\right],\tilde{E}=\left[\begin{array}{ccc}\overline{E}& 0& 0\end{array}\right]$

Equation (9) can be rewritten into the standard form of Petersen’s lemma. By Petersen’s Lemma, Equation (9) guarantees Equation (10).

$$\left[\begin{array}{ccc}-P+C^{\prime \mathrm{T}}{C}^{\prime }& C^{\prime \mathrm{T}}D& {\overline{A}}_c^{\mathrm{T}}\\ D^{\mathrm{T}}{C}^{\prime }& -{\gamma }^{2}I+D^{\mathrm{T}}D& B_2^{\prime \mathrm{T}}\\ {\overline{A}}_c& B_2^{\prime }& -P^{-1}\end{array}\right]<0$$

(10)

By invoking the Schur complement lemma, it follows that:

$$\Pi =\left[\begin{array}{cc}{\overline{A}}_c^{\mathrm{T}}P{\overline{A}}_c-P+C^{\prime \mathrm{T}}{C}^{\prime }& {\overline{A}}_c^{\mathrm{T}}P{B}_2^{\prime }+C^{\prime \mathrm{T}}D\\ B_2^{\prime \mathrm{T}}P{\overline{A}}_c+D^{\mathrm{T}}{C}^{\prime }& B_2^{\prime \mathrm{T}}P{B}_2^{\prime }+D^{\mathrm{T}}D-{\gamma }^{2}I\end{array}\right]<0$$

(11)

which implies the following inequality holds:

$${\overline{A}}_c^{\mathrm{T}}P{\overline{A}}_c-P<0$$

(12)

Selecting a Lyapunov function $V\left(k\right)=\chi {\left(k\right)}^{\mathrm{T}}P\chi \left(k\right)$, we calculate its difference $\Delta V\left(k\right)=V(k+1)-V\left(k\right)$ with $w\left(k\right)\equiv 0$ and can obtain the following equation:

$$\Delta V\left(k\right)=\chi {\left(k\right)}^{\mathrm{T}}({\overline{A}}_c^{\mathrm{T}}P{\overline{A}}_c-P)\chi \left(k\right)<0$$

(13)

which means the tracking error system (6) is asymptotically stable.

For $H_{\infty }$ performance, we consider the following expression:

$$\Delta V\left(k\right)+Z{\left(k\right)}^{\mathrm{T}}Z\left(k\right)-{\gamma }^{2}w{\left(k\right)}^{\mathrm{T}}w\left(k\right)={\left[\begin{array}{c}\chi \left(k\right)\\ w\left(k\right)\end{array}\right]}^{\mathrm{T}}\Pi \left[\begin{array}{c}\chi \left(k\right)\\ w\left(k\right)\end{array}\right]$$

(14)

From (11), it follows that:

$$\Delta V\left(k\right)+Z{\left(k\right)}^{\mathrm{T}}Z\left(k\right)-{\gamma }^{2}w{\left(k\right)}^{\mathrm{T}}w\left(k\right)<0$$

(15)

Summing (15) with respect to k from 0 to $N-1$ yields:

$$V\left(N\right)-V\left(0\right)+\sum _{k=0}^{N-1}Z{\left(k\right)}^{\mathrm{T}}Z\left(k\right)-{\gamma }^2\sum _{k=0}^{N-1}w{\left(k\right)}^{\mathrm{T}}w\left(k\right)<0$$

(16)

Note that $\chi \left(0\right)=0$ in the definition of $H_{\infty }$ performance and $\chi {\left(N\right)}^{\mathrm{T}}P\chi \left(N\right)\ge 0$. Let $N\to \infty$:

$$\sum _{k=0}^{\infty }Z{\left(k\right)}^{\mathrm{T}}Z\left(k\right)<{\gamma }^2\sum _{k=0}^{\infty }w{\left(k\right)}^{\mathrm{T}}w\left(k\right)$$

(17)

holds, which proves $H_{\infty }$ performance. Thus, Theorem 1 is proved.    □

Note that (8) involves certain bilinear coupling terms, making it be difficult to solve directly. To eliminate this nonlinearity, Theorem 2 will be proposed.

Theorem 2. 

The closed-loop tracking error system (6) is asymptotically stable with $H_{\infty }$ performance $\gamma >0$ if there exist matrices $X>0$ and V, and a scalar $\beta >0$, such that the following condition holds:

$$\left[\begin{array}{cccccc}-X& X{C}^{\prime \mathrm{T}}D& X{A}^{\prime \mathrm{T}}+V^{\mathrm{T}}{B}_1^{\prime \mathrm{T}}& X{E}_1^{\mathrm{T}}& X{E}_2^{\mathrm{T}}& X{C}^{\prime \mathrm{T}}\\ D^{\mathrm{T}}{C}^{\prime }X& -{\gamma }^{2}I+D^{\mathrm{T}}D& B_2^{\prime \mathrm{T}}& 0& 0& 0\\ A^{\prime }X+B_1^{\prime }V& B_2^{\prime }& -X+\beta \overline{H}{\overline{H}}^{\mathrm{T}}& 0& 0& 0\\ E_{1}X& 0& 0& -\beta I& 0& 0\\ E_{2}X& 0& 0& 0& -\beta I& 0\\ C^{\prime }X& 0& 0& 0& 0& -I\end{array}\right]<0$$

(18)

Accordingly, the robust controller is expressed as follows:

$$u_1\left(k\right)=V{X}^{-1}\chi \left(k\right)$$

(19)

Proof. 

Multiply both sides of Equation (18) by $diag\{X^{-1},I,I,I,I,I\}$:

$$\left[\begin{array}{cccccc}-X^{-1}& C^{\prime \mathrm{T}}D& A^{\prime \mathrm{T}}+X^{-1}{V}^{\mathrm{T}}{B}_1^{\prime \mathrm{T}}& E_1^{\mathrm{T}}& E_2^{\mathrm{T}}& C^{\prime \mathrm{T}}\\ D^{\mathrm{T}}{C}^{\prime }& -{\gamma }^{2}I+D^{\mathrm{T}}D& B_2^{\prime \mathrm{T}}& 0& 0& 0\\ A^{\prime }+B_1^{\prime }V{X}^{-1}& B_2^{\prime }& -X+\beta \overline{H}{\overline{H}}^{\mathrm{T}}& 0& 0& 0\\ E_1& 0& 0& -\beta I& 0& 0\\ E_2& 0& 0& 0& -\beta I& 0\\ C^{\prime }& 0& 0& 0& 0& -I\end{array}\right]<0$$

(20)

By defining

$$P=X^{-1},K=V{X}^{-1}$$

it follows that

$$A_c=A^{\prime }+B_1^{\prime }K=A^{\prime }+B_1^{\prime }V{X}^{-1}$$

and hence

$$A_c^{\mathrm{T}}=A^{\prime \mathrm{T}}+X^{-1}{V}^{\mathrm{T}}{B}_1^{\prime \mathrm{T}}$$

Moreover, since $X=P^{-1}$, the $(3,3)$-block satisfies

$$-X+\beta \overline{H}{\overline{H}}^{\mathrm{T}}=-P^{-1}+\beta \overline{H}{\overline{H}}^{\mathrm{T}}$$

and

$$\overline{E}=\left[\begin{array}{c}{E}_1\\ E_2\end{array}\right],{\overline{E}}^{\mathrm{T}}=\left[\begin{array}{cc}{E}_1^{\mathrm{T}}& E_2^{\mathrm{T}}\end{array}\right]$$

Then, the above inequality can be rewritten as follows:

$$\left[\begin{array}{ccccc}-P& C^{\prime \mathrm{T}}D& A_c^{\mathrm{T}}& {\overline{E}}^{\mathrm{T}}& C^{\prime \mathrm{T}}\\ D^{\mathrm{T}}{C}^{\prime }& -{\gamma }^{2}I+D^{\mathrm{T}}D& B_2^{\prime \mathrm{T}}& 0& 0\\ A_c& B_2^{\prime }& -P^{-1}+\beta \overline{H}{\overline{H}}^{\mathrm{T}}& 0& 0\\ \overline{E}& 0& 0& -\beta I& 0\\ C^{\prime }& 0& 0& 0& -I\end{array}\right]<0,$$

(21)

Thus, Theorem 2 is proved.    □

3.2. DDPG-Based Robust Controller Optimization

The $H_{\infty }$ controller $u_1\left(k\right)$ has been designed in section A, which guarantees closed-loop stability and achieves a certain level of disturbance attenuation under system uncertainties. However, its performance is inherently conservative due to considering the worst case of system uncertainty. To further improve the tracking accuracy and transient performance without sacrificing robustness, improved DDPG-based optimization is introduced in this section.

DDPG is a model-free, actor–critic method for continuous control, in which deep neural networks are employed as function approximators for both the deterministic policy and the action-value function. Specifically, the actor (policy network) outputs a parameterized deterministic action, and its parameters are updated using a policy-gradient signal provided by the critic. The critic (Q network) approximates the action–value function and is trained using targets constructed in a Q-learning manner, thereby evaluating state–action pairs and guiding the actor toward improved control performance. To enhance training stability and promote convergence, DDPG adopts a dual-network architecture consisting of a target policy network and a target critic network, which are maintained as delayed copies of the online actor and critic networks, respectively. The overall algorithmic structure is illustrated in Figure 2.

Figure 2. Structure of tracking-error-guided DDPG.

3.2.1. State, Action, and Reward Formulation

The following defines the environmental components of the DDPG algorithm in the context of the considered controller optimization problem.

State space: $s_t=[{\chi }_t,y_t,e_t,\Delta e_t],$ where ${\chi }_t$ is defined in the system (4), $y_t$ is defined in the system (1), $e_t$ denotes the tracking error, and $\Delta e_t=e_t-e_{t-1}$, t is the time instant during the training process.

Action space: action $a_t$ is the controller gain $\Delta K$ to be optimized, as output by the actor network.

Reward function: to ensure accurate and fast tracking performance, the reward function is designed by jointly considering tracking accuracy and transient response, as shown in (22):

$$\begin{array}{c}\hfill r_t={\delta }_1\ast e_t^{\mathrm{T}}{e}_t+{\delta }_2\ast \Delta e_t^{\mathrm{T}}\Delta e_t+{\delta }_3\ast t^2+{\delta }_4\ast max\{0,\parallel e_t\parallel -\epsilon \}+R_{ar}\end{array}$$

(22)

The first term $e_t^{\mathrm{T}}{e}_t$ penalizes the instantaneous tracking error, which encourages the agent to minimize the deviation between the system output and the reference signal, thereby improving steady-state accuracy. The second term $\Delta e_t^{\mathrm{T}}\Delta e_t$ penalizes the variation in the tracking error, promoting smooth system responses and suppressing abrupt control actions that may cause oscillations. The third term $t^2$ acts as a time penalty, which incentivizes the agent to achieve the desired tracking objective within fewer time. The fourth term $max\{0,\parallel e_t\parallel -\epsilon \}$ introduces an overshoot penalty when the tracking error exceeds a predefined tolerance $\epsilon$, effectively preventing excessive overshoot and enhancing system safety. Additionally, an extra constant reward $R_{ar}$ is granted when the output $y_t$ of system (1) is approaching the reference signal. ${\delta }_i,i=1,2,3,4$ are the weighting coefficients for each term.

3.2.2. Tracking-Error-Guided Exploration Strategy

In standard DDPG, exploration is typically achieved by injecting time-dependent random noise into the policy output. However, such exploration strategies are agnostic to the system performance evolution and the underlying structure of the control problem. As a result, exploration effort may be inefficiently distributed, generating a large number of low-informative samples in the replay buffer and slowing down learning convergence.

To address this issue, a tracking-error-guided exploration mechanism is introduced in this study, where the magnitude of exploration noise is adaptively modulated according to the system tracking error dynamics. The  variance of the exploration noise $\mathcal{N}\left({\mathcal{E}}_t\right)$ is defined as follows:

$$\sigma \left({\mathcal{E}}_t\right)={\sigma }_{min}+({\sigma }_{max}-{\sigma }_{min})exp(-\kappa {\mathcal{E}}_t),$$

(23)

where ${\mathcal{E}}_t=e_t^{\mathrm{T}}{e}_t+\lambda \Delta e_t^{\mathrm{T}}\Delta e_t$, and $\lambda$ is the weighting coefficient.

The parameters ${\sigma }_{min}$ and ${\sigma }_{max}$ are constant design parameters that define the minimum and maximum exploration intensities, respectively. Parameter $\kappa$ controls the sensitivity of the exploration noise to the energy function, governing the rate at which the exploration intensity decays as the system performance degrades. As a result, when the tracking error is small and the system exhibits favorable performance, a relatively larger exploration noise is introduced to encourage sufficient exploration, whereas the exploration intensity is automatically reduced when large tracking errors are observed, such that stronger exploration is encouraged when the system exhibits small tracking errors, while conservative behavior is enforced when the performance degrades.

3.2.3. Actor–Critic Network Architecture and Training

Based on the environment defined above, the learning framework is established using DDPG. The details of the actor network, critic network, and experience replay buffer are presented below.

(1) Actor Network

The input of the actor network is the current environment state $s_t$, and its output is an action $a_t=\mu \left(s_t\right|{\phi }_{\mu })$, where $\mu (·|{\phi }_{\mu })$ denotes the deterministic policy parameterized by ${\phi }_{\mu }$. ${\phi }_{\mu }$ and ${\phi }_{{\mu }^{\prime }}$ represent the parameters of the online and target actor networks, respectively. The parameters of the actor network are updated using the deterministic policy gradient, as given in (24):

$${\displaystyle {\nabla }_{{\phi }_{\mu }}\mu {\mid }_{s_i}\approx \frac{1}{N}\sum _i{\nabla }_{a}Q\left(s_i,a_i\mid {\phi }_Q\right){|}_{a_i=\mu \left(s_i\right)}\ast {\nabla }_{{\phi }_{\mu }}\mu (s_i\mid {\phi }_{\mu }){\mid }_{s_i}}$$

(24)

where the action–value function $Q(s,a)$ is approximated by the online critic network, while $Q^{\prime }(s,a)$ is produced by the target Critic network. During actor optimization, the online critic network is treated as a fixed, differentiable function that provides gradients with respect to the action. The target actor network is updated via a soft update:

$${\phi }_{u^{\prime }}\leftarrow \tau \ast {\phi }_u+(1-\tau )\ast {\phi }_{u^{\prime }}$$

(25)

where $\tau \in (0,1)$ is the soft update parameter.

(2) Critic Network

The critic network serves as a function approximator of the action–value function $Q(s,a)$, thereby scoring the actions output by the actor. The parameters ${\phi }_Q$ and ${\phi }_{Q^{\prime }}$ denote the online critic network and the target critic network, respectively. The parameters of the critic network are updated by minimizing the loss function $L_Q$, which is given by:

$$L_Q=\left({\displaystyle \frac{1}{N}}\right){\sum }_i{(Q(s_i,a_i|{\phi }_Q)-q_i)}^2$$

(26)

where $q_i=r_i+\xi \ast Q^{\prime }(s_{i+1},a_i^{\prime }\mid {\phi }_{Q^{\prime }})$, and $\xi$ is the discount factor. Similarly, the target critic network is updated as follows:

$${\phi }_{Q^{\prime }}\leftarrow \tau \ast {\phi }_Q+(1-\tau )\ast {\phi }_{Q^{\prime }}$$

(27)

(3) Experience Replay Buffer

The experience replay buffer stores past transition tuples ($s_t,a_t,r_t,s_{t+1}$). By sampling from this memory, temporal correlations are reduced so that training data are approximately i.i.d., which improves the stability and convergence of the algorithm.

Remark 1. 

Since the robust controller $\overline{K}$ has already incorporated $\Delta K$ into the uncertainty structure, further optimization via improved DDPG does not compromise system stability.

The pseudocode of the DDPG-based $H_{\infty }$ control algorithm is presented in Algorithm 1.

Algorithm 1: DDPG-Based $H_{\infty }$ Control Algorithm

Input: Initial optimization gain $\Delta K_0$, environment parameters, MaxEpisodes and                horizon T are set to 100.

Output: Optimized control gain $(K+\Delta K)$

Initialize: Actor network $\mu \left(s\right|{\phi }_{\mu })$, Critic network $Q(s,a|{\phi }_Q)$, target networks ${\mu }^{\prime }$, $Q^{\prime }$, and replay buffer D;

The $H_{\infty }$ controller gain K is obtained from (19)

for $episode=1$ to $MaxEpisodes$ do

Reset environment and obtain initial state;

$s_0=[X_0,y_0,e_0,\Delta e_0]$

for $t=1$ to T do

Select action $\Delta K_t=\mu \left(s_t\right|{\phi }_{\mu })$ with exploration noise $\mathcal{N}\left({\mathcal{E}}_t\right)$;

Execute action and the reward $r_t$ and next state $s_{t+1}$ are observed;

Store transition $(s_t,\Delta K_t,r_t,s_{t+1})$ into replay buffer D;

Sample N experiences from replay buffer D;

Update Actor network using Equation (24);

Update Critic network using Equation (26);

Update target networks using Equations (25) and (27).

4. Simulation Experiments

In this section, the effectiveness of the proposed method is validated through two simulation studies. In Section 4.1, the method is applied to an injection molding process [29,30], which represents a practical industrial scenario. In Section 4.2, an additional tracking task involving a sinusoidal reference signal is introduced to further evaluate the dynamic tracking capability and general applicability of the proposed approach.

4.1. Injection Molding Process

Injection molding generally consists of three operating phases: filling, packing (or holding), and cooling. During the filling stage, molten plastic is injected into the mold cavity at high pressure and velocity, requiring precise flow control to prevent short shots or flow marks. In the packing stage, cavity pressure is maintained to compensate for material shrinkage, thereby ensuring uniform product density and dimensional accuracy. Finally, in the cooling stage, the molten material gradually solidifies into the desired shape, followed by demolding.

The injection speed directly affects the mechanical properties, deformation, and dimensional accuracy of the molded product. Velocity control is therefore essential to ensure product quality consistency. Consequently, in this simulation, the filling stage is considered, where the injection speed is required to precisely track the prescribed setpoint to ensure a smooth transition to the subsequent production stage.

Following Reference [31], the injection process is modeled as in (1), where the input of the model is the valve opening and the output is the injection speed. The model parameters are given as follows:

$$A=\left[\begin{array}{cc}1.582& -0.5916\\ 1& 0\end{array}\right],B_1=\left[\begin{array}{c}1\\ 0\end{array}\right],B_2=1$$

$$C=\left[\begin{array}{cc}1.69& 1.419\end{array}\right],D=0$$

The uncertainty parameters in (2) are as follows:

$$H_1={\left[\begin{array}{cc}0.2& 0.2\end{array}\right]}^{\mathrm{T}},H_2=\left[\begin{array}{cccc}0.1& 0.2& 0.1& 0.4\end{array}\right]$$

$$F_1=1,F_2={\left[\begin{array}{cccc}1& 1& 1& 1\end{array}\right]}^{\mathrm{T}}$$

$$E_1=\left[\begin{array}{cc}0.2& 0.2\end{array}\right],E_2=\left[\begin{array}{cccc}0.2& 0.2& 0.6& 0.3\end{array}\right]$$

The target injection speed trajectory is generated by (3) with the following parameters:

$$A_d=diag\{1,1\},C_d=\left[\begin{array}{cc}1& 1\end{array}\right]$$

The target speed is set to 10 mm/s for $k\in [0,100]$, and 20 mm/s for $k\in [101,200]$.

By applying Theorem 2, the $H_{\infty }$ controller gain K is obtained:

$$K=\left[\begin{array}{cccc}-2.4364& 0.5881& 0.1444& 0.2881\end{array}\right]$$

(28)

Based on the $H_{\infty }$ controller in (28), the controller parameters are further optimized using an improved DDPG algorithm incorporating tracking-error-guided exploration. The relevant hyperparameters selection are shown in Table 1. The coefficients ${\delta }_i$ are selected as ${\delta }_1=-10$, ${\delta }_2=-4$, ${\delta }_3=-0.02$ and ${\delta }_4=-20$, with $\epsilon =0.5$ and $R_{ar}=20$.

Table 1. Hyperparameter settings for DDPG.

Hyperparameter	Value
Actor Network Learning Rate	${10}^{-6}$
Critic Network Learning Rate	${10}^{-5}$
Batch Size	256
Discount Factor ($\xi$)	$0.99$
Target Network Soft Update ($\tau$)	$0.001$
Replay Buffer Size	${10}^6$
${\sigma }_{min}$	0.01
${\sigma }_{max}$	0.4
$\kappa$	1
$\lambda$	0.3

To further investigate the contribution of the proposed exploration mechanism, a comparative study between the standard DDPG algorithm and the improved DDPG incorporating tracking-error-guided exploration is conducted. Both algorithms share identical network architectures and hyperparameter settings, except for the exploration strategy. The optimized robust controller gain parameters are as follows:

$${\overline{K}}_{DDPG}=\left[\begin{array}{cccc}-2.3939& 0.5670& 0.8310& 0.3520\end{array}\right]$$

(29)

$${\overline{K}}_{Improved}=\left[\begin{array}{cccc}-2.3927& 0.5678& 0.8289& 0.3517\end{array}\right]$$

(30)

The simulation comparison diagram with Gaussian white noise and the reward diagram are shown in Figure 3 and Figure 4, respectively. The light-colored lines indicate the raw reward of each episode during a single run, while the dark solid lines represent the reward smoothed by a moving average window within the same run. And the control input for the proposed method is also shown in Figure 5. It shows a brief initial oscillation and then gradually stabilizes around zero, indicating a normal and stable response.

Figure 3. Comparison between the DDPG and improved DDPG.

Figure 4. Reward comparison curve for the injection molding speed tracking task.

Figure 5. Control input.

To fully verify the effectiveness of the proposed control strategy, this paper first compares the tracking performance of the improved DDPG controller with the sparrow search algorithm optimized proportional-integral-derivative (SSA-PID) controller under disturbance-free conditions. Its output response and tracking error under ideal working conditions are shown in the Figure 6 and Figure 7.

Figure 6. Comparison between the SSA-PID and improved DDPG.

Figure 7. Tracking errors of SSA-PID and improved DDPG.

In the velocity tracking task of the injection molding process, the improved DDPG with tracking-error-guided exploration exhibits marginally superior performance relative to the original DDPG algorithm. Specifically, under identical training settings, it achieves smaller tracking errors and enhanced transient response. Furthermore, the cumulative reward evolution demonstrates a smoother and more stable convergence behavior, indicating reduced oscillations during policy updates and improved learning stability.

4.2. Sinusoidal Reference Signal Tracking Performance

This subsection evaluates the dynamic tracking capability of the proposed control framework by requiring the closed-loop system to follow a sinusoidal reference signal. Compared with the constant setpoint scenario used in Section 4.1, a sinusoidal trajectory provides a more stringent benchmark because it continuously varies with time and therefore reveals not only steady-state accuracy but also transient behaviors such as phase lag and disturbance sensitivity. Its model parameters are shown below:

$$A=\left[\begin{array}{cc}0.7& 0.2\\ 0.1& 0.6\end{array}\right],B_1=\left[\begin{array}{c}2\\ 1.6\end{array}\right],B_2=\left[\begin{array}{c}0.1\\ 0.2\end{array}\right]$$

$$C=\left[\begin{array}{cc}1& 2\end{array}\right],D=0$$

The uncertainty parameters in (2) are as follows:

$$H_1={\left[\begin{array}{cc}0.1& 0.1\end{array}\right]}^{\mathrm{T}},H_2=\left[\begin{array}{cccc}0.2& 0.1& 0.2& 0.2\end{array}\right]$$

$$F_1=1,F_2={\left[\begin{array}{cccc}1& 1& 1& 1\end{array}\right]}^{\mathrm{T}}$$

$$E_1=\left[\begin{array}{cc}0.1& 0.1\end{array}\right],E_2=\left[\begin{array}{cccc}0.1& 0.2& 0.3& 0.2\end{array}\right]$$

The amplitude of the reference signal is selected as 10, and its trajectory is generated by the following discrete reference system:

$$A_d=\left[\begin{array}{cc}cos(w\Delta t)& sin(w\Delta t)\\ -sin(w\Delta t)& cos(w\Delta t)\end{array}\right],C_d=\left[\begin{array}{cc}1& 0\end{array}\right]$$

where $\Delta t=0.01$, $w=24.88$ rad/s. Applying Theorem 2, the initial robust controller is obtained.

$$K=\left[\begin{array}{cccc}-0.1764& -0.2660& 0.0689& -0.0466\end{array}\right]$$

(31)

The training parameters are summarized in Table 1. The optimized controllers by the conventional DDPG and the improved DDPG are given below.

$${\overline{K}}_{DDPG}=\left[\begin{array}{cccc}-0.1779& -0.2704& 0.1739& 0.0588\end{array}\right]$$

(32)

$${\overline{K}}_{Improved}=\left[\begin{array}{cccc}-0.1779& -0.2684& 0.1875& 0.0476\end{array}\right]$$

(33)

To test the anti-disturbance performance of the proposed method, zero-mean Gaussian white noise with a standard deviation of 0.6 is added to the system as the process disturbance, and their control performance and reward profiles are compared in Figure 8 and Figure 9.

Figure 8. Sinusoidal reference signal tracking.

Figure 9. Reward comparison curve for the sinusoidal tracking task.

The simulation results show that the improved DDPG with tracking-error-guided exploration outperforms the original DDPG algorithm in terms of both tracking accuracy and dynamic response. The proposed approach yields smaller steady-state errors and reduced phase lag over the entire periodic trajectory, demonstrating stronger adaptability to time-varying reference signals. In addition, the cumulative reward remains consistently higher than that of the baseline method, reflecting improved policy quality and learning efficiency. These results illustrate that the tracking-error-guided exploration mechanism can effectively enhance tracking performance while facilitating more efficient reward optimization during the training process.

5. Conclusions

This paper develops a co-designed framework that combines robust $H_{\infty }$ synthesis with deep reinforcement learning for disturbed uncertain systems. The $H_{\infty }$ state-feedback controller serves as a robust baseline, ensuring closed-loop stability and effective disturbance attenuation in the presence of norm-bounded uncertainties. In addition to this baseline, DDPG is employed to further refine the feedback gains online so as to improve tracking performance and transient response. A key improvement lies in the proposed tracking-error-guided exploration strategy, which adapts the exploration intensity based on a tracking-error-guided mechanism, encouraging stabilizing exploration while suppressing aggressive actions when system performance deteriorates. The simulation results demonstrate that the proposed approach maintains strong robustness while achieving superior closed-loop performance.