Deep Reinforcement Learning-Based Water Jet Control for Robotic Manipulators Using an Improved Experience Replay Mechanism

Abstract

Existing robotic water jet control methods are limited by fixed spray configurations and low adaptability to complex or dynamic environments. These constraints hinder precise targeting in three-dimensional spaces. To overcome this, we propose a reinforcement learning-based water jet control framework that achieves accurate targeting without pose or angle restrictions. Specifically, we introduce Goal-Priority Hindsight Experience Replay (GPHER), a replay strategy that integrates the principles of Hindsight Experience Replay (HER), Prioritized Experience Replay (PER), and curriculum learning. GPHER dynamically adjusts sampling priorities based on goal-space distance, guiding training from simple to complex goals. Combined with Truncated Quantile Critics (TQCs), this approach accelerates convergence and enhances success rates. Both simulation and real-world experiments validate the robustness and adaptability of the proposed method, demonstrating its effectiveness for real-time robotic fluid control.

1. Introduction

Intelligent water-spraying robots have attracted increasing attention due to their potential applications in real-world scenarios such as firefighting, agricultural irrigation, and building façade cleaning. In firefighting tasks, robots are required to accurately direct water jets toward flames in complex and cluttered environments while maintaining a safe distance from hazardous areas [1]. In agricultural irrigation, crops are typically distributed irregularly in three-dimensional space, requiring precise control of water jet positions to ensure efficient watering while minimizing water waste [2]. Similarly, in building cleaning and industrial washing tasks, water jets must be directed toward specific surface locations at varying heights and orientations [3]. Achieving accurate water jet targeting in these scenarios is challenging because the landing position of the jet depends on multiple factors, including the nozzle position, orientation, jet velocity, and environmental disturbances such as air resistance. These factors introduce complex nonlinear dynamics that are difficult to model precisely using traditional analytical approaches.

Furthermore, studying robotic fluid manipulation is also important from the perspective of embodied intelligence. As robots are expected to operate in increasingly diverse environments, they must interact with objects with different physical properties, such as rigid bodies, deformable objects, and fluids. Compared with rigid object manipulation, fluid interaction introduces additional challenges due to its complex and dynamic behavior. Therefore, investigating robotic water jet control not only addresses specific application needs but also contributes to improving the general manipulation capabilities of robots in complex environments.

Water jet control therefore represents a complex robotic manipulation problem. Existing robotic fluid manipulation studies have largely focused on liquid pouring tasks [4,5], where fluids mainly flow downward under gravity. However, practical applications such as plant watering, firefighting, and high-altitude cleaning involve more complex spatial target distributions. To simplify the problem, many existing methods restrict certain control variables. For example, some approaches fix the spray angle while adjusting only the nozzle position [6]. In firefighting robotics, this strategy forces the robot to rely on base mobility for coarse positioning, which severely restricts its ability to reach targets in vertically cluttered environments such as multi-story buildings. Similarly, other studies keep the nozzle position fixed and vary only the spray angle to control the landing point [7]. While effective for planar targets, this approach cannot accommodate irregular three-dimensional crop distributions in agricultural settings, where plants vary in both height and horizontal offset. These simplified strategies fundamentally limit the controllable workspace and make it difficult to reach arbitrary target positions in three-dimensional space. Moreover, such control strategies often require significant manual tuning.

The choice of a robotic manipulator for water jet control is motivated by its inherent advantages in dexterity and flexibility. Unlike fixed spraying systems or those with limited mobility, a robotic arm provides multiple degrees of freedom (DOF), enabling simultaneous and continuous adjustment of both the nozzle’s position (X, Y, Z) and orientation (pitch, yaw). This capability is essential for directing a water jet to arbitrary points in three-dimensional space, a fundamental requirement in applications such as firefighting, where the robot must maintain a safe distance from hazardous areas while accurately targeting flames, or in agricultural irrigation, where crops are distributed irregularly across varying heights and angles. By leveraging the full pose control of a manipulator, the system can overcome the constrained workspace limitations of prior approaches [6,7] and achieve precise targeting under complex environmental conditions. Furthermore, studying robotic fluid manipulation with high-DOF arms contributes to the broader goal of embodied intelligence: as robots are expected to operate in increasingly diverse real-world scenarios, they must acquire the ability to interact with objects of varying physical properties, including rigid bodies, deformable materials, and fluids. Mastering water jet control thus serves not only a practical application need but also as a foundational step toward enhancing the generalization and adaptability of robotic systems in unstructured environments.

Reinforcement learning (RL) offers a novel research paradigm for water jet control tasks. Unlike traditional control methods that rely on accurate modeling, RL enables adaptive learning and optimization of control policies through interaction with simulation environments, without requiring explicit and complex fluid dynamic models. In contrast, conventional water control methods struggle with high-dimensional control problems due to their inherent limitations in control dimensionality. These methods often operate under constrained conditions, for instance, with fixed spray angles, which leads to limited adaptability and generalization capability. Despite the demonstrated flexibility and adaptability of RL in handling complex control tasks, existing replay mechanisms exhibit significant limitations in environments characterized by sparse rewards and complex goal spaces.

Hindsight Experience Replay (HER) mitigates the sparse reward problem by relabeling failed trajectories as successful experiences relative to achieved states, thereby improving learning efficiency. However, its uniform goal sampling neglects the varying importance of experiences for policy improvement, resulting in suboptimal sample utilization. In contrast, Prioritized Experience Replay (PER) assigns sampling weights based on Temporal Difference (TD) errors [8], enabling faster convergence but still suffering from inefficiency in sparse-reward scenarios due to the lack of valuable feedback samples. Integrating HER and PER has been explored to combine goal diversity with prioritized learning, yet the pseudo-success samples generated by HER distort the TD error distribution. Because these relabeled goals cluster in simpler regions of the task space, PER’s prioritization mechanism is weakened, ultimately reducing training effectiveness.

To overcome these limitations, this paper proposes Goal-Priority Hindsight Experience Replay (GPHER), a joint sampling framework that incorporates both goal-space distance and TD error. Adaptive weighting is applied to balance their influence across different training stages. Integrated with the Truncated Quantile Critics (TQC) algorithm, GPHER is evaluated in a simulated water jet control environment with air resistance modeling. Experimental results show that GPHER significantly accelerates convergence and enhances policy stability, achieving higher success rates, especially for high-difficulty targets, thereby improving both learning efficiency and generalization in complex robotic fluid control tasks.

Figure 1 illustrates the overall framework of the proposed robotic water jet control system. The framework includes three main stages: simulation environment construction for modeling the robotic manipulator and fluid dynamics, reinforcement learning–based policy training using the TQC algorithm combined with the proposed GPHER mechanism, and finally sim-to-real deployment with physical validation on a robotic platform. This pipeline illustrates how the water jet control task is formulated, how the learning agent is trained in simulation, and how the learned policy is transferred to the real system for evaluation.

2. Related Work

2.1. Robotic Water Manipulation Tasks

Robotic water flow manipulation is increasingly important in fields such as agriculture, firefighting, and cleaning. Most existing research, however, focuses on liquid pouring or fixed-point spraying in structured scenarios. For instance, Vatavuk et al. (2022) proposed a vineyard spraying method based on Task Space Model Predictive Control, which performs well in regularly planted vineyards but cannot adapt to complex three-dimensional target spaces [9]. Wei (2024) developed a six-axis robotic arm with a tracked chassis for automated irrigation, using Cartesian resolution and constrained joint optimization to reduce water deviation [10]. Nonetheless, its path planning relies on static modeling and cannot dynamically respond to plant morphology or obstacles. High-dimensional modeling complexity has led most studies to adopt limited control approaches, adjusting only spray angles or nozzle positions. Similarly, firefighting robots typically achieve coverage via base position and manipulator posture adjustments while maintaining fixed nozzle orientation [6], a strategy that is inefficient in cluttered or multi-level environments and prone to water wastage. In summary, traditional spraying methods are constrained by limited control degrees of freedom, poor environmental adaptability, and insufficient generalization.

These limitations highlight a fundamental challenge in robotic water manipulation: the difficulty of obtaining an accurate analytical model of the water jet’s behavior. The trajectory of a water jet is influenced by a complex set of parameters, which can be categorized into observable and unobservable factors. Observable parameters include the nozzle’s position and orientation, the initial jet velocity, and the target coordinates. These can be measured or controlled directly. However, numerous critical parameters are unobservable or highly uncertain in practice, such as the distribution of droplet sizes (which significantly affects air resistance), real-time fluctuations in flow rate and pressure, ambient air currents and turbulence, temperature-dependent variations in fluid viscosity and air density, surface tension effects, droplet breakup during flight, etc. The combined influence of these unobservable factors makes precise analytical modeling extremely challenging, as they introduce nonlinearities and random variations that are difficult to capture with deterministic equations. Conventional control methods for robotic water jet tasks can be broadly categorized into several approaches. Model-based methods, such as inverse kinematics combined with ballistic trajectory models, compute the required nozzle pose by solving analytical equations of motion under simplifying assumptions (e.g., neglecting air resistance or assuming uniform droplet size) [7]. While computationally efficient, these methods suffer from significant inaccuracies when unobservable factors are present, as their underlying models cannot capture the stochastic nature of real-world fluid behavior. Model Predictive Control (MPC) has been applied in structured agricultural settings [9], where it optimizes control inputs over a receding horizon based on a predefined dynamic model. However, MPC’s performance degrades substantially in dynamic environments where disturbances cannot be accurately predicted, and its computational complexity increases with model fidelity. Adaptive control techniques attempt to compensate for modeling errors online by estimating uncertain parameters during operation. Although they offer some robustness, these methods struggle with the high-dimensional, nonlinear dynamics of fluid–structure interaction and often require manual tuning of adaptation laws. More recently, learning-based approaches have emerged, but early attempts still relied on hybrid architectures that combine learned components with fixed analytical models [7], limiting their ability to fully adapt to complex environments. The fundamental limitation shared by these traditional methods is their reliance, whether explicit or implicit, on some form of analytical model that cannot adequately represent the unobservable factors discussed above. This motivates the adoption of a purely data-driven, model-free reinforcement learning framework that learns control policies directly from interaction experience, without requiring any prior analytical model of the fluid dynamics.

2.2. Experience Replay in Reinforcement Learning

Among existing experience replay mechanisms, HER provides an effective approach for goal-conditioned tasks under sparse rewards. By integrating with Universal Value Function Approximators [11], HER enables generalization to unseen desired goals. In long-horizon reaching tasks, HER demonstrates high sample efficiency [12,13]. To enhance HER’s performance, several research directions have been explored. These include improving the diversity of achieved goals through curriculum learning [14], leveraging demonstration data to boost training efficiency [15], generating more valuable desired goals [16], and incorporating curiosity-driven exploration mechanisms [17]. Further extending this line of work, some studies have combined continual reinforcement learning with HER in methods like RHER to enhance training efficiency [18]. Nonetheless, in complex environments such as the reaching sub-task within the water manipulation domain considered in this work, these methods, like the original HER, fail to account for the differential learning utility of various states. To provide a clearer understanding of the HER mechanism, Figure 2 illustrates the process of hindsight goal relabeling. As shown in Figure 2, failed trajectories can be relabeled with alternative goals that were actually achieved later in the episode, enabling the agent to learn from otherwise unsuccessful experiences.

While HER increases sample volume, PER emphasizes prioritized sampling. Recent methods aim to improve prioritization by incorporating informative signals, such as contact energy [19]. For example, Energy-Based Prioritization (EBP) [20] quantifies trajectory importance via an energy function, sampling high-energy episodes more frequently. Maximum Entropy-regularized Prioritization (MEP) [21] promotes exploration by encouraging uniform goal distributions, yet goal selection within trajectories remains uniform, allowing simpler samples to dominate. Studies have shown that directly combining HER and PER can sometimes perform worse than either alone [22], suggesting a mechanistic conflict. In complex water manipulation tasks with intricate dynamics and large goal spaces, such direct integration often fails to balance convergence speed and policy generalization.

3. Preliminaries

This section introduces the theoretical foundations used in this study. First, the Markov Decision Process (MDP) formulation of the reinforcement learning problem is presented. Then, the basic principles of experience replay mechanisms are introduced. Finally, the water jet motion model under air resistance is described, which provides the physical basis for the simulation environment used in the proposed framework.

3.1. Markov Decision Process

The core framework of RL is based on MDP, defined by the tuple $M=(S,A,P,R,\gamma ,T)$, where $S$ represents the state space, with $s_t\in S$ denoting the state at time t. $a_t\in A$ represents the action space, denoting the action taken at time t. $P\{·\}$ denotes the state transition probability function. The probability of the state at time t, $p_t$ can be described as:

$$p_t(s^{\prime }|s,a)=P\{S_t=s^{\prime }|S_{t-1}=s,A_{t-1}=a\},$$

(1)

the immediate reward $r_t$ can be computed as:

$$r_t=R(s_t,a_t),$$

(2)

where $R$ represents the reward function. The ultimate objective of reinforcement learning is to learn an optimal policy ${\pi }^{\ast }$ that maximizes the expected cumulative discounted reward:

$$J(\pi )=E_p\left[{\sum }_{t=0}^T{\gamma }^t{r}_t\mathsf{(}{s}_t,a_t,s_{t+1}\mathsf{)}\right],$$

(3)

where $J(\pi )$ denotes the expected return under policy $\pi$, where $E_p[·]$ denotes the expectation under the trajectory distribution induced by policy $\pi$ and the environment dynamics, the scalar $\gamma \in [0,1]$ represents the discount factor, t is the current number of episodes and T is the maximum episode length. The MDP framework establishes the mathematical basis for representing the robotic water jet control task within the reinforcement learning paradigm.

3.2. Experience Replay Algorithm

The Hindsight Experience Replay [12] introduces a goal relabeling mechanism that enables the agent to generate a large number of informative samples even from failed trajectories, thereby alleviating the inefficiency of learning in sparse-reward environments. HER relies on three key concepts: Desired goal ($g^d$), achieved goal ($g^a$) and hindsight goal ($g^h$).

Specifically, HER samples a subset of transitions from past trajectories and, with a given probability, replaces the original goal $g^d$ with a hindsight goal $g^h$ drawn from future achieved goals within the same episode. The reward is recomputed:

$$(s_t,a_t,r_t,s_{t+1}| g^d)\to (s_t,a_t,r_t^{\prime },s_{t+1}| g^h),$$

(4)

the reward $r_t$ for a failed step is the failure reward. After applying HER, those steps are reinterpreted as successful, and their reward value is converted to the success reward $r_t^{\prime }$. This process effectively transforms negative samples into informative positive samples without altering the task definition, thus greatly enhancing learning efficiency in sparse-reward environments.

PER assigns higher sampling probabilities to transitions with greater TD errors $\delta$, defined as:

$$\delta =Q_{\varphi }(s_t,a_t)-\left[r(s_t,a_t)+\gamma Q_{{\varphi }^{\prime }}(s_{t+1},a_{t+1})\right],$$

(5)

where $Q_{\varphi }(s_t,a_t)$ denotes the Q-value for the current state $s_t$ and action $a_t$, with $\varphi$ representing the parameters of the Q-network. The scalar $\gamma \in [0,1]$ represents the discount factor. The term $r(s_t,a_t)$ signifies the immediate reward received for executing action $a_t$ in state $s_t$. The sampling probability is calculated by the following formula:

$$p(i)={\displaystyle \frac{p_i^{\alpha }}{{\displaystyle \sum _{j=0}^N{p}_j^{\alpha }}}},$$

(6)

where N is the total number of samples in the buffer. $p_i$ is the priority value $p_i=|\delta (i)|+ϵ$, and $\alpha$ controls the degree of prioritization. Existing experience replay mechanisms such as HER and PER significantly improve learning efficiency, but their direct combination may still face challenges in complex environments with sparse rewards and large goal spaces, which motivates further improvements in replay strategies.

3.3. Water Jet Motion Modeling Under Air Resistance

In robotic water manipulation simulations, the trajectory of the water jet is influenced not only by gravity but also by aerodynamic drag, which becomes significant in the medium-to-low velocity range (several meters per second). The Reynolds number (Re) of droplets typically lies between 20 and 400, ignoring air resistance would result in substantial trajectory deviations, leading to discrepancies between simulation and reality. To address this issue, this study models air resistance using the classical drag force model [23]:

$${\overrightarrow{F}}_{\mathrm{drag}}=-{\displaystyle \frac{1}{2}}{\rho }_{a}A{C}_D|{\overrightarrow{v}}_0|{\overrightarrow{v}}_0=-\beta C_D|{\overrightarrow{v}}_0|{\overrightarrow{v}}_0,$$

(7)

where ${\rho }_a$ is the air density, $A$ is the droplet cross-sectional area with radius r, and $v_0$ is the velocity vector of the fluid particle. The parameters β and droplet mass m are expressed as functions of the particle radius r:

$$\beta (r)={\displaystyle \frac{1}{2}}{\rho }_{a}A={\rho }_a{\displaystyle \frac{\pi }{2}}{r}^2,$$

(8)

$$m(r)=\rho {\displaystyle \frac{4\pi }{3}}{r}^3,$$

(9)

where the drag coefficient $C_D$ for viscous droplets with dynamic viscosity contrast is obtained from Feng [24] as a function of Reynolds number:

$$Re(|\overrightarrow{v_0}|,r)={\displaystyle \frac{2r{\rho }_a|\overrightarrow{v_0}|}{{\mu }_a}},$$

(10)

where ${\mu }_a$ denotes the dynamic viscosity of air. The total force acting on each droplet in air can thus be expressed as:

$$m\dot{\overrightarrow{u}}=\overrightarrow{F}(t)={\overrightarrow{F}}_g+{\overrightarrow{F}}_{\mathrm{drag}}=-(m-m_a)g{\overrightarrow{e}}_{W_Z}-\beta C_D|{\overrightarrow{v}}_0|{\overrightarrow{v}}_0,$$

(11)

where $m_a$ is the mass of air displaced by the droplet (i.e., the buoyant force related mass), and g denotes the gravitational acceleration. Under the conditions specified in this study $T=20 ℃$, the parameter values are as follows: $g=9.81 {\mathrm{m}/\mathrm{s}}^2$, ${\rho }_a=1.204 {\mathrm{kg}/\mathrm{m}}^3$, ${\rho }_a=1.204 {\mathrm{kg}/\mathrm{m}}^3$ [25] and ${\mu }_a=1.813\cdot {10}^{-5} \mathrm{kg}/\mathrm{ms}$ [26].

As shown in Figure 3, the proposed water-jet motion model incorporates the effects of air resistance and other physical forces, enabling a more realistic representation of water jet dynamics in the simulation environment. This physical model provides the foundation for accurately evaluating the control policies learned by the reinforcement learning agent.

4. Methodology

4.1. State and Action Space of the Simulation Environment

The simulation environment for the robotic water jet control task was developed using the PyBullet physics engine, incorporating a particle-based fluid model to simulate the behavior of water droplets. All simulation experiments were conducted using Python 3.11 and PyBullet (version 3.2.7) on Ubuntu 22.04. The kinematic analysis of the water jet trajectory is formulated as follows:

As shown in Figure 4, let the world coordinate system be denoted as $\Sigma w:O-XYZ$, and the end-effector coordinate system of the robotic arm as $\Sigma z:o-xyz$. Its pose is represented by a rotation matrix $R\in SO(3)$ and a translation vector $P\in R^3$. The rotation matrix $R$ satisfies the definition of the Special Orthogonal Group:

$$SO(3) =\left\{R\in {ℝ}^{3\times 3}| R^{\top }R=I,\det (R)=1\right\},$$

(12)

where $P{=[\mathrm{p}}_x{,\mathrm{p}}_{y,}{\mathrm{p}}_z]$ represents the coordinates of the origin of the end-effector frame $\Sigma z$ expressed in the world frame $\Sigma w$. The column vectors of the rotation matrix R correspond to the projections of the basis vectors of $\Sigma z$ onto $\Sigma w$, given by:

$$R=\left[\begin{array}{ccc}{e}_x& e_y& e_z\end{array}\right],$$

(13)

here, $e_x$, $e_y$, $e_z$ represent the unit basis vectors of $\Sigma z$ projected in $\Sigma w$. The Z-axis of the end effector coordinate system corresponds to the direction of the water jet. The jet direction vector can be obtained from the third column of $R$:

$$d=e_z=R\left[:,3\right]={[d_X,d_Y,d_Z]}^T,$$

(14)

which satisfies the normalization condition:

$$‖d‖=\sqrt{d_X^2+d_Y^2+d_Z^2}=1,$$

(15)

where the pitch angle $\theta$ and yaw angle $\varphi$ are used to describe the orientation of the nozzle relative to the world coordinate:

$$\theta =\arcsin (d_Z),$$

(16)

$$\varphi =\arctan (d_Y/d_X).$$

(17)

For a desired target location ($d_x,d_{y,}{d}_z$), all simulated particles are sorted by their vertical distance from the target plane, and the first k particles with the smallest distances are selected to form a cluster. The centroid of this cluster $\overline{w}=({\overline{w}}_x,{\overline{w}}_y,{\overline{w}}_z)$ defines the landing position of the water jet:

$$({\overline{w}}_x,{\overline{w}}_y,{\overline{w}}_z)={\displaystyle \frac{1}{k}}{\displaystyle \sum _{i\in I_k}{D}_i}.$$

(18)

In the proposed robotic water jet control environment, the state space $s\in R^8$ consists of three components:

(1)

The Cartesian position of the end-effector ${[X_{arm},Y_{arm},Z_{arm}]}^T\in R^3$;

(2)

The nozzle orientation described by pitch and yaw angles $(\theta ,\varphi )\in R^2$;

(3)

The distribution of the landing points of the water droplets on the target plane $({\overline{w}}_x,{\overline{w}}_y,{\overline{w}}_z)\in R^3$.

The action space $a\in R^7$ corresponds to the continuous joint position commands for the seven joints of the robotic manipulator. Each action represents the target position increment applied to the corresponding joint at each control step. This formulation enables smooth motion control while maintaining compatibility with the physics simulation and facilitating sim-to-real transfer.

In summary, this section defines the state and action representations used in the robotic water jet control environment. The formulation captures the essential geometric relationship between the manipulator pose, jet direction, and landing position of the water particles.

For simplicity, several assumptions are adopted in the simulation environment, including a simplified particle-based representation of the water stream. While these simplifications improve computational efficiency during reinforcement learning training, they may not fully capture all complex fluid dynamics in real-world spraying scenarios. Nevertheless, the adopted model is sufficient for learning stable control policies and provides a reliable platform for evaluating the effectiveness of the proposed reinforcement learning framework.

4.2. Reward Function Design

Given the geometric nature of the water jet control task, the primary objective of the agent is to minimize the distance between the actual water jet landing point and the target location. Thus, a sparse reward function is defined as:

$$R=\left\{\begin{array}{l}0,\mathrm{if} ‖\mathsf{(}{w}_x,w_y,w_z\mathsf{)}-\mathsf{(}{d}_x,d_y,d_z\mathsf{)}‖\ge \epsilon \\ 1,\mathrm{if} ‖\mathsf{(}{w}_x,w_y,w_z\mathsf{)}-\mathsf{(}{d}_x,d_y,d_z\mathsf{)}‖<\epsilon \end{array}\right.,$$

(19)

where $({\overline{w}}_x,{\overline{w}}_y,{\overline{w}}_z)$ is the centroid of the selected water particle cluster, $(d_x,d_y,d_z)$ is the target position, and $\epsilon$ represents a small tolerance threshold. In this work, a sparse reward scheme is adopted. The use of sparse rewards reduces the risk of bias introduced by artificially designed dense reward functions [12].

The sparse reward formulation encourages the agent to focus on achieving accurate target reaching without introducing additional heuristic biases. However, this reward design simplifies the task objective by considering only the final landing position of the water jet. Other practical factors, such as water distribution efficiency, obstacle interference, or energy consumption, are not explicitly modeled in this study and may be considered in future work.

4.3. Goal-Priority Hindsight Experience Replay

Although HER has certain advantages in expanding the number of samples, the data it generates are not fully utilized by PER. The core idea of HER is to convert failed experiences into successful ones through goal relabeling, which can significantly increase the number of positive samples. Nevertheless, these ‘hindsight goals’ are essentially virtual successes computed based on substitute goals. Since trajectories rarely reach high-difficulty positions during training, simpler goals are more likely to be resampled by HER. When these samples are incorporated into PER’s priority calculation, the TD error may not accurately reflect their value for policy improvement, and they may even be assigned excessively high priorities, leading the agent to over-rely on pseudo-success samples during training and weakening the exploration of true goals. This phenomenon is particularly prominent in complex water jet control tasks, where the task goal space is large, samples of distant or high-difficulty goals are scarce, and the substitute samples generated by HER dilute the weight of these key experiences in PER, ultimately affecting the overall training effectiveness and policy reliability. Meanwhile, in water jet control task, the system has higher requirements for reaching remote or boundary target positions, and relying solely on HER and PER makes it difficult to balance efficiency and success rate. To address this limitation, this paper introduces an enhanced experience replay sampling strategy named GPHER. This method introduces a goal relevance measure on the basis of traditional PER and thereby optimizes the sample selection strategy to more effectively highlight experiences closely related to the task goals during sampling. As shown in Figure 5, the GPHER framework designed in this paper further demonstrates its improved mechanism and workflow when combining HER and PER.

Within the framework of traditional PER that allocates sampling priorities based on TD error, it further adds the goal-space center distance as a weighting factor, thereby increasing the sampling probability of samples far from the goal center and forcing the agent to focus more on learning in high-difficulty goal regions. Meanwhile, the priority calculation not only depends on the desired goal itself but also incorporates the spatial relationship between the achieved goal and the goal center. Under the same goal conditions, it prioritizes sampling trajectory samples that are closer to difficult goals. The revised priority formula is:

$$p_i=|{\delta }_i|\cdot (1-\lambda )+{\displaystyle \frac{\Vert g_i^{\prime }-g_m\Vert }{\Vert g_{\max }-g_m\Vert }}\cdot \lambda +{\displaystyle \frac{\Vert \overline{w}-g_m\Vert }{\Vert g_{\max }-g_m\Vert }}\cdot \lambda +ϵ,$$

(20)

where $g_i^{\prime }$ is the goal of the current state transition tuple, $g_m$ is the center of the goal space, m is the dimensionality of the goal space, $g_{\max }$ is the farthest distance in the goal space. $\overline{w}$ is the achieved position, The change in $\lambda$ is as follows:

$$l_n=l_0+\mathsf{(}{l}_N-l_0\mathsf{)}{\displaystyle \frac{n}{N}},$$

(21)

where ${\lambda }_{\mathrm{n}}$ represents the parameter value at training step n, the maximum number of steps is N, the initial value is ${\lambda }_0=0$, and the final value is ${\lambda }_N=0.4$. This dynamic adjustment mechanism is similar to the idea of Curriculum Learning, but the difference is that it acts on the experience replay mechanism rather than directly changing the task goals. Compared to achieving curriculum-like training by modifying goals, this method can gradually increase the sampling difficulty without introducing environmental mutations, thereby focusing more on learning difficult goals in the later stages of training.

Since the sampling probability of samples is proportional to the priority measure, the experience distribution deviates from the uniform sampling assumption, thereby introducing estimation bias. To eliminate this bias, PER introduces importance sampling weights:

$$w_i={\left({\displaystyle \frac{1}{N}}\cdot {\displaystyle \frac{1}{P(i)}}\right)}^{\beta },$$

(22)

where N represents the size of the experience pool, $P(i)$ is the sampling probability of sample i. The parameter $\beta \in [0,1]$ controls the correction strength. The weights are typically normalized:

$$w_j=\left({(N^{\ast }P(j))}^{-\beta }\right)/{\max }_i(w_i),$$

(23)

the pseudocode of the Algorithm 1 is presented as follows:

Algorithm 1 TQC-GPHER

Initialize the policy network TQC parameters, total training steps N, current step n, minibatch size k, replay period K, buffer D and $p_1=1$

for episode = 1, X do

Sample a goal g and an initial state $S_t$

for t = 0, T do

Select an action using the policy ${\mathit{\pi }}_{\mathit{\theta }}$ from Algorithm: $a_t\leftarrow {\pi }_{\theta }(S_t\Vert g)$

Execute action, interact with the environment and observe $S_{t+1}$

end for

for t = 0, T do

Compute reward $r(t)=R(s,a,s^{\prime },g)$

Store transition $R(s,a,s^{\prime },g)$ into buffer D with priority: $p_t={\max }_{i<t} p_i$

Sample a set of goals G from the current episode states

for $g^{\prime }\in G$ do

Compute relabeled reward $r(t)=R(s,a,s^{\prime },g^{\prime })$

Store relabeled transition $\mathsf{(}{s}_t,a_t,s_{t+1},r_t^{\prime },g^{\prime }\mathsf{)}$ into replay buffer D

end for

end for

for t = 1 mod K then

for j = 1 to k do

Sample transition i from the buffer with probability $p(i)$

Compute importance-sampling weight $w_j$ as defined in Equation (23)

Compute TD error according to Equation (5)

Update transition priority $p_i$ according to Equation (20)

end for

Update network parameters

end for

end for

In the subsequent experimental setup, this paper selects the currently well-performing Truncated Quantile Critics (TQC) algorithm as the basic reinforcement learning framework. On this basis, the proposed GPHER mechanism (Algorithm 1) is compared with various mainstream experience replay methods to systematically evaluate its performance improvement in water jet control task. The overall algorithm flow of TQC-GPHER is shown in Figure 6. The robotic manipulator interacts with the simulation environment to generate trajectories, which are stored in the replay buffer and processed by the proposed GPHER sampling strategy before being used to update the policy network.

Overall, the proposed GPHER mechanism improves experience replay efficiency by integrating TD-error prioritization with goal-space difficulty information. This design enables the agent to focus more effectively on challenging goal regions during training. Nevertheless, the current formulation assumes that goal difficulty can be reasonably approximated by the distance from the goal-space center. Although effective in the considered water jet control task, more complex environments may require alternative difficulty metrics or adaptive goal representations, which remain an interesting direction for future research.

5. Simulations and Experiments

5.1. Simulation Environment Setup for the Robotic Water Jet Control Task

Direct training in real-world environments often faces issues such as low sample efficiency, high safety risks, and prohibitive experimental costs [14]. Therefore, this paper chooses to train the robotic arm in a simulation environment to accomplish the water jet control task and validate the performance of the proposed TQC-GPHER algorithm. The experimental platform is built based on the PyBullet physics engine combined with the Gymnasium library to construct the reinforcement learning environment, as shown in Figure 7. The figure shows the training process of an episode.

In this environment, the agent adjusts the water outlet position and jet angle by controlling the pose of the robotic arm. To ensure the quantifiability and observability of the experiments while maintaining applicability to real-world scenarios, this paper adopts a design scheme where “the robotic arm gripper holds a water pipe for spraying.” It should be noted that the water flow is modeled and generated by the PyBullet particle system. As this model does not explicitly consider aerodynamic effects, this study further incorporates the modeling of air resistance on fluid particles, adding air resistance to the motion state of each fluid particle at each simulation time step to improve the physical plausibility of the simulation and the credibility of the experimental results. This provides a more reliable foundation for subsequently applying the trained policies to real-world water jet control tasks (such as firefighting, agricultural irrigation, or high-altitude cleaning). The details of the simulation setup are presented in

Table 1. Simulation environment parameters.

Parameter	Symbol
End-effector position of the manipulator	${[X_{arm},Y_{arm},Z_{arm}]}^T$
End-effector orientation of the manipulator	$R$
Three-dimensional outlet velocity of water jet	$v_0$
Pitch angle of the nozzle	$\theta$
Yaw angle of the nozzle	$\varphi$
Desired landing point position	$\mathsf{(}{d}_x,d_y,d_z\mathsf{)}$
Radius of fluid particles	r

.

First, the range of the water jet control task needs to be analyzed to determine a suitable target scope. The diameter d of natural raindrops is constrained to approximately 1–7 mm. In each simulation time step, fluid particles with a radius r = 3 mm are selected for jetting. The reachable space of the fluid is determined through Monte Carlo sampling to establish the set of reachable targets. According to Figure 8, the target range can be taken as $\mathcal{D}$. The initial velocity for particle jetting is set to 4 m/s. Data from the reachable target dataset where $d_x>0.5$ are used for training:

5.2. Comparison with Baseline

To comprehensively evaluate the proposed method in the water jet control task, comparative experiments were conducted in the simulation environment using TQC as the base RL framework against various baseline experience replay mechanisms. All experiments used the same hyperparameters, listed in

Table 2. GPHER-TQC algorithm hyperparameters.

Hyperparameter	Value	Hyperparameter	Value
Learning rate	0.01	Neurons per hidden layer	256
Discount factor	0.98	Number of hindsight relabeled goals	4
Soft update coefficient	0.95	Entropy coefficient	1
Replay buffer capacity	1 × 106	Entropy target	$H_0$
Batch size	512	Number of quantiles	25
Maximum episode length	200	Huber quantile loss parameter	1
Number of network layers	2	Prioritization exponent	0.6

.

For fair comparison, all methods were trained for the same total of 750,000 environment steps, and the success rate curves report the average performance over five independent training runs. Since the episode lengths vary across different runs, the success rate data were interpolated to a unified episode interval. A moving average was further applied to smooth the curves and reduce the influence of stochastic fluctuations. Figure 9 shows the learning curves of each method. The results indicate that when HER and PER are simply combined, as PER relies solely on TD error for importance sampling, the virtual goal samples introduced by HER relabeling are not fully utilized, instead weakening the prioritized sampling effect of PER, leading to overall performance even worse than using HER alone. After approximately episode 1500, the success rate of this method shows a significant decline, indicating a lack of policy stability. In contrast, the proposed GPHER exhibits faster convergence speed in the early stages of training and reaches a success rate of about 95% around episode 1500. Subsequently, its success rate remains stable at around 98%, significantly outperforming HER and Prioritized Hindsight Experience Replay (PHER) (85% with large fluctuations).

For the water jet control task, stable reaching performance across the entire reachable space is essential. To evaluate performance in challenging goal regions, comparative tests were conducted between PHER and GPHER, both outperforming HER. Each method was tested over 100 episodes on two representative planes: the horizontal plane (X = 1) and the vertical plane (Z = 0), where the fluid particle landing density is highest. For consistency, all target regions were defined as square areas and discretized into 16 configurations. On the X = 1 plane, regions I–IV and A–D corresponded to evenly spaced segments along the Z-axes and Y-axes, respectively, while on Z = 0, they represented divisions along the X- and Y-axes. As shown in Figure 10, GPHER achieves an average success rate of 96.8% in the edge and high-difficulty regions across the two evaluated scenarios, which is substantially higher than the 91.46% achieved by PHER. These results indicate that GPHER exhibits markedly improved performance in challenging boundary regions, demonstrating superior exploration efficiency, policy generalization, and robustness compared with PHER.

5.3. Ablation Experiment

To evaluate the contribution of each component in GPHER, ablation experiments were conducted in the same simulation environment, comparing TQC alone, HER, PER, and GPHER. As shown in Figure 11, HER accelerates early-stage learning, reaching 80% success around episode 1100, but its relabeling mechanism favors simpler goals, limiting exposure to difficult targets and yielding a final success rate of 82%. In contrast, GPHER maintains higher sample efficiency throughout training and achieves a stable final success rate of 98%, significantly outperforming HER. This confirms that the joint prioritization of goal difficulty and TD error in GPHER effectively mitigates HER’s bias toward simple samples, enhancing both policy learning efficiency and generalization.

5.4. Extending PGHER to Other Reaching Tasks

To further verify the general applicability of the proposed GPHER method, this section selects the relatively simple robotic arm Reach task as a supplementary experimental scenario. In the experimental setup, adjustments were made to the previous training details: During the same 250,000 training steps, the minibatch size was reduced from 512 to 256, and the PyTorch-based DDPG algorithm was used as the base learning framework.

In the Reach task, the robot end-effector was required to reach randomly sampled target positions within a predefined three-dimensional workspace. Specifically, the goal positions were uniformly sampled from a cubic region defined in the world coordinate frame, with ranges of [−1, 1] m along the x-axis, [0, 2] m along the y-axis, and [0.1, 0.3] m along the z-axis.

A reaching attempt was considered successful if the Euclidean distance between the end-effector position and the target goal was below a predefined threshold of 0.05 m. The success rate was computed as the ratio of successful episodes over the total number of episodes and was averaged across multiple independent training runs.

The experimental results are shown in Figure 12. As shown in the figure, GPHER achieves a success rate of approximately 0.95 within the first 500 training episodes, significantly outperforming the baseline methods during the early training stage. This indicates substantially higher sample efficiency compared with HER, PER, and their direct combination. Moreover, GPHER maintains a higher and more stable success rate throughout the remainder of the training process, with reduced performance fluctuations relative to the comparison algorithms.

5.5. Real-World Experiments

This paper further validates the effectiveness of the proposed method on a physical platform. The experiment adopts a two-stage deep reinforcement learning training framework, which separately models visual perception and the control policy. As shown in Figure 13, the visual perception module performs target detection and localization of the watering area and landing points, providing this as structured input to the control policy module for subsequent path planning and action decision training. This design effectively avoids the reliance of end-to-end methods on large-scale fine-grained modeling and high-quality annotated data while enhancing the transferability and generalization of the algorithm.

The real-world experiments were conducted using a Franka Emika Panda robotic manipulator, a widely used 7-degree-of-freedom (7-DOF) collaborative robot designed for research applications. The robot has a maximum reach of approximately 0.85 m and a payload capacity of 3 kg. A water nozzle connected to a pressurized water source was mounted at the end-effector of the robotic arm, allowing the jet direction to be controlled through the end-effector pose.

During the experiments, the robotic arm was used to adjust the position and orientation of the nozzle to direct the water jet toward specified target locations. The distance between the robotic arm base and the target surface was approximately 0–2 m, which lies within the reachable workspace considered in the simulation environment. The water jet velocity was maintained at approximately 4 m/s, consistent with the parameter settings used in the simulation environment.

The control commands were generated by the trained reinforcement learning policy and executed through the robot’s position control interface. The control framework was implemented on a workstation running Ubuntu, where the trained neural network model was deployed using PyTorch (version 2.5.1) for real-time inference.

All experiments were conducted in an indoor laboratory environment with relatively stable lighting conditions and minimal airflow to reduce disturbances to the water jet. Although the experimental setup simplifies certain environmental factors such as wind disturbances and water pressure variations, it is sufficient to validate the effectiveness and robustness of the proposed control framework.

For the sim-to-real validation, models trained with three different replay mechanisms (HER, PHER, and the proposed GPHER) were evaluated on a physical robotic platform under identical experimental conditions. Two representative difficult target regions were considered: the boundary regions of the horizontal plane (Z = 0) and the boundary regions of the vertical plane (X = 1). For each scenario, 100 target positions were randomly sampled and tested.

As summarized in

Table 3. Average success rates of models in the physical robotic experiments.

Trained Model	Difficult Target Region at X = 1	Difficult Target Region at Z = 0	Random Target Positions in the Goal Space
GPHER	0.93	0.95	0.96
PHER	0.83	0.84	0.85
HER	0.83	0.82	0.81

, the proposed GPHER method consistently achieves the highest success rates across all evaluation settings. In the difficult target regions on the vertical plane (X = 1), GPHER attains a success rate of 0.93, outperforming PHER and HER by 10 percentage points (0.83 for both). Similarly, on the horizontal plane (Z = 0), GPHER reaches a success rate of 0.95, compared with 0.84 for PHER and 0.82 for HER, corresponding to a relative improvement of approximately 13.1% and 15.9%, respectively.

When evaluated over randomly sampled target positions across the entire goal space, GPHER maintains a high success rate of 0.96, exceeding PHER (0.85) and HER (0.81) by 11 and 15 percentage points, respectively. These results indicate that GPHER not only improves overall task success but also demonstrates superior robustness and generalization capability in real-world conditions.

Notably, the performance gap between GPHER and the baseline methods is most pronounced in the difficult target regions, which correspond to the boundary areas of the workspace where control precision and trajectory stability are hardest to achieve. This observation aligns with the design motivation of GPHER, which explicitly increases the sampling priority of high-difficulty goals during training, enabling the learned policy to better handle edge cases that are underrepresented in standard replay strategies.

6. Conclusions and Future Works

Addressing the lack of techniques in the field of intelligent water jet control that enables directly hitting targets without imposing pose constraints, this paper proposes, for the first time, the use of reinforcement learning to control the robotic arm for this task. To address the issue where data generated by HER often concentrates within simple goal ranges, which results in an uneven goal distribution in the replay buffer, suboptimal learning efficiency, and low post-training policy success rates, particularly for critical edge targets in water jet control tasks, this paper proposes GPHER, an improved method that integrates HER and PER. Model validation shows that the proposed algorithm achieves a success rate of up to 99% in the simulated water control environment. Experiments conducted on a real robotic arm yield a success rate of 96% in the physical setup. Furthermore, validation performed on a Reach task demonstrates the method’s potential for generalization. Nevertheless, several limitations remain in the current study. The experimental setup does not explicitly model environmental disturbances such as wind or water pressure variations, which may influence the water jet trajectory in real-world applications.

Future work will focus on extending the proposed framework to more complex and realistic scenarios. In particular, we plan to incorporate fluid properties such as varying flow velocities and fluid viscosity into the state representation to better capture the physical characteristics of different fluids. Furthermore, the system will be integrated with a mobile robotic platform to expand the operational workspace and improve task flexibility. Finally, future studies will investigate the robustness of the proposed approach in outdoor environments with dynamic airflow and other environmental disturbances.