DRFW-TQC: Reinforcement Learning for Robotic Strawberry Picking with Dynamic Regularization and Feature Weighting

Abstract

Strawberry harvesting represents a labor-intensive agricultural operation where existing end-effector pose control algorithms frequently exhibit insufficient precision in fruit grasping, often resulting in unintended damage to target fruits. Concurrently, deep learning-based pose control algorithms suffer from inherent training instability, slow convergence rates, and inefficient learning processes in complex environments characterized by high-density fruit clusters and occluded picking scenarios. To address these challenges, this paper proposes an enhanced reinforcement learning framework DRFW-TQC that integrates Dynamic L2 Regularization for adaptive model stabilization and a Group-Wise Feature Weighting Network for discriminative feature representation. The methodology further incorporates a picking posture traction mechanism to optimize end-effector orientation control. The experimental results demonstrate the superior performance of DRFW-TQC compared to the baseline. The proposed approach achieves a 16.0% higher picking success rate and a 20.3% reduction in angular error with four target strawberries. Most notably, the framework’s transfer strategy effectively addresses the efficiency challenge in complex environments, maintaining an 89.1% success rate in eight-strawberry while reducing the timeout count by 60.2% compared to non-adaptive methods. These results confirm that DRFW-TQC successfully resolves the tripartite challenge of operational precision, training stability, and environmental adaptability in robotic fruit harvesting systems.

1. Introduction

This paper presents an integrated solution combining enhanced reinforcement learning with specialized end-effector control for the Franka Emika Panda robotic arm. Precision robotic harvesting in ridge-cultivated strawberry fields presents significant technical challenges due to the demanding requirements of end-effector pose control in dense, unstructured environments [1,2,3]. The development of reliable automated picking systems necessitates addressing three fundamental issues: precise end-effector trajectory planning for delicate fruit handling, stable training of deep reinforcement learning models under sparse reward conditions, and efficient adaptation to complex field environments. Traditional model-based control approaches, while effective in structured settings, often fail to meet these requirements due to their limited adaptability to the dynamic spatial configurations characteristic of ridge cultivation systems [4,5,6].

Recent advances in deep reinforcement learning have demonstrated promising results for robotic manipulation tasks [7], yet existing algorithms exhibit notable limitations when applied to strawberry harvesting scenarios. The high variability in fruit positioning [8], combined with frequent occlusions and delicate handling requirements, creates a challenging environment for conventional Deep Deterministic Policy Gradient and Soft Actor-Critic methods. These approaches typically suffer from training instability, slow convergence rates, and suboptimal performance in high-density picking scenarios, particularly when dealing with the seven-degree-of-freedom pose control required for precise end-effector alignment.

This paper introduces several key innovations to address the aforementioned challenges. The main contributions of this study are as follows:

• A novel posture optimization reward function is developed to guide precise end-effector alignment with target strawberries, incorporating both spatial positioning and orientation constraints to ensure optimal grasping postures in cluttered environments.
• The DRFW-TQC framework implements group-wise feature weighting and Dynamic L2 Regularization techniques within the Truncated Quantile Critics [9] algorithm to accelerate convergence and enhance training stability, effectively addressing the sparse reward problem inherent in complex picking scenarios.
• The proposed framework further incorporates a transfer strategy [10] that systematically migrates both the experience replay buffer and the learned networks to complex operational conditions, significantly improving training efficiency in challenging complex conditions.

2. Materials and Methods

This section delineates the comprehensive methodology employed in developing the robotic strawberry picking system. We establish the simulation environment, detailing its mechanical validation and spatial distribution. Subsequently, we present the posture optimization reward function that integrates spatial positioning and orientation constraints. The characterization of the state and action spaces follows, specifying the joint angle parameters and motion control mechanisms. In conclusion, we expound upon the DRFW-TQC framework’s innovative contributions, which collectively address the challenges of training, convergence, and policy optimization in complex agricultural environments.

2.1. Strawberry Picking Simulation Environment Modeling

To meet strawberry harvesting precision requirements while overcoming the cost and practical constraints of real-world training, we present a robotic strawberry harvesting system integrating DRFW-TQC with precision end-effector control for the Franka Emika Panda manipulator, utilizing a high-fidelity simulation environment constructed through Blender modeling software. The simulation environment has been rigorously validated to ensure its mechanical and spatial parameters fall within a 5% deviation from actual field measurements [11], with texture optimization balancing visual verisimilitude against computational efficiency. As shown in Figure 1a–d, the environment simulates a typical layout for strawberry picking, which mainly consists of a Franka Emika Panda manipulator, strawberry stalks, strawberry leaves, a ridge field, and multiple mature and immature strawberries in red and green hues, respectively. The simulation environment incorporates stochastic spatial distribution patterns, whereby individual strawberries and leaves exhibit randomized positional coordinates and quaternion-based orientation vectors. The combination of these elements results in the generation of random and naturalistic occlusion scenarios. Each part of the strawberry plant is equipped with a corresponding collision detection model that accurately reflects the mechanical properties in reality. In the splicing work of the simulation environment, this study adopts the splicing method of modularity and a coordinate system to organically combine several independent environmental elements and accurately locate these sub-modules by the spatial positions of the actual planting.

2.2. Posture Optimization Reward Function

In order to facilitate the precise alignment of the end-effector with the target strawberries, whilst incorporating both spatial positioning and orientation constraints, the posture optimization reward function is meticulously engineered within the DRFW-TQC framework. This sophisticated reward formulation directly supports the stringent success criteria established for the strawberry picking task, where completion requires the successful harvesting of all target fruits in the scene. The picking success evaluation incorporates multiple precision metrics, including the following:

• The attitude alignment condition is satisfied.
• The end-effector contacts the target strawberry.
• The gripper jaws successfully close.

In accordance with Figure 2c, the task is divided into two phases: approach and grasping. The robotic arm calculates the distance $d_t=\parallel P_i-T_i\parallel$ between the end-effector position $P_i$ and the target strawberry position $T_i$ and moves to the target so that $d_t$ is less than a preset threshold $d_{\varphi }$ while adjusting its attitude [12]. As shown in Figure 1e–h, the robotic arm approaches each strawberry in sequence. The attitude alignment condition is met when the angular error for grasping ${\theta }_{\mathrm{error}}$ satisfies

$${\theta }_{\mathrm{error}}=\pi -min({\theta }_P,{\theta }_{-P}),$$

(1)

where ${\theta }_P=2arccos|q_{P_i}·q_{T_i}|$ and ${\theta }_{-P}=2arccos|{-q}_{P_i}·q_{T_i}|$ represent the quaternion angular errors between the current orientation $q_{P_i}$ or its opposite $-q_{P_i}$ and the target orientation $q_{T_i}$. When ${\theta }_{\mathrm{error}}$ is less than the preset value, the attitude alignment condition is satisfied [13]. Upon entry of the gripper jaws of the end-effector into the preset grasping judgment area, the gripper jaws close. As shown in the orange boxes in Figure 1i–l, the robot arm gripper jaws close and pick each strawberry in order. Subsequently, if the sensor verification confirms the formation of an effective grip on the strawberry stalks, the system determines that the grasping phase is complete.

By separating the operation into a distinct approach and grasping phases, the system can first achieve precise positioning while avoiding collisions with adjacent fruits and foliage and then execute grasping with a proper judgment area to prevent bruising. The thresholds $d_{\varphi }$ and ${\theta }_{\mathrm{error}}$ ensure the end-effector maintains optimal positioning throughout both phases, theoretically minimizing damage risk while enabling efficient picking operations in dense strawberry canopies.

The reward function considers several factors comprehensively, including the distance from the target strawberry $r_{\mathrm{dis}}$, the success or failure of the grasping process $r_{\mathrm{suc}}$, the collision with the environment $r_{\mathrm{col}}$, and the timeout penalty $r_{\mathrm{tru}}$ [14]. The reward function is designed to guide the robot to approach the target strawberry with the correct attitude while avoiding unnecessary collisions and timeout behaviors. The distance and attitude alignment reward function between the robot end-effector and the target strawberry for each step is modeled as follows:

$$r_{\mathrm{dis}}=-{\delta }_1{(d_t+d_{\varphi })}^2-{\delta }_{2}max(z_{T_i}-z_{P_i},0)-{\delta }_3{\theta }_{\mathrm{error}}^2,$$

(2)

where $d_t$ denotes the distance between the end-effector position $P_i$ and the target strawberry position $T_i$. $d_{\varphi }$ is a distance threshold that prevents the agent from wandering near the edge of the strawberry. The terms $z_{T_i}$ and $z_{P_i}$ represent the positions on the Z-axis of the strawberry and the end-effector, respectively [15]. This methodology introduces an additional penalty for height differences, with the objective of encouraging the robot to complete the picking task at a higher position. This adaptation is intended to facilitate the robot’s performance in ridge cultivation environments. The parameter ${\theta }_{\mathrm{error}}$ is the angular error between the current pose and the target pose. The coefficients ${\delta }_1$, ${\delta }_2$, and ${\delta }_3$ are weighting factors that balance the learning process [16]. In the experiment, the values were set to 100, 150, and 20, respectively.

To prevent the robotic arm from excessively delaying the execution of the task, a timeout penalty $r_{\mathrm{tru}}$ is introduced. The task is considered truncated if the number of steps per episode exceeds 500. Each step performed before it is truncated or the task is completed is considered a timeout penalty. This design prevents the agent from wandering on invalid paths for extended periods. The timeout penalty for each step is defined as

$$r_{\mathrm{tru}}=-2\mathrm{if}\mathrm{not}\mathrm{truncated}\mathrm{or}\mathrm{completed}.$$

(3)

Collision detection is implemented using the pybullet.getContactPoints function [17], which penalizes the robot arm for collisions with non-target strawberries. The collision penalty $r_{\mathrm{col}}$ for each step is defined as

$$r_{\mathrm{col}}=\left\{\begin{array}{cc}-25\hfill & \mathrm{if}\mathrm{contact}\mathrm{with}\mathrm{non}-\mathrm{target}\mathrm{strawberry}\hfill \\ -\mathit{risk}\hfill & \mathrm{if}\mathrm{contact}\mathrm{with}\mathrm{obstacles}\hfill \end{array}\right.,$$

(4)

where risk represents the penalty severity for critical collisions that may damage the robotic system or crops. The hierarchical penalty structure ensures stronger constraints for high-risk interactions while allowing limited tolerance for non-target strawberry proximity during manipulation.

When the target strawberry meets the success criteria for picking, the reward function provides a significant reward to guide the agent toward learning an effective picking strategy. The success reward $r_{\mathrm{suc}}$ for each step is calculated as

$$r_{\mathrm{suc}}=\sum _{i=0}^{T_i}{R}_i\mathrm{if}\mathrm{success},$$

(5)

where $T_i\in \mathrm{sort}[T_1,T_2,\dots ,T_N]$ represents the sorted list of target strawberries according to the distance $d_t$ from the end-effector, as shown in Figure 2b. The final reward function $r_t$ for each step is the sum of all components:

$$r_t=r_{\mathrm{suc}}+r_{\mathrm{col}}+r_{\mathrm{tru}}+r_{\mathrm{dis}}.$$

(6)

By incorporating multiple factors such as distance, collision, and grasping success, the reward function enables the robot to learn an effective strategy for strawberry picking and gradually optimize the picking process.

2.3. State and Action Space Characterization

The observation space consists of the joint angles of the robot arm and the positions of the target strawberries ${\mathbf{s}}_{\mathbf{t}}=[P_{\theta },T_i]\in \mathcal{S}$. As demonstrated in Figure 2a, the joint angles represented by the sequence of numbers, denoted by $P_{\theta }=[{\theta }_1,{\theta }_2,{\theta }_3,\dots ,{\theta }_7]\in {\mathbb{R}}^7$, correspond to the current angular positions of the joints of the robot arm. These values constitute the internal state information of the robot. The end-effector position $P_i$ and strawberry positions $T_i=[x_i,y_i,z_i]\in {\mathbb{R}}^3$ are both acquired through the pybullet.getLinkState function, providing the 3D spatial information to determine the picking action [18]. In practical applications, when the robot completes the picking task of the current target strawberries, the system will re-select the next target according to the distance to ensure that the picking task can be carried out sequentially, which simplifies the problem that the state space is too large due to the simultaneous consideration of all strawberries [19,20].

As shown in Figure 2a, the action space is represented by the incremental movements of each joint ${\mathbf{a}}_{\mathbf{t}}=[a_1,a_2,\dots ,a_7]\in \mathcal{A}$. For strawberry picking tasks, the action space encompasses the motion control of the robot arm, enabling flexible movement in a three-dimensional space. The motion space consists of the control angles of seven joints, adjusted using the pybullet.setJointMotorControl2 function to reach and grasp the target strawberry [21].

2.4. DRFW-TQC Framework for Training Optimization

The DRFW-TQC framework addresses the critical challenges of slow convergence and training instability in complex strawberry harvesting scenarios through two key technical innovations. By incorporating the Group-Wise Feature Weighting Network in the Truncated Quantile Critics algorithm, abbreviated as TQC, the system prioritizes relevant state features during policy optimization, significantly improving exploration efficiency at the early stage of training, while Dynamic L2 Regularization techniques prevent overfitting to sparse reward signals. The effectiveness of group-wise feature weighting is demonstrated by the cumulative reward at the beginning of the training. Similarly, stable growth in the cumulative reward with low variance indicates effective regularization against sparse reward overfitting.

2.4.1. Truncated Quantile Critics Algorithm

The selection of TQC as the base algorithm is motivated by its distinctive advantages over alternative distributional RL approaches in continuous control domains. TQC’s quantile truncation mechanism has been demonstrated to exhibit superior computational efficiency in comparison to alternative distributional RL approaches of fixed quantile optimization [9]. In contrast to the methods of Quantile Regression-DQN and the Implicit Quantile Network, which necessitate the discretization of action spaces, TQC inherently accommodates continuous action spaces, thereby circumventing the loss of control precision that arises from the discretization of actions. This property is critical for robotic manipulation tasks that require precise motor control. The DRFW-TQC framework’s design intrinsically leverages the TQC’s unique quantile truncation mechanism to address the critical challenges of slow convergence and training instability in complex strawberry harvesting scenarios.

For balancing exploration and exploitation, the TQC algorithm enhances value function estimation through quantile truncation while maintaining the maximum entropy framework. The foundation lies in its reformulated objective function:

$$J\left(\alpha \right)={\mathbb{E}}_{\mathcal{D},{\pi }_{\varphi }}\left[log\alpha ·\left(-log{\pi }_{\varphi }\left({\mathbf{a}}_{\mathbf{t}}\right|{\mathbf{s}}_{\mathbf{t}})-{\mathcal{H}}_T\right)\right],$$

(7)

where $\mathcal{D}$ is the experience replay, and ${\pi }_{\varphi }$ denotes a parameterized stochastic policy that maps states to probability distributions over actions. This equation automatically adjusts the temperature parameter $\alpha$ to maintain stochasticity near the target entropy ${\mathcal{H}}_T$, ensuring adequate exploration during policy optimization.

Initially, as shown in the upper right part of Figure 3, each of the n target critic networks generates $N/n$ quantile-based approximations of the return distribution for the next state–action pair $({\mathbf{s}}_{\mathbf{t}}^{\prime },{\mathbf{a}}_{\mathbf{t}}^{\prime })$, producing M atoms per target critic [22]. These atoms are aggregated into a unified set:

$$\mathcal{Z}({\mathbf{s}}_{\mathbf{t}}^{\prime },{\mathbf{a}}_{\mathbf{t}}^{\prime })=\left\{{\theta }_{{\overline{\psi }}_n}^m({\mathbf{s}}_{\mathbf{t}}^{\prime },{\mathbf{a}}_{\mathbf{t}}^{\prime })\right\},$$

(8)

where$n\in \left[1,\dots ,N\right]\mathrm{and}m\in \left[1,\dots ,M\right]$, and ${\theta }_{{\overline{\psi }}_n}^m({\mathbf{s}}_{\mathbf{t}}^{\prime },{\mathbf{a}}_{\mathbf{t}}^{\prime })$ represents the parameters of the mth atom output by the nth target critic network. The approximations in this set are then sorted in ascending order, yielding an ordered sequence. To mitigate overestimation bias, the right tail of this combined distribution is truncated by discarding the $N-G$ largest approximations. The remaining G approximations are used to construct the target distribution. For each retained atom $z_{\left(i\right)}({\mathbf{s}}_{\mathbf{t}}^{\prime },{\mathbf{a}}_{\mathbf{t}}^{\prime })$, which is also the next quantile, the corresponding target quantile is computed as [23]

$$y_i({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}})=r({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}})+\zeta \left(z_{\left(i\right)}({\mathbf{s}}_{\mathbf{t}}^{\prime },{\mathbf{a}}_{\mathbf{t}}^{\prime })-\alpha log{\pi }_{\varphi }\left({\mathbf{a}}_{\mathbf{t}}^{\prime }\right|{\mathbf{s}}_{\mathbf{t}}^{\prime })\right),$$

(9)

where $\zeta \in [0,1)$ is the discount factor, and $r({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}})$ is immediate reward obtained after taking action ${\mathbf{a}}_{\mathbf{t}}$ in state ${\mathbf{s}}_{\mathbf{t}}$. As shown in the lower right part of Figure 3, the target quantile is used later in the computation of the critic loss.

The TQC algorithm integrates the maximum entropy reinforcement learning framework with the output of the quantile critic. The actor network’s objective function is given by

$$J_{\pi }\left(\varphi \right)={\mathbb{E}}_{\mathcal{D},{\pi }_{\varphi }}\left[\alpha log{\pi }_{\varphi }\left({\mathbf{a}}_{\mathbf{t}}\right|{\mathbf{s}}_{\mathbf{t}})-\frac{1}{NM}\sum _{n,m}{\theta }_{{\psi }_n}^m({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}})\right],$$

(10)

where ${\theta }_{{\psi }_n}^m({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}})$ represents the current quantile of the mth current atom output by the nth critic network.

While the next quantiles are trained on truncated targets, the actor network is optimized using the full, untruncated distribution to avoid compounding the truncation effect. This decoupling of value learning and policy updates is a critical design choice that contributes to the method’s robustness and performance.

2.4.2. Group-Wise Feature Weighting Network

In order to resolve the issues of sluggish convergence and inadequate key feature extraction inherent to the TQC algorithm when undertaking the early strawberry picking training, due to truncation, this paper proposes the introduction of a Group-Wise Feature Weighting Network, abbreviated as GFWN, within the actor network [24]. Specifically, the input features are divided into K groups along the channel dimension. As shown in Figure 4, for each group $x_k$, the group normalization module computes

$${\widehat{X}}_k={\displaystyle \frac{x_k-{\mu }_k}{\sqrt{{\sigma }_k^2+ϵ}}},$$

(11)

where $k\in$ [1, …, K], $x_k$ is the feature encoder for the kth group. ${\mu }_k$ and ${\sigma }_k^2$ denote the group-specific mean and variance, respectively.

A fully connected layer is applied to transform the normalized features, followed by a softmax activation function that generates a set of normalized attention weights $w_k$.

The final output combines the weighted features with a learnable scaling parameter $\gamma$:

$$w_{\mathrm{out}}=\gamma ·\sum _{k=1}^K{\omega }_k\odot {\widehat{X}}_k,$$

(12)

where ⊙ denotes element-wise multiplication, and the fused features $w_{\mathrm{out}}$ are subsequently fed into the actor network to generate the action distribution ${\pi }_{\varphi }$.

The Group-Wise Feature Weighting Network is an effective method of reducing the interference of irrelevant features. This enables the model to focus on task-relevant information during optimization. It is evident that by adjusting the feature weights during the training process, the DRFW-TQC is capable of resolving the TQC problem of losing early critical information due to truncation, thereby ensuring optimal exploration efficiency at the early stage of training.

2.4.3. Dynamic L2 Regularization

In order to address the issue of critic networks overfitting to sparse reward signals in strawberry picking training, which leads to poor generalization on the unseen state–action pairs and training instability, this paper introduces quantile Dynamic L2 Regularization, abbreviated as DL2 [25]. This regularization mechanism dynamically balances exploration and convergence by automatically adjusting its strength based on the training phase [26].

As demonstrated in the lower right quadrant of Figure 3, the critic network in TQC updates its parameters by minimizing the quantile Huber loss between the current quantiles and the target quantiles [27]. Specifically, given the current quantiles and the target quantiles from Equation (9), the quantile Huber loss is computed as

$$J_Q\left({\psi }_n\right)={\displaystyle \frac{1}{GM}}\sum _{i=1}^G\sum _{m=1}^M{\rho }_{{\tau }_m}^H\left(y_i({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}})-{\theta }_{{\psi }_n}^m({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}})\right),$$

(13)

where $y_i({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}})$ is the target quantile of the atoms output by the nth critic network, ${\theta }_{{\psi }_n}^m({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}})$ is the current quantile, ${\tau }_m={\displaystyle \frac{2m-1}{2M}}$ is the quantile position, and ${\rho }_{{\tau }_m}^H\left(\mu \right)$ is the Huber loss defined as

$${\rho }_{{\tau }_m}^H\left(\mu \right)=|{\tau }_m-\mathrm{II}(\mu <0)|·{\mathcal{L}}_H\left(\mu \right),$$

(14)

with ${\mathcal{L}}_H\left(\mu \right)$ given by

$${\mathcal{L}}_H\left(\mu \right)=\left\{\begin{array}{cc}{\displaystyle \frac{1}{2}}{\mu }^2\hfill & \mathrm{if}\left|\mu \right|\le \delta \hfill \\ \delta \left(\left|\mu \right|-{\displaystyle \frac{1}{2}}\delta \right)\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

(15)

where $\delta$ is a threshold parameter.

In addressing the problem of overfitting to sparse reward signals, a Dynamic L2 Regularization term is incorporated into the critic loss function. This regularization term penalizes large differences between different quantile estimates, encouraging smoother value distribution estimates while preventing overfitting to individual quantiles. The Dynamic L2 Regularization term is defined as

$${\mathcal{L}}_{L2}=\lambda ·{\displaystyle \frac{2}{N(N-1)}}\sum _{i=1}^N\sum _{j=i+1}^N{\parallel {\theta }_{{\psi }_i}({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}})-{\theta }_{{\psi }_j}({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}})\parallel }_2^2,$$

(16)

where ${\parallel ·\parallel }_2^2$ denotes the squared L2 norm, implemented as the MSE loss, and ${\theta }_{{\psi }_n}({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}})$ is the current quantile of the approximation output by the nth critic network. $\lambda$ is a dynamic regularization coefficient that controls the penalty strength. The coefficient $\lambda$ is dynamically adjusted during training as

$$\lambda =max\left(0,{10}^{-3}·\left(1-{\displaystyle \frac{t}{T}}\right)\right),$$

(17)

with t being the current training step and T being a predefined step threshold. The total critic loss function, incorporating both the quantile Huber loss and the Dynamic L2 Regularization term, is given by

$${\mathcal{J}}_Q\left({\psi }_n\right)=J_Q\left({\psi }_n\right)+{\mathcal{L}}_{L2}.$$

(18)

By introducing the DL2, the model’s tendency to overfit to training noise is suppressed in the early stages of training, while numerical instability caused by excessive constraints on the critic loss function is prevented in the later stages [28]. This ensures robust and efficient training throughout the learning process.

2.4.4. Multi-Objective Strawberry Picking Transfer Strategy

The challenge of state spaces in complex environments, which often hinder efficient model learning, is addressed by the employment of the transfer strategy in this paper, enabling rapid adaptation to complex environments [29]. As shown in Figure 5, a diverse set of offline datasets and network parameters are generated in the simple simulation environment, covering a wide range of task scenarios. Upon transfer to the complex environment, the model leverages the knowledge acquired from the simple environment to quickly adapt, thereby improving the success rate of strawberry picking.

The offline dataset is constructed from high-quality trajectory data generated by the reinforcement learning model, with each data point consisting of the state ${\mathbf{s}}_{\mathbf{t}}$, action ${\mathbf{a}}_{\mathbf{t}}$, reward $r_t$, and next state ${\mathbf{s}}_{\mathbf{t}+\mathbf{1}}$. The experimental results demonstrate that this approach significantly enhances the training efficiency of the model in complex environments.

3. Experimental Design and Evaluation Metrics

The task involves the sequential picking of multiple target strawberries, with the maximum number of training steps determined by the number of target strawberries. Each experiment runs for 1980 steps per epoch, with the results recorded and evaluated to ensure the stability and reproducibility of the experimental outcomes.

A series of ablation experiments have been conducted in order to evaluate the advantages of the DRFW-TQC in this particular picking task. The experiments have specifically introduced the DL2 and the GFWN. For the purpose of comparison, a selection has been made of widely used algorithms in continuous action space tasks. These include Soft Actor-Critic, also called SAC, and Deep Deterministic Policy Gradient, also called DDPG. It has been demonstrated that both of these algorithms exhibit strong performance in similar domains.

All strategy networks and value networks in this experiment consist of four hidden layers, each with 256 neurons [30]. The remaining parameters follow the default configurations of the stable-baselines3 library [31], which are based on extensive experimentation and tuning. These configurations provide a standardized setup applicable to a wide range of tasks, ensuring the fairness and consistency of the experiments.

The transition of the robotic arm from the initial phase to the terminated, truncated, or successful completion of the procedure is designated as an episode. In order to assess the performance of the algorithm in a rigorous manner, five quantitative metrics are adopted [32,33,34,35]. These are as follows:

• Average cumulative reward (AR), which measures the average of the cumulative rewards in the final ten epochs;
• Picking success rate (PS), which is the ratio of the number of successful episodes to the total episodes in the last ten epochs;
• Angular error (AE), or ${\theta }_{\mathrm{error}}$, is used to evaluate the accuracy of the end-effector’s grasping in the last ten epochs;
• First Success Step (FS), which is the total number of steps required for the first successful pick, indicates exploration efficiency;
• Timeout count (TO), the total number of truncated episodes, reflects the efficiency of policy convergence.

To ensure rigorous validation of the proposed DRFW-TQC framework’s performance, a comprehensive statistical analysis methodology was employed. Normality assumptions were first verified using the Shapiro–Wilk test statistic. For datasets satisfying normality assumptions, variance homogeneity was subsequently assessed through Levene’s test. When both normality and homoscedasticity conditions were met, ANOVA was employed for omnibus testing. For non-normal distributions or heterogeneous variances, the non-parametric Kruskal–Wallis test was utilized instead:

$$H_{KW}={\displaystyle \frac{12}{N(N+1)}}\sum _{i=1}^k{\displaystyle \frac{R_i^2}{n_i}}-3(N+1),$$

(19)

where N represents the total sample size across all k groups, $n_i$ denotes the sample size of the ith group, and $R_i$ corresponds to the sum of ranks for observations in the ith group when all N observations are jointly ranked.

Significant omnibus test results at the 0.05 threshold level initiated post hoc analyses, employing Tukey’s HSD test for parametric comparisons or Mann–Whitney U tests with Bonferroni correction for non-parametric cases. The effect magnitude quantification adopted Cohen’s d effect size measure:

$$d_{Ch}={\displaystyle \frac{{\overline{X}}_1-{\overline{X}}_2}{\sqrt{\frac{(n_1-1)s_1^2+(n_2-1)s_2^2}{n_1+n_2-2}}}},$$

(20)

where ${\overline{X}}_1$ and ${\overline{X}}_2$ represent the sample means of two independent groups, $n_1$ and $n_2$ are the sample sizes, and $s_1^2$, $s_2^2$ are the sample variances of the respective groups. The interpretation of the results considered both statistical significance with p-values below 0.01 indicating significant differences and effect sizes with Cohen’s d values exceeding 0.2 representing small effects and those above 0.5 indicating medium effects. The analytical approach ensured robust statistical inference while accommodating the varied distributional properties of different performance metrics.

4. Results and Discussion

This section presents a comprehensive analysis with comparative simulation studies that quantify improvements in five metrics relative to baseline methods. Furthermore, this section examines the framework’s transfer learning capabilities in complex environments and conducts hyperparameter sensitivity analyses to elucidate the contribution of key algorithmic components.

4.1. Results of Simulation Experiments

As illustrated in Figure 6a,b and

Table 1. Simulation results of various methods with 4 target strawberries. All values represent the mean ± standard error derived from 30 independent experimental trials.

Method	FS($\times {10}^3$)	TO	AE	AR ($\times {10}^4$)	PS
TQC	26.4 ± 3.7	294 ± 19	0.192 ± 0.015	20.87 ± 4.15	0.734 ± 0.060
TQC+DL2	16.4 ± 2.4	256 ± 16	0.175 ± 0.018	24.25 ± 3.93	0.807 ± 0.050
TQC+GFWN	25.9 ± 5.1	271 ± 17	0.179 ± 0.012	22.69 ± 4.01	0.851 ± 0.031
DRFW-TQC	15.1 ± 1.7	238 ± 17	0.153 ± 0.015	26.94 ± 3.62	0.851 ± 0.044
SAC	35.8 ± 5.5	301 ± 9	0.171 ± 0.016	9.52 ± 1.93	0.827± 0.039
DDPG	37.1 ± 4.7	419 ± 3	0.447 ± 0.031	−16.54 ± 1.25	0.074 ± 0.023

, the proposed DRFW-TQC framework, which integrates the DL2 and the GFWN, achieves superior performance across multiple metrics compared to baseline methods. The experimental results reveal that DRFW-TQC attains the highest average reward and the smallest angular error in the final training phase, demonstrating its robust optimization capability. Notably, while the baseline TQC achieves competitive cumulative rewards in later epochs, it had a picking success rate of only 73.39%, indicating suboptimal learning efficiency. In contrast, TQC+DL2 significantly enhances the early stages of exploration and renders the training process more stable, reducing the required steps for the first success to 16,420 while simultaneously improving the average cumulative reward to 242,501, suggesting reduced overfitting to sparse reward signals. Further refinement with GFWN yields additional improvements in long-term performance, elevating the picking success rate to 85.11% while maintaining high average cumulative rewards. As demonstrated in the reward graph depicted in Figure 6b, TQC+GFWN exhibits superior early average cumulative reward performance in comparison to competing approaches, which underscores its effectiveness in optimizing exploration of the early strawberry picking training and inadequate key feature extraction.

Comparative analysis with SAC and DDPG highlights DRFW-TQC’s superior convergence speed and stability, as evidenced by its consistently higher success rate progression. DDPG has been shown to exhibit diminished performance growth, whilst SAC has been demonstrated to demonstrate substandard cumulative reward performance. Conversely, DRFW-TQC has been shown to exhibit both accelerated initial learning and stable asymptotic performance. This validates the efficacy of DRFW-TQC in achieving a balance between exploration and exploitation, in addition to feature representation robustness.

The computational analysis demonstrates that the proposed enhancements introduce measurable yet justifiable processing requirements, with TQC+GFWN exhibiting minimal overhead at 8339 s, merely 0.7% longer than the baseline TQC’s 8281 s due to lightweight feature weighting operations. The TQC+DL2 configuration shows a more significant impact at 12,919 s, representing a 56% increase primarily attributed to dynamic regularization calculations during critic updates. The DRFW-TQC framework combining both components requires 13,041 s, exhibiting a 57% longer processing time compared to the baseline TQC but delivering greater performance improvements.

In the course of model training for the strawberry picking task, the final grasping point location of the end-effector for each episode was visualized for each algorithm. Figure 7 illustrates the 3D distribution of the grasping points, where red points denote correct locations, black points denote incorrect locations, and orange, green, and blue points represent projections on the XY, XZ, and YZ planes, respectively. As illustrated in Figure 7f, DRFW-TQC exhibits a higher concentration of red correct capture points in the proximity of the target area when compared to TQC+GFWN and TQC+DL2 in Figure 7d,e, accompanied by a reduced number of black error points. In contrast, DDPG and SAC in Figure 7a,b exhibit dispersed black error points, indicating lower localization accuracy. As illustrated in Figure 7c, TQC’s XY and YZ planes, the orange and blue points are more dispersed, suggesting that its final grasping point performance lags behind that of DRFW-TQC.

The proposed DRFW-TQC framework demonstrates statistically significant improvements across all evaluation metrics compared to baseline methods. As shown in

Table 2. Statistical significance test results compared with SAC, DDPG, and baseline TQC.

Comparison	Metric	Relative Improvement	p-Value	Effect Size
DRFW-TQC vs. TQC	PS	+16.0%	$5.19\times {10}^{-3}$	0.38
	AR	+29.1%	$7.23\times {10}^{-4}$	0.26
	AE	−20.3%	$9.50\times {10}^{-12}$	−0.46
	TO	−19.0%	$1.22\times {10}^{-9}$	−0.57
	FS	−42.7%	$5.84\times {10}^{-11}$	−0.71
DRFW-TQC vs. SAC	PS	+2.9%	$6.45\times {10}^{-8}$	0.09
	AR	+183.0%	$6.54\times {10}^{-21}$	0.93
	AE	−10.5%	$8.79\times {10}^{-4}$	−0.20
	TO	−20.9%	$2.28\times {10}^{-22}$	−0.84
	FS	−57.7%	$9.76\times {10}^{-20}$	−0.93
DRFW-TQC vs. DDPG	PS	+1050.0%	$2.94\times {10}^{-94}$	3.56
	AR	+262.9%	$1.41\times {10}^{-97}$	2.62
	AE	−65.8%	$6.38\times {10}^{-84}$	−2.19
	TO	−43.2%	$4.86\times {10}^{-90}$	−2.71
	FS	−59.2%	$1.57\times {10}^{-31}$	−1.15

, the experimental results demonstrate consistent and statistically significant improvements in the DRFW-TQC framework over the SAC, DDPG, and baseline TQC approach across all evaluation metrics. In terms of task completion quality, DRFW-TQC achieves a superior PS with a medium effect size of Cohen’s d equal to 0.38, where the p-value of 0.005 confirms statistical significance at the 1% level. The AR optimization performance of DRFW-TQC substantially outperforms TQC, achieving a 29.1% higher cumulative reward with extreme significance at p = 0.0007 and an effect size of 0.26. The AE further substantiates DRFW-TQC’s enhanced grasping accuracy, showing a reduction in error by 20.3% radians with an effect size of 0.46, extremely significant at p < 0.001. The training efficiency analysis reveals the particularly compelling advantages of DRFW-TQC.

This optimization in learning is accompanied by improved stability, as evidenced by a 19.0% reduction in TO, showing a medium effect size of 0.57 with p < 0.001. These comprehensive results establish DRFW-TQC as a superior approach to robotic strawberry harvesting, offering extremely significant improvements in both operational precision and learning efficiency compared to the baseline TQC method.

4.2. Experimental Results on Transfer Effects

The experimental results in

Table 3. Transfer performance of DRFW-TQC with 8 target strawberries. All values represent the average of 30 independent experimental trials, with PS and AR representing the average ± standard error.

Condition	FS	TO	AE	AR ($\times {10}^4$)	PS
With transfer strategy	44	111	0.091	62.21 ± 4.22	0.891 ± 0.032
Without	21,992	279	0.188	41.04 ± 5.35	0.774 ± 0.056

demonstrate that DRFW-TQC performance is significantly enhanced by leveraging the transfer strategy from a simple environment with 4 strawberries to a complex environment with 8 strawberries. DRFW-TQC with the transfer strategy achieves a substantial reduction in the FS and TO and reduces the AE to 0.091 rad. Moreover, it has been demonstrated that the AR is enhanced by 51.2%, elevating the performance of PS to 0.891. These results validate the effectiveness of the transfer strategy in complex environments, significantly improving DRFW-TQC efficiency and stability and providing strong support for the automated development of strawberry picking systems.

Upon analyzing each strawberry in contact with the robot arm for success and failure decomposition, it was determined that the accuracy characteristics remained consistent across the experimental conditions. As shown in Figure 8a, the transfer strategy configuration achieves angular errors below 0.05 radians in 76.42% of attempts, with only 0.55% exceeding this threshold. Non-target picking maintains comparable precision at a 22.70% success rate for low-error attempts. The non-transfer condition exhibits similar error distribution patterns, with 76.08% of target picks and 22.22% of non-target picks maintaining sub-0.05 radian errors in Figure 8b. The key distinction lies in the operational scale enabled by the transfer strategy, which facilitates a 2.6-fold increase in total picking attempts while preserving the angular precision characteristics. This scaling effect is particularly evident in the absolute numbers of high-precision attempts, which increase from 14,632 to 38,425 for target strawberries. The proportional increase in non-target picks from 4274 to 11,411 similarly reflects this expanded operational scope rather than any degradation in discrimination capability. These findings suggest that the transfer strategy effectively preserves the robotic arm’s precise properties when in contact with strawberries while concurrently enhancing its capacity to function in complex environments.

4.3. Experimental Results for Hyperparameters

4.3.1. Hyperparameter Analysis of Distance and Attitude Alignment Reward Function

Due to the stochastic nature of reinforcement learning, no single parameter set consistently outperforms all others in every experimental run; however, the selected weighting coefficients ${\delta }_1=100$, ${\delta }_2=150$, and ${\delta }_3=20$ in Equation (2) demonstrate robust performance across multiple evaluation metrics while exhibiting relative insensitivity to small parameter variations. The choice of ${\delta }_1=100$ effectively balances proximity incentives, as evidenced by a reduction in the AR of approximately 30% when higher values ranging from 500 to 1000 are employed. Lower values ranging from 50 to 95 result in diminished PS between $0.7819$ and $0.8495$. The optimal value for the horizontal attitude constraint of the end-effector is determined to be 150 because values less than 10 results in an increase in AE of 12.6%, while values greater than 500 lead to an unnecessary increase in TO by 12.2%. Implementing ${\delta }_3=20$ has been shown to result in a reduction of AE ranging from 8.8% to 27.5% compared to configurations lacking weighting or those with excessive weighting above 1000.

4.3.2. Hyperparameter Analysis of GFWN

As shown in Figure 9a,b, our architectural analysis reveals critical insights into parameter selection for the GFWN framework. The group number K demonstrates a non-monotonic relationship with model performance, where $K=4$ emerges as the optimal configuration. This setting captures sufficient environmental complexity without introducing the detrimental effects observed at $K=16$, where excessive subdivision leads to catastrophic overfitting. The superiority of $K=4$ persists throughout the training, showing both higher initial success rates and more stable convergence behavior.

5. Conclusions

Our study presents an efficient and accurate solution for strawberry picking automation. We enhance the TQC method by introducing DL2 and GFWN, enabling efficient migration from simple to complex environments through a transfer strategy. The experimental results demonstrate that DRFW-TQC achieves a significantly higher average picking success rate in simulated environments compared to existing methods, with improved stability through the transfer strategy.

The effectiveness of the proposed method is demonstrated through comprehensive experiments in simulated ridge cultivation environments. Comparative evaluations against approaches show substantial improvements in picking success rates and operational efficiency, particularly in high-density scenarios with multiple target strawberries. These results validate the framework’s potential for practical deployment in agricultural robotics applications, while the methodological innovations contribute to the broader field of reinforcement learning for robotic manipulation tasks.

While the simulation results validate the framework’s technical feasibility, we acknowledge the current limitation regarding real-world validation. Future research will focus on two critical steps for practical deployment: field testing under varying environmental conditions and the systematic evaluation of sim-to-real transfer performance. The simulation environment has been carefully designed with physically accurate parameters, including strawberry morphology and sensor noise models, to facilitate this transition. Subsequent studies will further explore the framework’s applicability to diverse agricultural harvesting tasks beyond strawberry picking.