Experience Extractor for Adaptive Tradeoff Between Exploration and Exploitation in Reinforcement Learning

Abstract

In Reinforcement Learning (RL), the agent cannot distinguish between exploratory and exploitative experience. Not all sequential experiences contribute equally to the agent’s optimization, and the same experience holds different importance at different learning stages. We propose the Extractor for Adaptive Tradeoff Between Exploration and Exploitation (EATBEE), a task-oriented tool that tightly couples with the agent’s current knowledge and enables adaptive knowledge acquisition. We compare the originally sampled data with the task-driven data distribution to clearly illustrate their deviation. Then, we show how EATBEE identifies and extracts beneficial data for the agent. The monotonic improvement policy is theoretically validated under the assumption that the experience trajectory keeps a high degree of trajectory similarity after extraction. EATBEE serves as an independent module that can be seamlessly integrated with most existing RL algorithms. We substantiate the efficacy and practical applicability of the EATBEE method through experiments conducted in both discrete and continuous environments.

1. Introduction

The analysis of sequential data plays a crucial role in Reinforcement Learning (RL), particularly in addressing the tradeoff between the exploration and exploitation in policies [1,2]. Specifically, if the RL model explores excessively, the policy may fail to converge; conversely, insufficient exploration can cause the policy to get trapped in local optima [3,4]. Consequently, the performance of an RL model is significantly influenced by the quality of the sampled data [5,6]. Currently, existing approaches primarily rely on expert data to improve the quality of RL samples, which significantly improves the performance of policy models [7,8,9]. However, such methods require prior knowledge and meticulous manual intervention. The ultimate objective of artificial intelligence (AI) has not only been to imitate human behavior, but also to enable autonomous exploration in unknown domains and to surpass existing human expertise [10,11].

The most representative class of exploration algorithms is based on curiosity-driven exploration. In this paradigm, exploration signals are intrinsically generated from the agent’s state prediction error, thereby decoupling it from the extrinsic reward provided by the environment. Representative methods include the Intrinsic Curiosity Module (ICM) [12], Random Network Distillation (RND) [13], and DreamGym [14]. This category of algorithms has also been extensively leveraged in vision-language models (VLMs) to improve policy performance [15]. Other major categories include methods based on state visitation counts [16,17,18], and state entropy maximization [19,20,21]. We find that exploration algorithms that effectively balance exploration and exploitation are better at boosting policy performance. To this end, we propose a key insight: policy sampling data should maintain an optimal tradeoff between exploration and exploitation. Rather than encouraging broad exploration, our approach extracts experiences that best align with the agent’s learning stage.

Exploration and exploitation in RL have long been extensively studied [22,23,24,25]. In sparse reward environments, which are common in real-world scenarios, a fundamental challenge lies in the tradeoff between exploration and exploitation. Specifically, agents often fail to perceive the direction of task progression, leading to excessive exploration that marginalizes the importance of exploitation data for policy optimization. We evaluate our approach in sparse-reward regimes, where infrequent feedback exacerbates the risks of stochastic exploration and underscores the necessity of an adaptive tradeoff between exploration and exploitation. (Sparse-reward regimes represent the prevalent real-world paradigm of RL, because reward engineering is notoriously intricate and rarely yields fine-grained, reliable shaping. This setting simultaneously offers a transparent testbed: the near-absence of external feedback strips away confounding supervisory signals and allows us to isolate, simplify, and theoretically ground the contribution of our method.) Notably, the term “extensive exploration” in our context refers to local-level exploration, which aligns with many existing studies [26,27,28]. While most prior work emphasizes balancing exploration and exploitation by encouraging agents to explore broader regions of the state space, our approach differs fundamentally. We focus not on expanding the scope of exploration, but rather on identifying and leveraging exploitation data from the sequential experiences [29]. By adaptively capturing the most beneficial data distribution from the original dataset, we aim to improve the policy more effectively while preserving its exploratory capabilities. Therefore, we propose the Extractor for Adaptive Tradeoff Between Exploration and Exploitation (EATBEE), a tool that facilitates a more effective and efficient policy optimization process.

Inspired by dense RL [30], we aim to dynamically adjust the proportion of exploration and exploitation data in sampled trajectories, based on the current policy and the agent’s task-specific epistemic. Specifically, we propose a data truncation mechanism that adaptively balances exploration and exploitation. We posit that effective exploration enhances the agent’s understanding of the underlying task structure, while proper exploitation allows it to perform the task more efficiently. Therefore, when the agent lacks sufficient understanding, we encourage increased exploration. Conversely, when exploration becomes excessive relative to the task knowledge acquired, we shift the focus toward exploitation. Our work differs from dense RL, where algorithms like Deep Q-Network (DQN) [31] are evaluated without explicitly considering the sequential dependencies in trajectory experience. In contrast, we introduce an adjacency network [32] that embeds truncated segments into a shared embedding space, preserving their contextual similarity—a property we refer to as reachability. This ensures that the temporal structure and semantic coherence among sequential data are minimally disrupted. Furthermore, building upon the policy gradient theorem, we theoretically demonstrate that our method guarantees monotonic policy improvement in sparse reward environments. Finally, we evaluate our approach across multiple benchmark tasks and demonstrate that EATBEE outperforms state-of-the-art baselines, validating both its theoretical soundness and practical effectiveness.

The main contributions of this paper are as follows:

1.

We propose an experience extractor, EATBEE, which can be integrated with most RL algorithms.

2.

We establish a theoretical foundation guaranteeing that EATBEE achieves monotonic improvement in sparse reward environments.

3.

We provide an intuitive visualization of the EATBEE mechanism to elucidate its underlying dynamics.

4.

We conduct evaluation experiments in both low- and high-dimensional state and action environments with sparse or dense rewards. The results show that EATBEE outperforms the baseline and exhibits greater robustness and stability.

The novelty of EATBEE lies in (1) extracting task–beneficial experiences instead of expanding exploration; (2) adaptive trajectory pruning with reachability preservation; (3) stage-aware sample selection via double value functions; (4) a plug–and–play, algorithm-independent module.

The remainder of this paper is organized as follows. After motivating our approach through a discussion of the exploration–exploitation tradeoff in Section 1, we review the relevant literature on experience replay and exploration strategies in Section 2. Section 3 then establishes the formal mathematical preliminaries necessary for the subsequent derivation. The core methodology is detailed across three subsequent sections: Section 4 elaborates on the operational principles of EATBEE, including its correlation metrics and a proof of monotonic performance improvement; Section 5 discusses data optimization under reachability constraints; and Section 6 describes how EATBEE adaptively refines sampled data to enhance policy training quality. These theoretical contributions are followed by Section 7, which provides comprehensive experimental validation, ranging from data distribution analysis to policy performance benchmarks and transferability tests. Finally, Section 8 concludes the paper with a summary of our findings.

2. Related Work

2.1. Exploration and Exploitation

The tradeoff between exploration and exploitation remains one of the foundational challenges in RL [33]. Existing RL policies often suffer from a lack of sufficient exploration, frequently leading them to converge to suboptimal policies [34]. The direct approach is to maximize policy entropy [35,36]. Specifically, the main goal of RL exploration is to visit unknown states, not merely to cover the entire state space. This realization has spurred the development of various auxiliary components designed to enhance a policy’s visitation of these unknown states. For instance, ICM constructs an intrinsic reward based on the prediction error of the next state to encourage the agent to explore novel state regions [37]. In addition to curiosity-driven exploration, methods like state visitation counts [38,39] and state entropy maximization [40] all primarily aim to facilitate the agent’s access to unknown states. This naturally raises a question: are these unknown states necessarily relevant information gains for the current task? To address this, we propose EATBEE, a novel data-driven method for the adaptive tradeoff between exploration and exploitation in a specific-task context. Its primary advantage lies in its ability to capture the underlying task distribution trend of the task and dynamically adjust the tradeoff strategy.

2.2. Experience Replay

The tradeoff between exploration and exploitation aims to obtain high-quality data for effective policy learning. Experience Replay (ER) [31] improves sample efficiency by uniformly sampling transitions from an experience buffer. However, it does not meet the requirements of the evolving policy. Prioritized Experience Replay (PER) [41] increases the sampling probability of transitions with higher temporal-difference error. This mechanism guides the agent to revisit informative experiences and explore less-visited regions of the state space. However, excessive exploration is inefficient in unknown regions, as many areas of the state space may be irrelevant to the specific task. Modern AI systems are designed to rapidly obtain task-relevant knowledge and outperform humans in task completion efficiency. An experience replay method based on frequency and similarity (FSER) [42] was proposed, which prioritizes transitions that are rarely sampled or highly relevant to the current policy, to alleviate imbalanced sampling. FSER and our approach, though conceptually related, pursue distinct functional goals. FSER focuses primarily on alleviating catastrophic forgetting; by contrast, we endeavor to balance the inherent tradeoff between exploration and exploitation through optimized experience selection, improving data quality.

Our research is primarily inspired by dense RL [30], which performs backpropagation within specific regions of the state space. However, it remains agnostic to the agent’s knowledge requirements, thereby lacking task-driven adaptability. Furthermore, we posit that not all states within a trajectory contribute equally to backpropagation or policy updates. By combining FSER and dense RL, our research aims to achieve an adaptive tradeoff between exploration and exploitation, enabling an agent to capture task-relevant knowledge from trajectories. EATBEE fundamentally differs from existing exploration–exploitation approaches in three key ways: (1) Instead of encouraging broader state-space exploration (e.g., curiosity, count, entropy maximization), EATBEE extracts task-aligned experiences from sampled trajectories to adaptively balance exploration and exploitation; (2) Unlike PER and FSER, which focus on sampling weights, EATBEE performs trajectory-level structured pruning while preserving Markovian reachability; (3) Different from dense RL, EATBEE uses policy-aware adaptive thresholds (double value functions, knowledge coefficient and latent state space) to match the agent’s learning stage, rather than fixed backpropagation regions.

3. Preliminaries

We consider the RL problem within the framework of Markov Decision Processes (MDPs). MDPs are usually defined by the tuple $<\mathcal{S},\mathcal{A},P,r,\gamma >$, where $\mathcal{S}$ is a state space, $\mathcal{A}$ is an action space, $P:\mathcal{S}\times \mathcal{A}\to \mathcal{S}$ is a state transition function, $r:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to \mathbb{R}$ is an immediate reward function, and $\gamma \in [0,1)$ is the discount factor. The objective of RL is to maximize the expected cumulative return:

$$J\left(\theta \right)={\mathbb{E}}_{{\pi }_{\theta }}\left[G\left(\tau \right)\right|s_0],$$

given the state $s_0$ under the policy ${\pi }_{\theta }\left(a\right|s)$ parameterized by $\theta$, where $G\left(\tau \right)={\sum }_{t=0}^{\infty }{\gamma }^t{r}_t$ and $\tau$ is the trajectory of the agent’s sampling. The agent samples a sequential experience trajectory

$$\tau =\{(s_0,a_0,r_0),(s_1,a_1,r_1),\cdots ,(s_T,a_T,r_T)\}{|}_{T\ge 0},$$

where the transition $(s_i,a_i,r_i)$ depends on the previous experience

$$\{(s_0,a_0,r_0),\cdots ,(s_{(i-1)},a_{(i-1)},r_{(i-1)})\}{|}_{i\ge 1}.$$

According to the policy gradient theorem, the gradient of the objective $J\left(\theta \right)$ is given by

$${\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{{\pi }_{\theta }}\left[\sum _{t=0}{\nabla }_{\theta }log{\pi }_{\theta }\left(a_t\right|s_t)G\left(\tau \right)\right].$$

Then, $\theta$ is updated as $\theta \leftarrow \theta +\alpha {\nabla }_{\theta }J\left(\theta \right)$, where $\alpha$ is the learning rate. To reduce the variance in the policy gradient, we introduce the state-value function $V\left(s\right)={\mathbb{E}}_{{\pi }_{\theta }}\left[G\left(\tau \right)\right|s]$ and the action-value function $Q(s,a)={\mathbb{E}}_{{\pi }_{\theta }}\left[G\left(\tau \right)\right|s,a]$, parameterized by ${\varphi }_V$ and ${\varphi }_Q$, respectively.

4. The Extractor of Sequentially Sampled Data

4.1. Sequential Experience Data Extraction

Let $\pi$ denote a stochastic policy that induces a trajectory $\tau$ over a finite time horizon T, defined by

$$\tau =\{(s_0,a_0,r_0),(s_1,a_1,r_1),\cdots ,(s_T,a_T,r_T)\}\in {(\mathcal{S}\times \mathcal{A}\times \mathbb{R})}^{T+1}.$$

We define a latent state space $\mathcal{X}$, and assume the existence of a forward mapping function $\mathcal{W}:\mathcal{S}\to \mathcal{X}$ that maps observations to the latent space, along with an inverse mapping $\mathcal{M}:\mathcal{X}\to \mathcal{S}$ such that $s=\mathcal{M}\left(\mathcal{W}\right(s\left)\right)$ for any observation s.

The latent state space $\mathcal{X}$ we construct exhibits structural properties analogous to those of the adjacency matrix space [32]. We introduce a hyperparameter c to indicate the agent’s capability to reach a specific state within c steps. A detailed description is provided in Section 5. By mapping a complete trajectory $\tau$ into the latent state space $\mathcal{X}$, i.e., ${\tau }_{\mathcal{X}}=\mathcal{W}\left(\tau \right)$, we construct an observation correlation matrix $\mathcal{C}\in {\mathbb{R}}^{(T+1)\times (T+1)}$ that encodes pairwise relationships among latent states along the trajectory. We also define a correlation threshold $\kappa$ (in our work, we set $\kappa =0.99$ by default). One effective application is to compute the correlation between any two arbitrary latent variables within the latent state space $\mathcal{X}$, aiming to identify the optimal data distribution along a trajectory. Its core advantage lies in improving data quality while preserving both the Markov property and the reachability of states. For the data $x_{\mathcal{X}}^j$ following the latent state $x_{\mathcal{X}}^i$ where $j>i$, if $corr(x_{\mathcal{X}}^i,x_{\mathcal{X}}^j)>\kappa =0.99$, then the segment $\{s^i,\cdots ,s^{j-1}\}$ can be eliminated, where $corr$ is the correlation calculation function. In our work, $corr$ is set as the Pearson correlation coefficient function, given by

$${\rho }_{X,Y}={\displaystyle \frac{cov(X,Y)}{{\sigma }_X{\sigma }_Y}}={\displaystyle \frac{\mathbb{E}\left[(X-{\mu }_X)(Y-{\mu }_Y)\right]}{\sqrt{\mathbb{E}\left[{(X-{\mu }_X)}^2\right]}\sqrt{\mathbb{E}\left[{(Y-{\mu }_Y)}^2\right]}}},$$

(1)

where $cov(X,Y)$ is the covariance, and $\mu$ and $\sigma$ denote the mean and standard deviation, respectively. Correlation is adopted as a pruning criterion for its efficiency in measuring state redundancy and preserving trajectory structural consistency. We emphasize that correlation is a choice for discrete environments, not a mandatory core of EATBEE. In continuous spaces, we switch to value-based significance filtering. The extraction is valid only when both similarity and reachability are satisfied, ensuring the Markov property and task reward are preserved. Correlation can be replaced by cosine similarity or Euclidean distance without breaking the framework. The conceptual framework of EATBEE is illustrated in Figure 1.

The EATBEE extracts the outcome from $\tau$:

$${\tau }^{EA}=\left\{(s_0^{EA},a_0^{EA},r_0^{EA}),(s_1^{EA},a_1^{EA},r_1^{EA}),\cdots ,(s_n^{EA},a_n^{EA},r_n^{EA})\right\}{|}_{n\le T},$$

where ${\tau }^{EA}$ represents the new trajectory obtained by processing $\tau$. Although $(s_i,a_i,r_i)=(s_i^{EA},a_i^{EA},r_i^{EA})$ may not hold in general, $r_T=r_n^{EA}$ must be satisfied; that is, the reward at the final step must be equal because the extractor must ensure that the same final state is reached. Consequently, the extractor only performs improvements on the intermediate steps of a trajectory. We define the cumulative return along trajectory $\tau$ at time t as $G_t={\sum }_{i=t}^T{\gamma }^{i-t}{r}_i$, and the cumulative return of the EATBEE-augmented trajectory ${\tau }^{EA}$ is given by $G_t^{EA}\left(\tau \right)={\sum }_{i=t}^n{\gamma }^{i-t}{r}_i^{EA}$, where $t<n\le T$. We observed that, in a continuous state space, the correlation matrix is computationally challenging to estimate, which prevents the experience extractor from achieving an adaptive tradeoff between exploration and exploitation. Therefore, we explored an alternative method for expansion. Details on how to obtain ${\tau }^{EA}$ in a continuous observation space are provided in Section 6.

4.2. EATBEE Conditional Monotonic Improvement Theorem

To quantify the effectiveness of EATBEE in improving data quality, we introduce a metric called the return difference ratio.

Definition 1 (Return Difference Ratio).

Given a trajectory τ under policy π, let $G_{\pi }\left(\tau \right)$ and $G_{\pi }^{EA}\left(\tau \right)$ denote the returns under the original and EATBEE-augmented trajectories, respectively. The return difference ratio is defined as

$$\delta ={\displaystyle \frac{G_{\pi }\left(\tau \right)}{G_{\pi }^{EA}\left(\tau \right)}}.$$

(2)

Thus, $\delta$ can be expressed as follows:

$$\delta ={\displaystyle \frac{G_t\left(\tau \right)}{G_t^{EA}\left(\tau \right)}}={\displaystyle \frac{{\sum }_{i=t}^T{\gamma }^{i-t}{r}_i{|}_{t<n\le T}}{{\sum }_{i=t}^n{\gamma }^{i-t}{r}_i^{EA}{|}_{t<n\le T}}}.$$

(3)

The metric reveals the following insights: (1) during the initial learning phase of the policy, $\delta \to 0$, indicating EATBEE achieves significant performance improvement; and (2) when the agent successfully completes the task, as indicated by $\delta \to 1$, the improvement effect gradually diminishes.

Suppose the agent receives a fixed reward e at each step, and the reward for successfully completing the task is denoted by $r_T$. After applying EATBEE, the number of steps along a trajectory $\tau$ with a finite horizon T is reduced to $T-k$, where $0\le k<T$. Consequently, the returns $G_t\left(\tau \right)$ and $G_t^{EA}\left(\tau \right)$ can be expressed, respectively, as follows:

$$G_t\left(\tau \right)={\displaystyle \sum _{i=t}^T}{\gamma }^{i-t}{r}_i=e+\gamma e+{\gamma }^{2}e+\cdots +{\gamma }^{T-t}{r}_T,$$

$$G_t^{EA}\left(\tau \right)={\displaystyle \sum _{i=t}^{T-k}}{\gamma }^{i-t}{r}_i^{EA}=e+\gamma e+{\gamma }^{2}e+\cdots +{\gamma }^{T-k-t}{r}_{T-k}^{EA}.$$

Clearly, when $k=0$, we have $G_t\left(\tau \right)=G_t^{EA}\left(\tau \right)$. If the environment provides sparse rewards (i.e., $e=0$), we obtain

$$\delta ={\displaystyle \frac{G_t\left(\tau \right)}{G_t^{EA}\left(\tau \right)}}={\displaystyle \frac{{\gamma }^{T-t}{r}_T}{{\gamma }^{T-k-t}{r}_{T-k}^{EA}}}={\gamma }^k(0\le \delta \le 1),$$

(4)

The intuition of $\delta$ is to quantify the scale-up of the return, i.e., $G_t^{EA}={\displaystyle \frac{G_t}{{\gamma }^k}}$. However, $\delta$ is not a fixed value—it is dynamically determined by EATBEE. We observe that the improvement in EATBEE grows exponentially.

Our method is based on a key insight: the policy $\pi =\{{\pi }_0,\cdots ,{\pi }_T\}$ can be viewed as a sequence of sub-policies, where the future sub-trajectory $\{s_t,\cdots ,s_T\}$ is generated by a sub-policy ${\pi }_t$. Since ${\pi }_t$ is a stochastic policy, the same sub-policy may lead to different state transitions. Consequently, in a trajectory $\tau =\{s_0,\cdots ,s_T\}$, the new trajectory ${\tau }^{EA}=\{s_0,\cdots ,s_{i-1},s_j,\cdots ,s_T\}$ extracted by EATBEE may result from the sub-policy ${\pi }_{i-1}$. Thus, we introduce a lemma on the policy set for a trajectory $\tau$.

Lemma 1 (Policy Distribution Mapping).

Given a trajectory $\tau =\{s_0,s_1,\cdots ,s_T\}$, the sub-trajectory $S_i=\{s_i,s_{i+1}\cdots ,s_T\}$ is generated by the sub-policy ${\pi }_i$ of the policy $\pi =\{{\pi }_0,{\pi }_1,\cdots ,{\pi }_T\}$.

The goal of EATBEE is to obtain the optimal policy,

$${\pi }^{\mathsf{*}}=\{{\pi }_0^{\mathsf{*}},{\pi }_1^{\mathsf{*}},{\pi }_2^{\mathsf{*}},\cdots ,{\pi }_{T-k}^{\mathsf{*}}|0\le k<T\},$$

along a trajectory $\tau$. Corresponding to ${\pi }^{\mathsf{*}}$, the optimal trajectory is defined as ${\tau }^{\mathsf{*}}=\{s_0,s_1,\cdots ,s_{T-k}|0\le k<T\}$ under the same condition. Therefore, the optimal policy ${\pi }^{\mathsf{*}}$ can be derived from the optimal trajectory ${\tau }^{\mathsf{*}}$, as illustrated in Figure 2.

The new objective function after experience extraction can be formally defined as follows, given the state s:

$$J^{EA}\left(\theta \right)=E_{{\pi }_{\theta }}\left[G^{EA}\left(\tau \right)\right|s]$$

(5)

A theorem [43,44] can be used to evaluate the performance improvement of the policy, and is formally stated as follows.

Theorem 1 

(Performance Difference Theorem [43,44]). For any two policies ${\pi }^{\prime }$ and π, the performance difference

$$J_{▵}({\pi }^{\prime },\pi )\triangleq J\left({\pi }^{\prime }\right)-J\left(\pi \right)$$

can be measured using the advantage function

$$J_{▵}({\pi }^{\prime },\pi )=E_{\tau \sim P_{{\pi }^{\prime }}\left(\tau \right)}\left[{\displaystyle \sum _{t=0}^T}{\gamma }^t{A}_{\pi }(s_t,a_t)\right]=E_{s\sim P_{{\pi }^{\prime }}(·),a\sim {\pi }^{\prime }(·|s)}\left[A_{\pi }(s,a)\right]$$

(6)

Based on Theorem 1 and Equation (5), we derived the EATBEE Monotonic Improvement Theorem as follows.

Theorem 2 (EATBEE Conditional Monotonic Improvement Theorem).

In a sparse reward environment, where routine actions yield a zero immediate reward ($r=0$) and the terminal action (task success or failure) yields a non-zero reward ($r_T\ne 0$), the monotonic improvement of EATBEE can be formally stated as follows:

$$J_{▵}({\pi }^{\prime },\pi )=J^{EA}\left(\theta \right)-J\left(\theta \right)=(1-\delta )J^{EA}\left(\theta \right)\ge 0$$

(7)

where $\delta ={\gamma }^k$, and k is a positive integer. The proof is provided in Appendix A.

If $J_{▵}({\pi }^{\prime },\pi )>0$, this indicates that the updated policy ${\pi }^{\prime }$ outperforms the original policy $\pi$. Therefore, maximizing $J_{▵}({\pi }^{\prime },\pi )$ leads to an improved policy ${\pi }^{\prime }$ that is guaranteed to perform at least as well as, and potentially better than, the current policy $\pi$. In this context, the term $J^{EA}\left(\theta \right)$ is equivalent to $J\left({\pi }^{\prime }\right)$, representing the performance of the updated policy. In a sparse reward environment, regardless of whether the task completion explicitly satisfies $J^{EA}\ge J\left(\theta \right)$, it still holds that ${\displaystyle \frac{J^{EA}\left(\theta \right)}{J\left(\theta \right)}}={\gamma }^k$. Meanwhile, the backpropagation mechanism of EATBEE is illustrated in Figure 3. We do not claim universal monotonic improvement for all scenarios. Instead, EATBEE provides conditionally monotonic policy improvement under sparse rewards, high trajectory similarity, and valid reachability constraints.

Let the distribution of the sampled data be denoted by $f\left(x\right)$, the optimal data distribution by $g\left(x\right)$, and the data deviation distribution by $h\left(x\right)$. These distributions are related by $g\left(x\right)=f\left(x\right)-h\left(x\right)$. The unknown data distribution ratio can be expressed as $\varphi ={\displaystyle \frac{f\left(x\right)}{g\left(x\right)}}={\displaystyle \frac{1}{1-ϵ}}$, where $ϵ={\displaystyle \frac{h\left(x\right)}{f\left(x\right)}}$ represents the relative deviation from the sampled distribution. The characteristics of these data distributions are illustrated in Figure 4. EATBEE effectively mitigates the issue of excessive exploration and extracts a task-driven data distribution from the sampled data, thereby addressing the fundamental challenge of balancing exploration and exploitation.

To preserve the generality of EATBEE, we implement the adaptive tradeoff with respect to the return $G\left(\tau \right)$, rather than the immediate reward r. If a state $s_t$ is excluded by EATBEE, the corresponding return at that timestep is set to $G_t^{EA}\left(\tau \right)=0$, as illustrated in Figure 5. Furthermore, EATBEE improves policy learning and stabilizes the value functions $Q(s,a)$ and $V\left(s\right)$ by reducing the learning burden on the neural network.

5. Trajectory Similarity

The experience extractor requires maintaining a high degree of similarity between trajectories $\tau$ and ${\tau }^{EA}$. We introduce a sequential data similarity metric, termed sequential data similarity.

Definition 2 (Sequential Data Similarity).

Consider sequential data $X=\{x_1,x_2,\cdots ,x_n\}$, where $x_i\in {\mathbb{R}}^m(m>0)$. For any two sequences $X_1$ and $X_2$, given a threshold $\xi \in [0,1]$, and $\exists N\in Z^{+}>0$ such that $f(X_1,X_2)>\xi$ iff $n>N$. Furthermore, $\underset{n\to \infty }{lim}f(X_1,X_2)\to 1$.

Evaluating the similarity between two sequences is significant, as it effectively preserves the inherent information. However, computing similarity over entire sequential data may incur substantial computational costs. To mitigate this, we observe that if a sequence X exhibits high similarity with and without a fragment $X_m=\{x_{i+1},\cdots ,x_j\}$, where $X_m\subseteq X,0\le i<j$, this indicates $x_i\approx x_j$. The sequence extracted by EATBEE is denoted as $X^{EA}$. Therefore, the similarity calculation $f(X,X^{EA})$ between X and $X^{EA}$ can be approximated by $f(x_i,x_j)$, i.e., $f(X,X^{EA})\approx f(x_i,x_j)$. Accordingly, we introduce a simplified similarity evaluation method.

Proposition 1 (Simplified Sequential Data Similarity).

Under Definition 2, the similarity of the sequence X can be denoted as $f(X,X^{EA})\approx f(x_i,x_j)$ with and without a segment $X_m=\{x_i,\cdots ,x_j\}{|}_{1\le i<j}$ where $X_m\subseteq X$, i.e., $x_i\approx x_j$.

Under Proposition 1, given a trajectory $\tau =\{s_0,s_1,\cdots ,s_T\}$, if $s_i\approx s_j$, then the similarity $f(\tau ,{\tau }^{EA})$ can be approximated by $f(s_i,s_j)$, i.e., $f(\tau ,{\tau }^{EA})=f(s_i,s_j)$. However, ${\tau }^{EA}$ suffers from a limitation: the sequential reachability of ${\tau }^{EA}$ may not be satisfied. This implies that, while the similarity condition $f(\tau ,{\tau }^{EA})\approx f(s_i,s_j)>\xi$ holds, the extracted trajectory ${\tau }^{EA}$ does not necessarily satisfy the Markov property in the original sequential data. To address this issue, we introduce the sequential data reachability metric.

Definition 3 (Sequential Data Reachability).

For any sequential data $X=\{x_1,x_2,\cdots ,x_n\}$ in the sequential data space Γ, adjacent elements must satisfy reachability, i.e., $x_{i-1}\to x_i$. Given a threshold η, adjacent elements must satisfy $h(x_{i-1},x_i)\le \eta$, where $h(·,·)$ is a distance function.

According to Definition 3, given sequential data $X=\{x_0,x_1,\cdots ,x_n\}$, for any elements $x_i$ and $x_j$, the transition $x_i\to x_{j+1}$ is considered reachable iff $h(x_i,x_{j+1})\le h(x_j,x_{j+1})\le \eta$. Under Proposition 1 and Definition 3, the sequential data similarity can be formalized as follows:

$$f(X,X^{\prime })\approx f(x_i,x_j)s.t.\left\{\begin{array}{c}i<j\\ h(x_j,x_{j+1})\ge h(x_i,x_{j+1})\end{array}\right.,$$

(8)

where the measure $h(x_i,x_j)$ is typically the Euclidean distance:

$$h(x_i,x_j)=\left|\right|x_i-x_j{\left|\right|}_2$$

(9)

For a trajectory $\tau =\{s_0,s_1,\cdots ,s_T|T>0\}$, the similarity can be formalized as follows within the reachability:

$$f(\tau ,{\tau }^{EA})\approx f(s_i,s_j)>\xi s.t.\left\{\begin{array}{c}1\le i<j\\ h(s_j,s_{j+1})\ge h(s_i,s_{j+1})\end{array}\right.$$

(10)

Under a deterministic constraint, directly computing the similarity along a trajectory $\tau$ is computationally expensive and often infeasible. To address this, we construct a latent space $\mathcal{X}$ to compute similarity for $\tau$, inspired by the adjacency matrix [32]. We define a mapping function $\mathcal{W}:\mathcal{S}\to \mathcal{X}$ and a reverse mapping function $\mathcal{M}:\mathcal{X}\to \mathcal{S}$, parameterized by ${\theta }_{\mathcal{W}}$ and ${\theta }_{\mathcal{M}}$, respectively. To satisfy reachability within c steps in the latent space $\mathcal{X}$, the loss function for the parameter ${\theta }_{\mathcal{W}}$ is defined as

$$Los{s}_{\mathcal{X}}=y·\left(h(·,·)-l_1\right)+(1-y)·\left(l_2-h(·,·)\right)$$

(11)

where $l_1>0$ and $l_2>l_1$ are constants defining the margin, and $y\in \{0,1\}$ is a binary label indicating reachability (1 if reachable, 0 otherwise). We embed the reachability constraint directly into the latent state space $\mathcal{X}$.

6. Adaptive Tradeoff

We introduced the experience extractor, which improves the data quality and policy learning by leveraging correlations under similarity and reachability conditions. However, directly using correlation presents two problems: (1) we need to manually remove the fragment $X_m=\{x_{i+1},\cdots ,x_j\}$, then concatenate $\{x_0,\cdots ,x_i\}$ with $\{x_{j+1},\cdots ,x_T\}$ when $x_i\approx x_j$, and (2) correlation cannot capture the adaptive tradeoff. This approach is accurate and theoretically sound, yet it is computationally expensive. And the experience extractor exhibits suboptimal performance in continuous state tasks and lacks adaptability. We define ‘adaptability’ specifically as the strategic alignment of data requirements with the current policy’s learning stage. Specifically, the agent should prioritize extensive explorative data during the initial phases rather than focusing solely on exploitative data. As the policy gains task-specific knowledge, the data distribution should shift to maintain a high degree of exploitation while retaining a calibrated level of exploration. We developed an adaptive tradeoff approach to address the aforementioned issues. We utilize Proximal Policy Optimization (PPO) [45] as our baseline RL algorithm, with the objective function defined as follows:

$$\mathcal{L}\left({\varphi }_V\right)=\mathbb{E}\left[{\left(V_{{\varphi }_V}^{\pi }\left(s_t\right)-G_t\left(\tau \right)\right)}^2\right]$$

(12)

$$\mathcal{L}\left(\theta \right)=\mathbb{E}\left[min\left({\displaystyle \frac{{\pi }_{\theta }(·|s)}{{\pi }_{old}\left(a\right|s)}}{A}_{{\pi }_{old}}(s,a),clip\left({\displaystyle \frac{{\pi }_{\theta }(·|s)}{{\pi }_{old}\left(a\right|s)}},1-2ϵ,1+2ϵ\right)A_{{\pi }_{old}}(s,a)\right)\right]$$

(13)

where ${\pi }_{old}$ represents the behavior policy from the previous iteration. The pseudocode for the EATBEE algorithm is presented in Algorithm 1. The actual performance is shown in Figure 6. In this scenario, there exists an expert who efficiently filters the data while preserving the MDP property, as expressed in Equations (4) and (7).

We posit that the adaptive tradeoff serves as the mediator between the policy and the task. We construct a parameterized function for the experience extractor. This function enables automated data pruning, thereby achieving an adaptive tradeoff between exploration and exploitation. Specifically, we design a neural network architecture, the state selection mechanism, to dynamically identify and select high-quality and informative data segments along the current policy $\pi$ within the latent state space $\mathcal{X}$.

Algorithm 1 EATBEE

1: Initialize: agent, replay buffer $\mathcal{D}$.   2: while not converged do   3:     Agent collects trajectories $\tau \to \mathcal{D}$.   4:     if episode terminates then   5:         Sample all trajectories $\tau \sim \mathcal{D}$.   6:         Update the adjacency latent space using Equation (11).   7:         ${\tau }^{EA}\leftarrow \tau$ using Equation (1).   8:         Update agent with ${\tau }^{EA}$.   9:     end if 10: end while

It is crucial to note that EATBEE is designed to be continuously adaptive to the evolving current policy $\pi$. This implies that the state selection mechanism must dynamically align with $\pi$ at all times. The most straightforward form of EATBEE is the value function. To formalize the update process, we introduce the concepts of the significant point and significant baseline to refine the state selection mechanism, as detailed in Definition 4.

Definition 4 (Significance Evaluation).

Given a trajectory

$$X=\{x_1,x_2,\cdots ,x_T\},$$

an element $x\in X$ is called a significant point if $f\left(x\right)>0$, where the function $f\left(x\right):X\to \mathbb{R}$ is referred as the significant baseline.

Considering a trajectory

$$\tau =\{(s_0,a_0,r_0),(s_1,a_1,r_1),\cdots ,(s_T,a_T,r_T)\}$$

collected by the current policy π, if a state–action pair $(s,a)$ satisfies

$$f(s,a)=Q^{true}(s,a)-V^{\pi }\left(s\right)>0,$$

(14)

then it is considered a significant point, where $f(·)$ serves as the significant baseline for π, $Q^{true}(s,a)$ is the true state–action value, and $V^{\pi }\left(s\right)$ is the predictive state value.

To mitigate the overestimation bias in value functions, which could lead to a severely distorted knowledge distribution, we employ a double state-value mechanism. This approach is conceptually analogous to Double Q-learning. We approximate the true value $Q^{true}(·,·)$ using the discounted cumulative returns from Monte Carlo rollouts. The knowledge filtering process of the approximated EATBEE is formulated as follows:

$${\tau }^{EA}\leftarrow \tau \left[\left(G\left(\tau \right)-min(V_{{\varphi }_{V_1}},V_{{\varphi }_{V_2}})\times \eta \right)>0\right]$$

(15)

where $\eta$ is the knowledge coefficient, which is empirically set to $\eta =0.8$ by default. A key advantage of EATBEE is its simplicity and seamless integration, requiring minimal modification to existing training pipelines. The pseudocode for the approximated EATBEE is detailed in Algorithm 2.

Why does Equation (15) enable an adaptive tradeoff between exploration and exploitation relative to the current policy? The residual between $G\left(\tau \right)$ and V serves as a critical evaluative metric for action quality, effectively capturing the advantage of specific decisions. As the agent approaches the task goal, the state-value function exhibits a monotonic increase, thereby refining the data extraction threshold. This consistency is illustrated in Figure 7.

Algorithm 2 Approximated EATBEE

1: Initialize: EATBEE with double state-value functions parameterized by ${\varphi }_{V_1},{\varphi }_{V_2}$, replay buffer $\mathcal{D}$, and knowledge coefficient $\eta$.   2: while not converged do   3:     Agent collects trajectories $\tau \to \mathcal{D}$.   4:     if episode terminates then   5:         Sample all trajectories $\tau \sim \mathcal{D}$.   6:         Update the adjacency latent space using Equation (11).   7:         Update ${\varphi }_{V_1},{\varphi }_{V_2}$ using Equation (12).   8:         ${\tau }^{EA}\leftarrow \tau$ using Equation (15).   9:         Update agent with ${\tau }^{EA}$. 10:     end if 11: end while

7. Experiment

We aim to answer the following questions: (1) Can EATBEE effectively achieve a tradeoff between exploration and exploitation within a trajectory? (2) How does EATBEE’s performance compare to state-of-the-art algorithms? (3) How does EATBEE capture the environmental knowledge distribution and enable effective transfer learning? (4) How do the fundamental components of EATBEE contribute to its overall performance?

7.1. Experimental Setup

We conducted experiments in both discrete and continuous environments. The discrete environments are illustrated in Figure 8, including LunarLander [46] and MiniGrid-DoorKey [47]. The continuous environments are MuJoCo [48], including Walker2d, Hopper, and Ant, as illustrated in Figure 9. We adopted five distinct random seeds for each experimental setting and trained the model for 20,000 episodes under each seed configuration. The total reward and step count recorded in each episode were taken as the per-episode evaluation metrics. The final performance results were obtained by averaging the corresponding metric values across all five random seeds. Meanwhile, the raw experimental results were smoothed during post-processing. Specifically, the average value was calculated for every consecutive 50 episodes, and data points were sampled at an interval of 20 steps to generate the evaluation results presented for each episode. The experimental hyperparameters are listed in

Table 1. Hyperparameter details.

	PPO	DQN
Optimizer	Adam [49]	Adam
learning rate	$3\times {10}^{-4}$	$3\times {10}^{-4}$
discount $\left(\gamma \right)$	$0.99$	$0.99$
Replay Buffer Size	None	$1\times {10}^6$
Number of Hidden Layers	2	2
Number of Hidden Units Per Layer	64	64
Number of Samples Per Minibatch	None	256
Nonlinearity	ReLU	ReLU

.

7.2. Distribution Analysis

To provide a clear explanation of EATBEE, we visualize the $\tau$ and ${\tau }^{EA}$ distributions using Principal Component Analysis (PCA), transforming the data from ${\mathbb{R}}^{d_s}$ to $\mathbb{R}$. To enhance a comprehensive understanding of the tradeoff between exploration and exploitation using EATBEE, we selected three types of state data distributions: (1) exploration with task failure (Type 1)—the agent takes up to 500 steps but fails to complete the task; (2) exploration with task success (Type 2)—the agent takes more than 200 steps and successfully completes the task; (3) exploitation with task success (Type 3)—the agent takes 50 steps or fewer and completes the task. The data distributions of the trajectories are randomly selected, as illustrated in Figure 10.

In Type 1, the agent performs excessive exploration, and EATBEE filters out most of the exploratory data. In Type 2, the agent performs finite exploration and successfully completes the task, and EATBEE retains most of the exploitative data. In Type 3, the agent has fully completed the task, and EATBEE retains all sampled data. The visualization demonstrates that EATBEE dynamically adjusts the distribution of $\tau$, achieving a tradeoff between exploration and exploitation.

7.3. Performance Evaluation of EATBEE

We compare EATBEE against the baseline algorithm (PPO). First, we introduce a selective indicator for EATBEE, termed Ratio, which quantifies the proportion between ${\tau }^{EA}$ and $\tau$:

$$Ratio={\displaystyle \frac{{\mathbb{E}}_{\pi }\left[\right|{\tau }^{EA}\left|\right]}{{\mathbb{E}}_{\pi }\left[\right|\tau \left|\right]}}.$$

(16)

where $|·|$ represents the length of a sequence.

The performance of EATBEE in the MiniGrid-DoorKey-8 × 8 environment is illustrated in Figure 11 and

Table 2. The performance comparison in the MiniGrid-DoorKey-8 × 8 environment.

	Min Step	Max Step	10% MS	20% MS	30% MS	40% MS	50% MS
PPO	$15.2\pm 2.48$	500	$134.12\pm 123.44$	$93.69\pm 58.24$	$80.27\pm 40.03$	$75.75\pm 31.48$	$74.62\pm 27.38$
PPO + LSTM	$15.2\pm 3.66$	500	$43.81\pm 4.17$	$47.44\pm 6.69$	$50.12\pm 11.5$	$50.40\pm 13.85$	$52.18\pm 18.13$
PPO + ICM	$143.4\pm 180.72$	500	$229.47\pm 220.91$	$232.66\pm 218.53$	$242.38\pm 211.11$	$242.94\pm 210.63$	$251.11\pm 204.80$
PPO + EATBEE	$14.8\pm 1.72$	500	$\mathbf{35}.\mathbf{68}\pm \mathbf{8}.\mathbf{45}$	$\mathbf{37}.\mathbf{53}\pm \mathbf{8}.\mathbf{98}$	$\mathbf{37}.\mathbf{50}\pm \mathbf{6}.\mathbf{25}$	$\mathbf{37}.\mathbf{62}\pm \mathbf{6}.\mathbf{89}$	$\mathbf{38}.\mathbf{14}\pm \mathbf{6}.\mathbf{80}$
PPO + LSTM + EATBEE	$\mathbf{14}.\mathbf{6}\pm \mathbf{3}.\mathbf{38}$	500	$41.74\pm 7.40$	$39.37\pm 4.62$	$40.27\pm 4.68$	$40.49\pm 3.78$	$42.49\pm 3.83$
DQN + ER	$412.0\pm 176.0$	500	$499.86\pm 0.28$	$499.85\pm 0.28$	$499.83\pm 0.32$	$499.84\pm 0.30$	$499.84\pm 0.30$
DQN + PER	$413.8\pm 172.4$	500	$499.39\pm 0.91$	$499.43\pm 0.82$	$499.39\pm 0.88$	$499.39\pm 0.89$	$499.43\pm 0.83$

. To further evaluate scalability and robustness, we extend the MiniGrid-DoorKey environment from an 8 × 8 grid to a more challenging 16 × 16 setting, and the results are illustrated in Figure 12 and

Table 3. The performance comparison in the MiniGrid-DoorKey-16 × 16 environment.

	Min Step	Max Step	10% MS	20% MS	30% MS	40% MS	50% MS
PPO	339.4 ± 198.52	500	496.47 ± 4.63	496.89 ± 3.28	496.77 ± 2.55	497.42 ± 2.05	497.79 ± 1.71
PPO + LSTM	$175.6\pm 163.00$	500	$388.34\pm 95.81$	$393.83\pm 96.07$	$403.21\pm 86.66$	$418.68\pm 70.56$	$431.58\pm 56.64$
PPO + ICM	$406.0\pm 188.00$	500	$497.95\pm 4.07$	$497.46\pm 5.07$	$497.96\pm 4.08$	$498.47\pm 3.06$	$498.77\pm 2.45$
PPO + EATBEE	$39.4\pm 7.00$	500	$\mathbf{143}.\mathbf{38}\pm \mathbf{75}.\mathbf{64}$	$\mathbf{153}.\mathbf{08}\pm \mathbf{79}.\mathbf{78}$	$\mathbf{167}.\mathbf{46}\pm \mathbf{77}.\mathbf{97}$	$\mathbf{204}.\mathbf{89}\pm \mathbf{69}.\mathbf{16}$	$\mathbf{235}.\mathbf{32}\pm \mathbf{69}.\mathbf{23}$
PPO + LSTM + EATBEE	$\mathbf{39}.\mathbf{0}\pm \mathbf{14}.\mathbf{46}$	500	$195.48\pm 68.74$	$201.83\pm 71.43$	$213.34\pm 65.39$	$234.57\pm 54.09$	$259.54\pm 40.23$
DQN + ER	426.2 ± 147.6	500	499.86 ± 0.28	500.0 ± 0.0	500 ± 0.0	500 ± 0.0	500 ± 0.0
DQN + PER	448.2 ± 103.60	500	499.98 ± 0.05	499.99 ± 0.03	499.99 ± 0.02	499.99 ± 0.02	499.99 ± 0.01

. These results demonstrate that EATBEE achieves better performance and robustness than the baseline in sparse environments, which is consistent with the theoretical foundation of EATBEE. The experiment in a continuous state and discrete action environment, LunarLander, is illustrated in Figure 13 and

Table 4. The performance comparison in the LunarLander environment.

	Min Reward	Max Reward	10% MR	20% MR	30% MR	40% MR	50% MR
PPO	−314.42 ± 101.27	290.33 ± 23.22	230.98 ± 42.04	231.10 ± 42.77	231.19 ± 43.22	230.71 ± 43.78	229.61 ± 43.92
PPO + LSTM	$-365.62\pm 70.73$	$302.93\pm 16.86$	$235.54\pm 28.65$	$234.37\pm 30.91$	$234.86\pm 30.64$	$232.62\pm 30.88$	$230.34\pm 29.68$
PPO + EATBEE	$-\mathbf{302}.\mathbf{37}\pm \mathbf{122}.\mathbf{28}$	$\mathbf{307}.\mathbf{87}\pm \mathbf{20}.\mathbf{62}$	$\mathbf{263}.\mathbf{49}\pm \mathbf{2}.\mathbf{90}$	$\mathbf{264}.\mathbf{06}\pm \mathbf{2}.\mathbf{03}$	$\mathbf{262}.\mathbf{04}\pm \mathbf{3}.\mathbf{58}$	$\mathbf{260}.\mathbf{54}\pm \mathbf{5}.\mathbf{79}$	$\mathbf{255}.\mathbf{87}\pm \mathbf{7}.\mathbf{73}$
PPO + LSTM + EATBEE	$-343.17\pm 132.99$	$300.16\pm 7.06$	$235.72\pm 11.95$	$229.14\pm 16.48$	$224.4\pm 17.07$	$221.19\pm 17.75$	$213.9\pm 19.82$
DQN + ER	−538.21 ± 1313.68	291.78 ± 9.14	108.86 ± 93.92	120.17 ± 99.12	141.25 ± 86.89	155.61 ± 72.13	156.20 ± 57.56
DQN + PER	−787.09 ± 1244.59	287.51 ± 13.34	−78.10 ± 111.93	−57.22 ± 67.68	−54.24 ± 49.29	−47.92 ± 39.76	−37.41 ± 36.66

. EATBEE autonomously discovers the knowledge necessary for the current policy.

MuJoCo presents a significant challenge in continuous control due to its high-dimensional, continuous state and action spaces. We utilize MuJoCo benchmarks to validate that EATBEE can be combined with any algorithm and autonomously distill the essential knowledge for the current policy, with results shown in Figure 14 and

Table 5. The performance comparison in MuJoCo.

	Min Reward	Max Reward	10% MR	20% MR	30% MR	40% MR	50% MR
PPO-Hopper	10.42 ± 3.41	1001.86 ± 981.75	555.89 ± 353.09	520.14 ± 302.63	506.22 ± 282.70	484.76 ± 260.26	463.50 ± 231.01
PPO + EATBEE-Hopper	$\mathbf{14}.\mathbf{22}\pm \mathbf{3}.\mathbf{93}$	$\mathbf{1849}.\mathbf{49}\pm \mathbf{1195}.\mathbf{62}$	$\mathbf{823}.\mathbf{12}\pm \mathbf{342}.\mathbf{79}$	$\mathbf{798}.\mathbf{23}\pm \mathbf{313}.\mathbf{54}$	$\mathbf{773}.\mathbf{11}\pm \mathbf{304}.\mathbf{48}$	$\mathbf{747}.\mathbf{69}\pm \mathbf{283}.\mathbf{99}$	$\mathbf{718}.\mathbf{53}\pm \mathbf{264}.\mathbf{58}$
PPO-Walker	−7.71 ± 9.17	612.83 ± 450.23	291.29 ± 41.78	289.34 ± 43.31	289.58 ± 43.20	290.96 ± 43.87	290.74 ± 46.67
PPO + EATBEE-Walker	$-\mathbf{5}.\mathbf{64}\pm \mathbf{15}.\mathbf{87}$	$\mathbf{738}.\mathbf{39}\pm \mathbf{454}.\mathbf{81}$	$\mathbf{420}.\mathbf{80}\pm \mathbf{99}.\mathbf{10}$	$\mathbf{404}.\mathbf{77}\pm \mathbf{85}.\mathbf{38}$	$\mathbf{388}.\mathbf{16}\pm \mathbf{73}.\mathbf{98}$	$\mathbf{374}.\mathbf{49}\pm \mathbf{64}.\mathbf{83}$	$\mathbf{362}.\mathbf{23}\pm \mathbf{55}.\mathbf{61}$
PPO-Ant	−1183.82 ± 872.45	66.14 ± 46.72	−1103.86 ± 975.20	−982.08 ± 898.34	−897.62 ± 883.52	−857.43 ± 888.60	−834.28 ± 894.73
PPO + EATBEE-Ant	$-\mathbf{68}.\mathbf{00}\pm \mathbf{106}.\mathbf{60}$	$\mathbf{137}.\mathbf{78}\pm \mathbf{67}.\mathbf{84}$	$\mathbf{54}.\mathbf{88}\pm \mathbf{1}.\mathbf{43}$	$\mathbf{44}.\mathbf{48}\pm \mathbf{2}.\mathbf{27}$	$\mathbf{39}.\mathbf{84}\pm \mathbf{2}.\mathbf{13}$	$\mathbf{37}.\mathbf{31}\pm \mathbf{2}.\mathbf{34}$	$\mathbf{35}.\mathbf{34}\pm \mathbf{2}.\mathbf{50}$

.

8. Analysis of Transfer Learning in EATBEE

One of the key features of EATBEE is its ability to identify a data distribution that benefits the current policy from sampled data. A new question arises: in similar or identical task environments, does EATBEE have efficient transfer learning capabilities while still being able to screen environmental sampling data? We validated EATBEE’s transfer learning capability in the MiniGrid-DoorKey environment, as shown in Figure 15. First, we trained EATBEE in the MiniGrid-DoorKey-5 × 5 environment for 2000 epochs to ensure a thorough understanding of the task before progressing to experiments in the MiniGrid-DoorKey-8 × 8 and MiniGrid-DoorKey-16 × 16 environments. Due to the limited information available in the MiniGrid-DoorKey-5 × 5 environment, it is not feasible to fully understand the exploratory data distribution present in the MiniGrid-DoorKey-16 × 16 environment. Therefore, we carried out transfer learning experiments to obtain a converged EATBEE model specifically tailored for the MiniGrid-DoorKey-8 × 8 environment, which was subsequently applied successfully in the more complex MiniGrid-DoorKey-16 × 16 setting. The EATBEE migration consists of two types: fine-tuning and non-fine-tuning.

9. Ablation Experiment

The approximated EATBEE comprises three core components: the number of state-value functions, the knowledge coefficient $\eta$, and the latent state space $\mathcal{X}$. All experiments are conducted in the MiniGrid-DoorKey-8 × 8 environment, with all other experimental settings consistent with those employed in previous work. The state-value function serves as an alternative approach to calculating state similarity, while also encoding the inherent knowledge of the environment. The corresponding ablation results are presented in Figure 16. Experimental results show that increasing the number of state-value functions adopted in EATBEE improves the stability of its feature extraction performance. However, this increase does not bring about substantial improvements in overall performance. Notably, EATBEE can achieve satisfactory and effective data screening results when equipped with only two state-value functions.

Another key component of EATBEE is the knowledge coefficient. This parameter determines the threshold and extent of environmental distribution knowledge captured by the EATBEE module. A higher value of $\eta$ imposes stricter criteria for extracting environmental knowledge, while a lower $\eta$ results in a more coarse-grained perception of environmental information. The knowledge coefficient, combined with the value function, jointly supports the measurement of state similarity. The ablation analysis of the knowledge coefficient $\eta$ is presented in Figure 17. Experiments demonstrate that, with $\eta =0.99$, the EATBEE framework captures environmental knowledge in a more rigorous manner. After policy convergence and stabilization, the Ratio metric fails to approach 1. This outcome indicates that stable policy convergence in the MiniGrid-DoorKey-8 × 8 environment can be achieved without full data utilization. The present study attributes this phenomenon to the partial observability inherent to the aforementioned environment. During key retrieval and door-opening tasks, persistent exploratory behaviour remains necessary even after policy optimisation, which aligns with the inherent structural properties of the target environment. This finding validates that the environmental knowledge encoded by EATBEE is highly consistent with the actual characteristics of the physical environment.

As a core module of EATBEE, the latent space maintains the reachability of pruned states, which constitutes a fundamental premise of our method. In particular, any pruned state should still be reachable while satisfying the similarity constraint. We design ablation experiments to examine the role of the reachable latent space, and the outcomes are reported in Figure 18. The results verify that the reachable latent space is indispensable to the efficacy of EATBEE. If the reachability constraint across pruned states is broken, apparent discontinuities will appear in truncated trajectories. These abrupt fluctuations contradict the basic properties of Markov Decision Processes. Accordingly, reachability is a necessary condition to guarantee the valid application of the EATBEE framework.

10. Discussion

The proportion of exploration–exploitation data required by an agent differs substantially across distinct learning phases, yet agents lack the capability to identify these phases effectively. To tackle the challenge of agents failing to distinguish between exploration and exploitation data properly, this paper introduces the EATBEE method. The theoretical result shows that EATBEE achieves asymptotic monotonic improvement under well-defined assumptions, rather than strict monotonicity in all environments. Empirically, we validate that EATBEE’s phase-adaptive experience extraction mechanism contributes significantly to policy enhancement by aligning with the current stage of policy learning.

In comparison to existing studies on exploration and exploitation, EATBEE does not prioritize extensive state space exploration by the agent; instead, it implements efficient empirical data filtering to cater to the demands of the agent’s ongoing learning phase. Unlike research centered on experience replay techniques, EATBEE functions as a policy-adaptive data screening framework, rather than a uniform or post-hoc sampling mechanism, allowing it to accurately capture the specific data types required by the policy. Moreover, EATBEE offers strong interpretability, which is reflected in the clear comparative plots illustrating data distributions before and after raw data processing via EATBEE. In experiments spanning both discrete and continuous control tasks, EATBEE achieves performance levels comparable to the optimal results of baseline algorithms.

However, this study on EATBEE has certain limitations. For example, the current data extraction approach relies on correlation coefficients and value functions. Future research efforts can focus on advancing data screening strategies, aiming to develop a more intuitive data extraction method that can dynamically track the evolving needs of policy learning.

11. Conclusions

In existing RL algorithms, agents often exhibit over-exploration, resulting in slow policy convergence and instability. The reason is that existing algorithms cannot capture the knowledge within the data distribution. We introduced a sequential data extractor, EATBEE. First, we visualized the data distribution shift problem within a trajectory, and illustrated the extractor mechanism of EATBEE. Second, we built a theoretical guarantee for EATBEE to ensure monotonic policy improvement. Finally, through evaluation experiments, the results demonstrated that the EATBEE can autonomously extract the knowledge in most tasks and stabilize the policy learning. However, extracting relational data solely through correlation metrics poses certain inherent limitations; for instance, in continuous data regimes, temporally adjacent samples often exhibit excessively high similarity and may be erroneously discarded. Consequently, a pivotal direction for future inquiry lies in the development of more sophisticated extraction mechanisms that transcend elementary correlation-based filters.