Efficient Classification-Based Constraints for Offline Reinforcement Learning

Abstract

Existing distribution-based constraints in offline reinforcement learning, such as bootstrapping error accumulation reduction (BEAR), can prevent policy deviation but often ignore action quality, leading to $O\left(n^2\right)$ complexity and limited interpretability. This study introduces a classification-based approach that employs pairwise action quality comparison to replace complex distributional constraints. A binary classifier learns the relative quality of two actions in the same state by comparing their Q-values, prioritizing value-aware selection while maintaining conservative behavior. Experiments on benchmark environments demonstrate that the proposed method consistently improves upon BEAR, achieving 3× on average and up to 5× in some environments. The algorithm reduces computational complexity from $O\left(n^2\right)$ to $O\left(n\right)$ while providing intuitive monitoring through classifier accuracy. These results indicate that efficient quality-based comparisons can serve as a practical and efficient alternative to computationally expensive distributional constraints in offline reinforcement learning, yielding practical gains in both performance and scalability.

1. Introduction

1.1. Research Background

Reinforcement learning (RL) enables agents to learn optimal policies through environmental interaction, with applications spanning robotic control, autonomous driving, and healthcare [1,2]. However, conventional online RL requires continuous data collection through trial and error, which can be expensive, dangerous, or infeasible in real-world scenarios such as autonomous vehicles, medical treatment optimization, or industrial process control [3].

Offline RL addresses this limitation by learning policies exclusively from pre-collected fixed datasets without environment interaction [4,5]. While offering critical advantages in safety, reproducibility, and cost-effectiveness, offline RL introduces the fundamental challenge of distributional shift [6]: when the learned policy π selects actions outside the behavior policy β’s distribution, Q-value estimates become unreliable due to extrapolation errors, leading to unstable learning and performance collapse [6,7].

One representative method for addressing this issue is bootstrapping error accumulation reduction (BEAR). BEAR employs maximum mean discrepancy (MMD) to measure the distributional similarity between samples and constrains the policy to remain within the action distribution of existing data [6,8]. Although BEAR is considered a standard approach for offline RL, it has the following limitations: (1) O(n²) computational complexity that degrades performance with increasing batch size, (2) indirect learning signals based on distributional statistics rather than action quality, (3) hyperparameter sensitivity requiring careful tuning of kernel functions and bandwidth, and (4) limited interpretability with opaque MMD values providing little insight into policy quality [9,10].

While existing methods ensure distributional safety through proximity constraints or penalize potentially unreliable actions through conservative value estimation, they cannot distinguish between good and bad actions within the data support. This study addresses this limitation by directly comparing action quality through classification-based evaluation.

1.2. Research Motivation and Core Idea

This study focuses on “how good actions are selected” instead of “how close the policy is to the data distribution.” Beyond simple distributional proximity, a new type of constraint is required to identify and prefer better actions within the data [11,12,13].

Accordingly, we propose Classification-BEAR, an offline RL algorithm that constrains policies through pairwise comparison of action quality. This approach learns which of two actions is better in a given state through a classifier and directly utilizes the classifier’s output for policy updates instead of the traditional MMD-based constraints.

To clarify the differences from existing methods, we note that BEAR’s MMD and BRAC’s KL divergence are distribution-based constraints that ensure actions remain within the data support. However, the relative quality of actions within the region is not evaluated. By contrast, the proposed approach is based on selective conservatism [14], which applies conservative constraints selectively to trustworthy actions, combined with orthogonal quality assessment [15].

BRAC ensures distribution-level safety but does not perform quality comparisons between individual actions. Conversely, the proposed method directly utilizes Q-value ordering information to explicitly compare the quality of individual actions. Although this may not guarantee distribution safety, it can provide rich learning signals. Therefore, BRAC and the proposed method are complementary but fundamentally different approaches. BRAC focuses on safety while maintaining distributional proximity, whereas Classification-BEAR realizes value-aware action selection [16,17] through quality comparison between actions.

This study focuses on direct comparison of action quality, designed to learn by directly comparing which actions are considered better. This approach has the following advantages:

• Intuitiveness: Just as humans intuitively judge action A to be better than action B, agents can learn to classify actions in a similar pairwise manner.
• Efficiency: Unlike MMD-based methods with $O\left(n^2\right)$ complexity, the proposed approach operates in $O\left(n\right)$ linear time.
• Interpretability: Learning progress can be directly monitored through classifier accuracy.
• Hyperparameter stability: Classification-BEAR is simple to configure and empirically stable across different environments.

The results presented herein demonstrate the versatility of classification-based approaches. In our previous study [18], classification-based Q-value estimation or classification-based constraint methods were confirmed to work effectively in online environments. That work was based on a continuous actor–critic online RL structure designed to directly maximize Q-values. By contrast, the current study focuses on constructing preference-based constraints that can replace distributional constraints in offline RL environments. When applied to different learning domains, the classification-based critic structure is effectively transferred. This not only extends the possible application of the proposed method but also introduces a paradigm shift from value estimation tools to constraint design tools, serving as an important case that empirically demonstrates the versatility of classification-based approaches [19]. Confirming that mechanisms successful in Q-value estimation are also effective as constraint mechanisms provides important validation supporting the generalizability of this approach [19,20].

To clarify this structural difference, we consider the following example. When humans compare two actions, we do not calculate distributional statistics. Instead, we intuitively consider which action might lead to better results. For example, if selecting action a₁ in a given state yields a reward of +5 and selecting action a₂ yields + 2, the classifier learns that “a₁ is better than a₂.” Similarly, the core idea of this study is to enable RL agents to directly compare and learn the relative quality of actions based on past experiences. This process produces quality relationships that replace complex distributional calculations.

1.3. Research Contributions

The main contributions of this study are summarized as follows:

• Efficient quality-based constraint design: Learning the relative quality between actions directly via a classifier rather than relying on distribution-based constraints.
• Consistent performance improvement: Achieving reliable performance gains over BEAR across various environments.
• Superior computational efficiency: Achieving approximately 3× faster training speed on average and up to 5× in some environments with $O\left(n\right)$ complexity compared with MMD’s O(n²).
• Versatility and generalizability: Demonstrating robust performance across environments with different reward structures and state dimensions.
• Interpretable learning structure: Enabling transparent monitoring of policy quality and learning progress through classification accuracy.

These contributions provide a practical direction for constraint design in offline RL and demonstrate extensibility that can be naturally integrated into diverse RL architectures in the future.

Section 2 reviews related work and positions our contribution within existing offline RL paradigms. Section 3 establishes the mathematical framework, defining pairwise classifiers and quality-based constraints. Section 4 presents the Classification-BEAR algorithm. Section 5 provides comprehensive experimental evaluation across diverse environments with statistical analysis. Section 6 discusses implications, limitations, and future directions. Section 7 concludes.

2. Related Work

2.1. Offline RL and Distributional Shift

Offline RL addresses the challenge of learning effective policies from fixed datasets without environment interaction, enabling applications where real-time data collection is expensive, dangerous, or infeasible [3,4,5]. The central challenge is distributional shift: when the learned policy π selects actions outside the behavior policy β’s distribution, Q-value estimates become unreliable due to extrapolation errors, leading to unstable learning and performance collapse [4,5,6].

This fundamental problem manifests in two ways: (1) extrapolation error, where Q-values are incorrectly estimated for state–action pairs absent from the dataset, and (2) compounding error, where inaccurate estimates accumulate during iterative policy improvement, progressively degrading performance. Addressing distributional shift has become the defining objective of offline RL research.

2.2. Existing Offline RL Approaches

Current offline RL methods address distributional shift through three main paradigms:

Methods such as conservative Q-learning (CQL) [21], implicit Q-learning (IQL) [9], and advantage-weighted actor–critic (AWAC) [22] modify value function objectives to penalize unseen actions or apply implicit behavioral constraints. CQL explicitly penalizes Q-values for out-of-distribution actions while elevating in-distribution estimates. IQL uses expectile regression to avoid distributional constraints entirely, learning conservative value functions through asymmetric regression. AWAC weights policy updates by advantage estimates, implicitly constraining the policy. While computationally efficient (O(n) complexity), these methods often exhibit excessive conservatism, limiting performance, or suffer from hyperparameter sensitivity that requires careful tuning per environment.

BEAR [6] and BRAC [11] explicitly constrain the learned policy to remain close to the behavior policy distribution. BEAR employs maximum mean discrepancy (MMD) to measure distributional similarity: MMD (π(·|s), β(·|s)) ≤ ε ensures policy actions stay within data support. BRAC replaces MMD with Kullback–Leibler (KL) divergence for improved efficiency $\left(O\right(n log n)$ vs. $O\left(n^2\right)$). Advantage-weighted regression (AWR) [23] applies advantage weighting with distributional proximity. While these methods provide stable performance through explicit safety constraints, they face three critical limitations: (1) high computational cost ($O\left(n^2\right)$ for MMD and $O(n log n)$ for KL); (2) inability to distinguish action quality within the data distribution—all in-distribution actions are treated equally regardless of their Q-values; and (3) poor interpretability as distance metrics (MMD values and KL divergence) provide limited insight into policy quality or learning progress.

MOPO [24] and COMBO [25] incorporate learned environment models with uncertainty penalties to avoid unreliable state–action regions. MOPO penalizes model uncertainty to discourage exploration into poorly modeled areas, while COMBO combines CQL’s conservative value estimation with model-based planning. These methods offer the potential to leverage environment structure but depend heavily on model accuracy: errors in learned dynamics propagate and compound, potentially undermining performance. Model learning adds O(model) computational overhead and complexity.

The limitations of existing methods motivate our quality-based approach: rather than focusing solely on distributional safety (staying within data) or conservative penalties (avoiding unseen actions), we directly compare action quality within the data support, achieving competitive performance with O(n) complexity and interpretable monitoring.

2.3. Classification-Based Approaches in RL

Classification paradigms have emerged across multiple RL contexts. Ranking learning methods from information retrieval—including RankNet [26] (pairwise ranking via neural networks) and ListNet [27] (listwise ranking optimization)—have inspired preference modeling in RL. Preference-based RL [28] learns from human preferences rather than reward functions, training policies to align with comparative feedback. Classification in deep RL includes architectural innovations such as dueling DQN [29], which decomposes Q-values into state value and advantage streams, and categorical DQN [30], which estimates return distributions through classification over discrete bins. Decision transformer [31] frames RL as sequence modeling, predicting actions conditioned on desired returns.

Recent work has explored classification for value estimation. Choi et al. (2024) [32] demonstrated that listwise ranking in preference-based RL enables more efficient learning than pairwise methods. Our previous work [18] showed that classification-based Q-value estimation outperforms regression-based approaches even in online settings, learning to classify Q-value bins rather than directly regressing values. The present study extends this paradigm from value estimation to constraint design, replacing distributional matching with direct quality comparison—a novel application that demonstrates the versatility of classification-based approaches across RL components.

2.4. Key Differences from Existing Methods

Table 1. Comparison with existing offline RL methods.

Aspect	Conservative (CQL, IQL)	Distributional (BEAR, BRAC)	This Study (Classification-BEAR)
Constraint philosophy	Penalize unseen actions	Stay within data distribution	Select better actions
Learning signal	Modified value objective	Distributional statistics	Q-value ordering
Computational complexity	$O\left(n\right)$	$O\left(n^2\right):MMD,O(n log n):KL$	$O\left(n\right)$
Action quality awareness	Indirect through penalties	None—all in-distribution actions equal	Direct through pairwise comparison
Interpretability	Penalty magnitude (opaque)	Distance metrics (opaque)	Classification accuracy (transparent)
Hyperparameter sensitivity	High (penalty coefficients)	High (kernels, bandwidth)	Low (temperature, smoothing)
Primary focus	Avoid overestimation	Distributional safety	Value-aware selection

summarizes the fundamental differences between our approach and existing offline RL paradigms.

The key distinction lies in constraint philosophy. Conservative methods focus on penalizing potentially unreliable actions, distributional methods ensure policy–behavior proximity, and our approach directly identifies and prefers higher-quality actions. While distributional constraints ask “Is this action in the data?” quality-based constraints ask “Is this action better than alternatives in the data?” This reframing enables finer-grained control: rather than treating all in-distribution actions equally, we distinguish their quality through pairwise Q-value comparison.

BRAC [11] and Classification-BEAR both move beyond MMD but pursue different objectives. BRAC replaces MMD with KL divergence to improve computational efficiency while maintaining distributional matching. Classification-BEAR replaces the entire distributional paradigm with quality comparison. BRAC ensures (π(·|s) ≈ β(·|s)); Classification-BEAR ensures $E[C_{\phi }\_(s, a_{\pi }\_, a_{\beta }\_\left)\right] > 0.5$. These are complementary: BRAC prioritizes safety through distribution-level proximity, while our method prioritizes performance through action-level quality assessment. Our experiments (Section 5) show that quality-based constraints alone can achieve competitive results, suggesting that distributional proximity may not be strictly necessary when action quality is explicitly modeled.

Quality-based constraints are orthogonal to conservative value estimation. CQL and IQL modify the value learning objective; our approach modifies the policy constraint mechanism. These paradigms could potentially be combined: conservative Q-learning could provide calibrated Q-values for classifier training, while quality-based constraints guide policy optimization. Exploring such hybrid approaches represents promising future work.

2.5. Recent Advances and Research Context

Contemporary offline RL research explores several directions complementary to our work.

Offline-to-online transfer methods [17,33,34,35] investigate how to leverage offline pretraining for improved online fine-tuning, addressing the offline–online gap. Selective regularization [14] applies conservative constraints only to trustworthy states identified through uncertainty estimation, avoiding excessive conservatism. Diffusion-based policies [15] use generative models for expressive policy representations in offline settings. Representation learning [19,20] improves generalization through contrastive learning and first-order dynamics modeling, particularly for multitask scenarios.

Our work aligns with the trend toward enhanced efficiency, interpretability, and practical deployability. By achieving O(n) complexity with transparent classification accuracy monitoring, Classification-BEAR addresses computational and interpretability challenges while introducing a novel constraint paradigm. The classification-based approach is complementary to these advances: it could enhance offline-to-online transfer through quality-aware initialization, combine with selective regularization by confidence-weighted constraints, or leverage improved representations for more accurate quality comparison.

Among recent methods, our work is most closely related to selective regularization [14] in philosophy (adapting conservatism based on uncertainty) but differs fundamentally in mechanism (pairwise quality comparison vs. state-dependent regularization). Compared with advantage-aware optimization [16] and first-order dynamics methods [20], our approach operates at the constraint level rather than value estimation, offering orthogonal contributions. This positions Classification-BEAR as a methodological contribution to constraint design rather than an improvement to specific algorithmic components, with potential for integration across diverse offline RL frameworks.

From a broader perspective, recent advances in offline RL can be broadly categorized along two major axes—explainability-oriented approaches such as debt collection recommender systems designed for human rationale generation [36] and value-update-level conservative methods such as imagination-limited Q-learning [37], which reduce extrapolation bias by constraining hypothetical rollouts during target estimation. In contrast, Classification-BEAR introduces a third and orthogonal methodological axis focused on policy-level support filtering with minimal computational overhead. Rather than altering value targets or generating explicit explanations, our method directly constrains the action selection process through efficient pairwise quality comparison. Consequently, it should not be viewed as a competing alternative but rather a scalable and composable constraint design paradigm that can be naturally integrated with both explainability-driven and value-conservative offline RL frameworks to achieve a complementary safety–efficiency balance.

3. Mathematical Framework

3.1. Problem Formulation

We consider the standard offline RL setting: given a Markov decision process (MDP) defined by state space $S$, action space $A$, transition dynamics $P$, reward function $R$, and discount factor $\gamma \in \left[\mathrm{0,1}\right)$ and a fixed dataset $\mathcal{D}={\left\{\right(s_{\mathrm{i}}, a_{\mathrm{i}}, r_{\mathrm{i}}, {s^{\prime }}_{\mathrm{i}}\left)\right\}}_{i=1}^N$ collected by behavior policy $\beta$, the objective is to learn a policy $\pi$ that maximizes expected return $J\left(\pi \right) = E\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t\right]$ without additional environment interaction.

The central challenge in offline RL is distributional shift: when the learned policy π selects actions outside the support of behavior policy $\beta$, the estimated Q-values $Q_{\pi }\left(s,a\right)$ become unreliable due to extrapolation errors, leading to unstable learning and performance degradation [4,5,6].

3.2. Core Definitions

Definition 1.

Pairwise Action Quality Classifier.

We define a binary classifier $C_{\phi }: S \times A \times A \to \left[\mathrm{0,\; 1}\right]$ parameterized by $\phi$ that estimates the probability that one action has a higher Q-value than another:

$$C_{\phi }(s,a_1,a_2)\approx P\left(Q^{\pi }\right(s,a_1)>Q^{\pi }(s,a_2\left)\right)$$

(1)

where $Q^{\pi }(s,a)$ denotes the action value function under policy $\pi$. Through training on Q-value comparisons, the classifier approximately satisfies natural ordering properties (anti-symmetry, transitivity, and self-consistency) that ensure consistent action quality rankings across pairwise comparisons.

Definition 2.

Quality-Based Policy Constraint.

A policy ${\pi }_{\psi }$ parameterized by ψ satisfies the quality-based constraint if

$${\mathbb{E}}_{\{s\sim {\rho }^{\pi }, a\sim {\pi }_{\psi }(·\left|s\right), a^{\prime }\sim \beta (·|s\left)\right\}} \left[C_{\phi }\right(s, a, a^{\prime }\left)\right] \ge {\tau }_{min}$$

(2)

where ${\rho }^{\pi }$ denotes the state distribution under policy $\pi$, $\beta$ is the behavior policy, and ${\tau }_{min}$ $\in [0.5, 1]$ is a threshold parameter. This constraint ensures that actions sampled from ${\pi }_{\psi }$ are predicted by the classifier to have higher quality than behavior policy actions with probability at least ${\tau }_{min}$, thereby encouraging value-aware action selection while maintaining conservative behavior.

In practice, ${\tau }_{min}$ regulates the degree of conservatism: ${\tau }_{min}= 0.5$ represents neutral baseline, while larger values enforce a stronger preference for high-quality actions. Empirical results (Section 5) indicate that ${\tau }_{min}\in [0.6, 0.8]$ achieves stable performance across diverse environments.

3.3. Comparison with Distributional Constraints

3.3.1. Approximation Quality

The classifier’s approximation quality is characterized by the following bound:

Proposition 1.

If classifier  $C_{\phi }$ achieves classification accuracy  $1-\epsilon$ on a validation set (where ε is the error rate), then for any state s and actions a₁, a₂,

$$\left|C_{\phi }\left(s, a^1, a^2\right)-P\left(Q^{\pi }\left(s, a^1\right)>Q^{\pi }\left(s, a^2\right)\right)\right|\le 2\epsilon .$$

(3)

This bound follows from standard binary classification error analysis: the expected absolute deviation between predicted probability and true probability is at most twice the classification error rate. Equation (3) guarantees that as the classifier improves $(\epsilon \to 0)$, its predictions converge to the true probability of Q-value ordering.

3.3.2. Monotonicity Property

Proposition 2.

For a well-trained classifier, $if Q^{\pi }(s, a_1)>Q^{\pi }(s, a_2)>Q^{\pi }(s, a_3)$, then

$$C_{\phi }\left(s, a_1, a_3\right)\ge max\left\{C_{\phi }\left(s, a_1,a_2\right), C_{\phi }\left(s, a_2,a_3\right)\right\}.$$

(4)

This monotonicity property ensures that pairwise quality comparisons compose consistently: if action a₁ is better than a₂ and a₂ is better than a₃, then the classifier assigns at least as high confidence to “a₁ better than a₃” as to the intermediate comparisons. This supports transitive reasoning about action quality and stable policy improvement.

3.3.3. Computational Complexity

Proposition 3.

Classification-BEAR has $O\left(n\right)$ time complexity per iteration, where $n$ denotes the batch size.

Each algorithmic component operates linearly on batch size: Q-network forward and backward passes $\left(O\right(n\left)\right)$, action pair generation from batch states $\left(O\right(n\left)\right)$, classifier training on $n$ pairs $\left(O\right(n\left)\right)$, and policy gradient computation $\left(O\right(n\left)\right)$. The total complexity per iteration is therefore $O\left(n\right)$.

In contrast, BEAR’s maximum mean discrepancy (MMD) computation requires evaluating kernel functions between all pairs of samples, resulting in $O\left(n^2\right)$ complexity. This yields

$$Speedup={\displaystyle \frac{O\left(n^2\right)}{O\left(n\right)}}=O(n)$$

which explains the empirically observed 3–5× speedup in practice in Section 5. Additionally, Classification-BEAR eliminates the need to store $n\times n$ kernel matrices, reducing peak memory usage by approximately 57% compared with BEAR in high-dimensional environments in Section 5.

3.3.4. Convergence Properties

Proposition 4.

Under standard assumptions for actor–critic methods (bounded rewards $R_{max}$, Lipschitz continuous policy ${\pi }_{\psi }$, diminishing learning rates $\left\{{\alpha }_t\right\}$, and bounded approximation error in Q-function estimation), Classification-BEAR converges to a stationary point of the policy objective within bounded error.

Specifically, if the classifier error ${\epsilon }_t$ decreases over time and the Q-function approximation error remains bounded, then the policy update operator induced by Equation (2) is a contraction in expectation, ensuring convergence. As the classifier accuracy improves $({\epsilon }_t$ $\to 0)$, policy updates increasingly align with true Q-value rankings, supporting stable policy improvement.

The detailed convergence analysis follows from combining standard actor–critic convergence theory [1,6] with the classification error bounds established in Proposition 1.

3.4. Conceptual Comparison with Distributional Methods

Quality-based constraints (Equation (2)) are fundamentally more expressive than distributional constraints used in BEAR and BRAC. Consider two actions a₁ and a₂, both within the support of behavior policy $\beta$ (i.e., $\beta \left(a_1\right|s) > 0, \beta (a_2\left|s\right) > 0$) but with different Q-values: $Q^{\pi }\left(s, a_1\right)> Q^{\pi }\left(s, a_2\right).$

Using BEAR’s MMD constraint, MMD $\left(\pi \right(·\left|s\right), \beta (·|s\left)\right) \le \epsilon$ ensures that $\pi (·|s)$ remains close to $\beta (·|s)$ in distribution but provides no mechanism to prefer $a_1$ over $a_2$. Both actions receive similar probability mass if they are similarly frequent in the dataset.

Using Classification-BEAR’s quality constraint, Equation (2) directly encodes the preference $C_{\phi }(s,a_1,a_2)$ $\to 1$, explicitly prioritizing higher-quality actions. This enables value-aware action selection rather than mere distributional proximity.

This represents a fundamental shift from “distributional safety” (stay within data support) to “quality-based selection” (choose better actions within the support). While distributional constraints prevent catastrophic out-of-distribution actions, they may be overly conservative by treating all in-distribution actions equally. Quality-based constraints provide finer-grained control, distinguishing action quality within the data support. In Table 1 in Section 2, the key differences are summarized. This comparison demonstrates that while distributional and quality-based constraints address different aspects of offline RL (safety vs. performance), the quality-based approach can achieve competitive or superior results with significantly reduced computational cost and enhanced interpretability. Formal proofs and extended mathematical analysis are provided in Appendix A.

4. Methodology

4.1. Overview of Classification-BEAR

Classification-BEAR implements the quality-based constraint framework established in Section 3 through three core components: (1) a Q-network $Q_{{\theta }^{\prime }}$ that estimates action values, (2) a pairwise classifier $C_{\phi }$ that learns action quality rankings from Q-value comparisons, and (3) a policy ${\pi }_{\psi }$ that is optimized to prefer higher-quality actions as predicted by the classifier. Unlike BEAR’s MMD-based distributional matching, our approach directly compares action quality through symmetric pairwise classification, achieving $O\left(n\right)$ computational complexity while maintaining value-aware action selection.

4.2. Classifier Training Procedure

4.2.1. Action Pair Generation and Q-Value Computation

For each state s in a training batch, we sample two transitions $(s,a_1,r_1,{s^{\prime }}_1)$ and $(s,a_2,r_2,{s^{\prime }}_2)$ from dataset $\mathcal{D}$. We compute bootstrapped Q-value estimates using the target Q-network $Q_{\theta }^{\prime }$:

$$Q_1=r_1+\gamma Q_{\theta }^{\prime }(s_1^{\prime },{\pi }_{\psi }(s_1^{\prime }\left)\right)$$

$$Q_2=r_2+\gamma Q_{\theta }^{\prime }(s_2^{\prime },{\pi }_{\psi }(s_2^{\prime }\left)\right)$$

where $\gamma$ is the discount factor, ${\pi }_{\psi }$ is the current policy, and $Q_{\theta }^{\prime }$ is the target network (updated via soft updates for stability).

4.2.2. Soft Label Generation

To train the classifier $C_{\phi }$, we convert Q-value differences into probabilistic labels using temperature-scaled sigmoid followed by label smoothing:

$$p = \sigma \left({\displaystyle \frac{Q_1 - Q_2}{\tau }}\right)$$

(5)

$$\mathrm{ŷ}=(1-\alpha ) · p+0.5 · \alpha ,$$

(6)

where σ denotes the sigmoid function, $\tau > 0$ is the temperature parameter controlling the sharpness of the probability distribution, and $\alpha \in \left[\mathrm{0,\; 1}\right]$ is the label smoothing factor. Equation (5) converts Q-value differences into soft probabilities: when $Q_1 \gg Q_2$, $p \to 1$; when $Q_1 \ll Q_2, p \to 0$; and when $Q_1 \approx Q_2, p \approx 0.5$. Equation (6) applies label smoothing to prevent overconfidence: the smoothed label $\mathrm{ŷ}$ is pulled toward $0.5$ by factor $\alpha$, improving classifier generalization and robustness.

The classifier is then trained to minimize binary cross-entropy loss:

$$L_c = -{[\mathrm{ŷ}\mathrm{l}\mathrm{o}\mathrm{g}C}_{\phi }(s, a_1, a_2)+(1-\mathrm{ŷ})\mathrm{l}\mathrm{o}\mathrm{g}(1-C_{\phi }\left(s, a_1, a_2\right))]$$

where the loss encourages $C_{\phi }(s, a_1, a_2)$ to approximate the soft label $\mathrm{ŷ}$, which in turn approximates $P(Q_1 > Q_2).$

4.2.3. Action Pair Generation in Continuous Spaces

In continuous state spaces where exact state matching is rare, we generate action pairs based on $\epsilon$-neighborhood similarity rather than strict state equality. Specifically, for each state sss sampled from the mini-batch, we identify alternative transitions whose states $s^{\prime }$ satisfy ${\parallel s-s^{\prime }\parallel }_2<\epsilon$, with $\epsilon =0.01$ fixed across all experiments. This ensures that pair formation reflects locally comparable decision contexts while preserving the theoretical $O\left(n\right)$ complexity of our algorithm as exactly one valid pair is constructed per state without requiring any cross-batch search or nearest-neighbor expansion. Unlike BEAR’s kernel-based MMD constraint, which requires $O\left(n^2\right)$ pairwise evaluations, our pairing mechanism scales linearly with batch size and incurs negligible computational overhead (formal proof provided in Appendix A).

A crucial property of this construction is that the resulting classifier naturally preserves preference transitivity across action pairs. Empirical validation over 10,000 triplets per environment showed perfect logical consistency in HalfCheetah (100%), with only negligible violations in Hopper (99.97%) and Walker2d (99.93%), all of which disappeared after the first few epochs of training. This confirms that our pairwise comparison process yields a stable and coherent implicit action ranking without requiring any additional cycle elimination or explicit transitivity regularization (empirical validation results reported in Appendix B).

In addition to preserving structural coherence, the $\epsilon$-similarity pairing design directly contributes to the practical efficiency of our method. Because only a single filtered alternative action is evaluated per state, the classifier-guided policy update operates strictly at $O\left(n\right)$ total cost per iteration. This design choice is the key reason why Classification-BEAR consistently achieves 3–5× faster wall-clock training speed than BEAR while maintaining stable and data-supported policy improvement.

4.3. Policy Optimization with Quality Constraints

The policy ${\pi }_{\psi }$ is optimized to maximize the expected classifier score, encouraging actions that the classifier predicts to have higher Q-values than behavior policy actions:

$$L_{\pi } = -\lambda ·{\mathbb{E}}_{S~D, a_{policy}~{\pi }_{\psi }\left(\cdot ∣\mathrm{s}\right),a_{data}~\mathrm{D}\left(\cdot ∣\mathrm{s}\right)}[C_{\phi }(s,a_{policy},a_{data})] + \beta ·H[{\pi }_{\psi }(·|s\left)\right]$$

(7)

where $\lambda >0$ controls the strength of the quality constraint, $a_{policy}$ is sampled from the current policy, $a_{data}$ is sampled from the dataset for the same state s, $H[·]$ denotes entropy, and $\beta \ge 0$ is the entropy regularization coefficient. The negative sign converts maximization of classifier scores into a minimization objective for gradient descent.

Equation (7) embodies the quality-based constraint (Definition 2): by maximizing $C_{\phi }(s,a_{policy},a_{data})$, the policy learns to generate actions that the classifier predicts will outperform dataset actions. The entropy term $\beta ·H\left[{\pi }_{\psi }\right(·\left|s\right)]$ encourages exploration and prevents premature convergence to deterministic policies.

4.4. Classification-BEAR Algorithm

We propose Algorithm 1, which summarizes the overall training pipeline of the Classification-BEAR method.

Algorithm 1: Classification-BEAR

Input:    $Offline dataset$ $D=\left\{\right(s,a,r,s^{\prime }\left)\right\}$    $Networks:$        $Q_{\theta } (Q-network)$$, {\pi }_{\psi } \left(Policy\right)$$, C_{\phi } \left(classifier\right)$$, Q_{{\theta }^{\prime }} (Target network)$    $Hyperparameters:$$\tau \left(temperature\right),\alpha \left(smoothing\right),$       $\lambda \left(quality weight\right), \beta \left(entropy weight\right), {\tau }_{target} (target update rate)$ $\mathbf{Initialize} Q_{\theta }$$, {\pi }_{\psi }$$, C_{\phi }$ randomly $\mathbf{Set} Q_{{\theta }^{\prime }}$ ← $Q_{\theta }$ for episode = 1 to max_episodes do:   $B \leftarrow SampleBatch \left(\mathcal{D}\right)$    $\mathit{U}\mathit{p}\mathit{d}\mathit{a}\mathit{t}\mathit{e}\mathit{Q}\mathit{N}\mathit{e}\mathit{t}\mathit{w}\mathit{o}\mathit{r}\mathit{k} (Q_{\theta }, B, {\pi }_{\psi }, Q_{{\theta }^{\prime }})$    // Q-Learning Update   $pairs \leftarrow \mathit{G}\mathit{e}\mathit{n}\mathit{e}\mathit{r}\mathit{a}\mathit{t}\mathit{e}\mathit{A}\mathit{c}\mathit{t}\mathit{i}\mathit{o}\mathit{n}\mathit{P}\mathit{a}\mathit{i}\mathit{r}\mathit{s} (B, \mathcal{D})$    $\mathit{T}\mathit{r}\mathit{a}\mathit{i}\mathit{n}\mathit{C}\mathit{l}\mathit{a}\mathit{s}\mathit{s}\mathit{i}\mathit{f}\mathit{i}\mathit{e}\mathit{r} (C_{\phi }, pairs, Q_{{\theta }^{\prime }}, \tau , \alpha )$    // Classification Training   $\mathit{U}\mathit{p}\mathit{d}\mathit{a}\mathit{t}\mathit{e}\mathit{P}\mathit{o}\mathit{l}\mathit{i}\mathit{c}\mathit{y} ({\pi }_{\psi }, C_{\phi }, B, \lambda , \beta )$      // Policy Improvement   $\mathit{S}\mathit{o}\mathit{f}\mathit{t}\mathit{U}\mathit{p}\mathit{d}\mathit{a}\mathit{t}\mathit{e} (Q_{{\theta }^{\prime }}, Q_{\theta }, {\tau }_{target})$       // Target Update end for $\mathbf{return} {\pi }_{\psi }$ $\mathbf{UpdateQNetwork} (Q_{\theta }, B, {\pi }_{\psi }, Q_{{\theta }^{\prime }})$:    $\mathrm{for} (s, a, r, s^{\prime })$$\mathrm{in} B$ do:      $a^{\prime }$$\leftarrow {\pi }_{\psi }\left(s^{\prime }\right)$           ${target}_q \leftarrow r + \gamma · Q_{{\theta }^{\prime }} (s^{\prime }, a^{\prime })$    end for          $L_Q \leftarrow MSE \left(Q_{\theta }\right(s, a), {target}_q)$    $\mathrm{UpdateParameters} (Q_{\theta }$$, {\nabla }_{\theta }, L_Q$) $\mathbf{GenerateActionPairs} (B, \mathcal{D}$):    pairs$\leftarrow \left[\right]$    $\mathrm{for} \mathrm{each} state s$$\mathrm{in} B$:          $if \left|\right\{transitions with state s in D\left\}\right| \ge 2 then$       $Sample two transitions: (s, a_1, r_1, {s^{\prime }}_1), (s, a_2, r_2, {s^{\prime }}_2) from D$       pairs.append $\left(\right(s,a_1,a_2,r_1, r_2,{s^{\prime }}_1,{s^{\prime }}_2\left)\right)$    $\mathrm{return} pairs$ $\mathbf{TrainClassifier} (C_{\phi }, pairs, Q_{{\theta }^{\prime }}, \tau , \alpha$):    $\mathrm{for} (s,a_1,a_2,r_1, r_2,{s^{\prime }}_1,{s^{\prime }}_2)$$\mathrm{in} pairs$:         $Q_1=r_1+\gamma Q_{\theta }^{\prime }(s_1^{\prime },{\pi }_{\psi }(s_1^{\prime }\left)\right)$         $Q_2=r_2+\gamma Q_{\theta }^{\prime }(s_2^{\prime },{\pi }_{\psi }(s_2^{\prime }\left)\right)$      $soft\_label \leftarrow \sigma \left(\right(Q_1-Q_2)/\tau ) \times (1-\alpha ) + 0.5\alpha$ // Equations (5) and (6)     end for       $L_{C }\leftarrow BinaryCrossEntropy\left(C_{\phi }\right(s,a_1,a_2), soft\_label)$    $\mathrm{UpdateParameters} (C_{\phi },{\nabla }_{\phi }, L_{C })$ UpdatePolicy $({\pi }_{\psi },C_{\phi },B, \lambda , \beta ):$    $\mathrm{for} \mathrm{each} state s$$\mathrm{in} B$:      $a_{policy}$ $\leftarrow {\pi }_{\psi }\left(s\right)$      $a_{data}$$\leftarrow SampleActionfromDataset(s, \mathcal{D})$      scores$\leftarrow C_{\phi }(B.states, a_{policy}, a_{data})$    end for    $L_{\pi }=-\lambda ·Mean\left(scores\right)+\beta ·Entropy\left({\pi }_{\psi }\right)$ // Equation (7)    UpdateParameters $({\pi }_{\psi },{\nabla }_{\psi },L_{\pi })$ SoftUpdate ($Q_{{\theta }^{\prime }}, Q_{\theta }, {\tau }_{target}):$       $Q_{{\theta }^{\prime }}$← ${\tau }_{target}$·$Q_{\theta }+(1-$${\tau }_{target})$·$Q_{{\theta }^{\prime }}$

4.5. Comparisons of Classification-BEAR and BEAR Algorithm

Figure 1 illustrates the fundamental architectural difference between BEAR and Classification-BEAR. BEAR constrains the learned policy $\pi$ through maximum mean discrepancy:

$$\mathrm{MMD} \left(\pi \left(·|s\right),\beta \left(·|s\right)\right)\le {\parallel E_{a\sim \pi }\left[\varphi \right(a\left)\right]-E_{a^{\prime }\sim \beta }\left[\varphi \right(a^{\prime }\left)\right]\parallel }_H$$

where $\phi$ maps actions to a reproducing kernel Hilbert space H. This requires computing kernel evaluations for all pairs of sampled actions, resulting in $O\left(n^2\right)$ complexity. The MMD constraint ensures distributional proximity but provides no mechanism to distinguish action quality within the behavior policy’s support.

The Hybrid approach combines both distributional and quality-based constraints:

$$-{\lambda }_1\mathrm{Mean} \left(C_{\phi }\right(s,a_1,a_2)) + {\lambda }_2\mathrm{MMD} (\pi \left(·|s\right),\beta \left(·|s\right)) + \beta ·\mathrm{Entropy}\left(\pi \right)$$

This dual-constraint design attempts to balance distributional safety (λ₂ term) with action quality guidance (λ₁ term).

Classification-BEAR replaces MMD with symmetric pairwise classifier $C_{\phi }(s,a_1,a_2),$ which directly encodes Q-value ordering through Equation (1). The classifier operates on individual action pairs with $O\left(1\right)$ evaluation per pair and $O\left(n\right)$ total complexity for n pairs. This paradigm shift from distributional matching to quality comparison enables the following:

• Linear computational scaling: $O\left(n\right)$ vs. $O\left(n^2\right);$
• Direct value information: Q-value ordering vs. distributional statistics;
• Interpretable monitoring: Classification accuracy vs. opaque MMD values;
• Reduced hyperparameter sensitivity: Simple temperature/smoothing vs. kernel selection/bandwidth tuning.

The computational efficiency of Classification-BEAR is analyzed in detail, as summarized in

Table 2. Time complexity.

Component	Operations	Complexity
Q-network update	Forward + backward pass	$O\left(n\right)$
Action pair generation	State grouping + sampling	$O\left(n\right)$
Classifier training	Binary classification × n pairs	$O\left(n\right)$
Policy update	Forward + backward pass	$O\left(n\right)$
Target update	Parameter copying	$O\left(1\right)$
Total per iteration		$O\left(n\right)$

. As summarized in

Table 3. Memory complexity.

Component	BEAR	Classification-BEAR	Savings
Constraint computation	$O\left(n^2\right)$ kernel matrix	$O\left(n\right)$ action pairs	$O\left(n\right)$
Intermediate storage	$n\times n$ similarity matrix	Temporary pair list	$~87\%$ reduction
Total memory scaling	$O\left(n^2\right)$	$O\left(n\right)$	Linear improvement

, unlike BEAR’s $O\left(n^2\right)$ kernel matrix storage for MMD computation, Classification-BEAR requires only temporary storage for action pairs, resulting in a significantly reduced memory footprint.

5. Experiments

5.1. Experimental Setup

We evaluate Classification-BEAR on three benchmark environments with diverse characteristics [38,39]:

• Pendulum (classic control) [40]: Continuous 3D state space ($cos \theta$, $sin \theta$, and angular velocity), 1D action space $[-2, 2]$, dense reward $r = -({\theta }^2 + 0.1{\theta }^2 + 0.001a^2)$, and 200 steps per episode. Tests basic continuous control performance.
• MountainCar (classic control) [41]: 2D state (position and velocity), 1D action $[-1, 1]$, sparse reward $(+100$ at goal and $-0.1a^2$ otherwise), and 999 steps per episode. Evaluates robustness in challenging sparse-reward settings.
• HalfCheetah (MuJoCo) [42]: High-dimensional 17D state (joint angles/velocities), 6D action (torques), dense reward (forward velocity $- 0.1a^2$), and 1000 steps per episode. Standard MuJoCo benchmark for scalability.

For each environment, we collected 100 K transitions using medium-quality policies (soft actor–critic trained for 1 M steps, achieving $60–80\%$ of expert performance) with ε-greedy exploration for diversity.

Networks use two-layer MLPs $[256, 256]$ for Q and policy and three-layer MLPs $[256, 256, 128]$ for classifier, all with ReLU activations. Adam optimizer has a learning rate $3\times {10}^{-4}$, batch size $256$, and gradient clipping ($1.0$ for Q/classifier and $0.5$ for policy). Hyperparameters are temperature τ = 0.5, label smoothing α = 0.1, quality weight λ = 1.0, entropy weight $\beta =0.01$, target update rate $0.005$, and discount $\gamma =0.99$. These values were selected via grid search on Pendulum and applied uniformly across environments. All state features are standardized using z-score normalization prior to training to ensure scale-invariant distance measurements for ε-neighborhood pairing.

We compare against (1) BEAR-Original, standard BEAR with MMD constraint, and (2) Hybrid, combined MMD + classifier approach with weights λ₁ = 0.8 (quality) and λ₂ = 0.5 (distribution).

Each algorithm ran for 1 M gradient steps with evaluation every 10 K steps. We report mean ± std over 10 independent runs (seeds $42, 123, 456, 789$, $1011, 1213, 1415, 1617, 1819, \mathrm{a}\mathrm{n}\mathrm{d} 2021$). Initial experiments (Pendulum, MountainCar, and HalfCheetah) used 10 seeds (42, 123, 456, 789, 1011, 1213, 1415, 1617, 1819, and 2021). Extended D4RL experiments (Hopper and Walker2d) [43] used five seeds (42, 123, 456, 789, and 999). Statistical significance was assessed using t-tests with Bonferroni correction (α = 0.05) and Cohen’s d effect size. The hardware was an NVIDIA RTX 3080 GPU, PyTorch 1.12.0, and Gymnasium 0.26.3.5.1.1 Experimental Environments and Dataset Generation.

5.2. Performance Results

5.2.1. Overall Comparison

Figure 2 presents a comprehensive performance comparison across four metrics.

Classification-BEAR consistently outperformed BEAR-Original across all environments, recording improvements of +120 (Pendulum), +2 (MountainCar), and +171 (HalfCheetah).

Classification-BEAR achieved the highest overall performance improvement (+97.7) compared with BEAR-Original (–37.0) and Hybrid (+91.0). This indicates consistent advantages beyond environment-specific factors.

Classification-BEAR achieved +146 (Pendulum), +38 (MountainCar), and +261 (HalfCheetah), demonstrating substantial improvements across different task complexities.

Classification-BEAR achieved a 100% win rate (3/3 environments) over BEAR-Original and Hybrid, whereas both baselines recorded no wins. This emphasizes the reliability and robustness of the proposed method across diverse environments.

5.2.2. Environment-Specific Analysis

Figure 3, Figure 4 and Figure 5 show detailed analyses for each environment including learning curves, statistical tests, effect sizes, and classification accuracy evolution.

• Pendulum (Figure 3): Classification-BEAR achieved +146 improvement over BEAR with a 60% success rate (episodes exceeding BEAR + 20 threshold). While statistical significance was marginal (p = 0.071), the very large effect size (Cohen’s d = 5.00) indicates substantial practical differences. Classification accuracy reached 75–80%, demonstrating reliable quality discrimination. The narrow reward distribution indicates consistent high performance.
• MountainCar (Figure 4): Despite challenging sparse rewards, Classification-BEAR achieved +38 improvement with very strong statistical significance (p < 0.001, d = 8.91 vs. BEAR). Classification accuracy stabilized at 70–75%. Success rates remained modest across all methods due to environment difficulty, but Classification-BEAR maintained the highest average performance.
• HalfCheetah (Figure 5): In this high-dimensional environment, Classification-BEAR achieved +261 improvement over BEAR (p = 0.040, d = 1.04) with 50–60% success rate. Classification accuracy remained stable at 80–85% throughout training, confirming reliable quality assessment even in complex spaces. Both learning stability and final performance surpassed baselines.

Figure 3. Detailed analysis for Pendulum environment: learning curves, statistical significance, effect size, classification accuracy, and success rate.

Figure 3. Detailed analysis for Pendulum environment: learning curves, statistical significance, effect size, classification accuracy, and success rate.

Figure 4. Detailed analysis for the MountainCar environment: learning curves, statistical significance, effect size, classification accuracy, and success rate. Asterisks indicate statistical significance (*** p < 0.001).

Figure 4. Detailed analysis for the MountainCar environment: learning curves, statistical significance, effect size, classification accuracy, and success rate. Asterisks indicate statistical significance (*** p < 0.001).

Figure 5. Detailed analysis for the HalfCheetah environment: learning curves, statistical significance, effect size, classification accuracy, and success rate. Asterisks indicate statistical significance (* p < 0.05).

Figure 5. Detailed analysis for the HalfCheetah environment: learning curves, statistical significance, effect size, classification accuracy, and success rate. Asterisks indicate statistical significance (* p < 0.05).

While Classification-BEAR achieved statistically significant improvement in the sparse-reward MountainCar environment (p < 0.001, d = 8.91), absolute success rates remained modest for all methods. This reflects a broader limitation of offline RL when datasets lack sufficient high-reward trajectories. The classification-based constraint depends on quality comparisons that require adequate representation of successful actions. Future work may explore approaches such as reward shaping, hierarchical subgoal decomposition, limited online data supplementation, or data augmentation to enrich sparse datasets and improve generalization in these domains.

5.2.3. Mechanism Isolation and Efficiency Interpretation

The $O\left(n\right)$ computational efficiency and the pairwise comparison mechanism contribute independently to the observed improvements. The implementation-level gain arises from evaluating one pair per state instead of all $O\left(n^2\right)$ pairs, producing about a 3–5× speedup without affecting policy quality (Section 5.6, Appendix B.4). In contrast, the algorithmic improvement stems from quality-aware action selection, as pure Classification-BEAR consistently outperforms both BEAR and Hybrid variants.

The Hybrid baseline retains the same $O\left(n\right)$ computational structure yet performs worse in four of five environments, indicating that the gain arises from the quality comparison mechanism rather than implementation efficiency. A naive random-pairing ablation would destroy the quality-ranking property and yield confounded results; hence, the Hybrid comparison provides a cleaner means to isolate algorithmic contribution while preserving computational structure.

5.3. Statistical Analysis of Performance Improvements

We aimed to rigorously validate the superiority of Classification-BEAR not only through average performance comparison but also through formal statistical testing. Depending on the distributional characteristics and variance of each environment, we applied independent t-tests, Welch’s t-tests, and Wilcoxon rank-sum tests where appropriate. To prevent inflated error risk due to multiple comparisons, Bonferroni correction was incorporated, and the significance level was set to $\alpha = 0.05$. Importantly, rather than relying solely on p-values, we jointly evaluated the practical effect size using Cohen’s d, where $d \ge 0.8$ indicates a large effect and $d \ge 2.0$ indicates a very large effect.

The final normalized return performance (mean ± standard deviation) for each environment is summarized in

Table 4. Normalized return performance (mean ± std) across environments.

Environment	BEAR	Classification-BEAR (The Proposed)	Observed Effect
Pendulum	−213 ± 18	−187 ± 12	−198 ± 16
MountainCar	45 ± 4	52 ± 3	48 ± 5
HalfCheetah	3163 ± 252	3424 ± 187	3351 ± 201

, while the detailed significance testing results are provided separately in Appendix B. Figure 6 presents an overall comparative visualization across the three environments (Pendulum, MountainCar, and HalfCheetah), clearly showing that our method consistently achieves higher mean performance with lower variance relative to baselines. Subsequently, Figure 7 (Pendulum), Figure 8 (MountainCar), and Figure 9 (HalfCheetah) provide more fine-grained visualizations of the return distribution and policy stability, confirming that Classification-BEAR maintains a clear performance advantage even in the later phases of training.

The MountainCar environment exhibited the strongest signal, with $p < 0.001$ and Cohen’s d exceeding $8.91$, indicating overwhelming statistical and practical superiority. In HalfCheetah, the comparison against BEAR yielded $p = 0.040$ with a large effect size (d = 1.04), demonstrating both statistical significance and practically meaningful improvement. Although Pendulum reported p = 0.071, the large effect size $(d = 5.00)$ suggests a favorable trend approaching statistical significance. Additionally, coefficient of variation (CV) analysis confirmed that Classification-BEAR achieved the lowest average variability across all environments (approximately $17\%$), demonstrating not only high performance but also excellent reproducibility across repeated runs.

In summary, Classification-BEAR consistently and decisively outperformed existing offline reinforcement learning algorithms across all three dimensions—statistical significance, effect size, and learning stability. These findings are further reinforced by extended analyses presented in Appendix B. Moreover, additional locomotion experiments on Hopper and Walker2d are included in Appendix B, where the same superiority trend is consistently observed. Further distributional and stability-oriented analyses are provided in Appendix B, offering visual confirmation of reproducibility and safety under extended evaluation.

5.4. Hyperparameter Sensitivity

We analyzed sensitivity to key hyperparameters.

Table 5. Hyperparameter sensitivity analysis.

Parameter	Role	Stable Range	Observed Effect
$\mathrm{Temperature} (\tau$)	Controls label sharpness	0.6~0.8	Too low → unstable; too high → slow learning
$\mathrm{Label} \mathrm{smoothing} (\alpha$)	Regularizes classifier	0.05~0.2	High values slow convergence
$\mathrm{Quality} \mathrm{weight} (\lambda$)	Constraint strength	0.5~1.0	Linear effect on conservatism
Learning rates (η, β)	Update speed	${10}^{-4}–{10}^{-3}$	Minor impact within range

summarizes observed effects and stable ranges.

Performance remained stable when parameters varied within moderate ranges. Temperature τ and smoothing α exerted the most noticeable effects, while other parameters showed minor impact. This robustness reduces hyperparameter tuning requirements compared with BEAR’s kernel selection and bandwidth tuning, supporting practical deployability across diverse environments.

5.5. Integrated Performance Visualization

Figure 10, Figure 11, Figure 12 and Figure 13 provide integrated visualizations that complement the detailed analyses, summarizing Classification-BEAR’s performance across multiple dimensions. Figure 10 presents overall performance improvements in two perspectives: Quantitative improvements showing absolute score gains of +146 (Pendulum), +38 (MountainCar), and +261 (HalfCheetah) over BEAR-Original and the relative improvement rates of 570%, 106%, and 292%, respectively, demonstrating substantial performance gains across diverse environment characteristics.

Figure 11, Figure 12 and Figure 13 present environment-specific distribution analyses, success rates, and statistical significance visualizations. Each figure includes violin plots showing reward distribution characteristics, success rate comparisons against predefined thresholds, and statistical test results. Figure 11 (Pendulum) shows Classification-BEAR achieved the highest average reward with narrow distribution indicating consistency, although statistical significance was marginal (p = 0.071) with large effect size (d = 5.00). The success rate reached 60% compared with BEAR’s 20%. Figure 12 (MountainCar) demonstrates very strong statistical significance (p < 0.001, d = 8.91) despite challenging sparse rewards. Classification-BEAR maintained the highest average performance with the most consistent distribution, although absolute success rates remained modest across all methods due to environment difficulty. Figure 13 (HalfCheetah) confirms significant improvements (p = 0.040, d = 1.04) in high-dimensional settings with a 50–60% success rate. The narrow reward distribution and high success rate demonstrate stable policy quality and robust learning dynamics even in complex robotic control tasks.

Together, these visualizations demonstrate that Classification-BEAR achieves consistent performance improvements, stable learning dynamics, and strong statistical support across environments with diverse characteristics—validating the effectiveness of quality-based constraints as a practical alternative to distributional matching in offline RL.

5.6. Comprehensive Comparison with State-of-the-Art Offline RL Methods and Practical Efficiency Evaluation

Section 5.1, Section 5.2, Section 5.3, Section 5.4 and Section 5.5 quantitatively demonstrated that Classification-BEAR holds structural advantages over BEAR-family algorithms in terms of performance, stability, and statistical effect sizes. However, to assess real-world applicability in offline reinforcement learning, one must consider not only performance metrics but also “how quickly and with how few resources the target performance can be achieved.”

Figure 14 presents an integrated comparison of training time, peak memory usage, scalability, and performance–efficiency balance among BEAR-Original, Hybrid, and Classification-BEAR in the HalfCheetah environment. These results clearly demonstrate that Classification-BEAR not only achieves higher performance but also possesses deployment-ready efficiency suitable for practical applications.

Building on this confirmed efficiency advantage, we expanded our comparison beyond the BEAR family to include state-of-the-art practical offline RL methods: CQL, IQL, and TD3 + BC. CQL employs value-based pessimism through conservative Q-value penalties, IQL applies implicit value regularization via expectile regression, and TD3 + BC combines policy improvement with weighted behavior cloning. Since these methods utilize fundamentally different mechanisms from Classification-BEAR’s support-filtering constraint, this comparison reveals practical gaps across algorithmic paradigms.

All methods were evaluated under strictly identical conditions: D4RL datasets (HalfCheetah/Hopper/Walker2d), random seeds {42, 123, 456, 789, 999}, hardware (RTX 3080), and uniform hyperparameter tuning budgets of eight configurations per method. For compatibility reasons, the replay buffer was loaded via a direct dataset interface rather than importing the D4RL package, but it is strictly identical to the official D4RL dataset specification.

Figure 15 visualizes the trade-off between convergence speed (Time-to-80%) and 30 min cumulative performance (AUC@30m). CQL converges quickly but suffers from low AUC due to early performance collapse, IQL achieves high final performance but with excessively slow convergence, and TD3 + BC shows average performance on both metrics. Classification-BEAR secures both speed and performance, positioning itself at the Pareto-optimal point—representing the most ideal efficiency balance for practical deployment. However, depending on the application context and the required level of risk aversion, more conservative approaches such as TD3 + BC may still be preferable in certain scenarios.

Figure 16 compares learning convergence patterns over time. CQL rises quickly but exhibits unstable oscillations after mid-training, IQL shows excessively slow ascent during the initial 5–15 min window, and TD3 + BC demonstrates pronounced performance saturation in later stages. In contrast, Classification-BEAR maintains both rapid initial convergence and monotonically stable patterns through late training, demonstrating the most effective resolution of the trade-off between early efficiency and long-term stability. Accordingly, this study does not claim that Classification-BEAR holds absolute superiority on a single metric but rather demonstrates that it serves as a highly effective option in practical settings where a balance between rapid initial convergence and long-term performance is required.

6. Discussion

6.1. Key Contributions and Implications

This study introduces a paradigm shift in offline RL constraint design, moving from distributional safety (“stay within data”) to quality-based selection (“choose better actions”). Three significant contributions emerge.

Reducing complexity from $O\left(n^2\right)$ to $O\left(n\right)$ achieves 3× training speedup and 57% memory reduction, enabling scalability to large datasets and real-time applications. This addresses a critical bottleneck in practical offline RL deployment, where computational resources often limit batch sizes and iteration speed.

Classification accuracy provides intuitive, real-time monitoring of learning progress. Unlike opaque MMD values that offer limited insight, accuracy (70–85% across experiments) directly indicates how reliably the system distinguishes action quality, facilitating debugging and deployment decisions in production environments.

Consistent improvements across diverse environments (simple control, sparse rewards, and high-dimensional robotics) with uniform hyperparameters demonstrate robustness. The approach achieves strong results despite structural simplicity, supporting Occam’s razor principles [43]: simpler mechanisms can be more effective than complex alternatives.

The finding that pure classification constraints outperformed the tested hybrid implementation reveals implementation-specific challenges when combining distributional safety and quality-based objectives. In our architecture, enforcing distributional proximity (via MMD) appeared to prevent exploitation of higher-quality actions that the classifier identified within the data support. While this suggests potential gradient conflicts in naive combinations of these constraints, it does not preclude more sophisticated hybrid designs that could leverage complementary strengths of both paradigms.

6.2. Limitations and Future Directions

Several limitations warrant acknowledgment and suggest promising research directions:

By removing explicit distributional constraints, Classification-BEAR may occasionally favor actions with less data support. While our experiments remained stable, formal analysis of out-of-distribution action frequency and safety guarantees compared with conservative baselines (CQL and IQL) requires further investigation. Quantifying this trade-off between performance and safety represents important future work.

High classification accuracy (70–85%) does not guarantee proportional policy improvement. This reflects the fundamental challenge that local pairwise comparisons may not fully capture global policy optimality. Investigating mechanisms to better align local comparison accuracy with global performance—such as weighting pairs by importance or incorporating transitivity constraints explicitly—could enhance the approach.

We used simple one-pair-per-state sampling. Alternative strategies warrant exploration: (1) multiple pairs per state to improve classifier coverage, (2) prioritized sampling by Q-difference magnitude to focus on informative comparisons, (3) adaptive selection based on learning stage to balance exploration and exploitation, and (4) curriculum learning that progressively increases comparison difficulty.

While Section 3 established convergence properties under standard assumptions, formal finite-sample guarantees and tighter sample complexity bounds remain open problems. Establishing PAC-style bounds for classification-based offline RL would strengthen theoretical foundations.

Extension to discrete action spaces (using softmax over pairwise scores), partially observable environments (incorporating recurrence in classifiers), multi-agent settings (pairwise comparisons across agents), and hierarchical RL (quality comparison at multiple temporal abstractions) represents promising directions for future work.

While our pure classification approach outperformed the tested hybrid, alternative combinations with conservative methods (CQL’s Q-penalties for classifier training) or adaptive distributional constraints (MMD only when classifier uncertainty is high) may achieve better safety–performance balance through careful design.

The λ parameter exhibits environment-dependent optimal values, requiring per-environment tuning for peak performance. While we fixed λ = 1.0 for fair comparison, this represents a limitation compared with distribution-free methods. Future work could explore adaptive λ-selection mechanisms based on online performance metrics or dataset characteristics.

We acknowledge that the hybrid baseline (combining distributional and classification constraints) was not exhaustively tuned. The potential for conflicting loss signals between MMD and classification objectives may have hindered its performance. A more systematic exploration of hybrid architectures could reveal synergistic benefits we did not observe in our initial experiments.

During mid-training, the classifier may label a high portion of actions as out of distribution (≈80–95%), reflecting temporary uncertainty rather than unsafe behavior. This rate decreases substantially by convergence (≈20–35%). While our experiments showed no instability or failure, deployments in safety-critical domains (e.g., autonomous driving and medical decision support) should implement additional safeguards, such as runtime OOD confidence monitoring and restricting use to fully trained policies.

These limitations and directions provide a roadmap for advancing classification-based approaches toward broader applicability and stronger theoretical foundations.

7. Conclusions

This study demonstrates that complex distributional constraints in offline reinforcement learning can be effectively addressed with simple pairwise action quality comparison. Classification-BEAR achieves three simultaneous objectives: improved performance, enhanced computational efficiency, and increased interpretability through transparent classification accuracy monitoring. This method also outperformed state-of-the-art methods (CQL, IQL, and TD3BC) while achieving 3× on average and up to 5× in some environments.

The paradigm shift from distributional safety to direct quality comparison represents a promising direction for offline RL research. This work might introduce a complementary and orthogonal constraint axis that prioritizes quality-aware action selection within the supported data. By demonstrating that simple quality comparison can replace complex distributional matching, we contribute both theoretical insights and practical tools for advancing offline RL beyond laboratory settings toward real-world deployment. Future work should explore enhanced classifier designs, adaptive λ-selection mechanisms, and combinations with imagination-limited Q-learning, as well as extensions to discrete actions or multi-agent settings-enabling dynamic balancing between distributional safety and quality prioritization depending on task properties. The classification-based paradigm offers a path forward for bridging the gap between academic research and industrial applications, where simplicity, efficiency, and interpretability are paramount.