Enhancing Adversarial Policy Learning via Value-Based Reward Shaping

Abstract

In adversarial reinforcement learning, designing dense reward functions is a traditional approach to address the sparsity of adversarial objectives. However, conventional reward design often relies on high-quality domain knowledge and may fail in practice, thereby inducing objective misalignment—a discrepancy between optimizing the designed reward and achieving the true adversarial utility. To reduce this discrepancy, a Value-Based Reward Shaping (VBRS) framework is proposed. VBRS integrates an intrinsic state-value estimate, which is a dynamic predictor of long-term utility, into the immediate reward function. As a result, exploration can be encouraged toward states predicted to be strategically advantageous, potentially avoiding some local optima in practice. Experiments demonstrate that VBRS outperforms a baseline that relies solely on the original reward function. The results confirm that the proposed method enhances adversarial performance and helps bridge the gap between designed reward guidance and the adversarial objective.

1. Introduction

Modern autonomous intelligent systems are required to make decisions in complex adversarial environments, with applications including robotics [1], autonomous driving [2], and multi-agent collaboration [3]. Adversarial decision-making represents a fundamental challenge in domains such as close-range unmanned aerial combat [4] and real-time strategy games [5]. A distinguishing characteristic of these domains is that agents must confront intelligent opponents within highly coupled interactions, while the opponent policy and the environment dynamics are unknown [6].

Traditional approaches often rely on rule-based controllers or tactical libraries designed by experts to approximate optimal behaviors [7]. Although effective in predictable scenarios, these static policies inherently fail to capture the complexity of unknown strategic objectives. Jiang et al. [8] attempted to learn opponent strategies through imitation learning and then applied dynamic programming. Nevertheless, data scarcity of trajectories in adversarial environments renders it impractical to reconstruct the environment and opponent policies [9].

Reinforcement Learning (RL) [10] provides a mechanism to optimize policies through direct interactions. Algorithms based on policy gradients [11], such as PPO [12], TRPO [13], and SAC [14], typically maximize a cumulative reward signal. In adversarial tasks, the true objective is defined by sparse goals, such as winning or losing [15,16]. Since optimizing solely on sparse signals is computationally inefficient, dense handcrafted rewards are frequently introduced to accelerate learning [17,18]. However, in adversarial settings, such task-specific rewards may deviate from the true adversarial utility and induce policy learning bias [19]. A fundamental discrepancy arises here: handcrafted reward functions are approximations that inevitably deviate from the true strategic objective [20,21].

Existing approaches, such as potential-based reward shaping (PBRS), enable reward shaping to improve the learning signal while preserving policy invariance under certain conditions. Classic extensions further relax the assumption of a fixed hand-crafted potential by allowing the potential function to change over time while maintaining PBRS-style guarantees, e.g., dynamic potential-based reward shaping [22]. Besides that, online learning of shaping rewards has been proposed to adapt shaping signals from experience, mitigating the need for expert-specified potentials [23]. Eureka leverages large language models to iteratively generate and refine executable rewards, but it still optimizes a proxy reward specification and offers no direct safeguard against misalignment with the true win/loss objective in adversarial settings [24]. Choi and Kim [25] demonstrate that incorporating a safety potential as a shaping signal can improve learning in autonomous driving. It has also been shown that PBRS can be reformulated as an equivalent initialization of action values, which has provided additional intuition on how state-dependent potentials influence learning dynamics without introducing net cyclic gains [26]. Recent bootstrapped reward shaping (BSRS) reduces reliance on expert-crafted potentials by using the agent’s current value estimate as the PBRS potential to densify sparse feedback [27], but it primarily targets exploration speed in sparse-reward settings and does not explicitly address objective misalignment in coupled adversarial interactions. Exploration-Guided Reward Shaping (EXPLORS) [28] integrates self-supervised intrinsic rewards with exploration bonuses to facilitate learning in environments with extremely sparse or noisy signals. Similarly, Automatic Intrinsic Reward Shaping (AIRS) addresses the biased-objective issue caused by auxiliary intrinsic rewards by adaptively selecting intrinsic shaping functions from a predefined set based on online return estimation [29]. More recently, the utilization of shaping rewards has been explicitly studied, where shaping signals have been adaptively leveraged while potentially unhelpful components are suppressed during training [30].

However, these approaches primarily target the scarcity of feedback by synthesizing auxiliary exploration signals or selecting from a pool of intrinsic modules. They do not explicitly address reward misalignment between artificial reward functions and the true strategic objective in adversarial settings. In such scenarios, the key challenge is not merely exploring the state space to obtain feedback, but correcting the optimization bias introduced by imperfect artificial priors that may actively mislead the agent away from the global optimum. Consequently, a critical unresolved issue remains: how can the discrepancy between handcrafted reward design and the true adversarial objective be reduced, such that optimizing rewards remains consistent with optimal adversarial goals?

To address this issue, a Value-Based Reward Shaping method (VBRS) is proposed. This method is designed to empirically mitigate the mismatch between optimizing a handcrafted reward and achieving the true adversarial objective in adversarial environments. The core mechanism integrates a value function, which serves as a dynamic estimate of long-term strategic utility, into the reward structure. By modulating the influence of handcrafted rewards based on value estimates, the gap between manually designed dense guidance and sparse adversarial outcomes is reduced. As training progresses and the value estimate improves, the bias introduced by imperfect reward engineering is partially rectified. The main contributions of this work can be summarized as follows:

• A Value-Based Reward Shaping (VBRS) framework is proposed to mitigate reward misalignment. By incorporating value-function guidance into reward shaping, exploration is driven by long-term strategic value rather than biased short-term heuristics.
• A two-dimensional adversarial simulation environment is constructed. Experimental results demonstrate that, compared with other baselines, VBRS achieves superior adversarial performance.

The remainder of this paper is organized as follows: Section 2 formalizes the adversarial decision-making problem. Section 3 analyzes the reward misalignment problem and describes the proposed value-based reward shaping method. Section 4 describes the experimental setup and discusses the results. Section 5 concludes the paper and outlines future research directions.

2. Problem Formulation

To address the core question raised in Section 1—how can the discrepancy between artificial reward design and the true adversarial objective be reduced, such that optimizing a handcrafted reward becomes more consistent with adversarial goals in practice?—the adversarial engagement between an ego agent A and a fixed opponent B is formalized. Both agents act simultaneously in the environment, producing a joint trajectory:

$$\tau =(s_0,a_0^A,a_0^B,s_1,a_1^A,a_1^B,\dots ,s_T).$$

At each time step t, both agents select actions $a_t^A\in {\mathcal{A}}_A$ and $a_t^B\in {\mathcal{A}}_B$ based on the current state $s_t\in \mathcal{S}$. State transitions follow a joint dynamics model:

$$s_{t+1}\sim P_{\mathrm{env}}(·\mid s_t,a_t^A,a_t^B).$$

The opponent agent B follows a fixed policy:

$$a_t^B\sim {\pi }_B(·\mid s_t).$$

2.1. Markov Decision Process

Since ${\pi }_B$ is fixed, the adversarial learning problem for agent A is modeled as a standard Markov Decision Process (MDP) [31]:

$$\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma ),$$

where

• $\mathcal{S}$ is the state space, encompassing the status of both agents and relevant environmental information.
• $\mathcal{A}$ is the action space of agent A.
• $P(s^{\prime }\mid s,a^A)$ is the state transition probability induced by the environment dynamics and the opponent’s policy, calculated as

  $$P(s^{\prime }\mid s,a^A)=\sum _{a^B\in {\mathcal{A}}_B}{P}_{\mathrm{env}}(s^{\prime }\mid s,a^A,a^B){\pi }_B(a^B\mid s).$$
• $r(s,a^A)$ is the immediate reward function for agent A.
• $\gamma \in [0,1)$ is the discount factor.

The behavior of agent A is defined by a policy $\pi \left(a^A\right|s)$, representing the probability of taking action $a^A$ in state s. Given a policy $\pi$, the value function $V_{\pi }\left(s\right)$ measures the expected cumulative discounted reward starting from state s:   

$$V_{\pi }\left(s\right)={\mathbb{E}}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k}\mid S_t=s\right],$$

(1)

where ${\mathbb{E}}_{\pi }$ denotes expectation over trajectories induced by policy $\pi$. The action-value function $Q_{\pi }(s,a)$ is

$$Q_{\pi }(s,a)={\mathbb{E}}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k}\mid S_t=s,A_t=a\right].$$

(2)

The optimal action-value function $Q^{*}(s,a)$ satisfies

$$Q^{*}(s,a)\ge Q_{\pi }(s,a),\forall s\in \mathcal{S},a\in \mathcal{A},\pi \in \Pi ,$$

(3)

where $\Pi$ denotes the set of all policies. The optimal policy is

$${\pi }^{*}\left(s\right)=\arg \underset{a\in \mathcal{A}}{\max }{Q}^{*}(s,a).$$

(4)

2.2. Policy Gradient Framework and Objective Analysis

Drawing upon theoretical results suggesting that reinforcement learning methods are fundamentally connected to policy gradient approaches [32,33,34], our algorithm is formulated in the policy gradient framework. The objective function is defined as the expected return:

$$\mathcal{J}\left(\theta \right)={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^T{\gamma }^t{r}_t\right].$$

(5)

$\mathcal{J}\left(\theta \right)$ is optimized via gradient ascent:

$${\nabla }_{\theta }\mathcal{J}\left(\theta \right)={\mathbb{E}}_{{\pi }_{\theta }}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(a^A\mid s)A_{\pi }(s,a^A)\right],$$

(6)

where $A_{\pi }(s,a)$ is the advantage function. The goal is to find the optimal policy parameters ${\theta }^{*}=\arg {\max }_{\theta }\mathcal{J}\left(\theta \right)$.

In adversarial tasks, the true objective is typically a sparse outcome reward (e.g., win/loss), which is difficult to optimize directly. To address this, practitioners use a designed dense reward function r. Unfortunately, the reward function often suffers from misalignment.

The existence of an ideal adversarial reward function, denoted as $r^{*}$, which perfectly captures the strategic structure of the task, is hypothesized. The corresponding ideal objective is

$${\mathcal{J}}^{*}\left(\theta \right)={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^T{\gamma }^t{r}_t^{*}\right].$$

(7)

Let ${\theta }^{*}=\arg {\max }_{\theta }{\mathcal{J}}^{*}\left(\theta \right)$ be the parameters of the ideal policy. For a generic handcrafted reward r, the resulting policy ${\theta }^l=\arg {\max }_{\theta }\mathcal{J}\left(\theta \right)$ is typically suboptimal with respect to the ideal objective:

$${\mathcal{J}}^{*}\left({\theta }^l\right)\le {\mathcal{J}}^{*}\left({\theta }^{*}\right).$$

(8)

Since $r^{*}$ is unknown, the goal is to perform reward shaping on the existing reward r, such that the optimized policy $\theta$ approximates the performance of the ideal policy, while minimizing the required interaction samples and enabling rapid learning of effective counter-strategies.

3. Methodology

In this section, the Value-Based Reward Shaping (VBRS) framework is proposed. A value-based reward shaping mechanism aimed at overcoming the discrepancy between handcrafted reward design and the true adversarial objective is introduced, followed by an analysis of the properties of the shaped advantages. Then, an implementation using a dual-critic Actor–Critic framework integrated with Generalized Advantage Estimation and an adaptive $\beta$-regularization schedule is presented. It is important to note that although we instantiate the actor update with PPO in this paper, PPO is not essential to VBRS. The main contribution of VBRS lies in the critic-side design—using intrinsic value estimates to construct a shaped reward/advantage signal that provides more informative learning guidance. Therefore, the actor component can be replaced by other compatible policy optimization schemes while retaining the same critic-driven shaping mechanism.

The overall architecture of the proposed VBRS framework is illustrated in Figure 1. As shown in the diagram, the framework comprises three integral components: the environment interaction module, the dual-critic evaluation mechanism consisting of the Main Value Network and the Auxiliary Value Network, and the actor policy update module. Notably, the actor update component is algorithm-agnostic and can be implemented using any value-based policy optimization method, such as traditional AC, Advantage Actor–Critic (A2C) [35], PPO, or SAC.

3.1. Value-Based Reward Shaping Objective

The agent is trained to maximize the expected discounted return under the designed environmental reward $r(s,a)$, as formulated in Section 2. In adversarial tasks, $r(s,a)$ is often constructed by combining sparse outcome rewards with heuristic dense terms. As a result, local improvements in $r(s,a)$ may not reliably correspond to progress toward the ideal adversarial objective.

To reduce this discrepancy, a shaped reward is defined by mixing the preset reward with a state-value signal that captures long-term task utility. For a fixed shaping coefficient $\beta \in [0,1]$, the shaped reward is defined as

$$r^{\prime }(s,a)=(1-\beta )r(s,a)+\beta V_{\pi }\left(s\right),$$

(9)

where $V_{\pi }\left(s\right)$ denotes the value function under policy $\pi$ with respect to the original reward scale. Since $V_{\pi }$ propagates delayed outcome information through Bellman recursion, the term $\beta V_{\pi }\left(s\right)$ serves as a dense signal that reflects the long-term desirability of visiting s.

The corresponding shaped objective is

$${\mathcal{J}}^{\prime }\left(\theta \right)={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^T{\gamma }^t{r}^{\prime }(s_t,a_t)\right].$$

(10)

Optimizing ${\mathcal{J}}^{\prime }\left(\theta \right)$ encourages both the immediate reward $r(s_t,a_t)$ and the visitation of states with high long-horizon utility measured by $V_{\pi }\left(s_t\right)$. The strength of this encouragement is controlled by $\beta$, which is required to balance the value term and the original reward. It should be noted that the shaping term $V\left(s\right)$ has the same physical scale as a discounted return. When it is injected into every time step, the cumulative shaped objective contains an additional state-occupancy term ${\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[{\sum }_t{\gamma }^{t}V\left(s_t\right)\right]$, which generally re-weights trajectories and may be interpreted as intentionally emphasizing visits to states that are predicted to lead to long-horizon success. This design is not introduced as a policy-invariant reward transformation. Instead, the trade-off between the original reward and the value-based shaping signal is controlled explicitly by the balancing coefficient $\beta$.

The shaping term changes the effective state-occupancy weighting, VBRS intentionally biases optimization toward states predicted to be strategically valuable. Accordingly, we evaluate alignment via outcome-level metrics (e.g., win rate) as evidence that proxy-induced optimization bias is reduced.

3.2. Dual-Critic Estimation and Adaptive Shaping Coefficient

The objective in Equation (10) is optimized by estimating the value terms with two critics. A Main Critic is used to estimate state values under the original reward. An Auxiliary Critic is used to estimate values under the shaped reward and to provide advantage estimates for actor optimization.

The Main Critic $V(s;\omega )$ is trained on the original reward $r_t=r(s_t,a_t)$ using the squared TD error

$${\mathcal{L}}_V\left(\omega \right)=\mathbb{E}\left[{\left(r_t+\gamma V(s_{t+1};\omega )-V(s_t;\omega )\right)}^2\right].$$

(11)

After the Main Critic has been updated, the shaped reward is instantiated by replacing $V_{\pi }\left(s_t\right)$ in Equation (9) with the Main Critic output. To avoid unintended gradient coupling, the value term is treated as a detached scalar shaping signal:

$$r_t^{\prime }=(1-{\beta }_k)r_t+{\beta }_k\mathrm{stopgrad}\left(V(s_t;\omega )\right),$$

(12)

where ${\beta }_k$ is held constant within iteration k for the trajectories collected in that iteration.

Consider a single trajectory $\tau ={\left\{(s_t,a_t,r_t^{\prime })\right\}}_{t=0}^{T-1}$ with the terminal or truncation state $s_T$. The Auxiliary Critic $V^{\prime }(s;{\omega }^{\prime })$ is trained under the shaped reward $r_t^{\prime }$ using a masked bootstrap target. Define the termination indicator $d_{t+1}\in \{0,1\}$, where $d_{t+1}=1$ if the episode terminates after step t, and $d_{t+1}=0$ for time-limit truncation so that bootstrapping is enabled.

Specifically, the bootstrap value is

$${\tilde{V}}_{t+1}^{\prime }=(1-d_{t+1})V^{\prime }(s_{t+1};{\omega }^{\prime }),$$

(13)

and the Auxiliary Critic is optimized by minimizing the squared TD error:

$${\mathcal{L}}_{V^{\prime }}\left({\omega }^{\prime }\right)=\mathbb{E}\left[{\left(r_t^{\prime }+\gamma {\tilde{V}}_{t+1}^{\prime }-V^{\prime }(s_t;{\omega }^{\prime })\right)}^2\right].$$

(14)

Advantages for the actor are computed from the Auxiliary Critic using GAE. The shaped TD error is

$${\delta }_t^{\prime }=r_t^{\prime }+\gamma {\tilde{V}}_{t+1}^{\prime }-V^{\prime }(s_t;{\omega }^{\prime }),$$

(15)

and the shaped GAE advantage is computed as

$${\widehat{A}}_t^{\prime }=\sum _{l=0}^{T-1-t}{\left(\gamma \lambda \right)}^l{\delta }_{t+l}^{\prime },$$

(16)

where $\lambda \in [0,1]$ is the GAE parameter.

The shaping coefficient ${\beta }_k$ is introduced to control the magnitude of value-based shaping, since value estimates can be unreliable in early training. An adaptive schedule is constructed from the Main Critic loss. Let

$$L_k=\mathbb{E}\left[{\left(r_t+\gamma V(s_{t+1};\omega )-V(s_t;\omega )\right)}^2\right]$$

(17)

denote the batch TD loss after the Main Critic update at iteration k. To reduce noise, an exponential moving average is maintained:

$${\overline{L}}_k=(1-\kappa ){\overline{L}}_{k-1}+\kappa L_k,\kappa \in (0,1].$$

(18)

A confidence score is defined as

$$c_k=\exp \left(-{\displaystyle \frac{{\overline{L}}_k}{L_{\mathrm{scale}}}}\right)\in (0,1],$$

(19)

where $L_{\mathrm{scale}}$ is a fixed scale for normalization. The target shaping coefficient is set to

$${\beta }_k^{★}={\beta }_{\min }+({\beta }_{\max }-{\beta }_{\min })c_k,$$

(20)

and the applied coefficient is updated smoothly and with a per-iteration change limit:

$$\begin{array}{cc}\hfill {\tilde{\beta }}_{k+1}& =(1-\tau ){\beta }_k+\tau {\beta }_k^{★},\tau \in (0,1],\hfill \\ \hfill {\beta }_{k+1}& =\mathrm{clip}\left({\tilde{\beta }}_{k+1},{\beta }_k-\Delta {\beta }_{\max },{\beta }_k+\Delta {\beta }_{\max }\right),\hfill \\ \hfill {\beta }_{k+1}& =\mathrm{clip}\left({\beta }_{k+1},0,{\beta }_{\max }\right).\hfill \end{array}$$

(21)

3.3. Actor Optimization with PPO

PPO is adopted to optimize the policy parameters $\theta$. The coupling between PPO and VBRS is restricted to the advantage estimator: the PPO surrogate objective is evaluated using the shaped advantages computed under the shaped reward function. The actor optimizer can be replaced by other policy optimization schemes without modifying the critic-driven shaping mechanism, as long as the update is driven by advantage estimates.

Let ${\pi }_{{\theta }_{\mathrm{old}}}$ denote the behavior policy and consider a single trajectory $\tau ={\left\{(s_t,a_t,r_t^{\prime })\right\}}_{t=0}^{T-1}$ with variable length T. The shaped TD errors ${\left\{{\delta }_t^{\prime }\right\}}_{t=0}^{T-1}$ are computed from the Auxiliary Critic according to Equation (15), with the termination/time-limit bootstrapping rule described in Section 3.2. The shaped advantages ${\left\{{\widehat{A}}_t^{\prime }\right\}}_{t=0}^{T-1}$ are then obtained by GAE truncated at the end of the trajectory as in Equation (16).

Given ${\widehat{A}}_t^{\prime }$, the PPO likelihood ratio is defined as

$${\rho }_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}(a_t\mid s_t)}}.$$

(22)

The clipped surrogate objective for the single trajectory $\tau$ is

$$L_{\tau }^{\mathrm{CLIP}}\left(\theta \right)={\displaystyle \frac{1}{T}}\sum _{t=0}^{T-1}\min \left({\rho }_t\left(\theta \right){\widehat{A}}_t^{\prime },\mathrm{clip}({\rho }_t\left(\theta \right),1-{ϵ}_{\mathrm{clip}},1+{ϵ}_{\mathrm{clip}}){\widehat{A}}_t^{\prime }\right),$$

(23)

this clipping limits the effective policy change: when ${\widehat{A}}_t^{\prime }>0$, the update is capped if ${\rho }_t\left(\theta \right)>1+{ϵ}_{\mathrm{clip}}$; when ${\widehat{A}}_t^{\prime }<0$, the update is capped if ${\rho }_t\left(\theta \right)<1-{ϵ}_{\mathrm{clip}}$. and the entropy bonus is computed over the same trajectory:

$$L_{\tau }^{\mathrm{ENT}}\left(\theta \right)={\displaystyle \frac{1}{T}}\sum _{t=0}^{T-1}\mathcal{H}\left({\pi }_{\theta }(·\mid s_t)\right).$$

(24)

The final PPO objective for $\tau$ is

$$L_{\tau }^{\mathrm{PPO}}\left(\theta \right)=L_{\tau }^{\mathrm{CLIP}}\left(\theta \right)+{\beta }_{\mathrm{ent}}{L}_{\tau }^{\mathrm{ENT}}\left(\theta \right),$$

(25)

and the policy parameters are updated by stochastic gradient ascent:

$$\theta \leftarrow \theta +{\alpha }_{\pi }{\nabla }_{\theta }{L}_{\tau }^{\mathrm{PPO}}\left(\theta \right).$$

(26)

During the actor update, ${\widehat{A}}_t^{\prime }$ is treated as a constant with respect to $\theta$, and no gradients are propagated into the critic networks through the advantage estimator.

For a batch of trajectories, the same construction is applied to each trajectory and the objectives are averaged over all collected time steps, which is equivalent to replacing the factor ${\displaystyle \frac{1}{T}}{\sum }_{t=0}^{T-1}$ by ${\displaystyle \frac{1}{{\sum }_i{T}_i}}{\sum }_i{\sum }_{t=0}^{T_i-1}$. The complete process of the algorithm is shown in Algorithm 1.

Algorithm 1 PPO + VBRS:Proximal Policy Optimization with Value-Based Reward Shaping

1: Initialize actor parameters $\theta$, Main Critic parameters $\omega$, Auxiliary Critic parameters ${\omega }^{\prime }$. 2: Set ${\beta }_0=0$, ${\overline{L}}_0=0$, and hyperparameters $\gamma ,\lambda ,{\alpha }_{\pi },{\alpha }_V,{\alpha }_{V^{\prime }}$. 3: Set PPO hyperparameters $K,B,{ϵ}_{\mathrm{clip}},{\beta }_{\mathrm{ent}}$. 4: Set shaping hyperparameters ${\beta }_{\min },{\beta }_{\max },\kappa ,\tau ,L_{\mathrm{scale}},\Delta {\beta }_{\max }$. 5: for iteration $k=0,1,\dots$ do 6:      Set ${\theta }_{\mathrm{old}}\leftarrow \theta$. 7:      Collect trajectories ${\mathcal{D}}_k$ using ${\pi }_{{\theta }_{\mathrm{old}}}$: 8:      Update $\omega$ by using ${\mathcal{D}}_k$: $\omega \leftarrow \omega -{\alpha }_V{\nabla }_{\omega }{\mathcal{L}}_V$. 9:      Compute shaped rewards for all steps in ${\mathcal{D}}_k$: 10:       $r_t^{\prime }\leftarrow (1-{\beta }_k)r_t+{\beta }_k\mathrm{stopgrad}\left(V(s_t;\omega )\right)$. 11:    Update Auxiliary Critic $V^{\prime }(s;{\omega }^{\prime })$ by minimizing Equation (14) with learning rate ${\alpha }_{V^{\prime }}$. 12:    Update ${\overline{L}}_k\leftarrow (1-\kappa ){\overline{L}}_{k-1}+\kappa L_k$ and compute $c_k$ as in Section 3.2. 13:    Update ${\beta }_{k+1}$ using Equations (20) and (21). 14:    Compute shaped TD errors ${\delta }_t^{\prime }$ by Equation (15). 15:    Compute shaped advantages ${\widehat{A}}_t^{\prime }$ by Equation (16). 16:    for epoch $=1,\dots ,K$ do 17:       for minibatch $\mathcal{B}\subset {\mathcal{D}}_k$ of size B do 18:         Compute ${\rho }_t\left(\theta \right)$ by Equation (22) for $t\in \mathcal{B}$. 19:         Compute $L^{\mathrm{PPO}}\left(\theta \right)$ on $\mathcal{B}$ by Equations (23)–(25) using ${\widehat{A}}_t^{\prime }$. 20:         Update actor by gradient ascent: 21:            $\theta \leftarrow \theta +{\alpha }_{\pi }{\nabla }_{\theta }{L}^{\mathrm{PPO}}\left(\theta \right)$. 22:       end for 23:    end for 24: end for

4. Experiment

4.1. Environment Setting

To evaluate the effectiveness of the proposed method, a two-dimensional adversarial simulation environment is constructed. All experiments are performed in a Cartesian plane defined over the region $[0,15]\times [0,15]$. Two static rectangular obstacles are placed inside the environment, whose axis-aligned bounding boxes are given by

$${\mathcal{O}}_1:x\in [4,8],y\in [7,11],{\mathcal{O}}_2:x\in [10,13],y\in [3,6].$$

(27)

Obstacles occlude the line-of-sight between the two agents and thus prevent successful aiming. When an agent collides with an obstacle, the velocity undergoes specular reflection with a damping effect. Specifically, let ${\mathit{v}}_{\mathrm{ref}}$ denote the mirror-reflected post-impact velocity. The actual post-collision velocity is defined as

$${\mathit{v}}^{+}={\kappa }_v{\mathit{v}}_{\mathrm{ref}},$$

(28)

where ${\kappa }_v\in (0,1)$ is a constant damping factor and we set ${\kappa }_v=0.8$.

Agents A and B share the same kinematic constraints and weapon configurations. At the beginning of each episode, agent A is initialized uniformly at random within the region $[8,15]\times [8,15]$ with zero initial velocity. Agent B is initialized within $[0,5]\times [0,5]$, also with zero initial velocity. A schematic illustration of the environment is provided in Figure 2. Each agent is equipped with a short-range weapon characterized by a maximum attack distance $r_{\max }=5$ m and a maximum attack angle ${\psi }_{\max }$. From each agent’s own perspective, its attack angle is defined as the absolute angle between its heading direction and the line connecting itself to its target. Specifically, the ego agent’s attack angle is denoted by $\psi$, and the opponent’s attack angle is denoted by $\varphi$. Each agent maintains an internal heading unit vector ${\mathit{h}}^t$. At episode initialization, the agent has an initialized heading ${\mathit{h}}^0$. When $\parallel {\mathit{v}}^t\parallel > 0$, the heading is aligned with the velocity direction. When $\parallel {\mathit{v}}^t\parallel = 0$, the heading is set to the last valid heading: ${\mathit{h}}^t={\mathit{h}}^{t-\Delta t}$. In particular, at $t=0$, the heading equals the initialized heading ${\mathit{h}}^0$. Attack angles $\psi$ and $\varphi$ are computed using ${\mathit{h}}^t$ so that the angle is well-defined even when the velocity magnitude is zero.

An agent is considered to satisfy the firing condition if and only if (i) the target lies within the maximum attack angle, (ii) the line-of-sight distance does not exceed $r_{\max }$, and (iii) no obstacle blocks the ray between the two agents. Formally, the firing condition is unified as

$$C_f=\{r\le r_{\max },\alpha \le {\psi }_{\max },I_{\mathrm{obs}}=0\},$$

(29)

where $\alpha$ denotes the agent’s own attack angle (i.e., $\alpha =\psi$ for agent A and $\alpha =\varphi$ for agent B), and $I_{\mathrm{obs}}=1$ indicates the presence of an obstacle between the two agents.

An episode terminates when either agent satisfies the firing condition $C_f$ or when the maximum episode length is reached. If both agents satisfy $C_f$ simultaneously or no firing event occurs before the time limit, the result is considered a draw. Otherwise, the agent satisfying $C_f$ first is regarded as the winner. In this paper, each episode corresponds to one trajectory $\tau$.

Each agent obeys second-order point-mass dynamics with control input represented by a two-dimensional acceleration vector $\mathit{a}=[a_x,a_y]$. The maximum allowable acceleration and velocity are constrained by ${\parallel \mathit{a}\parallel }_{\infty }\le a_{\max }$ and ${\parallel \mathit{v}\parallel }_{\infty }\le v_{\max }$, respectively. Let the state at time t be ${\mathit{p}}^t=[x^t,y^t]$ and ${\mathit{v}}^t=[v_x^t,v_y^t]$. The motion evolves in discrete time with step size $\Delta t$ using a semi-implicit Euler method:

We denote by $\mathrm{clip}(\mathit{x},x_{\max })$ the element-wise clipping operator that maps each component of $\mathit{x}$ to $[-x_{\max },x_{\max }]$. In our implementation, the policy first samples an unconstrained raw acceleration action ${\mathit{a}}^{t,\mathrm{raw}}$ from the Gaussian policy, and we enforce the acceleration bound using element-wise clipping:

$${\mathit{a}}^t=\mathrm{clip}({\mathit{a}}^{t,\mathrm{raw}},a_{\max }).$$

(30)

The environment transition uses the clipped action ${\mathit{a}}^t$, while PPO log-probabilities are evaluated w.r.t. the raw sampled action ${\mathit{a}}^{t,\mathrm{raw}}$ (see Algorithm 1).

$${\mathit{v}}^{t+\Delta t}=\mathrm{clip}\left({\mathit{v}}^t+{\mathit{a}}^t\Delta t,v_{\max }\right),$$

(31)

$${\mathit{p}}^{t+\Delta t}={\mathit{p}}^t+{\mathit{v}}^{t+\Delta t}\Delta t.$$

(32)

The complete environment state is defined as

$$s=[{\mathit{p}}_A^t,{\mathit{p}}_B^t,{\mathit{v}}_A^t,{\mathit{v}}_B^t,I_{\mathrm{env}}^t],$$

(33)

where ${\mathit{p}}_A^t$ and ${\mathit{p}}_B^t$ are the positions of agent A and agent B, respectively. The environment encoding $I_{\mathrm{env}}^t$ is a fixed vector that contains (i) obstacle positions and (ii) boundary information represented by the four corner points of the arena.

4.2. Reward Function Design

The reward function, denoted as r, consists of a terminal reward and a shaping reward [36]. This reward is implemented as the default environment reward across all methods, serving as the baseline reward for PPO, as well as the starting point for our proposed shaping mechanism. For a fair comparison, all methods employ PPO as the actor optimization algorithm. The terminal reward is defined as follows:

$$R=\left\{\begin{array}{cc}{R}_{\mathrm{win}},\hfill & \mathrm{if}\mathrm{the}\mathrm{ego}\mathrm{agent}\mathrm{satisfies}{C}_f\mathrm{and}\mathrm{the}\mathrm{opponent}\mathrm{does}\mathrm{not},\hfill \\ R_{\mathrm{lose}},\hfill & \mathrm{if}\mathrm{the}\mathrm{opponent}\mathrm{satisfies}{C}_f\mathrm{and}\mathrm{the}\mathrm{ego}\mathrm{agent}\mathrm{does}\mathrm{not},\hfill \\ R_{\mathrm{draw}},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(34)

The heuristic shaping reward is designed to provide dense tactical feedback that encourages the ego agent to (i) reduce its own attack angle $\psi$, (ii) increase the opponent’s attack angle $\varphi$, and (iii) maintain an appropriate engagement distance r. To obtain a bounded and numerically stable signal, a cosine-based angular term is adopted. Since $\psi ,\varphi \in [0,{\psi }_{\max }]$ in our setting and $\cos (·)$ is monotonically decreasing on $[0,\pi ]$, a smaller angle corresponds to a larger cosine value. Accordingly, the angular shaping reward is defined as

$$R_a(\psi ,\varphi )=\cos \psi -\cos \varphi ,$$

(35)

which yields a higher reward when the ego agent achieves a smaller attack angle while forcing the opponent to a larger attack angle.

To regulate spatial engagement, we further modulate the same tactical preference by a distance-dependent window centered at the safety radius $r_{\min }$:

$$R_d\left(r\right)=(\cos \psi -\cos \varphi )\exp \left(-{\displaystyle \frac{{(r-r_{\min })}^2}{2{\sigma }^2}}\right),$$

(36)

where $r_{\min }$ specifies the preferred safety distance, and $\sigma$ controls the width of the distance window.

The overall shaping reward is

$$R_{\mathrm{self}}=w_a{R}_a+w_d{R}_d,$$

(37)

where $w_a$ and $w_d$ are scalar coefficients that balance angular preference and distance regularization, respectively.

4.3. Opponent Policy Training

To obtain a competent opponent policy, an iterative training procedure is adopted for agent B. For both our agent A and the opponent B, the policy network and the value network are modeled as fully connected feedforward neural networks with one input layer, five hidden layers of 128 neurons each, and one output layer, using ReLU activation.

Importantly, both agents employ stochastic Gaussian policies. Specifically, the action is sampled from

$${\pi }_{\theta }\left(a\right|s)=\mathcal{N}\left({\mu }_{\theta }\left(s\right),{\sigma }_{\theta }{\left(s\right)}^2\right),$$

(38)

where ${\mu }_{\theta }\left(s\right)$ and ${\sigma }_{\theta }\left(s\right)$ are the state-dependent mean and standard deviation produced by the policy network. The log-standard-deviation is learned and bounded for numerical stability by clamping $\log {\sigma }_{\theta }\left(s\right)\in [-20,2]$. At interaction time, we first sample an unconstrained raw action $a_t^{\mathrm{raw}}\sim \mathcal{N}({\mu }_{\theta }\left(s_t\right),{\sigma }_{\theta }{\left(s_t\right)}^2)$, and then enforce the acceleration bounds using element-wise clipping: $a_t=\mathrm{clip}(a_t^{\mathrm{raw}},-a_{\max },a_{\max })$. We store $a_t^{\mathrm{raw}}$ and its log-probability $\log {\pi }_{\theta }(a_t^{\mathrm{raw}}\mid s_t)$ in the rollout buffer, while the environment transition uses the clipped action $a_t$. Accordingly, the PPO likelihood ratio is computed w.r.t. the same raw action distribution, ensuring consistency between sampling and optimization.

To obtain a competent opponent, we refine agent B’s policy ${\pi }_B$ via an auxiliary training partner, agent C. Specifically, agent C is introduced to provide interactive training signals for agent B, enabling ${\pi }_B$ to become progressively more effective through self-play style refinement. Agent B interacts with agent C to refine its policy ${\pi }_B$ through three stages:

• Stage 1 Agent C remains stationary, allowing agent B to learn basic aiming.
• Stage 2 Agent C moves in straight lines with reflection, providing moving targets.
• Stage 3 (Self-Play): Agent C adopts a frozen copy of agent B’s current policy. This process is repeated for 15 iterations.

For all subsequent evaluations in Section 4.4, the opponent agent employs the final converged policy obtained at the end of Stage 3 and remains frozen throughout testing. To ensure reproducibility, we fix the environment random seed and the opponent-policy sampling seed for each evaluation run, and we report results aggregated over 10 independent training seeds.

4.4. Experimental Results

To evaluate the effectiveness of the proposed VBRS algorithm, we report two complementary metrics. (1) Win rate against the opponent. We use the win rate against the opponent policy (the best-performing fixed opponent defined in Section 4.3) as the primary metric to reflect the final adversarial effectiveness of the learned policy. (2) Estimated return under the preset handcrafted environment reward. In addition, we report an estimate of the original objective $\mathcal{J}\left(\theta \right)$, the expected discounted cumulative return under the preset reward $r_t$ defined in Equations (34)–(37). This metric is reported to highlight that maximizing the handcrafted reward objective is not necessarily equivalent to maximizing adversarial performance, and a bias may exist between the two.

Every 50 episodes with the opponent, we freeze the current policy parameters and run a separate evaluation phase. The evaluation data are not used for learning updates; they are used only to compute the win rate and to estimate $\mathcal{J}\left(\theta \right)$ under the original reward. Each evaluation consists of $N_{\mathrm{eval}}=200$ episodes. An episode is counted as win/lose/draw according to the termination rule in Section 4, and the win rate is computed as

$$\mathrm{WinRate}={\displaystyle \frac{N_{\mathrm{win}}}{N_{\mathrm{eval}}}}.$$

(39)

During the evaluation phase at iteration k, we estimate the original discounted return using Monte-Carlo rollouts collected under the frozen policy:

$$\widehat{\mathcal{J}}\left({\theta }_k\right)={\displaystyle \frac{1}{|{\mathcal{D}}_k^{\mathrm{eval}}|}}\sum _{\tau \in {\mathcal{D}}_k^{\mathrm{eval}}}\sum _{t=0}^{T\left(\tau \right)-1}{\gamma }^t{r}_t,$$

(40)

where ${\mathcal{D}}_k^{\mathrm{eval}}$ is the set of evaluation trajectories and $T\left(\tau \right)$ is the horizon of trajectory $\tau$.

Four baselines are considered as follows:

(1) PPO with original reward. The first control group uses the preset environment reward $r_t$ and updates the actor using PPO.

(2) PBRS + PPO. The second control group applies potential-based reward shaping (PBRS) to the original reward and then performs PPO-based actor updates. Specifically, PBRS uses the standard shaping form

$$r_t^{\mathrm{PBRS}}=r_t+\gamma \Phi \left(s_{t+1}\right)-\Phi \left(s_t\right),\Phi \left(s_T\right)=0,$$

(41)

where we adopt a handcrafted state potential

$$\Phi \left(s\right)=k_{\varphi }\left(a_1\left(\pi -\psi \left(s\right)\right)+a_2\left(r_{\max }-r\left(s\right)\right)\right).$$

(42)

Here, $\psi \left(s\right)$ is the ego attack angle and $r\left(s\right)$ is the inter-agent distance, $a_1,a_2,r_{\max }$ are environment constants, and we set $k_{\varphi }=2, a_1=3, a_2=2$ in our experiments.

(3) RND + PPO. The third control group augments the original reward with an intrinsic shaping signal based on Random Network Distillation (RND) [37] and updates the actor with PPO.

(4) VBRS with fixed $\beta$. The fourth control group is an ablation of VBRS: while VBRS updates $\beta$ via an exponential moving average, this baseline fixes $\beta =0.05$ throughout training.

(5) VBRS. The experimental group employs the full VBRS algorithm and updates the actor using PPO.

For a fair comparison, all methods share the same network architecture, the same PPO-based update pipeline, and the same common hyperparameters (e.g., learning rates, number of updates, batch sizes, and evaluation settings), as summarized in

Table 1. Hyperparameter settings.

Description	Symbol	Value
General Training Parameters
Discount factor	$\gamma$	$0.96$
GAE parameter	$\lambda$	$0.95$
PPO clipping parameter	${ϵ}_{\mathrm{clip}}$	$0.2$
Entropy regularization coefficient	${\beta }_{\mathrm{ent}}$	$0.01$
Trajectory buffer size (steps)	N	2048
Minibatch size	B	128
PPO epochs per update	K	6
Optimizer	–	Adam
Policy/Value learning rates	${\alpha }_{\pi },{\alpha }_V,{\alpha }_{V^{\prime }}$	$3\times {10}^{-4},1\times {10}^{-3},1\times {10}^{-3}$
State/action dimensions	$d_s,d_a$	$8,2$
Total training episodes (per run)	–	20,000
Number of training runs (seeds)	–	10
Maximum episode length	$T_{\max }$	2000
Evaluation Parameters
Console print interval (episodes)	$F_{\mathrm{print}}$	100
Win-rate record interval (episodes)	$F_{\mathrm{win}}$	50
Win-rate window size (episodes)	$W_{\mathrm{win}}$	100
Avg. discounted reward record
interval (episodes)	$F_{\mathrm{ret}}$	50
Policy Network Parameters
Policy distribution	–	Gaussian (Equation (38))
Log-std clamp range	–	$[-20,2]$
Hidden layers	–	$5\times 128$ (MLP)
Activation	–	ReLU
VBRS-Specific Parameters
Initial shaping coefficient	${\beta }_0$	0
Minimum shaping coefficient	${\beta }_{\min }$	$0.0$
Maximum shaping coefficient	${\beta }_{\max }$	$0.05$
EMA update rate for TD loss	$\kappa$	$0.1$
Smoothing rate toward target ${\beta }_k^{★}$	$\tau$	$0.1$
TD-loss normalization scale	$L_{\mathrm{scale}}$	${10}^6$
Max per-update change of $\beta$	$\Delta {\beta }_{\max }$	$0.002$
Environment Parameters
Simulation time step	$\Delta t$	$0.3$ s
Weapon maximum range	$r_{\max }$	5 m
Maximum attack angle
(ego/opponent)	${\psi }_{\max }$	$\pi /4$ rad
Reward coefficients	$w_a,w_d$	$6,4$
Terminal rewards (Win/Lose/Draw)	R	$1200/-900/-600$
Safety radius	$r_{\min }$	1 m
Distance normalization factor	$\sigma$	$1.5$ m
Maximum velocity (per-axis)	$v_{\max }$	3 m/s
Maximum acceleration (per-axis)	$a_{\max }$	1 m/s2
Collision velocity damping factor	${\kappa }_v$	$0.8$
Kinetic-energy loss ratio	$1-{\kappa }_v^2$	$0.36$

.

Figure 3a compares VBRS with four control baselines. Across training, VBRS consistently attains higher win rates, suggesting an empirical reduction of proxy-induced bias toward the adversarial outcome objective in our setting. In the early phase (0–4 k episodes), Figure 4a shows that VBRS achieves a faster win-rate improvement than other baselines (PPO, PBRS, and RND) during the early training phase. Figure 3b visualizes the EMA-updated $\beta$ in the VBRS, which gradually increases and stabilizes near the target range. Moreover, as $\beta$ gradually increases in Figure 3b, the performance gain becomes more pronounced, indicating that the value-based shaping signal provides effective guidance for learning and steers the policy toward strategically advantageous behaviors earlier.

In the final convergence window (16 k–20 k), Figure 4b shows that VBRS stabilizes at a markedly higher win rate than other baselines.

Table 2. Final win rate over the last 10 evaluation points. For each seed, we first average the last 10 recorded win-rate values, then report the mean across seeds with a 95% confidence interval (t-interval) and the standard deviation (SD) across seeds.

Method	Final Win Rate (95% CI)	SD
Without VBRS (PPO)	71.41% [67.88, 74.94]%	4.93
PBRS	79.08% [76.59, 81.57]%	3.48
RND	75.66% [72.46, 78.86]%	4.47
VBRS (fixed $\beta =0.05$)	86.06% [83.29, 88.83]%	3.43
VBRS (dynamic $\beta :0\to 0.03$)	83.22% [80.60, 85.84]%	3.66
VBRS (dynamic $\beta :0\to 0.05$)	88.62% [86.62, 90.62]%	3.58
VBRS (dynamic $\beta :0\to 0.10$)	80.17% [75.75, 84.59]%	6.18

summarizes the final win rates by averaging the last 10 evaluation points per seed and then reporting mean ± 95% CI across seeds. The PPO Without VBRS achieves $71.41\%$ (95% CI: $[67.88,74.94]\%$), while the PPO with VBRS with a dynamic schedule $\beta :0\to 0.05$ achieves $88.62\%$ (95% CI: $[86.62,90.62]\%$), i.e., an absolute gain of $17.21\%$ in win rate. Among the shaping baselines, PBRS improves over PPO to $79.08\%$ (95% CI: $[76.59,81.57]\%$), whereas RND yields $75.66\%$ (95% CI: $[72.46,78.86]\%$), both remaining below VBRS. The fixed-$\beta$ ablation (VBRS with $\beta =0.05$ fixed) performs worse than the dynamic-$\beta$ variant, supporting the necessity of adapting $\beta$ during training.

We conduct hypothesis tests on the final win rates against the PPO baseline using seed-paired runs ($n=10$). For each method, we first average the last 10 evaluation points within each seed, and then apply a two-sided Wilcoxon signed-rank test on the paired seed-level results. As summarized in

Table 3. Statistical tests on the final win rate against the PPO baseline. Final win rate is computed by averaging the last 10 evaluation points for each seed. Since runs are seed-paired ($n=10$), we use a two-sided Wilcoxon signed-rank test. Effect size is reported as Cohen’s $d_z$ on paired differences. Stars: ** $p<0.01$.

Method	Test	p (vs. PPO)	Effect	Effect Type
VBRS ($\beta :0\to 0.03$)	Wilcoxon	0.00195 **	1.858	Cohen’s $d_z$
VBRS (Fixed $\beta =0.05$)	Wilcoxon	0.00195 **	2.374	Cohen’s $d_z$
VBRS ($\beta :0\to 0.05$)	Wilcoxon	0.00195 **	3.209	Cohen’s $d_z$
PBRS	Wilcoxon	0.00586 **	1.427	Cohen’s $d_z$
VBRS ($\beta :0\to 0.10$)	Wilcoxon	0.0645	0.803	Cohen’s $d_z$
RND	Wilcoxon	0.0840	0.663	Cohen’s $d_z$

, VBRS with a dynamic schedule $\beta :0\to 0.05$ yields a statistically significant improvement over PPO ($p=0.00195$) with a large effect size (Cohen’s $d_z=3.209$). PBRS also shows a significant gain ($p=0.00586$), while RND and VBRS with ${\beta }_{\max }=0.10$ are not statistically significant at the 0.05 level. Overall, the tests corroborate that the performance advantage of VBRS (especially with ${\beta }_{\max }=0.05$) is robust across random seeds.

Figure 5 further studies the effect of different ${\beta }_{\max }$ values. We observe that ${\beta }_{\max }=0.05$ yields the best overall win-rate performance, while too small ($0.03$) weakens the shaping effect and too large ($0.10$) may introduce excessive bias, leading to inferior convergence.

An interesting phenomenon emerges when comparing the win rate in Figure 3a, with the original objective $\mathcal{J}\left(\theta \right)$ in Figure 6. While VBRS improves the win rate, the resulting policy can attain a lower value of the original objective $\mathcal{J}\left(\theta \right)$ than the non-shaping baseline and policy-invariant shaping methods. This discrepancy empirically supports the premise in Section 3: the preset reward $r_t$ may be imperfectly aligned with the true adversarial goal. The PPO baseline greedily optimizes the dense heuristic components of $r_t$, which yields reasonable reward scores but does not necessarily secure victories. In contrast, VBRS leverages value estimates to prioritize states with a higher winning probability, potentially sacrificing short-term heuristic points in favor of the strategic objective.

5. Conclusions

This paper studies reward misalignment in adversarial reinforcement learning, where handcrafted dense rewards may deviate from the true task-level outcome objective and introduce optimization bias. To mitigate this issue, we proposed a Value-Based Reward Shaping (VBRS) framework that injects an intrinsic state-value estimate into the immediate reward structure. By adaptively blending the original environment reward with a learned long-horizon utility signal, VBRS encourages exploration and policy improvement toward states that are predicted to be strategically advantageous, rather than being overly driven by short-term heuristic terms. Results in a two-agent competitive simulation environment show that VBRS improves training efficiency and achieves higher final outcome performance (e.g., win rate) compared with baselines using the original reward or alternative shaping strategies, suggesting that value-based shaping can be a practical heuristic to alleviate reward-induced optimization bias in adversarial learning.

Despite these advantages, VBRS has several limitations. First, the adaptive shaping schedule introduces hyperparameters that may require tuning across tasks, and the best operating range of the shaping coefficient can be environment-dependent. Second, our evaluation focuses on learning against a fixed opponent policy; in settings with non-stationary or adaptive opponents, the learning dynamics and the stability of value-based shaping may change. Third, the current validation is conducted in a simplified low-dimensional simulation; extending to more complex observations, higher-dimensional control, or partially observable settings may require additional stabilizing techniques and systematic ablation studies.

Future work will explore several directions. First, we plan to develop more rigorous mathematical formulations to quantify the discrepancy between the shaped reward and the underlying task objective, with the goal of providing analytical or provable measures of how reward shaping affects optimality and policy bias. Second, we will extend VBRS beyond the current two-agent setting to broader multi-agent scenarios, investigating how value-based shaping interacts with non-stationarity, strategic coupling, and scalable training in competitive multi-agent benchmarks.