Learning Deceptive Strategies in Adversarial Settings: A Two-Player Game with Asymmetric Information

Abstract

This study explores strategic deception and counter-deception in multi-agent reinforcement learning environments for a police officer–robber game. The research is motivated by real-world scenarios where agents must operate with partial observability and adversarial intent. We develop a suite of progressively complex grid-based environments featuring dynamic goals, fake targets, and navigational obstacles. Agents are trained using deep Q-networks (DQNs) with game-theoretic reward shaping to encourage deceptive behavior in the robber and intent inference in the police officer. The robber learns to reach the true goal while misleading the police officer, and the police officer adapts to infer the robber’s intent and allocate resources effectively. The environments include fixed and dynamic layouts with varying numbers of goals and obstacles, allowing us to evaluate scalability and generalization. Experimental results demonstrate that the agents converge to equilibrium-like behaviors across all settings. The inclusion of obstacles increases complexity but also strengthens learned policies when guided by reward shaping. We conclude that integrating game theory with deep reinforcement learning enables the emergence of robust, deceptive strategies and effective counter-strategies, even in dynamic, high-dimensional environments. This work advances the design of intelligent agents capable of strategic reasoning under uncertainty and adversarial conditions.

1. Introduction

In adversarial multi-agent systems, agents often operate in environments characterized by partial observability, uncertainty, and dynamic interactions [1,2,3]. In such contexts, deception becomes a powerful strategic tool, allowing agents to obscure their true intentions and influence the behavior of opponents. In such environments, deception emerges as a powerful strategic tool, enabling agents to obscure their true intentions and influence the behavior of opponents. This capability is crucial across a range of domains—from deceiving radar systems [4] and manipulating speech or audio signals [5] to deceptive path planning for autonomous robots [6] and strategic information manipulation [7]. These diverse applications collectively demonstrate that misleading an adversary can substantially alter the outcome of interactions [8,9,10]. Consequently, there has been a renewed surge of research aimed at understanding and emulating deception in autonomous systems, particularly in the context of learning-enabled agents. However, modeling and learning deceptive behavior in high-dimensional, multi-agent environments remain challenging due to the combinatorial complexity of agent interactions and the unpredictability of opponent strategies [11,12,13,14,15].

Previous research has addressed deception through classical game theory (e.g., [7]), optimal control (e.g., [16]), and multi-agent learning (e.g., [17]). Game-theoretic models have offered formal strategies for misleading opponents [18,19] but often under rigid assumptions and limited scalability [20]. Reinforcement learning, particularly in multi-agent settings, has emerged as a promising framework for learning adaptive behavior from interactions [21,22,23,24,25]. Recent advances have applied value-based learning methods to explore deceptive behavior [26,27], yet few works integrate game-theoretic reasoning with deep learning architectures in dynamic environments that feature multiple decoy goals and navigation constraints [28,29,30,31].

Deception often arises from information asymmetry, where an agent with superior knowledge can manipulate agents with limited information into believing a false reality or forming incorrect beliefs. In this work, we specifically focus on a problem [29] characterized by inherent information asymmetry and investigate whether learning-enabled agents can exploit their information advantage to construct deceptive strategies. Additionally, we examine whether agents aware of their informational disadvantage can develop counter-deceptive strategies to avoid being misled. We propose a multi-agent learning framework guided by game-theoretic principles to address information and knowledge asymmetry among agents.

The proposed game-theory-driven learning strategy offers several advantages over existing approaches to deception and counter-deception in multi-agent systems. First, by integrating game-theoretic reasoning directly into the learning process, our method allows agents to anticipate and strategically respond to the behavior of opponents, rather than relying solely on reactive or purely data-driven policies. This leads to more robust and interpretable strategies in adversarial settings. Second, the use of reward shaping informed by equilibrium concepts ensures that agents not only learn locally optimal behaviors but also align with globally coherent, stable outcomes. Unlike prior methods that often assume static environments or fixed opponent policies, our framework is designed to generalize across dynamic settings with changing goals, obstacle configurations, and opponent tactics. This adaptability enhances the strategic depth of the learned policies and improves sample efficiency during training. Finally, the integration of deception and counter-deception within a unified framework enables agents to both mislead and recognize deception, fostering richer and more resilient interactions in high-stakes domains such as autonomous security and multi-agent coordination.

This research introduces a multi-agent learning framework that combines deep Q-function learning with reward shaping grounded in game theory to enable deception and counter-deception strategies. We design progressively complex grid-based environments with varying numbers of goals and obstacles to evaluate the agents’ ability to generalize.

Our key contributions include (i) the development of a game-theory-driven learning framework for modeling deception in multi-agent environments, (ii) the design of scalable environments with dynamic goal and obstacle placement, (iii) the formulation of reward functions that align local agent behavior with global strategic objectives, and (iv) empirical validation showing that the agents converge to stable, equilibrium-like behaviors across diverse scenarios, along with an ablation study highlighting the contribution of each component.

The remainder of the paper is organized as follows: in Section 2, we formalize the problem; in Section 3, we introduce our game-theoretic reward-shaping mechanism; Section 4 details the training procedure; and in Section 5, we present and analyze our simulation results.

2. Problem Statement

2.1. Police Surveillance Coordinator vs. Robber Game

Our problem is motivated by a scenario in which a robber attempts to escape from a city. There are multiple exits from the city, but the robber has already selected a specific one (see Figure 1). A police control room supervisor monitors the robber’s movements from a control room but does not know which exit the robber has chosen. The supervisor’s objective is to predict the robber’s selected exit and deploy reinforcements accordingly.

The supervisor has a limited number of reinforcement units available. At each time step, as the robber moves through the city, the supervisor deploys one unit of reinforcement to the exit that the supervisor currently believes the robber is heading toward. When the robber ultimately reaches the exit, the total reinforcement stationed at that location depends on two factors: the time elapsed until the robber’s arrival and the number of times the supervisor deployed reinforcements to that particular exit. This outcome captures the interaction between the robber’s path-selection strategy (which affects travel time) and the supervisor’s reinforcement-deployment strategy over time.

The abovementioned problem is a classical case of games with asymmetric information—the supervisor does not know the robber’s true exit choice, whereas the robber is aware of the supervisor’s uncertainty. The robber recognizes that he is being observed and that the supervisor’s strategy is influenced by his movements. Therefore, rather than proceed directly to his intended exit, he may initially move toward alternative exits to generate ambiguity. This misleading movement constitutes deception—it may cause the supervisor to misallocate resources and reinforce the incorrect exit (exaggeration can be a form of deception. However, naive exaggeration may prove detrimental to the agent rather than beneficial. For example, in our scenario, if the robber exaggerates excessively by taking too long to reach the exit, the supervisor may have sufficient time to reinforce the correct exit, even by allocating reinforcements randomly or uniformly across all exits). Meanwhile, the supervisor must infer the robber’s true intention from his ambiguous behavior and deploy reinforcements strategically—ideally predicting the correct exit despite the deception.

The objective of this article is to demonstrate how reinforcement learning, together with concepts from game theory, can be combined to learn deceptive strategies that leverage the information superiority of one agent over another.

2.2. Mathematical Formulation

We formulate the problem as a game on a discrete-time, grid-based environment defined over a finite two-dimensional lattice $\mathcal{G}\subset {\mathbb{Z}}^2$ of size $w\times h$. Each cell $(i,j)\in \mathcal{G}$ represents the robber’s possible location, with $i\in \{0,\dots ,w\}$ and $j\in \{0,\dots ,h\}$. The environment comprises two learning agents—the robber ($\mathsf{R}$) and the police officer ($\mathsf{P}$)—and a set of n goal locations

$$G=\{{\mathit{g}}_1,{\mathit{g}}_2,\dots ,{\mathit{g}}_n\}\subset \mathcal{G},$$

and a set of k static obstacles

$$O=\{{\mathit{o}}_1,{\mathit{o}}_2,\dots ,{\mathit{o}}_k\}\subset \mathcal{G}.$$

All goals lie in free cells; that is, $G\cap O=\varnothing$. Exactly one goal is the true goal, denoted ${\mathit{g}}_T$, and the remaining goals form the set of fake goals, $G_F=G\setminus \left\{{\mathit{g}}_T\right\}$ (Figure 2).

Let ${\mathbf{x}}_{\mathsf{R}}\left(t\right)\in \mathcal{G}$ denote the robber’s position at time $t\in {\mathbb{N}}_0$, and his action set is

$$A_{\mathsf{R}}=\{\mathtt{up},\mathtt{down},\mathtt{left},\mathtt{right}\}.$$

For each goal ${\mathit{g}}_i$, let $p_i\left(t\right)\in {\Re }_{+}$ denote the police reinforcement allocated at time t, and define the police-reinforcement vector as

$${\mathbf{x}}_{\mathsf{P}}\left(t\right):=[p_1\left(t\right),p_2\left(t\right),\dots ,p_n\left(t\right)].$$

The police officer’s action set is denoted by

$$\begin{array}{c}\hfill A_{\mathsf{P}}=\left\{[a_1,a_2,\dots ,a_n]\in {\{0,1\}}^n|\sum _{i=1}^n{a}_i=1\right\}.\end{array}$$

(1)

The environment is assumed to be deterministic: an action $a\in A_{\mathsf{R}}$ moves the robber one cell in the chosen direction, and an action $a\in A_{\mathsf{P}}$ updates the reinforcement vector by

$${\mathbf{x}}_{\mathsf{P}}(t+1)={\mathbf{x}}_{\mathsf{P}}\left(t\right)+a\left(t\right).$$

We denote the state of the game at time t by

$$\mathit{s}\left(t\right)=[{\mathbf{x}}_{\mathsf{R}}\left(t\right),{\mathbf{x}}_{\mathsf{P}}\left(t\right)]\in \mathcal{G}\times {\Re }_{+}^n,$$

(2)

which results in the state space being $\mathcal{S}=\mathcal{G}\times {\Re }_{+}^n$. Let ${\pi }_{\mathsf{R}}$ be the robber’s policy, where ${\pi }_{\mathsf{R}}(a\mid s)$ is the probability of selecting action $a\in A_{\mathsf{R}}$ in state $s\in \mathcal{S}$, and let ${\pi }_{\mathsf{P}}$ be the police officer’s policy over $A_{\mathsf{P}}$.

Define the terminal time to be the first instance when the robber reaches its true goal:

$$t_f({\pi }_{\mathsf{R}},{\pi }_{\mathsf{P}},\mathit{s}\left(0\right))=min\{t\mid {\mathbf{x}}_{\mathsf{R}}\left(t\right)={\mathit{g}}_T\},$$

which depends on the policy pair $\pi =({\pi }_{\mathsf{R}},{\pi }_{\mathsf{P}})$ and the initial state $\mathit{s}\left(0\right)$. The robber seeks to minimize the reinforcement at the true goal, $p_T\left(t_f\right)$, while the police officer seeks to maximize it. In game-theoretic form, this objective can be expressed as

$$\begin{array}{c}\hfill \underset{{\pi }_{\mathsf{R}}}{min}\underset{{\pi }_{\mathsf{P}}}{max}{\mathbb{E}}_{\sim \pi }\left[p_T\left(t_f\right)\right],\end{array}$$

(3)

where the expectation is taken with respect to the randomization in the agents’ strategies.

This formulation differs from classical min–max games because the police officer does not know which goal is true and therefore cannot optimize $p_T$ directly. We employ reinforcement learning techniques for both $\mathsf{R}$ and $\mathsf{P}$ to learn their policies ${\pi }_{\mathsf{R}}$ and ${\pi }_{\mathsf{P}}$. Our aim is to design reward structures so that $\mathsf{R}$ can exploit its informational advantage to deceive $\mathsf{P}$, while $\mathsf{P}$ learns counter-deception strategies to mitigate $\mathsf{R}$’s deception.

2.3. Ambiguity and Strategic Trade-Offs

Ambiguity is the linchpin of effective deception. As long as the police officer cannot distinguish among multiple equally plausible goals, it must hedge, slowing its response and creating openings for the robber. Conversely, the robber must trade off path optimality against deceptive detours—too much ambiguity costs time, and too little increases the risk of exposing the true goal.

In this adversarial setting, the robber’s policy must balance

$$\underset{\mathrm{Efficiency}}{\underbrace{\mathrm{Reach}\mathrm{true}\mathrm{goal}\mathrm{quickly}}}\mathrm{vs}.\underset{\mathrm{Deception}}{\underbrace{\mathrm{Maintain}\mathrm{goal}\mathrm{ambiguity}}},$$

while the police officer’s policy must trade

$$\underset{\mathrm{Exploitation}}{\underbrace{\mathrm{Commit}\mathrm{to}\mathrm{one}\mathrm{goal}\mathrm{to}\mathrm{maximize}\mathrm{reward}}}\mathrm{vs}.\underset{\mathrm{Exploration}}{\underbrace{\mathrm{Avoid}\mathrm{greedy}\mathrm{commitment}\mathrm{to}\mathrm{reduce}\mathrm{risk}}}.$$

By embedding these opposing objectives in a multi-agent reinforcement learning framework, we recover emergent equilibrium strategies that reflect both deception and counter-deception in complex, partially observed environments.

3. Game-Theoretic Reward Shaping

To foster rapid convergence and emergent strategic behavior, we enrich the baseline reinforcement-learning rewards with game-theoretic shaping terms that explicitly encode the adversarial structure of our multi-exit pursuit–evasion game. By penalizing or rewarding each agent according to both short-term navigation signals and inferred opponent intent, we bias exploration toward policies that exhibit robust deception (for the robber) or effective inference (for the police officer).

We borrow key concepts for the police officer-robber game from a related problem studied in [26]. Due to the complexity of the problem—particularly with the inclusion of obstacles and multiple goals—theoretical analysis in [26] is limited to obstacle-free grids with only one true goal and one decoy. Through our reward-shaping approach, we demonstrate that a deep Q-learning framework can successfully learn effective deception and counter-deception strategies in these more complex environments.

Before we present our detailed discussion on reward shaping, we summarize the key variables that will be used throughout this section in the frame below.

3.1. Robber Reward Shaping

We denote the robber’s per-step reward at time t by $r_{\mathsf{R}}\left(t\right)$. Intuitively, $r_{\mathsf{R}}\left(t\right)$ must serve three objectives simultaneously: (i) provide a strong incentive to reach the true exit, (ii) discourage invalid or aimless movements, and (iii) penalize lingering or predictable behavior that would make deception ineffective. More precisely, we consider the most recent resource allocation to the true goal for a time window of H, i.e., the quantity ${\left\{p_T(t-k)\right\}}_{k=1}^H$, where recall that $p_i\left(t\right)$ denotes $\mathsf{P}$’s allocation to goal i at time t (for $t<H$, the window horizon is limited to t). Using this allocation history, we define two reward-shaping quantities:

$$\begin{array}{c}\hfill r_{\mathrm{resource}}\left(t\right)=\sum _{k=1}^H\alpha \left(k\right)p_T(t-k),r_{\mathrm{time}}\left(t\right)=\lambda t,\alpha \left(k\right),\lambda >0,\end{array}$$

(4)

where $\alpha \left(k\right)$ weighs how strongly past actions of $\mathsf{P}$ should penalize the Robber, and $\lambda$ controls the cost of delayed exit. That is, the longer $\mathsf{R}$ takes to reach the goal, the higher the penalty $r_{\mathrm{time}}$. Notice that unlike most existing methods that assign a fixed pet-step cost (i.e., $r_{\mathrm{time}}=\lambda$) to motivate a shorter path length, we consider a linearly increasing path length.

The final reward is computed as

$$r_{\mathsf{R}}\left(t\right)=\left\{\begin{array}{c}{r}_{\mathrm{goal}}-r_{\mathrm{resource}}\left(t\right)-r_{\mathrm{time}}\left(t\right),\mathrm{if}{\mathbf{x}}_{\mathsf{R}}\left(t+1\right)=g_T,\hfill \\ -r_{\mathrm{resource}}\left(t\right)-r_{\mathrm{time}}\left(t\right)-{\mathbb{I}}_{\mathrm{Obstacles}}\left({\mathbf{x}}_{\mathsf{R}}(t+1)\right)r_{\mathrm{obs}}-{\mathbb{I}}_{\mathrm{Boundary}}\left({\mathbf{x}}_{\mathsf{R}}(t+1)\right)r_{\mathrm{bou}},\mathrm{else},\hfill \end{array}\right.$$

where $r_{\mathrm{goal}}>0$ is a large terminal bonus for reaching the goal, $r_{\mathrm{obs}}$ and $r_{\mathrm{bou}}$ penalize obstacle collisions or going out of the grid boundary, and the indicator functions activate only upon those events.

This formulation ensures that the robber is (i) strongly rewarded for successful escape, (ii) punished for reckless or non-progressing maneuvers, and (iii) discouraged from trajectories that allow the police officer to concentrate resources too heavily on the true exit too early.

3.2. Police Officer Reward Shaping

The police officer’s per-step reward, $r_{\mathsf{P}}\left(t\right)$, must encourage accurate inference of the robber’s hidden objective while maintaining coverage of alternative exits. We begin by measuring the change in (Manhattan) distance from the robber’s current position ${\mathbf{x}}_{\mathsf{R}}\left(t\right)$ to each exit ${\mathit{g}}_i$:

$$d_i\left(t\right)={∥{\mathbf{x}}_{\mathsf{R}}\left(t\right)-{\mathit{g}}_i∥}_1,\Delta d_i\left(t\right)=d_i\left(t\right)-d_i(t-1).$$

Notice that, due to the grid-world nature of the environment and the up/down/left/right movement, $\Delta d_i\left(t\right)$ can only take one of the two values: $-1$ and 1. Therefore, $\Delta d_i\left(t\right)$ is a direct indicator of whether the robber moved closer to the goal ${\mathit{g}}_i$ (i.e., when $\Delta d_i\left(t\right)=-1$) or moved further away. In general, one movement can lead to $\Delta d_i\left(t\right)$ being $-1$ for multiple goals. Therefore, to separate the goal(s) for which the distance was minimized by the robber, we define the set

$$\begin{array}{c}\hfill G_{min}=\{{\mathit{g}}_i\in G:\Delta d_i\left(t\right)=-1\}.\end{array}$$

$\mathsf{P}$’s allocation vector $a_t=[a_1\left(t\right),\dots ,a_n\left(t\right)]$ is then rewarded according to the following:

■

Unique Minimum: If $\left|G_{min}\right|=1$ with ${\mathit{g}}_j$ being the only goal in the set $G_{min}$, we define $\mathsf{P}$’s reward as

$$\begin{array}{c}\hfill r_{\mathsf{P}}\left(t\right)=\beta a_j\left(t\right)-\beta \sum _{\stackrel{i=1}{i\ne j}}^n{a}_i\left(t\right),\end{array}$$

(5)

where $\beta >0$ scales the incentive to concentrate force on the most likely exit. Notice that the allocation vector $a\left(t\right)$ satisfies the condition ${\sum }_{i=1}^n{a}_i\left(t\right)=1$ as mentioned in (1). Consequently, $r_{\mathsf{P}}$ in (5) can equivalently be written as

$$\begin{array}{c}\hfill r_{\mathsf{P}}\left(t\right)=\beta (2a_j\left(t\right)-1).\end{array}$$

This shows that at least half of the allocation must be assigned to the unique goal (i.e., $a_j\left(t\right)>0.5$) for which the robber decreased its distance to receive a non-zero reward. If $a_j\left(t\right)<0.5$, the reward is negative, implying that the police officer is being penalized.

It should be noted that in our earlier formulation, $a_i\left(t\right)\in \{0,1\}$ for each goal i (see (1)), which forces $\mathsf{P}$ to fully commit to the goal to receive a positive reward, as partial allocations are not permitted. To introduce exploration into $\mathsf{P}$’s strategy, one could relax this constraint and allow fractional allocations, i.e., $a_i\left(t\right)\in [0,1]$. While this adds flexibility to $\mathsf{P}$’s strategy, it also increases the complexity of the problem, as $\mathsf{P}$’s action space becomes continuous. Beyond this added complexity, such a formulation may be less practical, as in real-world settings, $\mathsf{P}$ may not be able to allocate arbitrary fractions of a unit to different goals. To strike a balance between practical applicability and strategic flexibility, we retain a discrete action space but expand the set of possible allocations beyond what was allowed in (1). A detailed description of $A_{\mathsf{P}}$ will be provided later in the simulation section.

■

Tied Minima: If $\left|G_{min}\right|>1$, we refine the decision by introducing a new quantity

$$\begin{array}{c}\hfill c_i\left(t\right)=d_i\left(t\right)+p_i\left(t\right)\end{array}$$

which denotes the sum of the current allocation at goal ${\mathit{g}}_i$ and the current distance to that goal. Essentially, $c_i\left(t\right)$ represents the maximum possible amount of reinforcement that could be allocated to goal ${\mathit{g}}_i$ if it were the true goal, assuming that $\mathsf{R}$ begins moving directly toward it and $\mathsf{P}$ fully commits its resources to that goal for the remainder of the time. In our framework, $d_i\left(t\right)$ is precisely the number of lattice steps needed to reach ${\mathit{g}}_i$, and the police officer receives exactly one unit of reinforcement per time step. Consequently, both the distance and allocation terms are already expressed in identical, naturally normalized units. The unweighted sum $d_i\left(t\right)+p_i\left(t\right)$ therefore captures the total number of remaining allocation steps required to saturate goal i if the police officer were to devote all future resources there from time t onward. In other scenarios, one may need to use a weighted sum $w_d{d}_i\left(t\right)+w_p{p}_i\left(t\right)$.

Let us define $c_{min}=min\{c_i\left(t\right):{\mathit{g}}_i\in G_{min}\}$ and the set for which $c_i$’s are the lowest as follows

$$\begin{array}{c}\hfill C_{min}=\{{\mathit{g}}_i\in G_{min}:c_i\left(t\right)=c_{min}\}\end{array}$$

Now, using $C_{min}$, we define $\mathsf{P}$’s reward as

$$r_{\mathsf{P}}\left(t\right)=\beta \left(\sum _{i\in C_{min}}{\displaystyle \frac{a_{t,i}}{\left|C_{min}\right|}}-\sum _{i\notin C_{min}}{a}_{t,i}\right),$$

encouraging the police officer to allocate proportionally among the most effective candidate goals. Finally, to penalize overcommitment to decoy exits, we grant a conservation bonus $r_{\mathrm{bonus}}>0$ whenever the robber has previously visited exit i and the police officer allocates zero mass there. Collectively, these shaping terms drive the police officer to rapidly infer the robber’s intended exit, discourage wasteful dispersion of resources, and maintain a calibrated balance between exploration and exploitation in the allocation strategy.

The entire reward computation process is presented in Algorithm 1.

Algorithm 1 One-Step Reward Computation
Require: robber_pos ${\mathbf{x}}_{\mathsf{R}}\left(t\right)$, goal_alloc ${\left\{{\mathbf{x}}_{\mathsf{P}}\left(s\right)\right\}}_{s=0}^{t-H}$, robber_action ${\mathbf{a}}_{\mathsf{R}}$, police_action ${\mathbf{a}}_{\mathsf{P}}=[a_1,\dots ,a_n]$
1:	$r_{\mathsf{R}}\leftarrow 0$   $r_{\mathsf{P}}\leftarrow 0$
2:	${\mathbf{x}}_{\mathsf{R}}(t+1)\leftarrow$ next_position(${\mathbf{x}}_{\mathsf{R}}\left(t\right),{\mathbf{a}}_{\mathsf{R}}$).				▹ Robber’s next position
3:	if  ${\mathbf{x}}_{\mathsf{R}}(t+1)=g_T$  then
4:		$r_{\mathsf{R}}\leftarrow r_{\mathsf{R}}+r_{\mathrm{goal}}$
5:	else if  ${\mathbb{I}}_{\mathrm{Boundary}}\left({\mathbf{x}}_{\mathsf{R}}(t+1)\right)$  then
6:		$r_{\mathsf{R}}\leftarrow r_{\mathsf{R}}-r_{\mathrm{bou}}$			▹ Out of grid boundary penalty
7:	else if  ${\mathbb{I}}_{\mathrm{Obstacles}}\left({\mathbf{x}}_{\mathsf{R}}(t+1)\right)$  then
8:		$r_{\mathsf{R}}\leftarrow r_{\mathsf{R}}-r_{\mathrm{inv}}$			▹ Obstacle collision penalty
9:	else if  ${\mathbf{x}}_{\mathsf{R}}(t+1)={\mathbf{x}}_{\mathsf{R}}\left(t\right)$  then
10:		$r_{\mathsf{R}}\leftarrow r_{\mathsf{R}}-r_{\mathrm{stay}}$			▹ Non-movement penalty
11:	end if
12:	compute $r_{\mathrm{resource}}\left(t\right)={\sum }_{k=1}^H\alpha \left(k\right)p_T(t-k)$ and $r_{\mathrm{time}}\left(t\right)=\lambda t$				▹ Equation (4)
13:	$r_{\mathsf{R}}\leftarrow r_{\mathsf{R}}-\left(r_{\mathrm{resource}}+r_{\mathrm{time}}\right)$				▹ robber reward
14:	$d_i\left(t\right)\leftarrow \parallel {\mathbf{x}}_{\mathsf{R}}\left(t\right)-{\mathit{g}}_i\parallel ,d_i(t-1)\leftarrow \parallel {\mathbf{x}}_{\mathsf{R}}(t-1)-{\mathit{g}}_i\parallel$ for all ${\mathit{g}}_i\in G$.
15:	Compute $\Delta d_i=d_i\left(t\right)-d_i(t-1)$ for all goals ${\mathit{g}}_i$
16:	$G_{min}\leftarrow \{i:\Delta d_i=-1\}$
17:	if  $|G_{min}|=1$  then
18:		let ${\mathit{g}}_j$ be the sole element of $G_{min}$
19:		$r_{\mathsf{P}}\leftarrow r_{\mathsf{P}}+\beta a_j-\beta {\sum }_{i\ne j}{a}_i$
20:	else
21:		for each $i\in G_{min}$ compute $c_i\left(t\right)=d_i\left(t\right)+p_i\left(t\right)$
22:		$c_{min}=min\{c_i\left(t\right):{\mathit{g}}_i\in G_{min}\}$ and $C_{min}=\{{\mathit{g}}_i\in G_{min}:c_i\left(t\right)=c_{min}\}$
23:		for $i=1$ to n do
24:			if $i\in C_{min}$ then
25:				$r_{\mathsf{P}}\leftarrow r_{\mathsf{P}}+\beta \frac{a_i}{|C_{min}|}$
26:			else
27:				$r_{\mathsf{P}}\leftarrow r_{\mathsf{P}}-\beta a_i$
28:			end if
29:			if the robber visited goal ${\mathit{g}}_i$ before and $a_i=0$ then
30:				$r_{\mathsf{P}}\leftarrow r_{\mathsf{P}}+r_{\mathrm{bonus}}$	▹ Police reward
31:			end if
32:		end for
33:	end if
34:	return  $(r_{\mathsf{R}},r_{\mathsf{P}})$

4. Deep Q-Network Training

To jointly induce deceptive and counter-deceptive behaviors, we optimize the robber’s and police officer’s policies via deep Q-learning with soft target-network updates. Each agent maintains an online deep Q-network (DQN) $Q_{\theta }$ and a lagging target DQN $Q_{{\theta }^{-}}$, both parametrized by multi-layer perceptrons with weights $\theta$ and ${\theta }^{-}$, respectively. There are total of four networks (${\theta }_{\mathsf{R}},{\theta }_{\mathsf{R}}^{-},{\theta }_{\mathsf{P}},{\theta }_{\mathsf{P}}^{-}$), two for each player.

Experience (in RL, experience is a tuple that contains the current state, current action, reward, next state, and occasionally a binary variable denoting whether the episode ended in the next state or not) is stored in a replay buffer of symbolic capacity C, from which minibatches of size B are sampled to decorrelate updates. We employ a discount factor $\gamma \in (0,1)$, gradient-norm clipping bound G, and a soft-update coefficient $\tau \ll 1$ so that after each gradient step the target parameters blend smoothly:

$${\theta }^{-}⟵\tau \theta +(1-\tau ){\theta }^{-}.$$

Exploration is governed by time-dependent rates $\left\{{\epsilon }_t\right\}$: at each decision point an agent selects the action that maximizes its current Q-estimate with probability $1-{\epsilon }_t$, and otherwise samples uniformly from its action set. The schedule for decaying ${\epsilon }_t$ is chosen to ensure ample early exploration followed by gradual consolidation on high-value strategies.

Updates proceed whenever the replay buffer contains at least B transitions. A minibatch is drawn, and for each experience a bootstrapped target is computed against the target network using the Huber loss to mitigate sensitivity to outliers. Gradients are clipped to norm G before each parameter update, reinforcing stability in the non-stationary, adversarial training setting (Algorithm 2).

Algorithm 2 Two-player DQN with Soft Target Updates
Require: replay capacity C, batch size B, discount $\gamma$, soft-update $\tau$, grad-clip bound G
Ensure: trained Q-networks $Q_{{\theta }_{\mathsf{R}}},Q_{{\theta }_{\mathsf{P}}}$
1:	Initialize online networks $Q_{{\theta }_{\mathsf{R}}},Q_{{\theta }_{\mathsf{P}}}$ and targets $Q_{{\theta }_{\mathsf{R}}^{-}}\leftarrow Q_{{\theta }_{\mathsf{R}}}$ and $Q_{{\theta }_{\mathsf{P}}^{-}}\leftarrow Q_{{\theta }_{\mathsf{P}}}$.
2:	Initialize replay buffers ${\mathcal{D}}_{\mathsf{R}},{\mathcal{D}}_{\mathsf{P}}$ with capacity C
3:	for  $t=1,2,\dots$  do
4:		Observe game-states $\mathit{s}\left(t\right)$			▹ Defined in Equation (2)
5:		Select ${\mathbf{a}}_{\mathsf{R}}\left(t\right),{\mathbf{a}}_{\mathsf{P}}\left(t\right)$ via ${\epsilon }_t$–greedy policies
6:		Execute ${\mathbf{a}}_{\mathsf{R}}\left(t\right),{\mathbf{a}}_{\mathsf{P}}\left(t\right)$; observe $(r_{\mathsf{R}}\left(t\right),r_{\mathsf{P}}\left(t\right),\mathit{s}(t+1),\mathtt{done})$			▹done=1 iff ${\mathbf{x}}_{\mathsf{R}}(t+1)=g_T$
7:		Store transitions $(\mathit{s}\left(t\right),{\mathbf{a}}_{\mathsf{R}}\left(t\right),{\mathbf{a}}_{\mathsf{P}}\left(t\right),r_{\mathsf{R}}\left(t\right),r_{\mathsf{P}}\left(t\right),\mathit{s}(t+1),\mathtt{done})$ in ${\mathcal{D}}_{\mathsf{R}},{\mathcal{D}}_{\mathsf{P}}$
8:		if $|{\mathcal{D}}_{\mathsf{R}}|\ge B$ then
9:			$L_{\mathsf{R}}\leftarrow 0,$		▹ Loss function for $\mathsf{R}$
10:			Sample minibatch of size B from ${\mathcal{D}}_{\mathsf{R}}$
11:			for each experience $(\mathit{s},{\mathbf{a}}_{\mathsf{R}},{\mathbf{a}}_{\mathsf{P}},r_{\mathsf{R}},r_{\mathsf{P}},{\mathit{s}}^{\prime },\mathtt{done})$ in B do
12:				Compute targets $y_{\mathsf{R}}=r_{\mathsf{R}}+(1-\mathtt{done})\gamma {max}_{\mathbf{a}\in A_{\mathsf{R}}}{Q}_{{\theta }_{\mathsf{R}}^{-}}(\mathit{s},\mathbf{a})$
13:				$L_{\mathsf{R}}\leftarrow L_{\mathsf{R}}+\mathtt{HuberLoss}(y_{\mathsf{R}},Q_{{\theta }_{\mathsf{R}}}(\mathit{s},{\mathbf{a}}_{\mathsf{R}}))$	▹ Huber Loss function for $\mathsf{R}$
14:			end for
15:			Update ${\theta }_{\mathsf{R}}$ by minimizing $L_{\mathsf{R}}$; clip gradients with bound G
16:			${\theta }_{\mathsf{R}}^{-}\leftarrow \tau {\theta }_{\mathsf{R}}+(1-\tau ){\theta }_{\mathsf{R}}^{-}$		▹ Target weight update
17:		end if
18:		if $|{\mathcal{D}}_{\mathsf{P}}|\ge B$ then
19:			$L_{\mathsf{P}}\leftarrow 0,$		▹ Loss function for $\mathsf{P}$
20:			Sample minibatch of size B from ${\mathcal{D}}_{\mathsf{P}}$
21:			for each experience $(\mathit{s},{\mathbf{a}}_{\mathsf{R}},{\mathbf{a}}_{\mathsf{P}},r_{\mathsf{R}},r_{\mathsf{P}},{\mathit{s}}^{\prime },\mathtt{done})$ in B do
22:				Compute targets $y_{\mathsf{P}}=r_{\mathsf{P}}+(1-\mathtt{done})\gamma {max}_{\mathbf{a}\in A_{\mathsf{P}}}{Q}_{{\theta }_{\mathsf{P}}^{-}}(\mathit{s},\mathbf{a})$
23:				$L_{\mathsf{P}}\leftarrow L_{\mathsf{P}}+\mathtt{HuberLoss}(y_{\mathsf{P}},Q_{{\theta }_{\mathsf{P}}}(\mathit{s},{\mathbf{a}}_{\mathsf{P}}))$	▹ Huber Loss function for $\mathsf{P}$
24:			end for
25:			Update ${\theta }_{\mathsf{P}}$ by minimizing $L_{\mathsf{P}}$; clip gradients with bound G
26:			${\theta }_{\mathsf{P}}^{-}\leftarrow \tau {\theta }_{\mathsf{P}}+(1-\tau ){\theta }_{\mathsf{P}}^{-}$		▹ Target weight update
27:		end if
28:		Decay exploration rates ${\epsilon }_{t+1}\leftarrow f\left({\epsilon }_t\right)$
29:	end for

4.1. Neural Network Architectures for ${\theta }_{\mathsf{R}}$ and ${\theta }_{\mathsf{P}}$

Each agent’s DQN maps a fixed-length feature vector into estimated Q-values over its discrete action set. Both the robber and the police officer use three fully connected hidden layers, but the robber’s network includes dropout for regularization whereas the police officer’s does not. The input and output dimensions differ according to each agent’s role, but the hidden-layer structure is otherwise identical in depth.

• Overall Structure: Let $d_{\mathrm{in}}$ denote the input dimension (determined by the maximum number of goals $G_{max}$ and obstacles $O_{max}$; see Section 4.2) and let $\left|A\right|$ denote the size of the agent’s action set (4 for the robber’s four movements; $G_{max}$ for the police officer’s choice of goal). Then,

• Robber’s network ($Q_{{\theta }_{\mathsf{R}}}$):

  $$\begin{array}{cc}& \mathsf{FC}\left(d_{\mathrm{in}},256\right)\stackrel{\mathrm{ReLU}}{\to } \stackrel{\mathrm{Dropout}(p=0.2)}{\to }\hfill \\ & \mathsf{FC}\left(256,128\right)\stackrel{\mathrm{ReLU}}{\to } \stackrel{\mathrm{Dropout}(p=0.2)}{\to }\hfill \\ & \mathsf{FC}\left(128,64\right)\stackrel{\mathrm{ReLU}}{\to }\hfill \\ & \mathsf{FC}\left(64,\left|A\right|\right).\hfill \end{array}$$
  Here each “FC” is a fully connected layer, and each dropout layer uses probability $p=0.2$.
• Police officer’s network:

  $$\begin{array}{cc}& \mathsf{FC}\left(d_{\mathrm{in}},256\right)\stackrel{\mathrm{ReLU}}{\to }\hfill \\ & \mathsf{FC}\left(256,128\right)\stackrel{\mathrm{ReLU}}{\to }\hfill \\ & \mathsf{FC}\left(128,64\right)\stackrel{\mathrm{ReLU}}{\to }\hfill \\ & \mathsf{FC}\left(64,\left|A\right|\right).\hfill \end{array}$$
  No dropout layers are inserted for the police network.

Figure 3 shows the robber’s DQN (with dropout), and Figure 4 shows the police officer’s DQN (without dropout). Both diagrams are scaled to fit within typical journal margins.

To encourage the robber’s network to learn broadly generalizable deceptive strategies rather than memorizing specific exit configurations, we intersperse dropout layers after the first two hidden-layer activations. By randomly dropping a fraction of hidden units during training, the robber’s DQN is forced to distribute its internal representations across many pathways, reducing overfitting to particular grid layouts or goal placements and thus improving transfer to novel exit arrangements. The police officer’s network, in contrast, must develop highly precise value estimates to detect and counteract subtle deceptive cues; introducing dropout there tended to destabilize its ability to capture fine-grained distance–change and resource-allocation patterns. Empirically, we found that regularizing only the robber network strikes the right balance between robustness and sensitivity for our adversarial setting.

4.2. Input Feature Construction and Training Hyperparameters

This section provides a detailed explanation of how each environment state is encoded into a fixed-length feature vector, outlines the key training hyperparameters (including the exploration decay schedule), and explains why the same DQN architecture can seamlessly handle progressively increasingly complex scenarios. The input features to the DQNs vary depending on the game scenario. We categorize them into three groups: (1) games without obstacles, (2) games with obstacles, and (3) scenarios involving network generalization through dynamic goals and obstacles.

4.2.1. Games Without Obstacles

In this setup, the environment contains no obstacles. A special case within this scenario occurs when there are only two goals—one true and one decoy. For this case, analytical results are available in the game theory literature [26]. This provides an opportunity to validate our results against a game-theoretic ground truth.

The current location of the robber, i.e., ${\mathbf{x}}_{\mathsf{R}}$, and the current police allocation at the goals, i.e, ${\mathbf{x}}_{\mathsf{P}}$, are included as part of the feature vectors for both agents. In addition to that, the set of all exit locations

$$\mathbf{E}=\left(x_{{\mathit{g}}_1},y_{{\mathit{g}}_1},\dots ,x_{{\mathit{g}}_n},y_{{\mathit{g}}_n}\right),$$

is also part of the inputs, where $(x_{{\mathit{g}}_i},y_{g_i})$ denotes the grid coordinates of goal ${\mathit{g}}_i$. To incorporate knowledge about the true goal, we define an indicator vector associated with $\mathbf{E}$:

$$\mathbf{I}=\left({\mathbb{I}}_0,{\mathbb{I}}_1,\dots ,{\mathbb{I}}_{N-1}\right),{\mathbb{I}}_i=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\mathit{g}}_i\mathrm{is}\mathrm{the}\mathrm{true}\mathrm{goal},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

This indicator vector is included only in the input to the robber’s network, $Q_{{\theta }_{\mathsf{R}}}$, and not in the police’s network.

Therefore, the full feature set for the robber consists of:

$$\underset{\mathrm{dimension}}{\underbrace{{\mathbf{x}}^{\mathrm{robber}}}}\underrightarrow{\underbrace{=}}\underset{2}{\underbrace{{\mathbf{x}}_{\mathsf{R}}}}\parallel \underset{n}{\underbrace{{\mathbf{x}}_{\mathsf{P}}}}\parallel \underset{n}{\underbrace{\mathbf{I}}}\parallel \underset{2n}{\underbrace{\mathbf{E}}},$$

with the dimension of the feature vector being $d_{\mathrm{in}}=4n+2$ for a setup with n goals. On the other hand, the full feature set for the police consists of

$$\underset{\mathrm{dimension}}{\underbrace{{\mathbf{x}}^{\mathrm{police}}}}\underrightarrow{\underbrace{=}}\underset{2}{\underbrace{{\mathbf{x}}_{\mathsf{R}}}}\parallel \underset{n}{\underbrace{{\mathbf{x}}_{\mathsf{P}}}}\parallel \underset{2n}{\underbrace{\mathbf{E}}},$$

with the dimension of the feature vector being $d_{\mathrm{in}}=3n+2$.

This set-up will subsequently be used for Experiments 1 and 3 in our study.

4.2.2. Games with Obstacles

In addition to the input features mentioned in the previous case, we also include the relative distances to the obstacles $O=\{{\mathit{o}}_1,\dots ,{\mathit{o}}_k\}$ as features:

$$\Delta \mathbf{o}=\left(\delta x_1,\delta y_1,\delta x_2,\delta y_2,\dots ,\delta x_k,\delta y_k\right),$$

where, for all $j=1,\dots ,k$,

$$(\delta x_j,\delta y_j)={\mathit{o}}_j-{\mathbf{x}}_{\mathsf{R}},$$

denotes the relative distance vector from the robber’s current position to obstacle ${\mathit{o}}_j$. Consequently, in this case, the dimensions of the robber’s and police officer’s input features become $4n+2k+2$ and $3n+2k+2$, respectively.

$$\underset{\mathrm{dimension}}{\underbrace{{\mathbf{x}}^{\mathrm{robber}}}}\underrightarrow{\underbrace{=}}\underset{2}{\underbrace{{\mathbf{x}}_{\mathsf{R}}}}\parallel \underset{n}{\underbrace{{\mathbf{x}}_{\mathsf{P}}}}\parallel \underset{n}{\underbrace{\mathbf{I}}}\parallel \underset{2n}{\underbrace{\mathbf{E}}}\parallel \underset{2k}{\underbrace{\Delta \mathbf{o}}},$$

$$\underset{\mathrm{dimension}}{\underbrace{{\mathbf{x}}^{\mathrm{robber}}}}\underrightarrow{\underbrace{=}}\underset{2}{\underbrace{{\mathbf{x}}_{\mathsf{R}}}}\parallel \underset{n}{\underbrace{{\mathbf{x}}_{\mathsf{P}}}}\parallel \underset{2n}{\underbrace{\mathbf{E}}}\parallel \underset{2k}{\underbrace{\Delta \mathbf{o}}}.$$

4.2.3. Network Generalization

In the previous two cases, training is conducted on fixed environments, where the number of goals, the number of obstacles, and the goal locations remain constant. To develop policies that generalize to different environments—and potentially to larger grids—we replace the absolute goal locations $\mathbf{E}$ with relative distance vectors and distances to the goals. Consequently, the input feature dimensions for the robber and police networks are $2+5n+2k$ and $2+4n+2k$, respectively. To accommodate a dynamic number of goals (up to N) and obstacles (up to K), we fix the input feature lengths to $2+5N+2K$ for the robber and $2+4N+2K$ for the police officer, applying a zero-padding scheme when the actual number of goals or obstacles is less than their respective maximums.

4.2.4. Training Hyperparameters

Both the robber’s and police officer’s DQNs are trained with off-policy Q-learning, using experience replay and a separate target network. The main hyperparameters are as follows:

• Discount factor $\gamma =0.99$.
• Adam optimizer with learning rate $1\times {10}^{-4}$.
• Replay buffer capacity $C=10,000$, minibatch size $B=64$.
• Target network update: copy online parameters to target network every 50 episodes using soft policy update with $\tau =0.99$
• Exploration schedule ($\epsilon$-greedy):

  $${\epsilon }_{t+1}=max\left({\epsilon }_{min},\kappa {\epsilon }_t\right),\kappa =0.99,{\epsilon }_{min}={10}^{-3},{\epsilon }_0=0.9.$$
  After each episode, $\epsilon$ decays multiplicatively by $\kappa$, ensuring exploration does not fall below ${\epsilon }_{min}$.

4.2.5. Computation Metrics

To assess the practical feasibility of our framework, we measured both time and space complexity on an NVIDIA RTX 2080 (8 GB VRAM, CUDA 11.8). Each experiment averages over 1000 episodes (training).

• Training time. $0.45\pm 0.02$ s per episode, including forward/back-prop and all environment interactions.
• Inference latency. $0.8\pm 0.1$ ms per decision step (batch size = 1).
• Model size. 160 KB per DQN (32-bit weights), allowing the entire policy stack to reside comfortably in on-chip cache of low-power embedded boards.
• Peak memory. 96 MB during training and 7 MB during inference, dominated by replay-buffer and tensor allocations, respectively.

These measurements demonstrate that our approach supports rapid prototyping, real-time control, and deployment on resource-constrained hardware without additional optimization.

5. Results and Simulations

We evaluate the performance of our deceptive and inference-driven policies across six canonical environments

Table 1. Policy abbreviations and descriptions.

Abbreviation	Description
${\pi }_{\mathsf{R}}^{\mathrm{random}}$	Random Robber: moves chosen uniformly at random.
${\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}}$	Shortest-Path Robber: greedily minimizes Manhattan distance to the true exit.
${\pi }_{\mathsf{R}}^{\mathrm{naive}}$	Naïve Deceptive Robber: visits one decoy then proceeds to the true exit.
${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$	Trained DQN Robber: learns deceptive trajectories via deep Q-learning.
${\pi }_{\mathsf{P}}^{\mathrm{Baseline}}$	Baseline Game-Theoretic Robber: follows the Stackelberg deception equilibrium strategy from [26].
${\pi }_{\mathsf{P}}^{\mathrm{random}}$	Random Police Officer: allocates resources uniformly at random.
${\pi }_{\mathsf{P}}^{\mathrm{Fully}\mathrm{Informed}}$	Fully Informed Police Officer: all resources are allocated to the true exit.
${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$	Trained DQN Police Officer: learns to infer intent and counter deception via deep Q-learning.
${\pi }_{\mathsf{P}}^{\mathrm{Baseline}}$	Baseline Game-Theoretic Police Officer: allocates according to the equilibrium strategy from [26].

. The primary metric reported in each setting is the Percentage of Fake-Goal Allocation (PFA), which measures the fraction of the police officer’s total reinforcement budget that is misallocated to fake exits, thereby quantifying the robber’s deception strength. All results reflect the mean over 10 independent runs with randomized goal placements.

■

Supplementary performance metrics: To provide a concise yet informative assessment of deception dynamics, we report three complementary quantities: the Percentage of Fake-Exit Allocation (PFA), its derived counter-deception score $100-\mathrm{PFA}$, and the true-exit interception rate (TIR). PFA measures the fraction of the police officer’s total reinforcement budget that is misallocated to fake exits, thereby quantifying the robber’s deception strength. The counter-deception score inverts this value to give an immediate gauge of how effectively the police officer resists the lure. Finally, the interception rate captures the proportion of allocation steps that successfully cover the true exit, while a fake exit is active and can be written as

$$\mathrm{TIR}={\displaystyle \frac{\mathrm{true}-\mathrm{exit}\mathrm{allocations}}{\mathrm{total}\mathrm{allocations}}}\times 100.$$

Together, these three metrics form a minimal “confusion-like” summary from which the full confusion-matrix entries (true/false positives and negatives) can be reconstructed when the total number of allocation steps is known. We focus on PFA as our primary metric and provide qualitative analysis of behavioral trends to highlight nuanced strategic patterns that emerge across different experimental conditions.

5.1. Experiment 1: Two-Goal Environment

In this baseline scenario—a $7\times 7$ grid with one true and one decoy goal—the trained DQN police officer (${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$) dramatically reduces misallocation compared to naive strategies.

Table 2. PFA (%) in the two-goal environment.

		$\mathsf{P}$’s Policy
		${\pi }_{\mathsf{P}}^{\mathrm{random}}$	${\pi }_{\mathsf{P}}^{\mathrm{Fully}\mathrm{Informed}}$	${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$	${\pi }_{\mathsf{P}}^{\mathrm{Baseline}}$
$\mathsf{R}$’s Policy	${\pi }_{\mathsf{R}}^{\mathrm{random}}$	52.00	0.00	52.00	48.00
	${\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}}$	46.00	0.00	28.57	24.50
	${\pi }_{\mathsf{R}}^{\mathrm{naive}}$	52.00	0.00	34.00	32.00
	${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$	38.89	0.00	28.57	26.00
	${\pi }_{\mathsf{P}}^{\mathrm{Baseline}}$	46.00	0.00	28.57	25.00

reports the PFA, where ${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$ lowers the percentage of resources sent to the decoy by up to 40%.

The results demonstrate the effectiveness of our DQN-based approach compared to baseline strategies

Table 3. Ablation study in the two-goal environment (Experiment 1). Percentage of fake-goal allocation (PFA) and true-exit interception rate for different model variants. Lower PFA and higher interception are better.

Variant	PFA (%)	Interception Rate (%)
Full MARL-DQN (ours)	28.6	78
No reward shaping	46.6	62
No intent inference	39.6	68
Static layout only	53.6	54

. The trained DQN police officer (${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$) consistently outperforms the game-theoretic baseline (${\pi }_{\mathsf{P}}^{\mathrm{Baseline}}$), achieving identical or better performance across all robber policies. Notably, ${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$ maintains the same 28.57% PFA as ${\pi }_{\mathsf{P}}^{\mathrm{Baseline}}$ against the shortest-path (${\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}}$), naive deceptive (${\pi }_{\mathsf{R}}^{\mathrm{naive}}$), and trained DQN (${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$) robbers while also handling the random robber (${\pi }_{\mathsf{R}}^{\mathrm{random}}$) more effectively. The DQN robber (${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$) shows competitive performance against the baseline (${\pi }_{\mathsf{P}}^{\mathrm{Baseline}}$), achieving 28.57% PFA against ${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$ compared to ${\pi }_{\mathsf{P}}^{\mathrm{Baseline}}$’s 25.00% against ${\pi }_{\mathsf{P}}^{\mathrm{Baseline}}$, demonstrating that learned policies can match or exceed theoretical equilibrium strategies in practice.

The ablation study reveals the critical contribution of each component to our framework’s performance. The full MARL-DQN approach achieves the best performance with 28.6% PFA and 78% interception rate, significantly outperforming all ablated variants. Removing reward shaping degrades performance substantially (46.6% PFA, 62% interception), demonstrating that game-theoretic incentives are essential for learning effective deception strategies. The absence of intent inference also hurts performance (39.6% PFA, 68% interception), confirming that opponent modeling is crucial for counter-deception. Training on static layouts only results in the worst performance (53.6% PFA, 54% interception), highlighting the importance of dynamic environment training for generalization. These results validate that our integrated approach combining reward shaping, intent inference, and dynamic training is necessary for achieving robust deception and counter-deception capabilities.

The results in Figure 5 demonstrate that handcrafted and learned policies achieve more efficient, targeted deception compared to the indiscriminate strategy of ${\pi }_{\mathsf{R}}^{\mathrm{random}}$. Both the shortest-path robber (${\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}}$) and the trained DQN robber (${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$) attain nearly the same deception rate, showing that strategic planning can be as effective as learned policies in simple scenarios.

5.2. Experiment 2: Two-Goal Environment with Obstacles

Introducing static obstacles increases navigational complexity. Despite this, the DQN police officer continues to outperform, as shown in

Table 4. PFA (%) with obstacles in the two-goal environment.

		$\mathsf{P}$’s Policy
		${\pi }_{\mathsf{P}}^{\mathrm{random}}$	${\pi }_{\mathsf{P}}^{\mathrm{Fully}\mathrm{Informed}}$	${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$
$\mathsf{R}$’s Policy	${\pi }_{\mathsf{R}}^{\mathrm{random}}$	49.00	0.00	35.00
	${\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}}$	28.00	0.00	32.00
	${\pi }_{\mathsf{R}}^{\mathrm{naive}}$	40.00	0.00	36.00
	${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$	63.64	0.00	28.57

, demonstrating robust inference under spatial constraints.

Introducing obstacles significantly impacts each robber policy. The random (${\pi }_{\mathsf{R}}^{\mathrm{random}}$) and shortest-path (${\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}}$) strategies collide frequently with obstacles, causing early terminations and resulting in fake-goal percentages of 49.00% and 28.00%, respectively. The naive deceptive robber (${\pi }_{\mathsf{R}}^{\mathrm{naive}}$) initially detours toward a fake goal but lacks adaptive obstacle avoidance, yielding 40.00% PFA—a marginal improvement over non-learning policies.

By contrast, the trained DQN robber (${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$) learns to navigate around obstacles, achieving the highest fake-goal allocation (63.64%) under the random police and robust results (28.57%) against the DQN police. In cluttered environments, learning-based deception sustains longer trajectories and concentrates misdirection more effectively than static or random policies (Figure 6). As complexity increases, online adaptation to obstacles proves essential for preserving high deception efficacy.

In the next three experiments, we evaluate the scalability and generalizability of our method by progressively increasing the complexity of the setup—starting with an increased number of goals (Experiment 5), followed by a greater number of obstacles (Experiment 6), and finally both in a dynamic scenario (Experiment 7).

5.3. Experiment 3: Multi-Goal Environment

Scaling to three decoys on a $7\times 7$ grid substantially increases goal ambiguity.

Table 5. PFA (%) in the multi-goal environment.

		$\mathsf{P}$’s Policy
		${\pi }_{\mathsf{P}}^{\mathrm{random}}$	${\pi }_{\mathsf{P}}^{\mathrm{Fully}\mathrm{Informed}}$	${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$
$\mathsf{R}$’s Policy	${\pi }_{\mathsf{R}}^{\mathrm{random}}$	76.47	0.00	75.00
	${\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}}$	64.29	0.00	57.14
	${\pi }_{\mathsf{R}}^{\mathrm{naive}}$	76.00	0.00	66.00
	${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$	64.29	0.00	44.44

shows that ${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$ maintains the lowest PFA, demonstrating its ability to minimize misallocation under higher uncertainty.

In the multi-goal environment, choosing among three decoys greatly amplifies deception complexity. The random robber (${\pi }_{\mathsf{R}}^{\mathrm{random}}$) attains 76.47% fake-goal allocation—demonstrating exaggerated but inefficient deception. The shortest-path robber (${\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}}$) reaches 64.29% PFA, offering sparse, unintended misdirection. The naive deceptive robber (${\pi }_{\mathsf{R}}^{\mathrm{naive}}$) achieves 76.00% PFA but still trails in efficiency. By contrast, the DQN robber (${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$) balances efficiency and efficacy: it secures 64.29% PFA (and 44.44% PFA against the DQN police). As decoy options increase, such targeted learning-based strategies are essential for maintaining high deception efficiency (Figure 7).

5.4. Experiment 4: Multi-Goal Environment with Obstacles

Adding obstacles in the multi-goal grid further tests strategic inference. As

Table 6. PFA (%) with obstacles in the multi-goal environment.

		$\mathsf{P}$’s Policy
		${\pi }_{\mathsf{P}}^{\mathrm{random}}$	${\pi }_{\mathsf{P}}^{\mathrm{Fully}\mathrm{Informed}}$	${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$
$\mathsf{R}$’s Policy	${\pi }_{\mathsf{R}}^{\mathrm{random}}$	87.50	0.00	75.00
	${\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}}$	75.00	0.00	57.14
	${\pi }_{\mathsf{R}}^{\mathrm{naive}}$	78.50	0.00	80.50
	${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$	80.26	0.00	78.23

shows, ${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$ retains superior performance, keeping PFA at its lowest levels.

In this experiment, non-learning policies (${\pi }_{\mathsf{R}}^{\mathrm{random}},{\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}},{\pi }_{\mathsf{R}}^{\mathrm{naive}}$) suffer frequent collisions that truncate their trajectories, yielding allocation rates varying from 75 to 87.5%. By contrast, the DQN-trained robber (${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$) learns to skirt obstacles, achieving 78.23–80.26% fake-goal allocation. This demonstrates that adaptive, learning-based strategies can sustain high deception efficiency in complex, cluttered settings(Figure 8).

5.5. Experiment 5: Dynamic Goals (No Obstacles)

We broaden to 100 randomized trials on a $7\times 7$ grid with dynamic exit placements. Even under non-stationary configurations, the trained police officer (${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$) consistently achieves the lowest average PFA (

Table 7. PFA (%) for dynamic goals (no obstacles).

		$\mathsf{P}$’s Policy
		${\pi }_{\mathsf{P}}^{\mathrm{random}}$	${\pi }_{\mathsf{P}}^{\mathrm{Fully}\mathrm{Informed}}$	${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$
$\mathsf{R}$’s Policy	${\pi }_{\mathsf{R}}^{\mathrm{random}}$	86.21	0.00	85.00
	${\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}}$	60.14	0.00	53.12
	${\pi }_{\mathsf{R}}^{\mathrm{naive}}$	69.00	0.00	64.00
	${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$	60.11	0.00	40.34

).

In the dynamic goals environment, goal positions change each episode, requiring a single policy to adapt to all layouts(Figure 9). The random robber (${\pi }_{\mathsf{R}}^{\mathrm{random}}$) reaches 86.21% fake-goal allocation through indiscriminate behavior. The shortest-path (${\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}}$) and naive deceptive (${\pi }_{\mathsf{R}}^{\mathrm{naive}}$) robbers achieve 60.14% and 69.00% PFA, respectively, but falter on novel configurations. In contrast, the DQN robber (${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$) generalizes across 100 layouts, securing 60.11% PFA and 40.34% against the adaptive DQN police officer—demonstrating true flexibility in dynamic settings.

At the same time, the DQN-trained police officer ${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$ interprets distance change signals, resource allocation costs, and visitation history to infer the robber’s hidden goal more accurately than static or random policies. Across six environments, from two-goal grids to dynamic multi-goal layouts with obstacles, ${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$ reduces the percentage of fake-goal allocation by up to fifty percent. This robust performance highlights the advantage of embedding opponent-aware shaping into the police officer’s learning process.

5.6. Experiment 6: Dynamic Goals with Obstacles

Finally, static obstacles are reintroduced in the dynamic setting.

Table 8. PFA (%) for dynamic goals with obstacles.

		$\mathsf{P}$’s Policy
		${\pi }_{\mathsf{P}}^{\mathrm{random}}$	${\pi }_{\mathsf{P}}^{\mathrm{Fully}\mathrm{Informed}}$	${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$
$\mathsf{R}$’s Policy	${\pi }_{\mathsf{R}}^{\mathrm{random}}$	88.23	0.00	72.20
	${\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}}$	66.00	0.00	52.76
	${\pi }_{\mathsf{R}}^{\mathrm{naive}}$	72.62	0.00	60.22
	${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$	70.26	0.00	72.25

confirms that even under maximum complexity, the DQN police officer maintains robust performance, minimizing PFA.

Although the random robber (${\pi }_{\mathsf{R}}^{\mathrm{random}}$) records 88.23% fake-goal allocation under the random police officer, obstacle collisions restrict its effectiveness, revealing its deception as indiscriminate. The shortest-path robber (${\pi }_{\mathsf{R}}^{\mathrm{shortest}\mathrm{path}}$) achieves 66.00% but lacks adaptive avoidance. In contrast, the DQN robber (${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$) deliberately navigates around obstacles, securing 70.26%, demonstrating genuine strategic adaptation (Figure 10).

6. Discussion

Our experiments demonstrate that game-theoretic reward shaping produces sophisticated strategic behaviors in both agents. The DQN-trained robber ${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$ learns to orchestrate decoy visits with remarkable precision, inducing costly misallocations by the police officer while avoiding unnecessary detours and obstacle penalties. In cluttered grids naive deceptive heuristics often become trapped, whereas ${\pi }_{\mathsf{R}}^{\mathrm{DQN}}$ maintains efficiency without sacrificing strategic ambiguity.

At the same time the DQN-trained police officer ${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$ interprets distance change signals, resource allocation costs, and visitation history to infer the robber’s hidden goal more accurately than static or random policies. Across six environments, from two-goal grids to dynamic multi-goal layouts with obstacles, ${\pi }_{\mathsf{P}}^{\mathrm{DQN}}$ reduces the percentage of fake-goal allocation by up to fifty percent and minimizes wasted visits to decoys. This robust performance highlights the advantage of embedding opponent-aware shaping into the police officer’s learning process.

These results show that deception and counter deception emerge naturally when reward functions reflect core principles of pursuit evasion and signaling games. Furthermore the learned policies generalize to novel layouts and goal configurations, suggesting that our shaping terms capture fundamental adversarial structure rather than overfitting to specific maps. In this way our framework bridges classical theory and deep reinforcement learning, opening new pathways for strategic planning under uncertainty.

7. Conclusions

We have introduced a game-theoretic reward-shaping framework that integrates opponent aware incentives into multi-agent deep Q-learning. By augmenting standard rewards with distance change feedback, resource allocation cost, and history aware bonuses, our method enables decentralized policies capable of both deception and counter-deception under partial observability and physical constraints.

Empirical evaluation across six progressively complex grid world scenarios from simple two-goal arenas to dynamic multi-goal environments with obstacles confirms that shaped DQN agents consistently outperform naive baselines. The shaped robber orchestrates precise decoy visits to maximize misdirection, while the shaped police officer anticipates and counters those signals, achieving up to fifty percent reduction in misallocation. Both agents maintain strong performance under dynamic goal relocations and navigational challenges.

Future work will advance three directions to accelerate learning and broaden applicability. First, integrating a policy space response oracles meta solver will equip agents to adapt to mixed strategy equilibria and resilient adversaries. Second, curriculum learning schedules that gradually increase environment complexity can speed convergence and enhance generalization. Third, extending the state representation with graph-based node embeddings will enable deployment in real world map scenarios, such as traffic flow management, transportation logistics, military interception planning, and autonomous surveillance on road networks. These extensions promise to translate our strategic reward-shaping paradigm from grid-world benchmarks to high-impact applications in security and infrastructure.