Entropy-Driven Action Randomization: A Deployment-Time Defense for Environment Privacy in Deep Reinforcement Learning

Abstract

Deep reinforcement learning (DRL) agents implicitly memorize the structure of their training environment, allowing an attacker to reconstruct it by querying the action outputs of a deployed model and causing environment-privacy leakage. To address this, this study proposes Entropy-Driven Action Randomization (EDAR), a deployment-time action randomization defense. Inspired by the exponential mechanism from differential privacy, EDAR replaces deterministic greedy action selection with probabilistic action sampling based on a normalized Q-value utility function, without altering the trained policy. A state-adaptive dynamic privacy budget is further designed, guided by the Shannon entropy of the action distribution. A simulated-annealing-based policy inversion attack is used to quantify privacy leakage, measured by the normalized recovery rate (NRR). Experiments on GridWorld environments ranging from 7 × 7 to 13 × 13 show that, at a privacy level comparable to a strong fixed budget, the proposed dynamic budget raises the reward retention from 6.2% to 78.9% while keeping a comparable NRR reduction. These results indicate that adapting the privacy budget to per-state policy determinism yields a markedly better privacy–utility trade-off than existing training-perturbation-based methods. We clarify that the differential privacy property invoked here is a per-query, mechanism-level guarantee of indistinguishability between neighboring observation states under a fixed trained model. The protection of the environment structure itself is established empirically through the policy inversion attack. Experiments are conducted on discrete GridWorld navigation tasks; the conclusions are scoped to small value-based navigation settings.

1. Introduction

The rapid proliferation of machine learning systems in safety-critical and privacy-sensitive domains has made the privacy of learned models a central concern [1,2]. Unlike classical data-privacy threats that arise from the direct exposure of stored records, trained models can implicitly encode sensitive information acquired during learning, rendering them susceptible to reconstruction attacks even when the model parameters remain hidden [3,4,5,6]. Deep reinforcement learning (DRL) presents a particularly acute instance of this problem. A DRL agent learns by repeatedly interacting with and observing its training environment, which causes the learned policy to implicitly memorize the structural information of that environment [7]. Once such an agent is deployed as a query-accessible service, an attacker with only black-box access can exploit this memorization to reconstruct the original training environment from the agent’s action outputs alone [7]. This threat is directly relevant to domains such as medical robotics, autonomous navigation, and intelligent infrastructure management, where the environment itself constitutes sensitive proprietary information.

1.1. Related Work

The broader risk of model memorization has been extensively documented in supervised learning. Membership inference attacks [3] demonstrate that an adversary can determine, through black-box queries, whether a given record was used for training. Model inversion attacks [4] show that the confidence scores exposed by a classifier are sufficient to reconstruct sensitive input features. Carlini et al. [5] established a rigorous likelihood-ratio framework that approaches the theoretical optimum of membership inference using only per-sample loss values, while Hu et al. [6] provided a comprehensive survey of this line of work. These studies collectively indicate that the memorization of training data is an inherent by-product of gradient-based learning and constitutes the root cause of privacy leakage—a conclusion that extends naturally to DRL, where the training environment plays the role of the sensitive data.

Differential privacy (DP) [8], with its rigorous mathematical guarantees, has become the de facto framework for privacy-preserving machine learning. Abadi et al. [9] introduced differentially private stochastic gradient descent (DP-SGD), which injects calibrated Gaussian noise into clipped gradients to ensure privacy during training. In the reinforcement learning setting, Wang and Hegde [10] proposed differentially private Q-learning with functional noise for continuous state spaces, and Abahussein et al. [11] extended this idea to Double DQN in continuous spaces via a DP-SGD approach. Qiao and Wang [12] designed private offline RL algorithms with instance-dependent sample-complexity bounds, showing that privacy protection incurs almost no additional utility loss under tabular and linear Markov Decision Process (MDP) settings. Vietri et al. [13] studied private episodic exploration under joint DP, Garcelon et al. [14] addressed regret minimization under local DP, and Qiao and Wang [15] further attained near-optimal sample complexity for reward-free exploration under joint DP. More recently, Rio et al. [16] introduced a differentially private policy-gradient algorithm based on trust-region methods, demonstrating that the privacy noise can be absorbed into the trust-region constraint with minimal sacrifice of theoretical guarantees. However, these works uniformly target the protection of user-level or trajectory-level personal data, rather than the structural information of the training environment itself.

The environment-privacy problem was first identified by Pan et al. [7], who showed that a GridWorld agent retains its training-environment trajectory even after all obstacles are removed, and demonstrated that a genetic-algorithm-based attacker can recover the original map by comparing candidate policies against the target policy. Mo et al. [2] and Ilahi et al. [1] subsequently provided systematic taxonomies of DRL security threats, both observing that defenses against environment-privacy leakage remain severely underdeveloped. The only dedicated defense to date is differentially private deep reinforcement learning (DP-DRL) [17], which perturbs the agent’s observation input during training using the exponential mechanism. This approach, however, has two significant limitations. First, it requires a forward pass over the full observation space before each training step, which becomes computationally prohibitive in high-dimensional settings. Second, after sufficient training the policy still converges to a representation that encodes genuine environment information, so the structural leakage is not fundamentally eliminated.

1.2. Contributions

The above limitations motivate a fundamentally different strategy: rather than intervening in the training process, randomization is applied directly at the deployment stage to the action outputs of an already-trained model. This strategy requires no modification to the training pipeline, incurs only $O(|A|)$ additional computation per step regardless of the observation dimensionality, and directly targets the query interface through which an attacker extracts environment information. The main contributions of this study are summarized as follows.

(1)

A deployment-time action randomization defense, termed Entropy-Driven Action Randomization (EDAR), is proposed, which replaces deterministic greedy action selection with probabilistic sampling based on a normalized Q-value utility function inspired by the exponential mechanism, without modifying the trained policy.

(2)

A state-adaptive dynamic privacy budget strategy is designed, guided by the Shannon entropy of the action distribution, applying stronger perturbation in states with highly deterministic outputs while preserving more utility in states with low determinism.

(3)

A simulated-annealing-based policy inversion attack is designed as a quantitative tool for evaluating environment-privacy leakage, using the normalized recovery rate (NRR) as the privacy metric. Experiments on GridWorld environments of four scales (7 × 7 to 13 × 13) show that, at a privacy level comparable to a strong fixed-budget baseline ($\epsilon = 1$), the proposed dynamic-budget method raises the reward retention rate from 6.2% to 78.9%, thereby achieving a markedly better privacy–utility trade-off than the existing training-perturbation-based method [17].

The $\epsilon$-differential-privacy property established for the proposed sampling mechanism (Equation (6)) is a per-query, mechanism-level guarantee of indistinguishability between neighboring observation states under a fixed trained model; it is not an end-to-end differential-privacy guarantee with respect to neighboring training environments. What is formally protected is therefore the indistinguishability of action outputs at inference, whereas the protection of the environment structure against reconstruction is demonstrated empirically through the policy inversion attack in Section 5, with a full end-to-end analysis left for future work.

1.3. Organization

The remainder of this paper is organized as follows. Section 2 introduces the preliminaries on Double DQN, differential privacy, and the exponential mechanism. Section 3 formulates the environment-privacy problem and the threat model. Section 4 presents the proposed deployment-time action randomization defense and the dynamic privacy budget strategy. Section 5 describes the simulated-annealing-based policy inversion attack and the privacy metric. Section 6 reports the experimental results, and Section 7 concludes the paper.

2. Preliminaries

2.1. Double DQN

Reinforcement learning formalizes sequential decision-making as a Markov Decision Process (MDP), defined by the tuple $(S, A, T, R, \gamma )$, where $S$ is the state space, $A$ is the action space, $T$ is the transition dynamics, $R$ is the reward function, and $\gamma \in (0, 1)$ is the discount factor. At each time step, the agent observes state $s$, selects action $a$ according to policy $\pi$, receives reward $r$, and transitions to $s^{\prime }$. The agent aims to find the optimal policy $\pi *$ that maximizes the expected cumulative discounted reward. The optimal action-value function $Q*(s, a)$ satisfies the Bellman optimality equation:

$$Q*(s, a) = E[r + \gamma {max}_{a^{\prime }} Q*(s^{\prime }, a^{\prime }) | s, a]$$

(1)

The Deep Q-Network (DQN) [18] approximates $Q*(s, a)$ with a neural network $Q(s, a; \theta )$ by minimizing the temporal-difference (TD) loss. Standard DQN suffers from overestimation bias because the same network is used for both action selection and evaluation. Double DQN [19] decouples these two operations: the online network $\theta$ selects the greedy action, while the target network ${\theta }^{-}$ evaluates it:

$$y = r + \gamma Q(s^{\prime }, {argmax}_{a^{\prime }} Q\left(s^{\prime }, a^{\prime }; \theta \right); {\theta }^{-})$$

(2)

This separation effectively suppresses overestimation and produces higher-quality policies, which is important for the present work: a well-converged policy provides a stable Q-value landscape on which the deployment-time perturbation mechanism can operate predictably. Double DQN (DDQN) is therefore adopted as the base algorithm; the proposed defense is, however, compatible with any value-based DRL algorithm.

2.2. Differential Privacy

Differential privacy (DP) [8] provides a rigorous, worst-case bound on the information an adversary can extract from the output of a randomized algorithm.

Definition 1 ($\epsilon$-Differential Privacy).

A randomized algorithm $M$ satisfies $\epsilon$-differential privacy if, for any two neighboring inputs $D$ and $D^{\prime }$differing in a single element, and for all measurable output sets $\Omega$:

$$Pr[M(D) \in \Omega ] \le e^{\mathsf{\epsilon }} · Pr[M(D^{\prime }) \in \Omega ]$$

(3)

The privacy budget $\epsilon$ governs the privacy–utility trade-off: a smaller $\epsilon$ enforces stronger privacy at the cost of greater output randomization. Achieving DP requires injecting noise calibrated to the global sensitivity of the query, defined for a query function f as $\Delta f = {max}_{D\sim D^{\prime }} \Vert f(D) - f(D^{\prime })\Vert$, i.e., the largest change in the output induced by altering one element of the input.

2.3. Exponential Mechanism

For non-numerical outputs such as discrete action selection, the exponential mechanism [20] is the canonical DP tool. Given a utility function $u$: $D \times R \to \mathbb{R}$ with global sensitivity $\Delta u = {max}_{D\sim D^{\prime }, r} |u(D, r) - u(D^{\prime }, r)|$, the exponential mechanism samples output $r \in R$ with probability:

$$Pr[M outputs r] \propto exp( {\displaystyle \frac{\epsilon ·u(D, r)}{2\Delta u}})$$

(4)

By construction, $M$ satisfies $\epsilon$-differential privacy [20].

3. Problem Formulation and Threat Model

3.1. Problem Definition

A DRL agent is trained to perform a task in an environment $E$ whose transition dynamics are determined by the structural configuration of $E$ (e.g., the obstacle layout of a grid map). Upon convergence, the policy ${\pi }_{\theta }$ and action-value function $Q(s, a; \theta )$ implicitly encode the structure of $E$: the learned state-to-action mapping reflects, at each location, the local layout of the surrounding environment. This encoding is the source of environment-privacy leakage—when the agent is deployed as a query-accessible service, an attacker can invert this mapping to reconstruct the structure of $E$. Unlike conventional privacy threats whose protected asset is personal data, here the protected asset is the environment structure itself.

3.2. Threat Model

This study assumes a black-box threat model reflecting realistic API-level access:

(1)

Query access. The attacker can place the agent in any valid state $s$ and observe its action output $a = {\pi }_{\theta (s)}$.

(2)

Repeated querying. For a given state, the attacker may issue $T$ repeated queries and record the empirical action frequencies; for a randomized policy, this estimates the underlying action distribution.

(3)

Prior knowledge. The attacker knows the state and action space definitions and boundary conditions, but has no access to the model parameters or the training process.

The defense goal is to prevent accurate reconstruction of $E$ from queried action sequences while preserving sufficient policy utility for the deployed task.

4. Deployment-Time Action Randomization Defense

This section presents the proposed defense, EDAR. In standard DRL, the agent acts deterministically as $a = {argmax}_a Q(s, a; \theta )$, making its behavior fully predictable and the training environment recoverable from action sequences. The proposed defense replaces this deterministic selection with probabilistic sampling at deployment time, without modifying the trained policy.

4.1. Action Randomization via the Exponential Mechanism

To apply the exponential mechanism to action selection, a utility function is needed. Since environment privacy concerns the indistinguishability of action outputs at neighboring observation states (states differing in a single observation component), and since a small structural change can induce large fluctuations in raw Q-values through the nonlinear network, we normalize the Q-values to a bounded range.

Definition 2 (Utility Function).

Given state $s$, the utility of selecting action $a$ is defined as the min–max-normalized Q-value:

$$u(s, a)={\displaystyle \frac{Q(s, a; \theta )-Q_{min}(s)}{Q_{max}(s)-Q_{min}(s)+\delta }}$$

(5)

where $Q_{max}(s)$ and $Q_{min}\left(s\right)$ are the maximum and minimum Q-values at $s$, and $\delta >0$ is a small smoothing constant that prevents division by zero when $Q_{max}(s)$ = $Q_{min}\left(s\right)$ ($\delta = {10}^{-6}$ in all experiments). Beyond this min–max range normalization and the smoothing constant $\delta$, Equation (5) introduces no additional temperature or scaling. By construction $u(s,a)\in [0,1]$, for any two neighboring states the utility differs by at most 1, giving a global sensitivity $\Delta u=1$ independent of the network architecture or training dynamics. This bound holds because the min–max normalization of Equation (5) maps all Q-values into [0, 1] before they enter the exponential mechanism; the sensitivity of the raw Q-network outputs to input perturbations therefore does not affect the DP guarantee, since any perturbation—however large in the raw Q-value space—is absorbed by the normalization and cannot push $u(s,a)$ outside [0, 1].

Applying the exponential mechanism with privacy budget $\epsilon$, the agent samples action $a$ according to

$$P(a | s)={\displaystyle \frac{exp(\frac{\epsilon · u\left(s, a\right)}{2})}{\sum _{a^{\prime }\in A}exp(\frac{\epsilon · u\left(s, a^{\prime }\right)}{2})}}$$

(6)

By the guarantee of the exponential mechanism, Equation (6) satisfies $\epsilon$-differential privacy with respect to neighboring observation states. A large $\epsilon$ concentrates the distribution on the greedy action, while a small $\epsilon$ flattens it toward uniform, trading utility for stronger perturbation.

The $\epsilon$-DP property established here is a per-query, mechanism-level guarantee with respect to neighboring observation states under a fixed trained model $\theta$. It does not directly constitute a differential-privacy guarantee with respect to neighboring training environments. A full end-to-end analysis spanning training and deployment is left for future work; the empirical defense effectiveness is instead evaluated through the policy inversion attack in Section 5.

4.2. State-Adaptive Dynamic Privacy Budget

A fixed $\epsilon$ treats all states uniformly, which is suboptimal: states whose Q-values are already near-uniform leak little environment information and need little perturbation, whereas states with sharply peaked Q-values are most informative to an attacker and require stronger protection.

As a heuristic motivation, the single-query leakage can be viewed through the mutual information $I(E; a) = H(a) - H(a | E)$, where $H(a | E)$ decomposes over states as an average of the per-state action entropy $H(a | s)$. Raising $H(a | s)$ in low-entropy states therefore tends to reduce leakage. A rigorous treatment would need to account for how randomization reshapes the marginal distribution $P(a)$, which is beyond the present scope. This motivates allocating the budget by the per-state entropy itself.

The normalized Shannon entropy of the Softmax-transformed Q-values serves as the proxy for the per-state determinism:

$$H(a\mid s)=-{\displaystyle \frac{ {\sum }_{a\in A} p(a\mid s)\ln p(a\mid s)}{ln\mid A\mid }}$$

(7)

where $p(a | s)$ is the Softmax distribution over Q-values with temperature 0.5 and the factor $1/ln|A|$ normalizes $H$ to $[0, 1]$. $H(a | s) \approx 0$ indicates a near-deterministic policy (high leakage risk) and $H(a | s) \approx 1$ a near-uniform one (low risk). The budget is then assigned by a linear mapping. The entropy is deliberately computed from the Softmax of the Q-values rather than from $p\left(a|s\right)$, because the budget $\epsilon (s)$ is itself determined by the per-state determinism, whereas $p\left(a|s\right)$ already depends on $\epsilon (s)$ through the exponential mechanism; using the entropy of $p\left(a|s\right)$ to set $\epsilon (s)$ would therefore be circular.

$$\epsilon (s)={\epsilon }_{min}+({\epsilon }_{max}-{\epsilon }_{min}) · H(a | s)$$

(8)

where ${\epsilon }_{min}=1$ and ${\epsilon }_{max} = 10$; these values are fixed across all experiments. The bounds are chosen such that ${\epsilon }_{max}$ yields an action distribution strongly biased toward the greedy action, while ${\epsilon }_{min}$ produces a substantially flattened distribution that meaningfully disperses probability mass across actions. A sensitivity analysis of these bounds is provided in Section 6.7. Substituting Equation (8) into Equation (6) gives the state-adaptive action sampling distribution:

$$P(a | s) = {\displaystyle \frac{exp(\frac{\epsilon (s) · u\left(s, a\right)}{2})}{\sum _{a^{\prime }\in A}exp(\frac{\epsilon \left(s\right)· u\left(s, a^{\prime }\right)}{2})}}$$

(9)

Thus, the dynamic strategy applies stronger perturbation precisely where the policy is most informative and relaxes it where the policy is already near-uniform, which is shown in Section 6 to yield a better privacy–utility trade-off.

4.3. Overall Algorithm

Training uses standard Double DQN with no modification; privacy protection acts only at inference, since leakage occurs after deployment. Figure 1 illustrates the overall workflow. The complete procedure is summarized in Algorithm 1. The per-step overhead beyond a single forward pass is $O(|A|)$, independent of the observation dimensionality—in contrast to the $O(n)$ observation-space perturbation of DP-DRL [17], where $n$ is the observation dimension.

Algorithm 1 Entropy-Driven Action Randomization (EDAR)

$\mathbf{\text{Require:}} \mathrm{trained} \mathrm{Q}-\mathrm{network} \mathrm{parameters} \theta ; \mathrm{action} \mathrm{set} A; \mathrm{budget} \mathrm{bounds} {\epsilon }_{min}, {\epsilon }_{\mathrm{m}\mathrm{a}\mathrm{x}}; \mathrm{smoothing} \mathrm{constant} \delta ; \mathrm{maximum} \mathrm{episode} \mathrm{length} T_{max}$

$\mathbf{\text{Ensure:}} \mathrm{a} \mathrm{privacy}-\mathrm{protected} \mathrm{action} a_t$ at each step

1: $\mathbf{for} \mathrm{t}=1, 2, \dots , T_{max}$ do

2:     $\mathrm{Receive} \mathrm{current} \mathrm{state} s_t$

3:     $\mathrm{Compute} Q(s_t, a; \theta ) \mathrm{for} \mathrm{all} a \in A$ via a single forward pass

4:     $Q_{max} \leftarrow max Q(s_t, a; \theta ); Q_{min} \leftarrow min Q(s_t, a; \theta )$

5:     $\mathbf{for} \mathrm{each} a \in A$ do

6:     $u(s_t, a) \leftarrow (Q(s_t, a; \theta )-Q_{min}) / (Q_{max}-Q_{min}+\delta )$

7:     end for

8:     $\mathrm{Compute} \mathrm{Softmax} \mathrm{distribution} p(a|s_t)$ over Q-values

9:     $H(a|s_t) \leftarrow -(1 / ln|A|) {\Sigma }_a p(a|s_t) ln p(a|s_t)$

10:    $\epsilon (s_t) \leftarrow {\epsilon }_{min}+({\epsilon }_{\mathrm{m}\mathrm{a}\mathrm{x}}-{\epsilon }_{min}) · H(a|s_t)$

11:    $\mathbf{for} \mathrm{each} a \in A$ do

12:        $P(a | s_t) \leftarrow exp(\epsilon (s_t)·u(s_t, a)/2) / {\Sigma }_{a^{\prime }} exp(\epsilon (s_t)·u(s_t, a^{\prime })/2)$

13:    end for

14:    $\mathrm{Sample} \mathrm{action} a_t ~ P(·|s_t)$

15:    $\mathrm{Output} a_t$

16:    $\mathrm{Execute} a_t; \mathrm{the} \mathrm{environment} \mathrm{returns} \mathrm{the} \mathrm{next} \mathrm{state} s_{t+1}$

17:    $\mathbf{if} s_{t+1}$ is terminal then break

18:    $s_t=s_{t+1}$

19: end for

5. Policy Inversion Attack via Simulated Annealing

To quantitatively evaluate the defense proposed in Section 4 under the black-box threat model of Section 3, we design an attack that attempts to reconstruct the training environment from action queries. The genetic-algorithm (GA) attack of Pan et al. [7] retrains a DRL agent for every candidate environment, which is computationally prohibitive when evaluating a defense across multiple maps. We instead adopt a simulated-annealing-based attack that operates entirely through black-box queries to the deployed agent, with no retraining.

5.1. Attack Principle

The attack exploits the locality of the encoding: in a navigation task, the agent’s action preference at a given position primarily reflects the local environment structure around that position. The attacker therefore searches for a candidate map whose induced state observations cause the target agent to produce action responses matching those of the true map; iteratively refining the candidate to maximize this matching score progressively reveals the original environment.

5.2. Attack Algorithm

The attack proceeds in two stages: reference-action collection and map search.

Reference-action collection. For every reachable position $c$ in the true environment, the attacker queries the agent $T$ times and records the majority-voted action as the reference action $a_{ref}(c)$. The full set $\mathcal{R} = \{(c, a_{ref}(c))\}$ serves as the matching target for the subsequent search. For a deterministic policy $T = 1$ suffices; for a randomized policy, repeated queries reduce per-query noise through majority voting.

Map search. The attacker searches over candidate maps satisfying four structural constraints: walls along the boundary, fixed start and goal positions, full connectivity among non-wall cells, and the absence of any 2 × 2 wall block. Given a candidate map $m$, the score is defined as the matching rate between the actions induced by $m$ and the reference actions:

$$Score(m,R)={\displaystyle \frac{1}{\mid R\mid }} \sum _{(c,a_{ref(c)})\in R}𝟙[{\pi }_{\theta } (o_m (c))=a_{ref(c)}]$$

(10)

where $o_m (c)$ is the observation reconstructed at position $c$ from $m$, ${\pi }_{\theta }(·)$ denotes a single black-box query to the target agent and $𝟙[·]$ is the indicator function. Here $Score(m, R) \in [0, 1]$, with a higher value indicating that $m$ induces observations more consistent with those of the true map.

We adopt simulated annealing [21] over the discrete map space; its temperature-controlled Metropolis rule balances global exploration and local exploitation. The neighborhood operator flips one or two interior cells (two-cell flips at high temperatures to escape local optima, single-cell flips at low temperatures for refinement); any flipped candidate violating the structural constraints is discarded. Multiple independent restarts are used, and the highest-scoring candidate across all restarts is returned as the inferred map. The complete procedure is summarized in Algorithm 2.

With reference to Algorithm 2, two mechanisms make the proposed defense effective against this attack. In the reference-collection stage (lines 2–5), the exponential mechanism causes repeated queries on the same state to return different actions, so the majority-voted reference action may deviate from the true greedy action—poisoning the matching target from the outset. In the scoring stage (via Equation (10)), even a candidate map identical to the true map can produce non-greedy actions, lowering the score and obscuring the optimum. A smaller $\epsilon (s)$ amplifies both effects; the dynamic budget concentrates this protection in precisely the low-entropy states that would otherwise leak the most information.

Algorithm 2 Simulated-Annealing-Based Policy Inversion Attack

$\mathbf{\text{Require:}} \mathrm{target} \mathrm{agent} (\mathrm{black}-\mathrm{box}); \mathrm{grid} \mathrm{size} H\times W; \mathrm{start} \mathrm{and} \mathrm{goal} \mathrm{positions}; \mathrm{query} \mathrm{count} T; \mathrm{initial} \mathrm{temperature} {\tau }_0; \mathrm{minimum} \mathrm{temperature} {\tau }_{min}; \mathrm{cooling} \mathrm{rate} \alpha \in (0, 1); \max \mathrm{iterations} N; \mathrm{number} \mathrm{of} \mathrm{restarts} K$

$\mathbf{\text{Ensure:}} \mathrm{inferred} \mathrm{map} m\ast$

▷ Stage 1: Reference-action collection

1: $\mathbf{for} \mathrm{each} \mathrm{reachable} \mathrm{position} c$ in the true map do

2:     $\mathrm{Reconstruct} \mathrm{the} \mathrm{observation} o(c); \mathrm{query} \mathrm{the} \mathrm{agent} T \mathrm{times} \mathrm{on} o(c)$

3:     $a_{ref}(c)\leftarrow \mathrm{majority}-\mathrm{voted} \mathrm{action} \mathrm{over} \mathrm{the} T$ queries

4: end for

5: $R \leftarrow \{(c, a_{ref}(c))\}$    ▷ reference set used as the matching target

▷ Stage 2: Map search with simulated annealing

6: $m* \leftarrow None; s* \leftarrow -\infty$

7: $\mathbf{for} k=1, 2, \dots , K$ do    ▷ K independent restarts to escape local optima

8:      $\mathrm{Generate} \mathrm{a} \mathrm{random} \mathrm{initial} \mathrm{map} m_0$ satisfying the four structural constraints

9:      $m \leftarrow m_0; s \leftarrow Score(m, R)$ by Equation (10)

10:     $m_{best} \leftarrow m; s_{best} \leftarrow s; \tau \leftarrow {\tau }_0$

11:     $\mathbf{for} n=1, 2, \dots , N$ do    ▷ simulated annealing iterations

12:           $\mathrm{Choose} \mathrm{flip} \mathrm{size}: 2 \mathrm{cells} \mathrm{if} \tau >{\tau }_0/2$, else 1 cell

13:           $\mathrm{Flip} \mathrm{the} \mathrm{selected} \mathrm{interior} \mathrm{cell} (s) \mathrm{to} \mathrm{obtain} \mathrm{candidate} m^{\prime }$

14:           $\mathbf{if} m^{\prime }$ violates any structural constraint then continue

15:           $s^{\prime } \leftarrow Score(m^{\prime }, R)$ by Equation (10)

16:           $\Delta \leftarrow s^{\prime }-s$

17:           $\mathbf{if} \Delta >0 \mathrm{or} Uniform(0,1)<exp(\Delta /\tau )$ then

18:               $m \leftarrow m^{\prime }; s \leftarrow s^{\prime }$   ▷ Metropolis acceptance

19:               $\mathbf{if} s>s_{best}$ then

20:                   $m_{best} \leftarrow m; s_{best} \leftarrow s$

21:               end if

22:            end if

23:            $\tau \leftarrow \alpha · \tau$ ▷ geometric cooling schedule

24:            $\mathbf{if} \tau <{\tau }_{min}$ then break

25:     end for

26:     $\mathbf{if} s_{best}>s* \mathbf{then} m* \leftarrow m_{best}; s* \leftarrow s_{best}$

27: end for

28: $\mathbf{return} m*$

5.3. Privacy Metric

To quantify the success of the attack and, conversely, the strength of the defense, we adopt the normalized recovery rate (NRR) as the privacy metric. Since boundary walls are public knowledge under the threat model of Section 3, only interior cells contribute. NRR is defined as the fraction of interior cells whose state matches between the inferred and the true maps:

$$NRR={\displaystyle \frac{1}{|I|}} \sum _{c\in I}𝟙[\widehat{m}(c)=m(c)]$$

(11)

where $I$ is the set of interior cells, $m(c)$ and $\widehat{m}(c)$ are the true and inferred states of cell $c$. $NRR \in [0, 1]$: a value of 1 indicates full interior recovery, while values close to the random-guessing baseline indicate that the defense has prevented meaningful reconstruction. NRR is the primary privacy metric used throughout Section 6.

The per-query $\epsilon$-DP guarantee and NRR are connected empirically rather than analytically: the DP property bounds the information revealed by each action query, which reduces the discriminative power of the simulated annealing (SA) scoring function and lowers NRR; Section 6.5 characterizes this relationship by varying the query count $T$. In the GridWorld setting, NRR directly measures leakage of the sensitive information—the interior wall layout—since the start and goal positions are assumed to be known to the attacker under the threat model of Section 3. NRR does not, however, capture coarser structural leakages such as room topology or connectivity patterns that an attacker might infer without precise cell-level reconstruction; this is a known limitation of cell-level metrics in more complex environments, noted in Section 7.

6. Simulations and Results

This section evaluates EDAR on GridWorld environments of four scales, using the simulated-annealing attack (Algorithm 2) as the evaluation tool and NRR as the primary privacy metric. We verify the convergence of the base Double DQN agent, quantify EDAR’s privacy and utility against fixed-budget baselines (Section 6.2 and Section 6.3), compare it with four training-stage DP methods (Section 6.4), assess its robustness to the attacker’s query budget (Section 6.5), validate the SA attack against the prior-art GA attacker (Section 6.6), and analyze the sensitivity of the results to the budget bounds ${\epsilon }_{min}$ and ${\epsilon }_{max}$ and the robustness of the advantage to the choice of budget signal (Section 6.7).

6.1. Experimental Setup

Following prior work on environment-privacy evaluation [7,17], we use GridWorld environments of four scales—7 × 7, 9 × 9, 11 × 11, and 13 × 13—with 25, 49, 81, and 121 interior cells, respectively. Each cell is either a wall or a free space; boundary cells are walls, and interior cells form the private structure the attacker tries to recover. The agent navigates from a fixed start to a fixed goal via four discrete actions (up, down, left, right). Observations are 8-dimensional normalized LiDAR distance vectors, more closely reflecting realistic robotic perception than absolute coordinates. Representative maps of the four scales are shown in Figure 2.

The base Double DQN agent uses a fully connected Q-network (8–64–64–4) with ReLU activations and Xavier initialization. Training runs for 1000 episodes per environment with the Adam optimizer and an ε-greedy schedule. The reward is +1 for reaching the goal, −0.05 for hitting a wall, and 0 otherwise. The Baseline Double DQN agent converges within 1000 episodes on all four scales, providing a stable reference policy for the subsequent privacy and utility evaluations. Full hyperparameters are listed in

Table 1. Training hyperparameters of the Double Deep Q-Network (DDQN) agent.

Parameter	Symbol	Value
Learning rate	$lr$	0.001
Discount factor	$\gamma$	0.99
Episodes	—	1000
Replay buffer size	—	15,000
Target network update interval	—	500
Initial exploration rate	—	1
Final exploration rate	—	0.05
Exploration decay rate	—	0.996

.

EDAR uses $\delta = {10}^{-6}$, ${\epsilon }_{min} = 1$, ${\epsilon }_{max} = 10$ (fixed across all scales). The simulated-annealing attack (Algorithm 2) uses parameters listed in

Table 2. Parameters of the simulated-annealing-based policy inversion attack.

Parameter	Symbol	Value
Initial temperature	${\tau }_0$	2.0
Minimum temperature	$\tau ₘᵢₙ$	0.01
Cooling rate	$\alpha$	0.98
Maximum iterations per restart	$N$	3000
Number of independent restarts	$K$	3
Queries per state	$T$	20(EDAR); 1(NoDefense)

; unless otherwise stated, the per-state query count is $T = 20$.

All experiments were implemented in Python 3.9.22 with PyTorch 2.5.1 (CUDA 12.1).

6.2. Privacy Protection

We compare EDAR against the no-defense baseline and three fixed-budget schemes corresponding to strong ($\epsilon = 1$), moderate ($\epsilon = 5$), and weak ($\epsilon = 10$) privacy regimes.

Figure 3 visualizes the inferred maps under each scheme on the four environments, with red crosses marking interior cells whose reconstructed state differs from the true map. Under no defense, the attacker fully recovers the interior structure on the 7 × 7, 9 × 9, and 11 × 11 maps (NRR = 1.000) and still reaches NRR = 0.843 on the 13 × 13 map, confirming that the deterministic greedy output leaks the training environment in considerable detail. The near-perfect recovery on the three smaller maps demonstrates that a black-box query-only attacker can invert the action-to-structure mapping almost exactly when no randomization is applied; the slight degradation on the 13 × 13 map reflects the larger search space rather than any inherent protection. As $\epsilon$ decreases, the red crosses become denser and increasingly spread across the interior, with the most pronounced improvement on larger maps where the cumulative effect of per-state randomization is greatest. The quantitative NRR values in

Table 3. NRR of the simulated-annealing attack under different defenses across the four GridWorld scales. Lower is better. Results report means ± standard deviations across three independent training seeds.

Scale	No Defense	$\mathit{\epsilon }\mathbf{=}\mathbf{1}$	$\mathit{\epsilon }\mathbf{=}\mathbf{5}$	$\mathit{\epsilon }\mathbf{=}\mathbf{10}$	EDAR
7 × 7	1.00 ± 0.00	0.95 ± 0.05	1.00 ± 0.00	1.00 ± 0.00	0.95 ± 0.02
9 × 9	1.00 ± 0.00	0.77 ± 0.01	0.90 ± 0.05	0.97 ± 0.01	0.82 ± 0.04
11 × 11	0.96 ± 0.04	0.69 ± 0.04	0.85 ± 0.08	0.86 ± 0.04	0.73 ± 0.01
13 × 13	0.84 ± 0.02	0.66 ± 0.03	0.78 ± 0.04	0.77 ± 0.02	0.68 ± 0.03
Average	0.95	0.77	0.88	0.90	0.79

corroborate this visual trend.

Table 3 reports per-map and average NRR. The strong fixed budget ($\epsilon = 1$) reaches an average NRR of 0.77; moderate and weak budgets ($\epsilon = 5, 10$) yield 0.88 and 0.90, confirming the monotonic relationship between $\epsilon$ and protection strength. On the 7 × 7 map, $\epsilon = 5$ and $\epsilon = 10$ both fail to deviate from the no-defense baseline (NRR = 1.00 ± 0.00), indicating that weak randomization is insufficient on small search spaces.

EDAR achieves an average NRR of 0.79—approaching the strongest fixed budget—and tends to concentrate reconstruction errors in corridors, corners, and dense-obstacle regions, which correspond to the low-entropy, high-leakage-risk states where EDAR assigns its smallest $\epsilon (s)$; open multi-path areas with higher action entropy, on the other hand, are perturbed less and remain easier to recover. This spatial pattern is consistent with the design intent of the entropy-driven budget allocation.

6.3. Utility Analysis

We evaluate four metrics covering both privacy and utility, summarized in

Table 4. Average privacy and utility metrics across the four GridWorld scales. RR: reward retention = mean episode return of the agent/mean episode return of the no-defense baseline × 100%; Avg. Return: mean episode return of the agent; SR: success rate (percentage of episodes reaching the goal); Steps: average number of steps per episode; NRR Red: NRR reduction (baseline NRR minus protected NRR; larger is better). Results report means ± standard deviations across three independent training seeds.

Scheme	RR	Avg. Return	SR	Steps	NRR Red
No Defense	100%	+1.00 ± 0.00	100%	15	—
ε = 1	6.2% ± 1.2%	+0.06 ± 0.01	99.3%	134	19.3%
ε = 5	99.4% ± 0.1%	+0.99 ± 0.00	100%	23	7.0%
ε = 10	100% ± 0.0%	+1.00 ± 0.00	100%	16	5.3%
EDAR	78.9% ± 1.2%	+0.79 ± 0.01	100%	58	16.4%

.

The weak fixed budget ($\epsilon = 10$) preserves utility almost perfectly (reward retention of 100%) but offers little protection (NRR reduction of only 5.3%). The strong fixed budget ($\epsilon = 1$) attains substantial protection (NRR reduction of 19.3%) at the cost of catastrophic utility loss—reward retention drops to 6.2% and the average step count balloons to 134, roughly 9× the baseline—because uniform strong perturbation forces the agent to wander even in low-risk open regions.

EDAR breaks this trade-off: it achieves an NRR reduction approaching that of $\epsilon =1$ with reward retention of 78.9%, success rate of 100%, and 58 average steps. EDAR thus delivers privacy approaching the strongest fixed budget while recovering most of the utility cost, by concentrating strong perturbation on low-entropy states and preserving near-optimal action selection elsewhere.

The reward-retention ratio alone can be misleading when step counts differ substantially between agents, because a policy that collides with walls frequently accumulates negative reward even when it eventually reaches the goal. The raw mean episode return is therefore reported alongside the ratio in Table 4 to allow direct comparison.

6.4. Comparison with Training-Stage DP Methods

Although the four training-stage methods compared below were originally designed for data or trajectory privacy rather than environment-structure privacy, they represent the closest existing DP-based defenses for DRL agents and are included to examine whether training-stage DP noise can inadvertently protect environment structure as a side effect. We compare EDAR with four representative training-stage methods that inject DP noise at different points in the training pipeline: (i) DDQN-DP-SGD [11], which adds noise to clipped Q-Network gradients (clip norm $C = 1.0$, noise multiplier $\sigma = 0.1$, batch size 64, 3000 training episodes); (ii) Proximal Policy Optimization (PPO)-DP-SGD [16], which applies the DP-SGD framework to PPO’s policy-gradient updates (clip norm $C = 1.0$, noise multiplier $\sigma = 0.1$, rollout length 128, 4 update epochs per rollout, learning rate 3 × 10⁻⁴, 3000 episodes); (iii) DQN-RewPerturb, which adds Laplace noise to per-step rewards, based on the Q-learning DP framework of Wang and Hegde [10] (DP budget ${\epsilon }_r = 21$, Laplace noise scale = $\Delta r/{\epsilon }_r \approx 0.05$ with reward sensitivity $\Delta r = 1.05$, 3000 episodes); and (iv) DP-DRL, which perturbs the agent’s observation input during training via the exponential mechanism (20,000 training episodes; all other hyperparameters follow the settings in [17]). For each baseline, the privacy hyperparameters follow the settings reported in its original work. All four methods produce deterministic policies and are evaluated under the same simulated-annealing attack as EDAR.

Table 5. Comparison between EDAR and four training-stage differential privacy (DP) methods on the four GridWorld scales, in terms of per-map NRR, average NRR (Avg. NRR), and utility metrics (RR, SR, Steps). Lower NRR and higher reward retention indicate better privacy and utility, respectively.

Method	DP Layer	7 × 7	9 × 9	11 × 11	13 × 13	Avg. NRR	RR	SR	Steps
No Defense	—	1.00	1.00	1.00	0.84	0.96	100%	100%	15
DDQN-DP-SGD	Gradient	1.00	1.00	0.97	0.85	0.95	100%	100%	16
PPO-DP-SGD	Policy Gradient	1.00	0.93	0.95	0.79	0.91	74.8%	75.0%	134
DQN-RewPerturb	Reward	1.00	1.00	1.00	0.78	0.94	100%	100%	15
DP-DRL	Observation	1.00	1.00	0.98	0.89	0.96	98.3%	100%	17
EDAR	Action	0.95	0.82	0.73	0.68	0.79	78.9%	100%	58

reports both NRR and the corresponding utility metrics.

Table 5 reveals a fundamental dilemma facing training-stage methods. DDQN-DP-SGD and DQN-RewPerturb retain near-perfect utility, but this very fact implies that the training noise was too mild to disturb the policy: the converged Q-network still encodes the environment accurately, and the deterministic argmax leaks it losslessly, leaving NRR around 0.95. PPO-DP-SGD goes to the opposite extreme—its utility is substantially reduced, owing to the inherently higher sensitivity of on-policy policy-gradient updates to additive DP noise and PPO’s stochastic inference rather than a failure to converge; yet, NRR remains 0.91 because the impaired policy output is still spatially distinguishable.

Training-stage DP thus faces an irreducible tension: too little noise and privacy fail; too much noise and the policy fail. EDAR escapes this dilemma by acting at the deployment stage: the trained Q-network is untouched, while deployment-time randomization directly disrupts the query channel that training-stage methods cannot reach.

6.5. Robustness to Query Budget

We vary the per-state query count $T$ from 1 to 150 on the 9 × 9 map and run the full attack independently at each value. Figure 4 shows the resulting NRR curves.

The no-defense baseline is flat: a deterministic policy reveals its action in a single query, so additional queries are redundant. Under EDAR, NRR increases with the per-state query count T as majority voting averages out the per-query randomness, rising from 0.694 at $T = 1$ to 0.959 at $T = 100$ with clearly diminishing returns thereafter; within the tested range (up to $T = 150$) it remains below the no-defense level, but the overall trend is upward.

This behavior is the empirical signature of differential-privacy composition rather than evidence of a permanent protective margin. Each query to a fixed state is an independent draw from the $\epsilon (s)$-differentially private mechanism of Equation (6), so by sequential composition [8] the T queries on that state consume a cumulative privacy budget of at most $T·\epsilon (s)$. In the low-entropy states to which EDAR assigns its smallest budget (${\epsilon }_{min} = 1$), the cumulative guarantee after $T = 100$ queries is therefore bounded only by $\epsilon \approx 100$, which is effectively vacuous: the greedy action remains the mode of the per-query distribution, so given enough queries the majority vote concentrates on it and recovery approaches the no-defense level. The reason the gap closes gradually rather than at once is that, at a small $\epsilon (s)$, the probability margin of the greedy action over its closest competitors is small, so many queries are needed for the majority vote to resolve these low-margin states. We therefore do not claim that majority voting is unable to neutralize the randomization, but rather that doing so requires a large per-state query budget.

Whether this limitation is operationally significant depends on whether the attacker is query-limited in the target deployment. A model served behind an authenticated, rate-limited, or monitored API—as is typical of clinical, navigation, and industrial-control endpoints—bounds the per-state query count and can flag anomalous streams of identical repeated queries, so the small-to-moderate $T$ regime in which EDAR retains a clear advantage is the operationally relevant one. An adversary with unrestricted offline black-box access is, by contrast, not query-limited, and against such an adversary EDAR in its present i.i.d.-resampling form—like any per-query DP mechanism—cannot prevent eventual reconstruction. We therefore position EDAR as a deployment-time defense for query-limited settings and state this assumption explicitly. A natural query-robust extension is to return a single cached sampled action per state, so that repeated queries to the same state reuse one draw and the cumulative budget remains $\epsilon (s)$ regardless of $T$; this, together with a formal cumulative-budget composition analysis, is left for future work.

6.6. Comparison of SA and GA Attacks

To verify that the SA-based attack provides a sufficiently strong evaluation baseline, we compare it directly with the genetic-algorithm (GA) attack of Pan et al. [7] on the no-defense agent.

Table 6. Comparison of simulated annealing (SA) and genetic-algorithm (GA) policy inversion attacks on the 7 × 7 GridWorld map under the no-defense DQN agent.

Attack	Retraining Required	Time (s)	NRR
GA	Yes	4511	0.895
SA	No	1.1	1.000

reports NRR and wall-clock time for both methods on the 7 × 7 map; the comparison is limited to this scale owing to GA’s prohibitive computational cost at larger map sizes.

SA achieves higher NRR (1.000 vs. 0.895) while reducing runtime by a factor of approximately 4100. The NRR difference is partly attributable to the query strategies of the two methods: SA evaluates each candidate map by issuing multiple direct queries to the frozen target policy per state, providing a dense fitness signal that guides the search; GA instead retrains a separate policy for each candidate and uses policy-level action matching as the scoring criterion, which introduces additional variance from the retraining process and yields a coarser fitness signal. The computational advantage of SA stems from this same design: avoiding retraining makes each fitness evaluation a lightweight forward-pass operation, whereas GA’s bottleneck is repeated policy training. These results show that SA is at least as strong as the prior-art GA attacker while being substantially more practical, and we therefore adopt it as the evaluation tool across all four map scales in this paper.

6.7. Sensitivity Analysis

To assess whether the reported advantage depends on the specific budget bounds, we vary each bound independently.

Table 7. Sensitivity of EDAR to the budget bounds ${\epsilon }_{min}$ and ${\epsilon }_{max}$ on the 9 × 9 map. Each configuration is evaluated over five independent SA attack trials. Lower NRR indicates stronger privacy, and higher reward retention indicates better utility. The configuration (${\epsilon }_{min} = 1$, ${\epsilon }_{max} = 10$) appears as the anchor row in both blocks; the two entries report independent sets of five trials, and the small numerical differences reflect sampling variance.

Configuration	NRR	RR	Steps
${\epsilon }_{max}=10$ fixed
${\epsilon }_{min}=1$	0.82 ± 0.06	74.0%	55
${\epsilon }_{min}=2$	0.83 ± 0.08	80.2%	51
${\epsilon }_{min}=3$	0.87 ± 0.07	90.5%	33
${\epsilon }_{min}=4$	0.91 ± 0.06	97.2%	29
${\epsilon }_{min}=1$ fixed
${\epsilon }_{max}=10$	0.83 ± 0.08	75.2%	54
${\epsilon }_{max}=9$	0.81 ± 0.07	71.6%	58
${\epsilon }_{max}=8$	0.79 ± 0.04	68.5%	64
${\epsilon }_{max}=7$	0.78 ± 0.03	52.6%	68

reports NRR and reward retention for a range of ${\epsilon }_{min}$ and ${\epsilon }_{max}$ configurations on the 9 × 9 map, with each configuration evaluated over five independent SA attack trials.

Varying ${\epsilon }_{min}$ over $\{1, 2, 3, 4\}$ with ${\epsilon }_{max}$ fixed at 10 produces a consistent privacy–utility trade-off: larger ${\epsilon }_{min}$ allocates a more relaxed budget to low-entropy states, yielding higher NRR and higher reward retention, while smaller ${\epsilon }_{min}$ strengthens protection at the cost of utility.

Varying ${\epsilon }_{max}$ over $\{10, 9, 8, 7\}$ with ${\epsilon }_{min}$ fixed at 1 shows a similar directional pattern: as ${\epsilon }_{max}$ decreases, NRR and reward retention both decline, indicating stronger privacy protection at higher utility cost. The modest overall NRR variation across this range indicates that EDAR is robust to the choice of ${\epsilon }_{max}$. Taken together, the results confirm that the conclusions of Section 6.2 and Section 6.3 hold across the examined range of parameter settings, and that varying the bounds shifts the operating point along the privacy–utility trade-off in a predictable and interpretable manner.

To verify that the advantage of EDAR is not specific to the entropy-based budget signal or the linear mapping in Equation (8), we compare two alternative per-state uncertainty proxies under the same experimental protocol (9 × 9 map, ${\epsilon }_{min} = 1$, ${\epsilon }_{max} = 10$, five independent SA attack trials). The first alternative replaces the linear mapping with a square-root mapping applied to the same entropy signal; the second replaces the entropy signal entirely with a top-2 Q-gap measure, defined as the normalized difference between the two highest Q-values, with the direction inverted so that a larger gap—indicating higher action certainty and greater privacy exposure—yields a smaller budget.

Table 8. Budget signal ablation on the 9 × 9 map (${\epsilon }_{min} = 1$, ${\epsilon }_{max} = 10$, five trials each). All three signals preserve the core EDAR advantage over fixed-budget baselines.

Budget Signal	NRR	RR
Entropy + Linear (default)	0.82 ± 0.06	74.0%
Entropy + Sqrt	0.84 ± 0.08	76.7%
Q-gap + Linear	0.85 ± 0.04	78.4%

reports the results. All three signals achieve NRR in the range 0.82–0.85, well below the no-defense baseline (NRR = 1.00), and reward retention in the range 74.0–78.4%, far above the fixed-budget baseline at $\epsilon = 1$. The NRR spread across the three signals is only 0.03, indicating that the EDAR advantage is robust to the choice of uncertainty measure and mapping function. The linear entropy mapping is therefore a practically convenient default rather than a critical design dependency.

7. Conclusions

This work addresses environment-privacy leakage in deep reinforcement learning, where an attacker with black-box query access can reconstruct the training environment from the agent’s action outputs. We propose Entropy-Driven Action Randomization (EDAR), a deployment-time defense that samples actions through the exponential mechanism with a state-adaptive privacy budget guided by the Shannon entropy of the action distribution. A simulated-annealing-based policy inversion attack is designed as the evaluation tool, and the normalized recovery rate (NRR) is adopted as the privacy metric. Across four GridWorld scales, EDAR approaches the privacy of the strongest fixed-budget baseline (Avg. NRR 0.79) while raising reward retention from 6.2% to 78.9%, substantially outperforming four representative training-stage DP methods.

The evaluation is currently confined to discrete GridWorld environments with goal-directed navigation tasks, and extending the entropy-driven mechanism to continuous state spaces and non-navigation objectives remains an open problem. Beyond the empirical scope, the differential-privacy analysis in this work is restricted to single-query, mechanism-level guarantees with respect to neighboring observation states.

The central open challenge for practical deployment is constraining and formally composing the per-query budget across a full black-box interaction: while the single-query $\epsilon$-DP guarantee of Equation (6) is rigorous, a realistic deployment in which the attacker can issue many queries requires a formal cumulative-budget composition analysis and, ultimately, an end-to-end privacy guarantee with respect to the training environment. We identify this as the primary direction for future work.

Secondary extensions include applying the entropy-driven budget mechanism to continuous-action algorithms and higher-dimensional perception inputs, developing query-robust variants such as returning a cached sampled action per state to maintain the cumulative budget at $\epsilon (s)$ regardless of query count, extending the framework to human–machine cooperative control settings where dynamic authority allocation strategies [22] share a conceptually similar state-adaptive design philosophy with the entropy-driven budget mechanism, and incorporating hierarchical attention-based uncertainty estimation techniques [23] to further refine per-state budget allocation.