Reinforcement Learning for Diffusion Policies in Robotics: A Survey and State-Based Locomotion Reproduction

Abstract

Diffusion policies model multimodal robot action sequences, but behavioral cloning does not directly optimize task return. We present a structured scoping review of reinforcement learning for generative robot policies and a bounded state-based locomotion reproduction. Four documented routes yielded 178 records, 162 unique candidates, and an 84-study evidence map. Hierarchical rules distinguish 41 direct reward-driven studies from 32 adjacent robotic, eight alternative-generator, and three non-robotic studies; a five-axis taxonomy codes initialization/data, interaction regime, optimized object, credit assignment, and generator. Under a fixed-final evaluation protocol on the Datasets for Deep Data-Driven Reinforcement Learning (D4RL) 1.1 Hopper benchmark, five diffusion policy policy optimization (DPPO) fine-tuning seeds improved over their run-recorded behavior-cloning initializations by a mean of 1261.2 return, with a seed-level standard deviation of 125.5 and a 95% confidence interval of 1105.3–1417.1; the five runs link to two recorded behavior-cloning checkpoints. A Gaussian-policy control also improved after proximal policy optimization, so the gain was not diffusion-specific. A full-chain backpropagation adaptation exhibited clear seed-dependent variation, a matched action-divergence intervention did not establish causal critical timesteps, and reducing denoiser evaluations from 20 to 2 lowered A100 latency from 30.97 to 3.85 ms while substantially reducing normalized score. The experiments are limited to state-based locomotion and do not validate visual manipulation.

1. Introduction

1.1. Motivation

Diffusion policies have reshaped imitation-based robot learning over the past few years. Chi et al. [1] showed that casting a robot’s visuomotor policy as a conditional denoising diffusion process—combined with receding-horizon control and visual conditioning—outperforms conventional behavioral-cloning architectures such as LSTM-GMM [2] and Implicit BC [3], by an average of $46.9\%$ over 15 manipulation tasks. Three properties of diffusion models underpin this result: they can flexibly approximate multimodal action distributions, scale gracefully to high-dimensional action sequences, and train stably under a regression-style denoising objective [4,5]. Together these make the diffusion policy a good match for the high-dimensional, multimodal action spaces of manipulation, locomotion, and navigation.

This strength comes with an inherited limitation. A policy trained only by behavioral cloning is strongly constrained by the quality and coverage of its demonstrations and commonly inherits their sub-optimal or inconsistent strategies [6,7]. Demonstrations are costly to collect and seldom reach the long tail of states a deployed agent will face. Specifying preferences through demonstrations alone is also awkward—asking a robot to align rather than stack objects, or to keep clear of certain regions, is hard to convey by example [7].

Reinforcement learning (RL) addresses this directly. Optimizing a policy against a scalar reward obtained from interaction can surpass expert performance, generalize beyond the demonstration manifold through exploration, and align behavior with task specifications or human preferences [8,9]. The marginal cost of the required interaction has fallen as well: massively parallel GPU simulators such as NVIDIA Isaac Lab [10] and ManiSkill3 [11], together with offline-to-online benchmarks [2,12], make online fine-tuning of diffusion policies practical.

Combining the two is, however, far from straightforward. Three structural conflicts recur in the literature [6,8,13]. First, the marginal likelihood ${\pi }_{\theta }(\mathit{a}\mid \mathit{s})$ of a diffusion policy generally does not admit a tractable closed-form expression, unlike the density of a standard Gaussian policy; this prevents the direct use of likelihood-ratio objectives such as proximal policy optimization (PPO) [14] and complicates exact entropy regularization as used by maximum-entropy actor-critic methods [15]. Second, obtaining the policy gradient by backpropagating through the full K-step denoising chain is expensive in memory and time and can exhibit the vanishing/exploding-gradient pathology familiar from long unrolled computations [16,17]. Third, each environment step requires several network evaluations, so the achievable control rate depends on the sampler, network, hardware, batching strategy, and surrounding control stack. We formalize these as challenges (C1)–(C3) in Section 2.4.

These tensions have driven a fast-growing body of work along several complementary fronts. One line replaces or bypasses the generally unavailable marginal likelihood—working directly with action gradients ${\nabla }_{\mathit{a}}Q(\mathit{s},\mathit{a})$ [18,19], reweighting the denoising loss with a critic-derived signal [20,21], or constructing tractable surrogates from normalizing-flow likelihoods [22] and flow-matching losses [23]. A second line treats the denoising process itself as a Markov decision process (MDP) and runs policy gradient at every denoising step [6], the route taken by the effective DPPO framework. A third targets alignment, fine-tuning a pre-trained policy under a reward model learned from human preferences while regularizing toward the original behavior [7,24]. A fourth attacks inference cost, either distilling the policy into a few-step or one-step generator [25,26,27] or allocating denoising steps adaptively by action criticality [13]. These fronts are often evaluated under different tasks and protocols; in our structured search, we did not identify a study that compares all four under one common protocol, and existing surveys cover only slices of the resulting design space (Section 1.5).

In the literature screened for this review, we did not identify prior work that pairs an auditable evidence map and a multi-axis technical taxonomy with a controlled state-based reproduction that dissects what RL fine-tuning concretely does to a diffusion policy—how large the gain over imitation is, and how the fine-tuned actions differ from the cloned ones. We address these two objectives at different scopes. The structured scoping review covers robotic diffusion policies conditioned on states, images, point clouds, or language and separates evidential eligibility from technical similarity. The focused empirical study (Section 6) reproduces and dissects the canonical DP-versus-DPPO comparison only on proprioceptive MuJoCo locomotion, with controlled checks across credit-assignment routes, two locomotion tasks, and inference configurations. It neither evaluates visual representations nor supports a general claim about vision-conditioned manipulation. Figure 1 uses mechanism and learning setting as two compact navigation coordinates; Section 3.1 supplies the formal taxonomy.

Figure 1. Overview of the survey. A behavior-cloned diffusion-policy core (center)—mapping observations through a conditional denoising model to an action sequence—is refined by RL. The top and right branches show two navigation coordinates used in Section 3: the policy-improvement mechanism (action-gradient, Q-weighting, proximity-based, BPTT) and the learning setting (online, offline, offline-to-online, preference-based). The formal taxonomy additionally distinguishes initial policy/data, optimized object, and inference generator (Section 3.1). The four left-hand labels map to three challenges (C1)–(C3) because denoising horizon and gradient conditioning are two aspects of C2; they denote design pressures, not universal failures. Evaluation (bottom) follows Section 5. Arrows trace the pre-training → fine-tuning → deployment flow toward robotic applications (right).Abbreviations: RL, reinforcement learning; BPTT, backpropagation through time.

1.2. Scope of This Survey

This survey sits at the intersection of diffusion-based policies, reinforcement learning, and robotic control, and we draw its boundaries deliberately. We consider diffusion models used as a policy—mapping states or observations to actions or action sequences—rather than as a planner [28], world model [29], or data synthesizer [30]; the complementary role-based taxonomy is treated by Zhu et al. [9]. We stay within robotics (manipulation, locomotion, navigation, and dexterous control) rather than RL fine-tuning of text-to-image diffusion models [31] or language models, though techniques cross over. We also concentrate on model-free RL; model-based approaches that learn a diffusion world model for synthetic rollouts [29,32] appear only for contrast. The scope of the review is therefore broader than that of the empirical study: the review retains state-, image-, point-cloud-, and language-conditioned robotic policies, whereas Section 6 is restricted to proprioceptive state inputs in two MuJoCo locomotion environments. The experiments do not include a visual encoder, camera observations, contact-rich object manipulation, or a real robot. They provide an interpretable reproduction and a set of mechanism-level diagnostics for the tested state-based configurations, not empirical validation of the full visuomotor literature summarized by the survey.

1.3. Review Methodology

We treat the review as a structured scoping review: its purpose is an auditable and updateable synthesis of reward-driven policy improvement, not a claim of database-exhaustive coverage of every use of diffusion in robotics. The final search date was 25 July 2026. Method eligibility used a 1 January 2020–25 July 2026 window; older works were retained only when needed for definitions or algorithmic background. Four source routes were logged before deduplication: the 98 records cited in the submitted manuscript; all 18 records in the reviewer literature package; a 50-record high-precision arXiv API search for diffusion-policy RL; and a 12-record arXiv API search targeted at consistency and flow-matching policies. These routes produced 178 raw records. We first matched DOI or arXiv identifiers, then normalized title, first author, and year for records without a shared identifier. Sixteen duplicates (four between the manuscript and reviewer pools and 12 between the arXiv searches and prior pools) were merged, leaving 162 unique candidates.

The exact arXiv API diffusion-policy query (A1) was:

(ti:diffusion AND ti:policy)

AND (abs:robot OR abs:manipulation OR abs:locomotion OR abs:"continuous control")

AND all:"reinforcement learning".

The consistency/flow query (A2) was:

(ti:consistency OR ti:"flow matching") AND ti:policy

AND (abs:robot OR abs:manipulation OR abs:locomotion)

AND (all:"reinforcement learning" OR all:"policy gradient" OR all:finetuning OR all:"fine-tuning").

The arXiv API supplied the discovery counts for A1–A2. Crossref/DOI records, PMLR, NeurIPS proceedings, IEEE, RSS, OpenReview, and arXiv primary pages were then used for title, year, venue, DOI, and publication-status verification, not as uncounted discovery pools. When both a preprint and a peer-reviewed record existed, the peer-reviewed metadata took precedence while the arXiv identifier was retained only when useful for version tracing. The route names, exact strings, time window, raw counts, duplicate count, and unique contribution are stated above, summarized in Figure 2, and provided in machine-readable form in Supplementary File S1. Because the discovery search is intentionally title-focused and does not use subscription databases such as Scopus or Web of Science, the review should be read as a high-precision structured scoping map rather than a systematic estimate of the entire literature.

Figure 2. Structured-review screening flow, updated on 25 July 2026. Four logged source routes produce 178 raw records: 98 manuscript citations, 18 reviewer-supplied records, 50 arXiv A1 records, and 12 arXiv A2 records. Identifier/title matching removes 16 duplicates, yielding 162 unique candidates. Title/abstract screening excludes 59 records; 103 enter full-text assessment, 19 are excluded, and 84 remain in the evidence map. Only Direct-RL studies support field-level claims about reward-driven optimization of robotic generative-policy parameters, and cross-paper numerical comparisons still require matched tasks and protocols.

Screening proceeded in two stages. Title/abstract screening excluded 59 records that were foundational theory, generic RL, surveys, benchmark descriptions, imitation-only architectures without a required boundary role, or false matches in which “consistency” did not denote a generative consistency policy. The remaining 103 records entered full-text assessment. Nineteen were excluded because diffusion served as a planner, world model, environment/data generator, reward model, or sampler for a non-diffusion policy, or because the evaluation domain lay outside robotic/continuous control. The final evidence map contains 84 studies. Figure 2 reports the complete flow, and Appendix A reports each included study’s classification, permitted evidentiary role, domain, modality, reward source, and principal limitation. Here, exclusion denotes exclusion from the structured evidence map, not necessarily removal from the manuscript’s background bibliography. Supplementary File S2 provides the record-level screening decision and reason for every unique candidate, while Supplementary File S3 provides the complete coding of all 84 retained studies.

Coding followed the operational definitions in Table 1 and the five-axis codebook described in Section 3.1.Before finalization, consistency checks verified unique bibliographic keys, completion of the required evidence fields, exact agreement between the 84 included keys in the screening log and evidence map, and the reported class totals. These checks make the decisions externally auditable but are not presented as independent duplicate screening; no inter-rater reliability statistic is claimed.

Table 1. Hierarchical eligibility labels and evidence-use rules. Rules are evaluated from top to bottom: D1 takes precedence when a reward/value/preference feedback directly updates the deployed generative-policy parameters, even when the generator is a flow or consistency model; G1 is used only when a changed generator has no such reward-driven update.

Label/Rule	Operational Definition	Typical Examples	Permitted Evidential Use
Direct-RL/ D1	A task reward, learned value, or preference objective directly updates the parameters or internal representation of the deployed diffusion, flow, consistency, or hybrid generative action policy, followed by robot/continuous-control evaluation.	DPPO, Diffusion-QL, FDPP, EDA, BDPO, DACER, ConRFT, ReinFlow	Supports direct method synthesis. Numerical aggregation is allowed only for matched tasks, metrics, and protocols.
Adj-Robot/ A1–A2	The robot study improves an auxiliary latent policy, residual, adaptor, dataset, reward model, or sampler while the base generator is frozen or supervised indirectly (A1), or it is an imitation/ pretraining comparator retained to delimit architecture or modality (A2).	DSRL, PA-RL, residual RL, AlignDiff, DISCO, TinyVLA	Supports boundary-setting and design transfer; excluded from Direct-RL counts and direct fine-tuning claims.
Alt-Gen/G1	A robotic action generator changes standard multi-step diffusion to a distilled, consistency, flow, streaming, or other inference family, but the paper does not apply reward/value/ preference optimization to that generator.	Consistency Policy, OneDP, ManiCM, flow-matching imitation policies	Supports generator and acceleration comparisons only. If reward directly updates the new generator, D1 takes precedence.
Non-Robot/ N1	Reward or preference feedback updates a diffusion or hybrid generator, but the reported evaluation does not include an eligible robotic or standard continuous-control task.	DDPO, SDPO, CHDP	Cross-domain or boundary precedent only; never direct robotic validation.

1.4. Eligibility Classes and Evidence Use

To prevent category errors, every retained study receives one of four eligibility labels (Table 1). The label answers a different question from the taxonomy of Section 3: eligibility states what kind of evidence a paper can provide, whereas the taxonomy states how its policy or generator is constructed and improved. A paper may therefore be technically informative without being direct evidence for robotic RL fine-tuning.

The retained counts are 41 Direct-RL, 32 Adj-Robot, eight Alt-Gen, and three Non-Robot studies. The hierarchy prevents generator and update mechanism from being conflated. Thus ConRFT [33], FPO [23], ReinFlow [34], and FlowDPG [35] remain Direct-RL because reward directly updates their consistency/flow action generators in robotic or eligible continuous-control evaluations. By contrast, DSRL [36] and LP-DS [37] train latent-noise steering while freezing the base generator; PA-RL [38] improves actions through a critic and subsequently retrains a policy with a policy-agnostic supervised objective; DISCO [39] and TinyVLA [40] provide constrained-sampling and architecture evidence without RL policy updates. Image-generation alignment such as DDPO [31] and SDPO [41] remains Non-Robot. CHDP [42] is also coded Non-Robot boundary evidence: it directly optimizes hybrid diffusion policies, but its reported evaluation consists of generic parameterized-action benchmarks rather than a robot or standard continuous-control experiment. Field-level numerical statements about success rates, sample requirements, or real-robot reliability use Direct-RL evidence only unless a broader denominator is named explicitly. Appendix A lists every retained paper with its label, five-axis coordinates, classification rationale, learning setting, domain, observation modality, reward source, sample requirements, real-robot evidence, platform, publication status, strengths, and limitations.

1.5. Comparison with Prior Surveys

Several recent surveys have addressed related but distinct slices of the literature. Table 2 summarizes the differences.

Table 2. Comparison with prior surveys on diffusion models in decision-making and robotics. Columns abbreviate whether the work treats DP + RL systematically, covers real-world deployment, acceleration, alignment, the VLA outlook, and whether it pairs the review with an own empirical study. Here, ✓ denotes coverage, × denotes no coverage, and “Partial” denotes limited coverage.

Survey	Year	Primary Scope	DP + RL	Real	Accel.	Align.	VLA	Own Exp.
Zhu et al. [9]	2024	DM roles in RL	Partial	×	×	×	×	×
Song et al. [43]	2025	DP for manipulation	As one strategy	✓	Partial	×	Partial	×
Ubukata et al. [44]	2024	DM for planning (SLR)	Partial	×	Partial	×	×	×
Urain et al. [45]	2024	Generative robot learning	Partial	✓	Partial	×	Partial	×
Wolf et al. [46]	2025	DM for manipulation	Partial	✓	✓	Partial	Partial	×
Choi et al. [8]	2026	Online DPRL only	✓	× (sim only)	Partial	×	×	✓
Ours	2026	Reward-driven policy improvement	✓ structured	✓	✓	✓	✓	✓ multi-axis

Our work departs from these in three ways. Where Zhu et al. [9] organize the literature by the diffusion model’s role (planner/policy/synthesizer), and Urain et al. [45] and Wolf et al. [46] cover the broader generative-policy and manipulation landscapes, we restrict the core evidence claim to reward-driven improvement of deployed robot policies. Where Choi et al. [8] taxonomize and benchmark online model-free DPRL, we additionally cover offline-to-online fine-tuning (e.g., DPPO [6]), preference-based alignment (e.g., FDPP [7]), latent steering (DSRL [36]), and alternative consistency/flow generators [33,47]. Finally, and unlike a pure survey, we pair the structured review with a controlled state-based locomotion reproduction and diagnostic controls. Those experiments inform the cross-cutting discussion but are not presented as visual-manipulation validation.

1.6. Contributions

The contributions are threefold. Survey: (1) an auditable structured scoping review that converts 178 raw records into 162 unique candidates and an 84-study evidence map under hierarchical evidence-use rules; (2) a five-axis technical taxonomy over initialization/data, interaction, optimized object, credit assignment, and inference generator; and (3) synthesis of likelihood/entropy, reward alignment, acceleration, and exploration stability without treating adjacent work as direct robotic RL evidence.

Methodology: (4) documented search routes, screening decisions, evidence coding, benchmarks, and metrics, together with desiderata D1–D3 for reproducible comparison.

Empirical study: (5) a fixed-final, five-fine-tuning-seed DPPO evaluation on state-based Hopper, pairing each policy with its run-recorded BC checkpoint for 20 common episodes and distinguishing fine-tuning seeds from the two reused BC files; (6) a 50-episode action-divergence analysis plus a controlled but inconclusive intervention; and (7) bounded checks using Gaussian BC-to-PPO, all four T4 credit-assignment routes including a five-seed BC-initialized DACER adaptation, a second locomotion task, and a five-checkpoint pure-NFE sweep. Interaction units, statistical units, missing provenance, and non-saturated reliability are reported explicitly. The experiments diagnose tested state-based configurations and do not establish visuomotor transfer or a universal algorithm ranking.

1.7. Organization

Section 2, Section 3, Section 4 and Section 5 cover foundations, taxonomy, cross-cutting themes, and evaluation; Section 6 reports the empirical study, Section 7 discusses open challenges, and Appendix A provides the evidence map.

2. Preliminaries

This section fixes notation and background. We summarize reinforcement learning (Section 2.1), denoising diffusion and flow-matching models (Section 2.2), and the diffusion policy with its variants (Section 2.3), then state the three challenges of combining diffusion policies with RL (Section 2.4) that motivate the methods of Section 3 and the experiment of Section 6.

2.1. Reinforcement Learning Fundamentals

We model agent–environment interaction as a Markov decision process (MDP) $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\gamma )$ with state and action spaces $\mathcal{S},\mathcal{A}$, reward $\mathcal{R}:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$, transition kernel $\mathcal{P}:\mathcal{S}\times \mathcal{A}\to \Delta \left(\mathcal{S}\right)$, and discount $\gamma \in [0,1)$ [48]. The aim is a policy $\pi (\mathit{a}\mid \mathit{s})$ maximizing the expected return $J\left(\pi \right)={\mathbb{E}}_{\tau \sim \pi }\left[{\sum }_{t=0}^{\infty }{\gamma }^t{r}_t\right]$.

2.1.1. Value Optimization

The action-value function $Q^{\pi }(\mathit{s},\mathit{a})$ satisfies the Bellman equation

$$Q^{\pi }(\mathit{s},\mathit{a})={\mathbb{E}}_{{\mathit{s}}^{\prime },r}\left[r+\gamma {\mathbb{E}}_{{\mathit{a}}^{\prime }\sim \pi }\left[Q^{\pi }({\mathit{s}}^{\prime },{\mathit{a}}^{\prime })\right]\right].$$

(1)

Deep Q-learning [49] approximates the action-value relation in Equation (1) with neural networks; DDPG [50] and TD3 [51] carry the idea to continuous control through deterministic policy gradients and double critics.

2.1.2. Policy Gradient Methods

For continuous actions, the policy gradient theorem [52] gives

$${\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[{\displaystyle \sum _{t=0}^{\infty }}{\nabla }_{\theta }log{\pi }_{\theta }({\mathit{a}}_t\mid {\mathit{s}}_t)·Q^{{\pi }_{\theta }}({\mathit{s}}_t,{\mathit{a}}_t)\right].$$

(2)

PPO [14] stabilizes (2) with a clipped surrogate built on the probability ratio $r_{\theta }={\pi }_{\theta }(\mathit{a}\mid \mathit{s})/{\pi }_{{\theta }_{\mathrm{old}}}(\mathit{a}\mid \mathit{s})$. Both forms need the policy log-probability—the quantity that is unavailable for a diffusion policy (Section 2.4). PPO is also the optimizer inside DPPO [6], the method at the center of our experiment.

2.1.3. Maximum-Entropy and Distributional RL

Maximum-entropy RL [15,53] adds a policy-entropy bonus to encourage exploration:

$$J_{\mathrm{MaxEnt}}\left(\pi \right)={\mathbb{E}}_{\pi }\left[\sum _t{r}_t+\beta \mathcal{H}\left(\pi (·\mid {\mathit{s}}_t)\right)\right].$$

(3)

In Equation (3), $\beta$ controls the entropy bonus. Distributional RL [54,55] instead models the return distribution $Z(\mathit{s},\mathit{a})$ with $Q(\mathit{s},\mathit{a})=\mathbb{E}\left[Z\right(\mathit{s},\mathit{a}\left)\right]$, improving value estimation when returns are multimodal.

2.1.4. Online, Offline, and Offline-to-Online RL

Online RL draws fresh samples from $\mathcal{P}$ during training, whereas offline RL is confined to a fixed dataset $\mathcal{D}=\left\{(\mathit{s},\mathit{a},r,{\mathit{s}}^{\prime })\right\}$ [56]. Datasets for Deep Data-Driven Reinforcement Learning (D4RL) provides standardized datasets and normalized scores for this setting [57]. Offline-to-online RL pre-trains from $\mathcal{D}$—typically by behavior cloning or offline RL such as IQL [58] or CQL [59]—and then fine-tunes online. This hybrid has become a common practical pattern for fine-tuning diffusion policies [6,38] and is the setting our experiment instantiates in simulation.

2.2. Diffusion and Flow-Matching Models

2.2.1. Denoising Diffusion Probabilistic Models (DDPM)

Diffusion models [4,60] learn to invert a Markov noising process that turns data ${\mathit{x}}^0$ into Gaussian noise ${\mathit{x}}^K\sim \mathcal{N}(\mathbf{0},\mathit{I})$:

$$q({\mathit{x}}^k\mid {\mathit{x}}^{k-1})=\mathcal{N}({\mathit{x}}^k;\sqrt{1-{\beta }_k}{\mathit{x}}^{k-1},{\beta }_k\mathit{I}),$$

(4)

Equation (4) uses a fixed schedule ${\left\{{\beta }_k\right\}}_{k=1}^K$. A network ${ϵ}_{\theta }({\mathit{x}}^k,k)$ predicts the added noise via

$${\mathcal{L}}_{\mathrm{DDPM}}\left(\theta \right)={\mathbb{E}}_{{\mathit{x}}^0,k,ϵ}\left[\parallel ϵ-{ϵ}_{\theta }(\sqrt{{\overline{\alpha }}_k}{\mathit{x}}^0+\sqrt{1-{\overline{\alpha }}_k}ϵ,k){\parallel }^2\right],$$

(5)

where Equation (5) uses ${\overline{\alpha }}_k={\prod }_{i=1}^k(1-{\beta }_i)$. Sampling applies the reverse step

$${\mathit{x}}^{k-1}={\displaystyle \frac{1}{\sqrt{1-{\beta }_k}}}\left({\mathit{x}}^k-{\displaystyle \frac{{\beta }_k}{\sqrt{1-{\overline{\alpha }}_k}}}{ϵ}_{\theta }({\mathit{x}}^k,k)\right)+{\sigma }_k\mathit{z}$$

(6)

by iterating Equation (6) for K steps.

2.2.2. Score-Based Formulation and DDIM

Equivalently, the reverse process can be written via the score ${\nabla }_{\mathit{x}}log{p}_k\left(\mathit{x}\right)\approx -{ϵ}_{\theta }(\mathit{x},k)/\sqrt{1-{\overline{\alpha }}_k}$, which yields a stochastic differential equation (SDE) view [5]. DDIM [61] defines a deterministic, non-Markovian reverse process that permits arbitrary noise-level strides, enabling few-step inference at some cost in sample diversity. We denote by NFE (number of function evaluations) the number of denoiser or vector-field evaluations required to generate one action chunk at a receding-horizon replan—equivalently, the number of denoising steps actually executed for that chunk; for a full DDPM sampler NFE = K, while DDIM and consistency models reduce it below K. NFE is an algorithmic cost proxy, so the empirical study additionally reports wall-clock latency per chunk and the corresponding compute-limited replanning ceiling.

2.2.3. Consistency Models

Consistency models [62] learn $f_{\theta }({\mathit{x}}^k,k)$ mapping any point on the probability-flow ODE trajectory directly to its endpoint, supporting one- or few-step generation. Consistency Policy [25] and OneDP [26] apply this to distill a multi-step diffusion policy into a near-real-time action generator.

2.2.4. Flow Matching

Flow matching [63] swaps the noise-prediction objective for velocity prediction along a learned vector field ${\mathit{v}}_{\theta }$. For the linear conditional path ${\mathit{x}}_t=(1-t)ϵ+t{\mathit{x}}^0$, with $t\sim \mathcal{U}[0,1]$ and $ϵ\sim \mathcal{N}(\mathbf{0},\mathit{I})$, the target velocity is ${\mathit{x}}^0-ϵ$:

$${\mathcal{L}}_{\mathrm{CFM}}\left(\theta \right)={\mathbb{E}}_{t,{\mathit{x}}^0,ϵ}\left[\parallel {\mathit{v}}_{\theta }({\mathit{x}}_t,t)-({\mathit{x}}^0-ϵ){\parallel }^2\right].$$

(7)

The conditional flow-matching objective is given in Equation (7).

It is the action-generation backbone of the VLA model ${\pi }_0$ [64]; the generator choice provides a continuous-time action trajectory and supports low-NFE numerical integration, although realized speed and stability depend on the trained model and solver.

2.3. Diffusion Policies for Robotic Control

Diffusion Policy (DP) [1] treats the conditional action distribution ${\pi }_{\theta }(\mathit{a}\mid \mathit{s})$ as a denoising process. Given the latest $T_o$ observations ${\mathit{O}}_t$, the policy denoises an action chunk ${\mathit{A}}_t=\{{\mathit{a}}_t,\dots ,{\mathit{a}}_{t+T_p-1}\}$ via

$${\mathit{A}}_t^{k-1}={\displaystyle \frac{1}{\sqrt{1-{\beta }_k}}}\left({\mathit{A}}_t^k-{\displaystyle \frac{{\beta }_k}{\sqrt{1-{\overline{\alpha }}_k}}}{ϵ}_{\theta }({\mathit{A}}_t^k,{\mathit{O}}_t,k)\right)+{\sigma }_k\mathit{z},$$

(8)

Equation (8) starts from ${\mathit{A}}_t^K\sim \mathcal{N}(\mathbf{0},\mathit{I})$, and the policy executes the first $T_a\le T_p$ actions before re-planning—a form of receding-horizon control. Chi et al. [1] report that position control consistently beats velocity control under DP, that an action horizon $T_a\approx 8$ trades off temporal consistency against reactivity well, and that the formulation tolerates several control steps of latency without losing performance.

2.3.1. Architectural Variants

Two denoiser backbones are common: a 1D temporal CNN with FiLM conditioning [65], and a transformer with cross-attention on observation embeddings [1]. Later work isolates design choices that affect stability, such as the DiT-block policy [66] and the modulated-attention design of MTDP [67]. For low-dimensional proprioceptive inputs—as in the locomotion benchmark we use—a lightweight MLP denoiser is standard [6].

2.3.2. 3D and Equivariant Variants

3D Diffusion Policy (DP3) [68] swaps image features for a sparse point-cloud encoding and generalizes from as few as ten demonstrations; iDP3 [69] carries this to humanoid manipulation with egocentric 3D perception. EquiBot [70] and Diffusion-EDFs [71] add SIM(3)- and SE(3)-equivariance, sharply improving sample efficiency on geometrically symmetric tasks.

2.3.3. From DP to VLA

A parallel line scales diffusion-based action heads onto vision-language-action (VLA) backbones. RDT-1B [72], Octo [73], DiffusionVLA [74], and the flow-matching-based ${\pi }_0$ [64] attach a diffusion or flow-matching head to a multimodal transformer and show cross-embodiment, cross-task generalization at scale. RL fine-tuning of such large policies is already an emerging demonstrated capability; safe, sample-efficient improvement without loss of language grounding or cross-embodiment competence remains an open frontier (Section 7).

2.4. Why Combining Diffusion Policies and RL Is Challenging

The structural mismatches between diffusion-based action generation and standard RL machinery reduce to three propositions, sketched in Figure 3.

Figure 3. Three recurring design pressures when fine-tuning diffusion policies with RL. (C1) The standard marginal action density is generally unavailable in closed form, blocking direct marginal-action PPO ratios and entropy bonuses. (C2) A long denoising unroll can be memory-intensive or poorly conditioned, with configuration-dependent severity. (C3) Iterative generation adds latency, while realized control rate depends on hardware, model, batching, sampler, action chunking, and the surrounding control stack. Text inside the schematic is shorthand: its rates are representative implementation values or illustrative targets from the acceleration literature [13,25,26,64], not universal requirements or measurements from our experiments, and “Resolved by” means “mitigated by.”

(C1)

Intractability of $log{\pi }_{\theta }(\mathit{a}\mid \mathit{s})$ and $\mathcal{H}\left({\pi }_{\theta }\right)$.

The marginal density of an action from a K-step reverse process requires integrating over all intermediate latents and is generally not analytically available. Consequently, neither the marginal-action PPO importance ratio $r_{\theta }$ [14] nor the corresponding exact policy-entropy term $\mathcal{H}\left({\pi }_{\theta }\right)$ [15] is directly evaluable for the standard construction. Section 4.1 surveys the workarounds.

(C2)

Cost and conditioning of BPTT through K denoising steps.

A deterministic policy gradient ${\nabla }_{\theta }Q(\mathit{s},{\mathit{a}}_{\theta }\left(\mathit{s}\right))$ backpropagates through the entire K-step chain in (8); representative diffusion-policy implementations commonly use tens of denoising steps. Such an unroll can be memory-intensive and, depending on the architecture, schedule, and stabilization procedure, may exhibit poorly conditioned, vanishing, or exploding gradients [16,17]. Section 3.5 reviews how BPTT-based methods manage these costs.

(C3)

Inference latency vs. control rate.

Reported multi-step samplers can be too slow for the control stack in which they are deployed [25,26]; the realized rate varies with hardware, model size, batching, action horizon, and perception/actuation overhead. This variability motivates the NFE and measured-latency reporting used in Section 4.3 and in our empirical study.

These three challenges (C1)–(C3) directly shape the top-level design of every RL-for-DP method that follows. The mechanism and learning setting provide a compact navigation view, but they do not by themselves distinguish what is initialized, what is optimized, how credit reaches that object, or which generator executes actions. Section 3 therefore supplements that view with a five-axis technical matrix.

3. Taxonomy of RL Methods for Diffusion Policies

The taxonomy has two levels. First, a compact mechanism–setting view provides the entry point used by prior DPRL discussions. Its mechanism coordinate asks how reward-derived credit reaches a trainable policy component: action-gradient, Q-weighting, proximity/density-ratio construction, or BPTT (Figure 4). Its setting coordinate asks where supervision and interaction originate: online from scratch, fixed-data offline RL, offline-to-online fine-tuning, or preference-based learning (Table 3). Second, Section 3.1 formalizes five functionally orthogonal technical axes: (T1) initial policy and data source, (T2) interaction regime, (T3) optimized object, (T4) credit-assignment mechanism, and (T5) inference generator. “Orthogonal” means that the axes answer distinct design questions; it does not imply statistical independence. This second level is necessary because methods that share a broad mechanism label can optimize fundamentally different objects: DPPO [6] updates denoising-policy weights, DSRL [36] freezes the base generator and learns a latent-noise steering policy, D2PPO [75] regularizes hidden representations during DPPO, EDA [76] aligns pretrained diffusion behavior with Q-labels, and ReinFlow [34] fine-tunes a flow-matching policy through an exact-likelihood stochastic path. Eligibility labels from Section 1.4 remain separate: they determine evidential use, not technical similarity.

Figure 4. The four policy-improvement mechanisms used as the credit-assignment coordinate (T4). (a) Action-gradient methods use ${\nabla }_{\mathit{a}}Q$ and depend on critic accuracy. (b) Q-weighting methods reweight the denoising loss and depend on reward scale. (c) Proximity/density-ratio methods construct a tractable likelihood, ratio, surrogate, or trajectory divergence. (d) BPTT methods differentiate through the generator and can be costly or poorly conditioned for long unrolls. Panel equations and italic limitations describe the listed exemplars, not entire families; accordingly, panel (c)’s online-rollout label does not exclude offline alignment methods.

Table 3. Mechanism–setting navigation view. The down arrow indicates that mechanisms vary by row, and the right arrow indicates that settings vary by column. Rows show the credit-assignment family (T4), columns show the interaction regime (T2), and cells list representative methods. Underlining marks eligible diffusion-policy methods instantiated in our empirical code path; AWR is omitted because it is a generic regression baseline rather than a diffusion-policy method. ^† Adj-Robot methods are retained for design comparison but not used as direct fine-tuning evidence. The complete five-axis coding is given in Table 4 and Appendix A.

Mechanism ↓ Setting →	Online from Scratch	Offline RL	Offline-to-Online Fine-Tuning	Preference-Based
Action-Gradient	DIPO [18], DDiffPG [77]	QSM [19]	Q-VGM [78], FlowDPG [35]	—
Q-Weighting	QVPO [20], DPMD/SDAC [21]	Diffusion-QL [79], IDQL [80], EDP [81], SfBC † [82]	—	—
Proximity/Density-Ratio	GenPO [22], FPO [23]	BDPO [83], EDA [76]	DPPO [6], D2PPO [75], ReinFlow [34], PA-RL † [38]	FDPP [7], FKPD [84], AlignDiff † [24]
BPTT-based	DACER [85], DACERv2 [86], DIME [87]	CPQL [88], DSAC-D [89]	FTD [90], D3P [13]	—

3.1. Five-Axis Technical Matrix

The mechanism–setting view is useful for navigation, but it collapses distinctions that determine what is actually learned and executed. We therefore code every retained study on five technical axes: T1, initial policy and data source (from-scratch interaction, fixed offline data, behavior-cloned policy, or pretrained VLA); T2, interaction regime (offline, online on-policy, online off-policy, offline-to-online, preference-based, distillation, or test-time); T3, optimized object (base generator weights, a subset of denoising steps, score/vector field, latent steering policy, residual/adaptor, representation, reward model, sampler, or data); T4, credit assignment (action/value gradient, Q-weighting, likelihood/density-ratio construction, BPTT, preference optimization, supervised distillation, or guidance); and T5, inference generator (multi-step diffusion, few-step or adaptive diffusion, consistency, flow matching, or a hybrid/VLA-dependent generator). Coding follows two reproducible rules. First, T3 names the parameterized object that receives the primary improvement signal; frozen conditioning modules are recorded in the rationale rather than treated as optimized objects. Second, T4 records the route by which reward-derived or supervised credit reaches T3, not the name chosen by the paper. Multi-component methods may receive compound values separated by “+”, but no axis is inferred from application domain alone. Table 4 shows representative coordinates; the full 84-study matrix, including paper-specific classification rationales, domain, observation modality, reward source, sample requirements, real-robot evidence, platform, publication status, strengths, and limitations, is provided in Appendix A.

Table 4. Representative five-axis technical matrix. Class labels follow Table 1: D = Direct-RL, A = Adj-Robot, and G = Alt-Gen. The matrix deliberately separates the optimized object (T3) from the route by which it receives credit or supervision (T4).

Method/Class	T1: Initial Policy/Data	T2: Regime	T3: Optimized Object	T4: Credit Assignment	T5: Generator
DPPO/D [6]	BC demonstrations	Offline-to-online	Final denoising steps/weights	Per-denoising-step surrogate PPO	Multi-step diffusion
DSRL/A [36]	Frozen BC or VLA policy	Online adaptation	Latent-noise steering policy	RL in latent-noise space	Frozen diffusion /flow base
ADP/A [91]	Offline demonstrations	Parameter-efficient adaptation	LoRA/temporal adaptor	Supervised adaptive fine-tuning	Multi-step diffusion
Residual RL/A [92]	Frozen BC policy	Offline-to-online	Residual action head	Off-policy RL on residual actions	Base policy plus residual
D2PPO/D [75]	BC demonstrations	Offline-to-online	Weights and hidden representations	DPPO plus dispersive representation loss	Multi-step diffusion
FDPP/D [7]	Pretrained BC policy	Preference-based	Diffusion-policy weights	Learned preference reward, PPO, and KL anchor	Multi-step diffusion
Diffusion-QL/D [79]	Fixed offline data	Offline	Diffusion-policy weights	BC regression plus Q maximization	Multi-step diffusion
BDPO/D [83]	Fixed offline data	Offline	Diffusion-policy weights	Actor–critic objective with analytic trajectory behavior KL	Multi-step diffusion
EDA/D [76]	Fixed offline data	Offline	Diffusion-policy weights	Align behavior likelihood with Q-derived labels	Multi-step diffusion
QSM/D [19]	Offline reward data	Offline	Policy score field	Match score to action-value gradient	Multi-step diffusion
DACER/D [85]	Online from scratch	Online off-policy	Full diffusion-policy weights	BPTT with a GMM entropy proxy	Multi-step diffusion
FTD/D [90]	Pretrained diffusion policy	Offline-to-online	Full denoising trajectory/weights	Full-trajectory differentiation with RL objective	Multi-step diffusion
FPO/D [23]	Scratch or pretrained policy	Online on-policy	Flow-policy weights	Conditional-flow-loss surrogate PPO	Flow matching
ReinFlow/D [34]	Pretrained flow policy	Offline-to-online	Stochastic flow-policy weights	Exact-likelihood policy gradient along stochastic flow	Flow matching
FlowDPG/D [35]	Pretrained flow policy	Offline-to-online off-policy	Flow vector field	Critic-gradient distillation/deterministic policy gradient	Flow matching
ConRFT/D [33]	VLA demonstrations	Offline-to-online	Consistency-policy/VLA parameters	Unified BC, Q-learning, and online RL	Consistency VLA
D3P/D [13]	Pretrained diffusion policy	Offline-to-online	Base policy and step adaptor	Two-layer RL with task/compute reward	Adaptive diffusion
OneDP/G [26]	Pretrained diffusion teacher	Distillation	One-step student policy	KL-based diffusion distillation	One-step diffusion
${\pi }_0$/A [64]	Large VLA corpus	Pretraining/imitation	Flow-matching action head	Supervised flow matching; no RL	Flow-matching VLA

Table 5 compares the likelihood/gradient strategies and distinctive features of representative methods after the five-axis matrix.

Table 5. Representative method properties. Class: D = Direct-RL and A = Adj-Robot. Mech.: AG = action-gradient, QW = Q-weighting, PB = proximity/surrogate-likelihood, BPTT = backprop-through-time, LS = latent steering, HY = hybrid objective. Setting: On = online from scratch, Off = offline, O→O = offline-to-online, Pref = preference-based. DPPO is the subject of our empirical study.

Method	Class	Mech.	Setting	Likelihood/Gradient Strategy	Distinctive Feature
DIPO [18]	D	AG	On	Overwrite buffer actions via ${\nabla }_{\mathit{a}}Q$, then supervised diffusion regression	No explicit likelihood; can break replay-buffer MDP consistency
DDiffPG [77]	D	AG	On	Per-mode ${\nabla }_{\mathit{a}}Q$ improvement with mode-specific critics	Clusters rollouts into modes to preserve multimodality
QSM [19]	D	AG	Off	Train the score field to match ${\nabla }_{\mathit{a}}Q$	Elegant, but highly sensitive to critic accuracy
Diffusion-QL [79]	D	QW	Off	BC regression loss + a Q-maximization term	First diffusion model used as an offline-RL policy class
IDQL [80]	D	QW	Off	Weighted regression with the IQL advantage as weight	Recasts implicit Q-learning as weighted regression
EDP [81]	D	QW	Off	Approximate action sampling bypassing the full chain	Accelerates Diffusion-QL training
QVPO [20]	D	QW	On	Q-weighted objective filtering negative-advantage actions	Strong off-policy baseline; preserves multimodality
DPMD/SDAC [21]	D	QW	On	Reweighted score matching with $exp(Q/\lambda )$ weights	Mirror-descent/SAC variants; reward-scale sensitive
EDA [76]	D	PB	Off	Convert Q-function comparisons into behavior-alignment labels	Offline alignment without on-policy rollouts
BDPO [83]	D	PB	Off	Actor–critic update with analytic trajectory-level behavior KL	Principled behavior regularization for diffusion trajectories
GenPO [22]	D	PB	On	Exact likelihood via diffusion inversion, then PPO clip	Doubled-dummy-action trick; K = 5 steps (borders BPTT)
FPO [23]	D	PB	On	PPO ratio from conditional flow-matching losses	Likelihood-free surrogate for flow/diffusion policies
FlowDPG [35]	D	AG	O→O	Distill deterministic critic gradients into the flow vector field	Avoids full ODE BPTT during real-robot fine-tuning
ReinFlow [34]	D	PB	O→O	Exact likelihood along a stochastic interpolant, then policy gradient	Reward fine-tuning of pretrained flow policies
DPPO [6]	D	PB	O→O	Per-step Gaussian likelihood of the denoising MDP + PPO	Reported near-data-support latent exploration; examined indirectly here
D2PPO [75]	D	PB	O→O	DPPO objective + dispersive representation regularization	Separates hidden features during fine-tuning
DSRL [36]	A	LS	O→O	RL trains a latent-noise steering policy; base is frozen	Adapts diffusion and flow policies without base-weight updates
PA-RL [38]	A	PB	O→O	Critic-improved buffer actions + supervised re-training	Policy-agnostic; applies to autoregressive/VLA policies
FDPP [7]	D	PB	Pref	PPO on a Bradley–Terry reward model + KL regularizer	Preference alignment; KL coefficient $\beta$ is critical
AlignDiff [24]	A	PB	Pref	Attribute-strength model used for guided sampling	Zero-shot; edits sampling, not policy parameters
DACER(v2) [85,86]	D	BPTT	On	BPTT with a GMM entropy proxy (v2 adds a normalized Q-grad term)	Tractable MaxEnt exploration; v2 uses fewer steps
DIME [87]	D	BPTT	On	Variational lower bound on MaxEnt over the full trajectory	Rigorous treatment of entropy intractability
FTD [90]	D	BPTT	O→O	Differentiate the RL objective through the full denoising trajectory	Direct full-trajectory fine-tuning of a pretrained policy
CPQL [88]	D	BPTT	Off	BPTT through a one-step consistency function	Avoids multi-step backprop cost
DSAC-D [89]	D	BPTT	Off	Diffusion value network modeling the full return distribution	Distributional critic for robust policy gradients
ConRFT [33]	D	HY	O→O	Unified BC, Q-learning, and online RL for consistency VLA	Real-robot reinforced VLA adaptation
D3P [13]	D	BPTT	O→O	BPTT with adaptive per-action denoising-step allocation	Inference speed-up; motivates our criticality probe

3.2. Action-Gradient Methods

Action-gradient methods sidestep the intractable likelihood by using the action gradient ${\nabla }_{\mathit{a}}Q(\mathit{s},\mathit{a})$ as a supervision signal: even without ${\nabla }_{\theta }log{\pi }_{\theta }$, a learned critic still indicates a direction in which actions improve.

3.2.1. DIPO

DIPO [18], an early online model-free DPRL algorithm, improves a sampled action by critic ascent,

$$\widehat{\mathit{a}}=\mathit{a}+{\eta }_a{\nabla }_{\mathit{a}}{Q}_{\varphi }(\mathit{s},\mathit{a}),$$

(9)

and uses the improved action in Equation (9) as a target for the standard diffusion regression loss (5). The update needs no explicit log-likelihood and keeps training in the familiar supervised regime. Choi et al. [8] note that overwriting $\mathit{a}$ with $\widehat{\mathit{a}}$ breaks MDP consistency in the replay buffer and can bias the critic.

3.2.2. DDiffPG

DDiffPG [77] targets multimodal behavior: it clusters rollouts into M modes, learns a per-mode critic $Q_{{\varphi }_m}$, and applies the mode-specific improvement

$${\widehat{\mathit{a}}}_m={\mathit{a}}_m+{\eta }_a{\nabla }_{\mathit{a}}{Q}_{{\varphi }_m}(\mathit{s},{\mathit{a}}_m).$$

(10)

The multimodal targets from Equation (10) are folded into one diffusion batch, exploiting the model’s ability to fit multimodal data.

3.2.3. QSM

Q-Score Matching (QSM) [19] is the cleanest of the three theoretically: matching the policy score ${\mathit{s}}_{\theta }(\mathit{s},\mathit{a})$ to the scaled action gradient drives the policy to the Boltzmann optimum ${\pi }^{★}\propto exp(Q/\beta )$,

$${\mathcal{L}}_{\theta }^{\mathrm{QSM}}={\mathbb{E}}_{(\mathit{s},\mathit{a})\sim \mathcal{D}}\left[\parallel {\mathit{s}}_{\theta }(\mathit{s},\mathit{a})-\alpha {\nabla }_{\mathit{a}}{Q(\mathit{s},\mathit{a})\parallel }^2\right].$$

(11)

The objective in Equation (11) is highly sensitive to critic accuracy [8]: any bias in Q propagates straight into the score field. DACERv2 [86] reuses a normalized variant of this objective as an auxiliary loss to stabilize BPTT.

3.3. Q-Weighting Methods

Q-weighting methods keep the standard diffusion regression loss but modulate it per sample with a critic-derived weight $w(\mathit{s},\mathit{a})$:

$${\mathcal{L}}_{\theta }^{Q\text{-}\mathrm{w}}={\mathbb{E}}_{\mathit{s},\mathit{a}}\left[w(\mathit{s},\mathit{a})·\parallel ϵ-{ϵ}_{\theta }(\sqrt{{\overline{\alpha }}_k}\mathit{a}+\sqrt{1-{\overline{\alpha }}_k}ϵ,\mathit{s},k){\parallel }^2\right].$$

(12)

Equation (12) retains the favorable optimization landscape of the diffusion loss while injecting reward-aware preference.

3.3.1. Diffusion-QL and IDQL (Offline)

Diffusion-QL [79] was among the earliest works to use a diffusion model as the offline-RL policy class, adding a Q-maximization term to the regression loss, $\mathcal{L}={\mathcal{L}}_{\mathrm{BC}}-\eta {\mathbb{E}}_{\mathit{a}\sim {\pi }_{\theta }}Q(\mathit{s},\mathit{a})$, with the BC term acting as a regularizer against out-of-distribution actions. IDQL [80] recasts implicit Q-learning [58] as weighted regression using the IQL advantage as the weight, and EDP [81] speeds up Diffusion-QL with an approximate action sampler that avoids the full chain.

3.3.2. QVPO (Online)

QVPO [20] shows that when $Q(\mathit{s},\mathit{a})\ge 0$ everywhere, the Q-weighted variational bound lower-bounds the RL objective. For the general case it uses the truncated advantage,

$$w(\mathit{s},\mathit{a})=max\left\{A(\mathit{s},\mathit{a}),0\right\},A(\mathit{s},\mathit{a})=Q(\mathit{s},\mathit{a})-V\left(\mathit{s}\right),$$

(13)

so the weight in Equation (13) filters out negative-advantage actions while keeping the model’s multimodal capacity. It is a consistently strong off-policy baseline in the benchmark of Choi et al. [8].

3.3.3. DPMD and SDAC

Ma et al. [21] propose reweighted score-matching objectives,

$${\mathcal{L}}_{\theta }^{\mathrm{RSM}}=\mathbb{E}\left[w(\mathit{s},\mathit{a})\parallel {\mathit{s}}_{\theta }({\mathit{a}}^k,\mathit{s},k)-{\nabla }_{{\mathit{a}}^k}log{p}_k({\mathit{a}}^k\mid \mathit{s}){\parallel }^2\right],$$

(14)

and instantiate Equation (14) in two forms—DPMD (mirror-descent) and SDAC (soft actor-critic)—both with exponential reweighting $w(\mathit{s},\mathit{a})=exp\left(Q\right(\mathit{s},\mathit{a})/\lambda )$. As Choi et al. [8] highlight, Q-weighting is reward-scale sensitive: because the weight couples to $\left|Q\right|$, ill-scaled rewards can destabilize training.

3.4. Proximity-Based and Surrogate-Likelihood Methods

Proximity-based methods attack (C1) head-on by constructing a tractable surrogate for the diffusion log-likelihood, which then admits trust-region or clipped-ratio objectives such as PPO.

3.4.1. GenPO

GenPO [22] computes ${\pi }_{\theta }(\mathit{a}\mid \mathit{s})$ exactly via diffusion inversion using a doubled dummy action that makes the forward process invertible. By the change-of-variables formula [93], the log-likelihood follows from the Jacobian determinant of the forward map, after which the PPO clip applies directly. Because inversion still needs gradients through the chain, GenPO keeps $K=5$ steps to stay tractable, which also places it on the boundary of the BPTT family (Section 3.5).

3.4.2. FPO

Flow Policy Optimization (FPO) [23] replaces the intractable likelihood ratio with a ratio of conditional flow-matching losses,

$$r_{\theta }^{\mathrm{FPO}}=exp\left({\mathcal{L}}_{\mathrm{old}}^{\mathrm{CFM}}(\mathit{s},\mathit{a})-{\mathcal{L}}_{\theta }^{\mathrm{CFM}}(\mathit{s},\mathit{a})\right).$$

(15)

Treating the loss ratio in Equation (15) as a proxy likelihood ratio lets the method drop into PPO while keeping the expressiveness of flow policies.

3.4.3. DPPO

Diffusion Policy Policy Optimization (DPPO) [6] takes a different route: instead of seeking a tractable marginal likelihood, it casts the denoising-plus-environment interaction as a two-layer MDP (Figure 5). The outer (environment) MDP unrolls policy actions; the inner (denoising) MDP unrolls one reverse-diffusion step at a time, each producing a Gaussian transition kernel

$${\pi }_{\theta }({\mathit{a}}^{k-1}\mid {\mathit{a}}^k,{\mathit{O}}_t)=\mathcal{N}\left({\mathit{a}}^{k-1};{\mu }_{\theta }({\mathit{a}}^k,{\mathit{O}}_t,k),{\sigma }_k^2\mathit{I}\right),$$

(16)

whose per-step log-likelihood in Equation (16) is closed-form. Standard policy gradient—PPO in practice—then optimizes the two-layer MDP, with the task reward propagating cleanly through the chain.

Figure 5. DPPO’s two-layer MDP. The outer MDP is the environment loop; the inner MDP unrolls ${\mathit{a}}^K\to \cdots \to {\mathit{a}}^0={\mathit{a}}_t$. Each Gaussian transition (16) has a closed-form log-likelihood, so PPO can operate at every denoising step without the generally unavailable marginal-action likelihood. The bottom “stays on-manifold” statement records DPPO’s reported motivation for intermediate-latent perturbations, not a guarantee of exact support membership, dynamic feasibility, or safety; our diagnostics do not establish that property (Section 4.4).

DPPO contributes three reported findings that recur below and motivate the experiment of Section 6. First, its experiments show that PPO can effectively fine-tune a pre-trained diffusion policy and outperform the evaluated Diffusion-QL-style offline-to-online baselines on most tested tasks [6]. Second, its analysis argues that perturbing intermediate denoising latents encourages exploratory behavior nearer to the training-data manifold than final action-space noise. Third, the reported fine-tuned policies transfer zero-shot to a Franka arm on a long-horizon, multi-stage task. We treat the second point as a method-specific interpretation rather than a property proved by our locomotion diagnostics.

3.4.4. Policy-Agnostic RL (PA-RL)

PA-RL [38] is deliberately model-agnostic: rather than design per-policy gradient estimators, it improves replay-buffer actions through the critic—much as DIPO does [18]—and re-trains the policy by supervised learning. The decoupling lets one recipe fine-tune diffusion, autoregressive, or VLA policies, and it has been applied to large pre-trained VLA backbones [38].

3.4.5. FDPP: Preference-Based Fine-Tuning

Unlike the methods above, which optimize an environment reward, FDPP [7] aligns a diffusion policy with human preferences in three stages: the pre-trained DP generates rollouts; a human ranks pairs of rollout segments; and a Bradley–Terry [94] reward model ${\widehat{r}}_{\psi }$ is fit to the labels by negative log-likelihood,

$$\begin{array}{c}\hfill \mathcal{L}\left(\psi \right)=-{\mathbb{E}}_{({\sigma }_0,{\sigma }_1,y)}[ylog{\widehat{p}}_{\psi }[{\sigma }_1\succ {\sigma }_0]\hfill \\ \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+(1-y)log{\widehat{p}}_{\psi }[{\sigma }_0\succ {\sigma }_1]],\end{array}$$

(17)

The loss in Equation (17) uses ${\widehat{p}}_{\psi }[{\sigma }_1\succ {\sigma }_0]=\frac{exp{\sum }_t{\widehat{r}}_{\psi }\left({\mathit{s}}_t^1\right)}{exp{\sum }_t{\widehat{r}}_{\psi }\left({\mathit{s}}_t^0\right)+exp{\sum }_t{\widehat{r}}_{\psi }\left({\mathit{s}}_t^1\right)}$. The policy is then fine-tuned by PPO under ${\widehat{r}}_{\psi }$ with a KL anchor to the pre-trained reference ${\pi }_{\mathrm{ref}}$:

$$J_{\mathrm{FDPP}}\left(\theta \right)={\mathbb{E}}_{{\pi }_{\theta }}\left[\sum _t{\widehat{r}}_{\psi }\left({\mathit{s}}_t\right)-\beta D_{\mathrm{KL}}({\pi }_{\theta }\parallel {\pi }_{\mathrm{ref}})\right].$$

(18)

In Equation (18), $\beta$ must be tuned carefully: too small and the policy over-fits the reward and forgets the task; too large and it never aligns with the new preference—the reward-hacking dilemma familiar from RLHF for language models [95].

3.4.6. AlignDiff

AlignDiff [24] predates FDPP on preference customization. It learns a multi-perspective attribute-strength model from human feedback and uses it as a classifier for guided sampling, achieving zero-shot behavior customization. Where AlignDiff edits the sampling process, FDPP edits the policy parameters, trading online interaction for more persistent behavioral change.

3.5. Backpropagation-Through-Time (BPTT) Methods

BPTT methods drop the surrogate loss and let gradients flow through the full reverse-diffusion chain. They differ mainly in how they contain the resulting cost and instability.

3.5.1. DACER and DACERv2

DACER [85] maximizes the expected Q-value end-to-end through BPTT,

$${\mathcal{L}}_{\theta }^{\mathrm{DACER}}=-{\mathbb{E}}_{\mathit{s}\sim \mathcal{D},\mathit{a}\sim {\pi }_{\theta }}\left[Q_{\varphi }^{\pi }(\mathit{s},\mathit{a})\right].$$

(19)

The loss in Equation (19) lacks an analytical entropy, so DACER approximates the policy by a Gaussian mixture and uses the GMM entropy as a MaxEnt regulator. DACERv2 adds a normalized Q-gradient term (in the spirit of QSM [19]) and stays competitive with far fewer denoising steps, at the price of sitting on the action-gradient boundary.

3.5.2. DIME

DIME [87] treats entropy intractability rigorously, deriving a tractable variational lower bound on the MaxEnt objective by viewing the full reverse-trajectory distribution ${\pi }_{\theta }^{0:K}({\mathit{a}}^{0:K}\mid \mathit{s})$ as the target of approximate inference [96], with the clean-step marginal taking the energy-based form ${\overline{\pi }}^0({\mathit{a}}^0\mid \mathit{s})\propto exp\left(Q(\mathit{s},{\mathit{a}}^0)\right)$. Minimizing $D_{\mathrm{KL}}({\pi }_{\theta }^{0:K}\parallel {\overline{\pi }}^{0:K})$ aligns the whole denoising trajectory with critic-preferred actions.

3.5.3. CPQL and DSAC-D

CPQL [88] sidesteps the BPTT cost by replacing the multi-step sampler with a one-step consistency function [62], backpropagating through a single evaluation. DSAC-D [89] goes further with a Diffusion Value Network that models the full return distribution $Z(\mathit{s},\mathit{a})$ [55], giving more robust critic estimates for the policy gradient.

3.5.4. D3P: Adaptive Denoising Under BPTT

D3P [13] is motivated by the observation, empirically supported on Robomimic and Franka Kitchen, that not all actions in a task are equally critical: a fixed number of denoising steps therefore wastes compute on routine actions and under-allocates capacity for crucial ones. D3P augments a base diffusion policy ${\pi }_{\theta }$ with a lightweight adaptor $K_{\omega }$ that predicts the noise-level stride per state. The denoising–environment interaction is formulated as a two-layer POMDP (echoing the DPPO formulation, but with an additional control over chain length), and the adaptor is trained with PPO under a reward

$$r_{K,\overline{t}}=\alpha {\widehat{A}}_{\Theta }({\mathit{X}}_t^0,{\mathit{o}}_t){\gamma }_s^{{\mathrm{sgn}}_t{\mathrm{stp}}_t}+\beta r_{s,\overline{t}}{\gamma }_s^{{\mathrm{stp}}_t},$$

(20)

where, in Equation (20), ${\gamma }_s\in (0,1)$ penalizes long denoising chains, ${\mathrm{stp}}_t$ counts denoising steps for the current action, and ${\widehat{A}}_{\Theta }$ is a critic-derived advantage. The base policy is concurrently fine-tuned with DPPO, and a three-stage curriculum ensures stable convergence. Empirically, D3P attains comparable success rates to the strongest DPPO baseline while delivering a $2.2\times$ average inference speed-up on simulated benchmarks and a $1.9\times$ speed-up on a Franka deployment. The action-criticality premise underlying D3P is one of the hypotheses our trajectory analysis (Section 6.4) examines directly.

3.6. Discussion: The Learning-Setting Axis

Prior surveys have largely organized the field by mechanism [8]; the learning-setting axis is no less consequential, and it forms the second dimension of our taxonomy. Four settings recur in the literature.

When the policy is learned entirely from online interaction (DIPO, QVPO, DACER, GenPO), algorithmic robustness is tested most directly, and scalability becomes a discriminating factor: as reported in [8], on-policy methods such as GenPO are competitive mainly under massive parallelization, whereas off-policy methods such as DIPO remain usable in resource-constrained regimes. The offline column (Diffusion-QL, IDQL, EDP, QSM, CPQL) instead trains on a fixed dataset, exploiting the capacity of diffusion models to represent multi-modal behavior policies—the property that first motivated their use in RL [79]. Their recurring weakness, visible once a policy is deployed on hardware, is fragility on the out-of-distribution states met at test time.

Offline-to-online fine-tuning is a common practical paradigm and is the setting we study empirically. DPPO [6], PA-RL [38], and residual schemes [92] all start from a BC-pretrained diffusion policy and refine it with online RL; the base-policy component of D3P [13] belongs here as well. The pretrained initialization can lower the cost of online exploration and provides an executable policy while updates accumulate. Preference-based fine-tuning, by contrast, aligns the policy with human judgments rather than an environment reward: FDPP [7] and AlignDiff [24] inherit the trade-off familiar from RLHF [95], in which a learned reward model can capture preferences that demonstrations cannot, at the price of reward hacking and policy drift—which is why a KL anchor (Equation (18)) is typically retained.

Two regularities cut across the cells of Table 3. First, no credit-assignment family is tied to a single interaction regime. Ratio-like optimization is most common online, but BDPO [83] and EDA [76] show that behavior-density or Q-label constructions can also support offline alignment without fresh rollouts. Conversely, BPTT spans online learning (DACER [85]), fixed-data offline RL (CPQL [88]), and offline-to-online fine-tuning (FTD [90]). Second, the generator axis cuts across the same mechanism labels: DPPO [6] applies a denoising-MDP likelihood to diffusion, FPO [23] uses a conditional-flow-loss surrogate, ReinFlow [34] constructs an exact-likelihood stochastic flow, and FlowDPG [35] propagates critic gradients through flow integration. These crossings are precisely why mechanism and setting are navigation coordinates rather than a complete taxonomy.

4. Cross-Cutting Themes

Beyond the choice of policy-improvement mechanism and learning setting, four structural problems recur across virtually every algorithm reviewed in Section 3. We examine each in turn and consolidate the design choices that the literature has explored; the empirical study in Section 6 then revisits three of them—likelihood/entropy intractability, inference acceleration, and exploration stability—from a measurement standpoint.

4.1. Handling Log-Likelihood and Entropy Intractability

The marginal density of an action drawn from a general K-step reverse diffusion process is usually not available in closed form, which obstructs standard marginal-likelihood RL objectives. The remedies surveyed in the literature trade off rigor against cost in recognizably different ways. The cheapest are surrogate proxies: DACER [85] and DSAC-D [89] fit a Gaussian mixture to action samples from ${\pi }_{\theta }$ and treat its entropy as a stand-in for $\mathcal{H}\left({\pi }_{\theta }\right)$, which is simple but biased and tends to under-estimate the entropy of strongly multimodal policies; SDAC [21] instead uses the log-probability of the additive exploration noise as a surrogate for $log{\pi }_{\theta }(\mathit{a}\mid \mathit{s})$, a heuristic that works well when the sampler is dominated by additive noise rather than multi-step structure. More principled but heavier options operate on the trajectory. DIME [87] derives a genuine lower bound on the MaxEnt objective from the joint distribution ${\pi }_{\theta }^{0:K}$, at the cost of careful trajectory-level KL approximations, while FPO [23] replaces the unavailable marginal-action importance ratio with $exp({\mathcal{L}}_{\mathrm{old}}^{\mathrm{CFM}}-{\mathcal{L}}_{\theta }^{\mathrm{CFM}})$, using the flow-matching loss as a proxy log-likelihood—a substitution exact only for small policy updates but robust in practice.

A third route avoids the marginal density entirely. DPPO [6] recasts the denoising–environment interaction as a two-layer MDP in which each denoising step is a closed-form Gaussian transition ${\pi }_{\theta }({\mathit{a}}^{k-1}\mid {\mathit{a}}^k,\mathit{O})$, so PPO applies at every step without approximating the marginal action likelihood; it is a widely used reference for offline-to-online fine-tuning and the route we exercise experimentally. GenPO [22] obtains the exact log-likelihood through the Jacobian determinant of an invertible forward map built from a doubled-dummy-action construction, yielding true PPO-equivalent updates but at a computational cost that confines it to small K (≤5).

4.2. Reward Modeling and Alignment

The choice of reward signal matters as much as the choice of update rule. Most methods (DPPO, DIPO, QVPO, DACER) assume a well-shaped environment reward, and as Choi et al. [8] observe, Q-weighting methods such as QVPO are then especially sensitive to its scale, since the weight $w(\mathit{s},\mathit{a})$ couples directly to Q. When no such reward is available, FDPP [7] and AlignDiff [24] learn one ${\widehat{r}}_{\psi }$ from human preferences in the tradition of deep PbRL [97,98]; the lesson FDPP draws is that a KL anchor to the pre-trained policy is indispensable, as without it the policy collapses into reward-maximizing but task-failing behavior [7]. A further option, raised in [8] and partly explored by [99], learns rewards from demonstrations via IRL [100,101] so that they generalize beyond the demonstrated pairs and extract signal even from suboptimal data; Diffusion Reward [99] is an early instance, building dense rewards from pre-trained video diffusion models.

Across these settings, reward optimization against a high-capacity diffusion policy is prone to reward hacking, and both FDPP [7] and large-policy frameworks such as DDPO [31] in image generation respond with KL anchoring to a frozen reference. We expect this anchor to remain a standard regularizer for future RL fine-tuning of large VLA models (Section 7).

4.3. Inference Acceleration

Inference latency directly bounds real-time deployability, and the literature attacks it from several angles. The least invasive operate at the solver level: replacing the stochastic reverse process with a deterministic ODE/SDE solver [61,102] cuts the NFE without retraining, and Chi et al. [1] report that DDIM with 10 inference steps reaches $0.1$ s latency on an NVIDIA 3080, enough for 10 Hz control. Distillation goes further by retraining for few-step sampling. Consistency Policy [25] distills a pre-trained DP into a consistency model [62] through consistency-trajectory training for 1–3-step inference; OneDP [26] adds a KL-based distillation that costs only 2–$10\%$ extra training and raises the action rate from $1.5$ Hz to 62 Hz on a Franka; and ManiCM [27] carries the idea to 3D policies. A related line builds the generator on flow matching [63], whose optimal-transport-style ODE typically needs fewer integration steps than diffusion for comparable quality—${\pi }_0$ [64] uses it to generate actions at 50 Hz.

Other strategies exploit temporal structure rather than the sampler. Streaming Diffusion Policy [103] reuses the unexecuted tail of the previous action chunk as a partially denoised initialization for the next, amortizing denoising over time, and Falcon [104] adds a thresholding rule that picks the best partial action prior from a latent buffer. D3P [13] takes a different view again: instead of lowering K uniformly, it spends denoising steps where they matter, allocating more to high-criticality actions and fewer to routine ones. In its benchmarks this produces a reported performance–efficiency trade-off that dominates the evaluated uniform-K baselines and provides a state-conditioned rule for allocating denoising computation.

4.4. Exploration and Training Stability

Recent fine-tuning work suggests that the point at which exploration noise enters the denoising chain can materially affect the resulting behavior. In the settings evaluated by DPPO [6], intermediate-latent perturbations produced behavior that the authors reported as closer to the pre-training-data support than behavior obtained by adding Gaussian noise directly to the final action. We use “near/on-manifold” only as shorthand for this method-specific empirical interpretation; it is not a general guarantee of support membership, dynamic feasibility, or safety. This reported mechanism motivates the action-divergence analysis in Section 6.4, but our state-based divergence diagnostic does not directly test manifold membership. DPPO additionally reports that the exploration-noise schedule should be re-tuned during fine-tuning because the variance appropriate for exploration need not coincide with that used for behavior-cloning denoising.

Stability concerns otherwise mirror conventional deep RL. Value overestimation [105] is countered with double-Q critics in DIPO [18], QVPO [20], CPQL [88], and DACERv2 [86], with DSAC-D [89] and DACERv2 additionally modeling the full return distribution to make the policy update more robust. To damp the variance of the stochastic diffusion sampler, several methods (Diffusion-QL [79], QVPO [20], SDAC and DPMD [21]) draw multiple action candidates per state and select or aggregate them through the critic.

4.5. Action Representation Considerations

Three choices about how actions are represented interact with the RL machinery and deserve note. On the control interface, Chi et al. [1] find that position control outperforms velocity control in their diffusion-policy evaluations, which they attribute to the more pronounced action multimodality of position space—which DP can exploit—and to less compounding error. Whether that ordering persists after RL fine-tuning is task- and implementation-dependent; long action horizons can additionally expose full-chain methods to less favorable gradient conditioning. On temporal granularity, the receding-horizon chunk ${\mathit{A}}_t=\{{\mathit{a}}_t,\dots ,{\mathit{a}}_{t+T_p-1}\}$ improves temporal consistency but raises the question of whether reward should be credited chunk-wise or step-wise; Q-Chunking [106] addresses it by running RL directly over action chunks in an offline-to-online setting, a formulation that meshes naturally with the chunked action space of diffusion policies even though it does not itself instantiate a diffusion policy (its integration with diffusion-specific machinery such as DPPO is flagged as open in Section 7). On the observation side, 3D Diffusion Policy (DP3) [68] and its equivariant variants [70,71] markedly improve sample efficiency by using point-cloud geometry; RL fine-tuning of such 3D policies is still little explored, though it follows naturally from the DPPO framework and is flagged as an open direction in Section 7.

5. Benchmarks, Metrics, and Evaluation Methodology

A recurring obstacle to cumulative progress in DP + RL is the absence of a shared evaluation methodology. This section consolidates what the literature actually uses—simulation platforms (Section 5.1), real-world deployments (Section 5.2), and metrics (Section 5.3)—and then makes the methodological gap explicit and distills a small set of desiderata for a reproducible reference protocol (Section 5.4). The empirical study of Section 6 is a deliberately transparent instantiation of that protocol.

5.1. Simulation Platforms

Empirical progress in DP + RL has been driven by a small set of simulation platforms; Table 6 summarizes the environments that recur in the literature and situates them by physics backend, degree of parallelization, and task type. These differences are a major source of incomparability across papers: a method tuned for massively parallel on-policy training on Isaac Lab [10] need not be competitive in the data-constrained, manipulation-oriented regime of Robomimic [2]. We deliberately select a low-dimensional, well-characterized continuous-control task for our own study (Section 6) so that the policy’s behavior can be inspected action-by-action without the confound of a high-dimensional perceptual front-end.

Table 6. Major simulation platforms used to evaluate DP + RL methods. The D4RL MuJoCo locomotion row identifies the setting that our empirical study (Section 6) adopts as a transparent reference benchmark.

Platform	Physics	Strength	Typical Use	Representative Methods
Isaac Lab [10]	PhysX, GPU	Massive parallelization	Online DPRL benchmark	GenPO, DPPO, Choi et al. [8]
Robomimic [2]	MuJoCo	Standard manipulation suite	Offline-to-online	DP, DPPO, D3P, FDPP
Franka Kitchen [12]	MuJoCo	Long-horizon multi-stage	Offline-to-online	DP, DPPO, D3P, Decision Diffuser [107]
Meta-World [108]	MuJoCo	Multi-task suite	Multi-task RL	MTDIFF, multi-task DPRL
ManiSkill3 [11]	PhysX, GPU	Large-scale, diverse tasks	Online/sim-to-real	EquiBot, modern VLA
D4RL Gym (MuJoCo) [57]	MuJoCo	Standard locomotion	Offline-to-online (ours)	Diffusion-QL, IDQL, EDP, DPPO
RLBench [109]	V-REP	Robotic multi-task	Vision-based manipulation	3D Diffuser Actor, GSC

5.2. Real-World Deployment Evidence

Real-world deployment is the ultimate test of a method’s practicality, and four case studies anchor the simulation benchmarks of Section 5.1. Chi et al. [1] deploy DP across UR5 (Push-T, sauce pouring and spreading), Franka (mug flipping), and bimanual platforms (egg-beater, mat unrolling, shirt folding), reporting $95\%$ success on Push-T against $20\%$ for an LSTM-GMM baseline, the value of haptic teleoperation for collecting contact-rich demonstrations such as the egg-beater task, and the need for mid-level controllers to avoid collisions in bimanual setups. Ren et al. [6] report zero-shot sim-to-real transfer of a DPPO-fine-tuned policy on a long-horizon, multi-stage manipulation task and attribute the evaluated policy’s increased robustness partly to structured latent-space exploration and stable policy-gradient fine-tuning. We treat this as evidence from that deployment rather than independent proof of a universal on-manifold or safety property. Chen et al. [7] run FDPP on Push-T, Stack-Dist, and Stack-Align with task-specific human preferences (for instance, aligning rather than stacking, or avoiding designated regions), and find that one pre-trained DP can be steered to several distinct preferences without sacrificing the underlying task success. Finally, Yu et al. [13] deploy D3P on a Franka for Square peg-insertion at $33.68$ Hz, a $1.92\times$ speed-up over a fixed 10-step diffusion policy, with sim-to-real bridged by a latent diffusion model that restyles real images toward the simulator and a curriculum over camera parameters.

5.3. Evaluation Metrics

Across simulation and real-world studies, four metrics recur. Success rate (SR) is the dominant scalar metric for manipulation, but it saturates once a policy reliably completes the task and then cannot distinguish a competent policy from an optimal one. Episodic return (and, on D4RL, its normalized variant [57]) supplies finer-grained credit. NFE is the dominant algorithmic cost axis for acceleration comparisons [13,25,26], but deployment also requires wall-clock latency and control-rate reporting. For locomotion and continuous-control evaluations, alive duration (time-until-falling) and linear/angular mean absolute errors provide robustness-oriented metrics [8]. In our experiment, a binary survival label was saturated whereas 50-episode time-until-falling and observation-perturbation curves remained discriminative (Section 6.3). Because success, return, and reliability can move independently, we recommend reporting a quality metric and at least one non-saturated reliability metric together; this is the substance of desideratum D3 below.

5.4. The Methodological Gap and Desiderata

As the metrics above and the platform diversity of Table 6 make clear, the community has yet to converge on a standardized protocol for comparing DPRL methods, particularly across the simulation–real-world boundary. Reported results differ in the base policy (image- vs. state-based), the denoising horizon K, the number of fine-tuned steps, the random-seed budget, and even the headline metric (SR vs. return vs. NFE), which makes cross-paper comparison unreliable. We argue that a useful reference protocol for the dominant offline-to-online setting should satisfy three desiderata: (D1) a transparent, low-dimensional base task on which the policy’s behavior can be inspected action-by-action, decoupled from a high-dimensional perceptual front-end; (D2) a sub-optimal demonstration source, so that the protocol genuinely tests whether RL can improve beyond the demonstrator rather than merely reproduce it; and (D3) multi-seed reporting of both a quality metric (episodic return/normalized score) and a non-saturated reliability metric (e.g., time-until-falling or perturbation robustness), since the two can move independently (Section 5.3). The empirical study of Section 6 instantiates D1–D3 on a transparent MuJoCo locomotion reference and documents the remaining limits to broader validity (Section 6.9).

6. Empirical Study: RL Fine-Tuning vs. Imitation for Diffusion Policies

This section tests RL improvement over diffusion behavior cloning under a controlled, state-based MuJoCo protocol. It reproduces the DP-versus-DPPO comparison [6] and adds bounded checks across credit-assignment routes, a second locomotion task, and NFE settings. The objective is an interpretable reproduction, not a new algorithm or state-of-the-art result. It instantiates D1–D3 but contains no visual encoder, image observation, manipulation task, or perception-to-action robustness test.

6.1. Research Questions

The study is organized around five research questions that connect directly to the themes of Section 3 and Section 4. The first two interrogate the primary DPPO-on-Hopper comparison in depth; the next three provide bounded checks along three coordinates that dominate cross-paper incomparability—credit assignment, a second task within the same locomotion domain, and inference budget (NFE):

• RQ1 (performance and reliability). By how much does DPPO fine-tuning improve over the behavior-cloned diffusion policy from which it is initialized, and does the result persist across training seeds, stochastic rollouts, and observation perturbations?
• RQ2 (behavioral change). Where and how much does fine-tuning alter the policy’s actions across episodes and training seeds? Is the change approximately uniform, or does its upper tail identify timesteps at which replacing the DPPO action is especially consequential?
• RQ3 (credit assignment and parameterization control). When the four T4 update routes—proximity-based DPPO [6], action-gradient QSM [19], the generic Q-weighting AWR baseline [110], and a controlled BPTT-based DACER adaptation [85]—start from the same diffusion BC policy, do their outcomes exhibit the route-specific failure modes discussed under the T4 coordinate? Separately, does a standard PPO policy initialized from non-diffusion Gaussian behavior cloning also improve under an interaction-matched Hopper protocol?
• RQ4 (within-domain locomotion robustness). Does the return gain persist on Walker2d-v2, a second, higher-dimensional but still state-based locomotion task?
• RQ5 (inference economy). How do return and measured per-chunk latency change when the same selected DPPO checkpoints are evaluated under deterministic DDIM grids with NFE $\in \{2,5,10,20\}$?

Scope of the Empirical Study

RQ1–RQ2 form the primary DPPO-on-Hopper probe; RQ3–RQ5 are targeted checks rather than taxonomy-wide validation. Exploration and stability are examined through action divergence, reliability, and four T4 routes. Likelihood handling is probed through DPPO, QSM, AWR, full-chain BPTT, and a separately initialized Gaussian BC-to-PPO control. Inference acceleration is evaluated by changing NFE while holding checkpoints and actor routing fixed. The study neither measures distributional mode coverage nor trains a recent acceleration baseline, and every run uses ground-truth environment reward rather than learned or preference-derived reward. Both tasks expose only proprioceptive states. The DACER result is a BC-initialized, four-action-chunk adaptation rather than an official from-scratch reproduction, and the Gaussian control has separate initialization records. Accordingly, RQ3 is a configuration-level probe and RQ4 is a within-domain locomotion check, not evidence of visuomotor generalization.

6.2. Experimental Setup

6.2.1. Task and Data

We use the legacy Gym task identifier Hopper-v2 (Figure 6, left), a MuJoCo one-legged hopper with a 3-dimensional continuous action and an 11-dimensional proprioceptive observation. The behavior-cloning dataset is the D4RL 1.1 identifier hopper-medium-v2 [57], generated by a partially trained (sub-optimal) policy. The retained data audit identifies the local train.npz by size (57,367,454 bytes) and MD5 digest (089a721d88cf384b41b271f29368f764). Its historical download file ID was not retained, so benchmark version, dataset name, size, and digest are reported instead of reconstructing an unverified command. The offline data provide BC initialization and normalization; subsequent fine-tuning uses online Hopper-v2 interaction. Following D4RL, we report $100\times (\overline{R}-R_{\mathrm{rand}})/(R_{\mathrm{expert}}-R_{\mathrm{rand}})$. The within-domain check uses the recorded Walker2d-v2/walker2d-medium-v2 pair (Figure 6, right). Both are low-dimensional, state-based locomotion benchmarks selected for action-level inspection. Neither task supplies images, point clouds, language, or other exteroceptive observations.

Figure 6. The two state-based D4RL 1.1 MuJoCo environments used in the empirical study. (Left): Hopper-v2/hopper-medium-v2 (action $\in {\mathbb{R}}^3$, observation $\in {\mathbb{R}}^{11}$), the testbed for RQ1–RQ3 and RQ5. (Right): Walker2d-v2/walker2d-medium-v2 (action $\in {\mathbb{R}}^6$, observation $\in {\mathbb{R}}^{17}$), the within-domain RQ4 check. This environment illustration contains no aggregate score or uncertainty statistic and is not a substitute for a perceptual or manipulation benchmark.

6.2.2. Pre-Trained Diffusion Policy (DP/BC)

The base policy is an $ϵ$-predicting residual MLP diffusion policy [1,6]: three 512-unit ReLU hidden layers, a 16-dimensional time embedding, one 11-dimensional conditioning observation, and a predicted four-step sequence of three-dimensional actions. Run-specific Hydra configurations reveal two archived BC initializations: seeds 7 and 42 load state_300.pt (MD5 _{ba16e47eef71ba6f469a6b3a7af76ab3}), whereas seeds 100, 2024, and 3407 load state_3000.pt (MD5 _{0b4a0e23d0458cba49b74fa3d814d59c}). The filenames are identifiers, not inferred epoch counts. The archived state_3000.pt pre-training configuration records seed 42, 200 epochs, batch size 128, learning rate ${10}^{-3}$, weight decay ${10}^{-6}$, cosine learning-rate scheduling with one warm-up epoch and minimum ${10}^{-4}$, and EMA decay 0.995; the complete historical pre-training log for state_300.pt was not retained. Both files share the recorded architecture, D4RL dataset identifier, normalization file, and sampler: 20-step DDPM, cosine $\beta$ schedule with $s=0.008$, $ϵ$ prediction, no DDIM subsampling, denoised/noise clipping at $1/3$, and minimum sampling standard deviation 0.1. At evaluation, BC executes all $K=20$ steps with no parameter update.

6.2.3. RL Fine-Tuning (DPPO)

The fine-tuned policy is obtained by running DPPO [6] on top of the run-recorded BC checkpoint. The recorded implementation is commit cc7234ad7f, and the configuration is ft_ppo_diffusion_mlp. We fine-tune the last 10 of 20 denoising steps using the per-step Gaussian likelihood of (16). The retained configuration records $\gamma =0.99$, GAE $\lambda =0.95$, five update epochs, batch size 50,000, actor/critic learning rates ${10}^{-4}/{10}^{-3}$, value-loss coefficient 0.5, target KL 1.0, denoising discount 0.99, and policy-loss clipping coefficients (current/base/rate) $0.01/0.01/3$. Running-return reward standardization is enabled with multiplier 1.0; no separate reward-clipping value appears in the retained DPPO configuration. The five fine-tuning seeds are 7, 42, 100, 2024, and 3407. They are independent fine-tuning runs but not independent BC repetitions because two BC files are reused. Each run has 100 outer iterations; $0,10,\dots ,90$ are validation iterations. Each non-validation iteration contains $20\times 500=10,000$ action-chunk decisions, corresponding to up to 40,000 primitive actions for $T_a=4$. Thus each seed has 900,000 training decisions and a nominal $3.6\times {10}^6$ primitive-step budget. The confirmatory comparison uses fixed state_99.pt; training-peak checkpoints at iterations 40–60 are used only for separately labeled exploratory probes.

6.2.4. Non-Diffusion Gaussian BC-to-PPO Control

To separate the existence of an RL fine-tuning gain from the use of a diffusion generator, we add a non-diffusion Gaussian control on the same Hopper-v2 task, hopper-medium-v2 dataset, normalization file, observation dimension, action dimension, and action chunk $T_a=4$. The Gaussian mean policy is a three-hidden-layer MLP with widths $[256,256,256]$, Mish activations, and a tanh output. Behavior cloning uses fixed standard deviation $0.1$, seed 42, and 200 epochs; its recorded checkpoint is state_200.pt. PPO initializes the complete mean network from this checkpoint and makes the global log-standard-deviation trainable from the same initial value. An explicit inheritance audit compares all eight mean-network tensors (137,740 parameters) in network, actor, and actor_ft; every comparison has zero relative ${\mathsf{\ell }}_2$ and zero maximum absolute difference at state_0.

Gaussian PPO uses seeds 42, 7, and 100, 20 parallel environments, 500 action-chunk decisions per outer iteration, 100 outer iterations with validation every 10 iterations, five PPO update epochs, actor and critic learning rates of ${10}^{-4}$ and ${10}^{-3}$, respectively, batch size 50,000, PPO clipping coefficient 0.1, and checkpoint saves every five iterations. Its recorded reward transform divides by the running standard deviation of discounted returns, applies a constant multiplier of 1.0, and clips the resulting reward to $[-10,10]$. The interaction units therefore match the primary DPPO configuration: 900,000 training action-chunk decisions and a nominal $3.6\times {10}^6$ primitive steps per seed. Checkpoints are selected from the training-reward peak, excluding iteration 0, by taking the nearest saved state; the selected iterations are 99, 55, and 90 for seeds 42, 7, and 100. Final fixed-policy evaluation uses the same 20 environment seeds (7000–7019), deterministic action selection, and a 1000-step episode cap for the Gaussian BC checkpoint and every selected Gaussian PPO checkpoint. The three PPO training-seed means are the top-level units; episode-level variability of the single fixed BC checkpoint is reported separately. This control was run on a Tesla T4, whereas the primary DPPO runs were recorded on an NVIDIA A100-SXM4-40GB; wall-clock durations are therefore not used for a cross-generator speed ratio.

6.2.5. Evaluation Protocol

The confirmatory RQ1 protocol evaluates every fixed state_99.pt DPPO checkpoint and the exact BC checkpoint named in that run’s Hydra configuration. Every checkpoint is evaluated without parameter updates for 20 deterministic episodes, using common environment seeds 7000–7019, NFE 20, and a 1000-step episode cap. For each fine-tuning seed, we first average the 20 episodes and form the paired difference between DPPO and its own BC initialization; the five fine-tuning-seed differences are the top-level inferential units. We report their mean, standard deviation, paired Student-t 95% confidence interval and two-sided test, and a 20,000-repetition hierarchical bootstrap confidence interval that resamples fine-tuning seeds and episodes. BC episode variability is not promoted to extra training replications, and the repeated mapping of two BC files to five runs is made explicit.

The original selected-checkpoint evaluator uses 10 parallel environments, deterministic action selection, evaluation seed 42, and a 2000-step rollout budget. It retained one aggregate return per fine-tuning seed but not completed-episode counts or raw episode returns; its $2787\pm 35$ result is therefore retained only as a secondary sensitivity result and as context for cached analyses that used those checkpoints, not as the Major-6 headline. For the separate stochastic reliability analysis, we evaluate 50 episodes for BC and 50 for DPPO (10 per fine-tuning seed) using common environment seeds. For RQ2, each of the five selected DPPO checkpoints controls 10 episodes while BC is queried counterfactually on the same states, yielding 50 episodes and 49,306 environment steps. Observation-perturbation and intervention probes use seeds 7, 42, and 100, with 10 episodes per seed and condition.

6.2.6. Additional Controlled Checks (RQ3–RQ5)

The additional checks reuse the same harness but not always the same seed budget. (RQ3, credit-assignment routes and parameterization control.) From the identical diffusion Hopper BC checkpoint we fine-tune, in addition to DPPO, QSM [19] (an eligible action-gradient diffusion-policy method) and AWR [110] (a canonical Q-weighting regression baseline), using the official codebase’s ft_qsm_diffusion_mlp and ft_awr_diffusion_mlp configurations. AWR is not entered in the evidence map as a diffusion-policy method; here it is an implementation-level control that applies a generic weighting objective to the shared diffusion-policy backbone. These three matched update routes retain seeds 7, 42, and 100 and an 80-iteration schedule with no per-method tuning; they characterize out-of-the-box behavior, not compute-matched best performance.

To instantiate the fourth, BPTT-based family, we add a controlled BC-initialized adaptation of DACER [85]. The adaptation preserves full-chain differentiation through all 20 reverse-diffusion steps, gradient checkpointing, a twin distributional critic, a three-component Gaussian-mixture entropy estimator, and the log-$\alpha$ exploration regulator. The actor retains the shared residual MLP, while each critic branch uses three 256-unit Mish layers. The actor and critic learning rates are both ${10}^{-4}$ with five warm-up iterations and cosine scheduling to ${10}^{-5}$; the actor/critic batch sizes are 128/256, and the actor is updated after every two critic updates. The replay capacity and warm-up are 100,000 and 30,000 chunk transitions, respectively. Four-action transitions use $\gamma ={0.99}^4=0.96059601$, reward scale 0.2, target-critic coefficient 0.005, and gradient-norm clipping at 1.0. The distributional critic uses standard-deviation update coefficient 0.005, Huber threshold 50, and standard-deviation bias 0.1. The entropy regulator starts at $\alpha =3.0$, uses learning rate 0.03 and bounds $[{10}^{-4},10]$, targets entropy $-10.8$, and scales external exploration noise by $0.1\alpha$. Each entropy update fits a three-component GMM to 200 actions for each of 20 states and occurs every 10,000 policy decisions.

The adaptation starts from the same state_3000.pt BC weights and normalization file as the shared-backbone three-route comparison. Before training, a per-seed inheritance audit compares all 12 actor tensors (553,020 parameters) against the BC checkpoint; both relative ${\mathsf{\ell }}_2$ error and maximum absolute difference are zero for every seed. The five independent training seeds (42, 7, 100, 1, and 2) follow the shared 80-iteration, 20-environment, 500-decision schedule with validation every 10 iterations. The 72 non-validation iterations therefore contain 720,000 action-chunk decisions and a nominal $2.88\times {10}^6$ primitive steps per seed. Each completed run records 1400 critic, 700 actor, and 70 entropy-regulator updates. To preclude post hoc selection, every seed is evaluated at the fixed final checkpoint state_79.pt. Final evaluation applies the evaluator’s deterministic flag, disables external $\alpha$-scaled exploration noise, and fixes the environment, NumPy, PyTorch, and CUDA seeds for the same 20 episodes (7000–7019) for BC and every adapted checkpoint. Seeded reverse-process noise is retained at denoising steps $t>0$, preserving the implemented sampler while tying every reported episode to recorded RNG seeds and the recorded software stack. The protocol is bound to SHA-256 signature 9f98fe91951660fe2c585522d50745d20b9ab6a0d5a8154156a234a529aa9551. This construction is interaction-matched to the three-route schedule but not compute-matched at the optimizer-update level. Because canonical DACER is trained online from scratch, the result is reported explicitly as a BC-initialized DACER adaptation, not as an official-setting reproduction.

The separately initialized Gaussian BC-to-PPO experiment described above adds a non-diffusion control under the primary 100-iteration interaction schedule; because its BC architecture and evaluation records differ, it is not inserted into the shared-diffusion-backbone ranking. (RQ4, within-domain locomotion.) We repeat the DP-versus-DPPO comparison on Walker2d-v2 from the official walker2d-medium-v2 BC checkpoint over three seeds. (RQ5, inference economy.) The original BC-only probe is retained as a bounded reference. The new DPPO analysis is an evaluation-only, pure-NFE experiment on the five previously selected checkpoints (states 50, 40, 60, 40, and 45 for training seeds 42, 7, 100, 2024, and 3407, respectively). Checkpoint selection is completed before the NFE loop, and no model is retrained or selected using these evaluation results. Every condition uses deterministic DDIM with NFE $\in \{2,5,10,20\}$. To avoid changing the policy definition when the DDIM grid is subsampled, actor routing is expressed in the original 20-step coordinates: the fine-tuned actor is used if and only if a retained original timestep satisfies $t<10$. The retained grids are $\{19,\dots ,0\}$, $\{18,16,\dots ,0\}$, $\{16,12,8,4,0\}$, and $\{10,0\}$, which mechanically produce 10, 5, 3, and 1 fine-tuned-actor calls. Thus checkpoint weights, actor-routing rule, normalization, action chunk, and environment seeds remain fixed; only the retained DDIM time grid changes. Each checkpoint–NFE condition is evaluated for 20 episodes on common environment seeds 7000–7019 with a 1000-step cap, and the five training-seed checkpoint means are the top-level units. Latency is measured for every checkpoint–NFE condition with batch size one on an NVIDIA A100-SXM4-40GB after 10 warm-up calls and 100 CUDA-synchronized forward passes. The timing covers one generated four-action chunk from the state-conditioned sampler only; it excludes sensing, visual encoding, communication, environment stepping, and actuator/control-loop delays. All four conditions use DDIM, so the NFE-20 point is the controlled DDIM reference and is not substituted for the DDPM headline result in RQ1. Walker and Hopper normalization use the D4RL references [57]; for Hopper, $R_{\mathrm{rand}}=-20.272$ and $R_{\mathrm{expert}}=3234.3$.

6.2.7. Reproducibility Record

The retained execution record links the source revision, package versions, Hydra settings, protocol signature, seeds, optimizer and replay settings, checkpoint rules, dataset and normalization hashes, run-to-BC mapping, evaluated-checkpoint hashes, and statistical units. It also contains the fixed-final episode table, inheritance and mechanism audits, checkpoint-selection table, per-seed compute audit, and pure-NFE timing records used in the analyses. Table 7 and the exact normalization block below provide a configuration-level reconstruction of the reported protocols, while cryptographic hashes bind the recorded settings to the executed source, initialization, normalization, and selected checkpoints.

Table 7. Reproducibility checklist for the primary Hopper-v2 experiment and controls. RQ3’s original DPPO/QSM/AWR comparison and RQ4 use three seeds; the primary fixed-final DPPO comparison and controlled DACER adaptation use five.

Item	Recorded Setting
Code revision	Repository irom-princeton/dppo, detached commit cc7234ad7ff39a8f32de3af903606723a16f0648. DACER protocol signature: SHA-256 9f98fe91951660fe2c585522d50745d20b9ab6a0d5a8154156a234a529aa9551. Custom DACER model/agent/configuration MD5 values: b8a54fbd2ea2002b5745cbf74639b1c0, 5c761f1984323e3b87cfb67700f2038e, and 1c484cc0a8efbc0e0ead9bc9de5e6c22
Software	Python 3.10.12; PyTorch 2.4.1+cu124; torchvision 0.19.1+cu124; NumPy 1.23.5; Gym 0.22.0; mujoco-py 2.1.2.14; D4RL 1.1; Hydra 1.3.2; OmegaConf 2.3.0; SciPy 1.10.1; scikit-learn 1.3.2; pandas 2.0.3. Pure-NFE evaluator: PyTorch 2.4.0+cu121
Hardware and duration	Primary DPPO, DACER, and pure-NFE: NVIDIA A100-SXM4-40GB; original CPU model not retained. The available DPPO duration logs mix three 80-iteration and two 100-iteration histories, so no unified five-seed full-run time is claimed. DACER: $0.326\pm 0.006$ h/seed. Gaussian control: Tesla T4. Latency: batch 1, 10 warm-ups, 100 synchronized repetitions/checkpoint/NFE
Task, data, and BC	Hopper-v2; D4RL 1.1 hopper-medium-v2; train.npz: 57,367,454 bytes, MD5 089a721d88cf384b41b271f29368f764; normalization.npz: 1130 bytes, MD5 9deb22db5e7a8d6426b5a15948940f9e. BC hashes: state_300.pt, ba16e47eef71ba6f469a6b3a7af76ab3; state_3000.pt, 0b4a0e23d0458cba49b74fa3d814d59c
Network and sampler	Actor: residual MLP $[512,512,512]$, ReLU, time embedding 16, observation/action dimensions $11/3$, condition horizon 1, $T_a=4$. Shared training sampler: $ϵ$-predicting 20-step DDPM, cosine $s=0.008$, no DDIM, denoised/noise clips $1/3$, minimum sampling s.d. 0.1
DPPO training	Seeds $7,42,100,2024,3407$; 100 iterations; 20 environments; 500 chunk decisions/iteration; validation every 10; last 10/20 denoising steps fine-tuned. $\gamma =0.99$; GAE $\lambda =0.95$; 5 epochs; batch 50,000; actor/critic LR ${10}^{-4}/{10}^{-3}$; value coefficient 0.5; target KL 1; denoising discount 0.99; clipping current/base/rate $0.01/0.01/3$; running-return standardization, multiplier 1.0
Selection and evaluation	Confirmatory RQ1: fixed state_99.pt; each DPPO policy and its run-recorded BC: 20 deterministic common-seed episodes (7000–7019), NFE 20, 1000-step cap, no updates. Selected-checkpoint probes: training-return peak excluding iteration 0, nearest saved state (40–60)
Gaussian control	BC seed 42, 200 epochs, state_200.pt; exact BC-to-PPO inheritance; PPO seeds $42,7,100$ and states $99,55,90$; 20 deterministic common-seed episodes/checkpoint
DACER optimization	Seeds $42,7,100,1,2$; full 20-step BPTT with gradient checkpointing; 80 iterations, 20 environments, 500 decisions/iteration, validation every 10; replay capacity/warm-up 100,000/30,000; actor/critic LR ${10}^{-4}/{10}^{-4}$, cosine minimum ${10}^{-5}$ after five warm-up iterations; actor/critic batches 128/256; actor updated every two critic updates; $\gamma =0.96059601$, reward scale 0.2, target coefficient 0.005, gradient clip 1.0
DACER critic and entropy	Twin three-layer $[256,256,256]$ Mish distributional critics; standard-deviation coefficient 0.005, Huber threshold 50, standard-deviation bias 0.1. GMM components 3; 20 entropy states and 200 action samples/state; ${\alpha }_0=3$, LR 0.03, bounds $[{10}^{-4},10]$, target entropy $-10.8$, exploration scale $0.1\alpha$; entropy update every 10,000 decisions
DACER integrity and evaluation	BC inheritance verified over 12 actor tensors/553,020 parameters for every seed (relative ${\mathsf{\ell }}_2=0$, maximum absolute difference $=0$); fixed state_79.pt; 720,000 decisions, $2.88\times {10}^6$ primitive steps, 1400 critic/700 actor/70 entropy updates per seed; 20 fixed-RNG common-seed episodes/checkpoint; external $\alpha$-scaled exploration disabled and seeded reverse-process noise retained at $t>0$. Resume restores model, optimizers, schedulers, replay, counters, and RNG at an iteration boundary
Score and provenance	$100(R-R_{\mathrm{rand}})/(R_{\mathrm{expert}}-R_{\mathrm{rand}})$, with Hopper $R_{\mathrm{rand}}=-20.272$, $R_{\mathrm{expert}}=3234.3$. The protocol-signed DACER manifest records optimization, replay, entropy, checkpoint-selection, evaluation, and compute settings. The historical D4RL download file ID and original CPU model were not retained; the dataset identifier, normalization values/hash, GPU model, package stack, and interaction counts are recorded

• Exact Hopper normalization constants. Observations use ${\mathit{x}}_{\mathrm{norm}}=2[(\mathit{x}-{\mathit{x}}_{min})/({\mathit{x}}_{max}-{\mathit{x}}_{min}+{10}^{-6})-0.5]$; normalized actions are mapped back by $\mathit{a}=({\mathit{a}}_{\mathrm{norm}}+1)({\mathit{a}}_{max}-{\mathit{a}}_{min})/2+{\mathit{a}}_{min}$. The retained float32 arrays are:
• x_min = [0.700007558, −0.194988877, −1.49306750, −1.48139060, −0.963951230,
        −0.503772974, −5.18923235, −3.20404005, −6.48566675, −10, −10]
• x_max = [1.72902501, 0.155243099, 0.0171932708, 0.0701925308, 0.957344055,
        5.01792908, 3.17441726, 3.83158350, 5.68633699, 9.22635174, 10]
• a_min = [−0.999967873, −0.999983489, −0.999968231]
• a_max = [0.999884307, 0.999948323, 0.999994516]

6.3. RQ1: Does RL Fine-Tuning Improve Beyond the BC Initialization?

Table 8 reports the confirmatory fixed-final comparison. The run-recorded BC checkpoint has a 20-episode mean of 1572.8 for fine-tuning seeds 7 and 42 and 1530.7 for seeds 100, 2024, and 3407. Mapping these two policies to the five runs gives a descriptive mean of $1547.5\pm 23.1$ (D4RL-normalized $48.2\pm 0.7$); this spread is not BC pre-training uncertainty. The five fixed state_99.pt DPPO checkpoints obtain $2808.8\pm 107.6$ across fine-tuning-seed means (normalized $86.9\pm 3.3$), with a Student-t 95% CI of $[2675.2,2942.3]$. Pairing each DPPO checkpoint with its actual initialization on identical environment seeds gives a gain of $1261.2\pm 125.5$ raw-return units, or 38.8 normalized-score points. The paired Student-t 95% CI is $[1105.3,1417.1]$ ($p=2.32\times {10}^{-5}$, two-sided), and the hierarchical seed-and-episode bootstrap gives $[1127.7,1381.2]$ over 20,000 repetitions. All five paired gains are positive. The result therefore provides seed-level evidence for a gain from DPPO fine-tuning under the recorded run-specific initializations, while not estimating variability from independently retraining BC five times.

Table 8. Confirmatory deterministic evaluation on Hopper-v2. Each fixed DPPO state_99.pt and its run-recorded BC initialization is evaluated for 20 episodes on common environment seeds 7000–7019. Values are mean ± s.d. across five run mappings or fine-tuning-seed means. Only two distinct BC files occur, so the BC spread is descriptive and is not five-seed pre-training uncertainty. FT is the number of fine-tuned denoising steps and Norm. is the D4RL-normalized score.

Policy	FT	Top-Level Unit	Return	Norm.	95% CI (Return)
Run-mapped DP (BC)	0	5 mappings/2 files	$1547.5\pm 23.1$	$48.2\pm 0.7$	–
DPPO, state 99	10	5 fine-tuning seeds	$2808.8\pm 107.6$	$86.9\pm 3.3$	$[2675.2,2942.3]$
Paired DPPO minus own BC initialization			$1261.2\pm 125.5$	$+38.8$	$[1105.3,1417.1]$

Figure 7 provides a qualitative view of one separately recorded deterministic rollout pair. It is included to make the state-based locomotion behavior visually concrete, not as additional statistical evidence: each policy controls its own environment, and the two displayed returns are single-episode outcomes. The five-seed paired inference in Table 8 remains the basis for RQ1.

Figure 7. Qualitative illustration of separate deterministic Hopper-v2 rollouts from a behavior-cloned diffusion checkpoint and its recorded DPPO-adapted counterpart at evaluation seed 42. The displayed BC and DPPO returns (1244 and 2919) refer only to these two illustrative episodes; they are not additional independent replications and are not used for the inferential comparison in Table 8. Both policies use the reported 20-step diffusion sampler, and each policy generates the trajectory shown in its own row.

The archived learning logs in Figure 8 show a rapid rise over their common prefix. Three logs end at outer iteration 79 and two end at 99; the plot therefore uses only their common prefix through iteration 79 (72 non-validation updates, 720,000 action-chunk decisions, and a nominal $2.88\times {10}^6$ primitive actions per seed). These curves are a descriptive training diagnostic and are not used to compute the fixed-state_99.pt estimate in Table 8. The fixed-final Hydra records specify the full 100-iteration, 900,000-decision protocol, but a complete five-seed set of matching 100-iteration curve and duration logs was not retained. We consequently report neither a unified five-seed 100-iteration AUC nor a fixed-final wall-clock mean. The Gaussian control matches the nominal interaction budget but was run on a T4 rather than the A100 used for DPPO, so no cross-generator speed ratio is inferred.

Figure 8. Archived full-20-step DPPO training dynamics over five seed labels (7, 42, 100, 2024, and 3407), restricted to the common prefix through outer iteration 79. Thin lines are individual training-rollout means, the solid line is the cross-seed mean, and the shaded band is the untruncated Student-t 95% confidence interval. The vertical marker at zero interactions identifies the run-specific BC initialization; the first reward observation follows 10,000 action-chunk decisions. The plotted 72 non-validation updates equal 720,000 action-chunk decisions (20 environments × 500 decisions/update) and a nominal $2.88\times {10}^6$ primitive actions for $T_a=4$. These heterogeneous archived histories are descriptive and distinct from the fixed 20-episode state_99.pt evaluation in Table 8.

Reliability Beyond Binary Success

The original binary survival rate was saturated and therefore masked meaningful differences. We consequently report time-until-falling (episode length, capped at 1000 steps) under stochastic action sampling. Across 50 episodes per policy group, BC survives $419.4$ steps on average whereas DPPO survives $900.7$; the 5th percentiles are 271.6 and 548.2 steps, respectively (Table 9). Mean stochastic return is $1287.1$ for BC (episode-level s.d. 292.7; 95% confidence-interval half-width 83.2) and $2754.2$ for DPPO (s.d. 249.4 and 95% confidence-interval half-width 309.7 across five training-seed means). Thus fine-tuning improves both return and stochastic rollout reliability under this separate protocol. These 50-episode samples are not the data underlying the deterministic headline statistic in Table 8.

Table 9. Non-saturated stochastic reliability on Hopper-v2. BC uses 50 episodes; DPPO pools 10 episodes from each of five fine-tuning seeds under the separate selected-checkpoint protocol. Episode length is time-until-falling with a 1000-step time limit. Return and length are descriptive episode-level aggregates.

Policy	Episodes	Mean Length	5th pct.	Minimum	At Cap	Mean Return
BC	50	419.4	271.6	170	0%	1287.1
DPPO	50	900.7	548.2	437	64%	2754.2

We additionally inject zero-mean Gaussian noise in normalized observation space at $\sigma \in \{0,0.05,0.10,0.20,0.30\}$ using deterministic action selection (Figure 9). DPPO retains longer episodes than BC up to $\sigma =0.20$; at $\sigma =0.30$ both fall to approximately 187 steps. At $\sigma =0.10$, for example, the DPPO seed-level mean survival is $582\pm 175$ steps (mean ± s.d. across three training-seed means), whereas the fixed BC checkpoint averages 378 steps over its 10 episodes. This probe measures robustness to a specified observation perturbation, not certified safety. Because the stochastic protocol of Table 9 and the deterministic perturbation protocol differ, their zero-noise values are not expected to coincide.

Figure 9. Reliability under normalized observation noise with the full NFE-20 sampler. BC uses 10 episodes per condition from one fixed checkpoint. DPPO uses 10 episodes per training seed and condition for seeds 7, 42, and 100 (30 DPPO episodes per condition); plotted DPPO error bars are mean ± s.d. across the three seed-level means. (Left): mean time-until-falling. (Right): mean episodic return. The dashed line marks the 1000-step time limit. Different noise levels use different environment-seed blocks, so the curve supports condition-wise comparisons but is not a fully paired dose–response experiment.

6.4. RQ2: How Does Fine-Tuning Reshape the Policy’s Actions?

A higher return tells us that the policy changed, not how. We define the step-level action-divergence score in de-normalized action space as

$$d_t={∥{\mathit{a}}_t^{\mathrm{BC}}-{\mathit{a}}_t^{\mathrm{DPPO}}∥}_2,$$

(21)

For Equation (21), DPPO controls the environment and BC is queried counterfactually on the same observation. The extension evaluates 10 episodes for each of five selected-checkpoint fine-tuning seeds (50 episodes, 49,306 steps) with the full NFE-20 sampler. With training seeds as the top-level units, mean divergence is $0.444\pm 0.053$ (mean ± s.d.; Student-t 95% CI $[0.378,0.509]$), and the seed means span $0.400$–$0.530$. Across the 50 nested episodes, the descriptive episode-level P95 and maximum average $1.099$ and $1.832$, respectively; the episodes are not treated as 50 independent training replications.

For an operational high-divergence definition, we set $\tau$ to the pooled P95, $\tau =1.115$, and label $d_t>\tau$ as a high-divergence step. By construction, 5% of pooled steps exceed this threshold; this fraction is a definition rather than evidence that exactly 5% of decisions are intrinsically “critical.” Across the 50 nested episodes, the exceedance fraction is $4.99\%$ with an episode-level s.d. of 2.84 percentage points. This dispersion is descriptive and is not promoted to an inferential interval because episodes are nested within five training seeds. Figure 10 combines a representative trajectory with all 50 per-episode cumulative distributions.

Figure 10. Selected-checkpoint, NFE-20 action-divergence analysis on Hopper-v2. (Left): one representative DPPO-controlled episode, with BC queried counterfactually on the same states; the dashed line is that episode’s mean and the dotted line is the pooled P95 threshold $\tau =1.115$. (Right): empirical CDFs for 50 episodes (10 nested episodes per each of five DPPO fine-tuning seeds). Actions are compared after de-normalization. The plot documents the distribution and cross-seed recurrence of large action differences but neither identifies action-distribution modes nor assigns causal task criticality.

The recorded state features provide only limited semantic localization. At high-divergence steps, mean torso height is $1.486$ versus $1.425$ elsewhere (standardized mean difference 0.327), and mean signed torso angle is $-0.0956$ versus $-0.1107$ (standardized mean difference 0.384). These are descriptive associations. Only five recorded episodes terminate before the time limit; in that subset, the late-50-step minus earlier high-divergence fraction is $0.054\pm 0.089$ (mean and 95% confidence-interval half-width), which is inconclusive. Contact indicators, per-step advantage, value estimates, and per-step rewards are unavailable in the cached divergence files. We therefore use “high-divergence” for the thresholded score and test consequentiality separately.

For the three primary training seeds, we first record an unintervened trajectory and select, within its first 200 steps, the 10 largest-divergence indices and 10 random indices. We then rerun the same environment seed under pure DPPO and under two scheduled replacements of the DPPO action by the BC action. All three conditions use the selected-checkpoint evaluator’s deterministic action-selection setting. This yields 10 paired environment seeds per training seed and 30 episodes per condition; all intended schedules execute exactly 10 replacements. Table 10 reports loss relative to pure DPPO. Across training-seed means, the high-divergence schedule has a loss of $1.6\pm 38.8$, the matched random schedule has a loss of $-20.4\pm 13.6$, and their difference is $22.1\pm 34.8$. The 95% confidence interval for the difference, $[-64.4,108.5]$, spans zero, and the seed-level difference is negative for one of three seeds. Under this equal intervention budget, we therefore detect no reliable evidence that the selected high-divergence timesteps are more consequential than random timesteps.

Table 10. Matched-budget intervention on three training seeds and 10 paired environment seeds per training seed. Paired loss is $R_{\mathrm{pure}\mathrm{DPPO}}-R_{\mathrm{intervened}}$; values are mean ± s.d. across the three training-seed means. Both intervention conditions replace exactly 10 scheduled actions within the first 200 steps. A negative loss means that the intervened rollout obtained a higher return than its pure-DPPO pair.

Intervention	Paired Loss	95% CI	Swaps/Episode
BC action at top-10 $d_t$ indices	$1.6\pm 38.8$	$[-94.7,98.0]$	10
BC action at 10 random indices	$-20.4\pm 13.6$	$[-54.1,13.2]$	10
High-divergence minus random loss	$22.1\pm 34.8$	$[-64.4,108.5]$	–

Accordingly, we withdraw the stronger causal “critical-timestep” interpretation: the experiment supports only a Hopper-specific localization of large BC–DPPO action differences. Without a resolved intervention effect, contact/advantage annotations, or cross-environment replication, this score should not be used as a validated adaptive-computation trigger.

6.5. RQ3: Does the T4 Credit-Assignment Coordinate Diagnose Fine-Tuning Outcomes?

RQ1–RQ2 fix the update route (proximity-based DPPO) and ask how much it helps; RQ3 fixes everything but that route and asks whether the T4 credit-assignment coordinate (Section 3.1) provides a useful diagnostic. We evaluate DPPO (proximity-based), QSM [19] (action-gradient), the generic AWR baseline [110] (Q-weighting), and the controlled BC-initialized DACER adaptation [85] (BPTT-based), all starting from the identical Hopper diffusion-BC checkpoint. Table 11 keeps the original three-route evaluator and the new five-seed BPTT extension in separate panels so that their statistical units and final-evaluation protocols are not conflated. Figure 11 and Figure 12 show the corresponding training dynamics. Because the methods are neither individually tuned nor compute-matched at the optimizer-update level, the comparison characterizes these configurations rather than ranking the mechanism families universally.

Table 11. Credit-assignment comparison on Hopper-v2; every update route starts from the same diffusion-BC checkpoint. Panel A retains the original evaluator and reports mean ± s.d. over the three common training seeds (7, 42, and 100). Panel B reports the separately controlled BPTT extension: BC is one fixed policy evaluated on 20 episodes, and DACER is mean ± s.d. across five training-seed means, each evaluated on the same 20 environment seeds. The BC episode-level s.d. is therefore not mixed with DACER training-seed variation. ΔBC is the relative change from the BC mean within each panel. AWR is a generic controlled baseline rather than an evidence-map entry for diffusion-policy RL; the DACER row is a BC-initialized adaptation rather than an official-setting reproduction.

Method	Mechanism	Top-Level Unit	Return	Norm.	$\mathbf{\Delta }$BC
Panel A: original shared-evaluator comparison
DP (BC)	Imitation	Fixed checkpoint	1421	$44.3$	–
AWR	Q-weighting	3 train seeds	$1423\pm 28$	$44.3\pm 0.9$	$+0.1\%$
QSM	Action-gradient	3 train seeds	$1869\pm 51$	$58.0\pm 1.6$	$+31.5\%$
DPPO	Proximity-based	3 train seeds	$2830\pm 42$	$87.6\pm 1.3$	$+99.1\%$
Panel B: fixed-final-checkpoint BPTT extension
DP (BC; re-evaluated)	Imitation	20 episodes	$1413.2$	$44.0$	–
DACER adaptation	BPTT	5 train seeds	$746.2\pm 855.3$	$23.6\pm 26.3$	$-47.2\%$

Figure 11. Original three-route mechanism comparison corresponding to Panel A of Table 11; all routes start from one shared NFE-20 Hopper-v2 BC checkpoint. (a) Final D4RL-normalized score (mean ± s.d. over seeds 7, 42, and 100); the dashed line is the BC level. The legacy evaluator retained seed aggregates but not raw episode counts, so no episode-level uncertainty is implied. (b) Logged training-rollout reward versus iteration (mean±s.d. across the same seeds). The tested DPPO route climbs smoothly, QSM undergoes an early near-collapse before partially recovering, and AWR remains near the imitation level under the shared untuned schedule. The separately evaluated BPTT extension is shown in Figure 12.

Figure 12. Periodic evaluation of the controlled BC-initialized DACER adaptation on Hopper-v2 with external $\alpha$-scaled exploration noise disabled. Thin lines are the five training seeds (42, 7, 100, 1, and 2), the thick line is their mean, the shaded band is the untruncated Student-t 95% confidence interval, and the dashed line is the separately fixed-evaluation BC mean of 1413.2. Each point at iterations $0,10,\dots ,70$ uses fixed evaluator RNG settings and a collection budget of 20 environments × 500 action-chunk decisions; because episode duration changes sharply after collapse, the number of completed episodes per point ranges from 64 to 5000. The pre-specified final checkpoint state_79.pt is evaluated separately for 20 common environment seeds and reported in Panel B of Table 11; those final values are not displayed here. All runs retain the full 20-step denoising chain. The plotted “noise-free” axis label denotes removal of external $\alpha$-scaled exploration noise; seeded reverse-process noise remains.

In Panel A, the proximity-based recipe obtains the highest score under the shared untuned evaluator (norm. $87.6\pm 1.3$, $+99.1\%$), consistent with the fixed-final five-fine-tuning-seed DPPO result of RQ1 ($86.9\pm 3.3$). The small offset reflects the mechanism comparison’s separate three-seed, 80-iteration schedule. QSM exhibits an early critic-sensitive collapse before partially recovering to $58.0$, while AWR remains near the BC level under the shared reward scale.

The BPTT extension yields a different and substantially more variable profile. The re-evaluated fixed BC checkpoint obtains $1413.2$ raw return (episode s.d. 190.8; 95% CI $[1323.9,1502.5]$), or 44.0 normalized. Across the five DACER training-seed means, the fixed-final-checkpoint return is $746.2\pm 855.3$ (Student-t 95% CI $[-315.8,1808.3]$), corresponding to a normalized score of $23.6\pm 26.3$. The mean seed-level DACER-minus-BC difference is $-667.0$ with a 95% CI of $[-1729.0,395.1]$ ($t_4=-1.74$, two-sided $p=0.156$). One of five adapted seeds exceeds the BC mean and four do not. Thus, the tested BPTT configuration shows high seed sensitivity and frequent behavioral collapse, but its mean difference from BC is not statistically resolved at the 0.05 level. A secondary pairing on the three common training-seed labels (7, 42, and 100) and the same 20 environment seeds gives a DPPO-minus-DACER difference of $1639.3\pm 974.7$ with a 95% CI of $[-781.9,4060.5]$ ($p=0.100$). Because the two methods do not share one verified initialization hash or optimizer-update budget, this wide interval is reported as a sensitivity check rather than a method ranking.

For periodic evaluation points $(u_j,R_j)$, where $u_j$ is cumulative action-chunk decisions and $R_j$ is the mean return with external $\alpha$-scaled exploration disabled, we use the interaction-normalized trapezoidal area

$${\mathrm{AUC}}_{\mathrm{int}}={\displaystyle \frac{1}{u_J-u_0}}{\displaystyle \sum _{j=1}^J}{\displaystyle \frac{R_{j-1}+R_j}{2}}(u_j-u_{j-1}).$$

(22)

Because the interaction axis in Equation (22) is normalized by its total span, ${\mathrm{AUC}}_{\mathrm{int}}$ retains raw-return units. Across the five DACER seeds it is $278.2\pm 177.6$. Each seed consumes 720,000 policy decisions and a nominal $2.88\times {10}^6$ primitive steps; measured runtime is $0.326\pm 0.006$ h per seed on an NVIDIA A100-SXM4-40GB. These runtime values characterize this implementation only and are not used as a wall-clock ranking against the other routes.

We verify that the low returns are not a silent failure to differentiate through the denoising process. In a retained mechanism audit, every one of the 20 reverse steps has a non-zero gradient norm, ranging from $8.57\times {10}^{-5}$ to $8.84\times {10}^{-3}$ (a $103.2\times$ span); the actor gradient norm is 0.386, the twin distributional critic has positive predicted standard deviations and a finite loss, and all logged entropy-regulator gradients have the intended sign. No NaN or Inf event occurs in the five runs, and external exploration noise is disabled in final evaluation. The observed collapses are therefore empirical outcomes of this controlled configuration rather than numerical exceptions. Nevertheless, the gradient range, unstable curves, and adaptation-specific protocol are consistent with the cost and conditioning concerns associated with full-chain BPTT in Section 3.5; they do not establish that tuned or official-setting DACER, DACERv2, or D3P must behave similarly.

Together, the two panels instantiate all four T4 families, but they do not make RQ3 a universal mechanism ranking. A tuned QSM, reward-normalized AWR, official-setting DACER, or recent accelerated diffusion-specific alternative could change the ordering. The non-diffusion control below addresses a different question and is therefore not added to the shared-backbone table. RQ3 supports T4 as a diagnostic lens rather than establishing DPPO as universally superior.

Non-Diffusion Parameterization Control

Table 12 reports the independently evaluated Gaussian BC-to-PPO control. The fixed Gaussian BC checkpoint obtains a 20-episode mean return of 1740.0 (episode s.d. 178.9; 95% CI $[1656.3,1823.7]$), corresponding to a normalized score of 54.1. The three Gaussian PPO training-seed means are 2893.4, 2652.2, and 2451.1. Their mean is $2665.6\pm 221.5$ (mean ± s.d. across training-seed means; Student-t 95% CI $[2115.3,3215.8]$), or $82.5\pm 6.8$ normalized. Every training seed improves over the common Gaussian BC reference. Subtracting the fixed BC mean from each PPO training-seed mean gives a mean difference of 925.6 raw-return units (Student-t 95% CI across the three differences $[375.3,1475.8]$), equivalent to a 53.2% increase in the mean raw-return point estimate and a 28.4-point normalized-score increase. Because all three differences use the same fixed BC estimate, this interval quantifies PPO training-seed variability and does not incorporate uncertainty from retraining the Gaussian BC policy.

Table 12. Non-diffusion Gaussian parameterization control on Hopper-v2. Gaussian BC is one fixed checkpoint evaluated for 20 deterministic episodes. Gaussian BC-to-PPO is summarized across three independent training-seed means, each evaluated on the same 20 environment seeds as BC. Thus the BC episode s.d. and PPO training-seed s.d. are different statistical units and are not pooled. Episode length is capped at 1000 primitive steps.

Policy	Top-Level Unit	Return	Norm.	Mean Length	At Cap
Gaussian BC	20 episodes	$1740.0$	$54.1$	$543.8$	$0/20$
Gaussian BC-to-PPO	3 train seeds	$2665.6\pm 221.5$	$82.5\pm 6.8$	$947.8\pm 57.3$	$52/60$

This result directly controls whether PPO can improve a behavior-cloned non-diffusion generator: the answer is affirmative under the tested interaction protocol. It does not show that diffusion is either necessary or superior. As a secondary comparison, pairing the fixed-final DPPO results with Gaussian PPO on the three common training-seed labels and the same 20 environment seeds gives a DPPO-minus-Gaussian-PPO mean of $99.6\pm 261.1$ with a 95% CI of $[-548.9,748.1]$ ($p=0.577$). The Gaussian checkpoints follow their recorded peak-based selection rule, the two generators have different BC architectures and starting returns, only three labels are shared, and training used different GPU models. The interval therefore does not resolve a cross-generator difference. We use the control to conclude that the RL gain is not unique to diffusion parameterization, not to rank Gaussian PPO against DPPO.

6.6. RQ4: Does the Gain Persist in a Second State-Based Locomotion Task?

To check whether the gain is confined to Hopper within the state-based locomotion domain, we repeat the DP-versus-DPPO comparison on Walker2d-v2 (Figure 6, right), a higher-dimensional locomotion task (6-D action, 17-D observation), from the official walker2d-medium-v2 BC checkpoint. DPPO lifts the normalized score from $62.1$ to $78.2\pm 1.2$ ($+25.8\%$ in raw return; Table 13). Within this original matched three-seed protocol, the Hopper gain is larger ($+92.9\%$ in raw return; BC 45.0 normalized). These RQ4 values are retained from the matched two-task locomotion check and are distinct from the separately rerun BC reference used in the five-seed RQ1 table. One plausible explanation is the stronger walker2d-medium initialization ($62.1$ versus $45.0$ normalized), which leaves less room for improvement; the two-task comparison is too small to establish this as a general law. Reliability was not re-measured on Walker2d, so RQ4 is limited to return persistence within state-based locomotion and does not test transfer to manipulation or visual observations.

Table 13. Within-domain locomotion check (RQ4): DP-versus-DPPO on two state-based D4RL MuJoCo tasks under a matched three-seed recipe. The gain over BC appears on both tasks, but two locomotion environments are insufficient to infer a general relationship between BC quality and improvement magnitude or transfer to visuomotor manipulation.

Task	Norm. Score		Return
	BC	DPPO	BC	DPPO
Hopper-v2	$45.0$	$86.3\pm 1.4$	1445	$2788\pm 47$
Walker2d-v2	$62.1$	$78.2\pm 1.2$	2854	$3592\pm 54$

6.7. RQ5: How Does Return Trade off Against the Inference Budget (NFE)?

The original BC-only probe suggested little change between 20 and 5 DDIM evaluations and a drop at 2, but it contained one checkpoint and one evaluation seed. We therefore test the fine-tuned policy under the controlled protocol of Section 6.2.6. The five selected DPPO checkpoints are fixed before the sweep, all conditions use deterministic DDIM, and only the retained original-timestep grid changes. The rule “use the fine-tuned actor iff $t<10$” mechanically yields 10, 5, 3, and 1 fine-tuned-actor calls at NFE 20, 10, 5, and 2.

Table 14 and Figure 13 show a nonlinear quality–latency frontier. NFE 20 gives a normalized score of $86.2\pm 1.7$ and a mean four-action-chunk latency of $30.97\pm 0.72$ ms. NFE 10 nearly halves latency to $15.51\pm 0.25$ ms ($2.00\times$ speed-up); its paired raw-return change relative to NFE 20 is $-3.9\pm 68.7$ with a 95% CI of $[-89.3,81.4]$. NFE 5 lowers latency to $8.95\pm 1.26$ ms ($3.46\times$), with a paired change of $-45.4\pm 152.1$ and a 95% CI of $[-234.2,143.4]$. These intervals do not resolve a mean decrement at NFE 10 or 5 with five checkpoints, but they also do not constitute equivalence or non-inferiority tests. At NFE 2, latency falls to $3.85\pm 0.83$ ms ($8.04\times$), whereas the normalized score falls to $51.1\pm 4.1$. The paired raw-return change is $-1142.2\pm 82.8$ with a 95% CI of $[-1245.0,-1039.4]$, resolving a substantial loss under this aggressive reduction. All means and standard deviations use the five training-seed checkpoint means as the top-level units.

Table 14. Pure-NFE, fixed-checkpoint DPPO sensitivity on an NVIDIA A100-SXM4-40GB. Each row re-evaluates the same five selected checkpoints for 20 common-seed episodes; score is mean ± s.d. across the five checkpoint means. $\Delta R$ is the paired raw-return difference from NFE 20 with a Student-t 95% CI. “FT calls” is mechanically induced by the fixed original-time rule $t<10$, not independently tuned. Latency is mean ± s.d. across checkpoints; each checkpoint mean uses 100 synchronized batch-one passes after 10 warm-ups. P95 is first computed per checkpoint and then averaged.

NFE	FT Calls	Norm. Score	$\mathbf{\Delta }$R vs. 20 (95% CI)	Latency (ms)	P95 (ms)	Ceiling (Hz)
20	10	$86.2\pm 1.7$	reference	$30.97\pm 0.72$	36.92	32.3
10	5	$86.1\pm 1.9$	$-3.9[-89.3,81.4]$	$15.51\pm 0.25$	18.03	64.5
5	3	$84.8\pm 3.7$	$-45.4[-234.2,143.4]$	$8.95\pm 1.26$	10.74	111.7
2	1	$51.1\pm 4.1$	$-1142.2[-1245.0,-1039.4]$	$3.85\pm 0.83$	4.98	259.5

Figure 13. Controlled DPPO NFE sensitivity with the same five selected checkpoints held fixed across conditions. (a) D4RL-normalized score, mean ± s.d. across five training-seed checkpoint means, under 20 common environment seeds per checkpoint and NFE. (b) CUDA-synchronized batch-one A100 latency per generated four-action chunk, mean ± s.d. across the five checkpoint-level latency means. (c) The resulting score–latency relation. Only the deterministic DDIM grid changes; weights, normalization, action chunk, environment seeds, and the original-time rule selecting the base or fine-tuned actor remain fixed. All points use DDIM, including NFE 20.

The reciprocal sampler latencies imply compute-limited replanning ceilings of 32.3, 64.5, 111.7, and 259.5 Hz. These are not physical robot control frequencies: they exclude sensing, image encoding, communication, environment dynamics, and actuation. Moreover, because the sweep reuses checkpoints chosen from training-reward peaks, it supports a paired within-checkpoint NFE effect but does not replace the fixed-final RQ1 estimate or remove possible checkpoint-selection optimism. A recent accelerated-policy baseline and a dedicated distributional mode-coverage metric remain necessary to determine whether an acceleration method preserves multimodal behavior rather than only mean return.

6.8. Interpretation and Connection to the Surveyed Themes

Taken together, the five questions touch all four cross-cutting themes of Section 4; we read them off in turn.

6.8.1. Empirical Findings on Exploration and Training Stability

Across 50 episodes, large BC–DPPO action differences recur across seed-conditioned rollouts, while time-until-falling and perturbation probes do not show lower DPPO reliability under the tested protocols. The matched intervention does not validate divergence as causal criticality. The four T4 routes exhibit distinct profiles: the five-seed DACER adaptation is highly variable (return s.d. 855.3), whereas the tested DPPO routes are more stable and Gaussian PPO improves in all three seeds. Because the routes are not jointly tuned and the latter controls use different protocols or initialization, this is configuration-level evidence, not a universal ordering or proof of on-manifold exploration.

6.8.2. Empirical Findings on Likelihood and Entropy Intractability

Under the shared-backbone schedules, DPPO’s per-step Gaussian construction is more stable than the tested action-gradient, generic Q-weighting, and full-chain BPTT configurations. The BPTT audit confirms gradients through all 20 reverse steps, so its seed sensitivity is not a silent differentiation failure. PPO also improves the separately initialized Gaussian BC policy; neither result establishes a generator ranking because tuning, compute, and initialization are unmatched.

6.8.3. Empirical Findings on Inference Acceleration

The pure-NFE sweep holds checkpoints, weights, and original-time actor routing fixed. NFE 10 and 5 reduce latency by $2.00\times$ and $3.46\times$ without a resolved return change, but their intervals do not establish equivalence. NFE 2 reduces latency by $8.04\times$ while losing 35.1 normalized-score points. Moderate reduction is therefore uncertain; aggressive reduction is not free. The divergence tail motivates adaptive computation [13] but is not itself a validated allocation signal.

6.8.4. Empirical Scope of Reward Modeling and Alignment

All runs use ground-truth environment reward, isolating RL-versus-imitation from reward-model error. Learned or preference-derived rewards remain outside the empirical scope and may change both optimization and safety (Section 7.6).

6.9. Limitations and Threats to Validity

Seven boundaries define the interpretation. (i) Scope. DPPO and DACER use five training seeds, whereas the Gaussian, original three-route, and Walker2d checks use three. All tasks are state-based locomotion; visual processing, contact-rich manipulation, and a real-robot loop remain outside the empirical scope. (ii) Baselines. RQ3 instantiates all four T4 families, while the acceleration study isolates NFE rather than training a separate accelerated policy. DACER is a BC-initialized, four-action-chunk adaptation and is interaction-matched rather than optimizer- or wall-clock-matched; Gaussian and diffusion policies also have different initializations. The comparisons therefore support configuration-level diagnosis rather than a universal mechanism ranking. (iii) Criticality. The pooled P95 exceedance rate is a threshold definition, and state association uses torso height and angle. The three-seed intervention gives 95% CI $[-64.4,108.5]$ and uses schedules derived from unintervened trajectories over the first 200 steps; it consequently evaluates the specified intervention without establishing a general causal marker. (iv) NFE. The pure-NFE sweep changes only the DDIM grid for five preselected checkpoints. Five paired means give limited precision for small differences, so the NFE-10/5 intervals are interpreted as unresolved rather than as equivalence tests. A100 timing covers the sampler, which is the component changed by the intervention. (v) Statistical units and initialization. Fixed-final DPPO inference uses five fine-tuning-seed means, 20 common episodes per checkpoint, and run-specific BC links, but two distinct BC files; the reported interval therefore targets fine-tuning variability under those recorded initializations. DACER uses a single audited BC reference, and Gaussian PPO uses three seeds and one fixed BC checkpoint. Episodes remain nested within training seeds, and action-chunk decisions are reported separately from primitive-action counts. (vi) Provenance. The source revision, protocol-signed DACER manifest, Hydra/YAML configurations, software stack, dataset and normalization identifiers, initialization and selected-checkpoint hashes, schedulers, fixed-seed episode table, checkpoint-selection table, and compute audit delimit the executed configurations. The historical D4RL download file ID and CPU model were not retained; the benchmark identifier, exact normalization, GPU model, interaction counts, and numerical records used in the analysis are available in the retained record. (vii) Reward and safety. The study evaluates environment return, time-until-falling, and controlled noise robustness; constraint satisfaction and real-robot safety require dedicated deployment tests. These boundaries preserve a clear distinction between the state-based empirical study and the broader survey synthesis.

7. Discussion and Open Challenges

Diffusion-policy RL is a fast-moving area. We first draw the empirical observations of Section 6 into a broader picture, then lay out the directions we expect to shape the field in the near term.

7.1. From Empirical Observations to Open Problems

The experiments support three bounded observations. First, fixed-final DPPO improves over its run-recorded BC initialization for all five Hopper fine-tuning seeds (95% CI $[1105.3,1417.1]$), but reuse of two BC files limits the claim to those recorded initializations. Gaussian PPO also improves, so the gain is not diffusion-exclusive; unmatched protocols preclude a generator ranking. Second, large action differences recur across seed-conditioned episodes, but the equal-budget intervention does not validate them as causal criticality. Third, the four T4 routes show distinct profiles, with marked seed sensitivity in the tested full-chain BPTT adaptation, while the NFE sweep exposes a measurable quality–latency trade-off.

These configuration-level findings jointly inform exploration/stability, likelihood handling, and acceleration. The BPTT audits verify the intended mechanism without making the adaptation a universal family comparison; the pure-NFE design isolates the DDIM grid and resolves the NFE-2 loss while leaving moderate reduction uncertain. Reward modeling remains outside the experiment because all runs use ground-truth reward. None of these state-based results tests visual error, camera shift, contact-rich dynamics, or end-to-end real-robot delay.

7.2. Sample Efficiency for Real-World RL

Despite progress in offline-to-online fine-tuning, several reported real-world DPRL deployments still rely on substantial demonstrations plus online interaction; for example, FDPP evaluates settings with 100–300 demonstrations [7]. Other recent methods use different initialization and interaction budgets, so no universal demonstration count is implied. Sample-efficient on-robot RL with diffusion policies nevertheless remains open. The fixed-final Hopper configurations make the nominal interaction cost explicit: each primary DPPO seed uses 900,000 training action-chunk decisions and up to $3.6\times {10}^6$ primitive actions. The archived curve files are less complete: only a 72-update common prefix is available across all five seed labels, so we do not attach a five-seed 100-iteration AUC or wall-clock estimate to the confirmatory result. Each DACER-adaptation seed uses 720,000 decisions and a nominal $2.88\times {10}^6$ primitive actions, with an implementation-specific runtime of $0.326\pm 0.006$ h on the A100. These values are not a compute-matched algorithm ranking because optimizer schedules and software paths differ. The Gaussian PPO control matches the primary DPPO interaction units, but only three seeds were run on a T4 and one stdout log is incomplete; it therefore answers the parameterization-control question at fixed-policy evaluation without providing a hardware-matched sample-efficiency or wall-clock ranking. A future experiment should cross demonstration fractions with capped online budgets, report return-versus-interaction curves and AUC under common action-chunk, primitive-action, wall-clock, hardware, and optimizer-update budgets, and compare full fine-tuning with residual or parameter-efficient adaptation [92]. Interaction focused on states identified by a validated consequence model is another testable lever, but divergence alone is not yet such a model.

7.3. Safety Constraints During RL Fine-Tuning

Choi et al. [8] note that aggressively reward-maximizing policies can overfit the source environment through unsafe behavior. Our additional evaluation gives two concrete reliability views: under stochastic sampling, mean time-until-falling rises from 419 steps for BC to 901 for DPPO, and under normalized observation noise DPPO retains a survival advantage through $\sigma =0.20$ before both policies converge near 187 steps at $\sigma =0.30$. These results show robustness in one simulator; they do not measure collisions, constraint violations, force limits, or recovery on hardware. A concrete safe-DPPO protocol should augment reward with a separately logged constraint cost, optimize a Lagrangian or constrained-PPO objective [111], and project executed actions through a control-barrier-function safety layer where a dynamics model is available [112]. Human intervention and a recovery policy should be treated as first-class outputs, with violation rate, intervention burden, recovery success, and performance under reward misspecification reported alongside return. Diffusion-specific SafeDiffuser [113] and LTLDoG [114] provide relevant constraint constructions, but integrating them into model-free DPPO fine-tuning remains open.

7.4. RL Fine-Tuning of Large VLA Models

Reinforced VLA fine-tuning should now be viewed as an emerging demonstrated capability rather than an entirely open possibility: ConRFT reports simulated and real-robot manipulation with a consistency-policy interface [33], while EXPO-FT [115] and RL Token [116] pursue more sample-efficient online adaptation interfaces. The unresolved questions are narrower and measurable. Future studies should report online interactions and resets per successful adaptation, human-intervention burden, constraint violations, reward-model error, retention on held-out pretraining tasks, cross-embodiment transfer, and language-grounding consistency before and after fine-tuning. Parameter-efficient updates may reduce compute, but catastrophic forgetting and reward-driven loss of instruction semantics require explicit retention suites rather than task return alone. Flow-matching VLA heads also need objectives compatible with their generator, for which FPO [23] is a relevant starting point.

7.5. Acceleration Without Sacrificing Multimodality

Consistency distillation [25,26] and flow matching [64] substantially reduce NFE, but preserving complex action distributions remains an open concern. In our pure-NFE DPPO sweep, reducing deterministic DDIM from 20 to 10 and 5 evaluations lowers A100 per-chunk latency from 30.97 to 15.51 and 8.95 ms while changing the normalized-score point estimate from 86.2 to 86.1 and 84.8. The corresponding paired return intervals include zero and are too wide to establish equivalence. At two evaluations, latency is 3.85 ms but the normalized score is 51.1, and the paired loss is statistically resolved. This result replaces the earlier confounded configuration comparison, but it still does not establish a multimodality frontier: no distributional mode-coverage metric, consistency-policy baseline, or adaptive-inference baseline is run. The observed action-difference distribution provides only a hypothesis for adaptive denoising such as D3P [13]; the null matched-intervention result shows that a consequence model must be validated before computation is allocated by the present divergence score.

7.6. Reward Specification, Hacking, and Alignment

FDPP [7] shows that KL anchoring is essential to preserve task competence during preference-based fine-tuning. As policies scale and rewards become more abstract—natural-language reward functions, or video-derived rewards [99]—the reward-hacking failure mode will only sharpen. We anticipate cross-pollination with the language-model RLHF literature [95]: DPO-style direct preference optimization adapted to diffusion policies, constitutional or rule-based reward shaping, and uncertainty-aware reward models that abstain rather than mislead.

7.7. Multi-Agent, Hierarchical, and Action-Chunked DPRL

Several structural extensions remain largely unexplored. Because diffusion policies naturally represent multi-modal joint action distributions, they suit multi-agent coordination: MADiff [117] takes a first step for offline multi-agent RL, but an online counterpart is open. Hierarchical schemes that pair a high-level diffusion sub-goal generator with a low-level controller [118,119] offer a natural decomposition of long-horizon tasks, though RL fine-tuning of such hierarchies demands careful credit assignment. And at the level of temporal granularity, Q-Chunking [106] provides an initial RL formulation for chunked actions whose integration with diffusion-specific machinery such as DPPO is a natural next step.

7.8. Theoretical Foundations

The two-layer MDP of DPPO [6] and the two-layer POMDP of D3P [13] hint at a unifying theoretical scaffold for DP + RL. Open questions include convergence guarantees for policy gradient on the two-layer MDP, the relationship between intermediate-latent exploration and standard exploration–exploitation bounds, and sample-complexity analyses that treat the denoising-chain depth K as an effective horizon. A rigorous account would both inform algorithm design and help explain why large state-conditioned action differences, such as those measured in Section 6.4, arise.

8. Conclusions

This paper combines an auditable structured scoping review with a focused empirical study. Four documented routes produce 178 records, 162 unique candidates, and an 84-study evidence map: 41 Direct-RL, 32 Adj-Robot, eight Alt-Gen, and three Non-Robot studies. Hierarchical eligibility rules separate evidential role from technical similarity, while the five-axis taxonomy distinguishes initialization/data, interaction regime, optimized object, credit assignment, and inference generator across four cross-cutting themes.

The empirical study reproduces DP-versus-DPPO on state-based MuJoCo locomotion. In the fixed-final protocol, five DPPO fine-tuning-seed means obtain $2808.8\pm 107.6$ versus $1547.5\pm 23.1$ for run-mapped BC; the paired gain is $1261.2\pm 125.5$ (Student-t 95% CI $[1105.3,1417.1]$). All gains are positive, but two reused BC files mean that BC retraining variability is not estimated. Gaussian PPO also improves over its fixed BC policy (normalized $82.5\pm 6.8$ versus 54.1), demonstrating an RL gain outside diffusion without resolving a generator ranking. A five-seed BC-initialized DACER adaptation with full BPTT yields $746.2\pm 855.3$ versus 1413.2 for fixed BC (difference 95% CI $[-1729.0,395.1]$), evidencing seed sensitivity rather than reliable degradation. Fifty-episode reliability rises from 419 to 901 mean steps, and large action differences recur across seed-conditioned episodes; however, the matched intervention CI $[-64.4,108.5]$ does not validate causal criticality. In the controlled A100 sweep, NFE 10/5 reduce latency without a resolved return change but do not establish equivalence; NFE 2 cuts latency from 30.97 to 3.85 ms while lowering normalized score from 86.2 to 51.1 (paired $\Delta R$ 95% CI $[-1245.0,-1039.4]$).

These are configuration-level results, not universal DPPO superiority or validation of the full taxonomy. The empirical contribution contains no visual encoder, contact-rich manipulation, camera shift, cross-embodiment test, or real-robot loop. Whether the observed fine-tuning gain, divergence structure, BPTT sensitivity, reliability, and quality–latency trade-off persist in visuomotor manipulation remains open; so do safe and sample-efficient VLA adaptation, reward verification, catastrophic-forgetting control, and preservation of language grounding.